Bulk loading keys to Redis using Python

date

Apr 16, 2023

slug

bulk-loading-redis

author

status

Public

Bulk Loading Keys to Redis using Python

If you are working with Redis, you might know that Redis is an in-memory data structure store that can be used as a database, cache, and message broker. Redis is known for its high performance and scalability, making it a popular choice among developers.

In this blog post, we will learn how to bulk load keys to Redis using Python. Bulk loading keys can be useful when you have a large number of keys that you want to add to Redis. It can also be faster than adding each key one by one.

Prerequisites

Before we begin, make sure you have Redis installed on your system. You can download Redis from the official website. You will also need to have the Redis Python client installed. You can install it using pip:

pip install redis

Redis pipeline

To bulk load keys to Redis using Python, we will use the pipeline method provided by the Redis Python client. The pipeline method allows us to execute multiple commands in a single request, which can be faster than sending each command separately.

Here is an example of how to use the pipeline method to bulk load keys:

import redis

# Connect to Redis
r = redis.Redis(host='localhost', port=6379, db=0)

# Create a pipeline
pipe = r.pipeline()

# Add keys to the pipeline
for i in range(10000):
    pipe.set(f'key_{i}', f'value_{i}')

# Execute the pipeline
pipe.execute()

In the example above, we first connect to Redis using the redis.Redis method. We then create a pipeline using the pipeline method. Next, we use a for loop to add 10,000 keys to the pipeline using the set method. Finally, we execute the pipeline using the execute method.

Here pipeline method has to wait for the reply from the previous insertion. For millions of keys this method will not scale well.

RESP Protocol

For scaling to millions of keys we will need to feed a text file encoded in the RESP protocol to the Redis-CLI. The RESP protocol is human-readable and can be used to communicate with Redis server for sending multiple commands at the same time. This is how the text file will look after encoding Redis commands in REST

*3
$3
SET
$3
key
$5
value
*2
$3
GET
$3
key

.txt file with commands encoded in RESP

We can generate this protocol file in Python 3.8

def encode_resp(command):
		resp_string = f"*{len(command)}\n"
		for word in command:
				resp_string += f"${len(word)}\n{word}\n"
		return resp_string

commands = [['SET', 'key', 'value'], ['GET', 'key']]
resp_file = open('resp_encoded_commands.txt', 'w')
for command in comands:
		resp_file.write(encode_resp(command))
resp_file.close()

Finally, we need to feed this file to the redis-cli using pipe mode.

cat resp_encoded_commands.txt  | sed 's/$/\r/g' | redis-cli --pipe

This command should execute in seconds for a large number of keys.

Conclusion

Here we explore how to update a large number of keys on your Redis server in a matter of seconds using python. This can be really useful for mass insertions/updations of a large number of key value pairs that need to be highly available with low latency.

Thanks for reading!!😊

References

Gojek Tech:

How We Pushed a Million Keys to Redis in Seconds - 3 min read

Dealing with a lot of keys? Redis’ Pipe Mode is your friend.

https://www.gojek.io/blog/how-we-pushed-a-million-keys-to-redis-in-seconds-2

How We Pushed a Million Keys to Redis in Seconds - 3 min read

Redis Docs

Bulk loading

Writing data in bulk using the Redis protocol

https://redis.io/docs/manual/patterns/bulk-loading/