Redis Clustering

This article is part of our Academy Course titled Redis a NoSQL key-value store.

This is a crash course on Redis. You will learn how to install Redis and start up the server. Additionally, you will mess around with the Redis command line. More advanced topics follow, such as replication, sharding and clustering, while the integration of Redis with Spring Data is also explained. Check it out here!

1. Introduction

This final part of the tutorial is dedicated to newest and coolest but still experimental (not ready for production) feature of Redis – clustering. The material of this part is heavily based on Redis documentation sections, http://redis.io/topics/cluster-tutorial and http://redis.io/topics/cluster-spec. Redis clustering (or simply Redis Cluster) is a distributed Redis deployment which is aiming to tackle the following principal goals:

• ability to automatically split the dataset among multiple nodes
• ability to provide high performance and linear scalability
• ability to retain all the writes originating from clients connected with the majority of the nodes (write safety/consistency)
• ability to survive the network partitions where the majority of the master nodes are reachable and there is at least a reachable slave for every master node that is no longer reachable (availability)

Redis Cluster is the alternative (but much more advanced) solution to data sharding (partitioning) which we have seen in part 4 of this tutorial, Redis Sharding (but instead of using third-party tools, all the functionality is provided by Redis itself with the additional configuration). In order to be highly available, Redis Cluster is also heavily relies on master – slave replication which we have seen in part 3 of the tutorial, Redis Replication.

2. Redis Clustering Limitations

First and foremost, all features related to Redis Cluster are in experimental mode, not ready for production usage just yet.

Building any highly available distributed system is very hard but Redis is trying to make it possible. There are a couple of limitations to be aware of and trade-offs to make, some of them we have already mentioned but it worth to repeat them here as well.

Firstly, commands dealing with multiple keys are not supported by the Redis Cluster (SINTER, SUNION, …). Such functionality would require moving data between Redis nodes which is going to make Redis Cluster not able to provide acceptable performances and predictable behavior under load. In general all the operations where keys are not available in the Redis node processing the command are not implemented.

Secondly, Redis Cluster does not support multiple databases like the standalone version of Redis. There is only one database 0 and SELECT is not allowed.

Thirdly, the nodes in the Redis Cluster do not proxy commands to the right nodes storing a given key but instead they redirect clients to the right nodes serving a given range of the key space (a hybrid form of so-called query routing). Eventually, clients obtain an up-to-date representation of the cluster topology and, knowing which node serves which subset of keys, may directly contact the right nodes in order to send a given command (effectively falling back to client side partitioning).

3. Sharding (Partitioning) Scheme

As we already know from part 4, Redis Sharding, there are a couple of data sharding (partitioning) schemes used for splitting the data with consistent hashing being the most advanced and widely used. Redis Cluster does not use consistent hashing but a different form of data split with every key being a part of what is called a hash slot.

There are 16384 hash slots in Redis Cluster and to compute what is the hash slot of a given key, the CRC16 function (http://en.wikipedia.org/wiki/Cyclic_redundancy_check) of the that key is calculated, and then the modulo of 16384 is applied to its result.

Every node in a Redis Cluster is responsible of a subset of the hash slots. As an example, let’s consider a cluster with four Redis nodes #1, #2, #3 and #4. That may give us the following hash slots distribution:

• Redis node #1 contains hash slots from 0 to 4096
• Redis node #2 contains hash slots from 4097 to 8192
• Redis node #3 contains hash slots from 8193 to 12288
• Redis node #4 contains hash slots from 12289 to 16383

Such a sharding (partitioning) scheme allows to change the topology (add and remove nodes) of the cluster easily. For example, if there is a need to add a new node #5, some hash slots from nodes #1, #2, #3 and #4 should be moved to node #5. Similarly if there is need to remove node #3 from the cluster, the hash slots served by node #3 should be moved to nodes #1 and #2. When the node #3 becomes empty, it could be removed from the cluster permanently.

Now the best part: because moving the hash slots from the one node to another does not require stopping the ongoing operations, adding and removing nodes (or changing the percentage of hash slots hold by the nodes) does not require any downtime.

Later in this tutorial we will get back to this example and build the actual cluster with three Redis master nodes (each backed by one slave). While Redis Cluster is operating, we will add and remove some nodes to see how hash slots could be redistributed live.

3.1 Keys hash tags

Very interesting feature supported by Redis sharding (partitioning) scheme is so called key hash tags. Hash tags are a technique to ensure that two (or more) keys are allocated in the same hash slot.

In order to support hash tags, the hash slot is computed in a different way. If the key contains a “{…}” pattern then only the substring between “{“ and “}” is hashed in order to obtain the hash slot (in case of the multiple occurrences of “{“ or” “}” in the key name, some rules are taken place and are described at http://redis.io/topics/cluster-spec).

Twemproxy (nutcracker) which we played with in part 4, Redis Sharding, also allows to configure hash tags to be used for keys hashing, following the same set of rules.

4. Redis Clustering in Nutshell

In the Redis Cluster all the nodes are holding some part of global keyset (shard or partition). Additionally, each node holds the state of the cluster including hash slots mappings so to redirect clients to right node for the given key. All nodes in the Redis Cluster are also able to auto-discover other nodes, detect nodes which are unreachable or don’t work as expected, and perform slave nodes to master election when required.

As for implementation details described at http://redis.io/topics/cluster-spec, all nodes in the cluster are connected using a TCP with a binary protocol (the cluster bus) so that every node is connected to every other node in the cluster using the cluster bus (it means that in the Redis Cluster of N nodes, every node has N – 1 outgoing TCP connections, and N – 1 incoming TCP connections). These TCP connections are kept alive all the time. Nodes use a gossip protocol (http://en.wikipedia.org/wiki/Gossip_protocol) to spread the cluster state, to discover new nodes, to make sure all nodes are working properly, and to propagate publish/subscribe messages across the cluster.

Every node in the Redis Cluster has a unique ID (name). The node ID (name) is the hex representation of a 160 bit random number, obtained the first time a node is started. The node will save its ID (name) in the node configuration file (by default, nodes.conf), and will use the same ID (name) forever (or at least as long as the node configuration file is not removed).

The node ID (name) is used to identify every node across the whole Redis Cluster. It is possible for a given node to change its IP address without any need to also change its ID (name). The cluster is also able to detect the change in IP or/and port and broadcast this information using the gossip protocol running over the cluster bus. Additionally, every node has some other associated with it information that all the other nodes in the Redis Cluster should know:

• the IP address and TCP port where the node is located
• a set of flags (master, slave, …)
• a set of hash slots served by the node (see please Sharding (Partitioning) Scheme)
• last time the ping packet was sent using the cluster bus
• last time the pong packet was received in reply
• the time at which the node was flagged as failing
• the number of slaves of this node
• the master node ID (name), if this node is a slave (or zeroes if it is a master node)

Some of this information is available using the CLUSTER NODES command (see please section Redis Cluster Commands).

5. Consistency, availability and scalability

Redis Cluster is a distributed system. Good distributed systems are scalable and are able to provide better performance at scale. But still, in any distributed system any component can fail at any time and the system should provide some guarantees in case such failures happen (especially if it is a data store). In this section we will just briefly look through some high-level trade-offs Redis makes with respect to consistency, availability and scalability. Much deeper insights and details could be found at http://redis.io/topics/cluster-spec and http://redis.io/topics/cluster-tutorial. Please note that Redis Cluster evolves very rapidly and some guarantees discussed in this section may not hold true anymore.

5.1 Consistency

Redis Cluster is not able to guarantee strong consistency but it tries hard to retain all the writes that are performed by the clients. Unfortunately, it is not always possible. Because Redis Cluster uses asynchronous replication between master and slave nodes, there are always time windows when it is possible to lose writes during network partitions. If the master node dies without the write reaching the slave nodes, the write is lost forever (in case the master is unreachable for a long period of time and one of its slaves is promoted to become a master).

5.2 Availability

Redis Cluster is not available in the minority side of the network partition. In the majority side of the network partition assuming that there are at least the majority of masters and a slave for every unreachable master, the Redis cluster is still available. It means that Redis Cluster can survive failures of a few nodes in the cluster, but it cannot survive the large network partitions. For the example, let us consider the Redis cluster with N master nodes (M1, M2, M3) and N slave nodes (S1, S2, S3, with every master having exactly one slave node). If any single master node becomes unreachable (let us assume it is M2) because of network partition, the majority of the cluster will still remain available (and S2 will be promoted to become a master). Later, if any other master or slave node becomes unreachable (except S2) the cluster will still be available. However note that if node S2 fails for some reason, Redis Cluster is not able to continue to operate anymore (because both master M2 and slave S2 are not available).

5.3 Scalability

We already know from section Sharding (Partitioning) Scheme that Redis Cluster nodes do not forward commands to the right node for a given key but redirect clients instead. The clients eventually obtain a complete mapping which nodes serve which subset of keys and may directly contact the right nodes. Because of that, Redis Cluster is able to scale linearly (adding more nodes leads to better performance) as all supported operations are handled exactly in the same way as in the case of a single Redis instance with no additional overhead.

6. Installing Redis with Clustering Support

Redis Cluster is currently available in unstable releases only. The latest unstable release at the moment of writing is 3.0.0-beta1 and can be downloaded from http://redis.io/download. Please note that only Linux distributions are provided, Windows port is not available yet.

The installation of Redis distribution with the clustering is no different from the regular Redis installation described in the part 1 of the tutorial, Redis Installation and follows the same steps:

wget https://github.com/antirez/redis/archive/3.0.0-beta1.tar.gz
tar xf 3.0.0-beta1.tar.gz
cd redis-3.0.0-beta1/
make
make test
sudo make install

After the last step, the usual Redis executables will be installed in /usr/local/bin folder.

7. Configuring Redis Cluster

Redis Cluster cannot be created using normal Redis instances and regular configuration. Instead, a couple of empty Redis instances should be running in special cluster mode. In order to do that, the instance should be run with cluster-specific configuration (the cluster-enabled directive should be set to “yes” in the configuration file) so to enable the cluster-specific features and commands.

The minimal set of settings required to run some Redis instance with cluster mode support includes the following ones.

• cluster-enabled yes (default: no)
  Enables Redis cluster mode for this instance
• cluster-config-file nodes.conf (default: nodes.conf)
  The path to a file where the configuration of this instance is stored. This file should be never touched and is simply generated at startup by the Redis Cluster instances and updated every time it is needed (see please section Redis Clustering in Nutshell)
• cluster-node-timeout 5000
  The timeout (in milliseconds) after which a non-responding instance is considered to be failing by the failure detection algorithm. As we mention in section Sharding (Partitioning) Scheme, we are going to configure and run a live Redis cluster with three Redis master nodes (master1, master2, master3) each backed by Redis slave node (slave1, slave2, slave3) as shown on a picture below.

Along the way we will explore most of the Redis cluster features but before that let us start with configuration for masters and slaves. To keep the configuration simple enough, we will start with bare minimum settings required for cluster to operate normally.

7.1 Configure Redis Cluster master nodes

The minimal configuration for Redis master nodes looks like that:

• Redis node master1 (redis-master1.conf)

      port 6379
      cluster-enabled yes
      cluster-config-file nodes.conf
      cluster-node-timeout 5000
      appendonly yes
  

• Redis node master2 (redis-master2.conf)

      port 6380
      cluster-enabled yes
      cluster-config-file nodes.conf
      cluster-node-timeout 5000
      appendonly yes
  

• Redis node master3 (redis-master3.conf)

      port 6381
      cluster-enabled yes
      cluster-config-file nodes.conf
      cluster-node-timeout 5000
      appendonly yes
  

Having configuration files ready, we can start our Redis master nodes one by one providing the configuration as command line parameter.

• redis-server redis-master1.conf

  

• redis-server redis-master2.conf

  

• redis-server redis-master3.conf

  

Comparing to console output of the standalone Redis instances, there are a couple of well-noticeable differences:

• upon start, each node generates its unique ID (name) as we discussed in Redis Clustering in Nutshell, please note that this value will be generated only at first run and then reused
• every instance is running in cluster mode
• also, for every running instance there is a file nodes.conf created with current node ID (name) and some additional information

At this moment we have three Redis master nodes running in a cluster mode but actually not yet forming a cluster (every Redis master node sees only itself but not others). To verify that, we can run CLUSTER NODES command (see please Redis Cluster Commands section) on each instance individually and observe it is really the case.

In order to form a cluster, the Redis nodes (running in cluster mode) should be connected together by CLUSTER MEET command (see please Redis Cluster Commands section). Unfortunately, the command accepts IP addresses only but not hostnames. In our topology master1 has IP address 192.168.1.105, master2 has 192.168.2.105 and master3 has 192.168.3.105. Having IP addresses, let us issue the command against master1 node.

Now, if we rerun CLUSTER NODES command, the result should be quite different.

The output of the CLUSTER NODES command is looking a bit cryptic and requires some explanation what each column means.

The last column, Hash Slots served, is not set in the output and there is a reason why: we have not assigned hash slot to the master nodes yet and that is what we are going to do now. Hash slots could be assigned to the nodes by using CLUSTER ADDSLOTS (see please Redis Cluster Commands) command on a particular cluster node (and unassigned using CLUSTER DELSLOTS respectively). Unfortunately, it is not possible to assign hash slot ranges (like 0-5400) but instead each hash slot (out of total 16384) should be assigned individually. One of the simplest ways to overcome this limitation is to use a bit of shell scripting. As we have only three Redis master nodes in the cluster, the range of 16384 hash slots could be split like this:

• Redis node master1 contains hash slots 0 – 5400

  for slot in {0..5400}; do redis-cli -h master1 -p 6379 CLUSTER ADDSLOTS $slot; done;
  

• Redis node master2 contains hash slots 5401 – 10800

  for slot in {5400..10800}; do redis-cli -h master2 -p 6380 CLUSTER ADDSLOTS $slot; done;
  

• Redis node master3 contains hash slots 10801 – 16383

  for slot in {10801..16383}; do redis-cli -h master3 -p 6381 CLUSTER ADDSLOTS $slot; done;
  

If we rerun CLUSTER NODES command once more, the last column will be filled with appropriate hash slots served by each master node (exactly matching the hash slot ranges we have assigned to the nodes before).

7.2 Configure Redis Cluster slave nodes and replication

To make our Redis cluster complete, we need to add to each running Redis master node exactly one slave node. Though part 3 of this tutorial, Redis Replication, covers replication configuration well enough, the Redis cluster does it differently. From the beginning, the procedure to run and configure slaves is no different from the master one (the only difference is the port number).

• Redis node slave1 (redis- slave1.conf)

      port 7379
      cluster-enabled yes
      cluster-config-file nodes.conf
      cluster-node-timeout 5000
      appendonly yes
  

• Redis node slave2 (redis-slave2.conf)

      port 7380
      cluster-enabled yes
      cluster-config-file nodes.conf
      cluster-node-timeout 5000
      appendonly yes
  

• Redis node slave3 (redis-slave3.conf)

      port 7381
      cluster-enabled yes
      cluster-config-file nodes.conf
      cluster-node-timeout 5000
      appendonly yes
  

Let us start all three slave instances followed by CLUSTER MEET command so every node will join our running Redis cluster.

redis-server redis-slave1.conf
redis-server redis-slave2.conf
redis-server redis-slave3.conf

As CLUSTER MEET requires IP address, our slave1 has IP address 192.168.4.105, slave2 has 192.168.5.105 and slave3 has 192.168.6.105.

redis-cli -h master1 -p 6379 CLUSTER MEET 192.168.4.105 7379
redis-cli -h master1 -p 6379 CLUSTER MEET 192.168.5.105 7380
redis-cli -h master1 -p 6379 CLUSTER MEET 192.168.6.105 7381

As always, using CLUSTER NODES command we can see the current nodes in the Redis cluster (with total of six). The output shows all nodes as being masters.

To configure the replication, the new CLUSTER REPLICATE command should be executed on each Redis slave by providing master Node ID (name). Following tables summarize all the required pieces for replication together (by consulting the result of CLUSTER NODES command output).

At this point our Redis cluster is configured properly and has the topology we have intended to create. The CLUSTER NODES command shows all the slaves connected to masters.

As we can see, all nodes are healthy, connected and have the right roles assigned (masters and slaves).

7.3 Verifying Redis Cluster is working properly

As always in case of Redis, the best way to ensure Redis cluster works as expected is to issue some commands using redis-cli. Please note, because nodes in the cluster do not proxy commands but redirect clients instead (see please Sharding (Partitioning) Scheme), the client must support such a protocol and that is why redis-cli should be run with -c command line option (with cluster support):

redis-cli -h master1 -p 6379 -c

Let us try to set stored keys (using SET command) and query them afterwards (using GET command). Because we have distributed hash slots among three nodes, the keys will also be distributed across all those nodes. The first key with name some-key is stored on the master1 node itself, which we are connected to.

But if we try to store the key with name some-another-key, the interesting thing is going to happen: the redis-cli is telling us that the value will be stored on the node with IP address 192.168.3.105 (master3) which holds the hash slot this key belongs to.

Please notice that after the command execution the redis-cli is automatically redirected to the node 192.168.3.105 (master3). Once we are on cluster node 192.168.3.105 (master3), we can verify that hash slot really contains the key some-another-key by issuing CLUSTER GETKEYSINSLOT command.

We also may verify that Redis slave node slave3 has replicated the key some-another-key from the master (master3) and returns its value.

7.4 Adding and removing nodes to the running Redis Cluster

We already mentioned in section Sharding (Partitioning) Scheme that Redis cluster could be reconfigured without downtime and usually involves hash slots migration. Let us add yet another master node master4 (with IP address 192.168.7.105) to the cluster and migrate the slot 15929 from the node master3 to master4 (it is the hash slot containing the key some-another-key). Her is Redis node master4 (redis- master4.conf) configuration:

    port 6384
    cluster-enabled yes
    cluster-config-file nodes.conf
    cluster-node-timeout 5000
    appendonly yes

    redis-server redis-master4.conf

   redis-cli -h master1 -p 6379 CLUSTER MEET 192.168.7.105 6384

The migrating hash slots procedure includes following stages:

• On the cluster node which owns the specific hash slot (master3), the command CLUSTER SETSLOT slot MIGRATING should be executed, where is the Node ID of the new node master4 (which is d8095be33a2b9d06affcb5583f7150b1341f4c96).

  redis-cli -h master3 -p 6381 CLUSTER SETSLOT 15929 MIGRATING
          d8095be33a2b9d06affcb5583f7150b1341f4c96
  

     When a slot is marked as MIGRATING, the node will accept all the requests for queries that are about this hash slot, but only if the given key exists, otherwise the query is forwarded to the node that is target of the migration.

• On the cluster node which should become a new owner of the specific hash slot (master4), the command CLUSTER SETSLOT slot IMPORTING, where is the Node ID of current owner master3 (which is d8a2ae6221624212b76d9cf6c1483452e3c26117).

  redis-cli -h master4 -p 6384 CLUSTER SETSLOT 15929 IMPORTING
          d8a2ae6221624212b76d9cf6c1483452e3c26117
  

• At this point, all the keys from hash slot should be migrated using MIGRATE command (see please http://redis.io/commands/migrate) from the current owner master3 to new owner master4. As we have only one key, it is easy.

  redis-cli -h master3 -p 6381 MIGRATE master4 6384 some-another-key 0 0
  

• Lastly, when hash slot becomes empty (which could be verified by issuing CLUSTER GETKEYSINSLOT command), it could be assigned to the new node (master4).

  redis-cli -h master3 -p 6381 CLUSTER SETSLOT 15929 NODE
      d8095be33a2b9d06affcb5583f7150b1341f4c96
  

Though it is quite useful to have an understanding what is going on in details, doing such a procedure manually is hard and error-prone. But Redis Cluster package provides a handy utility called redis-trib and located in src folder of the Redis distribution. It is written in Ruby and may be very helpful by simplifying the management of your Redis cluster (see please http://redis.io/topics/cluster-tutorial for more details).

8. Redis Cluster Commands

Redis Cluster adds an additional set of commands solely dedicated to cluster management, monitoring and configuration. Those commands have not been covered in part 2 of the tutorial, Redis Commands, as they are not available yet in stable releases. Also, there is not enough documentation on them available at Redis web site but at least we can describe each command briefly (many of them you have already seen in action).

9. Redis Sentinel

Another great but still experimental feature of Redis is Redis Sentinel. It is a system designed to help managing live Redis instances with following goals in mind:

• monitoring: Sentinel constantly checks if your master and slave instances are working as expected
• notification: Sentinel is able to notify if something is wrong with one of the monitored Redis instances
• automatic failover: If some master node is not working as expected, Sentinel can start a failover process where one of its slaves is promoted to master

Redis Sentinel is a very promising feature but it is currently being developed in the unstable branch of the Redis source code. It is not a part of Redis distributions yet.

For more details please look at http://redis.io/topics/sentinel.

10. What’s next

In this section we have covered a very appealing and highly demanded feature of Redis, clustering. Even though it is still in development, the feature is stable enough to start playing with it. In the next, the last part of the tutorial, we will cover the programmatic Java API for accessing Redis in different deployment scenarios.

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you our best selling eBooks for FREE!

1. JPA Mini Book

2. JVM Troubleshooting Guide

3. JUnit Tutorial for Unit Testing

4. Java Annotations Tutorial

5. Java Interview Questions

6. Spring Interview Questions

7. Android UI Design

and many more ....

I agree to the Terms and Privacy Policy

Sign up

Thank you!

We will contact you soon.