Issue when replicating data from Redis cluster to Redis cluster

Hi,

I am using rgsync to replicate data from Redis to Redis.
I have developed a new Redis connector and using the same.

I am able to successfully replicate data from:

  1. Standalone Redis to standalone Redis
  2. Standalone Redis to Redis cluster

But replication from
3. Redis cluster to Redis cluster [FAIL]

In replication from cluster to cluster data from only one node of Redis cluster is replicating.
I had used gears-cli to execute python script on all primary Redis cluster master nodes and registration was successful on each node.
/home/ratnesh1.tiwari/redis605_Gears_Cluster_Setup/python3_1.0.1/bin/gears-cli run --host 10.32.129.77 --port 7001 example-redis.py OK /home/ratnesh1.tiwari/redis605_Gears_Cluster_Setup/python3_1.0.1/bin/gears-cli run --host 10.32.129.73 --port 7001 example-redis.py
OK
/home/ratnesh1.tiwari/redis605_Gears_Cluster_Setup/python3_1.0.1/bin/gears-cli run --host 10.32.129.66 --port 7001 example-redis.py OK /home/ratnesh1.tiwari/redis605_Gears_Cluster_Setup/python3_1.0.1/bin/gears-cli run --host 10.32.129.67 --port 7001 example-redis.py
OK

I am using Redis cluster python client redis-py-cluster

List of other packages and version:

Package Version


click 6.7
gears-cli 1.1.0
redis 3.5.3
redis-py-cluster 2.1.0
rgsync 1.0.1
SQLAlchemy 1.3.18
wheel 0.34.2

I am not able to attach master nodes log. But could share on mail if needed.

Thanks

@ratnesh-tiwari can you share the output of rg.dumpregistrations command of each shard to make sure redistration reach all the shards. Notice that on oss cluster you need to execute rg.refreshcluster command on each shard to make sure gears is aware of the cluster topology.

@meirsh

Thanks
I was not aware of the RG.REFRESHCLUTSRE. Data is replicating successfully after using it.

Also While restarting node I am getting error
“The RDB file contains AUX module data I can’t load: no matching module ‘GEARS_DT0’”

While previously when module was loaded MODULE LIST output was:
10.32.129.73:7001> module list

    1. “name”
    2. “rg”
    3. “ver”
    4. (integer) 10001
      10.32.129.73:7001>

Are you sure that after the restart the RedisGears module is still loaded?

Output of module list is when first time I have loaded module before I am trying to restart redis server.

Steps are as following:

  1. Start server
  2. Load module
  3. Check output of module list
  4. Kill master to promote slave to master
  5. Try to restart killed server [failed with error: “The RDB file contains AUX module data I can’t load: no matching module ‘GEARS_DT0’”]

You need to make sure the module is loaded when the server starts, otherwise it will not be able to read the RDB file (the module contains a code that tells Redis how to read this part of the rdb). Try to load the module when you start the server (also on restart) using --loadmodule parameter (https://oss.redislabs.com/redisgears/quickstart.html#loading) and it should work.

Command to start server using config file and module

$ …/redis-server redis.conf --loadmodule /home/ratnesh1.tiwari/RedisGears/bin/linux-x64-release/redisgears.so PythonInstallationDir /home/ratnesh1.tiwari/RedisGears/bin/linux-x64-release

===From log file===

9521:M 11 Aug 2020 13:20:25.293 * Module ‘rg’ loaded from /home/ratnesh1.tiwari/RedisGears/bin/linux-x64-release/redisgears.so
9521:M 11 Aug 2020 13:20:25.294 * Loading RDB produced by version 6.0.5
9521:M 11 Aug 2020 13:20:25.294 * RDB age 14767 seconds
9521:M 11 Aug 2020 13:20:25.294 * RDB memory usage when created 27.73 Mb
9521:M 11 Aug 2020 13:20:25.339 * DB loaded from disk: 0.046 seconds
9521:M 11 Aug 2020 13:20:25.340 * Ready to accept connections
9521:M 11 Aug 2020 13:20:25.340 # Failed on XINFO command : got null reply

=== REDIS BUG REPORT START: Cut & paste starting from here ===
9521:M 11 Aug 2020 13:20:25.340 # === ASSERTION FAILED ===
9521:M 11 Aug 2020 13:20:25.340 # ==> src/readers/streams_reader.c:159 ‘ret’ is not true
9521:M 11 Aug 2020 13:20:25.340 # (forcing SIGSEGV to print the bug report.)
9521:M 11 Aug 2020 13:20:25.340 # Redis 6.0.5 crashed by signal: 11
9521:M 11 Aug 2020 13:20:25.340 # Crashed running the instruction at: 0x472c9a
9521:M 11 Aug 2020 13:20:25.340 # Accessing address: 0xffffffffffffffff
9521:M 11 Aug 2020 13:20:25.340 # Failed assertion: ret (src/readers/streams_reader.c:159)

------ STACK TRACE ------

@meirsh

In above case, am I missing anything?

Now you are fine, looks like a bug in RedisGears, is it possible to share the rdb file with me so it will be easier to reproduce and find the issue. Also if you can remind me the RedisGears version you are using?
Thanks.

I am using
RG version 1.0.1.
Following is from version.h file of RG
#define REDISGEARS_VERSION_MAJOR 1
#define REDISGEARS_VERSION_MINOR 0
#define REDISGEARS_VERSION_PATCH 1

Redis version 6.0.5

Link to rdb file
https://github.com/ratnesh-tiwari/RG-101-ERROR/blob/master/dump.rdb

Thanks, will check and get back to you.

So I am getting this error:

2134:M 11 Aug 2020 11:49:58.507 # Could not deserialize flat execution, error=‘Error occured when deserialized a python callback, error=Error type: <class ‘AttributeError’>, Value: Can’t get attribute ‘RedisClusterConnection’ on <module ‘rgsync.Connectors.redis_connector’ from ‘/var/opt/redislabs/modules/rg/.venv-/lib/python3.7/site-packages/rgsync/Connectors/redis_connector.py’>’

I did install your branch of RGSYNC but I guess you have some local changes that prevent me from loading it. Can you please push those changes?

For the next time if you specify your RGSYNC branch on the RG.PYEXECUTE command on the REQUIREMENTS section (https://oss.redislabs.com/redisgears/commands.html#rgpyexecute) then the current state of the code will also be saved to the rdb and this will not be needed. Also RedisGears will take care to install it for you on the virtual environment of all the shards.

I have pushed changes to


branch : redis_connector

Following files are modified:
examples/redis/example-redis.py
requirements.txt
rgsync/Connectors/init.py
rgsync/Connectors/redis_connector.py

I have not updated documentation yet. I will do it later.
From next time I will take care of installation of REQUIREMENTS.

I was able to start the redis successfully, no crashes. Any command you sent to trigger this issue?

Are you using the same command that I have used?

redis-server redis.conf --loadmodule /path/to/redisgears.so PythonInstallationDir /path/to/python_dir

Is it possible that there might be some configuration mistake in my config file ?

Yes I am running the same without the redis.conf file. I do not think it is related to the configuration file but if you can share it I can try run with this configuration file also. In addition, if you can enable memory dump I can try to analyze it:

ulimit -c unlimited
redis-server redis.conf --loadmodule /path/to/redisgears.so PythonInstallationDir /path/to/python_dir

This should create a core file that if you send it to me I can try to analyze with gdb.

I will check core and try myself first and if unable to fix I will share with you.

1 Like

@meirsh

At present firstly I install rgsync and dependency packages offline and then I am running gears-cli without passing ‘–requirements requirements.txt’ argument.
When I am paassing ‘–requirements requirements.txt’ it fails to install package in requirements.txt because plaform is not connected to internet.

How to install packages in requirements.txt offlne using gears-cli?

I want equivalent of following in gears-cli:

pip3 install rgsync --find-links=/home/ratnesh1.tiwari/pypi-org/

Here directory /home/ratnesh1.tiwari/pypi-org/ contains:
package-ver.tar.gz
package-ver-py3-none-any.whl

files for package being installed.

I tried to pass requirements.txt as extra arg but it filed.
[ratnesh1.tiwari@cm2 redis605_Gears_Cluster_Setup] /home/ratnesh1.tiwari/RedisGears/bin/linux-x64-release/python3_1.0.1/bin/gears-cli run --host 10.32.129.73 --port 7001 example-redis.py --requirements requirements.txt '--find-links=/home/ratnesh1.tiwari/pypi-org/' Error: no such option: --find-links [ratnesh1.tiwari@cm2 redis605_Gears_Cluster_Setup]

If your environment is not connected to the internet you can install it in an outside environment that does have internet using ‘gears-cli install-requirements’. Then you can export this requirement using ’
gears-cli export-requirements’, this will create a zip file that contains everything needed for the requirement to work properly. You can take this zip to your offline environment and import it using ’
gears-cli import-requirements’. After importing the requirement you should still give the --requirement option but gears will identify that it already has the requirement and it will not try to re-install it.