Execution block while running python script using gears-cli

Redis modules RedisGears
1

I am using gears-cli to run a python script to connect to redis and run command.
This is simply to learn the use of python script using gears-cli.

I am able to connect to redis but when I am trying to run a command like ping() It gets blocked.

I am using standalone Redis version Redis 6.0.5 64 bit.

Following is command to run gears-cli, python script and pstack trace when execution is blocked are attached.

gears-cli command:
$ /home/ratnesh1.tiwari/RedisGears/bin/linux-x64-release/python3_1.0.1/bin/gears-cli run --host 10.32.129.77 --port 7001 test_write_behind.py

Python script:
$ cat test_write_behind.py
#!/usr/bin/env python

import redis
import time
import argparse

parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter,
description=‘Gears Write Behind test’)

parser.add_argument(’–host’, default=‘localhost’, help=‘Redis host’)
parser.add_argument(’–port’, default=6379, type=int, help=‘Redis port’)
parser.add_argument(’–password’, default=None, help=‘Redis password’)

args = parser.parse_args()

try:
conn = redis.Redis(args.host, args.port, password=args.password, decode_responses=True, socket_connect_timeout=5)
#conn.ping() # this line is causing block
print(‘Connected to redis’)
except:
print(‘Cannot connect to Redis.’)
exit(1)

print(‘Done’)

pstack trace:
…
…
Thread 1 (Thread 0x7f603247ef80 (LWP 49041)):
#0 0x00007f6031553a9b in recv () from /lib64/libpthread.so.0
#1 0x00007f602134b316 in sock_recv_impl (data=, s=0x7f6024f50ec0) at /home/ratnesh1.tiwari/RedisGears/deps/cpython/Modules/socketmodule.c:3255
#2 sock_call_ex (err=0x0, timeout=-1000000000, connect=0, data=, sock_func=, writing=0, s=0x7f6024f50ec0) at /home/ratnesh1.tiwari/RedisGears/deps/cpython/Modules/socketmodule.c:884
#3 sock_call (data=, func=, writing=0, s=0x7f6024f50ec0) at /home/ratnesh1.tiwari/RedisGears/deps/cpython/Modules/socketmodule.c:936
#4 sock_recv_guts (flags=, len=65536, cbuf=0x7f602083fb28 “”, s=0x7f6024f50ec0) at /home/ratnesh1.tiwari/RedisGears/deps/cpython/Modules/socketmodule.c:3287
#5 sock_recv (s=0x7f6024f50ec0, args=) at /home/ratnesh1.tiwari/RedisGears/deps/cpython/Modules/socketmodule.c:3318
#6 0x00007f6027dacda5 in PyCFunction_Call () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#7 0x00007f6027d3e4f2 in _PyEval_EvalFrameDefault () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#8 0x00007f6027e91848 in _PyEval_EvalCodeWithName () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#9 0x00007f6027daa988 in _PyFunction_FastCallKeywords () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#10 0x00007f6027d3eba2 in _PyEval_EvalFrameDefault () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#11 0x00007f6027e91848 in _PyEval_EvalCodeWithName () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#12 0x00007f6027daa988 in _PyFunction_FastCallKeywords () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#13 0x00007f6027d3eb88 in _PyEval_EvalFrameDefault () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#14 0x00007f6027d353f0 in ?? () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#15 0x00007f6027d3eb88 in _PyEval_EvalFrameDefault () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#16 0x00007f6027e91848 in _PyEval_EvalCodeWithName () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#17 0x00007f6027daa988 in _PyFunction_FastCallKeywords () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#18 0x00007f6027d3eb88 in _PyEval_EvalFrameDefault () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#19 0x00007f6027d353f0 in ?? () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#20 0x00007f6027d3eb88 in _PyEval_EvalFrameDefault () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#21 0x00007f6027e91848 in _PyEval_EvalCodeWithName () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#22 0x00007f6027daa767 in _PyFunction_FastCallDict () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#23 0x00007f6027daba3e in _PyObject_Call_Prepend () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#24 0x00007f6027dacf47 in PyObject_Call () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#25 0x00007f6027d3a8e6 in _PyEval_EvalFrameDefault () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#26 0x00007f6027e91848 in _PyEval_EvalCodeWithName () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#27 0x00007f6027daa988 in _PyFunction_FastCallKeywords () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#28 0x00007f6027d3eb88 in _PyEval_EvalFrameDefault () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#29 0x00007f6027d353f0 in ?? () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#30 0x00007f6027d3eb88 in _PyEval_EvalFrameDefault () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#31 0x00007f6027e91848 in _PyEval_EvalCodeWithName () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#32 0x00007f6027e9199d in PyEval_EvalCodeEx () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#33 0x00007f6027e919eb in PyEval_EvalCode () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#34 0x00007f6027ecbea2 in PyRun_StringFlags () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#35 0x00007f6027d659ae in ?? () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#36 0x00007f6027d65bff in RedisGearsPy_Execute () from /home/ratnesh1.tiwari/redisGearSetup/redisgears.so
#37 0x00000000004a3b20 in RedisModuleCommandDispatcher (c=0x7f6030c7c340) at module.c:642
#38 0x000000000043455b in call (c=c@entry=0x7f6030c7c340, flags=flags@entry=15) at server.c:3268
#39 0x0000000000434f2d in processCommand (c=c@entry=0x7f6030c7c340) at server.c:3652
#40 0x0000000000442020 in processCommandAndResetClient (c=c@entry=0x7f6030c7c340) at networking.c:1785
#41 0x00000000004462d7 in processInputBuffer (c=0x7f6030c7c340) at networking.c:1867
#42 0x00000000004be74a in callHandler (handler=, conn=0x7f6030c150c0) at connhelpers.h:79
#43 connSocketEventHandler (el=, fd=, clientData=0x7f6030c150c0, mask=) at connection.c:281
#44 0x000000000042e391 in aeProcessEvents (eventLoop=eventLoop@entry=0x7f6030c0b540, flags=flags@entry=27) at ae.c:479
#45 0x000000000042e7ad in aeMain (eventLoop=0x7f6030c0b540) at ae.c:539
#46 0x000000000042b0ed in main (argc=, argv=0x7ffdab2de9f8) at server.c:5173
$

2

The problem is that you are trying to ping yourself from the Redis main thread you are holding the Redis Global Lock so Redis can not reply to ping. In order to run in a background thread try create a gears execution like this:

import redis
conn=redis.Redis()
GB(‘ShardsIDReader’).map(lambda x: conn.ping()).run()

By creating an execution the ping part will run in a background thread without holding the GIL.

Another option, if you want to execute commands on yourself, is to use the execute function:
https://oss.redislabs.com/redisgears/runtime.html#execute

3

Thanks meirsh for your help