RedisGears and C++ Client

Hi Team,

I am currently working on a project that requires the automation of data ingestion into a database, application of pre-processing techniques (ICA, Singular Spectrum Analysis) and post processing through AI. Speed is a major factor in choosing the tech stack for this project as it’s a crucial factor in determining the success or failure of the project. Examples of redis edge have been very appealing. Currently, all my work is in C++ and I can successfully ingest data into RedisTimeSeries and Redis Streams.

Redis Gears (RG) would form an essential part of the stack (event driven data processing is essential for the project) but I am currently having difficulty working with Redis Gears as most of the examples are given with python. To justify using python I must demonstrate that it will outperform something similar produced in C++.

Is there a resource available for generic RG commands that are independent of Python?
Where do I start with the C API as I may investigate C/C++ wrappers?
What work arounds would you recommend generally?

I may consider developing an event driven system in C++ independent of redis gears but I believe in the law of economy so this would be the very last resort.

Any help in the right direction would be much appreciated.

Regards,
Naeem

Hey @Naeem_Saleem,

RedisGears C API is not yet documented because it’s not yet fully finalize (though it’s really mature and I believe that it will be finalized and documented in the near coming versions). That said, it does not mean you can not use it, its just mean that you should keep in mind that there might be small changes in the API that will require changes in your code.

The main benefit you will gain in using RedisGears and not Redis directly is:

  1. Cluster support. So for example, if you use cluster or you choose to move to cluster one day, the fact that you are using Gears will make it very simple and almost smooth (maybe even entirely smooth if you follow some basic rules)
  2. A more reach notification API, you can specify keys prefix, whether or not to read values. On streams, you can specify whether or not to trim the stream, what do to on failures, and so on … All this logic is already implemented in Gears and you get it for free by the fact that you are using RedisGears.

Please notice that if most of the computing time is spent on running AI models then I believe the python will not be a big overhead, That said if you choose to try the RedisGears and RedisIA C API I will be very happy to guide you closely through the development process. I will also understand if you decide not to do it because the API is not yet finalized.

If you choose to give it a try then continue reading, if not then you can stop reading now :slight_smile:

So to get started I suggest you to take a look at this branch: https://github.com/RedisGears/RedisGears/pull/372
This branch (which will be merged to master once the review is finished) introduces the most recent changes to the API, but more importantly, it introduces Gears capability to load plugins (.so files), written in C or C++ (even RUST). Gears plugin can use all the Redis module C API but also the RedisGears C API. The RedisGears C API is given here: https://github.com/RedisGears/RedisGears/blob/gears_plugins_support_core_changes/src/redisgears.h
As I said it’s not yet documented but I believe that with some examples you can start work with it. The idea is that you need to put your compiled plugin in some directory and point RedisGears to look for plugins in this directory using PluginsDirectory configuration value (Read here on how to set RedisGears configuration value https://oss.redislabs.com/redisgears/configuration.html)

RedisGears will load your compiled .so file and call RedisGears_OnLoad function. This function must register your plugin using RedisGears_InitAsGearPlugin that gets RedisModuleCtx, plugin name, and plugin version. If this call success then your plugin was registered and you can continue.
See here for how we do it on the JVM plugin: https://github.com/RedisGears/RedisGears/blob/0d5aa2542c606b97b4e2bb9c303d1dd0408b4af9/plugins/gears_jvm/gears_jvm.c#L2898

After your plugin is registered you need to register callbacks functions. Those functions, after registered, will be accessible to be used on executions on the entire Redis cluster.
Again see the jvm example: https://github.com/RedisGears/RedisGears/blob/0d5aa2542c606b97b4e2bb9c303d1dd0408b4af9/plugins/gears_jvm/gears_jvm.c#L2951

Then you can build an execution plan and run it or register it on keyspace notification. This can be done like this:

char* err = NULL;
FlatExecutionPlan* rsctx = RGM_CreateCtx(KeysReader, &err);
RGM_Filter(rsctx, Example_Filter, NULL);
ExecutionPlan* ep = RGM_Run(rsctx, ExecutionModeAsync, readeArgs, NULL, NULL, NULL);

Again you can see how it was done on the jvm: https://github.com/RedisGears/RedisGears/blob/0d5aa2542c606b97b4e2bb9c303d1dd0408b4af9/plugins/gears_jvm/gears_jvm.c#L1184 https://github.com/RedisGears/RedisGears/blob/0d5aa2542c606b97b4e2bb9c303d1dd0408b4af9/plugins/gears_jvm/gears_jvm.c#L1694 https://github.com/RedisGears/RedisGears/blob/0d5aa2542c606b97b4e2bb9c303d1dd0408b4af9/plugins/gears_jvm/gears_jvm.c#L1926

You can also see how to use different readers: https://github.com/RedisGears/RedisGears/blob/0d5aa2542c606b97b4e2bb9c303d1dd0408b4af9/plugins/gears_jvm/gears_jvm.c#L1657 https://github.com/RedisGears/RedisGears/blob/0d5aa2542c606b97b4e2bb9c303d1dd0408b4af9/plugins/gears_jvm/gears_jvm.c#L1588

Now if your goal is to also integrate with RedisAI then take a look at RedisAI low-level API: https://github.com/RedisAI/RedisAI/blob/master/src/redisai.h

2 Likes

Hello meirsh,

I am currently investigating the resources you have provided. Thank you for your guidance.

Regards,
Naeem

2 Likes

meirsh,

I’ve run into an issue related to redisgears.h file whilst compiling a gears plugin:

! RedisModule_GetSharedAPI is throwing an undefined identifier error. I can see it’s declared here:

and here:

What do you suggest for this error? Not sure whether it’s a scoping issue?

Given the limited documentation I have been hesitant to mess with the gears header file, so I decided to test my new dev environment by building a tiny custom module using your resources as a template (I’ve transferred my dev environment from windows to 100% linux, now using vscode)

I’ve successfully compiled, loaded and registered a custom module with redis installed on a single machine. Furthermore, Redis TimeSeries and Redis Streams are both working fine.

Obviously, I would like to move ahead and develop a redisgears plugin as you have already suggested.

Regards,
Naeem

Hey @Naeem_Saleem what are the chances you can upload your code (or part of it) to github so I can see exactly what is the issue?

I’ve created a mini repo on git for you:

https://github.com/NMS0/RedisGears-Plugin.git

1 Like

Getting 404 reply. Maybe its permissions issue?

Sorry, please try again.

Hey @Naeem_Saleem,

I made a small PR to your example repository. The problem was that the RedisModule_GetSharedAPI is defined as experimental feature and so need to be compiled with -DREDISMODULE_EXPERIMENTAL_API.

I also change the artifact name to redisg.so and I run RedisGears with those parameters:
redis-server --loadmodule ./redisgears.so PluginsDirectory /path/to/RedisGears-Plugin

Your plugin was loaded successfully:

23914:M 19 Jul 2020 10:54:26.891 * Loading plugin from file /home/meir/work/RedisGears-Plugin/redisg.so
23914:M 19 Jul 2020 10:54:26.891 # Loading plugin mygplugin version 1

Please make sure to use the gears_plugins branch until it will be merged to master.

Also notice that you do not need all the RedisGears .h files, you only need the redisgears.h, redismodule.h, util/arr_rm_alloc.h and util/arr.h (I also modified the util/arr_rm_alloc.h to allow it to compiled without redisgears_memory.h)

Let me know if anything else is needed.

2 Likes

Dear meirsh,

I’ve implemented the changes you proposed and uploaded the compiled redisg.so file on git. This is the only reference to the plugin I get from the server:

23316:M 19 Jul 2020 20:13:14.083 * PluginsDirectory:./redisg.so

I get no confirmation that it’s loaded.

I normally load modules through config directives, the load modules section of the redis.conf file currently looks like this:

loadmodule ./redisai.so
loadmodule ./redisgears.so PluginsDirectory ./redisg.so
loadmodule ./redistimeseries.so

I tried the following commands too:

redis-server --loadmodule ./redisgears.so PluginsDirectory ./redisg.so

I get the same server response as above?

1 Like

@Naeem_Saleem the PluginsDirectory config value get a directory and looks for .so files in this directory (I might allow specify a specific .so file also but currently on this branch it’s not supported). So you need to give the directory in which the redisg.so file is located and RedisGears will find it there and load it.

1 Like

@meirsh

Yes tried the directory only approach too. I’m getting the same server response after pointing rg to the plugin directory, this is the only reference to the plugin:

39656:M 21 Jul 2020 23:03:32.939 * PluginsDirectory:./Documents/RedisGears-Plugin

All modules are loading successfully just not getting confirmation from the server that the rg plugin has loaded but I’m not getting an error either. I am using redis 6.0.6.

1 Like

Strange, it works for me.
Can you share the full log file?

For the benefit of forum members the problem is solved by switching from Master branch to gears_plugins branch.

Thanks for all your help Meir.

1 Like