Bulk loader locking application

I am using bulk loader to load data into a Graph A but while the loading is occurring reads/writes to different graphs seem to be prevented from occurring. I would think that bulk loader would just take a graph lock for the graph it’s loading but maybe my assumption is wrong.

Do you have any insight into why this might be occurring?

After investigating a little bit, I realized that bulk loader was sending huge requests (up to 2GB at a time) and I believe this was hogging the redis main thread somehow (either by taking up all the socket recieve buffers or by some other mechanism).

I ended up running 2 redis servers in parallel and I no longer observed this locking behavior.

If anybody has any ideas about exactly what’s going on (i.e. what happens when the redis main thread recieves such a huge request) I’d be very interested in learning more!

Hi @dsolnik,

The Python side of the bulk loader converts input CSVs into individual commands, each containing binary blobs (of up to 2 GBs total), which are transmitted to and processed by the Redis server in blocking mode. You can decrease the maximum size, and thus allow other commands to interrupt more frequently, with the bulk loader’s --max-buffer-size argument, but this will likely slow down the loading process.

On reflection, I think that this processing can probably be done asynchronously - if you open an issue on the GitHub repo, I will mark it as an enhancement and follow up when time permits!