REDIS Version: 4.0.14
CENT OS 7.9.2009
Please let me know if there is a more appropriate forum but I had an odd issue today. This past weekend we had some server patching down which consumed disk.
Today we ran into an issue where our redis instance went down. The snapshot was not able to complete. Looking at the logs I see the following repeated over and over prior to the outage
Can’t save in background: fork: Cannot allocate memory
10 changes in 300 seconds. Saving…
We also had this configured in our redis.conf file stop-writes-on-bgsave-error yes
The thing that I was hoping someone could answer was checking our disk stats at the time of the outage we were at 79% full with 5.3GB of free space at the time of the outage.
Our redis instance was at approximately 2.2GB in memory at the time of the outage and the snap shot size was only approximately 350 to 400MB in size ( I don’t have exact stats ). Once I cleaned up yum and we got another 5 GB of disk back the instance started to work perfectly again.
Is there some sort of limit built into redis where you can only snapshot if there is a certain % of disk free or some sort of hard limit? Based on the size of the files involved 5.3GB should have been sufficient. I just want to make sure that we have the appropriate monitoring in place to prevent this from happening again.