Redis process was killed by OS, Is there a bug?

Redis Process Killed by OS: Understanding and Preventing OOM Issues

Explore why your Redis process might be terminated by the Operating System's Out-Of-Memory (OOM) killer, how to diagnose the problem, and effective strategies to prevent it on Ubuntu.

Encountering a situation where your Redis server process is unexpectedly killed by the Operating System can be a frustrating experience. Often, this isn't a bug within Redis itself, but rather a consequence of resource exhaustion, specifically memory. The Linux kernel employs an Out-Of-Memory (OOM) killer to maintain system stability by terminating processes that consume excessive memory when the system runs low. This article delves into the common causes of Redis being targeted by the OOM killer and provides practical steps to diagnose and mitigate these issues on Ubuntu systems.

Understanding the Linux OOM Killer

The OOM killer is a crucial component of the Linux kernel designed to prevent system freezes when physical memory and swap space are fully utilized. When memory pressure becomes critical, the OOM killer selects a process (or processes) to terminate, aiming to free up enough memory to keep the system responsive. While its intention is good, it can be disruptive when it targets critical applications like Redis.

The OOM killer uses a heuristic algorithm to decide which process to kill. This algorithm considers factors such as the amount of memory a process is using, its oom_score, and its oom_score_adj. Processes with a higher oom_score are more likely to be selected. Redis, being a memory-intensive application, often becomes a prime candidate if not properly configured or if the system is under-provisioned.

flowchart TD
    A[System Memory Low] --> B{OOM Killer Activated}
    B --> C{Identify Candidate Processes}
    C --> D["Calculate oom_score (Memory Usage, oom_score_adj)"]
    D --> E{Select Highest Scoring Process}
    E --> F[Terminate Process]
    F --> G{Memory Freed}
    G --> H[System Stabilized (or repeat)]

Flowchart of the Linux OOM Killer Process

Diagnosing OOM Kills for Redis

The first step in resolving OOM issues is to confirm that Redis was indeed killed by the OOM killer. This information is typically logged in the system's kernel messages. On Ubuntu, you can check the syslog or kern.log files, or use the dmesg command.

Look for messages containing keywords like Out of memory, killed process, oom_reaper, or oom-killer. These logs will often explicitly mention the process name (e.g., redis-server) and the amount of memory it was consuming.

sudo grep -i 'killed process' /var/log/syslog
sudo dmesg | grep -i 'out of memory'

Commands to check system logs for OOM killer messages

Preventing Redis OOM Kills

Preventing Redis from being killed by the OOM killer involves a combination of proper resource provisioning, Redis configuration, and system-level adjustments. Here are several strategies:

1. Increase System Memory: The most straightforward solution is to ensure your server has enough RAM to accommodate Redis's dataset and operational overhead, plus other running applications.
2. Configure maxmemory in Redis: Redis allows you to set a maxmemory limit in its configuration file (redis.conf). When this limit is reached, Redis will start evicting keys according to a configured maxmemory-policy. This is crucial for preventing Redis from consuming all available system memory.
3. Choose an Appropriate maxmemory-policy: Redis offers several eviction policies (e.g., noeviction, allkeys-lru, volatile-ttl). Select one that suits your application's data access patterns. noeviction will cause write operations to fail when maxmemory is reached, which is safer than an OOM kill but might impact application functionality.
4. Adjust oom_score_adj: You can make Redis less likely to be killed by lowering its oom_score_adj. This can be done by writing a negative value to /proc/<pid>/oom_score_adj for the Redis process. However, use this with caution, as it might cause other critical processes to be killed instead.
5. Monitor Memory Usage: Regularly monitor Redis memory usage (using INFO memory command or tools like redis-cli monitor) and overall system memory to anticipate and address issues before they become critical.
6. Use Swap Space Wisely: While swap can prevent immediate OOM kills, excessive swapping can severely degrade Redis performance. Ensure you have adequate, but not overly large, swap space, and consider vm.swappiness settings.

# Example redis.conf settings
maxmemory 2gb
maxmemory-policy allkeys-lru

Example Redis configuration for memory limits and eviction policy

1. Identify Redis PID

Find the Process ID (PID) of your running Redis server using pgrep redis-server.

2. Adjust oom_score_adj

As root, write a negative value (e.g., -500) to the oom_score_adj file for the Redis PID. For example: echo -500 | sudo tee /proc/<REDIS_PID>/oom_score_adj. This makes Redis less likely to be killed by the OOM killer. Remember to replace <REDIS_PID> with the actual PID.

3. Verify Adjustment

Confirm the change by reading the file: cat /proc/<REDIS_PID>/oom_score_adj.