Out of memory on Linux: the OOM killer, what it killed and why

A process vanished, a container exited with 137, or memory alerts keep firing. How to read free correctly, find what the OOM killer killed, find the process that's growing, and fix it with limits, tuning or swap.

· 6 min read · By the Spot Downtime team

When Linux runs out of memory it doesn't slow down politely: the kernel's OOM killer picks a process, usually the biggest, and ends it with SIGKILL. Your database disappears, a worker restarts, a container exits with code 137. Before that, the server can crawl as it swaps. Here's how to see what happened and stop it happening again.

Read free correctly

bash
free -h
#                total   used   free   shared  buff/cache   available
# Mem:           7.7Gi  6.1Gi  210Mi    120Mi       1.4Gi       1.2Gi

Ignore free: Linux uses spare memory as disk cache and gives it back when programs need it, so "free" is always low on a healthy server. available is the number that matters: how much programs can still get. Spot Downtime's memory figure is the same idea, total minus available, so cache never sets off an alert.

Did the OOM killer run?

bash
journalctl -k --since "1 day ago" | grep -iE "out of memory|killed process"
# or
sudo dmesg -T | grep -iE "out of memory|killed process"

A line like this names the victim and how big it was:

text
Out of memory: Killed process 2140 (java) total-vm:6812340kB, anon-rss:5120324kB ...

For a container, docker inspect --format '{{.State.OOMKilled}}' myapp says true when it was killed for going over its own memory limit, even if the host had memory to spare. See exit code 137.

Find what's growing

bash
ps -eo pid,user,rss,etime,cmd --sort=-rss | head -10     # RSS in KB
docker stats --no-stream                                   # per container

The killed process isn't always the cause: the OOM killer picks the biggest, which may be your database, while the process that grew is something else. Watch the list over an hour; the one whose RSS only goes up is your leak.

Fixes

Tell the runtime how much it can use

Many runtimes size themselves for the whole machine. Cap them below the server's (or container's) memory: -Xmx for Java, --max-old-space-size for Node.js, shared_buffers and work_mem for PostgreSQL, innodb_buffer_pool_size for MySQL, maxmemory for Redis.

Fewer workers

Each PHP-FPM child, Gunicorn worker or Puma process holds its own copy of the app. Ten workers at 300 MB need 3 GB. Work out the memory per worker and size the pool to fit, with room left for the rest.

Limit containers

A memory limit per container (--memory, or deploy.resources.limits.memory in Compose) means a leak kills that container, which restarts, instead of the OOM killer choosing something important on the host.

Add a little swap

Swap doesn't make a server faster, but 1 to 2 GB gives the kernel somewhere to put idle pages and time before an OOM kill. Many cloud images have none:

bash
sudo fallocate -l 2G /swapfile && sudo chmod 600 /swapfile
sudo mkswap /swapfile && sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Swap that's always in heavy use means the server needs more memory. Spot Downtime charts swap next to memory, so you can see it.

Protect what matters

Make the OOM killer prefer other processes for a critical service with OOMScoreAdjust=-500 in its systemd unit. Use it sparingly: something still has to be killed.

Don't just restart it on a schedule

A nightly restart hides a leak until traffic grows and it fills memory by lunchtime. Find the leak, or at least the size it grows to, and set limits that match.

Alerts

  • Spot Downtime's server agent alerts when memory in use averages over your limit (90% by default) for the alert window, so a brief peak doesn't wake anyone.
  • Containers killed for memory alert on their own, with "was killed for running out of memory", whether or not the host was full.
  • Memory per running container is on the server's Containers table.

Set it up with the server agent. For CPU, see high CPU usage on Linux.

Get new guides by email

Practical posts on monitoring, outages, error fixes and on-call, when they're published. No spam.

Unsubscribe from any email with one click. See our privacy policy.

Keep reading