01
Preserve the failed run before restarting it
Capture the container ID, finish time, exit status, OOMKilled flag, restart count, and recent logs. The last application log may precede the failure because a forcibly killed process cannot complete its normal error handling or shutdown output.
Inspect the configured memory and swap limits rather than assuming the Compose file you are looking at matches the failed container. MemorySwap has specific combined-limit semantics; a zero value is not a reliable way to express no swap.
docker inspect --format 'exit={{.State.ExitCode}} oom={{.State.OOMKilled}} finished={{.State.FinishedAt}} restarts={{.RestartCount}}' your-worker
docker inspect --format 'memory={{.HostConfig.Memory}} memorySwap={{.HostConfig.MemorySwap}}' your-worker
docker logs --timestamps --tail=100 your-worker02
Make a small, bounded OOM reproduction
Use a new test container rather than changing an existing worker. This Python workload retains sixteen eight-megabyte allocations. A 64 MiB container limit is deliberately too small. Matching the memory and memory-swap values removes swap headroom for this controlled comparison.
The process can be killed before all allocations finish. Expect a nonzero docker run result and inspect the stopped container immediately. The editorial validation observed exit 137 with OOMKilled true under this limit.
docker run --name oom-proof --memory 64m --memory-swap 64m --cpus .5 python:3.12-slim python -c 'blocks=[bytearray(8*1024*1024) for _ in range(16)]; print("allocated",len(blocks)*8,"MiB")'
docker inspect --format '{{.State.ExitCode}} {{.State.OOMKilled}}' oom-proof03
Compare a different route to the same exit code
A process that receives SIGKILL for another reason can also end with 137. In our second test a sleeping container was explicitly killed and had OOMKilled false. That contrast prevents the exit code from becoming a root-cause label on its own.
Run this only against the new disposable container named in the example. In a real incident, investigate who or what issued a stop, kill, or restart, and whether a shutdown timeout preceded the forced termination.
docker run --detach --name kill-proof --memory 64m python:3.12-slim sleep 60
docker kill --signal KILL kill-proof
docker inspect --format '{{.State.ExitCode}} {{.State.OOMKilled}}' kill-proof04
Test a specific correction with the same workload
For this finite allocation test, a 256 MiB limit gives the same workload enough room to finish. Re-run it and require a zero exit as well as the expected allocation output. This demonstrates a corrected test configuration, not a recommended limit for an unrelated production service.
For an actual worker, also investigate concurrency, request sizes, retained objects, caches, and leaks. More headroom can postpone an unbounded growth problem. Set a limit from measured demand and host capacity, or reduce the workload that must be resident at once.
docker run --name memory-proof --memory 256m --memory-swap 256m --cpus .5 python:3.12-slim python -c 'blocks=[bytearray(8*1024*1024) for _ in range(16)]; print("allocated",len(blocks)*8,"MiB")'
docker inspect --format '{{.State.ExitCode}} {{.State.OOMKilled}}' memory-proof
docker rm oom-proof kill-proof memory-proof05
Know what a snapshot cannot establish
A docker stats sample taken after a restart does not show the memory peak from the failed process. Likewise, a stopped container with OOMKilled false does not identify the actor responsible for its SIGKILL.
When the container evidence is incomplete, investigate the host or VM and retain relevant memory-controller and process-lifecycle records. Keep the conclusion proportional to the evidence: confirmed memory-limit kill, confirmed external kill, or unresolved termination.
Know the limits
The lab uses bounded disposable Linux containers with explicit memory and swap caps. OOMKilled is evidence about the inspected container run, not a universal explanation of every 137 exit. A restart or recreation can change which state is available; retain incident evidence promptly. Reproduction checked on 26 October 2026 using Linux containers on Docker Engine 29.6.2 and Compose 5.3.1; this is the validation environment, not a minimum-version claim.
Sources & further reading
A technical publication by CoreTrailKit.
Browse all articles