MongoDB Docker Issue: /var Filesystem Full

Problem Summary

MongoDB container was failing with:

WiredTiger error: No space left on device (error 28)
fatal log failure
WT_PANIC: WiredTiger library panic

Root Cause

While the root filesystem (/) had 300GB+ free space, the /var filesystem was 100% full.

Why This Affected MongoDB

Docker stores all persistent data in /var/lib/docker:

Container volumes: /var/lib/docker/volumes/
Container images: /var/lib/docker/image/
Container logs: /var/lib/docker/containers/
OverlayFS layers: /var/lib/docker/overlay2/

MongoDB's /data/db is mapped to a Docker volume in /var/lib/docker/volumes/, so even with 300GB+ free on /, MongoDB couldn't write to /var.

How to Detect This Issue

Check All Filesystems

# Check all mounted filesystems
df -h

# Look for filesystems at 100%
df -h | grep -E '100%|Filesystem'

Check Docker Data Location

# Check where Docker stores data
docker system info | grep 'Docker Root Dir'

# Check space usage in Docker directory
sudo du -sh /var/lib/docker/*

Solutions

Immediate Fix: Free Up Space in /var

# Clean Docker (frees space in /var/lib/docker)
docker system prune -a --volumes -f

# Clean package caches
sudo apt clean
sudo apt autoclean

# Clean logs
sudo journalctl --vacuum-time=3d

Monitor /var Space

# Add to crontab for alerts
crontab -e
# Add this line:
*/5 * * * * df /var | tail -1 | awk '{print $5}' | grep -v Use | awk '{if($1+0 > 90) print "/var is " $1 " full"}'

Lessons Learned

Check all filesystems, not just root (/)
Docker data lives in /var by default
Separate mounts can have different space availability
Monitor /var separately when running Docker

Verification

After fixing /var space, verify:

# Check /var has free space
df -h /var

# Check MongoDB container is running
docker ps | grep mongodb

# Check MongoDB is healthy
docker inspect normogen-mongodb-dev --format='{{.State.Health.Status}}'

# Check MongoDB logs
docker logs normogen-mongodb-dev | grep "waiting for connections"

Expected Success

After fixing /var space:

$ df -h /var
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1        50G   15G   35G  30% /var

$ docker ps
CONTAINER ID   IMAGE         STATUS
abc123         mongo:6.0     Up 2 minutes (healthy)

$ docker logs normogen-mongodb-dev
{"msg":"Waiting for connections on port 27017"}

2.5 KiB Raw Blame History