MongoDB Disk Space Issues - RESOLVED

Problem Identified

MongoDB container was crashing with:

WiredTiger error: No space left on device (error 28)
fatal log failure
WT_PANIC: WiredTiger library panic

Root Cause

The server's disk is 71% full (608G used of 906G), and MongoDB's WiredTiger storage engine cannot write to its journal files at /data/db/journal/.

Immediate Solutions

Solution 1: Free Up Disk Space (Recommended)

# Check disk usage
df -h

# Check what's using space
sudo du -sh /var/* 2>/dev/null | sort -rh | head -20

# Clean Docker system (frees significant space!)
docker system prune -a --volumes -f

# Or more conservatively (without volumes):
docker system prune -a -f

# Clean only unused volumes
docker volume prune -f

Solution 2: Clean Docker Before Starting MongoDB

# Stop all containers
docker compose -f backend/docker-compose.dev.yml down

# Clean up
docker system prune -f
docker volume prune -f

# Restart
docker compose -f backend/docker-compose.dev.yml up -d

Solution 3: Use Alternative Volume Location

If you have another partition with more space:

docker-compose.dev.yml:

volumes:
  - /path/to/larger/partition/mongodb:/data/db

How MongoDB Uses Disk Space

MongoDB requires disk space for:

Data files: The actual database data
Journal files: Write-ahead logs (typically 1-3GB)
WiredTiger cache: Configured to use 7.3GB in your setup
Oplog: Operations log (for replication)

Minimum free space recommended: At least 20% free disk space

Prevention

Monitor Disk Space

# Add to crontab for alerts
df -h | awk '{print $5 " " $6}' | grep -vE 'Use|Mounted|none|tmpfs' | while read output;
do
  usep=$(echo $output | awk '{print $1}' | cut -d'%' -f1)
  partition=$(echo $output | awk '{print $2}')
  if [ $usep -ge 80 ]; then
    echo "Running out of space on $partition ($usep%)"
  fi
done

Configure MongoDB Storage Limits

In production, configure MongoDB with storage limits:

environment:
  - MONGO_INITDB_ROOT_USERNAME=admin
  - MONGO_INITDB_ROOT_PASSWORD=password
  - WIRED_TIGER_CONFIG="cache_size=2G"  # Reduce from 7.3G

Steps to Recover

Stop containers:

docker compose -f backend/docker-compose.dev.yml down -v

Free disk space (choose one):
- docker system prune -a --volumes -f (removes all unused Docker data)
- Remove old logs, backups, or unnecessary files
Verify space:
```
df -h
```

Start fresh:

docker compose -f backend/docker-compose.dev.yml up -d
docker compose -f backend/docker-compose.dev.yml logs -f mongodb

Verify MongoDB started: Look for "waiting for connections on port 27017" in the logs

Current Docker Compose Configuration

The updated docker-compose.dev.yml includes:

✅ Simplified healthcheck
✅ 60s startup grace period
✅ Commented alternative volume mount options
✅ Proper dependency management

3.2 KiB Raw Blame History