111 lines
2.5 KiB
Markdown
111 lines
2.5 KiB
Markdown
# MongoDB Docker Issue: /var Filesystem Full
|
|
|
|
## Problem Summary
|
|
|
|
MongoDB container was failing with:
|
|
```
|
|
WiredTiger error: No space left on device (error 28)
|
|
fatal log failure
|
|
WT_PANIC: WiredTiger library panic
|
|
```
|
|
|
|
## Root Cause
|
|
|
|
While the root filesystem (`/`) had 300GB+ free space, the **`/var` filesystem was 100% full**.
|
|
|
|
### Why This Affected MongoDB
|
|
|
|
Docker stores all persistent data in `/var/lib/docker`:
|
|
- Container volumes: `/var/lib/docker/volumes/`
|
|
- Container images: `/var/lib/docker/image/`
|
|
- Container logs: `/var/lib/docker/containers/`
|
|
- OverlayFS layers: `/var/lib/docker/overlay2/`
|
|
|
|
MongoDB's `/data/db` is mapped to a Docker volume in `/var/lib/docker/volumes/`,
|
|
so even with 300GB+ free on `/`, MongoDB couldn't write to `/var`.
|
|
|
|
## How to Detect This Issue
|
|
|
|
### Check All Filesystems
|
|
```bash
|
|
# Check all mounted filesystems
|
|
df -h
|
|
|
|
# Look for filesystems at 100%
|
|
df -h | grep -E '100%|Filesystem'
|
|
```
|
|
|
|
### Check Docker Data Location
|
|
```bash
|
|
# Check where Docker stores data
|
|
docker system info | grep 'Docker Root Dir'
|
|
|
|
# Check space usage in Docker directory
|
|
sudo du -sh /var/lib/docker/*
|
|
```
|
|
|
|
## Solutions
|
|
|
|
### Immediate Fix: Free Up Space in /var
|
|
|
|
```bash
|
|
# Clean Docker (frees space in /var/lib/docker)
|
|
docker system prune -a --volumes -f
|
|
|
|
# Clean package caches
|
|
sudo apt clean
|
|
sudo apt autoclean
|
|
|
|
# Clean logs
|
|
sudo journalctl --vacuum-time=3d
|
|
```
|
|
|
|
### Monitor /var Space
|
|
|
|
```bash
|
|
# Add to crontab for alerts
|
|
crontab -e
|
|
# Add this line:
|
|
*/5 * * * * df /var | tail -1 | awk '{print $5}' | grep -v Use | awk '{if($1+0 > 90) print "/var is " $1 " full"}'
|
|
```
|
|
|
|
## Lessons Learned
|
|
|
|
1. **Check all filesystems**, not just root (`/`)
|
|
2. **Docker data lives in `/var`** by default
|
|
3. **Separate mounts** can have different space availability
|
|
4. **Monitor `/var` separately** when running Docker
|
|
|
|
## Verification
|
|
|
|
After fixing /var space, verify:
|
|
|
|
```bash
|
|
# Check /var has free space
|
|
df -h /var
|
|
|
|
# Check MongoDB container is running
|
|
docker ps | grep mongodb
|
|
|
|
# Check MongoDB is healthy
|
|
docker inspect normogen-mongodb-dev --format='{{.State.Health.Status}}'
|
|
|
|
# Check MongoDB logs
|
|
docker logs normogen-mongodb-dev | grep "waiting for connections"
|
|
```
|
|
|
|
## Expected Success
|
|
|
|
After fixing /var space:
|
|
```
|
|
$ df -h /var
|
|
Filesystem Size Used Avail Use% Mounted on
|
|
/dev/sdb1 50G 15G 35G 30% /var
|
|
|
|
$ docker ps
|
|
CONTAINER ID IMAGE STATUS
|
|
abc123 mongo:6.0 Up 2 minutes (healthy)
|
|
|
|
$ docker logs normogen-mongodb-dev
|
|
{"msg":"Waiting for connections on port 27017"}
|
|
```
|