# MongoDB Docker Issue: /var Filesystem Full

## Problem Summary

MongoDB container was failing with:
```
WiredTiger error: No space left on device (error 28)
fatal log failure
WT_PANIC: WiredTiger library panic
```

## Root Cause

While the root filesystem (`/`) had 300GB+ free space, the **`/var` filesystem was 100% full**.

### Why This Affected MongoDB

Docker stores all persistent data in `/var/lib/docker`:
- Container volumes: `/var/lib/docker/volumes/`
- Container images: `/var/lib/docker/image/`
- Container logs: `/var/lib/docker/containers/`
- OverlayFS layers: `/var/lib/docker/overlay2/`

MongoDB's `/data/db` is mapped to a Docker volume in `/var/lib/docker/volumes/`,
so even with 300GB+ free on `/`, MongoDB couldn't write to `/var`.

## How to Detect This Issue

### Check All Filesystems
```bash
# Check all mounted filesystems
df -h

# Look for filesystems at 100%
df -h | grep -E '100%|Filesystem'
```

### Check Docker Data Location
```bash
# Check where Docker stores data
docker system info | grep 'Docker Root Dir'

# Check space usage in Docker directory
sudo du -sh /var/lib/docker/*
```

## Solutions

### Immediate Fix: Free Up Space in /var

```bash
# Clean Docker (frees space in /var/lib/docker)
docker system prune -a --volumes -f

# Clean package caches
sudo apt clean
sudo apt autoclean

# Clean logs
sudo journalctl --vacuum-time=3d
```

### Monitor /var Space

```bash
# Add to crontab for alerts
crontab -e
# Add this line:
*/5 * * * * df /var | tail -1 | awk '{print $5}' | grep -v Use | awk '{if($1+0 > 90) print "/var is " $1 " full"}'
```

## Lessons Learned

1. **Check all filesystems**, not just root (`/`)
2. **Docker data lives in `/var`** by default
3. **Separate mounts** can have different space availability
4. **Monitor `/var` separately** when running Docker

## Verification

After fixing /var space, verify:

```bash
# Check /var has free space
df -h /var

# Check MongoDB container is running
docker ps | grep mongodb

# Check MongoDB is healthy
docker inspect normogen-mongodb-dev --format='{{.State.Health.Status}}'

# Check MongoDB logs
docker logs normogen-mongodb-dev | grep "waiting for connections"
```

## Expected Success

After fixing /var space:
```
$ df -h /var
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1        50G   15G   35G  30% /var

$ docker ps
CONTAINER ID   IMAGE         STATUS
abc123         mongo:6.0     Up 2 minutes (healthy)

$ docker logs normogen-mongodb-dev
{"msg":"Waiting for connections on port 27017"}
```