docs(docker): Add /var space issue documentation and monitoring scripts

backend/docker/MONGODB-VAR-FULL-ISSUE.md

@@ -0,0 +1,111 @@
+# MongoDB Docker Issue: /var Filesystem Full
+
+## Problem Summary
+
+MongoDB container was failing with:
+```
+WiredTiger error: No space left on device (error 28)
+fatal log failure
+WT_PANIC: WiredTiger library panic
+```
+
+## Root Cause
+
+While the root filesystem (`/`) had 300GB+ free space, the **`/var` filesystem was 100% full**.
+
+### Why This Affected MongoDB
+
+Docker stores all persistent data in `/var/lib/docker`:
+- Container volumes: `/var/lib/docker/volumes/`
+- Container images: `/var/lib/docker/image/`
+- Container logs: `/var/lib/docker/containers/`
+- OverlayFS layers: `/var/lib/docker/overlay2/`
+
+MongoDB's `/data/db` is mapped to a Docker volume in `/var/lib/docker/volumes/`,
+so even with 300GB+ free on `/`, MongoDB couldn't write to `/var`.
+
+## How to Detect This Issue
+
+### Check All Filesystems
+```bash
+# Check all mounted filesystems
+df -h
+
+# Look for filesystems at 100%
+df -h | grep -E '100%|Filesystem'
+```
+
+### Check Docker Data Location
+```bash
+# Check where Docker stores data
+docker system info | grep 'Docker Root Dir'
+
+# Check space usage in Docker directory
+sudo du -sh /var/lib/docker/*
+```
+
+## Solutions
+
+### Immediate Fix: Free Up Space in /var
+
+```bash
+# Clean Docker (frees space in /var/lib/docker)
+docker system prune -a --volumes -f
+
+# Clean package caches
+sudo apt clean
+sudo apt autoclean
+
+# Clean logs
+sudo journalctl --vacuum-time=3d
+```
+
+### Monitor /var Space
+
+```bash
+# Add to crontab for alerts
+crontab -e
+# Add this line:
+*/5 * * * * df /var | tail -1 | awk '{print $5}' | grep -v Use | awk '{if($1+0 > 90) print "/var is " $1 " full"}'
+```
+
+## Lessons Learned
+
+1. **Check all filesystems**, not just root (`/`)
+2. **Docker data lives in `/var`** by default
+3. **Separate mounts** can have different space availability
+4. **Monitor `/var` separately** when running Docker
+
+## Verification
+
+After fixing /var space, verify:
+
+```bash
+# Check /var has free space
+df -h /var
+
+# Check MongoDB container is running
+docker ps | grep mongodb
+
+# Check MongoDB is healthy
+docker inspect normogen-mongodb-dev --format='{{.State.Health.Status}}'
+
+# Check MongoDB logs
+docker logs normogen-mongodb-dev | grep "waiting for connections"
+```
+
+## Expected Success
+
+After fixing /var space:
+```
+$ df -h /var
+Filesystem      Size  Used Avail Use% Mounted on
+/dev/sdb1        50G   15G   35G  30% /var
+
+$ docker ps
+CONTAINER ID   IMAGE         STATUS
+abc123         mongo:6.0     Up 2 minutes (healthy)
+
+$ docker logs normogen-mongodb-dev
+{"msg":"Waiting for connections on port 27017"}
+```