docs(docker): Add /var space issue documentation and monitoring scripts

backend/DAILY-SUMMARY.md

@@ -0,0 +1,154 @@
+# Today's Docker Issues - Summary and Solutions
+
+## Date: 2026-02-15
+
+## Issues Fixed
+
+### 1. Edition 2024 Error
+**Problem:** Rust 1.83-alpine didn't support Edition 2024
+**Solution:** Updated Dockerfiles to use Rust 1.93-slim
+**Files Modified:**
+- `backend/docker/Dockerfile`
+- `backend/docker/Dockerfile.dev`
+**Documentation:** `backend/docker/EDITION2024-FIX.md`
+
+### 2. MongoDB Healthcheck Configuration
+**Problem:** Healthcheck timing out, complex command
+**Solution:** Simplified healthcheck with 60s startup grace period
+**Files Modified:**
+- `backend/docker-compose.dev.yml`
+**Documentation:** `backend/docker/MONGODB-TROUBLESHOOTING.md`
+
+### 3. MongoDB Disk Space Issue
+**Problem:** MongoDB crashing with "No space left on device" error
+**Root Cause:** `/var` filesystem was 100% full (not root `/` filesystem)
+**Solution:** Freed up space in `/var` filesystem
+**Key Insight:** Docker stores volumes in `/var/lib/docker/volumes/`, so `/var` space matters more than root space for MongoDB
+**Documentation:** `backend/docker/MONGODB-VAR-FULL-ISSUE.md`
+
+## Lessons Learned
+
+1. **Always check all filesystems** with `df -h`, not just root (`/`)
+2. **Docker data location matters** - `/var/lib/docker` by default
+3. **Separate mounts have different space** - `/var` can be full while `/` has space
+4. **Monitor Docker space usage** regularly with `docker system df`
+
+## Prevention Setup
+
+### Regular Monitoring
+Add to crontab:
+```bash
+# Check disk space every hour
+0 * * * * /path/to/normogen/backend/scripts/check-disk-space.sh
+
+# Clean Docker weekly
+0 2 * * 0 docker system prune -f --filter "until=168h"
+```
+
+### Manual Checks
+```bash
+# Quick space check
+df -h
+
+# Docker space usage
+docker system df
+
+# Verify stack is running
+./backend/scripts/verify-stack.sh
+```
+
+## Documentation Created
+
+1. `backend/docker/EDITION2024-FIX.md` - Edition 2024 fix
+2. `backend/docker/MONGODB-TROUBLESHOOTING.md` - MongoDB issues
+3. `backend/docker/MONGODB-PERMISSIONS-FIX.md` - Permissions guide
+4. `backend/docker/MONGODB-DISKSPACE-FIX.md` - Disk space guide
+5. `backend/docker/MONGODB-VAR-FULL-ISSUE.md` - /var space issue
+6. `backend/docker/DOCKER-COMMANDS.md` - Docker commands reference
+7. `backend/scripts/check-disk-space.sh` - Space monitoring
+8. `backend/scripts/verify-stack.sh` - Stack verification
+9. `backend/diagnose-mongodb.sh` - MongoDB diagnostics
+
+## Quick Reference
+
+### Start the Stack
+```bash
+cd backend
+docker compose -f docker-compose.dev.yml up -d
+```
+
+### Check Status
+```bash
+docker compose -f docker-compose.dev.yml ps
+docker ps | grep normogen
+```
+
+### View Logs
+```bash
+# All services
+docker compose -f docker-compose.dev.yml logs -f
+
+# MongoDB only
+docker logs -f normogen-mongodb-dev
+
+# Backend only
+docker logs -f normogen-backend-dev
+```
+
+### Stop the Stack
+```bash
+docker compose -f docker-compose.dev.yml down
+```
+
+### Clean Restart
+```bash
+docker compose -f docker-compose.dev.yml down -v
+docker compose -f docker-compose.dev.yml up -d
+```
+
+## Success Indicators
+
+When everything is working, you should see:
+
+1. **Containers running:**
+   ```
+   $ docker ps | grep normogen
+   normogen-mongodb-dev   Up X minutes (healthy)   0.0.0.0:27017->27017/tcp
+   normogen-backend-dev   Up X minutes            0.0.0.0:6800->8000/tcp
+   ```
+
+2. **MongoDB logs:**
+   ```
+   {"msg":"Waiting for connections on port 27017"}
+   ```
+
+3. **Backend logs:**
+   ```
+   Server is running on http://0.0.0.0:8000
+   ```
+
+4. **Healthcheck:**
+   ```
+   $ docker inspect normogen-mongodb-dev --format='{{.State.Health.Status}}'
+   healthy
+   ```
+
+## Git Commits
+
+1. `d63f160` - fix(docker): Update to Rust 1.93 to support Edition 2024
+2. `b218594` - fix(docker): Fix MongoDB healthcheck configuration
+3. `b068579` - fix(docker): Simplify MongoDB healthcheck and add troubleshooting
+4. `f0b5109` - fix(docker): Document MongoDB disk space issue and solutions
+
+## Next Steps
+
+1. Build completed successfully
+2. `/var` space issue resolved
+3. Verify stack is running with `./backend/scripts/verify-stack.sh`
+4. Test API endpoints
+5. Continue with Phase 2.4 development
+
+---
+
+**Last Updated:** 2026-02-15
+**Status:** All issues resolved

backend/docker/MONGODB-VAR-FULL-ISSUE.md

@@ -0,0 +1,111 @@
+# MongoDB Docker Issue: /var Filesystem Full
+
+## Problem Summary
+
+MongoDB container was failing with:
+```
+WiredTiger error: No space left on device (error 28)
+fatal log failure
+WT_PANIC: WiredTiger library panic
+```
+
+## Root Cause
+
+While the root filesystem (`/`) had 300GB+ free space, the **`/var` filesystem was 100% full**.
+
+### Why This Affected MongoDB
+
+Docker stores all persistent data in `/var/lib/docker`:
+- Container volumes: `/var/lib/docker/volumes/`
+- Container images: `/var/lib/docker/image/`
+- Container logs: `/var/lib/docker/containers/`
+- OverlayFS layers: `/var/lib/docker/overlay2/`
+
+MongoDB's `/data/db` is mapped to a Docker volume in `/var/lib/docker/volumes/`,
+so even with 300GB+ free on `/`, MongoDB couldn't write to `/var`.
+
+## How to Detect This Issue
+
+### Check All Filesystems
+```bash
+# Check all mounted filesystems
+df -h
+
+# Look for filesystems at 100%
+df -h | grep -E '100%|Filesystem'
+```
+
+### Check Docker Data Location
+```bash
+# Check where Docker stores data
+docker system info | grep 'Docker Root Dir'
+
+# Check space usage in Docker directory
+sudo du -sh /var/lib/docker/*
+```
+
+## Solutions
+
+### Immediate Fix: Free Up Space in /var
+
+```bash
+# Clean Docker (frees space in /var/lib/docker)
+docker system prune -a --volumes -f
+
+# Clean package caches
+sudo apt clean
+sudo apt autoclean
+
+# Clean logs
+sudo journalctl --vacuum-time=3d
+```
+
+### Monitor /var Space
+
+```bash
+# Add to crontab for alerts
+crontab -e
+# Add this line:
+*/5 * * * * df /var | tail -1 | awk '{print $5}' | grep -v Use | awk '{if($1+0 > 90) print "/var is " $1 " full"}'
+```
+
+## Lessons Learned
+
+1. **Check all filesystems**, not just root (`/`)
+2. **Docker data lives in `/var`** by default
+3. **Separate mounts** can have different space availability
+4. **Monitor `/var` separately** when running Docker
+
+## Verification
+
+After fixing /var space, verify:
+
+```bash
+# Check /var has free space
+df -h /var
+
+# Check MongoDB container is running
+docker ps | grep mongodb
+
+# Check MongoDB is healthy
+docker inspect normogen-mongodb-dev --format='{{.State.Health.Status}}'
+
+# Check MongoDB logs
+docker logs normogen-mongodb-dev | grep "waiting for connections"
+```
+
+## Expected Success
+
+After fixing /var space:
+```
+$ df -h /var
+Filesystem      Size  Used Avail Use% Mounted on
+/dev/sdb1        50G   15G   35G  30% /var
+
+$ docker ps
+CONTAINER ID   IMAGE         STATUS
+abc123         mongo:6.0     Up 2 minutes (healthy)
+
+$ docker logs normogen-mongodb-dev
+{"msg":"Waiting for connections on port 27017"}
+```

backend/scripts/check-disk-space.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+# Monitor disk space on all filesystems
+# Run this periodically to catch space issues early
+
+echo "================================"
+echo "Disk Space Check - $(date)"
+echo "================================"
+echo ""
+
+# Check all filesystems
+echo "All Filesystems:"
+df -h
+echo ""
+
+# Check specifically /var
+echo "/var Filesystem:"
+df -h /var 2>/dev/null || echo "No separate /var filesystem"
+echo ""
+
+# Check Docker data location
+echo "Docker Data Location:"
+docker system info 2>/dev/null | grep "Docker Root Dir" || echo "Docker not accessible"
+echo ""
+
+# Check Docker space usage
+echo "Docker Space Usage:"
+docker system df 2>/dev/null || echo "Cannot get Docker stats"
+echo ""
+
+# Alert if any filesystem is > 90% full
+echo "Alerts (filesystems > 90% full):"
+df -h | awk 'NR>1 {gsub(/%/,""); if($5 > 90) print $6 " is " $5 "% full"}"
+if [ $(df -h | awk 'NR>1 {gsub(/%/,""); if($5 > 90)}' | wc -l) -eq 0 ]; then
+  echo "  No alerts"
+fi
+echo ""
+
+echo "================================"
+echo "Check complete"
+echo "================================"

backend/scripts/verify-stack.sh

@@ -0,0 +1,42 @@
+#!/bin/bash
+# Verify MongoDB and Backend are running correctly
+
+echo "================================"
+echo "Normogen Stack Verification"
+echo "================================"
+echo ""
+
+# Check containers are running
+echo "1. Checking containers..."
+docker ps --format "table {{.Names}}	{{.Status}}	{{.Ports}}" | grep normogen
+echo ""
+
+# Check MongoDB health
+echo "2. Checking MongoDB health..."
+MONGO_HEALTH=$(docker inspect normogen-mongodb-dev --format='{{.State.Health.Status}}' 2>/dev/null)
+echo "   MongoDB Health: $MONGO_HEALTH"
+echo ""
+
+# Check if MongoDB is accepting connections
+echo "3. Testing MongoDB connection..."
+docker exec normogen-mongodb-dev mongosh --eval 'db.runCommand({ping: 1})' --quiet 2>/dev/null && echo "   OK MongoDB is responding" || echo "   FAILED MongoDB not responding"
+echo ""
+
+# Check backend logs
+echo "4. Checking backend startup..."
+docker logs normogen-backend-dev 2>&1 | tail -5
+echo ""
+
+# Show recent MongoDB logs
+echo "5. Recent MongoDB logs..."
+docker logs normogen-mongodb-dev 2>&1 | grep -E '(waiting|ready|started|ERROR)' | tail -5
+echo ""
+
+# Check filesystem space
+echo "6. Checking filesystem space..."
+df -h | grep -E '(Filesystem|/var|/$)'
+echo ""
+
+echo "================================"
+echo "Verification complete"
+echo "================================"