diff --git a/backend/DAILY-SUMMARY.md b/backend/DAILY-SUMMARY.md new file mode 100644 index 0000000..0d1048e --- /dev/null +++ b/backend/DAILY-SUMMARY.md @@ -0,0 +1,154 @@ +# Today's Docker Issues - Summary and Solutions + +## Date: 2026-02-15 + +## Issues Fixed + +### 1. Edition 2024 Error +**Problem:** Rust 1.83-alpine didn't support Edition 2024 +**Solution:** Updated Dockerfiles to use Rust 1.93-slim +**Files Modified:** +- `backend/docker/Dockerfile` +- `backend/docker/Dockerfile.dev` +**Documentation:** `backend/docker/EDITION2024-FIX.md` + +### 2. MongoDB Healthcheck Configuration +**Problem:** Healthcheck timing out, complex command +**Solution:** Simplified healthcheck with 60s startup grace period +**Files Modified:** +- `backend/docker-compose.dev.yml` +**Documentation:** `backend/docker/MONGODB-TROUBLESHOOTING.md` + +### 3. MongoDB Disk Space Issue +**Problem:** MongoDB crashing with "No space left on device" error +**Root Cause:** `/var` filesystem was 100% full (not root `/` filesystem) +**Solution:** Freed up space in `/var` filesystem +**Key Insight:** Docker stores volumes in `/var/lib/docker/volumes/`, so `/var` space matters more than root space for MongoDB +**Documentation:** `backend/docker/MONGODB-VAR-FULL-ISSUE.md` + +## Lessons Learned + +1. **Always check all filesystems** with `df -h`, not just root (`/`) +2. **Docker data location matters** - `/var/lib/docker` by default +3. **Separate mounts have different space** - `/var` can be full while `/` has space +4. **Monitor Docker space usage** regularly with `docker system df` + +## Prevention Setup + +### Regular Monitoring +Add to crontab: +```bash +# Check disk space every hour +0 * * * * /path/to/normogen/backend/scripts/check-disk-space.sh + +# Clean Docker weekly +0 2 * * 0 docker system prune -f --filter "until=168h" +``` + +### Manual Checks +```bash +# Quick space check +df -h + +# Docker space usage +docker system df + +# Verify stack is running +./backend/scripts/verify-stack.sh +``` + +## Documentation Created + +1. `backend/docker/EDITION2024-FIX.md` - Edition 2024 fix +2. `backend/docker/MONGODB-TROUBLESHOOTING.md` - MongoDB issues +3. `backend/docker/MONGODB-PERMISSIONS-FIX.md` - Permissions guide +4. `backend/docker/MONGODB-DISKSPACE-FIX.md` - Disk space guide +5. `backend/docker/MONGODB-VAR-FULL-ISSUE.md` - /var space issue +6. `backend/docker/DOCKER-COMMANDS.md` - Docker commands reference +7. `backend/scripts/check-disk-space.sh` - Space monitoring +8. `backend/scripts/verify-stack.sh` - Stack verification +9. `backend/diagnose-mongodb.sh` - MongoDB diagnostics + +## Quick Reference + +### Start the Stack +```bash +cd backend +docker compose -f docker-compose.dev.yml up -d +``` + +### Check Status +```bash +docker compose -f docker-compose.dev.yml ps +docker ps | grep normogen +``` + +### View Logs +```bash +# All services +docker compose -f docker-compose.dev.yml logs -f + +# MongoDB only +docker logs -f normogen-mongodb-dev + +# Backend only +docker logs -f normogen-backend-dev +``` + +### Stop the Stack +```bash +docker compose -f docker-compose.dev.yml down +``` + +### Clean Restart +```bash +docker compose -f docker-compose.dev.yml down -v +docker compose -f docker-compose.dev.yml up -d +``` + +## Success Indicators + +When everything is working, you should see: + +1. **Containers running:** + ``` + $ docker ps | grep normogen + normogen-mongodb-dev Up X minutes (healthy) 0.0.0.0:27017->27017/tcp + normogen-backend-dev Up X minutes 0.0.0.0:6800->8000/tcp + ``` + +2. **MongoDB logs:** + ``` + {"msg":"Waiting for connections on port 27017"} + ``` + +3. **Backend logs:** + ``` + Server is running on http://0.0.0.0:8000 + ``` + +4. **Healthcheck:** + ``` + $ docker inspect normogen-mongodb-dev --format='{{.State.Health.Status}}' + healthy + ``` + +## Git Commits + +1. `d63f160` - fix(docker): Update to Rust 1.93 to support Edition 2024 +2. `b218594` - fix(docker): Fix MongoDB healthcheck configuration +3. `b068579` - fix(docker): Simplify MongoDB healthcheck and add troubleshooting +4. `f0b5109` - fix(docker): Document MongoDB disk space issue and solutions + +## Next Steps + +1. Build completed successfully +2. `/var` space issue resolved +3. Verify stack is running with `./backend/scripts/verify-stack.sh` +4. Test API endpoints +5. Continue with Phase 2.4 development + +--- + +**Last Updated:** 2026-02-15 +**Status:** All issues resolved diff --git a/backend/docker/MONGODB-VAR-FULL-ISSUE.md b/backend/docker/MONGODB-VAR-FULL-ISSUE.md new file mode 100644 index 0000000..ddc78f4 --- /dev/null +++ b/backend/docker/MONGODB-VAR-FULL-ISSUE.md @@ -0,0 +1,111 @@ +# MongoDB Docker Issue: /var Filesystem Full + +## Problem Summary + +MongoDB container was failing with: +``` +WiredTiger error: No space left on device (error 28) +fatal log failure +WT_PANIC: WiredTiger library panic +``` + +## Root Cause + +While the root filesystem (`/`) had 300GB+ free space, the **`/var` filesystem was 100% full**. + +### Why This Affected MongoDB + +Docker stores all persistent data in `/var/lib/docker`: +- Container volumes: `/var/lib/docker/volumes/` +- Container images: `/var/lib/docker/image/` +- Container logs: `/var/lib/docker/containers/` +- OverlayFS layers: `/var/lib/docker/overlay2/` + +MongoDB's `/data/db` is mapped to a Docker volume in `/var/lib/docker/volumes/`, +so even with 300GB+ free on `/`, MongoDB couldn't write to `/var`. + +## How to Detect This Issue + +### Check All Filesystems +```bash +# Check all mounted filesystems +df -h + +# Look for filesystems at 100% +df -h | grep -E '100%|Filesystem' +``` + +### Check Docker Data Location +```bash +# Check where Docker stores data +docker system info | grep 'Docker Root Dir' + +# Check space usage in Docker directory +sudo du -sh /var/lib/docker/* +``` + +## Solutions + +### Immediate Fix: Free Up Space in /var + +```bash +# Clean Docker (frees space in /var/lib/docker) +docker system prune -a --volumes -f + +# Clean package caches +sudo apt clean +sudo apt autoclean + +# Clean logs +sudo journalctl --vacuum-time=3d +``` + +### Monitor /var Space + +```bash +# Add to crontab for alerts +crontab -e +# Add this line: +*/5 * * * * df /var | tail -1 | awk '{print $5}' | grep -v Use | awk '{if($1+0 > 90) print "/var is " $1 " full"}' +``` + +## Lessons Learned + +1. **Check all filesystems**, not just root (`/`) +2. **Docker data lives in `/var`** by default +3. **Separate mounts** can have different space availability +4. **Monitor `/var` separately** when running Docker + +## Verification + +After fixing /var space, verify: + +```bash +# Check /var has free space +df -h /var + +# Check MongoDB container is running +docker ps | grep mongodb + +# Check MongoDB is healthy +docker inspect normogen-mongodb-dev --format='{{.State.Health.Status}}' + +# Check MongoDB logs +docker logs normogen-mongodb-dev | grep "waiting for connections" +``` + +## Expected Success + +After fixing /var space: +``` +$ df -h /var +Filesystem Size Used Avail Use% Mounted on +/dev/sdb1 50G 15G 35G 30% /var + +$ docker ps +CONTAINER ID IMAGE STATUS +abc123 mongo:6.0 Up 2 minutes (healthy) + +$ docker logs normogen-mongodb-dev +{"msg":"Waiting for connections on port 27017"} +``` diff --git a/backend/scripts/check-disk-space.sh b/backend/scripts/check-disk-space.sh new file mode 100755 index 0000000..b91a597 --- /dev/null +++ b/backend/scripts/check-disk-space.sh @@ -0,0 +1,40 @@ +#!/bin/bash +# Monitor disk space on all filesystems +# Run this periodically to catch space issues early + +echo "================================" +echo "Disk Space Check - $(date)" +echo "================================" +echo "" + +# Check all filesystems +echo "All Filesystems:" +df -h +echo "" + +# Check specifically /var +echo "/var Filesystem:" +df -h /var 2>/dev/null || echo "No separate /var filesystem" +echo "" + +# Check Docker data location +echo "Docker Data Location:" +docker system info 2>/dev/null | grep "Docker Root Dir" || echo "Docker not accessible" +echo "" + +# Check Docker space usage +echo "Docker Space Usage:" +docker system df 2>/dev/null || echo "Cannot get Docker stats" +echo "" + +# Alert if any filesystem is > 90% full +echo "Alerts (filesystems > 90% full):" +df -h | awk 'NR>1 {gsub(/%/,""); if($5 > 90) print $6 " is " $5 "% full"}" +if [ $(df -h | awk 'NR>1 {gsub(/%/,""); if($5 > 90)}' | wc -l) -eq 0 ]; then + echo " No alerts" +fi +echo "" + +echo "================================" +echo "Check complete" +echo "================================" diff --git a/backend/scripts/verify-stack.sh b/backend/scripts/verify-stack.sh new file mode 100755 index 0000000..a6b4b61 --- /dev/null +++ b/backend/scripts/verify-stack.sh @@ -0,0 +1,42 @@ +#!/bin/bash +# Verify MongoDB and Backend are running correctly + +echo "================================" +echo "Normogen Stack Verification" +echo "================================" +echo "" + +# Check containers are running +echo "1. Checking containers..." +docker ps --format "table {{.Names}} {{.Status}} {{.Ports}}" | grep normogen +echo "" + +# Check MongoDB health +echo "2. Checking MongoDB health..." +MONGO_HEALTH=$(docker inspect normogen-mongodb-dev --format='{{.State.Health.Status}}' 2>/dev/null) +echo " MongoDB Health: $MONGO_HEALTH" +echo "" + +# Check if MongoDB is accepting connections +echo "3. Testing MongoDB connection..." +docker exec normogen-mongodb-dev mongosh --eval 'db.runCommand({ping: 1})' --quiet 2>/dev/null && echo " OK MongoDB is responding" || echo " FAILED MongoDB not responding" +echo "" + +# Check backend logs +echo "4. Checking backend startup..." +docker logs normogen-backend-dev 2>&1 | tail -5 +echo "" + +# Show recent MongoDB logs +echo "5. Recent MongoDB logs..." +docker logs normogen-mongodb-dev 2>&1 | grep -E '(waiting|ready|started|ERROR)' | tail -5 +echo "" + +# Check filesystem space +echo "6. Checking filesystem space..." +df -h | grep -E '(Filesystem|/var|/$)' +echo "" + +echo "================================" +echo "Verification complete" +echo "================================"