154 lines
4 KiB
Markdown
154 lines
4 KiB
Markdown
# Today's Docker Issues - Summary and Solutions
|
|
|
|
## Date: 2026-02-15
|
|
|
|
## Issues Fixed
|
|
|
|
### 1. Edition 2024 Error
|
|
**Problem:** Rust 1.83-alpine didn't support Edition 2024
|
|
**Solution:** Updated Dockerfiles to use Rust 1.93-slim
|
|
**Files Modified:**
|
|
- `backend/docker/Dockerfile`
|
|
- `backend/docker/Dockerfile.dev`
|
|
**Documentation:** `backend/docker/EDITION2024-FIX.md`
|
|
|
|
### 2. MongoDB Healthcheck Configuration
|
|
**Problem:** Healthcheck timing out, complex command
|
|
**Solution:** Simplified healthcheck with 60s startup grace period
|
|
**Files Modified:**
|
|
- `backend/docker-compose.dev.yml`
|
|
**Documentation:** `backend/docker/MONGODB-TROUBLESHOOTING.md`
|
|
|
|
### 3. MongoDB Disk Space Issue
|
|
**Problem:** MongoDB crashing with "No space left on device" error
|
|
**Root Cause:** `/var` filesystem was 100% full (not root `/` filesystem)
|
|
**Solution:** Freed up space in `/var` filesystem
|
|
**Key Insight:** Docker stores volumes in `/var/lib/docker/volumes/`, so `/var` space matters more than root space for MongoDB
|
|
**Documentation:** `backend/docker/MONGODB-VAR-FULL-ISSUE.md`
|
|
|
|
## Lessons Learned
|
|
|
|
1. **Always check all filesystems** with `df -h`, not just root (`/`)
|
|
2. **Docker data location matters** - `/var/lib/docker` by default
|
|
3. **Separate mounts have different space** - `/var` can be full while `/` has space
|
|
4. **Monitor Docker space usage** regularly with `docker system df`
|
|
|
|
## Prevention Setup
|
|
|
|
### Regular Monitoring
|
|
Add to crontab:
|
|
```bash
|
|
# Check disk space every hour
|
|
0 * * * * /path/to/normogen/backend/scripts/check-disk-space.sh
|
|
|
|
# Clean Docker weekly
|
|
0 2 * * 0 docker system prune -f --filter "until=168h"
|
|
```
|
|
|
|
### Manual Checks
|
|
```bash
|
|
# Quick space check
|
|
df -h
|
|
|
|
# Docker space usage
|
|
docker system df
|
|
|
|
# Verify stack is running
|
|
./backend/scripts/verify-stack.sh
|
|
```
|
|
|
|
## Documentation Created
|
|
|
|
1. `backend/docker/EDITION2024-FIX.md` - Edition 2024 fix
|
|
2. `backend/docker/MONGODB-TROUBLESHOOTING.md` - MongoDB issues
|
|
3. `backend/docker/MONGODB-PERMISSIONS-FIX.md` - Permissions guide
|
|
4. `backend/docker/MONGODB-DISKSPACE-FIX.md` - Disk space guide
|
|
5. `backend/docker/MONGODB-VAR-FULL-ISSUE.md` - /var space issue
|
|
6. `backend/docker/DOCKER-COMMANDS.md` - Docker commands reference
|
|
7. `backend/scripts/check-disk-space.sh` - Space monitoring
|
|
8. `backend/scripts/verify-stack.sh` - Stack verification
|
|
9. `backend/diagnose-mongodb.sh` - MongoDB diagnostics
|
|
|
|
## Quick Reference
|
|
|
|
### Start the Stack
|
|
```bash
|
|
cd backend
|
|
docker compose -f docker-compose.dev.yml up -d
|
|
```
|
|
|
|
### Check Status
|
|
```bash
|
|
docker compose -f docker-compose.dev.yml ps
|
|
docker ps | grep normogen
|
|
```
|
|
|
|
### View Logs
|
|
```bash
|
|
# All services
|
|
docker compose -f docker-compose.dev.yml logs -f
|
|
|
|
# MongoDB only
|
|
docker logs -f normogen-mongodb-dev
|
|
|
|
# Backend only
|
|
docker logs -f normogen-backend-dev
|
|
```
|
|
|
|
### Stop the Stack
|
|
```bash
|
|
docker compose -f docker-compose.dev.yml down
|
|
```
|
|
|
|
### Clean Restart
|
|
```bash
|
|
docker compose -f docker-compose.dev.yml down -v
|
|
docker compose -f docker-compose.dev.yml up -d
|
|
```
|
|
|
|
## Success Indicators
|
|
|
|
When everything is working, you should see:
|
|
|
|
1. **Containers running:**
|
|
```
|
|
$ docker ps | grep normogen
|
|
normogen-mongodb-dev Up X minutes (healthy) 0.0.0.0:27017->27017/tcp
|
|
normogen-backend-dev Up X minutes 0.0.0.0:6800->8000/tcp
|
|
```
|
|
|
|
2. **MongoDB logs:**
|
|
```
|
|
{"msg":"Waiting for connections on port 27017"}
|
|
```
|
|
|
|
3. **Backend logs:**
|
|
```
|
|
Server is running on http://0.0.0.0:8000
|
|
```
|
|
|
|
4. **Healthcheck:**
|
|
```
|
|
$ docker inspect normogen-mongodb-dev --format='{{.State.Health.Status}}'
|
|
healthy
|
|
```
|
|
|
|
## Git Commits
|
|
|
|
1. `d63f160` - fix(docker): Update to Rust 1.93 to support Edition 2024
|
|
2. `b218594` - fix(docker): Fix MongoDB healthcheck configuration
|
|
3. `b068579` - fix(docker): Simplify MongoDB healthcheck and add troubleshooting
|
|
4. `f0b5109` - fix(docker): Document MongoDB disk space issue and solutions
|
|
|
|
## Next Steps
|
|
|
|
1. Build completed successfully
|
|
2. `/var` space issue resolved
|
|
3. Verify stack is running with `./backend/scripts/verify-stack.sh`
|
|
4. Test API endpoints
|
|
5. Continue with Phase 2.4 development
|
|
|
|
---
|
|
|
|
**Last Updated:** 2026-02-15
|
|
**Status:** All issues resolved
|