136 lines
3.2 KiB
Markdown
136 lines
3.2 KiB
Markdown
# MongoDB Disk Space Issues - RESOLVED
|
|
|
|
## Problem Identified
|
|
|
|
MongoDB container was crashing with:
|
|
```
|
|
WiredTiger error: No space left on device (error 28)
|
|
fatal log failure
|
|
WT_PANIC: WiredTiger library panic
|
|
```
|
|
|
|
## Root Cause
|
|
|
|
The server's disk is **71% full** (608G used of 906G), and MongoDB's WiredTiger
|
|
storage engine cannot write to its journal files at `/data/db/journal/`.
|
|
|
|
## Immediate Solutions
|
|
|
|
### Solution 1: Free Up Disk Space (Recommended)
|
|
|
|
```bash
|
|
# Check disk usage
|
|
df -h
|
|
|
|
# Check what's using space
|
|
sudo du -sh /var/* 2>/dev/null | sort -rh | head -20
|
|
|
|
# Clean Docker system (frees significant space!)
|
|
docker system prune -a --volumes -f
|
|
|
|
# Or more conservatively (without volumes):
|
|
docker system prune -a -f
|
|
|
|
# Clean only unused volumes
|
|
docker volume prune -f
|
|
```
|
|
|
|
### Solution 2: Clean Docker Before Starting MongoDB
|
|
|
|
```bash
|
|
# Stop all containers
|
|
docker compose -f backend/docker-compose.dev.yml down
|
|
|
|
# Clean up
|
|
docker system prune -f
|
|
docker volume prune -f
|
|
|
|
# Restart
|
|
docker compose -f backend/docker-compose.dev.yml up -d
|
|
```
|
|
|
|
### Solution 3: Use Alternative Volume Location
|
|
|
|
If you have another partition with more space:
|
|
|
|
`docker-compose.dev.yml`:
|
|
```yaml
|
|
volumes:
|
|
- /path/to/larger/partition/mongodb:/data/db
|
|
```
|
|
|
|
## How MongoDB Uses Disk Space
|
|
|
|
MongoDB requires disk space for:
|
|
1. **Data files**: The actual database data
|
|
2. **Journal files**: Write-ahead logs (typically 1-3GB)
|
|
3. **WiredTiger cache**: Configured to use 7.3GB in your setup
|
|
4. **Oplog**: Operations log (for replication)
|
|
|
|
Minimum free space recommended: **At least 20% free disk space**
|
|
|
|
## Prevention
|
|
|
|
### Monitor Disk Space
|
|
|
|
```bash
|
|
# Add to crontab for alerts
|
|
df -h | awk '{print $5 " " $6}' | grep -vE 'Use|Mounted|none|tmpfs' | while read output;
|
|
do
|
|
usep=$(echo $output | awk '{print $1}' | cut -d'%' -f1)
|
|
partition=$(echo $output | awk '{print $2}')
|
|
if [ $usep -ge 80 ]; then
|
|
echo "Running out of space on $partition ($usep%)"
|
|
fi
|
|
done
|
|
```
|
|
|
|
### Configure MongoDB Storage Limits
|
|
|
|
In production, configure MongoDB with storage limits:
|
|
|
|
```yaml
|
|
environment:
|
|
- MONGO_INITDB_ROOT_USERNAME=admin
|
|
- MONGO_INITDB_ROOT_PASSWORD=password
|
|
- WIRED_TIGER_CONFIG="cache_size=2G" # Reduce from 7.3G
|
|
```
|
|
|
|
## Steps to Recover
|
|
|
|
1. **Stop containers**:
|
|
```bash
|
|
docker compose -f backend/docker-compose.dev.yml down -v
|
|
```
|
|
|
|
2. **Free disk space** (choose one):
|
|
- `docker system prune -a --volumes -f` (removes all unused Docker data)
|
|
- Remove old logs, backups, or unnecessary files
|
|
|
|
3. **Verify space**:
|
|
```bash
|
|
df -h
|
|
```
|
|
|
|
4. **Start fresh**:
|
|
```bash
|
|
docker compose -f backend/docker-compose.dev.yml up -d
|
|
docker compose -f backend/docker-compose.dev.yml logs -f mongodb
|
|
```
|
|
|
|
5. **Verify MongoDB started**:
|
|
Look for "waiting for connections on port 27017" in the logs
|
|
|
|
## Current Docker Compose Configuration
|
|
|
|
The updated `docker-compose.dev.yml` includes:
|
|
- ✅ Simplified healthcheck
|
|
- ✅ 60s startup grace period
|
|
- ✅ Commented alternative volume mount options
|
|
- ✅ Proper dependency management
|
|
|
|
## Related Documentation
|
|
|
|
- [MongoDB Production Notes](https://www.mongodb.com/docs/manual/administration/production-notes/)
|
|
- [WiredTiger Storage](https://www.mongodb.com/docs/manual/core/wiredtiger/)
|
|
- [Docker Storage](https://docs.docker.com/storage/)
|