fix(docker): Simplify MongoDB healthcheck and add troubleshooting
This commit is contained in:
parent
b218594b53
commit
b068579671
2 changed files with 139 additions and 3 deletions
|
|
@ -35,12 +35,11 @@ services:
|
||||||
networks:
|
networks:
|
||||||
- normogen-network
|
- normogen-network
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: |
|
test: echo 'db.runCommand("ping").ok' | mongosh localhost:27017/test --quiet
|
||||||
mongosh --eval "db.adminCommand('ping').ok" --quiet
|
|
||||||
interval: 10s
|
interval: 10s
|
||||||
timeout: 5s
|
timeout: 5s
|
||||||
retries: 5
|
retries: 5
|
||||||
start_period: 40s
|
start_period: 60s
|
||||||
volumes:
|
volumes:
|
||||||
mongodb_dev_data:
|
mongodb_dev_data:
|
||||||
driver: local
|
driver: local
|
||||||
|
|
|
||||||
137
backend/docker/MONGODB-TROUBLESHOOTING.md
Normal file
137
backend/docker/MONGODB-TROUBLESHOOTING.md
Normal file
|
|
@ -0,0 +1,137 @@
|
||||||
|
# MongoDB Health Check Troubleshooting
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
MongoDB container failing health checks despite running properly.
|
||||||
|
|
||||||
|
## Root Cause Analysis
|
||||||
|
|
||||||
|
### Issue 1: Complex Healthcheck Command
|
||||||
|
The original healthcheck used shell script format which can be problematic:
|
||||||
|
```yaml
|
||||||
|
test: |
|
||||||
|
mongosh --eval "db.adminCommand('ping').ok" --quiet
|
||||||
|
```
|
||||||
|
|
||||||
|
### Issue 2: Insufficient Startup Time
|
||||||
|
Even with 40s start_period, MongoDB may need more time on:
|
||||||
|
- First run (data initialization)
|
||||||
|
- Slower systems
|
||||||
|
- Systems with high I/O wait
|
||||||
|
|
||||||
|
### Issue 3: Log Format Issues
|
||||||
|
The logs show extreme verbosity and duplication, suggesting the container is outputting logs in an unusual format.
|
||||||
|
|
||||||
|
## Solution: Simplified Healthcheck
|
||||||
|
|
||||||
|
### Updated Configuration
|
||||||
|
```yaml
|
||||||
|
healthcheck:
|
||||||
|
test: echo 'db.runCommand("ping").ok' | mongosh localhost:27017/test --quiet
|
||||||
|
interval: 10s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 5
|
||||||
|
start_period: 60s # Increased from 40s to 60s
|
||||||
|
```
|
||||||
|
|
||||||
|
### Key Changes
|
||||||
|
1. **Piped command instead of --eval**: More reliable with mongosh
|
||||||
|
2. **Explicit localhost**: Avoids DNS resolution issues
|
||||||
|
3. **Simple test database**: Uses /test instead of admin
|
||||||
|
4. **Longer start_period**: 60s gives MongoDB plenty of time
|
||||||
|
|
||||||
|
## Alternative: Disable Healthcheck for Development
|
||||||
|
|
||||||
|
If healthchecks continue to cause issues, you can disable them for development:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
healthcheck:
|
||||||
|
disable: true
|
||||||
|
```
|
||||||
|
|
||||||
|
Or remove the healthcheck entirely and use a simple dependency:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
depends_on:
|
||||||
|
- mongodb
|
||||||
|
# Remove: condition: service_healthy
|
||||||
|
```
|
||||||
|
|
||||||
|
## How to Apply
|
||||||
|
|
||||||
|
### Option 1: Pull and Restart (Recommended)
|
||||||
|
```bash
|
||||||
|
git pull origin main
|
||||||
|
docker compose -f docker-compose.dev.yml down -v
|
||||||
|
docker compose -f docker-compose.dev.yml up -d
|
||||||
|
docker compose -f docker-compose.dev.yml logs -f mongodb
|
||||||
|
```
|
||||||
|
|
||||||
|
### Option 2: Disable Healthcheck (Quick Fix)
|
||||||
|
Edit `docker-compose.dev.yml` and replace the healthcheck section with:
|
||||||
|
```yaml
|
||||||
|
healthcheck:
|
||||||
|
disable: true
|
||||||
|
```
|
||||||
|
|
||||||
|
Then restart:
|
||||||
|
```bash
|
||||||
|
docker compose -f docker-compose.dev.yml down -v
|
||||||
|
docker compose -f docker-compose.dev.yml up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
### Check Container Status
|
||||||
|
```bash
|
||||||
|
docker ps --format "table {{.Names}} {{.Status}}"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Check MongoDB Connection
|
||||||
|
```bash
|
||||||
|
docker exec normogen-mongodb-dev mongosh --eval "db.adminCommand('ping')"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Check Health Status
|
||||||
|
```bash
|
||||||
|
docker inspect normogen-mongodb-dev --format='{{json .State.Health}}' | jq
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common Issues and Fixes
|
||||||
|
|
||||||
|
### Issue: Port Already in Use
|
||||||
|
```bash
|
||||||
|
# Check what's using port 27017
|
||||||
|
sudo lsof -i :27017
|
||||||
|
|
||||||
|
# Kill the process if needed
|
||||||
|
sudo kill -9 <PID>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Issue: Corrupted Volume
|
||||||
|
```bash
|
||||||
|
# Remove the volume and start fresh
|
||||||
|
docker compose -f docker-compose.dev.yml down -v
|
||||||
|
docker compose -f docker-compose.dev.yml up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
### Issue: mongosh Not Found
|
||||||
|
This shouldn't happen with mongo:6.0, but if it does:
|
||||||
|
```bash
|
||||||
|
# Verify mongosh exists
|
||||||
|
docker exec normogen-mongodb-dev which mongosh
|
||||||
|
|
||||||
|
# If not found, try using mongo (legacy)
|
||||||
|
docker exec normogen-mongodb-dev which mongo
|
||||||
|
```
|
||||||
|
|
||||||
|
## Development vs Production
|
||||||
|
|
||||||
|
### Development (Current)
|
||||||
|
- Healthcheck enabled but with generous timeouts
|
||||||
|
- Focus on getting up and running quickly
|
||||||
|
- Can disable healthcheck if causing issues
|
||||||
|
|
||||||
|
### Production
|
||||||
|
- Healthcheck is critical
|
||||||
|
- Must use proper healthcheck with monitoring
|
||||||
|
- Consider using orchestration tools (Kubernetes, etc.)
|
||||||
Loading…
Add table
Add a link
Reference in a new issue