fix(docker): Simplify MongoDB healthcheck and add troubleshooting
This commit is contained in:
parent
b218594b53
commit
b068579671
2 changed files with 139 additions and 3 deletions
|
|
@ -35,12 +35,11 @@ services:
|
|||
networks:
|
||||
- normogen-network
|
||||
healthcheck:
|
||||
test: |
|
||||
mongosh --eval "db.adminCommand('ping').ok" --quiet
|
||||
test: echo 'db.runCommand("ping").ok' | mongosh localhost:27017/test --quiet
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 40s
|
||||
start_period: 60s
|
||||
volumes:
|
||||
mongodb_dev_data:
|
||||
driver: local
|
||||
|
|
|
|||
137
backend/docker/MONGODB-TROUBLESHOOTING.md
Normal file
137
backend/docker/MONGODB-TROUBLESHOOTING.md
Normal file
|
|
@ -0,0 +1,137 @@
|
|||
# MongoDB Health Check Troubleshooting
|
||||
|
||||
## Problem
|
||||
MongoDB container failing health checks despite running properly.
|
||||
|
||||
## Root Cause Analysis
|
||||
|
||||
### Issue 1: Complex Healthcheck Command
|
||||
The original healthcheck used shell script format which can be problematic:
|
||||
```yaml
|
||||
test: |
|
||||
mongosh --eval "db.adminCommand('ping').ok" --quiet
|
||||
```
|
||||
|
||||
### Issue 2: Insufficient Startup Time
|
||||
Even with 40s start_period, MongoDB may need more time on:
|
||||
- First run (data initialization)
|
||||
- Slower systems
|
||||
- Systems with high I/O wait
|
||||
|
||||
### Issue 3: Log Format Issues
|
||||
The logs show extreme verbosity and duplication, suggesting the container is outputting logs in an unusual format.
|
||||
|
||||
## Solution: Simplified Healthcheck
|
||||
|
||||
### Updated Configuration
|
||||
```yaml
|
||||
healthcheck:
|
||||
test: echo 'db.runCommand("ping").ok' | mongosh localhost:27017/test --quiet
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 60s # Increased from 40s to 60s
|
||||
```
|
||||
|
||||
### Key Changes
|
||||
1. **Piped command instead of --eval**: More reliable with mongosh
|
||||
2. **Explicit localhost**: Avoids DNS resolution issues
|
||||
3. **Simple test database**: Uses /test instead of admin
|
||||
4. **Longer start_period**: 60s gives MongoDB plenty of time
|
||||
|
||||
## Alternative: Disable Healthcheck for Development
|
||||
|
||||
If healthchecks continue to cause issues, you can disable them for development:
|
||||
|
||||
```yaml
|
||||
healthcheck:
|
||||
disable: true
|
||||
```
|
||||
|
||||
Or remove the healthcheck entirely and use a simple dependency:
|
||||
|
||||
```yaml
|
||||
depends_on:
|
||||
- mongodb
|
||||
# Remove: condition: service_healthy
|
||||
```
|
||||
|
||||
## How to Apply
|
||||
|
||||
### Option 1: Pull and Restart (Recommended)
|
||||
```bash
|
||||
git pull origin main
|
||||
docker compose -f docker-compose.dev.yml down -v
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
docker compose -f docker-compose.dev.yml logs -f mongodb
|
||||
```
|
||||
|
||||
### Option 2: Disable Healthcheck (Quick Fix)
|
||||
Edit `docker-compose.dev.yml` and replace the healthcheck section with:
|
||||
```yaml
|
||||
healthcheck:
|
||||
disable: true
|
||||
```
|
||||
|
||||
Then restart:
|
||||
```bash
|
||||
docker compose -f docker-compose.dev.yml down -v
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
### Check Container Status
|
||||
```bash
|
||||
docker ps --format "table {{.Names}} {{.Status}}"
|
||||
```
|
||||
|
||||
### Check MongoDB Connection
|
||||
```bash
|
||||
docker exec normogen-mongodb-dev mongosh --eval "db.adminCommand('ping')"
|
||||
```
|
||||
|
||||
### Check Health Status
|
||||
```bash
|
||||
docker inspect normogen-mongodb-dev --format='{{json .State.Health}}' | jq
|
||||
```
|
||||
|
||||
## Common Issues and Fixes
|
||||
|
||||
### Issue: Port Already in Use
|
||||
```bash
|
||||
# Check what's using port 27017
|
||||
sudo lsof -i :27017
|
||||
|
||||
# Kill the process if needed
|
||||
sudo kill -9 <PID>
|
||||
```
|
||||
|
||||
### Issue: Corrupted Volume
|
||||
```bash
|
||||
# Remove the volume and start fresh
|
||||
docker compose -f docker-compose.dev.yml down -v
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
```
|
||||
|
||||
### Issue: mongosh Not Found
|
||||
This shouldn't happen with mongo:6.0, but if it does:
|
||||
```bash
|
||||
# Verify mongosh exists
|
||||
docker exec normogen-mongodb-dev which mongosh
|
||||
|
||||
# If not found, try using mongo (legacy)
|
||||
docker exec normogen-mongodb-dev which mongo
|
||||
```
|
||||
|
||||
## Development vs Production
|
||||
|
||||
### Development (Current)
|
||||
- Healthcheck enabled but with generous timeouts
|
||||
- Focus on getting up and running quickly
|
||||
- Can disable healthcheck if causing issues
|
||||
|
||||
### Production
|
||||
- Healthcheck is critical
|
||||
- Must use proper healthcheck with monitoring
|
||||
- Consider using orchestration tools (Kubernetes, etc.)
|
||||
Loading…
Add table
Add a link
Reference in a new issue