137 lines
3.2 KiB
Markdown
137 lines
3.2 KiB
Markdown
# MongoDB Health Check Troubleshooting
|
|
|
|
## Problem
|
|
MongoDB container failing health checks despite running properly.
|
|
|
|
## Root Cause Analysis
|
|
|
|
### Issue 1: Complex Healthcheck Command
|
|
The original healthcheck used shell script format which can be problematic:
|
|
```yaml
|
|
test: |
|
|
mongosh --eval "db.adminCommand('ping').ok" --quiet
|
|
```
|
|
|
|
### Issue 2: Insufficient Startup Time
|
|
Even with 40s start_period, MongoDB may need more time on:
|
|
- First run (data initialization)
|
|
- Slower systems
|
|
- Systems with high I/O wait
|
|
|
|
### Issue 3: Log Format Issues
|
|
The logs show extreme verbosity and duplication, suggesting the container is outputting logs in an unusual format.
|
|
|
|
## Solution: Simplified Healthcheck
|
|
|
|
### Updated Configuration
|
|
```yaml
|
|
healthcheck:
|
|
test: echo 'db.runCommand("ping").ok' | mongosh localhost:27017/test --quiet
|
|
interval: 10s
|
|
timeout: 5s
|
|
retries: 5
|
|
start_period: 60s # Increased from 40s to 60s
|
|
```
|
|
|
|
### Key Changes
|
|
1. **Piped command instead of --eval**: More reliable with mongosh
|
|
2. **Explicit localhost**: Avoids DNS resolution issues
|
|
3. **Simple test database**: Uses /test instead of admin
|
|
4. **Longer start_period**: 60s gives MongoDB plenty of time
|
|
|
|
## Alternative: Disable Healthcheck for Development
|
|
|
|
If healthchecks continue to cause issues, you can disable them for development:
|
|
|
|
```yaml
|
|
healthcheck:
|
|
disable: true
|
|
```
|
|
|
|
Or remove the healthcheck entirely and use a simple dependency:
|
|
|
|
```yaml
|
|
depends_on:
|
|
- mongodb
|
|
# Remove: condition: service_healthy
|
|
```
|
|
|
|
## How to Apply
|
|
|
|
### Option 1: Pull and Restart (Recommended)
|
|
```bash
|
|
git pull origin main
|
|
docker compose -f docker-compose.dev.yml down -v
|
|
docker compose -f docker-compose.dev.yml up -d
|
|
docker compose -f docker-compose.dev.yml logs -f mongodb
|
|
```
|
|
|
|
### Option 2: Disable Healthcheck (Quick Fix)
|
|
Edit `docker-compose.dev.yml` and replace the healthcheck section with:
|
|
```yaml
|
|
healthcheck:
|
|
disable: true
|
|
```
|
|
|
|
Then restart:
|
|
```bash
|
|
docker compose -f docker-compose.dev.yml down -v
|
|
docker compose -f docker-compose.dev.yml up -d
|
|
```
|
|
|
|
## Verification
|
|
|
|
### Check Container Status
|
|
```bash
|
|
docker ps --format "table {{.Names}} {{.Status}}"
|
|
```
|
|
|
|
### Check MongoDB Connection
|
|
```bash
|
|
docker exec normogen-mongodb-dev mongosh --eval "db.adminCommand('ping')"
|
|
```
|
|
|
|
### Check Health Status
|
|
```bash
|
|
docker inspect normogen-mongodb-dev --format='{{json .State.Health}}' | jq
|
|
```
|
|
|
|
## Common Issues and Fixes
|
|
|
|
### Issue: Port Already in Use
|
|
```bash
|
|
# Check what's using port 27017
|
|
sudo lsof -i :27017
|
|
|
|
# Kill the process if needed
|
|
sudo kill -9 <PID>
|
|
```
|
|
|
|
### Issue: Corrupted Volume
|
|
```bash
|
|
# Remove the volume and start fresh
|
|
docker compose -f docker-compose.dev.yml down -v
|
|
docker compose -f docker-compose.dev.yml up -d
|
|
```
|
|
|
|
### Issue: mongosh Not Found
|
|
This shouldn't happen with mongo:6.0, but if it does:
|
|
```bash
|
|
# Verify mongosh exists
|
|
docker exec normogen-mongodb-dev which mongosh
|
|
|
|
# If not found, try using mongo (legacy)
|
|
docker exec normogen-mongodb-dev which mongo
|
|
```
|
|
|
|
## Development vs Production
|
|
|
|
### Development (Current)
|
|
- Healthcheck enabled but with generous timeouts
|
|
- Focus on getting up and running quickly
|
|
- Can disable healthcheck if causing issues
|
|
|
|
### Production
|
|
- Healthcheck is critical
|
|
- Must use proper healthcheck with monitoring
|
|
- Consider using orchestration tools (Kubernetes, etc.)
|