normogen/backend/docker/MONGODB-TROUBLESHOOTING.md

MongoDB Health Check Troubleshooting

Problem

MongoDB container failing health checks despite running properly.

Root Cause Analysis

Issue 1: Complex Healthcheck Command

The original healthcheck used shell script format which can be problematic:

test: |
  mongosh --eval "db.adminCommand('ping').ok" --quiet

Issue 2: Insufficient Startup Time

Even with 40s start_period, MongoDB may need more time on:

• First run (data initialization)
• Slower systems
• Systems with high I/O wait

Issue 3: Log Format Issues

The logs show extreme verbosity and duplication, suggesting the container is outputting logs in an unusual format.

Solution: Simplified Healthcheck

Updated Configuration

healthcheck:
  test: echo 'db.runCommand("ping").ok' | mongosh localhost:27017/test --quiet
  interval: 10s
  timeout: 5s
  retries: 5
  start_period: 60s  # Increased from 40s to 60s

Key Changes

1. Piped command instead of --eval: More reliable with mongosh
2. Explicit localhost: Avoids DNS resolution issues
3. Simple test database: Uses /test instead of admin
4. Longer start_period: 60s gives MongoDB plenty of time

Alternative: Disable Healthcheck for Development

If healthchecks continue to cause issues, you can disable them for development:

healthcheck:
  disable: true

Or remove the healthcheck entirely and use a simple dependency:

depends_on:
  - mongodb
  # Remove: condition: service_healthy

How to Apply

Option 1: Pull and Restart (Recommended)

git pull origin main
docker compose -f docker-compose.dev.yml down -v
docker compose -f docker-compose.dev.yml up -d
docker compose -f docker-compose.dev.yml logs -f mongodb

Option 2: Disable Healthcheck (Quick Fix)

Edit docker-compose.dev.yml and replace the healthcheck section with:

healthcheck:
  disable: true

Then restart:

docker compose -f docker-compose.dev.yml down -v
docker compose -f docker-compose.dev.yml up -d

Verification

Check Container Status

docker ps --format "table {{.Names}}	{{.Status}}"

Check MongoDB Connection

docker exec normogen-mongodb-dev mongosh --eval "db.adminCommand('ping')"

Check Health Status

docker inspect normogen-mongodb-dev --format='{{json .State.Health}}' | jq

Common Issues and Fixes

Issue: Port Already in Use

# Check what's using port 27017
sudo lsof -i :27017

# Kill the process if needed
sudo kill -9 <PID>

Issue: Corrupted Volume

# Remove the volume and start fresh
docker compose -f docker-compose.dev.yml down -v
docker compose -f docker-compose.dev.yml up -d

Issue: mongosh Not Found

This shouldn't happen with mongo:6.0, but if it does:

# Verify mongosh exists
docker exec normogen-mongodb-dev which mongosh

# If not found, try using mongo (legacy)
docker exec normogen-mongodb-dev which mongo

Development vs Production

Development (Current)

• Healthcheck enabled but with generous timeouts
• Focus on getting up and running quickly
• Can disable healthcheck if causing issues

Production

• Healthcheck is critical
• Must use proper healthcheck with monitoring
• Consider using orchestration tools (Kubernetes, etc.)