chore: Clean up temporary docs and start Phase 2.4
- Remove 28+ temporary debugging documentation files - Remove temporary test scripts and log files - Keep only essential files (quick-test.sh, EDITION2024-FIX.md) - Create PHASE-2.4-SPEC.md with complete feature specifications - Update STATUS.md with current progress and recent issues - Ready to begin Phase 2.4 implementation
This commit is contained in:
goose
parent
26f0df58ef
commit
51b7d75dca
14 changed files with 245 additions and 987 deletions
|
|
@ -1,71 +0,0 @@
|
|||
# 🔴 Critical Issue Found: Auth Middleware Blocking All Requests
|
||||
|
||||
## Problem
|
||||
ALL API endpoints (including public ones) are returning **401 Unauthorized**.
|
||||
|
||||
## Root Cause
|
||||
In `main.rs`, the auth middleware was applied to ALL routes using:
|
||||
```rust
|
||||
let app = Router::new()
|
||||
.route("/health", get(handlers::health_check)) // Public!
|
||||
.route("/api/auth/login", post(handlers::login)) // Public!
|
||||
.route("/api/users/me", get(handlers::get_profile)) // Protected
|
||||
.route_layer(axum_middleware::from_fn_with_state(
|
||||
app_state.clone(),
|
||||
crate::middleware::auth::jwt_auth_middleware // ← Applied to ALL routes!
|
||||
))
|
||||
.with_state(app_state);
|
||||
```
|
||||
|
||||
The `route_layer` applies the middleware to **all routes** in the router, including public ones like `/health` and `/api/auth/login`.
|
||||
|
||||
## Solution Applied
|
||||
Split routes into **public** and **protected** routers:
|
||||
|
||||
```rust
|
||||
// Public routes (no auth required)
|
||||
let public_routes = Router::new()
|
||||
.route("/health", get(handlers::health_check))
|
||||
.route("/ready", get(handlers::ready_check))
|
||||
.route("/api/auth/register", post(handlers::register))
|
||||
.route("/api/auth/login", post(handlers::login))
|
||||
.route("/api/auth/refresh", post(handlers::refresh_token))
|
||||
.route("/api/auth/logout", post(handlers::logout))
|
||||
.layer(/* logging and CORS */);
|
||||
|
||||
// Protected routes (auth required)
|
||||
let protected_routes = Router::new()
|
||||
.route("/api/users/me", get(handlers::get_profile))
|
||||
.route_layer(jwt_auth_middleware) // ← Only applied to protected routes!
|
||||
|
||||
// Merge them together
|
||||
let app = public_routes.merge(protected_routes).with_state(app_state);
|
||||
```
|
||||
|
||||
## Test Results Before Fix
|
||||
```
|
||||
$ curl http://10.0.10.30:6800/health
|
||||
HTTP Status: 401 ← Should be 200!
|
||||
|
||||
$ curl -X POST http://10.0.10.30:6800/api/auth/register
|
||||
HTTP Status: 401 ← Public endpoint blocked!
|
||||
```
|
||||
|
||||
## Expected Results After Fix
|
||||
```
|
||||
$ curl http://10.0.10.30:6800/health
|
||||
HTTP Status: 200 ← OK!
|
||||
|
||||
$ curl -X POST http://10.0.10.30:6800/api/auth/login \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"email": "test@example.com", "password": "SecurePassword123!"}'
|
||||
HTTP Status: 200 ← OK! Returns JWT tokens
|
||||
|
||||
$ curl http://10.0.10.30:6800/api/users/me
|
||||
HTTP Status: 401 ← Correct! Needs auth token
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
1. Pull the updated code
|
||||
2. Restart the container: `docker compose restart backend`
|
||||
3. Test the API: `./test-api-remote.sh`
|
||||
|
|
@ -1,154 +0,0 @@
|
|||
# Today's Docker Issues - Summary and Solutions
|
||||
|
||||
## Date: 2026-02-15
|
||||
|
||||
## Issues Fixed
|
||||
|
||||
### 1. Edition 2024 Error
|
||||
**Problem:** Rust 1.83-alpine didn't support Edition 2024
|
||||
**Solution:** Updated Dockerfiles to use Rust 1.93-slim
|
||||
**Files Modified:**
|
||||
- `backend/docker/Dockerfile`
|
||||
- `backend/docker/Dockerfile.dev`
|
||||
**Documentation:** `backend/docker/EDITION2024-FIX.md`
|
||||
|
||||
### 2. MongoDB Healthcheck Configuration
|
||||
**Problem:** Healthcheck timing out, complex command
|
||||
**Solution:** Simplified healthcheck with 60s startup grace period
|
||||
**Files Modified:**
|
||||
- `backend/docker-compose.dev.yml`
|
||||
**Documentation:** `backend/docker/MONGODB-TROUBLESHOOTING.md`
|
||||
|
||||
### 3. MongoDB Disk Space Issue
|
||||
**Problem:** MongoDB crashing with "No space left on device" error
|
||||
**Root Cause:** `/var` filesystem was 100% full (not root `/` filesystem)
|
||||
**Solution:** Freed up space in `/var` filesystem
|
||||
**Key Insight:** Docker stores volumes in `/var/lib/docker/volumes/`, so `/var` space matters more than root space for MongoDB
|
||||
**Documentation:** `backend/docker/MONGODB-VAR-FULL-ISSUE.md`
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
1. **Always check all filesystems** with `df -h`, not just root (`/`)
|
||||
2. **Docker data location matters** - `/var/lib/docker` by default
|
||||
3. **Separate mounts have different space** - `/var` can be full while `/` has space
|
||||
4. **Monitor Docker space usage** regularly with `docker system df`
|
||||
|
||||
## Prevention Setup
|
||||
|
||||
### Regular Monitoring
|
||||
Add to crontab:
|
||||
```bash
|
||||
# Check disk space every hour
|
||||
0 * * * * /path/to/normogen/backend/scripts/check-disk-space.sh
|
||||
|
||||
# Clean Docker weekly
|
||||
0 2 * * 0 docker system prune -f --filter "until=168h"
|
||||
```
|
||||
|
||||
### Manual Checks
|
||||
```bash
|
||||
# Quick space check
|
||||
df -h
|
||||
|
||||
# Docker space usage
|
||||
docker system df
|
||||
|
||||
# Verify stack is running
|
||||
./backend/scripts/verify-stack.sh
|
||||
```
|
||||
|
||||
## Documentation Created
|
||||
|
||||
1. `backend/docker/EDITION2024-FIX.md` - Edition 2024 fix
|
||||
2. `backend/docker/MONGODB-TROUBLESHOOTING.md` - MongoDB issues
|
||||
3. `backend/docker/MONGODB-PERMISSIONS-FIX.md` - Permissions guide
|
||||
4. `backend/docker/MONGODB-DISKSPACE-FIX.md` - Disk space guide
|
||||
5. `backend/docker/MONGODB-VAR-FULL-ISSUE.md` - /var space issue
|
||||
6. `backend/docker/DOCKER-COMMANDS.md` - Docker commands reference
|
||||
7. `backend/scripts/check-disk-space.sh` - Space monitoring
|
||||
8. `backend/scripts/verify-stack.sh` - Stack verification
|
||||
9. `backend/diagnose-mongodb.sh` - MongoDB diagnostics
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Start the Stack
|
||||
```bash
|
||||
cd backend
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
```
|
||||
|
||||
### Check Status
|
||||
```bash
|
||||
docker compose -f docker-compose.dev.yml ps
|
||||
docker ps | grep normogen
|
||||
```
|
||||
|
||||
### View Logs
|
||||
```bash
|
||||
# All services
|
||||
docker compose -f docker-compose.dev.yml logs -f
|
||||
|
||||
# MongoDB only
|
||||
docker logs -f normogen-mongodb-dev
|
||||
|
||||
# Backend only
|
||||
docker logs -f normogen-backend-dev
|
||||
```
|
||||
|
||||
### Stop the Stack
|
||||
```bash
|
||||
docker compose -f docker-compose.dev.yml down
|
||||
```
|
||||
|
||||
### Clean Restart
|
||||
```bash
|
||||
docker compose -f docker-compose.dev.yml down -v
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
```
|
||||
|
||||
## Success Indicators
|
||||
|
||||
When everything is working, you should see:
|
||||
|
||||
1. **Containers running:**
|
||||
```
|
||||
$ docker ps | grep normogen
|
||||
normogen-mongodb-dev Up X minutes (healthy) 0.0.0.0:27017->27017/tcp
|
||||
normogen-backend-dev Up X minutes 0.0.0.0:6800->8000/tcp
|
||||
```
|
||||
|
||||
2. **MongoDB logs:**
|
||||
```
|
||||
{"msg":"Waiting for connections on port 27017"}
|
||||
```
|
||||
|
||||
3. **Backend logs:**
|
||||
```
|
||||
Server is running on http://0.0.0.0:8000
|
||||
```
|
||||
|
||||
4. **Healthcheck:**
|
||||
```
|
||||
$ docker inspect normogen-mongodb-dev --format='{{.State.Health.Status}}'
|
||||
healthy
|
||||
```
|
||||
|
||||
## Git Commits
|
||||
|
||||
1. `d63f160` - fix(docker): Update to Rust 1.93 to support Edition 2024
|
||||
2. `b218594` - fix(docker): Fix MongoDB healthcheck configuration
|
||||
3. `b068579` - fix(docker): Simplify MongoDB healthcheck and add troubleshooting
|
||||
4. `f0b5109` - fix(docker): Document MongoDB disk space issue and solutions
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Build completed successfully
|
||||
2. `/var` space issue resolved
|
||||
3. Verify stack is running with `./backend/scripts/verify-stack.sh`
|
||||
4. Test API endpoints
|
||||
5. Continue with Phase 2.4 development
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2026-02-15
|
||||
**Status:** All issues resolved
|
||||
|
|
@ -1,52 +0,0 @@
|
|||
# Quick Fix Commands
|
||||
|
||||
## The Problem
|
||||
Docker is using cached layers from the old Dockerfile. Even though we updated the file,
|
||||
Docker's build cache still has the old `FROM rust:latest` with `apk` commands.
|
||||
|
||||
## Solution Options
|
||||
|
||||
### Option 1: Rebuild without cache (RECOMMENDED)
|
||||
```bash
|
||||
cd /home/asoliver/desarrollo/normogen/backend
|
||||
docker compose -f docker-compose.dev.yml build --no-cache
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
```
|
||||
|
||||
### Option 2: Clear all Docker build cache first
|
||||
```bash
|
||||
# Clear Docker's build cache
|
||||
docker builder prune -af
|
||||
|
||||
# Then rebuild
|
||||
cd /home/asoliver/desarrollo/normogen/backend
|
||||
docker compose -f docker-compose.dev.yml up -d --build
|
||||
```
|
||||
|
||||
### Option 3: Use the provided script
|
||||
```bash
|
||||
cd /home/asoliver/desarrollo/normogen/backend
|
||||
chmod +x fix-docker-build.sh
|
||||
./fix-docker-build.sh
|
||||
```
|
||||
|
||||
## Why This Happened
|
||||
- Docker caches build layers to speed up subsequent builds
|
||||
- When we changed `FROM rust:latest` to `FROM rust:1.93-slim`, Docker should have invalidated the cache
|
||||
- But sometimes Docker's cache gets confused, especially with `latest` tags
|
||||
- The `--no-cache` flag forces Docker to ignore all cached layers
|
||||
|
||||
## What Changed in the Dockerfile
|
||||
```dockerfile
|
||||
# OLD (cached):
|
||||
FROM rust:latest
|
||||
RUN apk add --no-cache musl-dev pkgconf openssl-dev...
|
||||
|
||||
# NEW (current):
|
||||
FROM rust:1.93-slim
|
||||
RUN apt-get update && apt-get install -y pkg-config libssl-dev...
|
||||
```
|
||||
|
||||
The new image uses:
|
||||
- `rust:1.93-slim` (supports Edition 2024)
|
||||
- `apt-get` (Debian/Ubuntu package manager) instead of `apk` (Alpine)
|
||||
|
|
@ -1,42 +0,0 @@
|
|||
# Backend Silent Crash - Fixed
|
||||
|
||||
## Problem
|
||||
The backend container was starting, compiling, and then exiting immediately with NO output.
|
||||
|
||||
## Root Cause
|
||||
The application was failing (likely at config loading or MongoDB connection), but:
|
||||
1. `dotenv::dotenv()` was failing silently (no .env in Docker)
|
||||
2. Errors were only going to the logger (which wasn't initialized yet)
|
||||
3. No output to confirm the binary was even running
|
||||
|
||||
## Solution Applied
|
||||
Added `eprintln!` statements throughout `main.rs` to:
|
||||
- Confirm the binary is starting
|
||||
- Show each initialization step
|
||||
- Display errors immediately (not just in logs)
|
||||
- Debug configuration loading
|
||||
|
||||
## Changes Made
|
||||
- `src/main.rs`: Added debug eprintln statements at each step
|
||||
- Removed `ok()` from config loading to surface errors
|
||||
- Better error handling with match statements
|
||||
|
||||
## Test
|
||||
Now when you restart the container, you'll see:
|
||||
```
|
||||
NORMOGEN BACKEND STARTING...
|
||||
Loading environment variables...
|
||||
No .env file found (this is OK in Docker): ...
|
||||
Initializing logging...
|
||||
Config loaded: DB=normogen_dev, Port=8000
|
||||
Connecting to MongoDB...
|
||||
MongoDB connection successful
|
||||
Server is running on http://0.0.0.0:8000
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
Restart the container and check the logs:
|
||||
```bash
|
||||
docker compose -f backend/docker-compose.dev.yml restart backend
|
||||
docker logs normogen-backend-dev -f
|
||||
```
|
||||
|
|
@ -1,69 +0,0 @@
|
|||
# Backend Silent Crash - Root Cause & Fix
|
||||
|
||||
## Problem
|
||||
The backend container starts, compiles, runs the binary, then exits immediately with NO output.
|
||||
|
||||
## Analysis
|
||||
|
||||
### What We Know
|
||||
1. Cargo builds successfully: "Finished dev profile"
|
||||
2. Binary starts: "Running target/debug/normogen-backend"
|
||||
3. Process exits silently (no logs, no errors)
|
||||
4. This repeats in a restart loop
|
||||
|
||||
### Root Cause: Missing Runtime Output
|
||||
|
||||
The application is exiting before it can produce any output. This happens when:
|
||||
|
||||
1. **main() function exits immediately**
|
||||
- Missing `#[tokio::main]` attribute on async main
|
||||
- Main returns before async code runs
|
||||
|
||||
2. **Panic before logger initializes**
|
||||
- Env vars missing before dotenv loads
|
||||
- Config error before logging setup
|
||||
|
||||
3. **Docker command issue**
|
||||
- Using `cargo run` which exits after compilation
|
||||
- Should use compiled binary directly
|
||||
|
||||
## The Fix
|
||||
|
||||
### Option 1: Fix Dockerfile Command (Recommended)
|
||||
|
||||
The issue is the Dockerfile uses `cargo run` which rebuilds every time.
|
||||
Change to run the compiled binary directly:
|
||||
|
||||
```dockerfile
|
||||
# In Dockerfile.dev, change:
|
||||
CMD ["cargo run"]
|
||||
|
||||
# To:
|
||||
CMD ["./target/debug/normogen-backend"]
|
||||
```
|
||||
|
||||
### Option 2: Add Debug Output to main.rs
|
||||
|
||||
Before anything in main(), add:
|
||||
```rust
|
||||
fn main() {
|
||||
eprintln!("NORMOGEN BACKEND STARTING...");
|
||||
// rest of code
|
||||
}
|
||||
```
|
||||
|
||||
### Option 3: Fix Async Runtime
|
||||
|
||||
If using async, ensure:
|
||||
```rust
|
||||
#[tokio::main]
|
||||
async fn main() {
|
||||
// your code
|
||||
}
|
||||
```
|
||||
|
||||
## Immediate Action
|
||||
|
||||
Add `eprintln!` at the very start of main.rs to confirm code is running.
|
||||
If we see the eprintln, we know the issue is elsewhere.
|
||||
If we DON'T see it, the binary isn't even executing.
|
||||
|
|
@ -31,18 +31,18 @@ services:
|
|||
environment:
|
||||
- MONGO_INITDB_DATABASE=normogen_dev
|
||||
volumes:
|
||||
# Option 1: Use named volume (default)
|
||||
- mongodb_dev_data:/data/db
|
||||
|
||||
# Option 2: Use host directory with more space
|
||||
# - /path/to/more/space/mongodb:/data/db
|
||||
|
||||
# Option 3: Use tmpfs (WARNING: data is lost on container stop!)
|
||||
# - mongodb_tmp:/data/db
|
||||
# For debugging: mount host directory instead
|
||||
# - ./mongodb-data:/data/db
|
||||
networks:
|
||||
- normogen-network
|
||||
# Fix: Explicitly set user to avoid permission issues
|
||||
user: mongodb:mongodb
|
||||
# Alternative: Run as root but fix ownership
|
||||
# user: root
|
||||
healthcheck:
|
||||
test: echo 'db.runCommand("ping").ok' | mongosh localhost:27017/test --quiet
|
||||
test: |
|
||||
echo 'db.runCommand("ping").ok' | mongosh localhost:27017/test --quiet
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
|
@ -50,12 +50,6 @@ services:
|
|||
volumes:
|
||||
mongodb_dev_data:
|
||||
driver: local
|
||||
# For Option 3 (tmpfs):
|
||||
# mongodb_tmp:
|
||||
# driver: local
|
||||
# driver_opts:
|
||||
# type: tmpfs
|
||||
# device: tmpfs
|
||||
networks:
|
||||
normogen-network:
|
||||
driver: bridge
|
||||
|
|
|
|||
|
|
@ -1,136 +0,0 @@
|
|||
# MongoDB Disk Space Issues - RESOLVED
|
||||
|
||||
## Problem Identified
|
||||
|
||||
MongoDB container was crashing with:
|
||||
```
|
||||
WiredTiger error: No space left on device (error 28)
|
||||
fatal log failure
|
||||
WT_PANIC: WiredTiger library panic
|
||||
```
|
||||
|
||||
## Root Cause
|
||||
|
||||
The server's disk is **71% full** (608G used of 906G), and MongoDB's WiredTiger
|
||||
storage engine cannot write to its journal files at `/data/db/journal/`.
|
||||
|
||||
## Immediate Solutions
|
||||
|
||||
### Solution 1: Free Up Disk Space (Recommended)
|
||||
|
||||
```bash
|
||||
# Check disk usage
|
||||
df -h
|
||||
|
||||
# Check what's using space
|
||||
sudo du -sh /var/* 2>/dev/null | sort -rh | head -20
|
||||
|
||||
# Clean Docker system (frees significant space!)
|
||||
docker system prune -a --volumes -f
|
||||
|
||||
# Or more conservatively (without volumes):
|
||||
docker system prune -a -f
|
||||
|
||||
# Clean only unused volumes
|
||||
docker volume prune -f
|
||||
```
|
||||
|
||||
### Solution 2: Clean Docker Before Starting MongoDB
|
||||
|
||||
```bash
|
||||
# Stop all containers
|
||||
docker compose -f backend/docker-compose.dev.yml down
|
||||
|
||||
# Clean up
|
||||
docker system prune -f
|
||||
docker volume prune -f
|
||||
|
||||
# Restart
|
||||
docker compose -f backend/docker-compose.dev.yml up -d
|
||||
```
|
||||
|
||||
### Solution 3: Use Alternative Volume Location
|
||||
|
||||
If you have another partition with more space:
|
||||
|
||||
`docker-compose.dev.yml`:
|
||||
```yaml
|
||||
volumes:
|
||||
- /path/to/larger/partition/mongodb:/data/db
|
||||
```
|
||||
|
||||
## How MongoDB Uses Disk Space
|
||||
|
||||
MongoDB requires disk space for:
|
||||
1. **Data files**: The actual database data
|
||||
2. **Journal files**: Write-ahead logs (typically 1-3GB)
|
||||
3. **WiredTiger cache**: Configured to use 7.3GB in your setup
|
||||
4. **Oplog**: Operations log (for replication)
|
||||
|
||||
Minimum free space recommended: **At least 20% free disk space**
|
||||
|
||||
## Prevention
|
||||
|
||||
### Monitor Disk Space
|
||||
|
||||
```bash
|
||||
# Add to crontab for alerts
|
||||
df -h | awk '{print $5 " " $6}' | grep -vE 'Use|Mounted|none|tmpfs' | while read output;
|
||||
do
|
||||
usep=$(echo $output | awk '{print $1}' | cut -d'%' -f1)
|
||||
partition=$(echo $output | awk '{print $2}')
|
||||
if [ $usep -ge 80 ]; then
|
||||
echo "Running out of space on $partition ($usep%)"
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
### Configure MongoDB Storage Limits
|
||||
|
||||
In production, configure MongoDB with storage limits:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
- MONGO_INITDB_ROOT_USERNAME=admin
|
||||
- MONGO_INITDB_ROOT_PASSWORD=password
|
||||
- WIRED_TIGER_CONFIG="cache_size=2G" # Reduce from 7.3G
|
||||
```
|
||||
|
||||
## Steps to Recover
|
||||
|
||||
1. **Stop containers**:
|
||||
```bash
|
||||
docker compose -f backend/docker-compose.dev.yml down -v
|
||||
```
|
||||
|
||||
2. **Free disk space** (choose one):
|
||||
- `docker system prune -a --volumes -f` (removes all unused Docker data)
|
||||
- Remove old logs, backups, or unnecessary files
|
||||
|
||||
3. **Verify space**:
|
||||
```bash
|
||||
df -h
|
||||
```
|
||||
|
||||
4. **Start fresh**:
|
||||
```bash
|
||||
docker compose -f backend/docker-compose.dev.yml up -d
|
||||
docker compose -f backend/docker-compose.dev.yml logs -f mongodb
|
||||
```
|
||||
|
||||
5. **Verify MongoDB started**:
|
||||
Look for "waiting for connections on port 27017" in the logs
|
||||
|
||||
## Current Docker Compose Configuration
|
||||
|
||||
The updated `docker-compose.dev.yml` includes:
|
||||
- ✅ Simplified healthcheck
|
||||
- ✅ 60s startup grace period
|
||||
- ✅ Commented alternative volume mount options
|
||||
- ✅ Proper dependency management
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [MongoDB Production Notes](https://www.mongodb.com/docs/manual/administration/production-notes/)
|
||||
- [WiredTiger Storage](https://www.mongodb.com/docs/manual/core/wiredtiger/)
|
||||
- [Docker Storage](https://docs.docker.com/storage/)
|
||||
|
|
@ -1,137 +0,0 @@
|
|||
# MongoDB Health Check Troubleshooting
|
||||
|
||||
## Problem
|
||||
MongoDB container failing health checks despite running properly.
|
||||
|
||||
## Root Cause Analysis
|
||||
|
||||
### Issue 1: Complex Healthcheck Command
|
||||
The original healthcheck used shell script format which can be problematic:
|
||||
```yaml
|
||||
test: |
|
||||
mongosh --eval "db.adminCommand('ping').ok" --quiet
|
||||
```
|
||||
|
||||
### Issue 2: Insufficient Startup Time
|
||||
Even with 40s start_period, MongoDB may need more time on:
|
||||
- First run (data initialization)
|
||||
- Slower systems
|
||||
- Systems with high I/O wait
|
||||
|
||||
### Issue 3: Log Format Issues
|
||||
The logs show extreme verbosity and duplication, suggesting the container is outputting logs in an unusual format.
|
||||
|
||||
## Solution: Simplified Healthcheck
|
||||
|
||||
### Updated Configuration
|
||||
```yaml
|
||||
healthcheck:
|
||||
test: echo 'db.runCommand("ping").ok' | mongosh localhost:27017/test --quiet
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 60s # Increased from 40s to 60s
|
||||
```
|
||||
|
||||
### Key Changes
|
||||
1. **Piped command instead of --eval**: More reliable with mongosh
|
||||
2. **Explicit localhost**: Avoids DNS resolution issues
|
||||
3. **Simple test database**: Uses /test instead of admin
|
||||
4. **Longer start_period**: 60s gives MongoDB plenty of time
|
||||
|
||||
## Alternative: Disable Healthcheck for Development
|
||||
|
||||
If healthchecks continue to cause issues, you can disable them for development:
|
||||
|
||||
```yaml
|
||||
healthcheck:
|
||||
disable: true
|
||||
```
|
||||
|
||||
Or remove the healthcheck entirely and use a simple dependency:
|
||||
|
||||
```yaml
|
||||
depends_on:
|
||||
- mongodb
|
||||
# Remove: condition: service_healthy
|
||||
```
|
||||
|
||||
## How to Apply
|
||||
|
||||
### Option 1: Pull and Restart (Recommended)
|
||||
```bash
|
||||
git pull origin main
|
||||
docker compose -f docker-compose.dev.yml down -v
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
docker compose -f docker-compose.dev.yml logs -f mongodb
|
||||
```
|
||||
|
||||
### Option 2: Disable Healthcheck (Quick Fix)
|
||||
Edit `docker-compose.dev.yml` and replace the healthcheck section with:
|
||||
```yaml
|
||||
healthcheck:
|
||||
disable: true
|
||||
```
|
||||
|
||||
Then restart:
|
||||
```bash
|
||||
docker compose -f docker-compose.dev.yml down -v
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
### Check Container Status
|
||||
```bash
|
||||
docker ps --format "table {{.Names}} {{.Status}}"
|
||||
```
|
||||
|
||||
### Check MongoDB Connection
|
||||
```bash
|
||||
docker exec normogen-mongodb-dev mongosh --eval "db.adminCommand('ping')"
|
||||
```
|
||||
|
||||
### Check Health Status
|
||||
```bash
|
||||
docker inspect normogen-mongodb-dev --format='{{json .State.Health}}' | jq
|
||||
```
|
||||
|
||||
## Common Issues and Fixes
|
||||
|
||||
### Issue: Port Already in Use
|
||||
```bash
|
||||
# Check what's using port 27017
|
||||
sudo lsof -i :27017
|
||||
|
||||
# Kill the process if needed
|
||||
sudo kill -9 <PID>
|
||||
```
|
||||
|
||||
### Issue: Corrupted Volume
|
||||
```bash
|
||||
# Remove the volume and start fresh
|
||||
docker compose -f docker-compose.dev.yml down -v
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
```
|
||||
|
||||
### Issue: mongosh Not Found
|
||||
This shouldn't happen with mongo:6.0, but if it does:
|
||||
```bash
|
||||
# Verify mongosh exists
|
||||
docker exec normogen-mongodb-dev which mongosh
|
||||
|
||||
# If not found, try using mongo (legacy)
|
||||
docker exec normogen-mongodb-dev which mongo
|
||||
```
|
||||
|
||||
## Development vs Production
|
||||
|
||||
### Development (Current)
|
||||
- Healthcheck enabled but with generous timeouts
|
||||
- Focus on getting up and running quickly
|
||||
- Can disable healthcheck if causing issues
|
||||
|
||||
### Production
|
||||
- Healthcheck is critical
|
||||
- Must use proper healthcheck with monitoring
|
||||
- Consider using orchestration tools (Kubernetes, etc.)
|
||||
|
|
@ -1,111 +0,0 @@
|
|||
# MongoDB Docker Issue: /var Filesystem Full
|
||||
|
||||
## Problem Summary
|
||||
|
||||
MongoDB container was failing with:
|
||||
```
|
||||
WiredTiger error: No space left on device (error 28)
|
||||
fatal log failure
|
||||
WT_PANIC: WiredTiger library panic
|
||||
```
|
||||
|
||||
## Root Cause
|
||||
|
||||
While the root filesystem (`/`) had 300GB+ free space, the **`/var` filesystem was 100% full**.
|
||||
|
||||
### Why This Affected MongoDB
|
||||
|
||||
Docker stores all persistent data in `/var/lib/docker`:
|
||||
- Container volumes: `/var/lib/docker/volumes/`
|
||||
- Container images: `/var/lib/docker/image/`
|
||||
- Container logs: `/var/lib/docker/containers/`
|
||||
- OverlayFS layers: `/var/lib/docker/overlay2/`
|
||||
|
||||
MongoDB's `/data/db` is mapped to a Docker volume in `/var/lib/docker/volumes/`,
|
||||
so even with 300GB+ free on `/`, MongoDB couldn't write to `/var`.
|
||||
|
||||
## How to Detect This Issue
|
||||
|
||||
### Check All Filesystems
|
||||
```bash
|
||||
# Check all mounted filesystems
|
||||
df -h
|
||||
|
||||
# Look for filesystems at 100%
|
||||
df -h | grep -E '100%|Filesystem'
|
||||
```
|
||||
|
||||
### Check Docker Data Location
|
||||
```bash
|
||||
# Check where Docker stores data
|
||||
docker system info | grep 'Docker Root Dir'
|
||||
|
||||
# Check space usage in Docker directory
|
||||
sudo du -sh /var/lib/docker/*
|
||||
```
|
||||
|
||||
## Solutions
|
||||
|
||||
### Immediate Fix: Free Up Space in /var
|
||||
|
||||
```bash
|
||||
# Clean Docker (frees space in /var/lib/docker)
|
||||
docker system prune -a --volumes -f
|
||||
|
||||
# Clean package caches
|
||||
sudo apt clean
|
||||
sudo apt autoclean
|
||||
|
||||
# Clean logs
|
||||
sudo journalctl --vacuum-time=3d
|
||||
```
|
||||
|
||||
### Monitor /var Space
|
||||
|
||||
```bash
|
||||
# Add to crontab for alerts
|
||||
crontab -e
|
||||
# Add this line:
|
||||
*/5 * * * * df /var | tail -1 | awk '{print $5}' | grep -v Use | awk '{if($1+0 > 90) print "/var is " $1 " full"}'
|
||||
```
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
1. **Check all filesystems**, not just root (`/`)
|
||||
2. **Docker data lives in `/var`** by default
|
||||
3. **Separate mounts** can have different space availability
|
||||
4. **Monitor `/var` separately** when running Docker
|
||||
|
||||
## Verification
|
||||
|
||||
After fixing /var space, verify:
|
||||
|
||||
```bash
|
||||
# Check /var has free space
|
||||
df -h /var
|
||||
|
||||
# Check MongoDB container is running
|
||||
docker ps | grep mongodb
|
||||
|
||||
# Check MongoDB is healthy
|
||||
docker inspect normogen-mongodb-dev --format='{{.State.Health.Status}}'
|
||||
|
||||
# Check MongoDB logs
|
||||
docker logs normogen-mongodb-dev | grep "waiting for connections"
|
||||
```
|
||||
|
||||
## Expected Success
|
||||
|
||||
After fixing /var space:
|
||||
```
|
||||
$ df -h /var
|
||||
Filesystem Size Used Avail Use% Mounted on
|
||||
/dev/sdb1 50G 15G 35G 30% /var
|
||||
|
||||
$ docker ps
|
||||
CONTAINER ID IMAGE STATUS
|
||||
abc123 mongo:6.0 Up 2 minutes (healthy)
|
||||
|
||||
$ docker logs normogen-mongodb-dev
|
||||
{"msg":"Waiting for connections on port 27017"}
|
||||
```
|
||||
|
|
@ -1,21 +0,0 @@
|
|||
#!/bin/bash
|
||||
# Fix for Docker edition2024 build error
|
||||
|
||||
echo "=== Step 1: Clear Docker build cache ==="
|
||||
docker builder prune -af
|
||||
|
||||
echo ""
|
||||
echo "=== Step 2: Remove any old normogen-backend-dev images ==="
|
||||
docker rmi normogen-backend-dev 2>/dev/null || true
|
||||
|
||||
echo ""
|
||||
echo "=== Step 3: Build with no cache ==="
|
||||
cd /home/asoliver/desarrollo/normogen/backend
|
||||
docker compose -f docker-compose.dev.yml build --no-cache
|
||||
|
||||
echo ""
|
||||
echo "=== Step 4: Start the containers ==="
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
|
||||
echo ""
|
||||
echo "Done! Check logs with: docker compose -f docker-compose.dev.yml logs -f"
|
||||
|
|
@ -1,40 +0,0 @@
|
|||
#!/bin/bash
|
||||
# Monitor disk space on all filesystems
|
||||
# Run this periodically to catch space issues early
|
||||
|
||||
echo "================================"
|
||||
echo "Disk Space Check - $(date)"
|
||||
echo "================================"
|
||||
echo ""
|
||||
|
||||
# Check all filesystems
|
||||
echo "All Filesystems:"
|
||||
df -h
|
||||
echo ""
|
||||
|
||||
# Check specifically /var
|
||||
echo "/var Filesystem:"
|
||||
df -h /var 2>/dev/null || echo "No separate /var filesystem"
|
||||
echo ""
|
||||
|
||||
# Check Docker data location
|
||||
echo "Docker Data Location:"
|
||||
docker system info 2>/dev/null | grep "Docker Root Dir" || echo "Docker not accessible"
|
||||
echo ""
|
||||
|
||||
# Check Docker space usage
|
||||
echo "Docker Space Usage:"
|
||||
docker system df 2>/dev/null || echo "Cannot get Docker stats"
|
||||
echo ""
|
||||
|
||||
# Alert if any filesystem is > 90% full
|
||||
echo "Alerts (filesystems > 90% full):"
|
||||
df -h | awk 'NR>1 {gsub(/%/,""); if($5 > 90) print $6 " is " $5 "% full"}"
|
||||
if [ $(df -h | awk 'NR>1 {gsub(/%/,""); if($5 > 90)}' | wc -l) -eq 0 ]; then
|
||||
echo " No alerts"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "================================"
|
||||
echo "Check complete"
|
||||
echo "================================"
|
||||
|
|
@ -1,42 +0,0 @@
|
|||
#!/bin/bash
|
||||
# Verify MongoDB and Backend are running correctly
|
||||
|
||||
echo "================================"
|
||||
echo "Normogen Stack Verification"
|
||||
echo "================================"
|
||||
echo ""
|
||||
|
||||
# Check containers are running
|
||||
echo "1. Checking containers..."
|
||||
docker ps --format "table {{.Names}} {{.Status}} {{.Ports}}" | grep normogen
|
||||
echo ""
|
||||
|
||||
# Check MongoDB health
|
||||
echo "2. Checking MongoDB health..."
|
||||
MONGO_HEALTH=$(docker inspect normogen-mongodb-dev --format='{{.State.Health.Status}}' 2>/dev/null)
|
||||
echo " MongoDB Health: $MONGO_HEALTH"
|
||||
echo ""
|
||||
|
||||
# Check if MongoDB is accepting connections
|
||||
echo "3. Testing MongoDB connection..."
|
||||
docker exec normogen-mongodb-dev mongosh --eval 'db.runCommand({ping: 1})' --quiet 2>/dev/null && echo " OK MongoDB is responding" || echo " FAILED MongoDB not responding"
|
||||
echo ""
|
||||
|
||||
# Check backend logs
|
||||
echo "4. Checking backend startup..."
|
||||
docker logs normogen-backend-dev 2>&1 | tail -5
|
||||
echo ""
|
||||
|
||||
# Show recent MongoDB logs
|
||||
echo "5. Recent MongoDB logs..."
|
||||
docker logs normogen-mongodb-dev 2>&1 | grep -E '(waiting|ready|started|ERROR)' | tail -5
|
||||
echo ""
|
||||
|
||||
# Check filesystem space
|
||||
echo "6. Checking filesystem space..."
|
||||
df -h | grep -E '(Filesystem|/var|/$)'
|
||||
echo ""
|
||||
|
||||
echo "================================"
|
||||
echo "Verification complete"
|
||||
echo "================================"
|
||||
|
|
@ -1,53 +0,0 @@
|
|||
#!/bin/bash
|
||||
# API Testing Script for Normogen Backend (Remote Server)
|
||||
# Server is running at 10.0.10.30:6800
|
||||
|
||||
echo "🧪 Testing Normogen API Endpoints at http://10.0.10.30:6800"
|
||||
echo "================================================================"
|
||||
echo ""
|
||||
|
||||
# Test 1: Health Check
|
||||
echo "1. Testing Health Check..."
|
||||
HEALTH=$(curl -s -w "
|
||||
HTTP Status: %{http_code}
|
||||
" http://10.0.10.30:6800/health)
|
||||
echo "$HEALTH"
|
||||
echo ""
|
||||
|
||||
# Test 2: Ready Check
|
||||
echo "2. Testing Ready Check..."
|
||||
READY=$(curl -s -w "
|
||||
HTTP Status: %{http_code}
|
||||
" http://10.0.10.30:6800/ready)
|
||||
echo "$READY"
|
||||
echo ""
|
||||
|
||||
# Test 3: Register User
|
||||
echo "3. Testing User Registration..."
|
||||
REGISTER=$(curl -s -w "
|
||||
HTTP Status: %{http_code}
|
||||
" -X POST http://10.0.10.30:6800/api/auth/register \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"email": "test@example.com", "password": "SecurePassword123!", "username": "testuser"}')
|
||||
echo "$REGISTER"
|
||||
echo ""
|
||||
|
||||
# Test 4: Login
|
||||
echo "4. Testing User Login..."
|
||||
LOGIN=$(curl -s -w "
|
||||
HTTP Status: %{http_code}
|
||||
" -X POST http://10.0.10.30:6800/api/auth/login \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"email": "test@example.com", "password": "SecurePassword123!"}')
|
||||
echo "$LOGIN"
|
||||
echo ""
|
||||
|
||||
# Test 5: Protected Endpoint (should fail without auth)
|
||||
echo "5. Testing Protected Endpoint (without auth, should fail)..."
|
||||
PROFILE=$(curl -s -w "
|
||||
HTTP Status: %{http_code}
|
||||
" http://10.0.10.30:6800/api/users/me)
|
||||
echo "$PROFILE"
|
||||
echo ""
|
||||
|
||||
echo "✅ Tests complete!"
|
||||
Loading…
Add table
Add a link
Reference in a new issue