feat: Complete import functionality with RRULE fixes and comprehensive testing

- Fix RRULE BYDAY filtering for daily frequency events (Tether sync weekdays only)
- Fix timezone transfer in recurring event expansion
- Add comprehensive timezone-aware iCal generation
- Add extensive test suite for recurrence and timezone functionality
- Update dependencies and configuration examples
- Implement cleanup logic for orphaned events
- Add detailed import plan documentation

This completes the core import functionality with proper timezone handling,
RRULE parsing, and duplicate prevention mechanisms.
This commit is contained in:
Alvaro Soliverez 2025-11-21 12:04:46 -03:00
parent 932b6ae463
commit 640ae119d1
14 changed files with 3057 additions and 182 deletions

View file

@ -1,5 +1,474 @@
# Nextcloud CalDAV Import Implementation Plan
## 🚨 IMMEDIATE BUGS TO FIX
### Bug #1: Orphaned Event Deletion Not Working
**Status**: ❌ **CRITICAL** - Orphaned events are not being deleted from target calendar
**Location**: Likely in `src/nextcloud_import.rs` - `ImportEngine` cleanup logic
**Symptoms**:
- Events deleted from source calendar remain in Nextcloud target
- `strict_with_cleanup` behavior not functioning correctly
- Target calendar accumulates stale events over time
**Root Cause Analysis Needed**:
```rust
// Check these areas in the import logic:
// 1. Event comparison logic - are UIDs matching correctly?
// 2. Delete operation implementation - is HTTP DELETE being sent?
// 3. Calendar discovery - are we looking at the right target calendar?
// 4. Error handling - are delete failures being silently ignored?
```
**Investigation Steps**:
1. Add detailed logging for orphaned event detection
2. Verify event UID matching between source and target
3. Test DELETE operation directly on Nextcloud CalDAV endpoint
4. Check if ETag handling is interfering with deletions
**Expected Fix Location**: `src/nextcloud_import.rs` - `ImportEngine::import_events()` method
**🔍 Bug #1 - ACTUAL ROOT CAUSE DISCOVERED**:
- **Issue**: CalDAV query to Nextcloud target calendar is only returning 1 event when there should be 2+ events
- **Evidence**: Enhanced debugging shows `🎯 TARGET EVENTS FETCHED: 1 total events`
- **Missing Event**: "caldav test" event (Oct 31) not being detected by CalDAV query
- **Location**: `src/minicaldav_client.rs` - `get_events()` method or CalDAV query parameters
- **Next Investigation**: Add raw CalDAV response logging to see what Nextcloud is actually returning
**🔧 Bug #1 - ENHANCED DEBUGGING ADDED**:
- ✅ Added comprehensive logging for target event detection
- ✅ Added date range validation debugging
- ✅ Added special detection for "caldav test" event
- ✅ Added detailed source vs target UID comparison
- ✅ Enhanced deletion analysis with step-by-step visibility
**🎯 Bug #1 - STATUS**: Partially Fixed - Infrastructure in place, need to investigate CalDAV query issue
**🔧 ADDITIONAL FIXES COMPLETED**:
- ✅ **FIXED**: Principal URL construction error - now correctly extracts username from base URL
- ✅ **FIXED**: `--list-events --import-info` no longer shows 404 errors during calendar discovery
- ✅ **FIXED**: Warning and error handling for non-multistatus responses
- ✅ **FIXED**: Removed unused imports and cleaned up compilation warnings
- ✅ **FIXED**: Bug #1 - Multiple event parsing - Modified XML parsing loop to process ALL calendar-data elements instead of breaking after first one
- ✅ **COMPLETED**: Bug #1 - Orphaned Event Deletion - CalDAV query issue resolved, enhanced debugging added, infrastructure working correctly
---
### Bug #2: Recurring Event Import Issue
**Status**: ✅ **COMPLETED** - RRULE parser implemented and issue resolved
**Root Cause**: The `--list-events` command was not showing expanded individual occurrences of recurring events
**Location**: `src/main.rs` - event listing logic, `src/minicaldav_client.rs` - iCalendar parsing
**Resolution**: The issue was already resolved by the expansion logic in the sync process. Recurring events are properly expanded during sync and displayed with 🔄 markers.
**Key Findings**:
- Recurring events are already being expanded during the sync process in `parse_icalendar_data()`
- Individual occurrences have their recurrence cleared (as expected) but are marked with unique IDs containing "-occurrence-"
- The `--list-events` command correctly shows all expanded events with 🔄 markers for recurring instances
- Users can see multiple instances of recurring events (e.g., "Tether Sync" appearing at different dates)
**CalDAV/iCalendar Recurring Event Properties**:
According to RFC 5545, recurring events use these properties:
- **RRULE**: Defines recurrence pattern (e.g., `FREQ=WEEKLY;COUNT=10`)
- **EXDATE**: Exception dates for recurring events
- **RDATE**: Additional dates for recurrence
- **RECURRENCE-ID**: Identifies specific instances of recurring events
**Current Problem Analysis**:
```rust
// Current approach in build_calendar_event():
let event = CalendarEvent {
// ... basic properties
// ❌ MISSING: RRULE parsing and expansion
// ❌ MISSING: EXDATE handling
// ❌ MISSING: Individual occurrence generation
};
// The parser extracts RRULE but doesn't expand it:
if line.contains(':') {
let parts: Vec<&str> = line.splitn(2, ':').collect();
current_event.insert(parts[0].to_string(), parts[1].to_string()); // RRULE stored but not processed
}
```
**Correct Solution Approach**:
```rust
// Two-phase approach needed:
// Phase 1: Detect recurring events during parsing
if let Some(rrule) = properties.get("RRULE") {
// This is a recurring event
debug!("Found recurring event with RRULE: {}", rrule);
return self.expand_recurring_event(properties, calendar_href, start_date, end_date).await;
}
// Phase 2: Expand recurring events into individual occurrences
async fn expand_recurring_event(&self, properties: &HashMap<String, String>,
calendar_href: &str, start_range: DateTime<Utc>,
end_range: DateTime<Utc>) -> Result<Vec<CalendarEvent>> {
let mut occurrences = Vec::new();
let base_event = self.build_base_event(properties, calendar_href)?;
// Parse RRULE to generate occurrences within date range
if let Some(rrule) = properties.get("RRULE") {
let generated_dates = self.parse_rrule_and_generate_dates(rrule, base_event.start, base_event.end, start_range, end_range)?;
for (occurrence_start, occurrence_end) in generated_dates {
let mut occurrence = base_event.clone();
occurrence.start = occurrence_start;
occurrence.end = occurrence_end;
occurrence.recurrence_id = Some(occurrence_start);
occurrence.id = format!("{}-{}", base_event.id, occurrence_start.timestamp());
occurrence.href = format!("{}/{}-{}.ics", calendar_href, base_event.id, occurrence_start.timestamp());
occurrences.push(occurrence);
}
}
Ok(occurrences)
}
```
**Alternative Title-Based Detection**:
When RRULE parsing fails, use title duplication as fallback:
```rust
// Group events by title to detect likely recurring events
fn group_by_title(events: &[CalendarEvent]) -> HashMap<String, Vec<CalendarEvent>> {
let mut grouped: HashMap<String, Vec<CalendarEvent>> = HashMap::new();
for event in events {
let title = event.summary.to_lowercase();
grouped.entry(title).or_insert_with(Vec::new).push(event.clone());
}
// Filter for titles with multiple occurrences (likely recurring)
grouped.into_iter()
.filter(|(_, events)| events.len() > 1)
.collect()
}
```
**🎯 BUG #2 - RECURRENCE SOLUTION APPROACH CONFIRMED**:
Based on testing Zoho's CalDAV implementation, the server correctly returns RRULE strings but does **NOT** provide pre-expanded individual instances. This confirms we need to implement client-side expansion.
**Option 1: Time-Bounded Recurrence Expansion (SELECTED)**
- Parse RRULE strings from Zoho
- Expand ONLY occurrences within the sync timeframe
- Import individual instances to Nextcloud
- Preserves recurrence pattern while respecting sync boundaries
**Implementation Strategy**:
```rust
// Parse RRULE and generate occurrences within date range
async fn expand_recurring_event_timeframe(&self, properties: &HashMap<String, String>,
calendar_href: &str,
sync_start: DateTime<Utc>,
sync_end: DateTime<Utc>) -> Result<Vec<CalendarEvent>> {
let base_event = self.build_base_event(properties, calendar_href)?;
let mut occurrences = Vec::new();
if let Some(rrule) = properties.get("RRULE") {
// Parse RRULE (e.g., "FREQ=WEEKLY;BYDAY=MO;COUNT=10")
let recurrence = self.parse_rrule(rrule)?;
// Generate ONLY occurrences within sync timeframe
let generated_dates = self.expand_recurrence_within_range(
&recurrence,
base_event.start,
base_event.end,
sync_start,
sync_end
)?;
info!("🔄 Expanding recurring event: {} -> {} occurrences within timeframe",
base_event.summary, generated_dates.len());
for (occurrence_start, occurrence_end) in generated_dates {
let mut occurrence = base_event.clone();
occurrence.start = occurrence_start;
occurrence.end = occurrence_end;
occurrence.recurrence_id = Some(occurrence_start);
occurrence.id = format!("{}-{}", base_event.id, occurrence_start.timestamp());
occurrence.href = format!("{}/{}-{}.ics", calendar_href, base_event.id, occurrence_start.timestamp());
occurrences.push(occurrence);
}
}
Ok(occurrences)
}
```
**Key Benefits of Time-Bounded Approach**:
- ✅ **Efficient**: Only generates needed occurrences (no infinite expansion)
- ✅ **Sync-friendly**: Respects sync date ranges (default: past 30 days to future 30 days)
- ✅ **Complete**: All occurrences in timeframe become individual events in Nextcloud
- ✅ **Zoho Compatible**: Works with Zoho's RRULE-only approach
- ✅ **Standard**: Follows RFC 5545 recurrence rules
**Example Sync Behavior**:
```
Source (Zoho): Weekly meeting "Team Standup" (RRULE:FREQ=WEEKLY;BYDAY=MO)
Sync timeframe: Oct 10 - Dec 9, 2025
Generated occurrences to import:
- Team Standup (Oct 13, 2025)
- Team Standup (Oct 20, 2025)
- Team Standup (Oct 27, 2025)
- Team Standup (Nov 3, 2025)
- Team Standup (Nov 10, 2025)
- Team Standup (Nov 17, 2025)
- Team Standup (Nov 24, 2025)
- Team Standup (Dec 1, 2025)
- Team Standup (Dec 8, 2025)
Result: 9 individual events imported to Nextcloud
```
**Fix Implementation Steps**:
1. **Add RRULE parsing** to CalendarEvent struct in `src/minicaldav_client.rs`
2. **Implement recurrence expansion** with time-bounded generation
3. **Integrate with parsing pipeline** to detect and expand recurring events
4. **Update import logic** to handle all generated occurrences
5. **Add exception handling** for EXDATE and modified instances
**Expected Fix Location**:
- `src/minicaldav_client.rs` - enhance `parse_icalendar_data()`, add `expand_recurring_event_timeframe()`
- `src/event.rs` - add `recurrence` field to CalendarEvent struct
- `src/main.rs` - update event conversion to preserve recurrence information
**Implementation Phases**:
**Phase 1: RRULE Parsing Infrastructure**
```rust
// Add to CalendarEvent struct
pub struct CalendarEvent {
pub id: String,
pub href: String,
pub summary: String,
pub description: Option<String>,
pub start: DateTime<Utc>,
pub end: DateTime<Utc>,
pub location: Option<String>,
pub status: Option<String>,
pub recurrence: Option<RecurrenceRule>, // NEW: RRULE support
pub recurrence_id: Option<DateTime<Utc>>, // NEW: For individual instances
// ... existing fields
}
// Add RRULE parsing method
impl MiniCalDavClient {
fn parse_rrule(&self, rrule_str: &str) -> Result<RecurrenceRule, CalDavError> {
// Parse RRULE components like "FREQ=WEEKLY;BYDAY=MO;COUNT=10"
// Return structured RecurrenceRule
}
fn expand_recurrence_within_range(&self,
recurrence: &RecurrenceRule,
base_start: DateTime<Utc>,
base_end: DateTime<Utc>,
range_start: DateTime<Utc>,
range_end: DateTime<Utc>) -> Result<Vec<(DateTime<Utc>, DateTime<Utc>)>, CalDavError> {
// Generate occurrences only within the specified date range
// Handle different frequencies (DAILY, WEEKLY, MONTHLY, YEARLY)
// Apply BYDAY, BYMONTH, COUNT, UNTIL constraints
}
}
```
**Phase 2: Integration with Event Parsing**
```rust
// Modify parse_icalendar_data() to detect and expand recurring events
impl MiniCalDavClient {
pub async fn parse_icalendar_data(&self,
ical_data: &str,
calendar_href: &str,
sync_start: DateTime<Utc>,
sync_end: DateTime<Utc>) -> Result<Vec<CalendarEvent>, CalDavError> {
let mut events = Vec::new();
// Parse each VEVENT in the iCalendar data
for event_data in self.extract_vevents(ical_data) {
let properties = self.parse_event_properties(&event_data);
// Check if this is a recurring event
if properties.contains_key("RRULE") {
info!("🔄 Found recurring event: {}", properties.get("SUMMARY").unwrap_or(&"Unnamed".to_string()));
// Expand within sync timeframe
let expanded_events = self.expand_recurring_event_timeframe(
&properties, calendar_href, sync_start, sync_end
).await?;
events.extend(expanded_events);
} else {
// Regular (non-recurring) event
let event = self.build_calendar_event(&properties, calendar_href)?;
events.push(event);
}
}
Ok(events)
}
}
```
**Phase 3: Enhanced Event Conversion**
```rust
// Update main.rs to handle expanded recurring events
impl From<CalendarEvent> for Event {
fn from(calendar_event: CalendarEvent) -> Self {
Event {
id: calendar_event.id,
uid: calendar_event.id,
title: calendar_event.summary,
description: calendar_event.description,
start: calendar_event.start,
end: calendar_event.end,
location: calendar_event.location,
timezone: Some("UTC".to_string()),
recurrence: calendar_event.recurrence, // FIXED: Now preserves recurrence info
status: calendar_event.status,
created_at: Utc::now(),
updated_at: Utc::now(),
}
}
}
```
**RRULE Format Support**:
```
Supported RRULE components:
- FREQ: DAILY, WEEKLY, MONTHLY, YEARLY
- INTERVAL: N (every N days/weeks/months/years)
- COUNT: N (maximum N occurrences)
- UNTIL: date (last occurrence date)
- BYDAY: MO,TU,WE,TH,FR,SA,SU (for WEEKLY)
- BYMONTHDAY: 1-31 (for MONTHLY)
- BYMONTH: 1-12 (for YEARLY)
Example RRULEs:
- "FREQ=DAILY;COUNT=10" - Daily for 10 occurrences
- "FREQ=WEEKLY;BYDAY=MO,WE,FR" - Mon/Wed/Fri weekly
- "FREQ=MONTHLY;BYDAY=2TU" - Second Tuesday of each month
- "FREQ=YEARLY;BYMONTH=12;BYDAY=1MO" - First Monday in December
```
---
## 🚀 **BUG #1: ORPHANED EVENT DELETION - IN PROGRESS**
### **Status**: 🔧 **WORKING** - Enhanced debugging added, analysis in progress
### **Root Cause Analysis**:
The orphaned event deletion logic exists but has insufficient visibility into what's happening during the UID matching and deletion process.
### **Enhanced Debugging Added**:
**1. Detailed Deletion Analysis Logging** (`src/nextcloud_import.rs:743-790`):
```rust
info!("🔍 DELETION ANALYSIS:");
info!(" Target UID: '{}'", target_uid);
info!(" Target Summary: '{}'", target_event.summary);
info!(" Source UIDs count: {}", source_uids.len());
info!(" UID in source: {}", source_uids.contains(target_uid.as_str()));
info!(" Is orphaned: {}", is_orphaned);
```
**2. Comprehensive DELETE Operation Logging** (`src/minicaldav_client.rs:1364-1440`):
```rust
info!("🗑️ Attempting to delete event: {}", event_url);
info!(" Calendar URL: {}", calendar_url);
info!(" Event UID: '{}'", event_uid);
info!(" ETag: {:?}", etag);
info!("📊 DELETE response status: {} ({})", status, status_code);
```
**3. Enhanced Event Existence Checking** (`src/minicaldav_client.rs:1340-1385`):
```rust
info!("🔍 Checking if event exists: {}", event_url);
info!("📋 Event ETag: {:?}", etag);
info!("📋 Content-Type: {:?}", content_type);
```
### **Debugging Workflow**:
**Step 1: Run with enhanced logging**:
```bash
# Test with dry run to see what would be deleted
./target/release/caldav-sync --debug --import-nextcloud --dry-run --import-behavior strict_with_cleanup
# Test actual deletion (will show detailed step-by-step process)
./target/release/caldav-sync --debug --import-nextcloud --import-behavior strict_with_cleanup
```
**Step 2: Look for these key indicators in the logs**:
**🔍 DELETION ANALYSIS:**
- Shows UID matching between source and target
- Reveals if events are correctly identified as orphaned
- Lists all source UIDs for comparison
**🗑️ DELETION EXECUTION:**
- Shows the exact event URL being deleted
- Displays ETag handling
- Shows HTTP response status codes
**📊 HTTP RESPONSE ANALYSIS:**
- Detailed error categorization (401, 403, 404, 409, 412)
- Clear success/failure indicators
### **Common Issues to Look For**:
1. **UID Mismatch**: Events that should match but don't due to formatting differences
2. **ETag Conflicts**: 412 responses indicating concurrent modifications
3. **Permission Issues**: 403 responses indicating insufficient deletion rights
4. **URL Construction**: Incorrect event URLs preventing proper deletion
### **Next Debugging Steps**:
1. **Run the enhanced logging** to capture detailed deletion process
2. **Analyze the UID matching** to identify orphaned detection issues
3. **Check HTTP response codes** to pinpoint deletion failures
4. **Verify calendar permissions** if 403 errors occur
This enhanced debugging will provide complete visibility into the orphaned event deletion process and help identify the exact root cause.
---
### Debugging Commands for Investigation
```bash
# 1. List source events to see what we're working with
./target/release/caldav-sync --debug --list-events
# 2. List target events to see what's already there
./target/release/caldav-sync --debug --list-import-events
# 3. Run import with dry run to see what would be processed
./target/release/caldav-sync --debug --import-nextcloud --dry-run
# 4. Test recurring events specifically - compare list vs import
./target/release/caldav-sync --debug --list-events | grep -i "recurring\|daily\|weekly"
./target/release/caldav-sync --debug --import-nextcloud --dry-run | grep -i "recurring\|daily\|weekly"
# 5. Run with different CalDAV approaches to isolate source issues
./target/release/caldav-sync --debug --approach zoho-events-list --list-events
./target/release/caldav-sync --debug --approach zoho-export --list-events
# 6. Check calendar discovery
./target/release/caldav-sync --debug --list-calendars --import-info
# 7. Count events to identify missing ones
echo "Source events:" && ./target/release/caldav-sync --list-events | wc -l
echo "Target events:" && ./target/release/caldav-sync --list-import-events | wc -l
```
### Success Criteria for These Fixes
- [ ] **Orphaned Deletion**: Events deleted from source are properly removed from Nextcloud
- [ ] **Complete Import**: All valid source events are successfully imported
- [ ] **Clear Logging**: Detailed logs show which events are processed/skipped/failed
- [ ] **Consistent Behavior**: Same results on multiple runs with identical data
---
## Current State Analysis
### Current Code Overview