mirror of
https://github.com/TZERO78/kopi-docka.git
synced 2026-06-19 07:37:12 +00:00
661440d4db
* modified: README.md new file: scripts/generate_architecture_json_and_extract_mermaid.py * new file: CLAUDE.md * deleted: docs/PROBLEM_1_PLAN.md deleted: docs/PROBLEM_2_PLAN.md * modified: CLAUDE.md * chore(release): bump version to 5.2.1, update changelog - Add changelog entries for v5.2.0 (run_command refactoring) and v5.2.1 (docs/CI) - Bump version to 5.2.1 in pyproject.toml, constants.py, __init__.py - Fix version drift between code and changelog 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add unit tests for error handling in backup/restore operations - Implement comprehensive unit tests for the BackupManager and KopiaRepository classes. - Cover scenarios including container stop timeouts, partial volume failures, Kopia connection issues, invalid input handling, Docker discovery errors, hook failures, and malformed data. - Ensure tests validate expected behavior and error recovery paths during backup and restore processes. - Enhance test coverage for snapshot creation, listing, and verification functionalities in the KopiaRepository. * refactor(tests): improve pytest configuration and add parallelization - Remove duplicate tests/pytest.ini (config now only in pyproject.toml) - Fix Makefile venv path (.venv -> venv) - Remove duplicate marker registration from conftest.py - Add pytest-xdist and pytest-randomly to dev dependencies - Add coverage configuration with 25% threshold (target: 60%) - Add reusable fixtures: mock_backup_config, mock_kopia_config, backup_unit_factory Test time improved from 35s to 10s with parallel execution (-n auto). All 262 tests passing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: add comprehensive unit tests for critical backup/restore paths Add 89 new unit tests covering previously untested critical components: - Backup workflow end-to-end (16 tests) * Complete flow: hooks → stop → backup → start → metadata * Pre/post hook integration and failure handling * Parallel volume backup and error recovery * Backup ID consistency and DR bundle integration - Dry-run workflow (23 tests) * Report generation with system info and estimates * Unit analysis and config review sections * Recovery bundle display and edge cases * Missing dependencies and disk space warnings - HooksManager (25 tests) * Hook execution success/failure scenarios * Script validation (missing, non-executable) * Timeout handling and environment variables * Executed hooks tracking across all 4 hook types - Volume restore DIRECT format (16 tests) * Format detection from snapshot tags * Safety backup creation and restore flow * Container stop/restart during restore * Rsync fallback and error handling - Network recreation conflicts (9 tests) * Skip/force recreate flags * Attached containers handling * Rollback on failure with reconnection * IPAM configuration and non-interactive mode All tests use proper mocking and follow existing test patterns. Test coverage increased significantly for stabilization phase. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * test: add comprehensive unit tests for disaster recovery and restore operations - Add 47 new tests for DisasterRecoveryManager covering: - Bundle creation and encryption for all backend types - Recovery script generation (filesystem, S3, B2, Azure, GCS, rclone) - Rclone config detection and bundle rotation - SHA256 checksums and version helpers - Add 11 new tests for RestoreManager covering: - Container network disconnect/reconnect/restart operations - Secret detection in inspect JSON files - Safety backup cleanup - User ID helpers for sudo operations Total: 112 passing tests (47 DisasterRecoveryManager + 65 RestoreManager) All tests validate critical disaster recovery and restore workflows. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * test: add comprehensive P1 tests for backup manager edge cases Add 28 new unit tests covering critical gaps in backup orchestration: - Container health check tests (11 tests): - _wait_container_healthy() with healthy/unhealthy/no healthcheck - Timeout handling and polling behavior - Exception handling and edge cases (empty responses, errors) - Parallel backup tests (7 tests): - ThreadPoolExecutor parallel execution with multiple volumes - Partial failures (None return and exceptions) - Task timeout handling - max_workers configuration - All volumes failing and empty volume list scenarios - Edge case tests (10 tests): - Empty container/volume lists - No compose files - Only stopped containers - Whitespace and special characters in volume names - Completely empty units - Mixed container states - Very long unit names (200 chars) - Zero-size volumes All tests pass. Coverage increased from 22 to 51 tests in test_backup_manager.py. Related to plan_0002_test-coverage-gaps.md (P1 tasks complete). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * test: add comprehensive TAR format tests for legacy backup/restore Add 12 new unit tests for deprecated TAR backup format (pre-v5.0): TAR Backup Tests (6 tests): - test_creates_tar_with_correct_flags - Verifies all tar command flags - test_includes_exclude_patterns_in_tar_command - Tests --exclude patterns - test_creates_snapshot_with_tar_format_tag - Validates backup_format=TAR tag - test_returns_none_when_tar_fails - Handles tar process failures - test_handles_exception_during_tar_backup - Exception handling - test_uses_temporary_file_for_stderr - Prevents pipe deadlock TAR Restore Tests (6 tests): - test_successful_tar_restore - Complete restore workflow - test_tar_file_not_found - Missing tar file handling - test_invalid_tar_file - Non-tar file detection - test_tar_extraction_fails - Extraction error handling - test_stops_and_restarts_containers - Container lifecycle - test_creates_safety_backup - Safety backup creation All tests pass. Ensures compatibility with legacy backups. Related to plan_0002_test-coverage-gaps.md (P2 task complete). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * test: add comprehensive full backup→restore integration tests Add 3 end-to-end integration tests using real Docker and Kopia: 1. test_full_backup_and_restore_cycle: - Creates volume with test data - Backs up using BackupManager - Clears volume completely - Restores using RestoreManager - Verifies data integrity (main file + nested files) - Tests complete workflow with 4 clear steps 2. test_multiple_backup_cycles: - Creates 3 sequential backups with different data - Verifies unique snapshot IDs - Tests incremental backup capability 3. test_backup_preserves_permissions: - Creates file with specific permissions (600) - Backs up and restores - Verifies permissions are preserved correctly Tests Requirements: - Marked as @pytest.mark.integration and @pytest.mark.slow - Automatically skipped if Docker/Kopia not available - Requires root access for volume operations - Uses ephemeral Kopia repository (auto-cleanup) These tests validate the complete backup/restore system works end-to-end with real filesystems, not just mocked components. Related to plan_0002_test-coverage-gaps.md (P2 task complete). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * test: add comprehensive integration tests for hooks and cross-machine restore - Add hook script integration tests (10 test methods) - Test pre/post backup/restore hooks with real side effects - Test environment variables (KOPI_DOCKA_HOOK_TYPE, KOPI_DOCKA_UNIT_NAME) - Test timeout handling, failure scenarios, missing/non-executable scripts - Test realistic maintenance mode scenario - File: tests/integration/test_hooks_integration.py (570 lines) - Add cross-machine restore integration test - Simulate DR scenario: backup on "machine-a", restore on "machine-b" - Test hostname mismatch detection and machine discovery - Verify data restores successfully despite hostname difference - File: tests/integration/test_backup_restore_cycle.py (+151 lines) Coverage: 39.53% (+10.5% from baseline) Tests: 476 passed, 19 skipped (require root/Docker) Completes P2 test coverage improvements from plan_0002 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: improve rclone config detection to distinguish permission errors (#29) **Problem:** When running `sudo kopi-docka admin config new`, the application could not detect existing rclone configurations due to permission restrictions. Instead of providing helpful guidance, it displayed a confusing prompt asking to create a new config even when the user already had a working rclone setup. **Root Cause:** The `_detect_rclone_config_path()` method silently swallowed `PermissionError` exceptions, making it impossible to distinguish between "file not found" and "file exists but permission denied" scenarios. **Solution:** 1. Added `ConfigStatus` enum with three states: FOUND, PERMISSION_DENIED, NOT_FOUND 2. Added `ConfigDetectionResult` NamedTuple to return path, status, and checked paths 3. Created `_detect_rclone_config_with_status()` method with explicit permission error handling 4. Refactored `_detect_rclone_config_path()` as backward-compatible wrapper 5. Updated `configure()` method to show helpful warning with 3 workarounds when permission denied: - Use `sudo -E` to preserve environment - Make config readable by root with `chmod 644` - Copy config to root's home directory **Testing:** - Added 7 comprehensive unit tests covering all scenarios - All tests pass (502/502) - Zero regressions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * docs: document rclone backend and sudo behavior (#29) **Added:** - Comprehensive Rclone backend documentation in CONFIGURATION.md - Updated backend list to include Rclone as option 8 (50+ cloud providers) - Detailed explanation of sudo permission issues and workarounds - Best practices for using rclone with kopi-docka - Example workflow showing complete setup process **Key Documentation:** - Config detection priority when using sudo - Three workarounds for permission denied scenarios - Why preserving user config is important (OAuth tokens, settings) - Complete example from rclone setup to backup verification This documentation helps users avoid the confusion reported in issue #29 and provides clear guidance for working with rclone configurations. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat: implement stable staging paths for metadata backups to fix retention policies This implements stable, reusable staging directories instead of random temporary directories for recipe and network backups. This fixes retention policies which were failing because Kopia saw each backup as coming from a different source path. Changes: - Add STAGING_BASE_DIR constant (/var/cache/kopi-docka/staging) - Add _prepare_staging_dir() helper method in BackupManager - Refactor _backup_recipes() to use stable path - Refactor _backup_networks() to use stable path - Add comprehensive unit tests (8 new tests in TestPrepareStagingDir) - Add integration test for Direct Mode retention - Fix existing tests to work with new staging implementation Benefits: - Kopia retention policies now work correctly for recipe/network snapshots - Prevents "ghost sessions" from accumulating in repositories - Cleaner restore interface (only actual backups shown) - Better debuggability (stable paths for inspection) - DRY code (eliminated duplicate directory preparation logic) All 74 backup_manager unit tests passing. Related to #40, #41 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * test: add integration tests for stable staging directory functionality Add comprehensive integration tests to verify that stable staging paths work correctly with real Docker containers, Kopia repository, and filesystem operations. Tests verify: - Staging directories are created at stable, predictable paths - Same directory is reused across multiple backups (not recreated) - Snapshots have consistent source paths (critical for retention policies) Three test scenarios: 1. test_staging_directory_created_with_stable_path - Verifies staging dir created at expected path - Verifies compose files copied to staging directory 2. test_staging_directory_reused_across_backups - Verifies same directory (same inode) reused - Verifies directory contents cleared and refreshed 3. test_snapshots_have_consistent_source_paths - Verifies all snapshots have identical source paths - Critical proof that retention policies can work correctly All tests use real Docker containers and Kopia filesystem repository. Tests require Docker daemon, root access, and Kopia binary. Properly marked as integration and slow tests. Related to #40, #41 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat: add repo-prune-empty-sessions command to clean up ghost sessions Add maintenance command to help users clean up legacy repositories that have accumulated "ghost sessions" (backup sessions with only recipe/network snapshots but no volume data). These ghost sessions accumulated when random temporary directories were used for metadata backups. Changes: - Add delete_snapshot() method to KopiaRepository class - Wraps kopia snapshot delete command - Supports unsafe-ignore-source flag for cross-machine cleanup - Add new CLI command: repo-prune-empty-sessions - Scans all snapshots and groups by backup_id - Identifies sessions with no volume snapshots - Displays results in Rich table (Backup ID, Recipes, Networks, Total) - Dry-run mode by default (safe preview) - Requires explicit --no-dry-run flag to delete - Asks for user confirmation before deletion - Shows progress bar during deletion - Suggests running maintenance afterward to reclaim space Usage: # Preview ghost sessions (dry-run) kopi-docka repo-prune-empty-sessions # Actually delete them kopi-docka repo-prune-empty-sessions --no-dry-run Benefits: - Helps clean up cluttered repositories - Improves restore UI (removes "Volumes: 0" entries) - Reclaims wasted metadata storage - Safe by default (dry-run mode) Related to #40, #41 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * chore(release): bump version to 5.3.0 - Update version in pyproject.toml, __init__.py, constants.py - Update CHANGELOG.md with comprehensive v5.3.0 release notes: - Critical fix: Direct Mode retention policies now work - Added: Stable staging paths for metadata backups - Added: repo-prune-empty-sessions command - Added: Massive test coverage improvements - Added: Documentation and CI infrastructure - Changed: Code quality improvements (run_command refactoring) - Fixed: Rclone backend improvements - Update documentation (USAGE.md, CONFIGURATION.md, ARCHITECTURE.md) - Update all version references from 5.2.x to 5.3.0 - Update CLAUDE.md and agent metadata to 5.3.0 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
77 lines
2.2 KiB
Makefile
77 lines
2.2 KiB
Makefile
# Makefile for Kopi-Docka
|
|
|
|
# Explicitly define the Python interpreter from our virtual environment.
|
|
# This makes sure 'make' always uses the right tools from the right workbench.
|
|
PYTHON := venv/bin/python3
|
|
|
|
.PHONY: all clean install install-dev install-system uninstall-system check-style format test build
|
|
|
|
# Default command when running 'make'
|
|
all: build
|
|
|
|
# Clean up build artifacts
|
|
clean:
|
|
rm -rf build/ dist/ *.egg-info .pytest_cache
|
|
|
|
# Install the package for production
|
|
install:
|
|
pip install .
|
|
|
|
# Install for development, including test tools
|
|
install-dev:
|
|
pip install -e ".[dev]"
|
|
|
|
# Run style and format checks
|
|
check-style:
|
|
$(PYTHON) -m flake8 kopi_docka/
|
|
$(PYTHON) -m black --check kopi_docka/
|
|
|
|
# Auto-format the code
|
|
format:
|
|
$(PYTHON) -m black kopi_docka/
|
|
|
|
# Run all tests
|
|
test:
|
|
$(PYTHON) -m pytest tests/
|
|
|
|
# Run only unit tests (fast)
|
|
test-unit:
|
|
$(PYTHON) -m pytest tests/unit/ -v
|
|
|
|
# Run only integration tests (slow)
|
|
test-integration:
|
|
$(PYTHON) -m pytest tests/integration/ -v
|
|
|
|
# Run tests with coverage report
|
|
test-coverage:
|
|
$(PYTHON) -m pytest tests/ --cov=kopi_docka --cov-report=html --cov-report=term-missing
|
|
|
|
# Run fast tests (unit only, no verbose)
|
|
test-fast:
|
|
$(PYTHON) -m pytest tests/unit/ -x --tb=short
|
|
|
|
# Run specific test file
|
|
test-file:
|
|
$(PYTHON) -m pytest $(FILE) -v
|
|
|
|
# Install system-wide wrapper (for sudo usage)
|
|
install-system:
|
|
@echo "Creating system wrapper for kopi-docka..."
|
|
@echo '#!/bin/bash' | sudo tee /usr/local/bin/kopi-docka > /dev/null
|
|
@echo 'exec $(shell pwd)/venv/bin/python -m kopi_docka "$$@"' | sudo tee -a /usr/local/bin/kopi-docka > /dev/null
|
|
@sudo chmod +x /usr/local/bin/kopi-docka
|
|
@echo "✓ Installed: /usr/local/bin/kopi-docka"
|
|
@echo "Usage: sudo kopi-docka [command]"
|
|
|
|
# Remove system-wide wrapper
|
|
uninstall-system:
|
|
@sudo rm -f /usr/local/bin/kopi-docka
|
|
@echo "✓ Removed: /usr/local/bin/kopi-docka"
|
|
|
|
# Build the standalone executable
|
|
build:
|
|
$(PYTHON) -m pyinstaller --onefile --name kopi-docka kopi_docka/__main__.py
|
|
@echo "Build complete. The executable is located in the 'dist' directory."
|
|
# Note: Ensure that PyInstaller is properly configured to include all necessary files and dependencies.
|
|
# You may need to create a spec file for more complex setups.
|