cleanup: Remove all legacy files, keep only PROJECT_DOCUMENTATION.md
Removed old log streaming utilities and sample logs to start fresh: - CLAUDE.md (legacy project instructions) - README.md (old readme) - error_handling_comparison.sh (rclone utility) - upload_log_file_fixed.sh (R2 upload script) - logs/ directory (sample log files) PROJECT_DOCUMENTATION.md retained as the single source of truth for the new Podman + Quadlet automation system. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
kappa
123
CLAUDE.md
123
CLAUDE.md
@@ -1,123 +0,0 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Project Overview
|
||||
|
||||
This is a utility collection for managing nginx-proxy-manager log streaming to Cloudflare R2 storage. The project contains bash scripts and configuration helpers for setting up real-time log monitoring and upload systems, particularly designed for integration with CrowdSec security analysis.
|
||||
|
||||
## Key Components
|
||||
|
||||
### Log Upload Utilities (`upload_log_file_fixed.sh`)
|
||||
|
||||
Core functionality for reliable file uploads to Cloudflare R2 using rclone:
|
||||
|
||||
- **`upload_log_file()`** - Robust upload function with retry logic and proper error handling
|
||||
- **`upload_log_file_minimal()`** - Simplified version focusing on exit code checking
|
||||
- **`monitor_and_upload_logs()`** - Continuous monitoring loop for log directory surveillance
|
||||
- **`test_rclone_upload()`** - Validation function for rclone configuration testing
|
||||
|
||||
**Key Implementation Details:**
|
||||
- Uses rclone exit codes (0=success) rather than parsing output text for reliability
|
||||
- Implements exponential backoff retry mechanism with configurable attempts
|
||||
- Includes timeout protection (300s default) to prevent hanging operations
|
||||
- Captures both stdout/stderr for detailed error reporting when failures occur
|
||||
|
||||
### Error Handling Patterns (`error_handling_comparison.sh`)
|
||||
|
||||
Documentation and examples showing:
|
||||
- Common anti-patterns when working with rclone output parsing
|
||||
- Why checking exit codes is more reliable than parsing "Transferred: X/Y" messages
|
||||
- Best practices for monitoring loop stability (don't exit on individual upload failures)
|
||||
|
||||
## Remote Server Integration
|
||||
|
||||
The scripts are designed to work with a remote nginx-proxy-manager setup running in Podman containers on Debian systems. The typical deployment includes:
|
||||
|
||||
- **System tuning**: Kernel parameters optimized for container workloads and proxy traffic
|
||||
- **Log streaming**: Real-time extraction from Podman containers and systemd journals
|
||||
- **R2 integration**: Direct upload of uncompressed log files for CrowdSec consumption
|
||||
- **Service automation**: systemd user services for continuous operation
|
||||
|
||||
## Development Commands
|
||||
|
||||
### Testing rclone Configuration
|
||||
```bash
|
||||
# Test basic upload functionality
|
||||
./upload_log_file_fixed.sh
|
||||
|
||||
# Source the functions for interactive testing
|
||||
source upload_log_file_fixed.sh
|
||||
upload_log_file "/path/to/logfile" "cloudflare-r2:bucket/path"
|
||||
```
|
||||
|
||||
### Remote Server Management
|
||||
When working with the remote server deployment:
|
||||
|
||||
```bash
|
||||
# Test log streaming script
|
||||
ssh user@server "/home/user/scripts/npm-log-streamer.sh test"
|
||||
|
||||
# Manual log sync
|
||||
ssh user@server "/home/user/scripts/npm-log-streamer.sh sync"
|
||||
|
||||
# Check service status
|
||||
ssh user@server "systemctl --user status npm-log-streamer.service"
|
||||
|
||||
# View streaming logs
|
||||
ssh user@server "journalctl --user -u npm-log-streamer.service -f"
|
||||
```
|
||||
|
||||
### Cloudflare R2 Operations
|
||||
```bash
|
||||
# List uploaded files
|
||||
rclone ls cloudflare-r2:npm-logs/
|
||||
|
||||
# Test connection
|
||||
rclone lsd cloudflare-r2:
|
||||
|
||||
# Manual file upload
|
||||
rclone copy localfile.log cloudflare-r2:npm-logs/path/
|
||||
```
|
||||
|
||||
## Architecture Considerations
|
||||
|
||||
### Reliability Design
|
||||
- **Fail-safe monitoring**: Individual upload failures don't terminate the monitoring service
|
||||
- **Retry mechanisms**: Built-in exponential backoff for transient network issues
|
||||
- **Timeout handling**: Prevents indefinite hangs on network problems
|
||||
- **Resource cleanup**: Automatic cleanup of temporary files and connections
|
||||
|
||||
### Integration Points
|
||||
- **Podman containers**: Log extraction from running nginx-proxy-manager containers
|
||||
- **systemd integration**: User-level services for automatic startup and restart
|
||||
- **CrowdSec compatibility**: Uncompressed log files uploaded in real-time for security analysis
|
||||
- **R2 bucket organization**: Hierarchical structure with hostname/date organization
|
||||
|
||||
### Performance Characteristics
|
||||
- **Upload frequency**: 1-minute intervals for real-time log availability
|
||||
- **Batch processing**: Multiple log files processed in single sync cycle
|
||||
- **Memory efficiency**: Streaming operations avoid large memory buffers
|
||||
- **Network optimization**: Configurable retry counts and timeout values
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
- **TLS handshake failures**: Usually indicate incorrect R2 credentials or endpoint configuration
|
||||
- **Exit code 126**: Typically permissions issues with script execution
|
||||
- **Service restart loops**: Often caused by missing dependencies or configuration errors
|
||||
|
||||
### Debugging Commands
|
||||
```bash
|
||||
# Check rclone configuration
|
||||
rclone config show cloudflare-r2
|
||||
|
||||
# Test direct R2 connection
|
||||
rclone lsd cloudflare-r2: --verbose
|
||||
|
||||
# Verify systemd service configuration
|
||||
systemctl --user status npm-log-streamer.service --no-pager
|
||||
|
||||
# Check recent service logs
|
||||
journalctl --user -u npm-log-streamer.service --since "1 hour ago"
|
||||
```
|
||||
Reference in New Issue
Block a user