4ac58a5568
- CLAUDE.md: Project guidance for Claude Code - PROJECT_DOCUMENTATION.md: Complete project documentation - upload_log_file_fixed.sh: Fixed rclone upload functions with proper error handling - error_handling_comparison.sh: Documentation of rclone error handling patterns This repository contains utility scripts for managing nginx-proxy-manager log streaming to Cloudflare R2 storage, designed for CrowdSec integration.
123 lines
4.7 KiB
Markdown
123 lines
4.7 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
## Project Overview
|
|
|
|
This is a utility collection for managing nginx-proxy-manager log streaming to Cloudflare R2 storage. The project contains bash scripts and configuration helpers for setting up real-time log monitoring and upload systems, particularly designed for integration with CrowdSec security analysis.
|
|
|
|
## Key Components
|
|
|
|
### Log Upload Utilities (`upload_log_file_fixed.sh`)
|
|
|
|
Core functionality for reliable file uploads to Cloudflare R2 using rclone:
|
|
|
|
- **`upload_log_file()`** - Robust upload function with retry logic and proper error handling
|
|
- **`upload_log_file_minimal()`** - Simplified version focusing on exit code checking
|
|
- **`monitor_and_upload_logs()`** - Continuous monitoring loop for log directory surveillance
|
|
- **`test_rclone_upload()`** - Validation function for rclone configuration testing
|
|
|
|
**Key Implementation Details:**
|
|
- Uses rclone exit codes (0=success) rather than parsing output text for reliability
|
|
- Implements exponential backoff retry mechanism with configurable attempts
|
|
- Includes timeout protection (300s default) to prevent hanging operations
|
|
- Captures both stdout/stderr for detailed error reporting when failures occur
|
|
|
|
### Error Handling Patterns (`error_handling_comparison.sh`)
|
|
|
|
Documentation and examples showing:
|
|
- Common anti-patterns when working with rclone output parsing
|
|
- Why checking exit codes is more reliable than parsing "Transferred: X/Y" messages
|
|
- Best practices for monitoring loop stability (don't exit on individual upload failures)
|
|
|
|
## Remote Server Integration
|
|
|
|
The scripts are designed to work with a remote nginx-proxy-manager setup running in Podman containers on Debian systems. The typical deployment includes:
|
|
|
|
- **System tuning**: Kernel parameters optimized for container workloads and proxy traffic
|
|
- **Log streaming**: Real-time extraction from Podman containers and systemd journals
|
|
- **R2 integration**: Direct upload of uncompressed log files for CrowdSec consumption
|
|
- **Service automation**: systemd user services for continuous operation
|
|
|
|
## Development Commands
|
|
|
|
### Testing rclone Configuration
|
|
```bash
|
|
# Test basic upload functionality
|
|
./upload_log_file_fixed.sh
|
|
|
|
# Source the functions for interactive testing
|
|
source upload_log_file_fixed.sh
|
|
upload_log_file "/path/to/logfile" "cloudflare-r2:bucket/path"
|
|
```
|
|
|
|
### Remote Server Management
|
|
When working with the remote server deployment:
|
|
|
|
```bash
|
|
# Test log streaming script
|
|
ssh user@server "/home/user/scripts/npm-log-streamer.sh test"
|
|
|
|
# Manual log sync
|
|
ssh user@server "/home/user/scripts/npm-log-streamer.sh sync"
|
|
|
|
# Check service status
|
|
ssh user@server "systemctl --user status npm-log-streamer.service"
|
|
|
|
# View streaming logs
|
|
ssh user@server "journalctl --user -u npm-log-streamer.service -f"
|
|
```
|
|
|
|
### Cloudflare R2 Operations
|
|
```bash
|
|
# List uploaded files
|
|
rclone ls cloudflare-r2:npm-logs/
|
|
|
|
# Test connection
|
|
rclone lsd cloudflare-r2:
|
|
|
|
# Manual file upload
|
|
rclone copy localfile.log cloudflare-r2:npm-logs/path/
|
|
```
|
|
|
|
## Architecture Considerations
|
|
|
|
### Reliability Design
|
|
- **Fail-safe monitoring**: Individual upload failures don't terminate the monitoring service
|
|
- **Retry mechanisms**: Built-in exponential backoff for transient network issues
|
|
- **Timeout handling**: Prevents indefinite hangs on network problems
|
|
- **Resource cleanup**: Automatic cleanup of temporary files and connections
|
|
|
|
### Integration Points
|
|
- **Podman containers**: Log extraction from running nginx-proxy-manager containers
|
|
- **systemd integration**: User-level services for automatic startup and restart
|
|
- **CrowdSec compatibility**: Uncompressed log files uploaded in real-time for security analysis
|
|
- **R2 bucket organization**: Hierarchical structure with hostname/date organization
|
|
|
|
### Performance Characteristics
|
|
- **Upload frequency**: 1-minute intervals for real-time log availability
|
|
- **Batch processing**: Multiple log files processed in single sync cycle
|
|
- **Memory efficiency**: Streaming operations avoid large memory buffers
|
|
- **Network optimization**: Configurable retry counts and timeout values
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
- **TLS handshake failures**: Usually indicate incorrect R2 credentials or endpoint configuration
|
|
- **Exit code 126**: Typically permissions issues with script execution
|
|
- **Service restart loops**: Often caused by missing dependencies or configuration errors
|
|
|
|
### Debugging Commands
|
|
```bash
|
|
# Check rclone configuration
|
|
rclone config show cloudflare-r2
|
|
|
|
# Test direct R2 connection
|
|
rclone lsd cloudflare-r2: --verbose
|
|
|
|
# Verify systemd service configuration
|
|
systemctl --user status npm-log-streamer.service --no-pager
|
|
|
|
# Check recent service logs
|
|
journalctl --user -u npm-log-streamer.service --since "1 hour ago"
|
|
``` |