# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

This is a utility collection for managing nginx-proxy-manager log streaming to Cloudflare R2 storage. The project contains bash scripts and configuration helpers for setting up real-time log monitoring and upload systems, particularly designed for integration with CrowdSec security analysis.

## Key Components

### Log Upload Utilities (`upload_log_file_fixed.sh`)

Core functionality for reliable file uploads to Cloudflare R2 using rclone:

- **`upload_log_file()`** - Robust upload function with retry logic and proper error handling
- **`upload_log_file_minimal()`** - Simplified version focusing on exit code checking
- **`monitor_and_upload_logs()`** - Continuous monitoring loop for log directory surveillance
- **`test_rclone_upload()`** - Validation function for rclone configuration testing

**Key Implementation Details:**
- Uses rclone exit codes (0=success) rather than parsing output text for reliability
- Implements exponential backoff retry mechanism with configurable attempts
- Includes timeout protection (300s default) to prevent hanging operations
- Captures both stdout/stderr for detailed error reporting when failures occur

### Error Handling Patterns (`error_handling_comparison.sh`)

Documentation and examples showing:
- Common anti-patterns when working with rclone output parsing
- Why checking exit codes is more reliable than parsing "Transferred: X/Y" messages
- Best practices for monitoring loop stability (don't exit on individual upload failures)

## Remote Server Integration

The scripts are designed to work with a remote nginx-proxy-manager setup running in Podman containers on Debian systems. The typical deployment includes:

- **System tuning**: Kernel parameters optimized for container workloads and proxy traffic
- **Log streaming**: Real-time extraction from Podman containers and systemd journals  
- **R2 integration**: Direct upload of uncompressed log files for CrowdSec consumption
- **Service automation**: systemd user services for continuous operation

## Development Commands

### Testing rclone Configuration
```bash
# Test basic upload functionality
./upload_log_file_fixed.sh

# Source the functions for interactive testing
source upload_log_file_fixed.sh
upload_log_file "/path/to/logfile" "cloudflare-r2:bucket/path"
```

### Remote Server Management
When working with the remote server deployment:

```bash
# Test log streaming script
ssh user@server "/home/user/scripts/npm-log-streamer.sh test"

# Manual log sync
ssh user@server "/home/user/scripts/npm-log-streamer.sh sync"

# Check service status
ssh user@server "systemctl --user status npm-log-streamer.service"

# View streaming logs
ssh user@server "journalctl --user -u npm-log-streamer.service -f"
```

### Cloudflare R2 Operations
```bash
# List uploaded files
rclone ls cloudflare-r2:npm-logs/

# Test connection
rclone lsd cloudflare-r2:

# Manual file upload
rclone copy localfile.log cloudflare-r2:npm-logs/path/
```

## Architecture Considerations

### Reliability Design
- **Fail-safe monitoring**: Individual upload failures don't terminate the monitoring service
- **Retry mechanisms**: Built-in exponential backoff for transient network issues
- **Timeout handling**: Prevents indefinite hangs on network problems
- **Resource cleanup**: Automatic cleanup of temporary files and connections

### Integration Points
- **Podman containers**: Log extraction from running nginx-proxy-manager containers
- **systemd integration**: User-level services for automatic startup and restart
- **CrowdSec compatibility**: Uncompressed log files uploaded in real-time for security analysis
- **R2 bucket organization**: Hierarchical structure with hostname/date organization

### Performance Characteristics
- **Upload frequency**: 1-minute intervals for real-time log availability
- **Batch processing**: Multiple log files processed in single sync cycle
- **Memory efficiency**: Streaming operations avoid large memory buffers
- **Network optimization**: Configurable retry counts and timeout values

## Troubleshooting

### Common Issues
- **TLS handshake failures**: Usually indicate incorrect R2 credentials or endpoint configuration
- **Exit code 126**: Typically permissions issues with script execution
- **Service restart loops**: Often caused by missing dependencies or configuration errors

### Debugging Commands
```bash
# Check rclone configuration
rclone config show cloudflare-r2

# Test direct R2 connection
rclone lsd cloudflare-r2: --verbose

# Verify systemd service configuration
systemctl --user status npm-log-streamer.service --no-pager

# Check recent service logs
journalctl --user -u npm-log-streamer.service --since "1 hour ago"
```