Access Log Plugin

The Access Log plugin provides comprehensive HTTP access logging with multiple format options and flexible output destinations.

Overview

The Access Log plugin is a critical component for monitoring and analyzing HTTP traffic in Rusty Beam. It captures detailed information about every request and response, supporting standard log formats (Common, Combined) as well as structured JSON logging for modern log analysis pipelines. The plugin is designed for high performance with minimal impact on request processing.

Key Features

Multiple Log Formats: Common Log Format (CLF), Combined Log Format, and JSON
Flexible Output: Log to file or stdout
Real Client IP Detection: Handles proxy headers (X-Forwarded-For, X-Real-IP, CF-Connecting-IP)
Authenticated User Tracking: Logs authenticated usernames from auth plugins
Automatic Directory Creation: Creates log directories if they don't exist
Log Buffering: Configurable buffering for improved I/O performance
Log Rotation: Size-based and daily rotation support
Request Timing: Tracks request processing time in milliseconds
Error Resilience: Continues serving even if logging fails

Configuration

To add the Access Log plugin to your pipeline:

<td itemprop="plugin" itemscope itemtype="https://rustybeam.net/schema/AccessLogPlugin">
    <div class="plugin-config">
        <div class="config-property">
            <label class="property-label">library:</label>
            <span itemprop="library" class="property-value">file://./plugins/librusty_beam_access_log.so</span>
        </div>
        <div class="config-property">
            <label class="property-label">log_file:</label>
            <span itemprop="log_file" class="property-value">./logs/access.log</span>
        </div>
        <div class="config-property">
            <label class="property-label">format:</label>
            <span itemprop="format" class="property-value">combined</span>
        </div>
    </div>
</td>

Configuration Parameters

Parameter	Type	Required	Default	Description
`log_file`	String	No	stdout	Path to log file (e.g., "/var/log/rusty-beam/access.log")
`format`	String	No	common	Log format: "common", "combined", or "json"
`buffer_size`	Number	No	1	Number of entries to buffer before writing to disk
`rotate_size_mb`	Number	No	0	Rotate log when it reaches this size in MB (0 = disabled)
`rotate_daily`	Boolean	No	false	Enable daily log rotation

Plugin Pipeline Placement: The Access Log plugin should be placed early in the plugin pipeline to ensure all requests are logged, even those that might be rejected by authentication or other plugins. However, it must be placed after any plugins that set the authenticated_user metadata if you want to log authenticated usernames.

Log Formats

Common Log Format (CLF)

The default format, widely supported by log analysis tools:

127.0.0.1 - alice [10/Oct/2024:13:55:36 +0000] "GET /index.html HTTP/1.1" 200 2326

Fields: remote_ip - user [timestamp] "method uri version" status size

Combined Log Format

Extends Common Log Format with referer and user agent:

127.0.0.1 - alice [10/Oct/2024:13:55:36 +0000] "GET /index.html HTTP/1.1" 200 2326 "http://example.com/" "Mozilla/5.0"

Additional fields: referer user_agent

JSON Format

Structured format for modern log processing pipelines:

{
  "timestamp": "10/Oct/2024:13:55:36 +0000",
  "remote_ip": "127.0.0.1",
  "user": "alice",
  "method": "GET",
  "uri": "/index.html",
  "version": "HTTP/1.1",
  "status": 200,
  "size": 2326,
  "user_agent": "Mozilla/5.0",
  "referer": "http://example.com/",
  "request_time_ms": 42
}

Client IP Detection

The plugin checks the following headers in order to determine the real client IP:

X-Forwarded-For - Standard proxy header (first IP if multiple)
X-Real-IP - Common nginx header
X-Client-IP - Alternative proxy header
CF-Connecting-IP - Cloudflare header
True-Client-IP - Cloudflare Enterprise header

If no proxy headers are found, the plugin falls back to the direct connection IP.

Log Rotation

Size-Based Rotation

Configure rotate_size_mb to automatically rotate logs when they reach a certain size:

<span itemprop="rotate_size_mb">100</span> <!-- Rotate at 100MB -->

Rotated files are named: access.log.20241010_135536

Daily Rotation

Enable rotate_daily for automatic daily log rotation:

<span itemprop="rotate_daily">true</span>

Performance Optimization

Buffering

Increase buffer_size to reduce disk I/O:

<span itemprop="buffer_size">100</span> <!-- Buffer 100 entries -->

Note: Buffered entries are automatically flushed when the buffer is full or when the server shuts down gracefully. In case of crashes, buffered entries may be lost.

Examples

Basic File Logging

<td itemprop="plugin" itemscope itemtype="https://rustybeam.net/schema/AccessLogPlugin">
    <span itemprop="log_file">./logs/access.log</span>
</td>

JSON Logging with Rotation

<td itemprop="plugin" itemscope itemtype="https://rustybeam.net/schema/AccessLogPlugin">
    <span itemprop="log_file">/var/log/rusty-beam/access.json</span>
    <span itemprop="format">json</span>
    <span itemprop="rotate_size_mb">50</span>
    <span itemprop="rotate_daily">true</span>
</td>

High-Performance Configuration

<td itemprop="plugin" itemscope itemtype="https://rustybeam.net/schema/AccessLogPlugin">
    <span itemprop="log_file">/var/log/rusty-beam/access.log</span>
    <span itemprop="format">common</span>
    <span itemprop="buffer_size">1000</span>
    <span itemprop="rotate_size_mb">1000</span>
</td>

Integration with Other Plugins

Basic Auth Plugin: Set authenticated_user metadata for user logging
OAuth Plugin: Set authenticated_user for OAuth identities
Rate Limit Plugin: Analyze access logs for rate limiting decisions
Error Handler Plugin: Error responses are logged with appropriate status codes
Directory Plugin: Logs show full request paths for nested routing

Log Analysis

Common tools for analyzing access logs:

GoAccess: Real-time web log analyzer
AWStats: Advanced web statistics
Elasticsearch/Logstash: For JSON format logs
grep/awk: Command-line analysis

Example: Find Top IPs

# Using awk with Common Log Format
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10

# Using jq with JSON format
jq -r .remote_ip access.json | sort | uniq -c | sort -rn | head -10

Troubleshooting

Issue	Possible Cause	Solution
No logs being written	Permission issues or invalid path	Check file permissions and ensure parent directory exists
Wrong client IPs logged	Proxy headers not configured	Ensure proxy is sending appropriate headers
Missing user information	Auth plugin not setting metadata	Place access log after authentication plugins
High memory usage	Buffer size too large	Reduce buffer_size configuration
Logs not rotating	File permissions or size calculation	Check write permissions for log directory

Security Considerations

Log File Permissions: Ensure log files are not world-readable if they contain sensitive data
Log Injection: The plugin properly escapes special characters to prevent log injection attacks
Disk Space: Monitor disk usage and configure rotation to prevent filling the disk
Privacy: Consider data retention policies and GDPR compliance for IP addresses

Privacy Warning: Access logs contain IP addresses and potentially other personal information. Ensure compliance with privacy regulations in your jurisdiction and implement appropriate data retention policies.