Logging and Monitoring

Project Code：https://github.com/hyperlane-dev/hyperlane

Observability is a critical aspect of running production web servers. Without proper logging and monitoring, diagnosing performance issues, tracking errors, and understanding traffic patterns becomes nearly impossible. In this article, we'll explore how to set up comprehensive logging and monitoring for Hyperlane applications using the hyperlane-log ecosystem.

The Importance of Logging

Logging serves multiple purposes in a production system:

• Debugging: When something goes wrong, logs provide the context needed to understand what happened and why.
• Auditing: Logs create a record of who did what and when, which is essential for security compliance.
• Performance analysis: By logging request durations and response sizes, you can identify slow endpoints and optimize them.
• Alerting: Monitoring systems can watch logs for error patterns and trigger alerts when anomalies are detected.

Introduction to hyperlane-log

The hyperlane-log crate is the recommended logging solution for Hyperlane applications. It provides structured logging with configurable output formats, log levels, and output destinations. Unlike unstructured logging (plain text messages), structured logging produces logs in a machine-readable format like JSON, making them easy to parse and analyze.

Key Features

• Configurable log levels: TRACE, DEBUG, INFO, WARN, ERROR
• Multiple output destinations: stdout, file, syslog
• Structured output: JSON-formatted log entries
• Contextual information: Automatic inclusion of request IDs, timestamps, and source locations
• Performance: Minimal overhead with async logging

Setting Up Basic Logging

Getting started with hyperlane-log is straightforward. Add it to your project and initialize it at the start of your application:

use hyperlane::*;

#[tokio::main]
async fn main() {
    // Initialize the logging system
    // This sets up a basic logger that outputs to stdout
    hyperlane::init();

    let server = Server::default();

    server.route::<IndexRoute>("/").await;
    server.route::<UsersRoute>("/api/users").await;

    server.run().await;
}

The hyperlane::init() function sets up the global logger with sensible defaults. Once initialized, all log messages throughout the application will be captured and formatted consistently.

Structured Logging in Route Handlers

Within your route handlers, you can use the standard Rust logging macros to emit log entries. These macros automatically include contextual information:

use hyperlane::*;

#[route("/api/users/{id}")]
async fn get_user(ctx: &mut Context) -> Result<(), RequestError> {
    let user_id = ctx.get_route_param("id");

    // Log at different levels
    log::info!("Fetching user with ID: {}", user_id);

    let user = fetch_user(user_id).await?;

    log::debug!("User data: {:?}", user);

    ctx.get_mut_response()
        .set_status_code(200)
        .set_body(serde_json::to_string(&user).unwrap());

    Ok(())
}

The log::info! and log::debug! macros produce structured log entries that include the timestamp, log level, source file and line number, and the message itself.

Logging in Middleware

Middleware is an excellent place to add request-level logging. You can log every request that passes through your server, capturing important metadata:

use hyperlane::*;

#[request_middleware(1)]
async fn request_logger(ctx: &mut Context) -> MiddlewareResult {
    let request = ctx.get_request();
    let method = request.get_method();
    let path = request.get_path();
    let host = request.get_host();

    log::info!(
        "Incoming request: {} {} (Host: {})",
        method, path, host
    );

    // Record the start time for duration calculation
    let start = std::time::Instant::now();

    // Continue to the next middleware or handler
    let result = ctx.next().await;

    let duration = start.elapsed();
    let status = ctx.get_response().get_status_code();

    log::info!(
        "Response: {} {} -> {} ({:?})",
        method, path, status, duration
    );

    result
}

This middleware logs both the incoming request details and the response status with timing information. The ctx.next().await call passes control to the next middleware in the chain or the route handler, and when it returns, we can measure the elapsed time.

Error Logging

Hyperlane provides dedicated error handling through the request_error and task_panic middleware types. These are ideal locations for error logging:

use hyperlane::*;

#[request_error]
async fn error_handler(ctx: &mut Context) {
    if let Some(error_data) = ctx.try_get_request_error_data() {
        log::error!(
            "Request error occurred: {:?}",
            error_data
        );
    }
}

#[task_panic]
async fn panic_handler(ctx: &mut Context) {
    if let Some(panic_data) = ctx.try_get_task_panic_data() {
        log::error!(
            "Task panic occurred: {:?}",
            panic_data
        );
    }
}

The try_get_request_error_data() and try_get_task_panic_data() methods return Option<RequestError> and Option<PanicData> respectively, allowing you to inspect error details and log them appropriately.

Monitoring Key Metrics

Beyond logging, monitoring numerical metrics is essential for understanding your server's health. Here are the key metrics to track:

Request Rate

Track the number of requests per second. Hyperlane's benchmark data shows what's achievable:

• Keep-Alive disabled: 51,031 QPS
• Keep-Alive enabled: 334,888 QPS
• ab 1 million requests: 316,211 QPS

These numbers serve as reference points for your own application's performance.

Response Time

Monitor response time percentiles (p50, p95, p99). The p99 latency tells you how slow the worst 1% of requests are, which is often more important than the average.

Error Rate

Track the percentage of requests that result in 4xx or 5xx responses. A sudden spike in error rates often indicates a deployment issue or upstream service failure.

Connection Count

Monitor the number of active connections. This helps you understand your server's concurrency level and identify connection leaks.

Implementing a Metrics Endpoint

You can expose a /metrics endpoint that returns current server statistics:

use hyperlane::*;

#[route("/metrics")]
async fn metrics(ctx: &mut Context) -> Result<(), RequestError> {
    let stats = collect_server_stats();

    let response = ctx.get_mut_response();
    response.set_status_code(200);
    response.add_header("Content-Type", "application/json");
    response.set_body(serde_json::to_string_pretty(&stats).unwrap());

    Ok(())
}

fn collect_server_stats() -> serde_json::Value {
    serde_json::json!({
        "server": "hyperlane",
        "uptime_seconds": get_uptime(),
        "active_connections": get_connection_count(),
        "total_requests": get_total_requests(),
        "requests_per_second": get_rps(),
        "error_rate": get_error_rate(),
    })
}

Logging Configuration for Production

In production, you'll want more control over logging behavior:

Log Rotation

For file-based logging, configure log rotation to prevent disk space exhaustion:

use hyperlane::*;

#[tokio::main]
async fn main() {
    // Configure logging with file output and rotation
    // Logs are written to files with daily rotation
    hyperlane::init();

    let server = Server::default();
    server.run().await;
}

Environment-Based Configuration

Use environment variables to control log levels without recompiling:

use hyperlane::*;

#[tokio::main]
async fn main() {
    // Log level can be configured via environment variables
    // Default to INFO if not specified
    hyperlane::init();

    let server = Server::default();
    server.run().await;
}

Docker and Container Monitoring

When deploying Hyperlane in Docker containers, you can use the provided Docker Compose configurations:

# Development environment
docker compose -f ./resources/docker/dev/server_docker_compose.yml up -d

# Production environment
docker compose -f ./resources/docker/release/server_docker_compose.yml up -d

In containerized environments, direct logs to stdout/stderr so that Docker's logging driver can capture them:

use hyperlane::*;

#[tokio::main]
async fn main() {
    // In containerized environments, log to stdout
    // Docker will capture and route these logs
    hyperlane::init();

    let server = Server::default();
    server.run().await;
}

Structured Error Responses

When errors occur, return structured error responses that include enough information for debugging without leaking sensitive details:

use hyperlane::*;

#[request_error]
async fn structured_error_handler(ctx: &mut Context) {
    if let Some(error_data) = ctx.try_get_request_error_data() {
        log::error!("Request error: {:?}", error_data);

        let response = ctx.get_mut_response();
        response.set_status_code(500);
        response.add_header("Content-Type", "application/json");
        response.set_body(
            serde_json::json!({
                "error": "Internal Server Error",
                "message": "An unexpected error occurred",
            }).to_string()
        );
    }
}

Best Practices for Production Logging

1. Use appropriate log levels: Reserve ERROR for actual problems, WARN for concerning but recoverable conditions, INFO for significant events, and DEBUG for development-time diagnostics.

2. Include request context: Always log enough context to trace a request through the system. Route parameters, method, and path are essential.

3. Avoid logging sensitive data: Never log passwords, tokens, or personal information. Be especially careful with request bodies and headers.

4. Use structured logging: JSON-formatted logs are easier to search and analyze in log aggregation systems.

5. Set up log rotation: Unbounded log files will eventually fill your disk. Configure rotation based on file size or time.

6. Monitor log volume: A sudden increase in log volume often indicates a problem. Set up alerts for unusual log patterns.

7. Correlate logs with metrics: When investigating issues, having both logs and metrics for the same time period makes diagnosis much faster.

Conclusion

Logging and monitoring are essential components of a production Hyperlane deployment. The hyperlane-log crate provides a solid foundation for structured logging, while Hyperlane's middleware system makes it easy to add request-level logging and error tracking.

By implementing the patterns described in this article — structured logging in route handlers, request logging middleware, error logging, and metrics collection — you'll have the observability you need to run Hyperlane applications confidently in production.

Remember that good logging is not about logging everything; it's about logging the right things. Focus on the information that helps you understand your application's behavior, diagnose issues quickly, and make informed decisions about performance optimization.

---

Project Code：https://github.com/hyperlane-dev/hyperlane