How to set up Rust logging in AWS Lambda for AWS CloudWatch

AWS Lambda automatically sends logs to AWS CloudWatch, but how do we make sure our Rust logs are formatted correctly? The AWS official documentation explains the basic setup with the Tracing crate. In this article, we dive deeper into how to set up Rust logging outputs properly.

JSON format setup

The most common JSON format for logging is bunyan. When using bunyan, we can use the standard tooling for processing it, such as the Rust CLI port.

The crate tracing-bunyan-formatter lets us set up the bunyan log output format with the tracing framework. We assume we have the Cargo.toml dependencies set up:

[dependencies]
# ...
tracing = "0.1"
tracing-bunyan-formatter = "0.3"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }

We can either hard-code the package name or use the CARGO_PKG_NAME environment variable or the module_path!() macro:

    let package_name = package_name
        .split("::")
        .next()
        .expect("splitn should always return at least one element")
        .replace('-', "_");

Next, we can check if the RUST_LOG environment variable exists. If it doesn’t, we can set it to a default value. Here’s how we can do that:

    let env_filter = if std::env::var("RUST_LOG").is_err() {
        EnvFilter::builder().parse_lossy("info,aws=warn")
    } else {
        EnvFilter::from_default_env()
    };
    let formatting_layer =
        BunyanFormattingLayer::new(package_name.to_string(), std::io::stdout);
    let subscriber = Registry::default()
        .with(env_filter)
        .with(JsonStorageLayer)
        .with(formatting_layer);

As with the standard tracing setup, if we want to capture SpanTraces, we can add the ErrorLayer:

    let mut subscriber = subscriber.with(tracing_error::ErrorLayer::default());

To customize the logging setup for local testing, we can check the AWS_LAMBDA_RUNTIME_API environment variable. If it’s not set, we can use a different logging format instead of the one used on AWS Lambda. Here’s an example:

    if std::env::var("AWS_LAMBDA_RUNTIME_API").is_err() {
        subscriber = Registry::default()
            .with(EnvFilter::from_default_env())
            .with(tracing_subscriber::fmt::layer());
    }

And finally, we set this subscriber as the global one:

    tracing::subscriber::set_global_default(subscriber).unwrap();

Stack overflow from enabling debug logging

Even something as simple as logging can cause complex issues. For us, it was when we enabled debug logging and Rust-based AWS Lambdas suddenly started crashing with stack overflows. The root cause was an issue here about debug logging with the BunyanFormattingLayer potentially causing an infinite loop.

The debug log statements have been removed from the tracing-bunyan-formatter, so you should not encounter this issue anymore. If you are still worried about it being reintroduced, you can disable logging of the tracing_bunyan_formatter itself:

    let bunyan_logging_off = "tracing_bunyan_formatter::formatting_layer=off"
        .parse()
        .expect("invalid directive");
    let subscriber = Registry::default()
        .with(EnvFilter::from_default_env().add_directive(bunyan_logging_off))
        .with(JsonStorageLayer)
        .with(formatting_layer);

Excluding secrets

Now, let’s talk about excluding certain data from the logs, such as API keys or personal identifiable information. We can do this by creating wrappers around types that we don’t want to accidentally leak in the logs. These wrapper types don’t implement the Display or Debug traits or implement them with a fixed string, like this:

pub struct Secret<T>(T);

impl<T> Debug for Secret<T> {
    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
        write!(f, "*** redacted ***")
    }
}

One way to do this is using the wrapper types from the secrecy crate. This crate also ensures that the content is properly zeroed out when the value goes out of scope.

Error logging

When logging errors, using the alternate display format {:#} is generally preferred, because it contains the full source error information. For example, in AWS SDK, a lot of errors fall under the Unhandled variant (e.g. AccessDeniedException) where the default error format does not show the error context, while the alternate display contains the error metadata (e.g. the actual resources being accessed) as with DisplayErrorContext.

One common issue happens with the popular thiserror crate. The thiserror crate currently only supports the default {} display format, which loses the error context. One workaround for this is to wrap the errors in anyhow or eyre that support the alternate display format.

Conclusion

While AWS CloudWatch captures AWS Lambda logging outputs automatically, there are a few things to keep in mind. First, make sure your logging outputs are in the right format, such as bunyan. Second, avoid leaking sensitive info such as API keys in your logs. Third, when logging errors, try to include as much context as possible to help diagnose them. With these tips, you can set up your logging with some peace of mind.