CloudWatch to Loki via Lambda: cutting log spend without losing fidelity

This is a write-up of a small-team project from a couple of years ago. The team was a few engineers, the AWS footprint was a handful of accounts rather than dozens, and several of the design choices below — credentials baked into Lambda env vars at deploy time, no DLQ, one Loki stream per log line — are sized for that context and would be wrong for a larger one. I’m calling them out as we go rather than pretending the post is a generic recipe.

CloudWatch was fine for the first while, just expensive and awkward to query. The bill scaled in a way that didn’t match the value we were getting, and CloudWatch Logs Insights is account-scoped, so any cross-account question — “what was every service doing in the hour before the alert fired?” — needed N tabs, N queries, and manual stitching. We already ran Grafana for metrics; the obvious move was to land logs there too, behind the same query surface engineers were already using.

The fix was a centralised log backend. We picked Loki because the team was already running Grafana for metrics, the LogQL syntax was familiar to anyone who’d used PromQL, and the storage model — chunked logs labelled with metadata, indexed by labels rather than full-text — fit the way our engineers actually searched. The piece this article is about is the bridge: a serverless Lambda that subscribed to CloudWatch log groups and pushed events into Loki via its HTTP push API.

The architecture in one sentence

Each CloudWatch log group had a subscription filter that fired a Lambda on every batch. The Lambda gunzipped the events, extracted a few well-known JSON fields as Loki labels, formatted a push payload, and POSTed it to Loki. That’s the whole shape.

Log group → Subscription filter → Lambda → HTTP POST → Loki
                                     │
                                     ├─ gunzip + b64 decode
                                     ├─ jmespath label extraction
                                     ├─ ns timestamp conversion
                                     └─ stream-per-line formatting

No queue, no DLQ, no retry layer of our own. CloudWatch’s subscription delivery has its own retry, and we accepted that if Loki was down long enough for those retries to exhaust, we’d have a bigger problem on our hands than missing log lines. At our scale this trade was right; in a larger setup the missed-logs cost would justify the extra moving parts.

The decode dance

CloudWatch sent events to subscribers as base64-encoded gzip payloads — historical reasons, presumably. The handler entry had to decode this before doing anything else:

def _decode_log_data(event: dict) -> dict:
    compressed = base64.b64decode(event["awslogs"]["data"])
    return json.loads(gzip.decompress(compressed))

The decoded payload contained a logGroup, the originating logStream, and a logEvents array — each event with a millisecond timestamp and the message itself. Most of our messages were JSON; some weren’t. The handler tried to parse, fell back to treating the message as a plain string, and moved on.

Decision: extract labels from the message body, not the log group

The most important call in the design was what to use as Loki labels. Loki indexes by labels — they’re cheap to filter on, expensive to vary on (each unique combination creates a new stream). Pick a high-cardinality label and the active stream count climbs faster than you’d expect, eventually past tenant limits or self-hosted-cluster comfort levels.

The obvious labels (logGroup, logStream) are static across an event batch, but they only let you slice by infrastructure, not by what the application is actually doing. So the handler also pulled a configurable list of content labels out of each JSON message:

def _stream_labels(label_paths: list[str], doc: dict) -> dict:
    labels = {}
    for path in label_paths:
        value = jmespath.search(path, doc)
        if value is None:
            continue
        # Loki forbids dots in label names
        labels[path.replace(".", "_")] = str(value)
    return labels

The default extraction included level, hostname, and res.statusCode — that last one a jmespath expression that drilled into a nested response object. The substitution from . to _ mattered: Loki rejects dots in label names, and finding that out at runtime in a 3am incident is not the way to find out.

The implication was that the handler created one Loki stream per log line rather than batching lines with the same labels into a single stream. A deliberate cardinality trade-off: simpler code, no merge logic, every line carrying its own labels. The metric that matters for Loki here isn’t events per second but the number of distinct label combinations active in the same time window — for our setup (one log group, a handful of level values, a handful of hostname values, a small set of res.statusCode values) the active stream count stayed in the low hundreds, well under Grafana Cloud’s tenant limits. The pattern would have scaled badly if we’d added a high-cardinality label like userId or requestId; the framework didn’t stop you from doing that, the operational discipline did. The follow-on plan, never needed in our case, was to batch lines with identical label sets within an event batch if active stream count ever climbed past a few thousand.

Decision (with hindsight): bake secrets at deploy time, not at invocation

Loki’s HTTP push API takes basic auth credentials. The naive path is to fetch them from Secrets Manager on every invocation. We didn’t:

# At deploy time, in the Pulumi program:
loki_secret = secrets_client.get_secret_value(SecretId=f"{env}/loki")
loki_creds = json.loads(loki_secret["SecretString"])

lambda_func = aws.lambda_.Function(
    "log-shipper",
    environment={
        "variables": {
            "LOKI_ENDPOINT": loki_creds["endpoint"],
            "LOKI_USER": loki_creds["userid"],
            "LOKI_TOKEN": loki_creds["token"],
        }
    },
    ...
)

The credentials ended up baked into the Lambda’s environment variables at deploy time. The Secrets Manager API call happened once per pulumi up, not once per Lambda invocation. At ~10 invocations per second that saved hundreds of dollars a year in secretsmanager:GetSecretValue calls and shaved a few hundred milliseconds off the cold start, at the cost of needing a redeploy to rotate the Loki credentials.

Calling this honestly: it’s the wrong shape, and I wouldn’t ship it again. The credential ends up visible to anyone with lambda:GetFunction on the function’s IAM resource, and Lambda environment variables aren’t encrypted at rest with a customer-managed KMS key unless you configure one explicitly. The right approach is the one we should have built in the first place — fetch the secret from Secrets Manager at module load and cache it in module scope; Lambda’s container reuse amortises the cost across the warm container’s lifetime, so you keep the cold-start saving without putting the credential into an IAM-readable env var. The reason we didn’t was that “no cache for secrets yet” was a TODO that never bubbled up the queue, and the Loki write token’s blast radius (send junk into our own log store) made the operational cost of leaving it tolerable. Tolerable isn’t the same as right; this part of the design is on the list of things to redo, and a fresh build would do it that way from day one.

Decision: explicit subscription, not auto-discovery

Some implementations of this pattern walk every account and auto-subscribe every log group. We took the opposite approach: log group subscriptions were declared in IaC, one Pulumi resource per group:

aws.cloudwatch.LogSubscriptionFilter(
    f"{name}-subscription",
    name="loki-shipper",
    log_group=log_group.name,
    filter_pattern="",        # forward everything
    destination_arn=lambda_func.arn,
)

filter_pattern="" meant every event; if a service was generating noise, that was a problem to fix at the source, not in the shipper. Adding a new log group was a one-line addition and a deploy.

The argument for auto-discovery is less work and fewer human errors. The argument for explicit subscription is that every line of log volume is a line of cost, and we wanted that to show up in code review. We took the explicit route.

What we accepted, what we didn’t

The handler had minimal retry logic. If Loki was unreachable, the POST raised, the Lambda crashed, and CloudWatch’s subscription delivery retried. After CloudWatch’s retries exhausted, the events were lost. We accepted this because:

• Loki was up for the bulk of the project. When it wasn’t, we had a bigger fire to fight than missing log shipping.
• Adding a DLQ-and-replay layer would have doubled the operational surface for a failure mode we hadn’t actually hit.
• The cost of the missed logs was bounded; the cost of operating a more complex shipping pipeline wasn’t.

What we wouldn’t accept was silently losing logs while the system claimed to be healthy. The handler emitted a CloudWatch metric on every failure, which fed a separate alerting pipeline; if shipping broke, an oncall engineer knew within minutes.

Was it worth it?

Per-account CloudWatch retention dropped to a few days. Loki held the longer history, on object storage, at a fraction of the cost. Engineers used logs again — a single Grafana panel showed logs across every account, with autocomplete on the labels, and queries finished in tenths of a second. The shipper itself was ~150 lines of Python and a Pulumi block.

The setup kept CloudWatch’s on-ramp — writing logs there is the default in every AWS service and SDK — while replacing the destination, which was where the cost and the bad query experience both lived. The shipper was the cheap part of that swap; the query experience was what made it worth doing. For a very small team across a handful of accounts, the right shape; for an organisation an order of magnitude larger, the env-var-baked secret and the no-DLQ acceptance both stop being defensible, and the rebuild starts.

What we could have used

The Lambda-as-shipper isn’t the only way to land CloudWatch logs in Loki. Promtail is the Loki-native answer, fine for host-heavy fleets but awkward when your workloads are already on Lambda and managed services. Vector or Fluent Bit as a per-account aggregator is the flexible answer at higher scale, with more substrate to operate. Kinesis Firehose with an HTTP destination is the managed-AWS answer, with proper buffering and DLQ-to-S3 baked in.

A Lambda shipper had real merits against all three for our context: no long-lived process to operate, no per-account aggregator to scale, no Firehose running 24/7, and the substrate was already a primitive we knew. The label-design and secret-handling decisions discussed above are what carry; the substrate is the easier piece to swap when the scale changes.