Async ML inference under three hostile constraints

This is a write-up of a project from 2024 — the team’s first real attempt at putting an LLM on a production workload. The data-science team had stood up a draft pipeline locally to classify incoming support tickets, using Azure OpenAI as the inference backend, and wanted to ship it. The architecture they’d sketched was sound; what they didn’t have was the production-engineering layer that makes a webhook-triggered Lambda chain survive contact with a help-desk product that retries aggressively and an LLM API that occasionally falls over. My contribution was the second part. The two-Lambda shape below is essentially what the data scientists proposed, and the rest of the post is about what survived contact with production and what didn’t.

A note on dating, because it matters here more than usual. The LLM provider was the Azure OpenAI Assistants v1 API, which was the canonical “give me a stateful agent with threading and file citations” surface in early 2024 — and the obvious choice for a team that wanted an enterprise-friendly Azure deployment rather than direct OpenAI. The Assistants v1 API has since been superseded by the Responses API, with structured outputs / JSON mode and synchronous calls for low-latency models. The production code below is honest about what shipped, including bugs that didn’t get fixed at the time and would be fixed now; there’s a “what we’d do today” section at the end.

The three constraints

Constraint 1. The help-desk product (Zendesk) sent a webhook on every new ticket and expected a 200 or 202 response within about twelve seconds. If you didn’t respond fast enough it retried, up to three times, then gave up. If you responded with a 5xx it considered the delivery failed and might retry harder. The system had to acknowledge the webhook fast, regardless of how long the actual classification took.

Constraint 2. The LLM exposed inference asynchronously: you created a thread, posted a message into it, started a run, and polled a status endpoint until the run reported completed. Total time: ~25 seconds for a typical ticket. There was no synchronous mode for that model on that API at the time.

Constraint 3. The compute ran on AWS Lambda, with a 15-minute wall-clock limit per invocation. No warm cache between invocations beyond what runtime container reuse gave us.

Twelve seconds wasn’t enough for the LLM call. Lambda’s wall was plenty for one classification. The async pattern had to live somewhere.

The architecture, inherited

Two Lambdas, no queue.

help-desk webhook ──► API Gateway ──► Webhook Lambda ──► (invoke async) ──► Worker Lambda
                                            │                                       │
                                       returns 202                              calls LLM,
                                       (sub-second)                             polls until done,
                                                                                writes back to ticket

The webhook Lambda’s only job was to validate the request, fire-and-forget the worker Lambda, and return a 202 in well under a second. The worker received the original webhook payload, ran the LLM classification synchronously within itself by polling the LLM’s async API, and wrote the results back to the ticket via a normal REST call when done.

There was no SQS queue between the two Lambdas — Lambda’s InvocationType="Event" does fire-and-forget invocation as a primitive, queueing the invocation in the Lambda service itself with built-in retries (twice, then drop) and an optional DLQ destination. For a few thousand tickets a day, an extra SQS would have been one more moving part for a problem we didn’t have.

The webhook handler: get out fast

def handler(event, context):
    try:
        lambda_client.invoke(
            FunctionName=os.environ["PROCESSOR_LAMBDA_NAME"],
            InvocationType="Event",          # fire-and-forget
            Payload=json.dumps(event),
        )
        return {
            "statusCode": 202,
            "body": json.dumps({"message": "OK: Request received and is being processed"}),
        }
    except Exception as e:
        return {"statusCode": 500, "body": json.dumps({"message": f"An error occurred: {str(e)}"})}

The PROCESSOR_LAMBDA_NAME was injected at deploy time by the IaC. The 202 was non-negotiable — 200 would have been a lie (no work had happened), 5xx would have triggered a retry storm.

Two weak points in this handler worth flagging, both shipped as written:

• The catch-all except Exception returns 5xx. A transient invoke failure becomes a Zendesk retry rather than a logged-and-acknowledged blip. The cleaner form catches the specific transient exception, emits a CloudWatch metric on the failure, and still returns 202.
• No request-shape validation. The handler trusts the payload and passes it straight through. A malformed webhook produces a downstream worker failure rather than being rejected up front.

The worker: poll the async LLM

The original draft polled the LLM in a tight while True loop with no time.sleep() and no handling of non-completed terminal statuses. It worked — the typical 25s response always landed inside the Lambda’s 30s timeout — but the production form should be bounded and sleep between polls, raising on failed, cancelled, or expired rather than looping until the runtime kills the invocation:

deadline = time.monotonic() + LLM_DEADLINE_SECONDS  # ~90s, well under the Lambda wall
while time.monotonic() < deadline:
    run = client.beta.threads.runs.retrieve(thread_id=thread.id, run_id=run.id)
    if run.status == "completed":
        return parse_response(client.beta.threads.messages.list(thread_id=thread.id))
    if run.status in {"failed", "cancelled", "expired"}:
        raise RuntimeError(f"LLM run {run.status}: {run.last_error}")
    time.sleep(POLL_INTERVAL)
raise TimeoutError(f"LLM run did not complete in {LLM_DEADLINE_SECONDS}s")

A bounded deadline raising on slow runs is the right behaviour: a slow run usually means the model is confused or rate-limited, and a runaway invocation blocks the next ticket on the same warm container.

The Lambda’s runtime configuration was 30-second timeout and 512 MB of memory — adequate for the typical 25s response, with a small safety margin.

What else was wrong with the worker

A few more things the production code shipped with, worth being explicit about:

• PII anonymisation was thin. The anonymise_ticket() function used regex for emails, phone numbers, URLs (good), plus spaCy NER (nl_core_news_sm) for personal names. The repo’s README acknowledges it: “the current anonymization method is rather poor”. spaCy NER misses surnames it hasn’t seen, addresses, IBANs, account numbers — content that does end up in support tickets. The data was processed by an enterprise-contracted Azure OpenAI deployment in the EU, which is a defensible context for sending PII through; the anonymiser was a defence-in-depth layer rather than the primary protection. A future pass would lean on Presidio or an LLM-based PII pre-pass for tighter coverage.
• The response parser was free-text, not structured. The Assistants API returned text-form messages, so the code parsed them as text — clunkier than we’d have liked at the time. Today’s equivalent uses structured outputs / JSON mode and removes the parser entirely.
• No DLQ on the worker’s async invoke. AWS Lambda async-invoke retries twice, then drops. At our throughput a silently-lost ticket every once in a while wasn’t operationally significant, but the explicit DLQ destination is the right default and would be cheap to add.
• No idempotency. Worth mentioning for completeness, but on this workload double-processing produces the same classification a second time and writes the same custom-field values to the same ticket — operationally a no-op rather than a real bug.

A lesson hidden in the Dockerfile

The two Lambdas shipped as different container images. The webhook image was around 50 MB — boto3, requests, almost nothing else. The worker image was around 400 MB — it included the spaCy model for the local anonymiser plus the OpenAI SDK plus a few dependency layers.

Early on, both Lambdas shared a single Dockerfile. Cold starts on the webhook were ~3 seconds, which under load made the 12-second window uncomfortable. Splitting the images cut the webhook cold start to under a second:

85f5182  reduce webhook cold start: separate ECR repo, drop unused deps
         processor: 401 MB → 401 MB (no change)
         webhook:    398 MB →  47 MB
         cold start: 3.1 s   →  0.9 s (p99)

Authentication, secrets, the boring stuff

The pipeline depended on three external credentials: the Azure OpenAI API key, the Zendesk API token, and a Slack bot token for failure notifications. All three lived in AWS Secrets Manager.

The production code fetched each secret with a separate secretsmanager:GetSecretValue call at import time. The cleaner form is to bundle them under one secret ARN and fetch once with @functools.cache:

@functools.cache
def secrets() -> dict[str, str]:
    raw = sm_client.get_secret_value(SecretId=os.environ["SECRETS_ARN"])
    return json.loads(raw["SecretString"])

Same outcome, half the cold-start latency, one API call instead of three.

What we’d do today

A 2026 rebuild of the same pipeline would change four things:

1. Replace the OpenAI Assistants v1 API with the Responses API, with structured outputs (or JSON mode) for the classification result. The polling loop disappears — Responses supports synchronous calls for low-latency models — and so does the free-text parser.
2. Add a DLQ on the worker and a CloudWatch alarm on its depth, so events that fail twice fail loudly rather than silently.
3. Have the webhook return 202 with a CloudWatch metric on transient lambda_client.invoke failures rather than 5xx. The intent isn’t to prevent duplicate processing (which is operationally a no-op here, since both runs would produce and write the same fields) — it’s to stop the help-desk’s retry path from being triggered by something the platform should absorb silently.
4. Use Step Functions Standard for the wait if a polling pattern is still needed for some reason. Step Functions doesn’t bill compute during wait states, so for any workload where the LLM is slower than ~25s on average the cost case tips quickly toward Standard.

The overall shape would stay — fast-ack webhook, fire-and-forget worker, async-invoke between them — because the constraints that drew it haven’t moved.

What the pipeline accomplished anyway

The pipeline was very much a prototype — duct-taping a SaaS help-desk, an AWS Lambda chain, and a service on Azure together while the team was still learning the ropes of all three — but it shipped and ran for the better part of a year. The category and summary fields on classified tickets became reliable enough that the support team built routing and reporting workflows around them, and the latent issues stayed latent: average LLM response time held in the 25s range, the help-desk’s invoke chain didn’t glitch, and the polling loop never blew the Lambda timeout in production.