Skip to content

AWS Updated 2026-07-04

Lambda dead letter queues and destinations: catching failed events before they vanish

Async Lambda events that fail all retries are silently dropped unless you configure a dead letter queue or an on-failure destination. How the two differ, which to pick, and how to set them up.

When an asynchronous Lambda invocation fails all its retries, the event is deleted. Not logged somewhere convenient, not parked — gone. A dead letter queue (DLQ) or an on-failure destination is the only thing standing between a bad deploy and a few hours of silently lost S3 notifications, EventBridge events, or SNS messages. Destinations are the newer, richer mechanism and the right default; DLQs remain fine where they exist. What matters is that every async function has one of them, and that something watches it.

What’s happening

Lambda has three invocation models, and failure handling differs sharply between them — most confusion about DLQs is really confusion about which model is in play.

Synchronous invocations (API Gateway, ALB, direct calls) return errors to the caller; Lambda retries nothing and neither DLQs nor destinations apply. Event source mappings — the poll-based model for SQS, Kinesis, and DynamoDB Streams — handle failure at the source: an SQS queue uses its own redrive policy to a queue-level DLQ, and Kinesis/DynamoDB mappings have their own retry, bisect, and on-failure destination settings. A function DLQ configured on an SQS-driven function does nothing; the queue’s redrive policy is what catches poison messages.

That leaves asynchronous invocations — S3 notifications, SNS, EventBridge, and anything invoked with InvocationType=Event. Lambda queues these internally, attempts delivery, and on function error retries up to two more times (configurable 0-2 via MaximumRetryAttempts) with backoff, discarding events older than MaximumEventAgeInSeconds (up to 6 hours). After the last failure, the event goes to the DLQ or on-failure destination if configured, and evaporates otherwise.

The two catch mechanisms differ in payload and reach. A DLQ is an SQS queue or SNS topic that receives the original event only — no error message, no context. Destinations (OnFailure and also OnSuccess) can target SQS, SNS, another Lambda function, or EventBridge, and deliver a wrapper containing the original event plus the error type, message, stack trace summary, and request context. For debugging at 3 a.m., that wrapper is the difference between knowing why it failed and re-running it blind to find out.

What to check first

  • Which invocation model each function actually uses. Trigger type determines whether function-level failure capture even applies.
  • Whether async functions have anything configured. Most accounts we audit have several async functions with no DLQ, no destination, and no idea events have been dropping.
  • Whether the existing DLQ has ever been read. A DLQ with 14,000 messages and no consumer or alarm is a landfill, not a safety net.
  • IAM permissions. The function’s execution role needs sqs:SendMessage, sns:Publish, lambda:InvokeFunction, or events:PutEvents on the target — without it, failure capture itself fails, silently.

How to verify

Inspect the async configuration of a function:

aws lambda get-function-event-invoke-config --function-name my-function

A ResourceNotFoundException means everything is at defaults: two retries, six-hour max age, no destination. Check for a legacy DLQ:

aws lambda get-function-configuration --function-name my-function \
  --query 'DeadLetterConfig'

In the console, both live under Lambda > Functions > (function) > Configuration > Asynchronous invocation. The CloudWatch metrics that tell you events are being lost are AsyncEventsDropped and DeadLetterErrors (failures to write to the DLQ itself) in the AWS/Lambda namespace. The invocation model details are documented at https://docs.aws.amazon.com/lambda/.

The runbook

  1. Inventory async functions. List functions and their triggers; anything fed by S3, SNS, or EventBridge, or invoked as Event by your own code, is in scope.
  2. Create one failure queue per function (or per service). An SQS queue with a 14-day retention gives you the maximum window to notice and replay. Per-function queues make attribution trivial; a shared queue per service is acceptable if the wrapper identifies the source.
  3. Configure an OnFailure destination pointing at it.
aws lambda put-function-event-invoke-config \
  --function-name my-function \
  --maximum-retry-attempts 2 \
  --maximum-event-age-in-seconds 21600 \
  --destination-config '{"OnFailure":{"Destination":"arn:aws:sqs:ca-central-1:123456789012:my-function-failures"}}'
  1. Grant the execution role permission to deliver. Add sqs:SendMessage on the queue ARN to the function’s role. Test by deploying a deliberately failing version in staging and confirming the wrapped event lands.
  2. Alarm on the queue, not the vibe. A CloudWatch alarm on ApproximateNumberOfMessagesVisible > 0 for the failure queue turns silent data loss into a page. Fold it into the same alarm baseline as everything else — see the CloudWatch alarm baseline.
  3. Build the replay path before you need it. A small script that reads the destination wrapper, extracts the original event, and re-invokes the function turns a bad deploy into “redrive the queue” instead of a data reconstruction project.
  4. Tune retries per workload. Payment webhooks may warrant MaximumRetryAttempts=0 and immediate capture (retries duplicate side effects); idempotent enrichment jobs can keep the full two retries and six hours.

When this is the wrong move

For SQS-driven functions, configure the redrive policy on the source queue instead — a function DLQ there is dead configuration. For Kinesis and DynamoDB streams, use the event source mapping’s own on-failure destination and bisect settings, which understand batches. And if a workload genuinely tolerates loss — sampled analytics pings, best-effort cache warming — a destination plus alarms plus replay tooling is process for its own sake; make the loss decision explicitly and write it down.

Common mistakes

  • Configuring a DLQ on a function whose trigger is an SQS event source mapping and believing poison messages are handled.
  • Forgetting the IAM permission for the destination, so failure capture fails — visible only in the DeadLetterErrors metric nobody graphs.
  • No alarm on the failure queue, discovering three weeks of dropped events during an unrelated investigation.
  • Choosing plain DLQs for new setups and losing the error context that destinations would have included for free.

Prevention

  • Make an OnFailure destination part of the function module in your IaC, so every new async function gets one without anyone remembering.
  • Alarm on AsyncEventsDropped account-wide — it should always be zero.
  • Include the replay script in the service’s runbook and test it quarterly, like a backup restore.
  • Review MaximumEventAgeInSeconds against real downstream tolerance; six hours of retries into a broken dependency can be worse than failing fast.

Failed events are inevitable; losing them is a choice you make by leaving the destination config empty. If nobody on your team truly owns the AWS environment end to end, that is the root of most of these problems — Stack Harbor runs AWS accounts daily under AWS Operations, and you can book an AWS review.