Carrier Webhook DLQ Redrive Without Duplicate Shipments

Why Carrier Webhooks End Up in a Dead-Letter Queue

A carrier webhook DLQ fills up because your consumer failed the same message too many times, not because SQS is broken. Amazon SQS moves a message once its delivery count crosses a threshold you set yourself: the number of times a message is delivered to the source queue before being moved to the dead-letter queue, with a default of 10. When the ReceiveCount for a message exceeds the maxReceiveCount for a queue, Amazon SQS moves the message to the dead-letter-queue.

In carrier middleware, the failures that fill this queue are rarely exotic. A carrier ships a schema change in a tracking-status payload and your handler throws on an unexpected field. A downstream service returns 500s during a deploy window. A label-void event arrives for a shipment ID your database hasn't committed yet because of replication lag. Each retry increments the receive count, and after ten attempts (or whatever you've configured) the message lands in the DLQ, usually alongside a few hundred siblings from the same bad hour.

The redrive itself is mechanically simple. The risk is what happens when you replay a "label created" or "shipment status: delivered" event into a system that has already half-processed it, or processed it and then rolled back. Get this wrong and you don't get a stuck queue, you get a duplicate label purchase or a duplicate billing event on a carrier's platform. This tutorial walks through doing it without that outcome.

Before You Start: What You Need in Place

Four things need to exist before you touch the redrive button, not during the incident.

• A confirmed RedrivePolicy on the source queue. Check that the source queue's RedrivePolicy attribute already has a deadLetterTargetArn and maxReceiveCount set, since RedrivePolicy is the string that includes the parameters for the dead-letter queue functionality of the source queue as a JSON object, including the ARN of the dead-letter queue to which Amazon SQS moves messages after maxReceiveCount is exceeded.
• IAM permissions for the redrive path. If your queues sit behind VPC endpoint restrictions, a plain redrive will fail with AccessDenied, because SQS calls the API on your behalf from outside your VPC. The fix is documented directly: add the aws:CalledViaLast condition to your queue policy, which allows Amazon SQS to make API calls on your behalf while maintaining VPC restrictions for direct access.
• An idempotency store, separate from SQS. Redis or DynamoDB, keyed on the carrier's own event ID or tracking number, not the SQS MessageId. SQS message IDs are per-delivery; carrier event IDs are the only stable de-duplication key across a redrive.
• A tested cancellation path. Know the command before you start the task, not after messages are already flowing into a broken consumer.

Redriving the DLQ Without Duplicating Shipments

Follow these in order. Skipping step 2 or 3 to "save time" is exactly how a redrive turns into an incident.

1. Inspect the DLQ without deleting anything. Use ReceiveMessage with a short VisibilityTimeout and no delete call, so you can read event types and payloads without pulling them off the queue permanently. Group by carrier and event type (label created, status update, void confirmed) so you know what you're about to replay.
2. Fix the bug that put the messages there before you redrive. If a consumer bug caused the failures, deploying the redrive into the same broken code just refills the DLQ. This sounds obvious. It's still the most common mistake in DLQ incidents.
3. Verify the idempotency guard is live in the consumer path. Before any message flows back, confirm your consumer checks the carrier's event ID against your idempotency store and short-circuits on a hit, rejecting or no-op'ing rather than re-executing shipment or billing logic.
4. Pick a destination: source queue, or a staging queue first. AWS gives you the choice directly in the redrive task: to redrive messages to their source queue, choose Redrive to source queue(s), or to redrive messages to another queue, choose Redrive to custom destination and enter the ARN of an existing destination queue. For anything touching billing or label creation, redrive to a staging queue first and let a human or a script verify a sample before promoting to production.
5. Set a velocity you can watch. Don't default to maximum throughput on a first redrive. AWS's own console offers System optimized, which redrives dead-letter queue messages at the maximum number of messages per second, or Custom max velocity, which redrives at a custom maximum rate of messages per second. Via the API, the same control exists as a rate cap: the number of messages to be moved per second, defining a fixed message movement rate, with a maximum value of 500 messages per second. Start low, e.g. 5-10/sec for anything with billing side-effects, and raise it once you trust the numbers.
6. Start the task with StartMessageMoveTask and know its limits. This is the actual API call: aws sqs start-message-move-task --source-arn <dlq-arn> --destination-arn <dest-arn> --max-number-of-messages-per-second 10. Two limits matter operationally: a dead-letter queue redrive task can run a maximum of 36 hours, and Amazon SQS supports a maximum of 100 active redrive tasks per account. Also note that only one active message movement task is supported per queue at any given time, so you can't run two redrives on the same DLQ in parallel to go faster.
7. Monitor with ListMessageMoveTasks and cross-check against your idempotency store. The task response gives you ApproximateNumberOfMessagesMoved and ApproximateNumberOfMessagesToMove. Don't stop there. Compare that count against hits in your idempotency store. If SQS says 4,200 moved but your store only shows 3,800 unique accepted events, something is either being rejected silently or double-counted, and you need to know which before declaring the redrive done.

Failure Mode: The Redrive That Fires a Second Label Purchase

Here's the scenario worth war-gaming before it happens to you. A "label created" webhook sat in the DLQ because your handler was down for twenty minutes during a deploy. During that window, a retry mechanism elsewhere in the stack (or a manual API call from an ops engineer trying to "just get the shipment moving") already created the label directly against the carrier. Now you redrive the DLQ. Without an idempotency check keyed on the carrier's shipment or label ID, the handler processes the webhook as if it's new, and triggers a second label purchase against the same order. On some carrier accounts, that's a second charge, not just a duplicate PDF.

The immediate remediation is to stop the bleeding, not diagnose live. If you want to cancel the message redrive task, on the Details page for your queue, choose Cancel DLQ redrive; when canceling an in-progress message redrive, any messages that have already been successfully moved to their move destination queue will remain in the destination queue. That last part matters: cancelling doesn't undo what already moved. You still need to reconcile.

Reconciliation means walking the idempotency store for the affected time window, cross-referencing against the carrier's actual shipment/label records via their tracking or shipment-status API, and voiding or crediting the duplicates you find. This is exactly why the envelope-vs-payload distinction matters in your message schema: if your envelope carries a stable carrier event ID separate from the payload body, you can de-duplicate on the envelope even when the payload has drifted between carrier API versions.

How You Know the Redrive Worked

Three signals, checked together, not individually:

• CloudWatch's ApproximateNumberOfMessagesVisible on the DLQ returns to zero (or to the count of messages you deliberately excluded, e.g. malformed ones sent to a quarantine queue instead).
• Your idempotency store's replayed-event count matches the accepted-event count, with no spike in "duplicate key rejected" events that actually correspond to real duplicate side-effects downstream (a rejected duplicate is success; a duplicate that got through is not).
• No new support tickets referencing double-billed labels, duplicate tracking numbers, or "why do I have two labels for one order" in the hours after the redrive completes.

If any of these three disagree with the others, don't close the incident. A clean DLQ count with a support ticket spike means messages moved successfully into a system that still produced duplicates. That's the failure mode you were trying to avoid, just delayed by a few hours.

This Isn't a Problem Any Carrier API Solves for You

Every platform that aggregates carrier webhooks at volume runs into this same DLQ-plus-idempotency problem, regardless of which carrier sits behind it. Enterprise TMS suites like MercuryGate, Descartes, and Blue Yonder deal with it. Multi-carrier shipping tools like ShipEngine, EasyPost, Shippo, and Sendcloud deal with it. Shipper-side connectivity platforms like Cargoson deal with it too. None of them make the queue-plus-idempotency-store decision for you architecturally; it's a design choice you own once webhook volume crosses a few hundred events per minute, independent of which carrier API is generating the events.

Next Steps

Before your next DLQ fills up, do two things now rather than during the incident. First, add the aws:CalledViaLast condition to your queue policies if you're running inside a VPC, so redrive doesn't fail on permissions when you need it most, following the pattern in AWS's DLQ redrive guide. Second, audit whether your webhook consumers actually check an idempotency key before executing shipment or billing logic, not just before returning a 200. If you're unsure, that's the gap a redrive will find for you, at the worst possible time. For webhook authenticity checks alongside idempotency, Shippo's webhook security documentation is a useful reference for the signature side of this problem.