Lambda SQS: retry only failed batch records

Inspect what the function reports for the batch

Separate a record-processing failure from a handler exception that escapes the entire invocation. AWS documents that the default failure path can make successfully processed records in that failed batch visible again. Partial batch handling changes which records are reported for retry.

Our synthetic batch contained two valid records and one controlled failure. The local handler returned only the failing message’s messageId. This checks its response construction; AWS’s documented event-source behavior still needs a live integration test in the target environment.

Return failed message IDs for the intended queue type

The following standard-queue example calls process_record for each record and collects known record-level failures. Replace that function with the real processing contract and record appropriate non-sensitive failure evidence. Unexpected errors that escape the handler still fail the whole invocation.

Use the SQS messageId as itemIdentifier. Do not substitute a body field, receipt handle, or array index. In our local test the returned list contained bad-2, while an empty batch returned an empty failure list.

python · EXAMPLE

def handler(event, context):
    failures = []
    for record in event["Records"]:
        try:
            process_record(record)
        except (ValueError, KeyError):
            failures.append({"itemIdentifier": record["messageId"]})
    return {"batchItemFailures": failures}

Enable the matching event-source response contract

Returning the JSON object is only half the configuration. Verify that the intended mapping has FunctionResponseTypes containing ReportBatchItemFailures. Apply a change through the existing infrastructure workflow rather than updating a similarly named mapping by guesswork.

This read-only CLI inspection shows the selected mapping’s function, source, and response setting. Replace the example UUID and profile with the intended values. Our local SDK validation checked the shape of an update request but did not deploy it.

sh · EXAMPLE

aws lambda get-event-source-mapping --uuid YOUR_MAPPING_UUID \
  --profile work --region eu-west-1 \
  --query '{Function:FunctionArn,Source:EventSourceArn,Responses:FunctionResponseTypes}'

Treat FIFO ordering as a separate requirement

Do not reuse the standard-queue loop unchanged for FIFO processing. AWS directs FIFO handlers to stop after the first failure and report failed and unprocessed messages, preserving ordering. Choose and test the contract for the actual queue type.

Keep visibility timeout, batch size, function timeout, and dead-letter handling consistent with the processing budget. Partial responses do not make slow processing fit inside a visibility window or resolve a permanently invalid record.

Verify outcomes, not just a successful invocation

In a controlled cloud test, send distinguishable records, force one known failure, and inspect which business operations occur and which records return. Use idempotency keys suitable for the operation so duplicate delivery cannot repeat a charge or another side effect.

Our local result proves that the handler isolates one failed identifier. It does not prove that a deployed queue is configured correctly or that a business operation is idempotent. Record those as separate integration checks instead of labeling all duplicates an SQS bug.

Know the limits

Validated by executing a local Python handler over three synthetic records and checking the mapping request against the Boto3 service model. Only bad-2 was returned as failed. No SQS queue, cloud event source mapping, visibility timeout, or redelivery behavior was exercised.

Sources & further reading

A technical publication by CoreTrailKit.

Browse all articles