Replaced a burst-sensitive ECS webhook endpoint with a serverless ingestion pipeline that preserved large payloads and reduced estimated monthly cost by more than half.
lower estimated monthly infrastructure cost
estimated monthly cost after redesign
payloads safely offloaded
Context
The original webhook path accepted traffic on an ECS server and forwarded each event to Celery. Sudden bursts could overwhelm or restart the server, causing webhooks to be lost.
Improve burst resilience without simply increasing always-on ECS capacity. The pipeline also had to account for occasional GHL payloads larger than SQS's 1 MB message limit.
Architecture
The ECS server acted as both the public entry point and the first handoff boundary, so a traffic burst could overwhelm it before Celery accepted the work.
API Gateway absorbs bursts, Lambda checks payload size, and S3 safely holds bodies that exceed the SQS message limit before asynchronous processing.
Webhook flow
Moved the public webhook endpoint to Amazon API Gateway so sudden traffic bursts would not depend on a fixed application server.
Added a Lambda function that evaluates payload size before enqueueing each webhook.
Stored payloads larger than 1 MB in S3 and placed the S3 reference on SQS; smaller events were written directly to the queue.
Used a small ECS consumer to drain SQS and route events to Celery workers for asynchronous business processing.
Outcome