Reorganized Celery workloads around task characteristics instead of client groups, cutting overall cloud cost by 34% and latency by more than 63%.
lower cloud cost
lower latency
buyers in a single deal flow
Context
SendFlow filters buyer buy boxes, generates personalized deal messages, delivers them through GHL, and schedules optional SMS, email, and follow-ups. A single deal may need to reach more than 5,000 buyers quickly.
The previous architecture assigned one Celery queue to every five clients. Supporting only 115 clients already required 23 worker tasks, and worker count would continue to grow linearly. GHL rate limits and large SendFlow bursts made simple consolidation unsafe.
Architecture
Queues and workers scaled with client groups instead of workload type, mixing SendFlow, CPU-bound, and I/O-bound tasks within the same partitions.
Routing moved from client identity to workload type, allowing each worker pool to scale and tune concurrency independently.
SendFlow
I/O-bound
CPU-bound
Routed every client's SendFlow request to one purpose-built queue instead of maintaining client-specific queue groups.
Used 8-10 consumers for the shared queue, providing enough throughput for large buyer lists while retaining room to autoscale.
Separated CPU-bound and I/O-bound jobs into dedicated queues to prevent different workload profiles from competing for the same worker capacity.
Ran I/O-bound workers at 12-16 concurrent jobs and CPU-bound workers at four, matching resource allocation to the real execution profile.
Outcome