ADR 003: Choice of Message Broker for Asynchronous Job Orchestration
Google Cloud Tasks vs. Google Cloud Pub/Sub
This record documents the architectural reasoning for transitioning the Deal Engine platform away from its current database-polling model toward a modern, event-driven task queue. It summarises the comparison between the two candidate Google Cloud messaging technologies and the rationale for the selected option.
1. Summary
| Field | Value |
|---|---|
| Status | Proposed |
| Scope | Deal Engine — asynchronous job orchestration |
| Decision | Adopt Google Cloud Tasks; deprecate NoSQL polling |
| Affected Services | Radar, Papi Regional Worker, PAPI Scheduler, NoSQL task store |
2. Context: The Database-as-a-Queue Bottleneck
Currently, the regional communication services (Radar and Papi) actively poll the NoSQL database to claim and execute ticket processing tasks. [6, 8, 12] This database-as-a-queue model suffers from several distributed systems pain points:
- Severe concurrency risks: Multiple workers can claim the same ticket and trigger double-refunds. [4, 6]
- Continuous read overhead: High-frequency polling imposes constant load on the NoSQL instance irrespective of task volume. [6]
- No native backpressure: There is no queue-level mechanism to protect fragile third-party airline APIs from traffic spikes. [12]
3. Options Evaluated
The architectural evaluation compared two primary GCP messaging technologies to resolve these bottlenecks:
| Evaluation Criteria | Option 1: Google Cloud Tasks (Recommended) | Option 2: Google Cloud Pub/Sub |
|---|---|---|
| Messaging Model | Target-Specific Task Queue (Message-to-Endpoint) — the broker manages queues and actively pushes tasks to specific worker HTTP endpoints. | Publish/Subscribe (Message-to-Subscription) — high-throughput topic-based broadcasting where multiple subscribers pull or receive pushes. |
| Rate-Limiting & Backpressure | Excellent (Built-in) — allows native configuration of maximum dispatch rates (tasks/second) and concurrent dispatches per queue. [5] | Poor (Client-Side) — throttling and flow control must be manually programmed and managed at the individual subscriber/worker level. [5] |
| Deduplication | Native & Long-Lived — automatically rejects duplicate tasks matching a custom Task Name within a 20-hour window. [5] | No Native Deduplication — requires manual implementation of lock tables or state validation inside the worker code or database layer. [5] |
| Workload Alignment | Perfect Fit — ticket processing is naturally a point-to-point task (Scheduler → Worker) rather than a multi-system fan-out. [5] | Overkill — high-throughput, horizontal event streaming is designed for massive event fan-outs rather than single-worker task execution. |
| Infrastructure Overhead | Zero — already a listed, pre-existing asset in Deal Engine’s core GCP infrastructure inventory. [10] | Medium — requires provisioning new Pub/Sub topics, subscriptions, and access permissions. |
4. Decision & Justification
We select Google Cloud Tasks to replace the NoSQL polling mechanism.
The driving factor is that rate-limiting and strict deduplication are paramount when dealing with fragile GDS partners and high-stakes financial operations. [5, 12] Cloud Tasks allows us to:
- Enforce hard rate limits at the queue level, protecting third-party airline endpoints from being overwhelmed during traffic spikes. [5]
- Utilize native deduplication by naming the task using our deterministic
idempotency_key. If a duplicate execution is triggered, Cloud Tasks drops it at the broker level before it can ever reach Radar, entirely preventing accidental duplicate ticket transactions. [5] - Minimize infrastructure procurement delays by leveraging a resource already approved and listed in our cloud stack. [10]
5. Consequences
- Positive: The NoSQL database is immediately freed from polling CPU and IOPS overhead [6], backpressure is enforced at the queue level, and duplicate executions are programmatically prevented. [5]
- Neutral: Regional workers (Radar and Papi) must be refactored to act as HTTP target endpoints, secured via private VPC networks and IAM authentication. [8]
6. Migration Strategy
A four-phase migration path will be executed to ensure zero downtime for active transactional traffic:
| Phase | Stage | Description |
|---|---|---|
| Phase 1 | Shadowing | Tasks are dispatched to Cloud Tasks alongside the existing polling flow, without executing the final GDS action, to validate connectivity and payload handling. |
| Phase 2 | Canary Routing | A small share of live transactional traffic is routed through Cloud Tasks while the legacy polling mechanism remains available as a fallback. |
| Phase 3 | Scale-Up | Traffic through Cloud Tasks is increased incrementally while dispatch stability, retry rates, and success rates are monitored. |
| Phase 4 | Polling Deprecation | Once stability is confirmed, the legacy polling logic is decommissioned and the associated NoSQL schemas are retired. |