ADR 003: Message Broker Selection for Asynchronous Job Orchestration

Replacing NoSQL Polling

The current reliance on NoSQL polling for core transactional workflows has reached its scaling limit, necessitating a shift to a push-based orchestration model. This ADR outlines the transition to a managed message broker to support Deal Engine’s multi-cloud evolution, ensuring regional reliability while safely decomposing the Aleron monolith.

1. Status and Meta-Data

2. Context and Problem Statement

The Deal Engine architecture currently utilizes a “Database-as-a-Queue” anti-pattern. In this model, the Aleron core and the PAPI Coordinator (the logic engine) write task states to a NoSQL database. Regional workers—specifically Radar and the Papi Regional Worker—continuously poll this database to identify pending tasks. Once a task is retrieved, the workers execute the action against external Global Distribution Systems (GDS) and airline APIs via the Twingate Client.

Operational Pain Points

This “pull-based” model introduces significant systemic risks:

• Concurrency and Race Conditions: The lack of atomic message visibility leads to multiple workers claiming the same task. In the travel domain, this results in duplicate ticket transactions and double-refunds.
• Resource Inefficiency: High-frequency polling against the NoSQL instance creates constant, unnecessary CPU and I/O load, regardless of actual task volume.
• Backpressure Deficits: There is no native mechanism to throttle outgoing requests. This leaves external GDS endpoints vulnerable to traffic spikes, risking IP blocking and rate-limit penalties.
• Non-Deterministic Latency for Rudder: The polling delay negatively impacts Rudder (the Tax Calculator), creating inconsistent timing for critical tax and fee calculations that must precede ticket finalization.
• Fragile Error Recovery: Retries and crash recovery are managed manually via database state. If a worker fails mid-execution, tasks often hang in a “Processing” state without native visibility timeouts.

3. Decision Drivers

The selection of a replacement broker is governed by the specialized constraints of the travel industry rather than raw throughput. Our primary requirement is the protection of fragile external endpoints.

Key Differentiators

• Granular Rate Limiting: We must strictly enforce Transactions Per Second (TPS) limits at the queue level to honor GDS constraints.
• Guaranteed Idempotency: The system must support deterministic keys to prevent duplicate execution of any ticket action.
• Scheduled Execution: Many Deal Engine jobs require delayed execution (e.g., “process refund in 2 hours”), a feature currently managed poorly by database timestamps.
• Operational Simplicity: We will prioritize existing Google Cloud Platform (GCP) assets to minimize security review friction and leverage our established GKE and Cloud Run footprints.

4. Options Evaluated

Option 1: Google Cloud Tasks (The Targeted Queue)

Cloud Tasks is a distributed queue service designed for “message-to-endpoint” push communication.

• Push Model: Cloud Tasks explicitly targets HTTP endpoints, allowing it to trigger workers residing on our existing Google Kubernetes Engine (GKE) and Cloud Run clusters.
• Native Scheduling: It supports a schedule_time parameter, allowing the Scheduler to offload future-dated jobs directly to the broker.
• Deduplication: It provides deterministic deduplication via the “Task Name” feature.
• Rate Control: Offers native max_dispatch_rate and max_concurrent_dispatches to protect downstream APIs and the Twingate egress point.

Option 2: Google Cloud Pub/Sub (The Event-Driven Broker)

Pub/Sub is a high-scale asynchronous messaging service designed for fan-out patterns.

• Limitations: Pub/Sub lacks native, broker-side rate limiting and does not support scheduled message delivery (delayed messages).
• Delivery Model: It follows an “at-least-once” delivery model without the deterministic deduplication required to prevent double-refunds.
• Contextual Fit: While excellent for high-throughput analytics ingestion, it is ill-suited for the point-to-point, task-oriented coordination required between the PAPI Coordinator and regional workers.

5. Decision Outcome & Justification

The engineering team will implement Google Cloud Tasks as the primary orchestration engine for core transactional operations.

Strategic Alignment

• TPS Protection: The max-dispatch-rate feature in Cloud Tasks acts as a native “Control Tower,” ensuring we never exceed GDS TPS limits regardless of internal traffic spikes.
• Deterministic Idempotency: We will implement a standardized naming convention for tasks: [service]-[ticketID]-[transactionID]-[attemptCount]. Because Cloud Tasks prevents the creation of two tasks with the same name within a queue, this effectively eliminates the “double-worker” race condition at the infrastructure level.
• Virtual Control Tower: While the legacy Control Tower UI remains for business visibility, Cloud Tasks provides an operational Control Tower via the GCP Console, offering native visibility into queue depth, retry rates, and execution latency.
• Infrastructure Synergy: Cloud Tasks is already an approved asset within the Deal Engine Primary Infrastructure list. It integrates seamlessly with our current GKE/Cloud Run execution environments without requiring new vendor procurement.

6. Consequences and Impact Analysis

Positive Impacts

• Reduced Database Pressure: Removing polling will significantly lower NoSQL I/O, allowing that resource to scale more effectively for state persistence.
• Stabilized Rudder Calculations: Transitioning to a push model ensures that tax calculations are triggered immediately upon task creation, reducing non-deterministic wait times.
• Operational Reliability: Native dead-letter queues and configurable retry backoffs replace fragile, manual database state management.

Negative / Neutral Impacts

• Worker Refactoring: The Radar and Papi workers must be updated to serve as secured HTTP targets. This requires configuring Identity-Aware Proxy (IAP) or service-to-service OIDC authentication.
• Retention Management: Cloud Tasks has a 31-day task retention limit. Long-term auditing of task outcomes must still be persisted to Cloud SQL.

7. Migration Roadmap

To adhere to our Zero-Interruption mandate, the transition will follow a four-phase approach: