Architecting a Resilient, Real-Time Sales Order Feed with Boomi & Azure Service Bus
A deep dive into designing a scalable, resilient, near real-time Sales Order integration using Boomi and Azure Service Bus across heterogeneous ERP systems.
Architecting a Resilient, Real-Time Sales Order Feed with Boomi & Azure Service Bus
Overall System Architecture: End-to-End Flow
The diagram below illustrates the complete event-driven architecture, including change detection, deduplication, fan-out delivery, and the resiliency mechanisms that enable manual reprocessing without data loss.
Introduction: Bridging the Latency Gap
In modern enterprise architecture, the synchronization gap between core ERPs and downstream platforms (E-commerce, CX, Logistics) is often where customer experience breaks down. Delays in inventory allocation or order status updates can result in overselling, shipping delays, and increased support tickets.
This case study describes the design of a sub-5-minute, near real-time Sales Order feed that unifies Oracle Cloud, JD Edwards (JDE), and E-Business Suite (EBS) into a single canonical event stream using Boomi and Azure Service Bus (ASB).
The Challenge: Heterogeneity & Noise
The Mission
Ingest Sales Orders from three distinct ERP platforms and broadcast full-payload updates to consuming applications while ensuring resilient delivery and aggressively filtering out non-critical updates ("noise").
Key Architectural Constraints
- Heterogeneous sources with different integration patterns:
- Oracle Cloud accessed via REST/BIP (Business Intelligence Publisher)
- JD Edwards (JDE) accessed via AIS (Application Interface Services)
- E-Business Suite (EBS) accessed via SQL/JDBC
- No middleware staging database to avoid operational overhead
- Noise reduction to prevent downstream API throttling and wasted processing
Architectural Decision: Sliding Window vs. CDC
Change Data Capture (CDC) was evaluated but rejected.
Why Polling Was Chosen
- CDC introduces operational risk on legacy JDE and EBS platforms
- Business SLA required minutes, not milliseconds
- A polling model enabled consistent logic across cloud and on-prem ERPs
Decision: A 5-minute sliding window polling approach with a small lookback buffer.
The Solution: A Decoupled "Claim Check" Pattern
This design follows a three-stage micro-integration pattern, separating change detection from data processing and delivery.
Why Claim Check?
Instead of placing large payloads on the bus immediately, the system first publishes a lightweight "claim ticket" (Sales Order ID + source). This keeps the messaging layer fast, cheap, and scalable.
Stage 1: Change Detection (Claim Check)
Boomi pollers detect changes and publish a minimal message containing:
- Sales Order ID
- Source ERP identifier
Crucially, the extraction method varies by ERP: Oracle uses REST/BIP, JDE uses AIS, and EBS uses SQL. This abstraction is managed entirely within the dedicated Boomi poller processes.
These messages are published to Topic 1 (so-id-distribution).
Stage 2: Hydration & Deduplication (Core Logic)
Each ERP has a dedicated subscription and Boomi processor.
Hash-Based Deduplication
- Retrieve the full Sales Order (hydration)
- Canonicalize the payload
- Generate a SHA-256 hash of critical fields (quantity, ship date, line changes)
- Compare against the stored hash
- Publish only meaningful changes to Topic 2 (
so-events)
This eliminates noise while preserving data integrity.
Stage 3: Fan-Out Delivery
Topic 2 enables pub/sub fan-out.
- Each consumer has its own subscription
- Each integration scales independently
- Failures in one consumer do not impact others
Built-In Scalability: Why This Works
1. Shock Absorber (Backpressure)
Azure Service Bus acts as a durable buffer. Downstream slowness does not impact upstream systems.
2. Competing Consumers
Boomi listeners scale horizontally to drain backlogs during peak load.
3. Independent Scaling Dimensions
Each stage and consumer scales independently, preventing cascading failures.
Production Resiliency: No Data Left Behind
Failure Handling
- Transient failures are automatically handled via ASB retry policies.
- Poison messages (those that consistently fail processing) are automatically routed to the Dead Letter Queues (DLQs) attached to Topics 1 and 2.
Recovery Loop: Alerting and Manual Reprocessing
The DLQs are a key component of the resiliency model, enabling manual intervention without data loss.
- DLQ Monitor: Dedicated monitoring tools continuously check the DLQ depth.
- Queue Alert: When the DLQ count exceeds a critical threshold, an Ops Alert (e.g., PagerDuty, email) is triggered.
- Diagnosis: Operations teams diagnose the root cause (e.g., incorrect data, API outage, certificate expiration).
- Manual Reprocessor: Once the issue is resolved, a dedicated Boomi Reprocessor flow is manually invoked to retrieve the failed messages from the DLQ.
- Re-Injection: The messages are re-injected into the original topic:
- Topic 1 DLQ messages are re-injected into Topic 1 for ID-level recovery.
- Topic 2 DLQ messages are re-injected into Topic 2 for full event recovery.
This pattern ensures every message is accountable and can be recovered, achieving the "No Data Left Behind" guarantee.
Conclusion
This architecture moves beyond point-to-point integration. By combining proven integration patterns with Boomi's orchestration and Azure Service Bus's durability, it delivers a scalable, resilient, and self-healing Sales Order event stream suitable for modern digital platforms.