RFC: Broker (v1)
Status: Draft
Version: v1
Scope: Real-time event transport, consumer coordination, run execution bindings
Non-goals: Persistence formats, execution semantics
1. Purpose
This RFC defines how XMachines runs are driven by streaming broker backends.
Unlike Git, which solves for durability and offline distribution, Broker solves for:
- Liveness: pushing events to machines in real-time
- Coordination: managing concurrency via consumer groups
- Transport: bridging the gap between topics and run streams
2. Separation of Concerns
This RFC establishes Broker as orthogonal to Git.
| Feature | Git (Persistence) | Broker (Execution) |
|---|---|---|
| Role | the ledger (storage) | the pipe (transport) |
| Model | pull (fetch/clone) | push (subscribe) |
| Concurrency | optimistic locking (branching) | pessimistic locking (consumer groups) |
| Latency | batch / high latency | streaming / sub-second |
| Primary Artifact | commit (immutable record) | message (trigger) |
A typical setup uses Broker for the hot path (real-time execution) and Git for the cold path (archival/audit).
3. Package Model
The Broker implementation follows the adapter pattern:
@xmachines/broker— core interfaces forBrokerConsumerandBrokerProducer@xmachines/broker-kafka— Kafka adapter@xmachines/broker-redpanda— Redpanda adapter
4. Execution Model
The Broker acts as the drive mechanism for a run. It does not replace the run; it feeds it.
Consumer Binding
- The Broker subscribes to a topic
- It maps the partition key to a specific run ID
- It instantiates the run (if not already running)
Stream Piping
Inbound:
The Broker pipes the topic partition into the run’s in stream (WritableStream)
Outbound:
The Broker pipes the run’s out stream (ReadableStream) back to a result topic or dead-letter queue
Backpressure
Because XMachines uses Web Streams, backpressure propagates automatically.
If the run is slow to process, the Broker stops fetching from the partition.
5. Concurrency & Ordering
Run RFC mandates that a run is deterministic given an ordered sequence of events.
Broker RFC guarantees this via partition leasing:
- A specific run ID is pinned to a specific partition
- The Broker’s consumer group protocol ensures only one worker node holds the lease at any time
This provides the single writer guarantee required by the run without needing a distributed lock manager.
6. Invariants
- Transport agnostic — the run does not know it is being driven by Kafka
- Ordering preserved — messages from a partition are delivered in strict order
- At-least-once — offsets are committed only after the run has successfully processed the event
- Partition alignment — a single run must never span multiple partitions
7. Lock statement
The Broker is the active transport binding that drives XMachines runs from real-time streams.
It handles coordination and delivery, delegating logic to the run and persistence to the configured durability backend.
This is Broker v1.