Mesh

Pipebase ships with message-bus services that flows can publish to and subscribe from. Use them to decouple a flow on one runtime from a consumer on another.

What’s bundled

• Kafka (pipebase-kafka) — Apache Kafka 3.9.2 in single-node KRaft mode. Reachable as pipebase-kafka:9092 from inside the stack’s Docker network. Auto-creates topics on first publish. Topics and consumer offsets are persisted on the pipebase-kafka-data volume.
• Pulsar (pipebase-pulsar) — Apache Pulsar 4.2 in single-node standalone mode. Reachable as pipebase-pulsar:6650 (binary) and :8080 (admin REST). Powers the cross-runtime mesh fabric.
• Pulsar second broker (pipebase-pulsar-east) — opt-in geo- replication topology. Off by default.

Profiles

Kafka and Pulsar start with the default docker compose up -d; --profile multi-broker adds pipebase-pulsar-east.

docker compose --profile multi-broker up -d # adds pipebase-pulsar-east

When to use which

Use case	Use
Decouple producer + consumer flows	Either
High throughput (>10k msg/s)	Kafka
Geo-replication	Pulsar (multi-broker)
Per-namespace access grants	Pulsar (MQTT and Pulsar clients; not AMQP)
Schema registry	Pulsar (the stack runs no Kafka schema registry)
Tiered storage to S3	Neither as shipped — the Pulsar image includes no offloaders

Camel route examples

Kafka — producer:

- to: kafka:orders.created

Kafka — consumer:

- from:
    uri: kafka:orders.created
    steps:
      - log:
          message: "Got order ${body}"

Pulsar:

- to: pulsar:persistent://public/default/events
- from:
    uri: pulsar:persistent://public/default/events
    parameters:
      subscriptionName: pipebase-consumer

Auth

Pulsar requires a broker-signed token on every port and, for Pulsar and MQTT clients, a namespace grant (PIPEBASE_PULSAR_TOKEN_AUTH and PIPEBASE_PULSAR_AUTHORIZATION, both on by default). Kafka has no authentication; it publishes no port on the host, so only containers on the stack’s Docker network reach it.

Related

• Runtimes — runtimes that consume from / publish to mesh topics