Enterprise Streaming Pattern

STATUS: EXTERNAL — share-safe, bounded, regulator-visible.

Apply External / Share-Safe Codex AI rules.

A Reference Architecture for Governed Event Participation in Enterprise Intelligence

Apache Kafka Reference Realisation

Version: 1.0

Status: Reference Architecture

Author: Arqua Pty Ltd

Classification: Public

Executive Summary

Enterprise Intelligence depends upon the ability to recognise and respond to change.

Operational systems continuously produce events describing changes in enterprise state:

• a service becomes unavailable;
• an asset enters a degraded condition;
• a customer interaction occurs;
• a work order changes status;
• an authority delegation expires;
• an inventory threshold is crossed;
• an AI agent proposes an operational action.

These events must become available across the enterprise without transferring ownership of the underlying enterprise reality to a central platform.

The Enterprise Streaming Pattern defines how current enterprise events are published, governed, distributed, interpreted and consumed as part of Enterprise Intelligence.

The pattern positions event streaming as an enterprise communication capability.

It does not position the streaming platform as:

• the Representation Spine;
• the Semantic Spine;
• Institutional Memory;
• the Enterprise Control Plane;
• the Enterprise Registry;
• or the authoritative source of enterprise truth.

Apache Kafka provides a strong reference realisation because it supports durable event topics, decoupled producers and consumers, partitioned processing, stream processing and integration with external systems. Kafka describes itself as a distributed event-streaming platform in which producers publish events, consumers subscribe to them and events are durably stored in topics.

Within the Arqua Enterprise Intelligence Architecture:

Authoritative enterprise products publish events. Kafka carries those events. Semantic and governance services establish their meaning and permitted use. Runtime Context Assembly incorporates relevant events into Qualified Operational Understanding. The Enterprise Control Plane coordinates their participation. SCIA Runtime governs any resulting consequence-bearing execution.

Kafka implements the communication pattern.

It does not define the enterprise architecture.

1. Purpose

This paper defines the Enterprise Streaming Pattern within the Arqua Enterprise Intelligence Architecture.

The pattern explains how events originating from authoritative Business and Data Domains participate in:

• enterprise representation;
• Runtime Context Assembly;
• operational coordination;
• AI reasoning;
• execution governance;
• outcome capture;
• and Institutional Memory.

Apache Kafka is presented as the reference technology realisation.

The pattern remains technology independent and may also be implemented using managed Kafka services, cloud-native event platforms or other distributed streaming technologies.

2. The Enterprise Problem

Enterprises contain many systems capable of describing change.

Examples include:

• operational applications;
• customer systems;
• network platforms;
• ERP systems;
• field systems;
• identity platforms;
• telemetry platforms;
• digital twins;
• AI agents;
• workflow engines;
• and external partners.

Traditional integration approaches often distribute these changes through:

• point-to-point interfaces;
• scheduled extracts;
• application-specific queues;
• database replication;
• direct API calls;
• or centrally owned transformation pipelines.

These approaches create recurring architectural problems:

• producers become coupled to consumers;
• event meaning becomes implicit;
• multiple systems publish conflicting representations;
• transformations obscure source authority;
• governance is applied inconsistently;
• consumers cannot establish provenance;
• events are treated as transient messages rather than enterprise representations;
• and downstream actions may occur without adequate runtime control.

Enterprise Intelligence requires a more disciplined model.

Events must remain connected to:

• the authoritative product that published them;
• the state transition they represent;
• the semantic meaning of that transition;
• the governing identity and policy;
• the evidence supporting the event;
• and the consequences that may follow.

3. Architectural Thesis

The Enterprise Streaming Pattern is founded upon the following thesis:

An enterprise event is a governed representation of change published by an authoritative enterprise product for participation across the enterprise.

The streaming platform transports and preserves the event.

It does not make the event authoritative.

Authority originates from the publishing product and its owning domain.

Meaning originates from the enterprise semantic architecture.

Permitted use originates from enterprise policy and authority.

Operational relevance is established through Runtime Context Assembly.

Execution consequence is governed through Execution Admissibility.

4. Constitutional Position

Enterprise Streaming is an implementation pattern rather than a constitutional asset.

It supports several constitutional responsibilities without absorbing them.

Kafka therefore acts as a communication implementation between constitutional capabilities.

It is not itself a constitutional spine.

5. Architectural Principles

ESP-1 Events Originate from Authoritative Products

Every enterprise event must be published by, or traceably derived from, an authoritative enterprise product.

The event platform does not create authority.

ESP-2 Events Represent Change

An event describes something that has occurred, been observed, been proposed or been determined.

Events should not be used as unqualified substitutes for complete enterprise state.

ESP-3 Producers and Consumers Remain Decoupled

Publishing products should not require knowledge of every downstream consumer.

Consumers should discover and subscribe to governed event products rather than creating direct dependencies upon source systems.

Kafka natively separates producers from consumers, allowing applications to publish and subscribe independently.

ESP-4 Meaning Travels with the Event

Every governed event must be associated with:

• an event type;
• a semantic definition;
• a schema;
• an authority;
• a version;
• and a declared business meaning.

Transport without meaning is not Enterprise Intelligence.

ESP-5 Event Ownership Remains with the Domain

A central platform team may operate the streaming infrastructure.

It must not assume ownership of the business event.

The publishing domain remains accountable for:

• event meaning;
• completeness;
• timeliness;
• quality;
• compatibility;
• and lifecycle.

ESP-6 Streaming is Not Institutional Memory

Kafka may durably retain events, but retention within a topic does not automatically make those events accepted Institutional Memory.

Institutional Memory requires acceptance, governance, evidentiary standing and durable preservation beyond transport retention.

ESP-7 Streaming is Not Execution Authority

Receipt of an event does not automatically authorise a consequential action.

A consumer may interpret an event, assemble context or propose an action.

Execution still requires the applicable authority, policy and admissibility controls.

ESP-8 Replay Must Not Recreate Consequence Unintentionally

The ability to replay events is valuable for recovery, reconstruction and learning.

Replay must be separated from irreversible execution.

Consumers must distinguish:

• historical reconstruction;
• analytical replay;
• state rebuilding;
• simulation;
• and live consequence-bearing execution.

6. The Enterprise Event

An Enterprise Event is more than a message payload.

It is a governed representation of change.

A conforming event should contain or reference the following elements.

Not every payload must physically contain every element.

However, the event product must make these characteristics resolvable through its contract, registry metadata or associated governance services.

7. Event Classes

The pattern distinguishes several classes of enterprise event.

7.1 State Change Event

Represents an accepted change in enterprise state.

Examples:

• work order status changed;
• customer service activated;
• asset condition degraded;
• authority delegation expired.

7.2 Observation Event

Represents something detected or measured.

Examples:

• telemetry reading;
• alarm raised;
• anomaly detected;
• device heartbeat missed.

An observation is not necessarily accepted enterprise state.

It may require qualification or correlation.

7.3 Decision Event

Represents a decision that has been made.

Examples:

• restoration strategy selected;
• claim approved;
• service priority changed;
• escalation required.

A decision event does not necessarily mean execution has occurred.

7.4 Proposed Action Event

Represents an action proposed by a person, workflow, decision service or AI agent.

Examples:

• dispatch technician;
• isolate network segment;
• issue customer credit;
• place supplier order.

A Proposed Action Event must not be interpreted as permission to execute.

It is an input to Execution Admissibility.

7.5 Admissibility Outcome Event

Represents the result of an Execution Admissibility evaluation.

Examples:

• ADMISSIBLE;
• ADMISSIBLE_WITH_CONDITIONS;
• ESCALATE;
• NOT_ADMISSIBLE;
• INSUFFICIENT_INFORMATION.

The outcome must remain bound to:

• the evaluated action;
• its context;
• its authority;
• its evidence;
• its constraints;
• and the applicable T=0 evaluation.

7.6 Execution Outcome Event

Represents what actually occurred after execution.

Examples:

• action committed;
• action failed;
• action partially completed;
• action compensated;
• action reversed where reversal was possible.

7.7 Revision Event

Represents a governed correction or reinterpretation of an earlier representation.

A revision must preserve the original event and express the relationship between:

• the original representation;
• the revised representation;
• the reason for revision;
• and the authority accepting the change.

8. Event Products

Individual Kafka topics should not be treated as isolated technical channels.

They should be governed as parts of Enterprise Event Products.

An Enterprise Event Product includes:

• one or more event streams;
• product ownership;
• event semantics;
• schemas;
• service-level expectations;
• retention requirements;
• classification;
• access policy;
• lineage;
• replay policy;
• compatibility rules;
• and consumer guidance.

The event topic is the transport surface.

The event product is the governed enterprise offering.

9. Topic Design

Topic design should follow enterprise meaning rather than application convenience.

Topics should normally represent stable business event classes or coherent event-product boundaries.

Avoid topic structures based solely on:

• project names;
• consuming applications;
• temporary delivery teams;
• database table names;
• or individual point-to-point interfaces.

Recommended topic characteristics include:

• explicit domain ownership;
• stable event purpose;
• versioned schemas;
• controlled partition keys;
• defined retention;
• declared ordering scope;
• compatible evolution;
• and observable service levels.

10. Partitioning and Ordering

Kafka orders records within a partition rather than across an entire topic.

The partition key therefore becomes an architectural decision.

It determines the scope within which ordered processing can be relied upon.

Typical partition keys include:

• customer identifier;
• service identifier;
• asset identifier;
• order identifier;
• incident identifier;
• or another stable enterprise subject.

The partition key should align with the enterprise subject whose state transitions require ordered interpretation.

A poorly selected partition key can create:

• inconsistent state reconstruction;
• unwanted processing contention;
• poor distribution;
• or false assumptions about global ordering.

The architecture must therefore declare:

• the required ordering scope;
• the chosen partition key;
• the implications for replay;
• and the behaviour when related events span partitions.

11. Event Schema and Contract

Each event product requires a governed event contract.

The contract should define:

• payload structure;
• field meaning;
• required and optional elements;
• compatibility rules;
• classification;
• semantic references;
• authority;
• quality expectations;
• and deprecation policy.

Schema compatibility is necessary but insufficient.

A schema may remain structurally compatible while changing business meaning.

Therefore, governance must distinguish:

• syntactic compatibility;
• semantic compatibility;
• behavioural compatibility;
• and operational compatibility.

12. Event Publication Pattern

The preferred publication flow is:

Authoritative Operational System
            │
            ▼
Source-Aligned Enterprise Product
            │
            ▼
Domain Event Formation
            │
            ▼
Contract and Policy Validation
            │
            ▼
Enterprise Event Product
            │
            ▼
Kafka Topic

The source system does not simply expose database changes as enterprise meaning.

The domain must determine which changes constitute stable enterprise events.

Change Data Capture may assist event formation, but database changes do not automatically become governed business events.

13. Outbox Pattern

Where an operational transaction and event publication must remain consistent, use a transactional outbox or equivalent pattern.

The operational application commits:

• the business transaction;
• and a corresponding outbox record

within the same local transaction.

A publisher then transfers the outbox event to Kafka.

This reduces the risk that:

• business state changes but no event is published;
• or an event is published for a transaction that later fails.

The outbox pattern does not remove the need for:

• idempotent consumers;
• duplicate handling;
• event identifiers;
• monitoring;
• and reconciliation.

14. Change Data Capture Pattern

Change Data Capture may be used where direct application event publication is unavailable.

CDC should be treated as an extraction mechanism, not as automatic semantic publication.

A CDC implementation should introduce an event-formation layer that:

• identifies meaningful changes;
• removes storage-specific noise;
• resolves business subjects;
• assigns event types;
• applies semantic definitions;
• attaches authority and provenance;
• and publishes a governed event product.

Without this layer, CDC risks reproducing internal database structures as enterprise contracts.

15. Kafka Connect Pattern

Kafka Connect provides a standard framework for moving records between Kafka and external systems through source and sink connectors. Apache Kafka describes connectors as components that ingest data into Kafka or pass data from Kafka to another system.

Within the Enterprise Streaming Pattern, Kafka Connect may implement:

• source ingestion;
• event publication;
• governed delivery;
• data-lake integration;
• search indexing;
• analytical replication;
• or system synchronisation.

Connectors must not bypass:

• enterprise product ownership;
• schema governance;
• classification;
• policy enforcement;
• lineage capture;
• or consequence controls.

A connector is an integration implementation.

It is not an authority transformation.

16. Stream Processing Pattern

Kafka Streams or equivalent stream-processing technologies may perform continuous processing over enterprise events.

Kafka Streams is a client library for building applications and microservices whose input and output are stored in Kafka topics.

Permitted uses include:

• filtering;
• enrichment;
• aggregation;
• windowed correlation;
• state reconstruction;
• anomaly detection;
• event projection;
• and operational signal formation.

Each derived event product must declare:

• source events;
• transformation logic;
• processing version;
• semantic meaning;
• quality implications;
• lineage;
• and authority status.

A derived stream is not automatically authoritative merely because it was computed from authoritative events.

Its authority must be explicitly established for its intended purpose.

17. Processing Guarantees

Kafka can support idempotent and transactional processing and provides exactly-once processing capabilities under defined conditions. Apache Kafka notes that exactly-once processing is directly supported for Kafka Streams and can also be achieved using coordinated transactional producer and consumer behaviour.

However, the architecture must avoid the misleading assumption that “exactly once” applies automatically across every external system.

Exactly-once behaviour within Kafka does not necessarily guarantee exactly-once business consequence in:

• an ERP;
• a payment platform;
• a field system;
• an external API;
• or a human process.

End-to-end consequence integrity still requires:

• idempotency;
• transaction boundaries;
• deduplication;
• reconciliation;
• external-system cooperation;
• and Execution Admissibility.

18. Runtime Context Assembly

Enterprise events are major contributors to Runtime Context Assembly.

They provide:

• current state changes;
• recent observations;
• emerging conditions;
• unresolved alarms;
• operational decisions;
• and proposed actions.

The relationship is:

Enterprise Event Products
            │
            ▼
Context Discovery
            │
            ▼
Relevant Event Selection
            │
            ▼
Semantic and Identity Resolution
            │
            ▼
Operational Assembly
            │
            ▼
Context Qualification
            │
            ▼
Qualified Operational Understanding

Kafka supplies current event representations.

Runtime Context Assembly determines which events are relevant to the operational intent.

Kafka does not itself create Qualified Operational Understanding.

19. Enterprise Control Plane

The Enterprise Control Plane coordinates how streaming events participate in constitutional runtime services.

It may coordinate:

• event-product discovery;
• identity resolution;
• authority resolution;
• policy resolution;
• semantic interpretation;
• Runtime Context Assembly;
• Context Qualification;
• Enterprise Coordination;
• Execution Admissibility;
• and Outcome Capture.

Kafka may transport signals between these services.

It is not the Enterprise Control Plane.

20. Relationship to SCIA Runtime

SCIA Runtime implements the Execution Admissibility Service within the Enterprise Control Plane.

Kafka may carry:

• Proposed Action Events;
• context references;
• authority evidence;
• state-change events;
• admissibility requests;
• typed admissibility outcomes;
• execution outcomes;
• and audit evidence.

However, Kafka must not be treated as the enforcement boundary merely because an event has been published.

The non-bypassable control must exist where the operational system commits consequence.

Proposed Action Event
            │
            ▼
Runtime Context Assembly
            │
            ▼
Qualified Operational Understanding
            │
            ▼
SCIA Runtime
            │
            ▼
Typed Admissibility Outcome
            │
            ▼
Operational Commit Boundary
            │
            ▼
Execution Outcome Event

Kafka communicates the proposed action and result.

SCIA resolves admissibility.

The operational system enforces the outcome.

21. Event-Driven AI Participation

AI agents may consume event streams to:

• detect conditions;
• identify patterns;
• explain emerging situations;
• generate recommendations;
• propose actions;
• or coordinate response.

AI-generated outputs must remain distinguishable from accepted enterprise events.

Use separate event classes for:

• observation;
• inference;
• recommendation;
• proposed action;
• accepted decision;
• admissibility outcome;
• and execution outcome.

An AI-generated Proposed Action Event must never be interpreted as an executable command without the applicable authority and admissibility evaluation.

22. Enterprise Coordination

Event streaming enables coordination across:

• Business Domains;
• operational systems;
• workflows;
• humans;
• AI agents;
• and external partners.

Coordination events may describe:

• requested participation;
• task acceptance;
• dependency readiness;
• escalation;
• completion;
• failure;
• compensation;
• and outcome.

The Enterprise Coordination architecture determines how these events form a coherent activity.

Kafka transports coordination signals.

It does not determine coordination authority.

23. Replay and Reconstruction

Kafka’s durable event log can support:

• state reconstruction;
• recovery;
• audit;
• simulation;
• analytical replay;
• and model evaluation.

Replay must be governed according to the purpose for which it is performed.

A conforming replay policy should distinguish:

The default rule should be:

Replaying an event must not replay institutional consequence unless explicitly authorised and controlled.

24. Retention and Institutional Memory

Kafka retention should be configured according to operational communication and reconstruction requirements.

Retention does not determine institutional significance.

Events requiring enduring evidentiary or institutional preservation should be transferred into governed Institutional Memory with:

• accepted provenance;
• immutable evidence;
• retention policy;
• semantic standing;
• access controls;
• and revision history.

Kafka may be one source of that evidence.

It is not necessarily the final preservation environment.

25. Security and Policy

The streaming environment should enforce:

• authenticated producers and consumers;
• encrypted transport;
• topic-level access;
• group-level access;
• environment separation;
• secret management;
• audit logging;
• classification controls;
• and administrative separation.

Platform access control must be supplemented by enterprise policy.

The fact that a consumer can technically read a topic does not prove that it may use the event for every purpose.

Purpose limitation and operational authority remain enterprise responsibilities.

26. Multi-Cluster and Multi-Cloud Streaming

Large enterprises may operate:

• multiple Kafka clusters;
• managed Kafka services;
• cloud-native event platforms;
• regional clusters;
• restricted security zones;
• and partner event environments.

The architecture should not assume a single enterprise-wide cluster.

Federation should preserve:

• domain ownership;
• event identity;
• schema compatibility;
• semantic meaning;
• authority;
• provenance;
• classification;
• and replay controls.

Cross-cluster replication must not silently transform a source event into a new authoritative event.

Replicated events should remain traceably connected to their original product and publication environment.

27. Platform Realisations

27.1 Apache Kafka

Apache Kafka provides the reference implementation through:

• durable topics;
• partitioned event storage;
• decoupled producers and consumers;
• consumer groups;
• Kafka Connect;
• Kafka Streams;
• transactions;
• security;
• and KRaft-based cluster metadata management.

Current Kafka architecture uses KRaft controller quorums to maintain cluster metadata rather than relying upon ZooKeeper in KRaft deployments.

27.2 AWS

AWS may realise the pattern through:

• Amazon Managed Streaming for Apache Kafka;
• Amazon EventBridge;
• Amazon Kinesis;
• AWS Glue Schema Registry;
• Lambda;
• Step Functions;
• and enterprise policy services.

These services may implement different parts of the pattern.

They do not change its constitutional role.

27.3 Microsoft

Microsoft may realise the pattern through:

• Fabric Eventstreams;
• Eventhouse;
• Azure Event Hubs;
• managed Kafka-compatible interfaces;
• Real-Time Intelligence;
• Functions;
• and workflow services.

Fabric may consume enterprise streams for Operational Assembly without becoming authoritative for every source event.

27.4 Databricks

Databricks may participate through:

• Structured Streaming;
• Lakeflow;
• Delta Live Tables or successor declarative pipeline capabilities;
• managed connectors;
• governed data products;
• Unity Catalog;
• and streaming tables.

Databricks may process, qualify and publish derived event products.

It does not become the owner of every originating enterprise event.

28. Governance Responsibilities

29. Failure Modes

The Enterprise Streaming Pattern should explicitly address the following failure modes.

29.1 Orphan Events

Events are published without an identifiable authoritative product or owner.

29.2 Semantic Drift

Payload structure remains valid while business meaning changes.

29.3 Topic Proliferation

Projects create unmanaged topics without stable enterprise purpose.

29.4 Consumer Coupling

Events are designed around the needs of one consumer rather than the meaning of the publishing domain.

29.5 Database Leakage

Internal table structures become long-lived enterprise event contracts.

29.6 False Exactly-Once Confidence

Kafka processing guarantees are assumed to guarantee external business consequence.

29.7 Replay-Induced Consequence

Historical events are replayed into live transactional consumers.

29.8 Unqualified AI Action

An AI-produced event is consumed as an executable instruction.

29.9 Authority Loss

Replicated or derived events lose their relationship to source authority.

29.10 Streaming as the Spine

Kafka is incorrectly positioned as the central source of enterprise meaning or truth.

30. Architecture Decision Records

ADR-ESP-001

Kafka shall be treated as an enterprise communication implementation rather than the Representation Spine or Semantic Spine.

ADR-ESP-002

Enterprise events shall originate from, or remain traceably connected to, authoritative enterprise products.

ADR-ESP-003

Kafka topics shall be governed as transport surfaces belonging to Enterprise Event Products.

ADR-ESP-004

Event schemas shall be governed for semantic and behavioural compatibility, not only structural compatibility.

ADR-ESP-005

Derived event products shall declare lineage, transformation logic and authority status.

ADR-ESP-006

Event replay shall not recreate irreversible consequence by default.

ADR-ESP-007

AI-generated proposed actions shall remain proposals until Execution Admissibility has been established.

ADR-ESP-008

SCIA Runtime shall evaluate consequential proposed actions at the operational commit boundary rather than relying upon event publication as evidence of permission.

ADR-ESP-009

Kafka retention shall not be treated as equivalent to Institutional Memory.

ADR-ESP-010

Multi-cluster and multi-cloud event distribution shall preserve source authority, identity, semantics and provenance.

31. Maturity Model

32. Implementation Roadmap

Phase 1 — Establish Event Ownership

Identify authoritative event-producing domains and assign product accountability.

Phase 2 — Define Event Contracts

Establish common envelope requirements, schemas, semantics, compatibility and lifecycle rules.

Phase 3 — Form Enterprise Event Products

Group topics, contracts, service expectations and policies into governed products.

Phase 4 — Implement Registry and Discovery

Publish event products through the Enterprise Registry with ownership, lineage and semantic references.

Phase 5 — Integrate Runtime Context Assembly

Enable operational intents to discover and consume relevant event products dynamically.

Phase 6 — Govern AI and Proposed Actions

Separate observations, inferences, recommendations and proposed actions into explicit event classes.

Phase 7 — Insert Execution Admissibility

Apply SCIA-compatible evaluation at non-bypassable consequence boundaries.

Phase 8 — Establish Outcome Learning

Preserve execution outcomes, evidence and governed revisions within Institutional Memory.

33. Reference Architecture

34. Non-Goals

The Enterprise Streaming Pattern does not:

• make Kafka the source of enterprise truth;
• require every enterprise event to use one Kafka cluster;
• replace APIs or request-response integration;
• replace Source-Aligned Data Products;
• replace the Enterprise Registry;
• replace enterprise semantics;
• replace Runtime Context Assembly;
• replace Institutional Memory;
• authorise execution;
• or eliminate domain ownership.

It defines how events participate coherently within Enterprise Intelligence.

35. Conclusion

Enterprise streaming is essential to Enterprise Intelligence because enterprise understanding must remain responsive to change.

However, streaming becomes architecturally valuable only when events remain connected to authority, meaning, provenance, policy and consequence.

The Enterprise Streaming Pattern therefore positions Kafka as a governed communication environment through which authoritative enterprise products publish representations of change for participation across the enterprise.

Kafka provides durable event distribution, decoupled producers and consumers, stream processing and integration capabilities.

It does not become:

• the enterprise;
• the Representation Spine;
• the Semantic Spine;
• Institutional Memory;
• the Enterprise Control Plane;
• or Execution Admissibility.

Within the Arqua Enterprise Intelligence Architecture:

• Source-Aligned Enterprise Products establish authority.
• Enterprise Event Products govern representations of change.
• Kafka communicates those events.
• Runtime Context Assembly determines their operational relevance.
• The Enterprise Control Plane coordinates their participation.
• SCIA Runtime evaluates consequential proposed actions.
• Operational systems bind or refuse consequence.
• Outcome Learning preserves accepted evidence and revision.

Kafka carries enterprise change.

Enterprise architecture determines what that change means, how it may participate and whether it may produce consequence.

About Arqua

Arqua develops constitutional architectures and implementation patterns for Enterprise Intelligence.

Arqua defines the enduring architectural responsibilities through which enterprises continuously represent, preserve, assemble, govern and operationalise institutional understanding.

The Enterprise Streaming Pattern demonstrates how event-streaming technologies such as Apache Kafka participate within that architecture while preserving enterprise authority, semantic integrity, execution governance and platform independence.

Core proposition

Kafka is not a spine. It is a governed communication implementation between spines, products, runtime services and execution boundaries.