Skip to content

Latest commit

 

History

History
447 lines (370 loc) · 21.8 KB

AGENTS.md

File metadata and controls

447 lines (370 loc) · 21.8 KB

Zilla — Agent Guide

Zilla is a multi-protocol event-driven gateway that bridges HTTP, WebSocket, SSE, gRPC, MQTT, and MCP to Apache Kafka. It is configured entirely via zilla.yaml, with no code changes required for protocol mapping. All runtime behavior is defined by a pipeline of named bindings.


Repository layout

runtime/                     # Core engine and all bindings
  engine/                    # EngineWorker, config loader, stream model
  binding-tcp/               # TCP server/client binding
  binding-tls/               # TLS binding
  binding-http/              # HTTP/1.1 + HTTP/2 binding
  binding-kafka/             # Kafka cache + client binding
  binding-mqtt/              # MQTT binding
  binding-grpc/              # gRPC binding
  binding-sse/               # Server-Sent Events binding
  binding-http-kafka/        # HTTP↔Kafka proxy binding
  binding-mqtt-kafka/        # MQTT↔Kafka proxy binding
  binding-grpc-kafka/        # gRPC↔Kafka proxy binding
  ...
specs/                       # Integration test specifications (IT)
incubator/                   # Bindings under active development
build/flyweight-maven-plugin/ # Code generator for flyweight types (local peer module, not a remote artifact)

Where deeper guidance lives

This file holds repo-wide guidance only. When you start working in a subtree, its AGENTS.md is loaded automatically with more detail:

Path Scope
runtime/AGENTS.md Java module system, stream model, binding kinds, EngineWorker threading, factory + flyweight pattern, server/client/proxy patterns, per-stream field naming, buffer slots, decode strategy, unit tests
runtime/engine/AGENTS.md Engine SPI conventions (Javadoc neutrality, null defaults), EngineContext implementer fan-out, test implementations for engine concepts
runtime/binding-kafka/AGENTS.md Kafka local cache (mmap segments, IoUtil.unmap(), retention timestamps)
runtime/binding-llm/AGENTS.md llm section contract: body members and attributes, ranges, guardrail pass-through, model-visible value per section, streaming vs non-streaming
specs/AGENTS.md .rpt script structure, folder layout, IT method naming, XxxFunctions builders/matchers, JUEL typed variants, k3po + JUnit 4 migration, JSON schema patches, required spec coverage

Build

Zilla uses Maven with Java 25.

Every new Maven project directory must include mvnw and mvnw.cmd copied from an existing module — this applies to all new projects regardless of type (runtime/, specs/, incubator/, etc.).

# Add license headers to new files — run this first after creating new source
# files, otherwise the build will fail on the license check before compilation
./mvnw license:format

# Full build with tests
./mvnw install

# Skip integration tests (faster)
./mvnw install -DskipITs

# Skip all tests
./mvnw install -DskipTests

# Build a specific module
./mvnw install -pl runtime/binding-http -am

# Run a single unit test class (*Test.java, no k3po) — class names are type-prefixed: Http*
./mvnw test -pl runtime/binding-http -Dtest=HttpConfigurationTest

*IT.java classes require verify, never test — this is not optional. Every k3po-based integration test class is named *IT.java and depends on the k3po:start/k3po:stop goals, which are bound to the pre-integration-test/ post-integration-test lifecycle phases. Those phases only run as part of verify (or integration-test); the test phase never reaches them. Running ./mvnw test -Dtest=SomeIT will still find and attempt the class (-Dtest bypasses Surefire's normal *Test.java-only file pattern), but the k3po control server is never started, so every test fails with a misleading Failed to connect. Is K3PO ready? — which looks like a test failure but is actually a wrong-command error. If you see that error, the fix is almost never in the test or the code — it's the command:

# WRONG for *IT.java — k3po never starts, fails with "Is K3PO ready?"
./mvnw test -pl runtime/binding-http -Dtest=HttpServerIT

# RIGHT — runs the full lifecycle including k3po:start/k3po:stop
./mvnw verify -pl runtime/binding-http -Dit.test=HttpServerIT

# RIGHT — no -Dit.test filter runs every IT in the module
./mvnw verify -pl runtime/binding-http

A crash or hang from test -Dtest=SomeIT (SIGSEGV, "forked VM terminated without properly saying goodbye", event-assertion mismatches) is a symptom of this same root cause, not a real bug — before spending time diagnosing a crash from an *IT run, confirm it was launched with verify.

Build cache

.mvn/extensions.xml enables the Maven build cache extension. A module whose inputs and reactor dependencies are unchanged is restored from ~/.m2/build-cache without running its tests or ITs; a changed module rebuilds, and so does every module that depends on it.

  • Use clean install for local builds — a rebuilt module fails in moditect if its target/modules jar already exists
  • Build a subset with -pl <module> -am; without -am its siblings resolve from ~/.m2 and the module misses entries from a full build
  • Force a full rebuild with -Dmaven.build.cache.skipCache=true, or bypass the extension with -Dmaven.build.cache.enabled=false
  • CI restores ~/.m2/build-cache from the OCI artifact ghcr.io/aklivity/zilla/build-cache:<branch>: a pull request restores its base branch and then develop, and a push to develop or support/** publishes it from a separate job, the only one with packages: write (.github/scripts/build-cache.sh). A missing artifact is a cold build, never an error. Each publish leaves the previous package version untagged, so the same job deletes untagged versions beyond the latest three; that needs the Admin role for the repository under the package's Manage Actions access, and a failure does not fail the publish
  • A local session (e.g. a sandboxed agent environment with no pre-populated ~/.m2) can restore the same cache manually instead of building the whole reactor from source — do this before any full ./mvnw install at the repo root, especially when a dependent repo (e.g. zilla-plus) needs zilla artifacts this session hasn't built yet:
    bash .github/scripts/build-cache.sh install   # installs oras into ~/.local/bin
    apt-get install -y zstd                       # only if zstd isn't already present
    bash .github/scripts/build-cache.sh restore "$HOME/.m2/build-cache" \
        ghcr.io/aklivity/zilla/build-cache develop
    Restoring falls back silently to a cold build if the ref/tag isn't found, so it's always safe to attempt first. Once restored, a full ./mvnw clean install -Dskip.checks=true from the repo root is a near-instant cache-hit build for every unchanged module instead of a from-source rebuild.
  • Do not add a direct plugin goal (for example jacoco:report-aggregate) to a cached build command — it disables caching for the whole build
  • A module that must always build opts out with <maven.build.cache.enabled>false</maven.build.cache.enabled> in its own <properties>: cloud/docker-image and cloud/helm-chart (a cache hit would skip the image build)
  • A module that shades its dependencies marks them <optional>true</optional>, sets createDependencyReducedPom to false and excludeOptional to false on maven-notice-plugin, so the installed pom is the source pom and the NOTICE still lists the shaded libraries (manager, incubator/command-log)

Docker

Cloud sessions have Docker installed but the daemon is not running at start, so docker info fails with Cannot connect to the Docker daemon. That does not mean Docker is unavailable: start the daemon yourself and carry on.

(nohup dockerd > /tmp/dockerd.log 2>&1 &); sleep 8; docker info

Do this before concluding that an example (examples/*/.github/test.sh, docker compose) or a Docker image build cannot be run locally. Run the example's own check rather than only the ITs when a change affects what the example exercises.

The Maven plugin in build/flyweight-maven-plugin/ generates flyweight Java classes from .idl files. Always run a full build after modifying any .idl file. It is a peer reactor module, not a published remote artifact — it comes from this same checkout, not from a Maven registry.


Test-first discipline

Zilla follows a strict test-first discipline. For every new feature or bug fix:

  1. Write the spec script(s) and/or unit tests first, before any implementation
  2. Confirm the tests fail for the right reason against the current code
  3. Implement until the tests pass
  4. Do not open a PR with implementation code that has no corresponding tests

This is especially important for new bindings. The spec scripts define the correct protocol behavior; the Java implementation exists to satisfy them. Never write implementation code and retrofit tests to match it — that defeats the purpose.

Spec script conventions and unit-test conventions are detailed in specs/AGENTS.md and runtime/AGENTS.md.

IT structure: one IT per component, one method per scenario

Each component under test (binding, guard, store, catalog, vault, exporter, ...) has one dedicated IT class named for it (for example HttpServerIT, EngineIT); every scenario for that component, including shutdown, error and event-ordering scenarios, is a test method in that class with its own @Configuration and @Specification. Do not add an IT class per scenario or concern (XxxCloseIT, XxxEventIT, XxxRegressionIT).

When a scenario needs extra fixtures (a held HTTP endpoint, extra engine properties), keep them in a small helper in the module's test tree that the test method creates and closes itself, or that stays inert until a test arms it, instead of forking the class. A separate class is for a genuinely different component or protocol side, not for another scenario of the same component: spec-level peer-to-peer ITs are split per protocol side (NetworkIT, ApplicationIT) or per protocol (McpIT, KafkaIT), see specs/AGENTS.md. If you believe a new IT class is justified, say why in the PR description.


Adding a new binding

Repo-level scaffolding (everything before module-specific implementation):

  1. Open a GitHub Issue to discuss the design before writing any code
  2. Create specs/binding-<n>.spec/ and write .rpt scripts for the happy path and key error scenarios, derived from the relevant protocol specification — see specs/AGENTS.md
  3. Create runtime/binding-<n>/ and specs/binding-<n>.spec/ following the existing module layout. Every new project directory (both runtime/ and specs/) must include these top-level files copied from an existing module: COPYRIGHT, LICENSE, NOTICE, NOTICE.template, mvnw, mvnw.cmd. All new components use the Aklivity Community License — copy LICENSE-AklivityCommunity, COPYRIGHT-AklivityCommunity, and NOTICE-AklivityCommunity from the top-level repository directory, renaming them to LICENSE, COPYRIGHT, and NOTICE.template respectively in the new module. Then generate NOTICE by running ./mvnw notice:generate --projects <path/to/project> from the repository root; do not copy NOTICE from another module as it must reflect the new module's actual dependencies. Never edit NOTICE files directly — always regenerate via ./mvnw notice:generate --projects <path/to/project>; manual edits will be overwritten. Source file headers must carry the Aklivity Community License copyright notice (Copyright 2021-2024 Aklivity Inc); run ./mvnw license:format to apply the correct header automatically
  4. Add the module to runtime/pom.xml and the root pom.xml

Then proceed with the runtime-side implementation steps in runtime/AGENTS.md and the spec/IT steps in specs/AGENTS.md. Confirm ./mvnw install passes including all ITs before opening the PR.


Code style

All Java code must pass the project checkstyle rules defined in conf/src/main/resources/io/aklivity/zilla/conf/checkstyle/configuration.xml. Run ./mvnw checkstyle:check to verify before committing. Key rules to be aware of: 4-space indentation (no tabs), 130-character line limit, opening braces on a new line (LeftCurly option nl), closing braces alone on their own line (RightCurly option alone), no trailing whitespace, imports ordered by group (java, javax, jakarta, org, com) with a blank line between groups and no star imports.

  • YAML files use 2-space indentation, no tabs; JSON files use 4-space indentation, no tabs
  • Prefer non-block lambdas (expression lambdas) over block lambdas ({ return ...; }) — even when the expression spans multiple lines via a builder chain, keep it as a single expression without braces or an explicit return
  • Method parameters are each on their own line, indented 4 spaces relative to the method declaration, with the closing ) on the same line as the last parameter:
private void onNetworkData(
    long traceId,
    long authorization,
    int reserved,
    OctetsFW payload)
{
  • Methods should have a single return statement at the end where possible; avoid early returns except for guard clauses at the very top of a method
  • Avoid the ...IfNecessary method naming suffix (e.g., doEndIfNecessary, cleanupDecodeSlotIfNecessary) — name methods for what they do (doEnd, cleanupDecodeSlot); internal conditionality based on stream state or slot value is an implementation detail that does not belong in the name
  • Name fields, parameters, and locals for the role they play in production, not for how they feel to write. Avoid names that read as placeholder or debug scaffolding (scratch, tmp, foo, data2) — even a reusable buffer with no domain meaning of its own needs a name describing its role (e.g. transformBuffer, not scratch). Equally, avoid names so verbose they restate the type or the enclosing class (kafkaCacheModelTransformer inside KafkaCacheModel, httpServerFactoryFlyweightBuffer as a field on HttpServerFactory) — once the surrounding type and class already carry that meaning, repeating it in the name is noise, not clarity. A short, precise name beats both extremes
  • Prefer self-describing code — clear names and small, well-named methods — over comments that explain what the code does. Default to no comment; add one only when removing it would lose a non-obvious WHY that the code itself cannot express (a hidden constraint, a workaround for a specific bug, a subtle invariant). The license header is the only comment every file requires
  • Java 21; no preview features
  • No Lombok
  • Use jakarta.json APIs (e.g., JsonObject, JsonReader, JsonParser) for JSON processing — do not introduce Jackson (com.fasterxml.jackson)
  • Prefer interface types over implementation classes for field, parameter, and return types where a suitable interface exists (e.g., List over ArrayList, Map over HashMap, ConcurrentMap over ConcurrentHashMap)
  • Never use fully qualified class names as field, parameter, or variable types — add an import and use the simple type name. The only exception is a naming collision where two different packages define the same class name; in that case qualify the less-frequently-used type
  • Package-private classes preferred over public where there is no SPI contract
  • final on all fields; immutable config objects
  • Flyweight field names use the *RO / *RW suffix convention consistently
  • Error paths must call cleanup() and release any acquired resources before returning
  • Log via the Zilla event system (BindingEvent), not java.util.logging or SLF4J, on the hot path

Import ordering

Checkstyle enforces a strict import order. Violations cause build failures, so always sort imports alphabetically by fully-qualified package name within each group, and separate groups with a blank line in this order:

  1. java.*
  2. javax.*
  3. jakarta.*
  4. org.*
  5. com.*
  6. io.* (covers all io.aklivity.zilla.* imports)

Within the io.aklivity.zilla.runtime.engine.* sub-packages the alphabetical rule means, for example:

import io.aklivity.zilla.runtime.engine.catalog.CatalogHandler;   // c
import io.aklivity.zilla.runtime.engine.guard.GuardHandler;        // g
import io.aklivity.zilla.runtime.engine.model.ValidatorHandler;    // m
import io.aklivity.zilla.runtime.engine.poller.PollerKey;          // p  ← before store
import io.aklivity.zilla.runtime.engine.store.StoreHandler;        // s  ← before vault
import io.aklivity.zilla.runtime.engine.vault.VaultHandler;        // v

When adding a new import, insert it at the correct alphabetical position — do not append it at the end of the group.


Key dependencies

Dependency Purpose
agrona Lock-free ring buffers, flyweight buffer access, IoUtil for mmap
zilla:maven-plugin Generates flyweight Java from .idl type definitions
junit5 Unit and integration tests
mockito Mocking in unit tests

Contribution workflow

  1. Fork the repo and create a branch using gitflow naming conventions: feature/<short-description> for new features, or fix/<issue-number>-<short-description> for bug fixes
  2. Make changes; ensure ./mvnw install passes with no failures
  3. Open a PR against the develop branch (not main)
  4. PRs require at least one approving review from a maintainer
  5. Commit messages follow Conventional Commits: feat(binding-http): add trailers support fix(engine): release mmap'd segment on log rotation
  6. Do not include generated sources (target/) or IDE files in commits

Diagnosing PR build failures

When a PR build fails, always fetch the actual CI logs before attempting to diagnose or fix the failure. Do not guess at the cause based on the change diff alone — retrieve the logs first using GitHub MCP tools, or ask the user to provide them. Build failures often have non-obvious root causes (e.g., a checkstyle violation, a transitive module-info issue, or a flaky unrelated test) that are impossible to diagnose without the log output.

For significant new bindings or behavior changes, open a GitHub Issue first to discuss the design before writing code.

Justifying changes to shared components

Many modules in this repo exist to be depended on by other modules — both inside this repo (common-avro, common-json, common-protobuf, the engine SPI, any module under runtime/*/ext) and beyond it. When filing an issue or writing a commit/PR description for a change to one of these, justify it entirely in terms of that component's own contract or capability gap — never by naming or describing a consumer that depends on it: not a sibling module, not a downstream binding, not a commercial extension, not a specific product or feature.

Test before writing: strip away everyone who currently depends on this component — present and future, in-tree and external. Does the description still stand on its own as "this component's contract had gap Y, now it has capability Z"? If it only makes sense once the reader knows who wanted it, the description is coupling this component to a consumer's specifics, and should be rewritten in terms of the gap itself.

This applies in both directions across the dependency graph: a common-avro change should never cite model-avro's use case as its motivation, an engine SPI change should never cite a specific binding's or model extension's use case, and this repo's own changes should never cite a downstream or commercial consumer's feature as their reason for existing.


Implementing against established patterns

Get it right the first time by adopting the pattern the codebase already uses for the concern in front of you, not by inventing a mechanism and iterating.

  • Before writing code for a concern (flow control, buffering, decode/encode slots, stream pipelines, section handling, window sizing), find the closest existing implementation in the same module and in sibling modules, read how it solves that exact problem, and mirror it. Say which implementation you are mirroring before you start.
  • A first draft that needs a reserve, a fixed slack, a fudge fraction, a "reject if it overshoots" guard, a flag that only exists to dodge a case, or a change to a test script so it fits the implementation, means the established pattern was missed. Stop and find the pattern. Do not ship the workaround and do not offer it as an option to be replaced later.
  • Correctness comes from honoring the protocol contract, not from margins. Flow control credit is a contract in every direction; see runtime/AGENTS.md for how it is honored.
  • Do not defer a correctness fix as a "larger change, say if you want it". Build the proper behavior in the first change.
  • Do not add a boolean field to track which phase you are in. Phase is expressed by decoder states (the decode<StateName> methods) and the XxxState bitmask; values carried to a later decode step stay as named fields. See runtime/AGENTS.md. Before adding any field, check whether that phase already exists somewhere and derive it instead.
  • If you cannot find an established pattern for the concern, ask before inventing one.
  • Fix a spec script only when it violates the protocol, such as a frame that carries a new extension without the init flag; never bend a script to fit implementation behavior.

Useful references