Skip to content

feat(esi): streaming processor with parallel includes, bump Go to 1.27 - #16

Merged
darkweak merged 4 commits into
masterfrom
feat/stream-parallel-esi
Nov 10, 2026
Merged

darkweak merged 4 commits into
masterfrom
feat/stream-parallel-esi

Conversation

@darkweak

@darkweak darkweak commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • esi.NewStream(dst, req): an io.WriteCloser that processes ESI tags while the document is written. Static content goes out immediately, includes are fetched concurrently and written in document order, and the destination is flushed whenever output waits on an include or more input. Tags split across writes are handled; Write never blocks on an include.
  • esi.Parse: same signature, now fetches includes in parallel and builds the output in one pre-sized allocation instead of re-appending the whole buffer after each tag.
  • writer.Writer: thin http.ResponseWriter over the stream. Middlewares become NewWriter → next.ServeHTTP → Close, removing the busy-wait goroutine (and its data race) from caddy, roadrunner and traefik.
  • Efficiency: byte search instead of regexes for tag/closing detection (offline parse ~2× faster, ~90× less memory); includes share a pooled transport (64 idle conns/host), use the request context, and no longer leak the response body on alt fallback.
  • Go 1.27 across all modules and CI; caddy upgraded v2.6.4 → v2.11.9 (old quic-go doesn't build on Go ≥ 1.21); golangci-lint config migrated to v2; traefik vendor refreshed.

Breaking changes / behaviour differences

  • writer.NewWriter(rw, rq) no longer takes a *bytes.Buffer; AsyncBuf, Done, Iteration removed; Close() must be called.
  • Upstream status codes are passed through (previously always 200).
  • Content-Length is dropped properly; the invalid Content-Encoding: chunked header is no longer set.
  • Unsupported tags (e.g. <esi:try>) are emitted verbatim instead of crashing Parse.
  • A failing include without onerror="continue" is removed instead of truncating the rest of the document.
  • Parse no longer mutates its input.

Test plan

  • go test -race ./... in the root module (tests now start their own fixture server on :9080, or reuse the CI caddy)
  • New tests: every chunk split / byte-by-byte streaming vs Parse, parallel includes (20 × 200 ms in ~210 ms), flush before slow include, write errors, writer status/Content-Length handling
  • caddy middleware integration test and roadrunner tests under -race
  • golangci-lint run (v2.13): 0 issues on the root module
  • Traefik plugin loads under Yaegi (traefik:latest, middleware/traefik stack) and processes vars, remove, include and alt fallback. Yaegi does not support generic stdlib types nor the clear/min builtins, so the core package avoids them.

darkweak and others added 3 commits November 9, 2026 18:15
- Add esi.Stream, an io.WriteCloser that processes ESI tags incrementally:
  static content is forwarded immediately, include tags are fetched
  concurrently and written in document order, and the destination is
  flushed whenever the output waits for an include or more input.
- Parse now fetches independent includes in parallel and assembles the
  output in a single pre-sized allocation.
- Share one incremental scanner between Parse and Stream; replace the
  tag/closing regexes with byte searches.
- Includes reuse a pooled transport, honour the request context and no
  longer leak the failed response body on alt fallback.
- Rewrite writer.Writer as a thin http.ResponseWriter over esi.Stream
  (passes status codes through, drops Content-Length, no busy loop);
  simplify the caddy, roadrunner and traefik middlewares accordingly.
- Bump Go to 1.27 across modules and CI, upgrade caddy to v2.11.9,
  migrate golangci-lint config to v2, re-vendor the traefik plugin.
- Tests start their own fixture server and cover chunk boundaries,
  parallelism, early flushing and write errors.

BREAKING CHANGE: writer.NewWriter(rw, rq) no longer takes a buffer, the
AsyncBuf/Done/Iteration fields are removed and Close must be called.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- actions/checkout v7, actions/setup-go v7 (Node 24 runtime)
- golangci-lint-action v9 with golangci-lint v2.15: the v2.13.2 binary
  cannot read the Go 1.27.3 export data ("export data version 5 is
  greater than maximum supported version 4")
- goreleaser-action v7 pinned to GoReleaser "~> v2"; --rm-dist was
  removed in v2, use --clean

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Yaegi rejected the plugin with "stream.go:128:1: cannot define new
methods on non-local type struct {}", then "undefined: clear" and
"undefined: min".

- Guard the stream error with a mutex instead of atomic.Pointer[error]:
  Yaegi cannot resolve generic standard library types.
- Replace the clear and min builtins with explicit code.
- Refresh the stale traefik vendor copy.

Validated with traefik:latest through middleware/traefik: the plugin
loads and processes vars, remove, include and alt fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@darkweak
darkweak force-pushed the feat/stream-parallel-esi branch 3 times, most recently from c138265 to 86ec746 Compare November 9, 2026 18:16
@darkweak
darkweak force-pushed the feat/stream-parallel-esi branch from 86ec746 to 0af7fb5 Compare November 10, 2026 09:50
@darkweak
darkweak merged commit 8d86ced into master Nov 10, 2026
3 checks passed
@p0358

p0358 commented Nov 10, 2026

Copy link
Copy Markdown

Dang it, I did local ESI reimplemention 2 weeks ago and forgot to PR it xD

Should I still push it somewhere for comparison in case there's anything useful there to salvage?

@darkweak

Copy link
Copy Markdown
Owner Author

I think you can push a PR 😄

@darkweak
darkweak deleted the feat/stream-parallel-esi branch November 10, 2026 12:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants