Egil.Orleans.Messaging - #58
Conversation
|
Review findings from the feature branch review:
Signed-off-by: GPT-5 |
|
@copilot resolve the merge conflicts in this pull request |
Conflicts resolved in |
egil
changed the title
Comprehensive API design document covering all 8 design sections: - §0: Scope, packaging (one NuGet, split-ready), name settled as Egil.Orleans.Messaging - §1: IStateManager<T> atomic writes with in-flight recovery - §2: Outbox<T> storage placement (co-located on grain state) - §3: Outbox<T> sealed class shape, epoch semantics, fingerprint equality, configurable max depth - §3a: VersionedState<TSelf> for ImmutableArray equality trap - §4: MessageTracker sealed class, dual-dictionary dedup - §5: StreamManager fluent builder facade - §6: Opinionated functional-grain pattern (guidance, not enforced) - §7: OutboxProcessor<T> grain-scoped timer+reminder dispatch with callback-based postmen, per-attempt error tracking - §8: Naming (flat namespace), serialization ([Alias] with fully qualified prefix, sequential [Id]), telemetry conventions All decisions settled via structured grilling sessions. Directory still named Egil.Orleans.CQRS — rename deferred to project scaffold. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rename directory from Egil.Orleans.CQRS to Egil.Orleans.Messaging. Add solution file, Directory.Packages.props, version.json, global.json, src csproj (library), and test csproj. Package description: toolbox of composable building blocks — not a framework. Pick what you need, leave the rest. [skip notes] Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Limit StreamCursorJsonConverter to Orleans built-in stream sequence token types and remove custom token adapter serialization paths. Unsupported token kinds now throw a NotSupportedException with an actionable message that explains the stability rationale and points users to the GitHub issues page for feature requests. Update StreamCursorJsonConverterTests to assert unsupported custom tokens and unknown token kinds fail with the new guidance message while keeping built-in token round-trip coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Enable JSON converter attribute on OutboxSequenceToken and implement OutboxSequenceTokenJsonConverter read/write paths. The converter now round-trips sequence, sender GrainId shape, timestamp, and epoch without requiring caller-side serializer options. Add focused tests for converter attribute registration, round-trip behavior (including multiple GrainId key shapes), and payload validation for missing sender. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Enable JSON converter factory attribute on OutboxMessageEnvelope<T> and implement OutboxMessageEnvelopeJsonConverterFactory with closed generic converters for read/write round-trip serialization. Add tests for attribute wiring, CanConvert behavior, round-trip serialization, and missing-token payload validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implement StateManager<T> state cache, read/write/clear operations, force-write ETag override, and conflict handling that reconciles equivalent persisted state before rethrowing non-equivalent failures. Add focused tests for read refresh, successful writes, write-failure reconciliation behavior, rollback on conflicting writes, rollback when post-failure read fails, and VersionedState version stamping. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implement AsStateManager<T> to validate input and wrap IPersistentState<T> with StateManager<T> for grain-facing usage. Add focused tests for successful wrapping and null-argument validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Enable StreamCursor JSON converter attribute and implement TryGetEnqueuedTime, TryGetStreamProviderName, and TryGetTraceParent to expose enrichment metadata when backed by EnrichedEventHubSequenceToken. Add focused tests for positive and negative cases across enqueued time, stream provider name, and traceparent access. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implement UseEnrichedDataAdapter extension to validate input and register EnrichedEventHubAdapter through the Event Hub configurator using Orleans Serializer from DI. Add focused tests for null-argument validation and configurator callback registration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implement EnrichedEventHubAdapter overrides to stamp traceparent on queued messages, create EnrichedEventHubSequenceToken from cached Event Hub messages, and return enriched stream positions that preserve offset/sequence/event index and optional traceparent. Add focused tests for traceparent stamping behavior and enriched token construction from cached message metadata. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add InitializeStateManager(...) that resolves keyed IStateManagerFactory<T> by storage name, with IServiceCollection and ISiloBuilder helper APIs for keyed singleton registrations. Refactor state management to StateManagerBase<T>/DefaultStateManager<T>, remove WritePolicy and force-write behavior, and drop reflection-based ETag mutation. Update state manager tests and API design docs to match the new factory wiring and write semantics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
StreamManager now separates subscription configuration from activation-time subscription setup via RegisterStreamManager, AddSubscription, and SubscribeAsync. This lets grains await Orleans stream subscription establishment without blocking activation and keeps subscription failures visible. MessageTracker now records receive-lag histograms for accepted enriched stream cursors and outbox tokens, with shared telemetry plumbing kept in MessagingTelemetry. The Messaging test suite adds an in-process Orleans cluster with memory streams covering ValueTask, Task, default error handling, and resume-token behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rename InitializeStateManager to RegisterStateManager so state and stream activation helpers use the same registration language before the package ships. Update state manager docs and tests to match the new API name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Capture the current OutboxProcessor design direction, including background posting, timer interleaving options, postman execution modes, and related telemetry guidance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix stream subscription initialization so handles are collected without concurrent List mutation, repeated SubscribeAsync calls are rejected, and stream handlers receive nullable sequence tokens instead of relying on null-forgiving calls. Recover StateManager clear failures with a read-back path, clamp negative receive-lag telemetry, validate malformed outbox JSON with JsonException, and document the intentional outbox equality fingerprint invariant. Rename the deferred outbox processor activation API references to RegisterOutboxProcessor to align with the other activation-time helpers while leaving the implementation deferred. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implements the outbox processor registration and draining path, including postman dispatch, success acknowledgement, failure reconciliation, retry timer/reminder scheduling, and stream-backed integration coverage. Adds outbox and stream receive telemetry plus focused Orleans in-process tests for stream delivery, error paths, resume behavior, JSON converter branches, and tracker semantics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds targeted tests for high-risk converter and tracker branches, including stream cursor token variants, malformed persisted JSON, and same-count MessageTracker equality mismatches. These tests raise branch coverage around runtime serialization and dedup semantics without padding low-value argument validation paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Stream cursors now carry namespace and provider metadata instead of using the Orleans stream id as the persisted tracking identity. Provider names are populated from subscription handles, with enriched Event Hub tokens taking precedence when present. This keeps tracking grain-local while still supporting provider-aware deduplication for non-conventional multi-provider namespace usage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document the Messaging package entry points, add it to the repository package list, and register its conventional commit scope. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the Messaging package workflow, include the package README in the NuGet artifact, and make Messaging tests consume Egil.Orleans.Testing as a package dependency instead of a sibling project reference. [skip notes] Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Release the drain gate between background dispatch and non-interleaving reconciliation, and let a foreground drain finish any pending batch before taking a new snapshot. This prevents non-reentrant grain calls from blocking the reconciliation turn while preserving direct PostAsync completion semantics.
Accept an explicit UTC append timestamp while preserving the system-clock convenience overload. Persisted outboxes no longer retain a TimeProvider, so serialization and rehydration need no clock re-registration.
Keep reconciliation ownership visible until callbacks complete so an interleaving empty post cannot cancel acknowledged work. Processing timeouts now accept a scoped TimeProvider, and tests use deterministic gates and clocks.
Fail immediately when a grain activation attempts to register a second processor, preserving the original reminder component and its durable retry path. Document the one-processor invariant and cover different-type registration through the reminder DIM.
Retain a reminder handle loaded from a previous activation before unregistering it, so a transient unregister failure can be retried by the next empty outbox drain.
Adopt durable state after a successful recovery read before rethrowing mismatches and concurrency conflicts. This keeps recovered state paired with the refreshed ETag so retries cannot overwrite concurrent writes.
Read cursor, discriminator-envelope, built-in token, and Event Hub token properties by name while ignoring unknown fields. Required fields, known value types, and duplicate properties remain strict so rolling upgrades gain forward tolerance without accepting ambiguous payloads.
Pin every private tracker model property to its canonical wire name so serializer naming policies cannot change persisted state. Require both root collections so mismatched or malformed payloads fail instead of silently clearing deduplication state.
Generate plain-text release notes from Messaging project commits and inject them into all three NuGet packages during Release packing.
Add Egil.Orleans.Messaging to the manual release workflow so its stable release branch can be prepared consistently with sibling projects.
Keep only Azure.Core as a runtime dependency of the Azure Storage state-manager companion. Move Table and Blob SDK references to tests and leave Orleans provider selection to consumers, avoiding unwanted provider and version coupling.
Annotate IStateManager<T>.State for storage providers that can expose null after reads or clears, and handle null recovery values without dereferencing them. Preserve Orleans activated defaults while documenting and testing both nullable and non-null missing-record shapes.
Keep InconsistentStateException observable when clear recovery finds a missing record, while adopting the provider's recovered state. Document the throwing-path state contract and cover it with a deterministic regression test.
Wrap Event Hubs batch containers so stream consumers receive broker enqueue time, provider name, and trace context on batch and per-event sequence tokens. Cover the cached-message and Orleans serialization pipeline deterministically.
Derive namespace-based explicit subscription IDs from the complete Orleans GrainId and expose the same helper to producers. Reject textual identities that do not round-trip instead of silently collapsing them. Cover compound keys, hex-parseable strings, cross-kind collisions, Orleans binary serialization, and StreamId text parsing. Document producer wiring and migration. BREAKING CHANGE: Namespace-based explicit subscriptions now use the complete GrainId as the StreamId key. Publishers must call StreamManager.CreateStreamId with the receiving GrainId, and existing durable key-only subscriptions must be recreated or configured through the explicit StreamId overload.
Pin same-epoch higher-sequence acceptance and subsequent duplicate rejection so an inverted comparison fails deterministically.
|
Regarding the first finding in #issuecomment-4679273951: fixed in 62ad15e and documented in 2689d0b. The README now configures OutboxProcessor with OutboxMessageEnvelope, acknowledges successful items by their OutboxSequenceToken values, and explicitly warns that the success batch need not be a contiguous prefix. Outbox.RemoveRange removes matching tokens across gaps, with deterministic coverage in RemoveRange_removes_matching_tokens_after_gap_and_preserves_remaining_order. This prevents a successful later dispatch from causing the failed FIFO head to be removed. Current Release suite passes 242/242; pack passes. |
Document the intentional O(1) recovery fingerprint with the exact removal-gap collision from review. Removal-only divergence can cause duplicate delivery but cannot lose an item which still required delivery.
|
Regarding the second finding in #issuecomment-4679273951: the [1,2,4] versus [1,3,4] collision is factual but intentional, so no production change. Outbox.Equals is documented as an O(1) state-recovery fingerprint, not structural collection equality. If two histories diverge only through removals, every token absent from either branch was already delivered or intentionally reconciled by that branch. Adopting the other history can retain an already handled token and cause duplicate delivery, but cannot lose an item which still required delivery. Append divergence is protected by the full first/last tokens; direct ETag conflicts are rethrown; VersionedState recovery compares its stamped version instead. Commit a2432dc pins the exact [1,2,4] / [1,3,4] equivalence and matching hash codes in a deterministic test. Release tests pass 243/243; pack passes. |
|
Regarding the third finding in #issuecomment-4679273951: fixed in fb53276. MessageTracker.OutboxEntry now stores the sender-stamped LastTimestamp separately from receiver-side Received. LatestOutbox reconstructs the token with LastTimestamp, while Received remains dedicated to eviction timing. The STJ model persists LastTimestamp and falls back to the legacy Received value when reading payloads written before the field existed. LatestOutbox_preserves_sender_timestamp_when_receiver_clock_differs deterministically sets the receiver one hour ahead and asserts exact token equality. Current Release suite passes 243/243; pack passes. |
|
Resolved by 74b6de7. The prepare-release project matrix now includes Egil.Orleans.Messaging, and .github/workflows/egil-orleans-messaging-ci.yml targets the same project. No additional change is needed for this finding. |
|
Resolved. InconsistentStateException recovery now has deterministic coverage: 11693ba covers adopting the persisted value and always rethrowing a write conflict (including coincidental equality); 0da0d8a covers recovery-read failure restoring the prior state and rethrowing the original conflict; and 22d8f8b covers a clear conflict remaining visible even when the recovery read observes deletion. These tests use fake persistent state and have no timing dependency. |
|
Resolved by 3a44a76. Added deterministic coverage for an ambiguous ClearAsync failure whose recovery read shows that the record still exists. The test verifies that the manager adopts the persisted state and refreshed ETag, preserves RecordExists, and rethrows the original clear exception. Full Release suite passes: 244/244. |
|
Resolved by 41ca928. Added an Orleans-hosted test against the fixture's real in-memory reminder service: a failed foreground dispatch returns only after registering the durable reminder, which the test verifies through GetReminder; a successful retry drains the outbox and returns only after unregistering it, which the test verifies is absent. This avoids waiting for Orleans' wall-clock scheduler. Reminder-name callback routing remains covered separately. Full Release suite passes: 245/245. |
Pin Microsoft.OpenApi 2.7.5 for .NET 10 so restores avoid the high-severity circular-reference parser vulnerability while .NET 9 remains on its unaffected 1.x API line.
|
CI blocker resolved by 51b4893. The .NET 10 abstractions target now pins Microsoft.OpenApi 2.7.5, the first patched 2.x release for GHSA-v5pm-xwqc-g5wc; .NET 9 remains on unaffected 1.6.17. NuGet reports no vulnerable packages across the StronglyTypedPrimitives solution, local Release tests pass 213/213, Release pack succeeds, and all restarted GitHub checks now pass. |
No description provided.