Conversation
🦋 Changeset detectedLatest commit: 644a4c0 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
theomonnom
reviewed
Sep 15, 2026
theomonnom
reviewed
Sep 15, 2026
theomonnom
approved these changes
Sep 15, 2026
u9g
added a commit
to livekit/livekit-cli
that referenced
this pull request
Sep 15, 2026
Follows livekit/protocol#1799 renaming the proto field.
u9g
changed the title
u9g
added a commit
to livekit/livekit-cli
that referenced
this pull request
Sep 15, 2026
Follows livekit/protocol#1799 renaming Job.attempt.
u9g
force-pushed
the
jason/simulation-repeats
branch
3 times, most recently
from
September 15, 2026 19:23
5c16cbf to
0ef9b88
Compare
A run today holds exactly one job per scenario, so a single flaky attempt is indistinguishable from a broken scenario. `Create.Request.repeats` asks for k attempts of every scenario in one run; each Job carries the `scenario_id` and 1-based `attempt` it belongs to so jobs group back into scenarios. `RunMetrics.pass_at_k` (at least one attempt passed) and `pass_pow_k` (every attempt passed) summarize the run over scenarios.
u9g
force-pushed
the
jason/simulation-repeats
branch
from
September 15, 2026 19:34
66effb8 to
be5b074
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A run today holds exactly one job per scenario, so a single flaky attempt is indistinguishable from a broken scenario.
SimulationRun.Create.Request.samples: attempts per scenario (0/unset = 1), requiresscenario_group. Mirrored onSimulationRun.samples.SimulationRun.Job.scenario_id/sample: every attempt of one scenario shares the id, so a run's jobs group back into scenarios. New field numbers; 9 stays reserved.Consumers: livekit/agents-private (agent-service fan-out and aggregation) and livekit/livekit-cli (
lk agent simulate --samples), PRs to follow.