Skip to content

ET: fail fast after runtime errors - #18

Closed
AFOliveira wants to merge 129 commits into
aifoundry-org:etfrom
AFOliveira:agent/fail-fast-et-runtime
Closed

AFOliveira wants to merge 129 commits into
aifoundry-org:etfrom
AFOliveira:agent/fail-fast-et-runtime

Conversation

@AFOliveira

@AFOliveira AFOliveira commented Jul 17, 2026 •

Copy link
Copy Markdown
Member

What changed

  • record ET stream and kernel-abort callbacks in backend state
  • stop scheduling graph nodes after a runtime failure
  • propagate every ET operation dispatch failure from graph compute
  • treat callback-only failures and synchronization timeouts as fatal
  • drain each ET stream after at most 4,096 queued kernels

Why

The ET runtime can deliver an error through its callback and leave
retrieveStreamErrors() empty. The backend then reported success and continued
scheduling later graphs against an unhealthy card. It also ignored boolean
failures returned by individual ET operation dispatchers.

EventId is 16-bit. A long decode could keep enough work in flight to reuse an
active ID at wraparound, leading to Couldn't dispatch event: 65535 and
firmware errors. The 4,096-kernel drain bound prevents that collision even on
hosts with the older runtime. The allocator-side fix is tracked separately in
aifoundry-org/et-platform#134.

Validation

  • compiled llama-bench with GGML_ET=ON against the ET SDK on aifoundry3
    at the first commit
  • the added stream-bound change will be rebuilt before the controlled
    post-power-cycle hardware test
  • no card reset is used; CI quarantines the card on the first runtime or
    firmware error

glguida and others added 30 commits October 19, 2025 10:02
This commit adds a stub implementation of the ET backend.

Doesn't do anything beside creating the infrastructure that can
be later expanded to effectively implement a backend.

It builds, registers the backend and finds no devices at
initialisation time.
Device with offboard memory is considered GPU.
ACCEL type is for accelerators like Blas.
This change initializes the runtime, and in the process fixes the
initialization of the backend in general.

Currently only supported operation is listing of devices.
GUID is a required property and is checked very early during
initialization.
Buffer usage is decided during model loading, before any interrogation
for ops support. Even if ops are not supported it should be possible
to allocate/set_tensor/get_tensor on device buffers.

Until we implement device buffer management we need to declare host
buffer support, so that fallback to CPU backend works correctly.
The integer device used by ggml for device id is now called devidx.

This will help separate it more clearly from the _runtime_ id.
For now, we can't get the amount of free memory. This will be
fixed in the future through modifications to runtime.

We use free == total, _for_ _now_.
This will help us, in the future, to create a better description
thanks to the runtime device properties.
This will be very useful in future calls to the runtime, to identify
our device.
Allocate reg before initialising devices so that the 'reg' pointer
is valid.
- Allocate/free memory on device
- Implement host-to-device and device-to-host memory transfers
ggml checks existence of these functions and calls then instead
of synchronous ones thus braking inference (specifically
it needs to transfer logits from device to host for sampling).
@github-actions github-actions Bot added documentation Improvements or additions to documentation testing ggml labels Jul 17, 2026
@AFOliveira AFOliveira closed this Jul 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation ggml testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants