Repository navigation
ET: fail fast after runtime errors - #18
Closed
AFOliveira wants to merge 129 commits into
Closed
AFOliveira wants to merge 129 commits into
AFOliveira wants to merge 129 commits into
Conversation
This commit adds a stub implementation of the ET backend. Doesn't do anything beside creating the infrastructure that can be later expanded to effectively implement a backend. It builds, registers the backend and finds no devices at initialisation time.
Device with offboard memory is considered GPU. ACCEL type is for accelerators like Blas.
This change initializes the runtime, and in the process fixes the initialization of the backend in general. Currently only supported operation is listing of devices.
GUID is a required property and is checked very early during initialization.
Buffer usage is decided during model loading, before any interrogation for ops support. Even if ops are not supported it should be possible to allocate/set_tensor/get_tensor on device buffers. Until we implement device buffer management we need to declare host buffer support, so that fallback to CPU backend works correctly.
The integer device used by ggml for device id is now called devidx. This will help separate it more clearly from the _runtime_ id.
For now, we can't get the amount of free memory. This will be fixed in the future through modifications to runtime. We use free == total, _for_ _now_.
This will help us, in the future, to create a better description thanks to the runtime device properties.
This will be very useful in future calls to the runtime, to identify our device.
Allocate reg before initialising devices so that the 'reg' pointer is valid.
- Allocate/free memory on device - Implement host-to-device and device-to-host memory transfers
ggml checks existence of these functions and calls then instead of synchronous ones thus braking inference (specifically it needs to transfer logits from device to host for sampling).
Enhance ET backend and sync upstream llama.cpp changes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Why
The ET runtime can deliver an error through its callback and leave
retrieveStreamErrors()empty. The backend then reported success and continuedscheduling later graphs against an unhealthy card. It also ignored boolean
failures returned by individual ET operation dispatchers.
EventIdis 16-bit. A long decode could keep enough work in flight to reuse anactive ID at wraparound, leading to
Couldn't dispatch event: 65535andfirmware errors. The 4,096-kernel drain bound prevents that collision even on
hosts with the older runtime. The allocator-side fix is tracked separately in
aifoundry-org/et-platform#134.
Validation
llama-benchwithGGML_ET=ONagainst the ET SDK onaifoundry3at the first commit
post-power-cycle hardware test
firmware error