Skip to content
View vito-lin-dev's full-sized avatar

Block or report vito-lin-dev

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
vito-lin-dev/README.md

Vito Lin

I work on the open-source LLM serving stack: inference routing, GPU fault handling, and the vLLM engine. Most of my contributions start from a failure I traced to its root cause and then fixed upstream.

In progress

llm-d-router

  • The SLO headroom tier filter trusted latency predictions for endpoints whose metrics had never been scraped. The fix skips those endpoints (#4293, in review), and a follow-up stops the predicted-latency producer from publishing predictions for them in the first place (#4333, assigned).
  • The latency SLO admitter's idle check reads a different signal than its README documents (#4246, assigned).

NVSentinel

  • The MongoDB store refuses to start on Linux kernel 6.19 and later, including Ubuntu kernels that already carry the upstream fix. I documented the workaround the maintainers chose (#2416, in review; issue #2410).

vLLM: engine lifecycle

  • Requests still running at the --shutdown-timeout deadline never received a final chunk, and the API server never exited. The fix aborts the remaining work at the deadline (#64991, in review), with a matching drain-shutdown section for the Kubernetes guide (#64995, in review).

vLLM: Rust frontend parity with Python

  • A partially cached Hub snapshot made the frontend skip generation_config.json, which broke stop tokens (#64768, in review).
  • An auto tool parser that cannot resolve for the served model now warns at startup instead of failing on every request (#64069, in review).

Merged

  • vllm-project/vllm #63580: Rust frontend returns 400 instead of 500 when tool parsing is disabled, matching Python
  • llm-d/llm-d-router #4244: guidance on choosing onMissing for load metrics in the endpoint attribute filter
  • NVIDIA/NVSentinel #2415: demo setup script shows the Helm error when an install fails before any pod is created
  • apache/spark #59352, #59377: PySpark test coverage and typing cleanup
  • NVIDIA/cuda-rust #38, #73, #75: installation docs

Links

Blog · LinkedIn

Pinned Loading

  1. vllm-project/vllm vllm-project/vllm Public

    A high-throughput and memory-efficient inference and serving engine for LLMs

    Python 97.3k 25.1k

  2. llm-d/llm-d-router llm-d/llm-d-router Public

    llm-d Router: The intelligent entry point for inference requests

    Go 473 557

  3. NVIDIA/cuda-rust NVIDIA/cuda-rust Public

    NVIDIA's CUDA platform for Rust. Host runtime crates plus Tile (cutile-rs) and SIMT (cuda-oxide) kernel programming models in idiomatic Rust.

    Rust 4.1k 378

  4. NVIDIA/NVSentinel NVIDIA/NVSentinel Public

    NVSentinel detects and remediates GPU faults on Kubernetes nodes

    Go 444 179