test_detect_backend_* fail on a clean checkout when GOOGLE_API_KEY (et al.) is set — backend-detection tests don't isolate ambient env

[AirRocker]

Version: v8 @ adf814f (0.9.57), clean checkout, Python 3.12 · Type: test isolation / contributor experience

Summary

The backend-detection tests call detect_backend() (graphify/llm.py:3147) against whatever the developer's shell happens to export, and each one clears only a hand-picked subset of the variables it reads. detect_backend() consults 13 environment variables, and ten of them can break these tests (the two Azure ones only in combination). With GOOGLE_API_KEY set, 3 of the 4 test_ollama.py::test_detect_backend_* tests fail on an untouched checkout. Other variables break one or two of them, and the Azure pair and OLLAMA_HOST also break a fourth test in test_provider_registry.py. CI never sees any of this, because no workflow sets any of these variables.

Repro

From a clean checkout of v8:

uv sync --all-extras --frozen
env -i HOME="$HOME" PATH="$PATH" GOOGLE_API_KEY=dummy uv run --frozen pytest tests/test_ollama.py -k detect_backend

tests/test_ollama.py FF.F                                                [100%]

=================================== FAILURES ===================================
__________________________ test_detect_backend_ollama __________________________

monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x10b7a7530>

    def test_detect_backend_ollama(monkeypatch):
        monkeypatch.delenv("MOONSHOT_API_KEY", raising=False)
        monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
        monkeypatch.setenv("OLLAMA_BASE_URL", "http://localhost:11434/v1")
>       assert detect_backend() == "ollama"
E       AssertionError: assert 'gemini' == 'ollama'
E         
E         - ollama
E         + gemini

tests/test_ollama.py:62: AssertionError
____________________ test_detect_backend_kimi_beats_ollama _____________________

monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x10b7a70b0>

    def test_detect_backend_kimi_beats_ollama(monkeypatch):
        monkeypatch.setenv("MOONSHOT_API_KEY", "test-key")
        monkeypatch.setenv("OLLAMA_BASE_URL", "http://localhost:11434/v1")
        monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
>       assert detect_backend() == "kimi"
E       AssertionError: assert 'gemini' == 'kimi'
E         
E         - kimi
E         + gemini

tests/test_ollama.py:69: AssertionError
___________________ test_detect_backend_none_without_envvars ___________________

monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x10c00e600>

    def test_detect_backend_none_without_envvars(monkeypatch):
        monkeypatch.delenv("MOONSHOT_API_KEY", raising=False)
        monkeypatch.delenv("OLLAMA_BASE_URL", raising=False)
        monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
>       assert detect_backend() is None
E       AssertionError: assert 'gemini' is None
E        +  where 'gemini' = detect_backend()

tests/test_ollama.py:87: AssertionError
=========================== short test summary info ============================
FAILED tests/test_ollama.py::test_detect_backend_ollama - AssertionError: ass...
FAILED tests/test_ollama.py::test_detect_backend_kimi_beats_ollama - Assertio...
FAILED tests/test_ollama.py::test_detect_backend_none_without_envvars - Asser...
================== 3 failed, 1 passed, 9 deselected in 0.26s ===================

Control, the same command with the variable left out:

env -i HOME="$HOME" PATH="$PATH" uv run --frozen pytest tests/test_ollama.py -k detect_backend

tests/test_ollama.py ....                                                [100%]

======================= 4 passed, 9 deselected in 0.21s ========================

The env -i matters. It keeps anything else in the shell from leaking in, and as the table below shows, the list of things that can leak in is long.

The full affected set

I added each of the 13 variables detect_backend() reads on its own, using a placeholder value, to an otherwise empty environment. Each time I ran the four test files that call detect_backend() (test_ollama.py, test_extract_cli.py, test_provider_registry.py and test_llm_backends.py, 158 tests). With nothing set, all 158 pass.

Variable present (each on its own)	test_detect_backend_ollama	test_detect_backend_kimi_beats_ollama	test_detect_backend_none_without_envvars	test_provider_registry.py:: test_detect_backend_custom_provider_after_builtins
GEMINI_API_KEY / GOOGLE_API_KEY	fail	fail	fail	pass
OPENAI_API_KEY	fail	pass	fail	pass
DEEPSEEK_API_KEY	fail	pass	fail	pass
AWS_PROFILE / AWS_REGION / AWS_DEFAULT_REGION	fail	pass	fail	pass
AZURE_OPENAI_API_KEY + AZURE_OPENAI_ENDPOINT	fail	pass	fail	fail
OLLAMA_HOST	pass	pass	fail	fail

These don't break anything: MOONSHOT_API_KEY, ANTHROPIC_API_KEY and OLLAMA_BASE_URL, because the tests set or clear them themselves; AZURE_OPENAI_API_KEY without the endpoint, because detection needs both; and OLLAMA_API_KEY, because detection doesn't read it. test_detect_backend_claude_beats_ollama passes in every row, because it clears both backends that rank ahead of claude.

The root cause is the same in every failing case. Each test clears a different hand-kept subset, and none of them clears all 13. The most thorough one, test_detect_backend_custom_provider_after_builtins, clears ten and still misses the Azure pair and OLLAMA_HOST (honoured since #1940).

Why it matters

This hits anyone who keeps a provider key in their shell. For a tool that exists to call LLM backends, that's most of the people likely to contribute. The AWS and OLLAMA_HOST rows widen it further: those aren't LLM keys at all, just ordinary settings for anyone who uses AWS or runs Ollama.

What those contributors get is a red suite on code they haven't touched. So they either stop trusting the suite or blame their own branch. That's exactly what happened to me while rebasing #2162 onto adf814f. Three failures showed up in tests my branch can't reach (its diff is markdown only), and it took toggling the variable to prove they came from my shell's GOOGLE_API_KEY and not from the rebase.

I'm far from the first to hit it. At least 19 PRs from 8 contributors since June have noted or excluded test_ollama failures as "pre-existing" or "environment-dependent" in their descriptions (#1447, #2686, #2776, #3298 and #3448, among others). Where those reports name the backend-detection tests, they count two failures or three, which fits the table: it depends on which variable the contributor happens to export. As far as I can find, it has never been tracked as an issue.

Fix shape

Before each of these tests sets the variables it's actually testing, monkeypatch.delenv every detect_backend() input, either in the tests' own setup or in an autouse fixture next to _sandbox_home in tests/conftest.py. That's the same move the suite already makes for HOME since #2168. Deriving the key names from BACKENDS would also keep the list from drifting again as backends are added.

Notes for triage

• detect_backend() silently routes extraction to Bedrock on the mere presence of AWS_PROFILE/AWS_REGION, with no env var or config key to override it #3300 (open) is the runtime sibling. It asks whether the mere presence of AWS_PROFILE/AWS_REGION should route extraction to Bedrock at all. This report doesn't depend on how that's decided: if detection changes, only the AWS rows of the table move.
• CI can't catch this, because none of the workflows set any of these variables. That's why it has only ever shown up locally.

Happy to submit a PR if that's useful. Same offer as before: I didn't want to presume where you'd like the fixture to live.