fix(http2): wire custom HTTP/2 fingerprints to the transport - #85
Closed
smeinecke wants to merge 150 commits into
Closed
fix(http2): wire custom HTTP/2 fingerprints to the transport#85smeinecke wants to merge 150 commits into
smeinecke wants to merge 150 commits into
Conversation
…flow permissions - Use debugLogger instead of log for request headers/body to avoid clear-text logging of sensitive data - Add bounds check after strconv.Atoi in parseError to prevent unsafe int→uint8 narrowing - Add permissions: contents: read to all workflow jobs missing explicit GITHUB_TOKEN permissions
* feat: add Python 3.9 support and follow-up fixes
…les, and CI capture pipeline - Add BrowserFamily and Platform enums for typed registry lookups - Add FingerprintRegistry.by_family(family, platform=None), latest(), random() with optional family/platform filters; refactor shared _candidates() helper - Rename all built-in profiles in fingerprints.json with platform suffix derived from user-agent (_win/_linux/_mac/_android); update _PROFILE_EXPORTS accordingly - Add _platform_suffix() to capture script; all new Playwright captures get OS-tagged names (e.g. chrome_146_0_0_0_linux, chrome_146_0_0_0_win) - Add Android ADB capture mode (--android-only) and chrome_*_android profiles - Add GitHub Actions fingerprint-tests workflow: Playwright capture across Linux/macOS/Windows, Android emulator capture, fingerprint replication tests - Add make fingerprint-tests target for local end-to-end runs - Add full unit test coverage for new enum/registry methods (TestBrowserFamilyAndPlatform)
pw.android is a JS-only Playwright API not exposed in the Python SDK. Replace with: - subprocess.run(["adb", "devices"]) to discover connected devices - adb shell am start to launch Chrome on the device - adb forward tcp:9222 localabstract:chrome_devtools_remote for CDP - pw.chromium.connect_over_cdp() + Security.setIgnoreCertificateErrors via CDP session to navigate and capture the fingerprint Also switch the CI emulator target from "default" to "google_apis" since the default AOSP image does not include Chrome.
…container Avoids re-downloading Chrome (~200 MB) on every run and the headed-Chrome hang inside xvfb-run. The host runner already has chromium/chrome/firefox/ webkit installed from the previous Install step. TrackMe is port-mapped to localhost:8443 so --url https://localhost:8443 reaches it directly.
…xes Pyright errors)
… now exists); fix Pyright errors
Bumps [github.com/andybalholm/brotli](https://github.com/andybalholm/brotli) from 1.2.0 to 1.2.1. - [Commits](andybalholm/brotli@v1.2.0...v1.2.1) --- updated-dependencies: - dependency-name: github.com/andybalholm/brotli dependency-version: 1.2.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
--- updated-dependencies: - dependency-name: aiohttp dependency-version: 3.13.4 dependency-type: direct:production dependency-group: uv ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
- actions/checkout@v4 → v6 - actions/setup-python@v5 → v6 - actions/setup-go@v5 → v6 - actions/upload-artifact@v4 → v7 - actions/download-artifact@v4 → v8
New fingerprints: chromium_145_0_0_0_mac,chromium_145_0_0_0_win
Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 7 to 8. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](peter-evans/create-pull-request@v7...v8) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-version: '8' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [github.com/valyala/fasthttp](https://github.com/valyala/fasthttp) from 1.69.0 to 1.70.0. - [Release notes](https://github.com/valyala/fasthttp/releases) - [Commits](valyala/fasthttp@v1.69.0...v1.70.0) --- updated-dependencies: - dependency-name: github.com/valyala/fasthttp dependency-version: 1.70.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.52.0 to 0.53.0. - [Commits](golang/net@v0.52.0...v0.53.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-version: 0.53.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.37.6 to 4.37.8. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v4.37.6...v4.37.8) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.37.8 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
New fingerprints: chrome_152_0_0_0_linux,chrome_152_0_0_0_mac,chrome_152_0_0_0_win
New fingerprints: msedge_152_0_0_0_linux,msedge_152_0_0_0_mac,msedge_152_0_0_0_win
…lang.org/x/net-0.58.0 build(deps): bump golang.org/x/net from 0.57.0 to 0.58.0 in /golang
…l-sh/setup-uv-10.0.1 build(deps): bump astral-sh/setup-uv from 9.0.0 to 10.0.1
…b/codeql-action-4.37.8 build(deps): bump github/codeql-action from 4.37.6 to 4.37.8
Request.to_dict() was clearing `ja3` whenever `ja4r`, `http2_fingerprint`, or `quic_fingerprint` was set. This caused `fingerprint="..."` to fall back to the Go default Chrome JA3 for most built-in profiles. Only `ja4r` should take precedence over `ja3`; http2/quic are independent layers. - Change `Request.ja3` default from a hard-coded Firefox 65 string to `Optional[str] = None` so a missing/unspecified ja3 lets the Go backend use its own default. - Only include `ja3` in the payload when it is explicitly set and `ja4r` is not present.
- Sync `cycletls/__version__.py` and `pyproject.toml` to 0.0.9. - Update CHANGELOG with the schema fix and merged dependency bumps. - Refresh `uv.lock` for the new package version.
- Correct the 0.0.8 entry to only the Brotli fix and its regression test. - Expand 0.0.9 with the actual schema fix, dependency bumps, and fingerprint registry updates from `v0.0.8..v0.0.9`. - Add missing 0.0.7 and 0.0.6 sections; move the `local_address`, `Do()`, `TLS13AutoRetry`, and `dispatchSSEAsync` items from 0.0.8 to 0.0.6 where they were actually introduced.
This reverts commit 944d9f9.
`scripts/capture_browser_fingerprints.py` only recorded `header_order` but dropped the actual header values. Newly captured profiles therefore lost the `headers` dictionary (e.g. default `accept`, `accept-language`, `accept-encoding`, `sec-fetch-*`) that older registry entries had. - Parse both header name order and header values from HTTP/2 HEADERS frames in `_parse_sent_headers`. - Return both from `capture_fingerprint` and `_capture_android_cdp` so the output JSON contains a `headers` map. - Pass `headers` through `_normalize_trackme_payload` so `load_trackme_fingerprints` persists them in the registry. This restores parity between newly captured profiles and manually maintained ones in `cycletls/data/fingerprints.json`.
… headers Add CDP support to `scripts/capture_browser_fingerprints.py` so it can connect to versioned browser containers (e.g. Selenium standalone Chrome) instead of only launching Playwright-managed browsers. Detect browser and platform from the user-agent, sanitize dynamic `sec-ch-ua`/`user-agent` values from headers, and force `en-US` locale for deterministic `Accept-Language`. Add `scripts/backfill_headers.py` to merge captured `headers` and `header_order` into `cycletls/data/fingerprints.json`, matching by browser/version prefix so one Linux capture can backfill the Windows and macOS variants. Backfill `headers` for the `chrome_152_0_0_0_*` profiles from a live Chrome 152 CDP capture. Remaining missing headers can be filled the same way from appropriate browser Docker images.
Run `scripts/backfill_headers.py` after merging new captured fingerprints so the registry auto-backfills `headers` for matching browser/version profiles (e.g. a Linux capture also fills the Windows and macOS variants).
Add Selenium/WebDriver capture mode so the script can drive selenium/standalone-firefox, standalone-chrome, and standalone-edge containers. Disable built-in JSON viewers (Firefox devtools preference, Chromium/Edge --disable-blink-features=PrettyPrintJSONDocument) so the /api/all JSON body can be read directly.
Add `.github/workflows/backfill-headers.yml` which discovers profiles missing `headers`, runs versioned `selenium/standalone-*` Docker containers for each Chrome/Chromium/Edge/Firefox base, captures the headers with `scripts/capture_browser_fingerprints.py --selenium-url`, and backfills `cycletls/data/fingerprints.json` via `scripts/backfill_headers.py` in a single PR. The capture script now also supports `--browser-override` and `--version-override` so a `standalone-chrome` image can be used to backfill both `chrome_*` and `chromium_*` registry entries.
Add `selenium` to the dev dependency group so `uv run` can drive Selenium containers in CI. Update `backfill-headers.yml` to `uv sync` instead of the broken `uv pip install` and add Docker image version fallbacks for versions without a standalone image (e.g. Edge 152).
Co-authored-by: smeinecke <7316652+smeinecke@users.noreply.github.com>
…ight and emulator
Co-authored-by: smeinecke <7316652+smeinecke@users.noreply.github.com>
Co-authored-by: smeinecke <7316652+smeinecke@users.noreply.github.com>
The http2_fingerprint option was parsed but never actually emitted on the wire. The old HTTP2Fingerprint.Apply only wrote Transport.Settings, which fhttp's Transport.ApplyPreset immediately overwrote with hardcoded Firefox/ Chrome defaults derived from the User-Agent. The fhttp HTTP2Settings struct (ConnectionFlow, PriorityFrames, HeaderPriority, and Settings) was left empty, so connection flow, initial priority frames, and the HEADERS stream ID stayed driven by the UA regardless of the supplied fingerprint. Rewrite HTTP2Fingerprint parsing to match the canonical SETTINGS|WINDOW_UPDATE|PRIORITY_FRAMES|PSEUDO_HEADERS format and populate http2.HTTP2Settings. Apply that to the transport and use the fingerprint to pick the remaining fhttp defaults (e.g. the implicit HEADERS stream ID when no priority frames are present). Also map the pseudo-header order letters onto http.PHeaderOrderKey so the request header ordering matches the fingerprint too. This makes http2_fingerprint a first-class option and lets profiles that ship both ja4r and http2_fingerprint produce a coherent TLS + HTTP/2 identity.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+21
to
+203
| name: Discover missing headers | ||
| runs-on: ubuntu-latest | ||
| outputs: | ||
| matrix: ${{ steps.matrix.outputs.matrix }} | ||
| has_work: ${{ steps.matrix.outputs.has_work }} | ||
| docker_matrix: ${{ steps.matrix.outputs.docker_matrix }} | ||
| docker_has_work: ${{ steps.matrix.outputs.docker_has_work }} | ||
| playwright_matrix: ${{ steps.matrix.outputs.playwright_matrix }} | ||
| playwright_has_work: ${{ steps.matrix.outputs.playwright_has_work }} | ||
| android_matrix: ${{ steps.matrix.outputs.android_matrix }} | ||
| android_has_work: ${{ steps.matrix.outputs.android_has_work }} | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
|
|
||
| - name: Generate capture matrix | ||
| id: matrix | ||
| run: | | ||
| python - <<'EOF' | ||
| import json, re, os | ||
|
|
||
| # Known selenium/standalone-* image versions available on Docker Hub. | ||
| # Used to fall back to a nearby image when an exact version is not published. | ||
| KNOWN_IMAGE_VERSIONS = { | ||
| "chrome": ["151.0", "150.0", "149.0", "148.0", "147.0"], | ||
| "edge": ["151.0", "150.0", "149.0", "148.0", "147.0"], | ||
| "firefox": ["153.0", "152.0", "151.0", "150.0", "149.0", "148.0"], | ||
| } | ||
|
|
||
| # Map a Safari/webkit version to the Playwright release that ships it. | ||
| # Playwright controls the exact browser build it uses, so we must pin it. | ||
| PLAYWRIGHT_VERSIONS = { | ||
| "26.0": "1.58.0", | ||
| "26.4": "1.60.0", | ||
| "26.5": "1.61.0", | ||
| } | ||
|
|
||
| def image_for(image_browser: str, tag_version: str) -> str: | ||
| known = KNOWN_IMAGE_VERSIONS.get(image_browser, []) | ||
| # Prefer an exact major.minor match. | ||
| if tag_version in known: | ||
| return f"selenium/standalone-{image_browser}:{tag_version}-20260808" | ||
| # Fall back to the closest older version that is known to exist. | ||
| parts = tag_version.split(".") | ||
| major = int(parts[0]) | ||
| minor = int(parts[1]) if len(parts) > 1 else 0 | ||
| candidates = [v for v in known] | ||
| candidates.sort(key=lambda v: tuple(int(x) for x in v.split(".")), reverse=True) | ||
| for cand in candidates: | ||
| cmajor, cminor = (int(x) for x in cand.split(".")) | ||
| if (cmajor, cminor) <= (major, minor): | ||
| return f"selenium/standalone-{image_browser}:{cand}-20260808" | ||
| if known: | ||
| return f"selenium/standalone-{image_browser}:{known[0]}-20260808" | ||
| return f"selenium/standalone-{image_browser}:nightly" | ||
|
|
||
| def safe_name(base: str) -> str: | ||
| return re.sub(r"[^0-9A-Za-z]+", "-", base).strip("-") | ||
|
|
||
| registry_path = "cycletls/data/fingerprints.json" | ||
| registry = json.load(open(registry_path)) | ||
| missing = [fp for fp in registry["fingerprints"] if "headers" not in fp] | ||
|
|
||
| seen = set() | ||
| docker_include = [] | ||
| playwright_include = [] | ||
| android_include = [] | ||
|
|
||
| for fp in missing: | ||
| name = fp["name"] | ||
| parts = name.rsplit("_", 1) | ||
| base = parts[0] if len(parts) == 2 else name | ||
| if base in seen: | ||
| continue | ||
| seen.add(base) | ||
|
|
||
| ua = fp.get("user_agent", "") | ||
| m = re.search(r"(?:Chrome|Chromium|HeadlessChrome|Firefox|Edg|Version|wv)/([0-9.]+)", ua) | ||
| version = m.group(1) if m else "latest" | ||
|
|
||
| if base.startswith("safari_"): | ||
| tag_version = ".".join(version.split(".")[:2]) | ||
| playwright_version = PLAYWRIGHT_VERSIONS.get(tag_version, "1.61.0") | ||
| playwright_include.append({ | ||
| "base": base, | ||
| "version": version, | ||
| "playwright_version": playwright_version, | ||
| "artifact": f"capture-{safe_name(base)}", | ||
| }) | ||
| elif base.startswith("chrome_android_"): | ||
| android_include.append({ | ||
| "base": base, | ||
| "version": version, | ||
| "artifact": f"capture-{safe_name(base)}", | ||
| }) | ||
| elif base.startswith("chrome_"): | ||
| image_browser = "chrome" | ||
| selenium_browser = "chrome" | ||
| registry_browser = "chrome" | ||
| tag_version = ".".join(version.split(".")[:2]) | ||
| image = image_for(image_browser, tag_version) | ||
| docker_include.append({ | ||
| "base": base, | ||
| "image": image, | ||
| "selenium_browser": selenium_browser, | ||
| "registry_browser": registry_browser, | ||
| "version": version, | ||
| "artifact": f"capture-{safe_name(base)}", | ||
| }) | ||
| elif base.startswith("chromium_"): | ||
| image_browser = "chrome" | ||
| selenium_browser = "chrome" | ||
| registry_browser = "chromium" | ||
| tag_version = ".".join(version.split(".")[:2]) | ||
| image = image_for(image_browser, tag_version) | ||
| docker_include.append({ | ||
| "base": base, | ||
| "image": image, | ||
| "selenium_browser": selenium_browser, | ||
| "registry_browser": registry_browser, | ||
| "version": version, | ||
| "artifact": f"capture-{safe_name(base)}", | ||
| }) | ||
| elif base.startswith("msedge_"): | ||
| image_browser = "edge" | ||
| selenium_browser = "MicrosoftEdge" | ||
| registry_browser = "msedge" | ||
| tag_version = ".".join(version.split(".")[:2]) | ||
| image = image_for(image_browser, tag_version) | ||
| docker_include.append({ | ||
| "base": base, | ||
| "image": image, | ||
| "selenium_browser": selenium_browser, | ||
| "registry_browser": registry_browser, | ||
| "version": version, | ||
| "artifact": f"capture-{safe_name(base)}", | ||
| }) | ||
| elif base.startswith("firefox_"): | ||
| image_browser = "firefox" | ||
| selenium_browser = "firefox" | ||
| registry_browser = "firefox" | ||
| tag_version = ".".join(version.split(".")[:2]) | ||
| image = image_for(image_browser, tag_version) | ||
| docker_include.append({ | ||
| "base": base, | ||
| "image": image, | ||
| "selenium_browser": selenium_browser, | ||
| "registry_browser": registry_browser, | ||
| "version": version, | ||
| "artifact": f"capture-{safe_name(base)}", | ||
| }) | ||
| else: | ||
| print(f"[skip] unsupported missing profile: {base}") | ||
| continue | ||
|
|
||
| docker_matrix = {"include": docker_include} | ||
| playwright_matrix = {"include": playwright_include} | ||
| android_matrix = {"include": android_include} | ||
|
|
||
| docker_has_work = "true" if docker_include else "false" | ||
| playwright_has_work = "true" if playwright_include else "false" | ||
| android_has_work = "true" if android_include else "false" | ||
| has_work = "true" if (docker_include or playwright_include or android_include) else "false" | ||
|
|
||
| with open(os.environ["GITHUB_OUTPUT"], "a", encoding="utf-8") as fh: | ||
| fh.write(f"matrix={json.dumps(docker_matrix)}\n") | ||
| fh.write(f"docker_matrix={json.dumps(docker_matrix)}\n") | ||
| fh.write(f"docker_has_work={docker_has_work}\n") | ||
| fh.write(f"playwright_matrix={json.dumps(playwright_matrix)}\n") | ||
| fh.write(f"playwright_has_work={playwright_has_work}\n") | ||
| fh.write(f"android_matrix={json.dumps(android_matrix)}\n") | ||
| fh.write(f"android_has_work={android_has_work}\n") | ||
| fh.write(f"has_work={has_work}\n") | ||
|
|
||
| print(f"Generated capture jobs: {len(docker_include)} docker, {len(playwright_include)} playwright, {len(android_include)} android") | ||
| for item in docker_include: | ||
| print(f" docker: {item['base']}: {item['image']} -> {item['registry_browser']} {item['version']}") | ||
| for item in playwright_include: | ||
| print(f" playwright: {item['base']}: playwright=={item['playwright_version']} -> {item['version']}") | ||
| for item in android_include: | ||
| print(f" android: {item['base']}: {item['version']}") | ||
| EOF | ||
|
|
||
| capture-docker: |
Comment on lines
+204
to
+263
| name: Capture ${{ matrix.base }} headers | ||
| needs: discover | ||
| runs-on: ubuntu-latest | ||
| if: needs.discover.outputs.docker_has_work == 'true' | ||
| continue-on-error: true | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: ${{ fromJSON(needs.discover.outputs.docker_matrix) }} | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
|
|
||
| - name: Setup Python | ||
| uses: actions/setup-python@v7 | ||
| with: | ||
| python-version: "3.12" | ||
|
|
||
| - name: Setup uv | ||
| uses: astral-sh/setup-uv@v10.0.1 | ||
| with: | ||
| enable-cache: true | ||
|
|
||
| - name: Start Selenium container | ||
| run: | | ||
| docker run -d --name selenium-${{ matrix.base }} -p 4444:4444 --shm-size=2g ${{ matrix.image }} | ||
| for i in $(seq 1 60); do | ||
| if curl -sf http://localhost:4444/wd/hub/status | grep -q '"ready": true'; then | ||
| echo "Selenium ready after $((i*2))s" | ||
| break | ||
| fi | ||
| sleep 2 | ||
| done | ||
|
|
||
| - name: Install Python dependencies | ||
| run: uv sync --locked --dev --no-install-project | ||
|
|
||
| - name: Capture fingerprint | ||
| env: | ||
| SERVER_URL: ${{ github.event.inputs.server_url }} | ||
| run: | | ||
| uv run --no-project python scripts/capture_browser_fingerprints.py \ | ||
| --selenium-url http://localhost:4444/wd/hub \ | ||
| --selenium-browser ${{ matrix.selenium_browser }} \ | ||
| --browser-override ${{ matrix.registry_browser }} \ | ||
| --version-override ${{ matrix.version }} \ | ||
| --url "$SERVER_URL" \ | ||
| --output /tmp/capture.json | ||
|
|
||
| - name: Upload capture artifact | ||
| if: always() | ||
| uses: actions/upload-artifact@v7 | ||
| with: | ||
| name: ${{ matrix.artifact }} | ||
| path: /tmp/capture.json | ||
| if-no-files-found: warn | ||
|
|
||
| - name: Stop Selenium container | ||
| if: always() | ||
| run: docker stop selenium-${{ matrix.base }} || true; docker rm -f selenium-${{ matrix.base }} || true | ||
|
|
||
| capture-playwright: |
Comment on lines
+264
to
+311
| name: Capture ${{ matrix.base }} headers (Playwright) | ||
| needs: discover | ||
| runs-on: ubuntu-latest | ||
| if: needs.discover.outputs.playwright_has_work == 'true' | ||
| continue-on-error: true | ||
| timeout-minutes: 25 | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: ${{ fromJSON(needs.discover.outputs.playwright_matrix) }} | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
|
|
||
| - name: Setup Python | ||
| uses: actions/setup-python@v7 | ||
| with: | ||
| python-version: "3.12" | ||
|
|
||
| - name: Setup uv | ||
| uses: astral-sh/setup-uv@v10.0.1 | ||
| with: | ||
| enable-cache: true | ||
|
|
||
| - name: Install Python dependencies | ||
| run: uv sync --locked --dev --no-install-project | ||
|
|
||
| - name: Install Playwright WebKit | ||
| run: | | ||
| uv pip install --force-reinstall playwright==${{ matrix.playwright_version }} | ||
| uv run --no-project python -m playwright install --with-deps webkit | ||
|
|
||
| - name: Capture fingerprint | ||
| env: | ||
| SERVER_URL: ${{ github.event.inputs.server_url }} | ||
| run: | | ||
| uv run --no-project python scripts/capture_browser_fingerprints.py \ | ||
| --require-browsers safari \ | ||
| --url "$SERVER_URL" \ | ||
| --output /tmp/capture.json | ||
|
|
||
| - name: Upload capture artifact | ||
| if: always() | ||
| uses: actions/upload-artifact@v7 | ||
| with: | ||
| name: ${{ matrix.artifact }} | ||
| path: /tmp/capture.json | ||
| if-no-files-found: warn | ||
|
|
||
| capture-android: |
Comment on lines
+312
to
+372
| name: Capture ${{ matrix.base }} headers (Android) | ||
| needs: discover | ||
| runs-on: ubuntu-latest | ||
| if: needs.discover.outputs.android_has_work == 'true' | ||
| continue-on-error: true | ||
| timeout-minutes: 45 | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: ${{ fromJSON(needs.discover.outputs.android_matrix) }} | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
|
|
||
| - name: Enable KVM | ||
| run: | | ||
| echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules | ||
| sudo udevadm control --reload-rules | ||
| sudo udevadm trigger --name-match=kvm | ||
|
|
||
| - name: Setup Python | ||
| uses: actions/setup-python@v7 | ||
| with: | ||
| python-version: "3.12" | ||
|
|
||
| - name: Setup uv | ||
| uses: astral-sh/setup-uv@v10.0.1 | ||
| with: | ||
| enable-cache: true | ||
|
|
||
| - name: Install Python dependencies | ||
| run: uv sync --locked --dev --no-install-project | ||
|
|
||
| - name: Install Playwright Chromium | ||
| run: uv run --no-project python -m playwright install --with-deps chromium | ||
|
|
||
| - name: Start Android emulator and capture Chrome fingerprint | ||
| uses: ReactiveCircus/android-emulator-runner@v2 | ||
| with: | ||
| api-level: 34 | ||
| arch: x86_64 | ||
| target: google_apis | ||
| profile: pixel_7 | ||
| emulator-boot-timeout: 300 | ||
| script: | | ||
| set -eux | ||
| echo "==> Emulator booted. Waiting 30 s for Play Services to settle..." | ||
| sleep 30 | ||
| echo "==> ADB devices:" | ||
| adb devices | ||
| echo "==> Chrome package version:" | ||
| adb shell dumpsys package com.android.chrome | grep versionName || true | ||
| . .venv/bin/activate && python3 scripts/capture_browser_fingerprints.py --android-only --url "${{ github.event.inputs.server_url }}" --output /tmp/capture.json --version-override "${{ matrix.version }}" | ||
|
|
||
| - name: Upload capture artifact | ||
| if: always() | ||
| uses: actions/upload-artifact@v7 | ||
| with: | ||
| name: ${{ matrix.artifact }} | ||
| path: /tmp/capture.json | ||
| if-no-files-found: warn | ||
|
|
||
| merge-and-backfill: |
Comment on lines
+373
to
+425
| name: Merge captures and backfill registry | ||
| needs: [discover, capture-docker, capture-playwright, capture-android] | ||
| runs-on: ubuntu-latest | ||
| if: always() && needs.discover.outputs.has_work == 'true' | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
|
|
||
| - name: Setup Python | ||
| uses: actions/setup-python@v7 | ||
| with: | ||
| python-version: "3.12" | ||
|
|
||
| - name: Setup uv | ||
| uses: astral-sh/setup-uv@v10.0.1 | ||
| with: | ||
| enable-cache: true | ||
|
|
||
| - name: Install Python dependencies | ||
| run: uv sync --locked --dev --no-install-project | ||
|
|
||
| - name: Download capture artifacts | ||
| uses: actions/download-artifact@v7 | ||
| with: | ||
| path: /tmp/captures | ||
| pattern: capture-* | ||
|
|
||
| - name: Backfill registry | ||
| run: | | ||
| for capture_file in $(find /tmp/captures -type f -name capture.json); do | ||
| echo "Backfilling from $capture_file" | ||
| uv run --no-project python scripts/backfill_headers.py \ | ||
| --registry cycletls/data/fingerprints.json \ | ||
| --capture "$capture_file" | ||
| done | ||
|
|
||
| - name: Show changes | ||
| run: | | ||
| git diff --stat cycletls/data/fingerprints.json || true | ||
|
|
||
| - name: Create backfill PR | ||
| if: github.event.inputs.dry_run != 'true' | ||
| uses: peter-evans/create-pull-request@v8 | ||
| with: | ||
| add-paths: | | ||
| cycletls/data/fingerprints.json | ||
| branch: ci/backfill-missing-headers | ||
| commit-message: "chore: backfill missing fingerprint headers" | ||
| title: "chore: backfill missing fingerprint headers" | ||
| body: | | ||
| This automated PR backfills `headers` (and `header_order` where missing) | ||
| for browser profiles that were missing them, using Selenium/Playwright | ||
| and an Android emulator where needed. | ||
| delete-branch: true |
| return True | ||
|
|
||
| page_text = xml_text.lower() | ||
| if "https://policies.google.com/" in page_text or "google terms of service" in page_text: |
| context.load_cert_chain(cert_path, key_path) | ||
|
|
||
| with socketserver.TCPServer(("", port), Handler) as httpd: | ||
| httpd.socket = context.wrap_socket(httpd.socket, server_side=True) |
| context.load_cert_chain(cert_path, key_path) | ||
|
|
||
| server = ThreadingHTTPServer(("127.0.0.1", 0), TrailingBrotliHandler) | ||
| server.socket = context.wrap_socket(server.socket, server_side=True) |
| if '/html' in url or 'example.com' in url: | ||
| assert isinstance(response.body, str), \ | ||
| f"HTML response should be string for {url}" | ||
| if "/html" in url or "example.com" in url: |
| } | ||
| log.Println(headers) | ||
| log.Println(body) | ||
| debugLogger.Println(headers) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
http2_fingerprintso the supplied HTTP/2 identity is actually emitted on the wire.golang/http2.goto parse the canonicalSETTINGS|WINDOW_UPDATE|PRIORITY_FRAMES|PSEUDO_HEADERSformat and populatefhttp'shttp2.HTTP2Settings.golang/roundtripper.goto apply those settings to the transport and to set the pseudo-header order from the fingerprint.Background
The Go layer parsed
http2_fingerprintbut only assigned it tohttp2.Transport.Settings. That field is overwritten byTransport.ApplyPreset, which derives the real HTTP/2 frame profile from theNavigator(parsed from the User-Agent). As a result, every request was emitted with the default Firefox- or Chrome-style frame sequence regardless of the suppliedhttp2_fingerprint.What changed
HTTP2Fingerprintnow carries the full fingerprint:Settingsfrom the first partConnectionFlowfrom the second partPriorityFramesfrom the third partHeaderPriority(reserved for future use)PriorityOrderpseudo-header map from the fourth partApplysetshttp2.Transport.HTTP2Settingsinstead of justSettings.roundTripperparses the fingerprint once and uses it when creating thehttp2.Transportand when preparing each request (http.PHeaderOrderKey).HTTP2Fingerprint.Navigator()infers the minimal fhttp browser default needed for the implicit HEADERS stream ID when no explicit priority frames are present.Test plan
go test ./...ingolang/passes.libcycletlsand verified with the matrix of Firefox/Chrome/Edge/customhttp2_fingerprintstrings that the wire Akamai fingerprint now matches the supplied value.firefox_124_win,chrome_149_*) now produce coherent HTTP/2 frames.m,p,a,svsm,a,s,p) follows the fingerprint, not just the User-Agent.Notes
This does not change how
ja3/ja4rTLS fingerprints are handled; it only makeshttp2_fingerprintan independent, working lever. Profiles that already ship bothja4randhttp2_fingerprintnow produce a matching TLS + HTTP/2 identity.