PEP 503 sdists for IDF full dependency tree

[jakub-kocka]

Description

• PEP 503 sdists: After assembling the full ESP-IDF dependency tree, emit_sdist_requirements.py computes which packages cannot be built as wheels on any supported platform/Python combination (using the same exclude_list.yaml merge logic as wheel builds). Those packages get a source distribution on https://dl.espressif.com/pypi next to wheels—not every exclude-list entry, only the union subset that has no buildable wheel path anywhere in the matrix.
• CI wiring: Linux x86_64 (stage-1) uploads idf-requirements-bundle (requirements.txt + sdist_requirements.txt). The upload job downloads sdists from PyPI into downloaded_wheels/; upload_wheels.py / create_index_pages.py publish them under pypi/<project>/. Post-upload verify_s3_sdists.py checks that each listed package has at least one sdist on S3.
• ARMv7 native wheels: repair_wheels.py skips auditwheel manylinux repair for guarded packages (native_import_guard.yaml, force_no_binary_linux.txt) and removes duplicate manylinux_*_armv7l wheels when a linux_armv7l sibling exists (avoids pip preferring broken manylinux cffi). Pre-upload checks run in test_wheels_install.py on native ARMv7 CI (no separate post-upload S3 import audit).
• Docs: README section on sdist policy; ARMv7/piwheels notes unchanged (piwheels used at build time in ARMv7 Docker, Espressif index for install).

What is not in scope

• Mirroring all PyPI sdists for the full tree—only computed fallback packages.
• Replacing wheels with sdists where wheels are still built on at least one platform.

Related

• Internal tracker: IDF-15426
• An ad-hoc script for verifying Espressif's PyPI about broken wheels was used to find and remove the broken packages manually
• Cryptography >=49.. on MacOS Intel is possible to build wheels for with this PR (ESPTOOL-1343)

Testing

• Workflow run: https://github.com/espressif/idf-python-wheels/actions/runs/27940574176/job/82710018979

---

Checklist

Before submitting a Pull Request, please ensure the following:

• 🚨 This PR does not introduce breaking changes.
• All CI checks (GH Actions) pass.
• Documentation is updated as needed.
• Tests are updated or added as necessary.
• Code is well-commented, especially in complex areas.
• Git history is clean — commits are squashed to the minimum necessary.

[github-actions]

	Messages
📖	This PR seems to be quite large (total lines of code: 2108), you might consider splitting it into smaller PRs

👋 Hello jakub-kocka, we appreciate your contribution to this project!

---

Click to see more instructions ...

This automated output is generated by the PR linter DangerJS, which checks if your Pull Request meets the project's requirements and helps you fix potential issues.

DangerJS is triggered with each push event to a Pull Request and modify the contents of this comment.

Please consider the following:
- Danger mainly focuses on the PR structure and formatting and can't understand the meaning behind your code or changes.
- Danger is not a substitute for human code reviews; it's still important to request a code review from your colleagues.
- Addressing info messages (📖) is strongly recommended; they're less critical but valuable.
- To manually retry these Danger checks, please navigate to the Actions tab and re-run last Danger workflow.

Review and merge process you can expect ...

We do welcome contributions in the form of bug reports, feature requests and pull requests.

1. An internal issue has been created for the PR, we assign it to the relevant engineer.
2. They review the PR and either approve it or ask you for changes or clarifications.
3. Once the GitHub PR is approved we do the final review, collect approvals from core owners and make sure all the automated tests are passing.
- At this point we may do some adjustments to the proposed change, or extend it by adding tests or documentation.
4. If the change is approved and passes the tests it is merged into the default branch.

Generated by 🚫 dangerJS against f6fbff4

[Copilot]

Pull request overview

Adds PEP 503 sdist fallback support to the IDF dependency publishing pipeline (compute which packages need sdists, download/upload them alongside wheels, and verify presence on S3), and improves ARMv7 handling by skipping problematic manylinux repairs and adding native import probes to catch broken native wheels.

Changes:

• Generate sdist_requirements.txt during wheel assembly and wire CI to download/upload sdists to the Espressif simple index.
• Add post-upload verification that required sdists exist on S3.
• Add ARMv7-specific safeguards: skip auditwheel for guarded packages, prune conflicting manylinux wheels, and run native import probes in CI.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
emit_sdist_requirements.py	New logic to compute and write sdist fallback requirements across the platform/Python matrix.
build_wheels.py	Emits sdist_requirements.txt as part of the main wheel build job.
download_sdists.py	New helper to pip download sdists listed in sdist_requirements.txt.
upload_wheels.py	Uploads sdists alongside wheels to S3 under pypi/<project>/.
create_index_pages.py	Stabilizes per-project simple-index page ordering by sorting filenames.
verify_s3_sdists.py	New post-upload S3 verification for required sdist presence.
repair_wheels.py	ARMv7: skip manylinux repair for guarded/native packages; prune conflicting manylinux wheels when linux-tag siblings exist.
native_import_probe.py	New subprocess-based import probe runner (segfault-safe).
native_import_guard.yaml	Defines per-distribution import probes for ARMv7 native wheel validation.
test_wheels_install.py	Runs guarded native import probes on ARMv7 after installation.
test_emit_sdist_requirements.py	Unit tests for sdist requirement computation and file emission.
test_native_import_guard.py	Unit test verifying guard YAML loads and contains expected cryptography probes.
yaml_list_adapter.py	Adds windows/macos → marker platform mapping for sys_platform evaluation.
_helper_functions.py	Adds native import guard loader + improves wheel name parsing/canonicalization helpers.
README.md	Documents sdist fallback policy and native import guard usage.
.github/workflows/build-wheels-platforms.yml	Uploads sdist_requirements.txt bundle; adds sdist verification step.
.github/workflows/upload-python-wheels.yml	Downloads requirements bundle and sdists before uploading to S3.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 7 comments.

[Copilot]

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated 6 comments.