Skip to content

Add asyncio support to the Python SDK - #57

Open
kvz wants to merge 63 commits into
py310-v2from
asyncio-v2
Open

kvz wants to merge 63 commits into
py310-v2from
asyncio-v2

Conversation

@kvz

@kvz kvz commented May 20, 2026 •

Copy link
Copy Markdown
Member

Why

Python SDK 2.0 includes asyncio and the full documented API surface. This PR adds AsyncTransloadit and all 44 documented operations to both clients while preserving the high-level Assembly, Template and resumable-upload helpers. It stays based on #56 (py310-v2) so the runtime and async changes ship together as 2.0.

Closes #9.

What changed

  • Merged current Prepare Python SDK 2.0 with Python 3.12+ #56 (including current SDK main); reconciled the 2.0 Unreleased changelog and migration docs.
  • Pinned API2's canonical contract at 27622f3ecd09ffef6fc86a43dcfe14b95deaaad6, preserving its bytes and SHA-256. A standard-library generator emits both endpoint blocks, transport metadata and the coverage table. CI checks generated bytes offline on Ubuntu and Windows.
  • Completed all 38 documented API2 endpoints: auth keys, token grants, DAM (retaining alpha stages), statistics, canonical multipart Assembly replacement and managed SSE, alongside existing endpoint families.
  • Added all six raw TUS operations, preserving binary download bytes and response headers. Returned URL validation permits trusted regional handoffs, rejects foreign/ambiguous destinations and disables account credentials and redirects on capability requests.
  • Added signed PATCH, multipart PUT, optional bearer-token requests and contract-selected token-grant authentication. The existing client-credentials default remains compatible; token refresh is explicit.
  • SSE parsing is incremental and bounded. Sync/async contexts clean up on early exit, errors and cancellation. Terminal messages do not imply processing success; the docs explain final-status checks.
  • Reused the raw TUS transport in the handwritten upload/resume helpers; normal high-level resumable Assemblies still use tuspy (offloaded from the async event loop).
  • Kept inherited Console-only helpers as compatibility methods outside public generation. Deferred the 16 explicitly separate-product storage/device/OAuth routes to the agreed 2.1 follow-up, Python 2.1: cover storage, device authorization and OAuth APIs #60.
  • Updated README, Sphinx docs, the full coverage table and CHANGELOG 2.0.0, including a runnable asyncio example.

Validation and council review

  • Regression tests were observed failing before each implementation/fix. The full local Docker matrix passes 265 tests per Python version (3.12, 3.13, 3.14), plus 14 opt-in live skips. Coverage is 93.02%; Node CLI signature parity passes on 3.14.
  • Behavioral coverage includes signed request bytes, all token grants, bearer/netrc precedence, multipart PUT and text-mode uploads, repeated cancellation/file ownership, proxy and connector configuration, Assembly polling, SSE framing/cleanup, the six raw TUS operations and binary ranges, and destination/header rejection.
  • Live test-account validation covers both clients: auth-key scopes, all six raw TUS operations, SSE subscription, cancellation and final-status retrieval. The existing CI upload and runnable-example suite also passes (18 live/example tests). Final CI on 8faf570 is green: all six Ubuntu/Windows matrix jobs and the live E2E job, run 38090452996.
  • Strict Sphinx build, Poetry metadata checks, offline generation check, and wheel/sdist build pass. Source-archive generation reproduces the committed bytes. A release-helper dry run builds artifacts without publishing.
  • Council round 1 fixed multipart and repeated TUS cancellation cleanup, preserved proxy/TLS configuration, bounded rate-limit retries, and added metadata validation and clear Assembly errors. It also led to direct Git storage of the contract pin and enforcement of unsupported TUS headers.
  • Council round 2 fixed ambient credential leakage on Assembly capability URLs, explicit bearer precedence, async text-mode uploads, custom-service worker polling and ASSEMBLY_REPLAYING handling. Regression tests cover all five findings.
  • Council round 3 fixed actual worker completion through global event-loop shutdown, CRLF/UTF-16 multipart byte lengths, explicit proxy preservation, legacy HTTP Assembly URL upgrades and consistent regional host validation. The text-length regression was also reproduced in Windows CI.
  • Council round 4 fixed environment proxy behavior (including proxy-only netrc authentication), async filename preservation and safe multipart parameter quoting.
  • Council round 5 found no remaining behavior fixes. One reviewer returned No issues found; arbitration retained only a quoting-comment clarification. The comment now records aiohttp’s extra backslash escaping, which API2 decodes equivalently. No additional review round is needed for that comment-only change. Final CI is green; the PR is ready with tim-kos requested.

Intentional boundaries: the legacy tuspy/custom-service workflows retain worker destinations returned by the configured service. Default Transloadit Assembly requests and generated raw capabilities remain destination-restricted; capability requests isolate credentials and disable redirects. SSE EOF/native transport errors remain observable, with final Assembly Status determining success. Async support is a standard dependency, already present in 1.x.

Friction: API2’s old generator/devdock wrapper commands were removed, so generation now uses its canonical checked-in contract and validation uses the repo’s existing live-test path. Live testing exposed cross-regional TUS/Assembly URLs. Windows CI caught an SSE test timing assumption and text newline/byte-length differences; both received portable regression fixes. Five council rounds covered cancellation, proxy/credential handling and upload compatibility.

Release handoff

Ready for review with tim-kos. No merge into main, tag or publication. Kevin's SM precedes release. #57 must stay stacked on #56 until the combined 2.0 changes are ready to land. The 44-operation boundary is approved; the 16 additional product routes are recorded in #60.

🤖 Claimed by kvz on framework1/python-sdk since 2026-10-10 · resume: codex resume 01a12780-713d-7b83-af02-b7d64ec08d37

@kvz kvz self-assigned this May 20, 2026
@codecov-commenter

codecov-commenter commented May 20, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.67787% with 76 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.96%. Comparing base (2397ea5) to head (8faf570).

Files with missing lines Patch % Lines
transloadit/client.py 91.74% 18 Missing ⚠️
transloadit/async_client.py 93.27% 16 Missing ⚠️
transloadit/assembly.py 88.54% 11 Missing ⚠️
transloadit/async_request.py 95.66% 11 Missing ⚠️
transloadit/endpoint_transport.py 93.42% 10 Missing ⚠️
transloadit/upload.py 93.18% 3 Missing ⚠️
transloadit/api_url.py 94.28% 2 Missing ⚠️
transloadit/request.py 97.59% 2 Missing ⚠️
transloadit/async_template.py 94.11% 1 Missing ⚠️
transloadit/stream.py 98.64% 1 Missing ⚠️
... and 1 more
Additional details and impacted files
@@             Coverage Diff              @@
##           py310-v2      #57      +/-   ##
============================================
+ Coverage     86.60%   94.96%   +8.36%     
============================================
  Files             7       17      +10     
  Lines           209     1550    +1341     
  Branches         19      214     +195     
============================================
+ Hits            181     1472    +1291     
- Misses           28       78      +50     
Flag Coverage Δ
unittests 94.96% <94.67%> (+8.36%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@kvz

kvz commented May 21, 2026

Copy link
Copy Markdown
Member Author

Setting this back to draft while we expand the feature scope: the v2 PR should also cover the full Transloadit API endpoint surface before we treat it as ready. I added a matching @todo to the 2.0.0 CHANGELOG section so the release notes get updated once the endpoint coverage is complete.

@kvz
kvz marked this pull request as draft May 21, 2026 14:07
kvz and others added 18 commits June 3, 2026 01:01
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The API2 Python generator now converts contract identifiers to
snake_case for local variable names, so generated methods read like
idiomatic Python. No public API or behavior change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The generator no longer emits function-local imports; base64, urljoin,
and requests (sync only) now live in the handwritten module headers
where Python convention expects them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
resume_tus_upload() is generated from the API2 resumeUpload TUS protocol
contract in both the sync and asyncio clients: it discovers the server
offset with a HEAD request, PATCHes the remaining bytes from that offset,
asserts the final offset matches the content length, and waits for the
Assembly to finish. The new api2-devdock-tus-resume-upload example
interrupts an upload after the first chunk like a dropped connection
would, then resumes it through the public SDK method.

The lifecycle example now polls the Assembly list briefly because the API
acknowledges creation before the list storage row lands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts:
#	.github/workflows/ci.yml
@kvz
kvz marked this pull request as ready for review October 10, 2026 22:14
@kvz
kvz requested a review from tim-kos October 10, 2026 22:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants