Skip to content

feat: support generic http(s) urls#8

Open
coroa wants to merge 5 commits intomainfrom
feat/support-generic-http-urls
Open

feat: support generic http(s) urls#8
coroa wants to merge 5 commits intomainfrom
feat/support-generic-http-urls

Conversation

@coroa
Copy link
Member

@coroa coroa commented Feb 26, 2026

Fixes #5 .

With this change we re-introduce general http(s) urls based on a http head request, so that cached-http becomes a complete replacement for the original http plugin.

Of note: if head requests are unsupported, the plugin assumes the requested url exists with an mtime of 0.

@coroa coroa requested a review from lkstrp February 26, 2026 11:05
@coroa
Copy link
Member Author

coroa commented Feb 26, 2026

We need to be a bit careful about releasing a version here, since i removed the monkey patching of snakemake-storage-plugin-http, so this would give us, if both are installled:

WorkflowError in file "/home/coroa/work/pypsa-eur/rules/retrieve.smk", line 28:
Multiple suitable storage providers found for query https://data.pypsa.org/workflows/eur/eurostat_balances/2026-02/nrg_bal_c: http, cached-http. Explicitly specify the storage provider.

I guess i could leave the monkey patching in place for a grace period and warn about the future incompatibility?

@coroa coroa mentioned this pull request Feb 26, 2026
@lkstrp
Copy link
Member

lkstrp commented Feb 26, 2026

I mean this is currently used for PyPSA-Eur and forks only. We can just pin the current version in Eur, release here a new version, and remove the pin and remove the default storage provider

@coroa coroa force-pushed the feat/support-generic-http-urls branch from ef5edfd to 4eade7f Compare February 26, 2026 12:57
@coroa
Copy link
Member Author

coroa commented Feb 26, 2026

Yes, but since the update cycles of the soft-forks are somewhat irregular, let's just take the safe route and keep the monkey-patching around a bit longer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add general url support

2 participants