Implement GLOFAS dataset by StuberSimon · Pull Request #498 · PyPSA/atlite

StuberSimon · 2026-04-06T10:18:18Z

Changes proposed in this Pull Request

Implement GLOFAS dataset for hydro functionality
Add option to use glofas data for plant inflow

Checklist

Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
Unit tests for new features were added (if applicable).
Newly introduced dependencies are added to environment.yaml, environment_docs.yaml and setup.py (if applicable).
A note for the release notes doc/release_notes.rst of the upcoming release is included.
I consent to the release of this PR's code under the MIT license.

StuberSimon · 2026-04-06T10:28:37Z

Hey there, I'm still new to the github workflow, so please let me know how I can improve :)
This is part of my bachelors thesis supervised by @doneachh.
Thank you @ekatef and @euronion for offering your support.
This pr is a work in progress, next step for me is to make river discharge provided by glofas selectable in cutout.hydro()

for more information, see https://pre-commit.ci

ekatef

Great @StuberSimon! Looks a very good progress, and happy to see GloFAS is being implemented. Have taken a liberty to do a preliminary code review and added a few comments hoping to assist you in getting used to github. My general impression that you are getting things perfectly right and the implementation looks quite neat in general.

ekatef · 2026-04-08T20:11:44Z

+    monthly_requests : bool, optional
+        If True, the data is requested on a monthly basis. This is useful for
+        large cutouts, where the data is requested in smaller chunks. The
+        default is False


Given EWDS limitations, I'd expect monthly_requests=True could be a more reasonable default. Could be good to test if monthly_requests=False actually works for GloFAS retrieval

As explained in the comment above, monthly_requests=False does work quite well for GloFAS retrieval.
monthly_requests=True makes the retrieval slower, as atlite has to wait for each api call seperately.
So I would suggest to not change the default to True.
But happy to discuss this further if you want :)

ekatef · 2026-04-08T20:18:41Z

+    Get inflow time-series for `plants` by extracting the discharge time series for
+    the nearest grid points.


Completely agree that it's worth to make sure the naming reflects different nature of variables in ERA5 and GloFAS datasets.
It could be a good idea also to mention both datasets in docstrings of the respective functions (that is mention here that _hydro_from_discharge is intended for usage on GloFAS data)

ekatef · 2026-04-08T20:28:11Z

Hey there, I'm still new to the github workflow, so please let me know how I can improve :) This is part of my bachelors thesis supervised by @doneachh. Thank you @ekatef and @euronion for offering your support. This pr is a work in progress, next step for me is to make river discharge provided by glofas selectable in cutout.hydro()

Thank you @StuberSimon, looks a great contribution! From my users' perspective, I strongly confirm that having GloFAS would be an amazing feature 🙂

Have added a few preliminary technical comments while leaving more in-depth analysis for @euronion who has much deeper understanding of atlite architecture.

For the next step, do you need any support?

ValeMTo · 2026-04-10T13:10:25Z

Hi @StuberSimon, @ekatef, and @euronion

First off, great work on this PR. I completely agree with @ekatef that having GloFAS natively integrated is a fantastic and much-needed feature for the community.

I'm jumping into this conversation since @Asdominet34 and I have worked on improving the hydromodeling in PyPSA-Eur, by firstly connecting it with GloFAS, "calibrating" it, and then validating it with the real hydro production in Europe. Before pushing the work, we would like to finalize the pumped-storage logic and the working paper we had in mind. I think that there is space for collaboration. What do you think about organising a call next week?

StuberSimon · 2026-04-14T12:40:30Z

Hi @ValeMTo that sounds great!
This week was already quite full, what do you think about next Tuesday, April 21st between 11:45 and 15:15 (UTC+2)?
You can reach me on discord under the name simonstuber.

Asdominet34 · 2026-04-17T07:20:59Z

Hi @ValeMTo that sounds great! This week was already quite full, what do you think about next Tuesday, April 21st between 11:45 and 15:15 (UTC+2)? You can reach me on discord under the name simonstuber.

Hi @simon, thanks for your message!

Unfortunately next week is quite busy on our side as well, so we won’t be available then. We’ll get back to you shortly with our availability starting from the following week.

Sorry again and looking forward to connecting!

coroa

Sorry for dragging my feet on this PR. Actually, i retract my earlier statement that this has no space within atlite. I think you showed that this can be accommodated here in a way that is useful and backwards compatible.

A couple of changes would be good:

abstract common code shared between era5.py and glofas.py in some cds_helper.py module
clean up the coordinates and variable names in use (ie. comments by @ekatef )
maybe an evaluation of potential heuristics for selecting the correct discharge grid cells

coroa · 2026-06-02T13:10:48Z

+    for plant in plants.itertuples():
+        # Extract the discharge time series for the nearest point
+        inflow.loc[dict(plant=plant.Index)] = discharge.sel(
+            x=plant.lon, y=plant.lat, method="nearest"
+        )


My previous look into these datasets suggested that it is quite easy to miss the correct river cells due to small misalignments in the datasets, so i'd expect one would like to have some more sophisticated find closest river cells which are actually part of the river, rather than find closest cell heuristic. And that this would need testing.

After some quick thinking probably the best thing would be a snapping to the largest value in the static uparea map (gridded data with the total catchment area of each cell).

That is available from here: https://confluence.ecmwf.int/display/CEMS/Auxiliary+Data

This could also be a pre-processing step on the powerplant dataset for which we provide a helper function that has to be called before.

The only problem is then that when your plant is right before a fork you would latch onto the larger river.

Hey @coroa thanks for the tip!

I found solving this (especially the fork/larger river problem) quite difficult.
I think I found a working, though not perfect and quite complicated solution.
As the full solution requires some major code additions, I'm not sure if it fits in atlite or this pr.
You can find a first draft and results of these changes here: StuberSimon#1

Note that those changes will still need some preprocessing done in pypsa-earth:

Calculating expected upstream area of plants (needs head from GHR)

Getting river names for plants and mapping them (from GHR to OSM)

Passing those to atlite snapping function, which will combine area matching and river name snapping to find the right glofas cell for each plant

-> This shouldn't be too hard, as GHR and OSM are already used in pypsa-earth. Only the name mapping needs to be done by the user.

I would suggest one of the following options:

Keep snapping to nearest cell, leave it for future improvements to be solved

Option 1, implement solution for this in pypsa-earth

Add only largest upstream area snapping functionality, leave fork/larger river problem to future improvements

Implement whole package (upstream area and river name matching) in this pr

Option 1, but I'll implement the whole package in a follow up pr

I would personally prefer option 5. It would keep this pr small and provide a more accurate river cell snapping. I am planning substantial hydro functionality improvements in pypsa-earth after this, and fixing the snapping problem before would be important for those upgrades too.

I would love to hear what you and others (@ekatef) think about that!

ekatef · 2026-06-02T22:25:26Z

Hello!
@coroa thank you so much for reviewing! It helps a lot to have your insights.

@StuberSimon if you'd need any support with additional testing, happy to assist. We are currently working with GloFAS for one of the projects, and there is a bit of struggle exactly with finding the river route as @coroa has mentioned.

# Conflicts: # .gitignore # atlite/convert.py # atlite/datasets/era5.py

FabianHofmann · 2026-07-17T09:15:08Z

+    else:
+        ds = xr.open_dataset(target, chunks=sanitize_chunks(chunks))
+        if tmpdir is None:
+            add_finalizer(target)


the signature of add_finalizer suggest that here is a bug, should be add_finalizer(ds, target). encountered while merge the latest master on this branch. will push the fix

FabianHofmann

to give this a little push, I am merging this latest master into here. I will follow up with more comments later

- fix add_finalizer(ds, target) call in glofas retrieve_data (was missing ds) - make zip extraction tmpdir-safe via Path(target).parent - align cds_helper helpers with master's typed, lint-clean versions - add type annotations, Returns/Raises docstrings, and logging/pathlib fixes

- restore hydrobasins as 3rd positional arg, make module/time keyword-only so existing positional callers (cutout.hydro(plants, basins)) keep working; collapse the duplicated auto/glofas/era5 dispatch into one path - point era5 request-tracking url back to the CDS server (was pointing at the EWDS/GloFAS server) - drop code-restating comments in glofas retrieve_data

StuberSimon and others added 3 commits April 8, 2026 15:46

add GLOFAS dataset module

61810fa

split hydro functionality into runoff and discharge based

5f3cfec

[pre-commit.ci] auto fixes from pre-commit.com hooks

33ef0bf

for more information, see https://pre-commit.ci

StuberSimon force-pushed the glofas-support branch from 79878e8 to 33ef0bf Compare April 8, 2026 13:52

ekatef reviewed Apr 8, 2026

View reviewed changes

ekatef mentioned this pull request Apr 14, 2026

[PARENT] Enable using custom data for hydro potential open-energy-transition/pypsa-zambia#162

Closed

coroa requested changes Jun 2, 2026

View reviewed changes

StuberSimon added 8 commits June 9, 2026 09:25

remove debug prints from _hydro_from_discharge

8d7b645

build glofas retrieval_times dates with strftime

000d840

return glofas discharge at native resolution

3fee4df

extract shared CDS helpers into cds_helper.py

e397aea

snap plants to nearest cell and interpolate hydro discharge

5dcc176

use cutout.data.data_vars consistently for hydro data checks

989c0de

move hydro inflow helpers from convert.py into hydro.py

befe311

update docstrings

aa0bc40

ekatef mentioned this pull request Jul 6, 2026

Enable using GloFAS inflow to calculate hydro potential open-energy-transition/pypsa-zambia#267

Merged

10 tasks

StuberSimon mentioned this pull request Jul 13, 2026

snap plants to corresponding river StuberSimon/atlite_hydro#1

Draft

5 tasks

Merge remote-tracking branch 'origin/master' into glofas-support

630b818

# Conflicts: # .gitignore # atlite/convert.py # atlite/datasets/era5.py

FabianHofmann reviewed Jul 17, 2026

View reviewed changes

FabianHofmann added 2 commits July 17, 2026 11:21

Merge branch 'master' into glofas-support

65baad1

FabianHofmann marked this pull request as ready for review July 17, 2026 10:09

		Get inflow time-series for `plants` by extracting the discharge time series for
		the nearest grid points.

Uh oh!

Conversation

StuberSimon commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes proposed in this Pull Request

Checklist

Uh oh!

StuberSimon commented Apr 6, 2026

Uh oh!

ekatef left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ekatef Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

StuberSimon Jun 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ekatef Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

ekatef commented Apr 8, 2026

Uh oh!

ValeMTo commented Apr 10, 2026

Uh oh!

StuberSimon commented Apr 14, 2026

Uh oh!

Asdominet34 commented Apr 17, 2026

Uh oh!

coroa left a comment

Choose a reason for hiding this comment

Uh oh!

coroa Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

coroa Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

StuberSimon Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ekatef commented Jun 2, 2026

Uh oh!

FabianHofmann Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

FabianHofmann left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

StuberSimon commented Apr 6, 2026 •

edited

Loading

coroa Jun 3, 2026 •

edited

Loading