Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
4d2bc2b
Added longtermdataset
Slmtdhy Nov 5, 2024
a53bbbf
Change the default region to "Caravan"
Slmtdhy Nov 12, 2024
bf271e7
Merge branch 'dev' into balstm
OuyangWenyu Nov 12, 2024
72f5ec5
Change the unit of average precipitation to "mm/M" and add unit conve…
Slmtdhy Nov 13, 2024
2d45dd0
Fixed bug reading basicid
Slmtdhy Nov 14, 2024
f8d8f1f
refine datasource
Slmtdhy Dec 30, 2024
4337fe4
Merge branch 'main' into balstm
Slmtdhy Dec 30, 2024
fbf4621
Merge branch 'dev' into balstm; debugging for gmspa but data need to be
OuyangWenyu Jul 3, 2025
6534e21
Merge branch 'dev' into balstm
OuyangWenyu Oct 30, 2025
98f03eb
tg_gnn test verson0.1
mahaoran2022 Nov 20, 2025
c7aae02
Merge branch 'dev' into tg
OuyangWenyu Nov 20, 2025
ca6e67d
remove useless mdfiles
OuyangWenyu Nov 22, 2025
aaf54a1
update tghydrodatasource init
OuyangWenyu Nov 22, 2025
088c5d1
updata tggnn version 1.0
mahaoran2022 Dec 12, 2025
84f851c
add floodeventdatasource with score
zhuanglaihong Dec 17, 2025
58fcf3a
dataaug
zyb1988 Apr 2, 2026
a24b52d
Merge branch 'dev' of gitcode.com:dlut-water/hydrodatasource into dev
zyb1988 Apr 2, 2026
b49a908
Merge branch 'main' into tg
OuyangWenyu Jun 17, 2026
9d15857
Merge branch 'main' into balstm
OuyangWenyu Jun 17, 2026
f529fdb
Merge branch 'tg' of gitcode.com:dlut-water/hydrodatasource into tg
OuyangWenyu Jun 17, 2026
3c7b7f4
Merge branch 'balstm'
OuyangWenyu Jun 17, 2026
8482bdf
feat: add data_resolver.py with READER_ALIASES for all 11 hydrodataso…
OuyangWenyu Jun 18, 2026
8614e9b
chore: add local hydrodataset source and unified data interface spec
OuyangWenyu Jun 18, 2026
931880a
feat: add configs/datasets.yml with hydrodatasource dataset registry
OuyangWenyu Jun 18, 2026
0fce21f
refactor: clean up config.py — remove DB/minio, add new APIs
OuyangWenyu Jun 18, 2026
30385c8
feat: add URI-accepting constructors to all 11 reader classes
OuyangWenyu Jun 18, 2026
3550269
feat: export READER_ALIASES, resolve_data_path, DatasetResolutionErro…
OuyangWenyu Jun 18, 2026
e9f8065
test: add 51 unit tests for URI constructors, config API, and package…
OuyangWenyu Jun 23, 2026
16ceb2c
fix: align data_resolver with hydrodataset public API, fix S3/os.path…
OuyangWenyu Jun 23, 2026
8bcda80
refactor: lazy-load config globals, unify CACHE_DIR with hydrodataset
OuyangWenyu Jun 23, 2026
606983f
refactor: unify data interface — URI-only constructors across all 11 …
OuyangWenyu Jun 23, 2026
43de2b0
refactor: remove library-level datasets.yml, clarify YAML is user-lay…
OuyangWenyu Jun 23, 2026
d8ee24b
refactor: rename _HDS_DATASETS to HDS_DATASETS, add __all__
OuyangWenyu Jun 24, 2026
956776d
feat: add open_dataset wrapper and align data_resolver with ResolverC…
OuyangWenyu Jun 25, 2026
436ac80
feat: support NetCDF attributes and make basins.shp optional
zym0123 Jul 5, 2026
7397166
fix: create cache dir if missing to avoid NetCDF write failure
zym0123 Jul 5, 2026
0fee67e
fix: harden data interface and adapt to hydroutils 0.2.0
OuyangWenyu Aug 15, 2026
8aac3c9
fix: harden cleaners, fix error masking, and remove dead code
OuyangWenyu Aug 15, 2026
b998881
docs: revise docs for unified URI-only interface and pin hydrodataset…
OuyangWenyu Aug 15, 2026
02991f8
ci: ignore local hydrodataset source in uv sync (--no-sources)
OuyangWenyu Aug 15, 2026
4cd8083
ci: add --no-sources to uv run steps in build/docs workflows
OuyangWenyu Aug 15, 2026
cb6c990
fix: pin h5py<3.15 to avoid HDF5 wheel mismatch with netCDF4 1.7.3
OuyangWenyu Aug 15, 2026
dd2c30f
fix: make tomllib import Python 3.10-compatible in compat test
OuyangWenyu Aug 15, 2026
d87992a
fix: address review findings on resolver robustness and cleanup
OuyangWenyu Aug 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,8 @@ jobs:

- name: Install dependencies
run: |
uv sync --dev
uv sync --no-sources --dev

- name: Run tests
run: |
uv run pytest -m "not internal_data"
uv run --no-sources pytest -m "not internal_data"
4 changes: 2 additions & 2 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,6 @@ jobs:
- name: Setup Python
run: uv python install 3.11
- name: Install dependencies
run: uv sync --all-extras
run: uv sync --no-sources --all-extras
- name: Deploy documentation
run: uv run mkdocs gh-deploy --force
run: uv run --no-sources mkdocs gh-deploy --force
2 changes: 1 addition & 1 deletion .github/workflows/pypi.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
run: uv python install 3.11

- name: Install dependencies
run: uv sync --all-extras # Install all dependencies including dev
run: uv sync --no-sources --all-extras # Install all dependencies including dev

- name: Build package
run: uv build
Expand Down
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -171,3 +171,9 @@ CLAUDE.md
.claude/
hydrodatasource/_version.py
GEMINI.md

# Cursor code graph
.codegraph/

# Specify / Speckit local tooling
.specify/
135 changes: 100 additions & 35 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,78 +11,144 @@ While libraries like [hydrodataset](https://github.com/OuyangWenyu/hydrodataset)

**`hydrodatasource`** is designed to solve this problem. It provides a flexible framework to read, process, and clean these custom datasets, preparing them for hydrological modeling and analysis.

The core of this framework is the `SelfMadeHydroDataset` class, which allows you to easily access your own data by organizing it into a simple, predefined directory structure.
`hydrodatasource` uses a unified **URI-only** design: every reader is constructed by pointing it at a directory or an `s3://` URI, and a one-line `open_dataset()` factory resolves a dataset id to a ready-to-use reader. The design is compatible with [hydrodataset](https://github.com/OuyangWenyu/hydrodataset).

## Reading Custom Datasets with `SelfMadeHydroDataset`
## Quick Start

This is the primary use case for `hydrodatasource`. If you have your own basin-level time series and attribute data, you can use this class to load it seamlessly.
If you have a **registered dataset** and a `~/hydro_setting.yml` (see [Configuration](#configuration)), resolve and open it in one line:

```python
from hydrodatasource.configs.data_resolver import open_dataset

# "songliao_event" is a hydrodatasource-registered dataset (flood-event reader)
ds = open_dataset("songliao_event")
```

To **open a custom dataset** by path, construct the reader directly:

```python
from hydrodatasource.reader.data_source import SelfMadeHydroDataset

reader = SelfMadeHydroDataset(uri="/path/to/my_dataset", time_unit=["1D"])
```

## Reading Custom Datasets

This is the primary use case for `hydrodatasource`. If you have your own basin-level time series and attribute data, you can use `SelfMadeHydroDataset` to load it seamlessly.

### 1. Prepare Your Data Directory

First, organize your data into the following folder structure:

```
/path/to/your_data_root/
└── my_custom_dataset/ # Your dataset's name
├── attributes/
│ └── attributes.csv
├── shapes/
│ └── basins.shp
└── timeseries/
├── 1D/ # Sub-folder for each time resolution (e.g., daily)
│ ├── basin_01.csv
│ ├── basin_02.csv
│ └── ...
└── 1D_units_info.json # JSON file with unit information
/path/to/my_dataset/
├── attributes/
│ └── attributes.csv
├── shapes/
│ └── basins.shp
└── timeseries/
├── 1D/ # Sub-folder for each time resolution (e.g., daily)
│ ├── basin_01.csv
│ ├── basin_02.csv
│ └── ...
└── 1D_units_info.json # JSON file with unit information
```

- **`attributes/attributes.csv`**: A CSV file containing static basin attributes (e.g., area, mean elevation). Must include a `basin_id` column that matches the filenames in the `timeseries` folder.
- **`shapes/basins.shp`**: A shapefile with the polygon geometry for each basin.
- **`timeseries/1D/`**: A folder for each time resolution (e.g., `1D` for daily, `3h` for 3-hourly). Inside, each CSV file should contain the time series data for a single basin and be named after its `basin_id`.
- **`timeseries/1D_units_info.json`**: A JSON file defining the units for each variable in your time series CSVs (e.g., `{"precipitation": "mm/d", "streamflow": "m3/s"}`).
- **`timeseries/1D/`**: A folder for each time resolution (`1h`, `3h`, `1D`, `8D`, `1M`). Inside, each CSV file should contain the time series data for a single basin and be named after its `basin_id`.
- **`timeseries/1D_units_info.json`**: A JSON file defining the units for each variable in your time series CSVs (e.g., `{"precipitation": "mm/d", "streamflow": "m3/s", "temperature": "degC"}`). Every variable you read must be listed here.

Extended datasets may add optional directories:

- **`intermediate/`** — interval-basin (区间流域) data with topological relationships (`attributes/`, `timeseries/`, `shapes/`), used by the TG basin reader.
- **`stations/`** — gauging-station data and adjacency matrices, used by the station reader.
- **`forecasts/`** — forecast time series, used by the forecast reader.

### 2. Read the Data in Python

Once your data is organized, you can use `SelfMadeHydroDataset` to read it with just a few lines of code.
Once your data is organized, point a URI-only reader at it:

```python
from hydrodatasource.reader.data_source import SelfMadeHydroDataset

# 1. Define the path to your data's parent directory and the dataset name
data_path = "/path/to/your_data_root/"
dataset_name = "my_custom_dataset"

# 2. Initialize the reader
# Specify the time units you want to work with
reader = SelfMadeHydroDataset(data_path=data_path, dataset_name=dataset_name, time_unit=["1D"])
# Pass the absolute path (or s3:// URI) of your dataset directory as `uri`
reader = SelfMadeHydroDataset(uri="/path/to/my_dataset", time_unit=["1D"])

# 3. Get a list of all available basin IDs
# Get a list of all available basin IDs
basin_ids = reader.read_object_ids()

# 4. Define the time range and variables you want to load
# Define the time range and variables you want to load
t_range = ["2000-01-01", "2010-12-31"]
variables_to_read = ["precipitation", "streamflow", "temperature"]

# 5. Read the time series data
# The result is a dictionary of xarray.Datasets, keyed by time unit
# Read the time series data (a dict of xarray.Datasets keyed by time unit)
timeseries_data = reader.read_ts_xrdataset(
gage_id_lst=basin_ids,
t_range=t_range,
var_lst=variables_to_read,
time_units=["1D"]
time_units=["1D"],
)

daily_data = timeseries_data["1D"]

print("Successfully loaded data:")
print(daily_data)

# You can also read the static attributes
# Static attributes are equally easy to read
attributes_data = reader.read_attr_xrdataset(gage_id_lst=basin_ids, var_lst=["area", "mean_elevation"])
print("\nAttributes:")
print(attributes_data)
```

> **Note on the old API.** Earlier versions accepted `data_path=` / `dataset_name=` constructor arguments.
> These were removed in favor of the unified `uri=` interface and now raise `ValueError`.

## Reader Aliases

All `hydrodatasource` readers are registered as aliases in `READER_ALIASES`, which downstream
projects (e.g. hydromodel) consume alongside hydrodataset's aliases:

| Alias | Class | Directory convention |
|-------|-------|---------------------|
| `selfmade` | `SelfMadeHydroDataset` | standard dataset (`attributes/`, `timeseries/`, `shapes/`) |
| `longterm` | `LongTermDataset` | self-made dataset with long-term support |
| `forecast` | `SelfMadeForecastDataset` | standard + `forecasts/` |
| `station` | `StationHydroDataset` | standard + `stations/` |
| `tghydro` | `TgHydroDatasource` | standard + `intermediate/` + LSTM predictions |
| `floodevent` | `FloodEventDatasource` | flood-event data with per-basin event markers |
| `gages` | `Gages` | GAGES-II public dataset |
| `grdc` | `Grdc` | GRDC public dataset |
| `rainfall` | `RainfallReader` | cleaned station rainfall |
| `crd` | `Crd` | China reservoir database |
| `rsvrinflow` | `RsvrInflowReader` | reservoir inflow data |

`hydrodataset`'s public datasets (e.g. `camels_us`) are also resolvable through the same
`open_dataset()` / `resolve_data_path()` interface.

## Configuration

`hydrodatasource` reads a shared `~/hydro_setting.yml` (same file used by `hydrodataset` and
`hydromodel`) using the unified `storage.*` format:

```yaml
storage:
default_source: local # 'local' or 'cloud' — fallback when source is not given
local:
root: 'D:\data\hydrodatasource' # main data root
cache: data\cache
s3: # optional — cloud (MinIO/S3) access
endpoint_url: 'http://minio:9000'
key: 'access_key'
secret: 'secret_key'
bucket: hydro-data
prefix: hydromodel
```

- When `default_source: local`, `resolve_data_path()` / `open_dataset()` resolve against `storage.local.root`.
- When `default_source: cloud`, they resolve against `storage.s3`.
- If `~/hydro_setting.yml` is missing, a default root `~/hydrodatasource_data` is used (with a warning).

## Other Features

Beyond reading data, `hydrodatasource` also includes modules for:
Expand All @@ -102,16 +168,15 @@ pip install hydrodatasource

### Development Setup

For developers, it is recommended to use `uv` to manage the environment, as this project has local dependencies (e.g., `hydroutils`, `hydrodataset`).
For developers, it is recommended to use `uv` to manage the environment, as this project has local dependencies (e.g., `hydroutils>=0.2.0`, `hydrodataset>=0.3.0`).

1. **Clone the repository:**
```bash
git clone https://github.com/iHeadWater/hydrodatasource.git
cd hydrodatasource
```

2. **Sync the environment with `uv`:**
This command will install all dependencies, including the local editable packages.
2. **Sync the environment with `uv`** (installs all extras, including dev tooling):
```bash
uv sync --all-extras
```
```
Loading
Loading