Skip to content

Commit 9ed7993

Browse files
fix: nullable integer streaming stats on pandas < 2 (#192)
test-matrix (3.9) has failed on main since #187: with pandas 1.5, the streaming state converted numeric columns via pd.to_numeric(s).to_numpy(dtype="float64"), which raises "cannot convert to 'float64'-dtype NumPy array with missing values" for nullable (masked) integer columns with NA. fd.clean_timeseries and StreamingCleaner hit it on any Int*/UInt* column with gaps. Pass na_value=np.nan there and in the time-series interpolation helper, which has the same conversion. Also pin the alert step #188 added to fetch-fixtures.yml, which #190's SHA sweep predated.
1 parent 20c9d3a commit 9ed7993

3 files changed

Lines changed: 11 additions & 8 deletions

File tree

‎.github/workflows/fetch-fixtures.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ jobs:
3333
run: pytest tests/test_online_datasets.py -m "not online" -q --no-cov
3434
- name: Open/refresh alert issue on failure
3535
if: failure() && github.event_name == 'schedule'
36-
uses: actions/github-script@v7
36+
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
3737
with:
3838
script: |
3939
const title = "weekly online fixture refresh failing";

‎src/freshdata/streaming/_state.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -107,7 +107,9 @@ def update(self, s: pd.Series) -> None:
107107
self.nunique_stat.update(np.array([float(s.nunique(dropna=True))]))
108108

109109
if is_numeric_dtype(s) and not is_bool_dtype(s):
110-
values = pd.to_numeric(s, errors="coerce").to_numpy(dtype="float64")
110+
# na_value: nullable (masked) columns with missing cells refuse a
111+
# plain float64 conversion on pandas < 2.
112+
values = pd.to_numeric(s, errors="coerce").to_numpy(dtype="float64", na_value=np.nan)
111113
values = values[~np.isnan(values)]
112114
if values.size:
113115
self._welford.update(values)

‎src/freshdata/streaming/_timeseries.py‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -428,18 +428,19 @@ def _interp_series(s: pd.Series, ts: pd.Series, method: str) -> pd.Series:
428428
return s.ffill()
429429
if method == "bfill":
430430
return s.bfill()
431+
# na_value: nullable (masked) columns with missing cells refuse a plain
432+
# float64 conversion on pandas < 2.
433+
values = s.to_numpy(dtype="float64", na_value=np.nan)
431434
if method == "time":
432-
tmp = pd.Series(s.to_numpy(dtype="float64"),
433-
index=pd.DatetimeIndex(ts))
435+
tmp = pd.Series(values, index=pd.DatetimeIndex(ts))
434436
try:
435437
out = tmp.interpolate(method="time", limit_area="inside")
436438
except ValueError: # non-monotonic / non-datetime index → linear fallback
437-
out = pd.Series(s.to_numpy(dtype="float64")).interpolate(
438-
method="linear", limit_area="inside")
439+
out = pd.Series(values).interpolate(method="linear", limit_area="inside")
439440
return pd.Series(out.to_numpy(), index=s.index)
440441
# linear
441-
return pd.Series(s.to_numpy(dtype="float64"),
442-
index=s.index).interpolate(method="linear", limit_area="inside")
442+
return pd.Series(values, index=s.index).interpolate(
443+
method="linear", limit_area="inside")
443444

444445
# -- step 3: seasonal imputation -------------------------------------------
445446

0 commit comments

Comments
 (0)