Repository navigation
Widen overflowing float magnitudes to Decimal instead of losing order - #198
afonsojanu wants to merge 2 commits into
Conversation
Under ns.REAL/ns.FLOAT, the pure Python fallback parses numeric tokens with float(), which silently rounds a finite number to +/-inf once the magnitude is too large instead of raising. Two distinct values like "1e400" and "1e500" both end up as inf and compare equal, so which one sorts first depends on the input order rather than their actual size. Widen such values to Decimal when this happens, since it has a much larger exponent range and compares correctly against plain floats. A literal "inf"/"nan" spelling is left as a plain float, since it is not the result of an overflow and existing callers rely on that. This only covers the bundled fallback used when the optional fastnumbers accelerator is not installed, since that C extension has the same float() based limit and isn't something natsort controls.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #198 +/- ##
==========================================
- Coverage 97.80% 94.26% -3.54%
==========================================
Files 10 10
Lines 637 645 +8
==========================================
- Hits 623 608 -15
- Misses 14 37 +23 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| """ | ||
| if ret not in (float("inf"), float("-inf")): | ||
| return ret | ||
| if x.strip().lstrip("+-")[:3].lower() in ("inf", "nan"): |
There was a problem hiding this comment.
Use the _NAN_INF set for this.
| it to keep such values distinguishable. A literal "inf"/"nan" spelling | ||
| is left alone as a plain float, since it isn't the result of an overflow. | ||
| """ | ||
| if ret not in (float("inf"), float("-inf")): |
There was a problem hiding this comment.
Could we pre-define float("inf") and float("-inf") so that we don't have to create this object on each call?
The previous fix only widened overflowing magnitudes to Decimal inside natsort's fallback fast_float implementation. When fastnumbers>=5.0.0 is actually installed, natsort binds its try_float straight to the real library's try_float, which bypasses that fallback entirely and never widens anything, so two different overflowing values like "1e400" and "1e500" both collapsed back to plain +/-inf and lost their relative order (they'd only keep their input order via list stability, which is why reversing the input flipped the sorted result). natsort.compat.fastnumbers.try_float is now always natsort's own function, regardless of which backend supplied the raw float, and it post-processes every result to widen a magnitude that overflowed to Decimal, whichever backend produced it. Decimal is threaded through the rest of the typing (StrBytesNum, on_fail's return type) so this type checks cleanly under mypy --strict. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
The diagnosis in #192 is right and >>> from natsort import natsorted, ns
>>> natsorted(['1e400', '5'], alg=ns.REAL)
TypeError: '<' not supported between instances of 'str' and 'decimal.Decimal'Any overflowing value next to any other element does it — Root cause The keys come out structurally different: >>> k = natsort_keygen(alg=ns.REAL)
>>> k('1e400')
(Decimal('1E+400'),)
>>> k('5')
('', 5.0)
types = (int, float)
first = next(iterator)
if type(first) in types:
yield sep
Adding This is shared with #194 Worth coordinating, because #194 returns >>> natsorted(['1' * 4301, '5']) # on #194
TypeError: '<' not supported between instances of 'str' and 'decimal.Decimal'That is the very error #193 reports, so #194 currently only fixes the case where the digit run follows other text ( So whichever of the two Verified against this branch at a7ded7c; |
|
Closing this one too, same reason: starting fresh and clearing out everything I had open. |
What's wrong
Under
ns.REAL/ns.FLOAT, when the optionalfastnumbersaccelerator isn't installed, natsort parses numeric tokens through the pure Python fallback innatsort/compat/fake_fastnumbers.py, which uses plainfloat().float()silently rounds a finite number to+inf/-infonce the magnitude is too large for a double, instead of raising.That means two clearly different values that both overflow end up comparing equal:
Both calls "succeed" with no error or warning, but they disagree about which value is larger, because internally both
"1e400"and"1e500"parse to the sameinfkey. Whichever value happened to come first in the input wins, which is a silent correctness bug rather than a crash.Fix
When
fast_float()sees thatfloat(x)overflowed to infinity, it now checks whether the original string was actually a literal"inf"/"nan"spelling (which should keep behaving exactly as before) or genuinely just too large a finite number. In the latter case it widens the result toDecimal(x)instead ofinf.Decimalhas a much bigger exponent range, so"1e400"and"1e500"keep their real relative order, andDecimalcompares correctly against the plain floats used everywhere else in the key.This only covers the bundled fallback, since the optional
fastnumbersC extension has the same underlyingfloat()-based limit and isn't something this project controls from here.Tests
fast_floatdirectly showing"1e400"and"1e500"no longer compare equal, and that literal"inf"/"-inf"strings still come back as plain floats.natsorted(..., alg=ns.REAL)regression test with both input orders, matching how the bug was originally reported.ruff/mypyare clean on the changed files.