Skip to content

Widen overflowing float magnitudes to Decimal instead of losing order - #198

Closed
afonsojanu wants to merge 2 commits into
SethMMorton:mainfrom
afonsojanu:fix-overflowing-float-magnitude-ordering
Closed

afonsojanu wants to merge 2 commits into
SethMMorton:mainfrom
afonsojanu:fix-overflowing-float-magnitude-ordering

Conversation

@afonsojanu

Copy link
Copy Markdown

What's wrong

Under ns.REAL/ns.FLOAT, when the optional fastnumbers accelerator isn't installed, natsort parses numeric tokens through the pure Python fallback in natsort/compat/fake_fastnumbers.py, which uses plain float(). float() silently rounds a finite number to +inf/-inf once the magnitude is too large for a double, instead of raising.

That means two clearly different values that both overflow end up comparing equal:

>>> from natsort import natsorted, ns
>>> natsorted(["1e400", "1e500"], alg=ns.REAL)
['1e400', '1e500']
>>> natsorted(["1e500", "1e400"], alg=ns.REAL)
['1e500', '1e400']

Both calls "succeed" with no error or warning, but they disagree about which value is larger, because internally both "1e400" and "1e500" parse to the same inf key. Whichever value happened to come first in the input wins, which is a silent correctness bug rather than a crash.

Fix

When fast_float() sees that float(x) overflowed to infinity, it now checks whether the original string was actually a literal "inf"/"nan" spelling (which should keep behaving exactly as before) or genuinely just too large a finite number. In the latter case it widens the result to Decimal(x) instead of inf. Decimal has a much bigger exponent range, so "1e400" and "1e500" keep their real relative order, and Decimal compares correctly against the plain floats used everywhere else in the key.

This only covers the bundled fallback, since the optional fastnumbers C extension has the same underlying float()-based limit and isn't something this project controls from here.

Tests

  • Added a unit test on fast_float directly showing "1e400" and "1e500" no longer compare equal, and that literal "inf"/"-inf" strings still come back as plain floats.
  • Added an end-to-end natsorted(..., alg=ns.REAL) regression test with both input orders, matching how the bug was originally reported.
  • Full existing test suite (358 tests) still passes, and ruff/mypy are clean on the changed files.

Under ns.REAL/ns.FLOAT, the pure Python fallback parses numeric tokens
with float(), which silently rounds a finite number to +/-inf once the
magnitude is too large instead of raising. Two distinct values like
"1e400" and "1e500" both end up as inf and compare equal, so which one
sorts first depends on the input order rather than their actual size.

Widen such values to Decimal when this happens, since it has a much
larger exponent range and compares correctly against plain floats. A
literal "inf"/"nan" spelling is left as a plain float, since it is not
the result of an overflow and existing callers rely on that.

This only covers the bundled fallback used when the optional
fastnumbers accelerator is not installed, since that C extension has
the same float() based limit and isn't something natsort controls.
@codecov

codecov Bot commented Sep 19, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.26%. Comparing base (607eeee) to head (2e301cd).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #198      +/-   ##
==========================================
- Coverage   97.80%   94.26%   -3.54%     
==========================================
  Files          10       10              
  Lines         637      645       +8     
==========================================
- Hits          623      608      -15     
- Misses         14       37      +23     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

"""
if ret not in (float("inf"), float("-inf")):
return ret
if x.strip().lstrip("+-")[:3].lower() in ("inf", "nan"):

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the _NAN_INF set for this.

it to keep such values distinguishable. A literal "inf"/"nan" spelling
is left alone as a plain float, since it isn't the result of an overflow.
"""
if ret not in (float("inf"), float("-inf")):

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we pre-define float("inf") and float("-inf") so that we don't have to create this object on each call?

The previous fix only widened overflowing magnitudes to Decimal inside
natsort's fallback fast_float implementation. When fastnumbers>=5.0.0 is
actually installed, natsort binds its try_float straight to the real
library's try_float, which bypasses that fallback entirely and never
widens anything, so two different overflowing values like "1e400" and
"1e500" both collapsed back to plain +/-inf and lost their relative
order (they'd only keep their input order via list stability, which is
why reversing the input flipped the sorted result).

natsort.compat.fastnumbers.try_float is now always natsort's own
function, regardless of which backend supplied the raw float, and it
post-processes every result to widen a magnitude that overflowed to
Decimal, whichever backend produced it. Decimal is threaded through the
rest of the typing (StrBytesNum, on_fail's return type) so this type
checks cleanly under mypy --strict.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@itzzdev09

Copy link
Copy Markdown

The diagnosis in #192 is right and widen_if_overflowed reads well — leaving literal inf/nan spellings alone as plain floats is a nice touch. But as it stands this branch turns the silent mis-ordering into a hard TypeError on very ordinary input:

>>> from natsort import natsorted, ns
>>> natsorted(['1e400', '5'], alg=ns.REAL)
TypeError: '<' not supported between instances of 'str' and 'decimal.Decimal'

Any overflowing value next to any other element does it — ['1e400', '2.5'], ['1e400', 'a'] and ['1e500', '1e400', '5'] all raise. Sorting a homogeneous list of overflowing values works, which is why the added tests pass.

Root cause

The keys come out structurally different:

>>> k = natsort_keygen(alg=ns.REAL)
>>> k('1e400')
(Decimal('1E+400'),)
>>> k('5')
('', 5.0)

sep_inserter in natsort/utils.py is what normally puts that leading '' in front of a numeric first element, and it decides with an exact type check:

types = (int, float)
first = next(iterator)
if type(first) in types:
    yield sep

Decimal isn't in types, and because this is type(x) in types rather than isinstance, a Decimal won't match however it's registered. So the sentinel is skipped and position 0 ends up comparing Decimal against str — exactly the failure mode the tuple keys exist to avoid.

Adding Decimal to that tuple looks like the missing piece. The comment there notes type is used deliberately for speed, so widening it to isinstance probably isn't wanted.

This is shared with #194

Worth coordinating, because #194 returns Decimal from fast_int and hits the identical wall:

>>> natsorted(['1' * 4301, '5'])        # on #194
TypeError: '<' not supported between instances of 'str' and 'decimal.Decimal'

That is the very error #193 reports, so #194 currently only fixes the case where the digit run follows other text ('f' + '1'*4301), not a bare numeric string. #197 takes the other route — it lifts the int() limit rather than changing the return type — and natsorted(['1'*4301, '5']) sorts correctly there, with key(...)[0] still a str.

So whichever of the two Decimal approaches is preferred, sep_inserter needs to learn about Decimal first, and a test mixing an oversized value with an ordinary one would catch this for both.

Verified against this branch at a7ded7c; pytest tests/ is 362 passed, so nothing here is caught today.

@afonsojanu

Copy link
Copy Markdown
Author

Closing this one too, same reason: starting fresh and clearing out everything I had open.

@afonsojanu afonsojanu closed this Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants