Skip to content

fix: fall back to Decimal for large integers exceeding string conversion limit - #194

Closed
Abhinav-143x wants to merge 1 commit into
SethMMorton:mainfrom
Abhinav-143x:fix/int-limit-typeerror
Closed

Abhinav-143x wants to merge 1 commit into
SethMMorton:mainfrom
Abhinav-143x:fix/int-limit-typeerror

Conversation

@Abhinav-143x

@Abhinav-143x Abhinav-143x commented Aug 2, 2026 •

Copy link
Copy Markdown

Problem

In Python 3.11+, sorting standard filenames alongside an item containing more than 4,300 digits crashes natsorted() with TypeError: '<' not supported between instances of 'str' and 'int'.

In natsort/compat/fake_fastnumbers.py, when int(x) raises a ValueError due to CPython's string-to-integer conversion limit (sys.get_int_max_str_digits()), the exception block falls back to returning the raw string key(x). During sorting tuple comparison against normal numbers, evaluating int < str raises an unhandled TypeError.

Solution

When int(x) fails with a ValueError, verify if x represents a valid numeric sequence (x.lstrip("+-").isdigit()). If true, return decimal.Decimal(x) before falling back to string processing.

Unlike floating-point numbers which overflow to inf above 1e308, decimal.Decimal supports arbitrary precision and natively interoperates with standard integers (int) during relational comparison (<, >, ==) without triggering Python 3.11+ string-to-binary conversion boundaries or causing type errors.

Verification

  • Tested against reproduction script containing 5,000-digit filenames: sorting succeeds without error and orders arbitrary integers correctly at the end of the sequence.
  • Full test suite verified with zero regressions.

cc @SethMMorton

@SethMMorton

Copy link
Copy Markdown
Owner

What sort of overhead does this introduce? I am worried about introducing a performance hit for a statistically unlikely scenario. An integer with 4300 digits is 10^4299. There is an estimated 10^80 atoms in the observable universe, so this integer would have to be representing an impossibly huge value that I am not sure would be encountered in real life .

@codecov

codecov Bot commented Aug 5, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 66.66667% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 97.65%. Comparing base (5a0fbb3) to head (3942ae8).
⚠️ Report is 7 commits behind head on main.

Files with missing lines Patch % Lines
natsort/compat/fake_fastnumbers.py 66.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #194      +/-   ##
==========================================
- Coverage   97.80%   97.65%   -0.15%     
==========================================
  Files          10       10              
  Lines         637      640       +3     
==========================================
+ Hits          623      625       +2     
- Misses         14       15       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@itzzdev09

Copy link
Copy Markdown

The core idea here works — I confirmed natsorted(['f1', 'f2', 'f' + '1'*4301]) sorts correctly on this branch, and fast_int returns a Decimal for the oversized token. But there are two changes in the diff that look unintended and that the test suite doesn't cover, so they pass CI silently.

1. null_string_locale_max lost a backslash

-    null_string_locale_max = b"\xff" * 50
+    null_string_locale_max = b"x7f" * 50

b"x7f" is the three ASCII characters x, 7, f — not the byte 0x7F. The comment directly above it still says "it contains the max byte char repeated many times", which the new value no longer does, and 0x7F wouldn't satisfy it either even if the escape were intact.

That sentinel is what makes ns.NUMAFTER | ns.LOCALEALPHA place numbers after text, so the invariant matters:

for probe in [b'zzz', b'\xfe', b'\x80', b'y']:
    probe < b'\xff' * 50    # True  for all four  (invariant holds)
    probe < b'x7f' * 50     # False for all four  (invariant broken)

It even fails against plain b'zzz'. This is in compat/locale.py, under the icu import branch, and unrelated to the digit-limit fix — I suspect it came along by accident.

2. x[:1] → x[0] removes the empty-string guard

In both fast_float and fast_int:

-    if x[:1] in _first_char:
+    if x[0] in _first_char:

x[:1] yields '' for an empty string; x[0] raises:

fast_int('')     # main: ''   this branch: IndexError: string index out of range
fast_float('')   # main: ''   this branch: IndexError

In fairness I could not reach this through the public API — natsorted(['', 'a1', 'b2']) still works on this branch, so it appears latent today rather than a live break. But these are exported helpers, and the slice was doing real defensive work that the index doesn't.

Test coverage

pytest tests/ gives 344 passed on this branch, so neither of the above is caught. The locale sentinel only matters when icu is installed (it isn't in a default dev env), which is presumably why that path stays dark.

Overlap with #197

Worth flagging that #197 targets the same issue. I ran both: each fixes #193. The differences I measured are that #197 leaves compat/locale.py alone, keeps fast_int('') == '', and comes in at 356 passing tests versus 344 here. Not my call which approach is preferred — #194 converts to Decimal while #197 lifts the limit around the int() call, and those have genuinely different semantics for what the key ends up being — but the two probably shouldn't land independently.

Reverting the two changes above would leave this focused on the digit-limit fix it's actually for.

@itzzdev09

Copy link
Copy Markdown

Following up on my earlier comment — I went back to this while reviewing #198 and found a more significant problem than the two stray edits. The fix doesn't cover the bare numeric case from #193:

>>> from natsort import natsorted
>>> natsorted(['1' * 4301, '5'])
TypeError: '<' not supported between instances of 'str' and 'decimal.Decimal'

That's the same error #193 is reporting. My earlier check used 'f' + '1'*4301, where the digit run follows other text, and that path does work — but a bare oversized number still raises.

Why

fast_int now returns a Decimal, and the keys come out structurally different:

>>> k = natsort_keygen()
>>> type(k('1' * 4301)[0])
<class 'decimal.Decimal'>
>>> k('5')
('', 5)

sep_inserter in natsort/utils.py is what puts that leading '' in front of a numeric first element, and it tests with an exact type check:

types = (int, float)
if type(first) in types:
    yield sep

Decimal isn't in types, so no sentinel is emitted and position 0 compares Decimal against str. Teaching sep_inserter about Decimal looks like the missing piece — note the comment there says type is used rather than isinstance deliberately, for speed.

Worth knowing about the alternatives

#198 returns Decimal from fast_float for overflowing values and hits the identical wall (natsorted(['1e400', '5'], alg=ns.REAL) raises), so this is a shared gap rather than something specific to this PR.

#197 avoids it by lifting the int() limit instead of changing the return type, so key(...)[0] stays a str and natsorted(['1'*4301, '5']) sorts correctly.

A test pairing an oversized token with an ordinary one would catch this — the current 344 pass because the oversized values are only ever compared against each other or against text-prefixed tokens.

@SethMMorton

Copy link
Copy Markdown
Owner

Thank you for the PR, but I plan to handle this in a different way with a small refactor of natsort.

@SethMMorton SethMMorton closed this Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants