Skip to content

Fix TypeError sorting a numeric token longer than the int() str-digit limit - #197

Closed
afonsojanu wants to merge 1 commit into
SethMMorton:mainfrom
afonsojanu:fix-fast-int-digit-limit
Closed

afonsojanu wants to merge 1 commit into
SethMMorton:mainfrom
afonsojanu:fix-fast-int-digit-limit

Conversation

@afonsojanu

Copy link
Copy Markdown

Fixes #193.

The bug

On Python 3.11+, int() raises ValueError when the string it is given has more digits than sys.get_int_max_str_digits() (4300 by default). In natsort/compat/fake_fastnumbers.py, fast_int catches that ValueError the same way it catches a genuine "this isn't a number" failure, and falls back to returning the original text as a plain string.

That means an oversized numeric token quietly becomes a str while an ordinarily-sized numeric token elsewhere in the same input stays an int. natsort builds its sort keys as tuples that alternate string and number, counting on same-shaped inputs producing same-shaped tuples, so once one entry's key has a str sitting where another entry's key has an int, the actual sorted() call blows up:

import natsort
natsort.natsorted(["item_10.png", "item_2.png", "item_" + "9" * 5000 + ".png"])
# TypeError: '<' not supported between instances of 'str' and 'int'

This only affects the pure-Python fallback used when the optional fastnumbers C extension isn't installed. fastnumbers has no equivalent digit limit, so the same input sorts fine with it installed, which is what makes this particularly confusing to hit.

The fix

fast_int now checks, on the ValueError path, whether the text (after an optional leading sign) is a plain run of decimal digits. If it is, the failure was the digit-count guard rail rather than an actual non-numeric string, so it converts the value anyway with the limit temporarily disabled for that call (restored immediately after via a small context manager), rather than downgrading a valid integer into a string.

sys.set_int_max_str_digits only exists on 3.11+, so the whole thing is a no-op (behaves exactly as before) on earlier versions, which this repo still supports.

Tests

Added a direct unit test for fast_int on a 5000-digit string (covering unsigned, +, and -), including a check that the digit limit is restored to its prior value afterward, plus an end-to-end natsorted test that reproduces the crash from the report. Both new tests fail against the old code and pass with the fix. The full existing suite (356 tests) still passes.

On Python 3.11+ int() raises ValueError for a digit string longer than
sys.get_int_max_str_digits() (4300 by default). natsort's pure-Python
fast_int fallback caught that ValueError the same as any other failed
conversion and returned the original string, so an oversized numeric
token quietly turned into a str while its shorter siblings stayed int.
Comparing the two during the actual sort then raised

    TypeError: '<' not supported between instances of 'str' and 'int'

for input that has nothing wrong with it, just one unusually long run
of digits (a huge serial number, a hash-like filename component, etc).

fast_int now recognizes this specific failure (the leftover text after
an optional sign is a plain decimal run) and converts it anyway with
the digit limit briefly lifted for that one call, restoring the
previous limit right after. This mirrors what already happens when the
optional fastnumbers C extension is installed, since that extension
has no equivalent limit.

Added a unit test for fast_int directly and an end-to-end natsorted
test reproducing the crash from the bug report.
return int(x)
except ValueError:
unsigned = x[1:] if x[:1] in "+-" else x
if unsigned.isdecimal():

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about x.lstrip("-+").isdecimal()?

@codecov

codecov Bot commented Sep 10, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.83%. Comparing base (70745f5) to head (8803981).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
natsort/compat/fake_fastnumbers.py 85.00% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #197      +/-   ##
==========================================
- Coverage   97.80%   92.83%   -4.97%     
==========================================
  Files          10       10              
  Lines         637      656      +19     
==========================================
- Hits          623      609      -14     
- Misses         14       47      +33     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@SethMMorton

Copy link
Copy Markdown
Owner

This is similar to #194 but takes an approach that does not introduce a new return type.

I am curious - did this error arise on real world data, or was this in search of a bug to fix? As I pointed out in #194 an integer with 4300 digits is 10^4299, but there is an estimated 10^80 atoms in the observable universe, so this integer would have to be representing an impossibly huge value that I am not sure would be encountered in real life.

Can you do some performance testing to demonstrate that this does not introduce a performance hit for real-world data?

@afonsojanu

Copy link
Copy Markdown
Author

Part of a general cleanup on my end, closing this along with the rest of what I had open.

@afonsojanu afonsojanu closed this Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unhandled TypeError when sorting strings containing >4,300 digits on Python 3.11+

3 participants