Repository navigation
Fix TypeError sorting a numeric token longer than the int() str-digit limit - #197
afonsojanu wants to merge 1 commit into
Conversation
On Python 3.11+ int() raises ValueError for a digit string longer than
sys.get_int_max_str_digits() (4300 by default). natsort's pure-Python
fast_int fallback caught that ValueError the same as any other failed
conversion and returned the original string, so an oversized numeric
token quietly turned into a str while its shorter siblings stayed int.
Comparing the two during the actual sort then raised
TypeError: '<' not supported between instances of 'str' and 'int'
for input that has nothing wrong with it, just one unusually long run
of digits (a huge serial number, a hash-like filename component, etc).
fast_int now recognizes this specific failure (the leftover text after
an optional sign is a plain decimal run) and converts it anyway with
the digit limit briefly lifted for that one call, restoring the
previous limit right after. This mirrors what already happens when the
optional fastnumbers C extension is installed, since that extension
has no equivalent limit.
Added a unit test for fast_int directly and an end-to-end natsorted
test reproducing the crash from the bug report.
| return int(x) | ||
| except ValueError: | ||
| unsigned = x[1:] if x[:1] in "+-" else x | ||
| if unsigned.isdecimal(): |
There was a problem hiding this comment.
How about x.lstrip("-+").isdecimal()?
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #197 +/- ##
==========================================
- Coverage 97.80% 92.83% -4.97%
==========================================
Files 10 10
Lines 637 656 +19
==========================================
- Hits 623 609 -14
- Misses 14 47 +33 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
This is similar to #194 but takes an approach that does not introduce a new return type. I am curious - did this error arise on real world data, or was this in search of a bug to fix? As I pointed out in #194 an integer with 4300 digits is 10^4299, but there is an estimated 10^80 atoms in the observable universe, so this integer would have to be representing an impossibly huge value that I am not sure would be encountered in real life. Can you do some performance testing to demonstrate that this does not introduce a performance hit for real-world data? |
|
Part of a general cleanup on my end, closing this along with the rest of what I had open. |
Fixes #193.
The bug
On Python 3.11+,
int()raisesValueErrorwhen the string it is given has more digits thansys.get_int_max_str_digits()(4300 by default). Innatsort/compat/fake_fastnumbers.py,fast_intcatches thatValueErrorthe same way it catches a genuine "this isn't a number" failure, and falls back to returning the original text as a plain string.That means an oversized numeric token quietly becomes a
strwhile an ordinarily-sized numeric token elsewhere in the same input stays anint.natsortbuilds its sort keys as tuples that alternate string and number, counting on same-shaped inputs producing same-shaped tuples, so once one entry's key has astrsitting where another entry's key has anint, the actualsorted()call blows up:This only affects the pure-Python fallback used when the optional
fastnumbersC extension isn't installed.fastnumbershas no equivalent digit limit, so the same input sorts fine with it installed, which is what makes this particularly confusing to hit.The fix
fast_intnow checks, on theValueErrorpath, whether the text (after an optional leading sign) is a plain run of decimal digits. If it is, the failure was the digit-count guard rail rather than an actual non-numeric string, so it converts the value anyway with the limit temporarily disabled for that call (restored immediately after via a small context manager), rather than downgrading a valid integer into a string.sys.set_int_max_str_digitsonly exists on 3.11+, so the whole thing is a no-op (behaves exactly as before) on earlier versions, which this repo still supports.Tests
Added a direct unit test for
fast_inton a 5000-digit string (covering unsigned,+, and-), including a check that the digit limit is restored to its prior value afterward, plus an end-to-endnatsortedtest that reproduces the crash from the report. Both new tests fail against the old code and pass with the fix. The full existing suite (356 tests) still passes.