Repository navigation
fix: fall back to Decimal for large integers exceeding string conversion limit - #194
Abhinav-143x wants to merge 1 commit into
Conversation
|
What sort of overhead does this introduce? I am worried about introducing a performance hit for a statistically unlikely scenario. An integer with 4300 digits is 10^4299. There is an estimated 10^80 atoms in the observable universe, so this integer would have to be representing an impossibly huge value that I am not sure would be encountered in real life . |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #194 +/- ##
==========================================
- Coverage 97.80% 97.65% -0.15%
==========================================
Files 10 10
Lines 637 640 +3
==========================================
+ Hits 623 625 +2
- Misses 14 15 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
The core idea here works — I confirmed 1. - null_string_locale_max = b"\xff" * 50
+ null_string_locale_max = b"x7f" * 50
That sentinel is what makes for probe in [b'zzz', b'\xfe', b'\x80', b'y']:
probe < b'\xff' * 50 # True for all four (invariant holds)
probe < b'x7f' * 50 # False for all four (invariant broken)It even fails against plain 2. In both - if x[:1] in _first_char:
+ if x[0] in _first_char:
fast_int('') # main: '' this branch: IndexError: string index out of range
fast_float('') # main: '' this branch: IndexErrorIn fairness I could not reach this through the public API — Test coverage
Overlap with #197 Worth flagging that #197 targets the same issue. I ran both: each fixes #193. The differences I measured are that #197 leaves Reverting the two changes above would leave this focused on the digit-limit fix it's actually for. |
|
Following up on my earlier comment — I went back to this while reviewing #198 and found a more significant problem than the two stray edits. The fix doesn't cover the bare numeric case from #193: >>> from natsort import natsorted
>>> natsorted(['1' * 4301, '5'])
TypeError: '<' not supported between instances of 'str' and 'decimal.Decimal'That's the same error #193 is reporting. My earlier check used Why
>>> k = natsort_keygen()
>>> type(k('1' * 4301)[0])
<class 'decimal.Decimal'>
>>> k('5')
('', 5)
types = (int, float)
if type(first) in types:
yield sep
Worth knowing about the alternatives #198 returns #197 avoids it by lifting the A test pairing an oversized token with an ordinary one would catch this — the current 344 pass because the oversized values are only ever compared against each other or against text-prefixed tokens. |
|
Thank you for the PR, but I plan to handle this in a different way with a small refactor of |
Problem
In Python 3.11+, sorting standard filenames alongside an item containing more than 4,300 digits crashes
natsorted()withTypeError: '<' not supported between instances of 'str' and 'int'.In
natsort/compat/fake_fastnumbers.py, whenint(x)raises aValueErrordue to CPython's string-to-integer conversion limit (sys.get_int_max_str_digits()), the exception block falls back to returning the raw stringkey(x). During sorting tuple comparison against normal numbers, evaluatingint < strraises an unhandledTypeError.Solution
When
int(x)fails with aValueError, verify ifxrepresents a valid numeric sequence (x.lstrip("+-").isdigit()). If true, returndecimal.Decimal(x)before falling back to string processing.Unlike floating-point numbers which overflow to
infabove1e308,decimal.Decimalsupports arbitrary precision and natively interoperates with standard integers (int) during relational comparison (<,>,==) without triggering Python 3.11+ string-to-binary conversion boundaries or causing type errors.Verification
cc @SethMMorton