Repository navigation
Widen overflowing float magnitudes to Decimal instead of losing order #198
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
afonsojanu
wants to merge
2
commits into
SethMMorton:main
from
afonsojanu:fix-overflowing-float-magnitude-ordering
Closed
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,6 +7,7 @@ | |
| from __future__ import annotations | ||
|
|
||
| import unicodedata | ||
| from decimal import Decimal, InvalidOperation | ||
| from typing import Callable, Union | ||
|
|
||
| from natsort.unicode_numbers import decimal_chars | ||
|
|
@@ -34,10 +35,31 @@ | |
| ASCII_NUMS = "0123456789+-" | ||
| POTENTIAL_FIRST_CHAR = frozenset(decimal_chars + list(ASCII_NUMS + ".")) | ||
|
|
||
| StrOrFloat = Union[str, float] | ||
| StrOrFloat = Union[str, float, Decimal] | ||
| StrOrInt = Union[str, int] | ||
|
|
||
|
|
||
| def widen_if_overflowed(x: str, ret: float) -> StrOrFloat: | ||
| """ | ||
| Widen a float that overflowed to +/-inf back into something orderable. | ||
|
|
||
| float() silently rounds a finite number with too large a magnitude to | ||
| +/-inf instead of raising, so two different numbers like "1e400" and | ||
| "1e500" would otherwise both come out as inf and compare equal, losing | ||
| their relative order. Decimal has a much larger exponent range, so use | ||
| it to keep such values distinguishable. A literal "inf"/"nan" spelling | ||
| is left alone as a plain float, since it isn't the result of an overflow. | ||
| """ | ||
| if ret not in (float("inf"), float("-inf")): | ||
| return ret | ||
| if x.strip().lstrip("+-")[:3].lower() in ("inf", "nan"): | ||
|
Owner
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Use the |
||
| return ret | ||
| try: | ||
| return Decimal(x) | ||
| except InvalidOperation: # pragma: no cover | ||
| return ret | ||
|
|
||
|
|
||
| def fast_float( | ||
| x: str, | ||
| key: Callable[[str], str] = lambda x: x, | ||
|
|
@@ -75,7 +97,7 @@ def fast_float( | |
| except TypeError: # pragma: no cover | ||
| return key(x) | ||
| else: | ||
| return nan if ret != ret else ret | ||
| return nan if ret != ret else widen_if_overflowed(x, ret) | ||
| else: | ||
| try: | ||
| return _uni(x, key(x)) if len(x) == 1 else key(x) | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| """Test the fastnumbers compatibility layer.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| from decimal import Decimal | ||
|
|
||
| from natsort.compat.fastnumbers import try_float | ||
|
|
||
|
|
||
| def test_try_float_widens_overflowing_magnitude_to_decimal_example() -> None: | ||
| # "1e400" and "1e500" both overflow float()'s range to inf, so if the | ||
| # backend's raw result were used as-is they would compare equal and | ||
| # lose their real order. This must hold regardless of whether the | ||
| # real fastnumbers package (any supported version) or natsort's own | ||
| # fallback is providing the underlying conversion. | ||
| small, large = try_float(["1e400", "1e500"], map=True) | ||
| assert isinstance(small, Decimal) | ||
| assert isinstance(large, Decimal) | ||
| assert small != large | ||
| assert small < large | ||
|
|
||
|
|
||
| def test_try_float_orders_reversed_overflowing_magnitudes_too() -> None: | ||
| # Same as above, but make sure the result doesn't just happen to match | ||
| # because of the order the values were given in. | ||
| large, small = try_float(["1e500", "1e400"], map=True) | ||
| assert isinstance(small, Decimal) | ||
| assert isinstance(large, Decimal) | ||
| assert small < large | ||
|
|
||
|
|
||
| def test_try_float_still_returns_a_plain_float_for_literal_infinity() -> None: | ||
| # A string that spells out infinity is not the result of an overflow, | ||
| # so it should still come back as a plain float, same as before. | ||
| pos_inf, neg_inf = try_float(["inf", "-inf"], map=True) | ||
| assert pos_inf == float("inf") | ||
| assert neg_inf == float("-inf") | ||
| assert isinstance(pos_inf, float) | ||
| assert isinstance(neg_inf, float) | ||
|
|
||
|
|
||
| def test_try_float_nan_substitution_is_not_treated_as_overflow() -> None: | ||
| # The "nan" substitution value can legitimately be +/-inf (depending on | ||
| # ns.NANLAST), and that is a deliberate substitution, not an overflow, | ||
| # so it must not get widened to Decimal. | ||
| (result,) = try_float(["nan"], map=True, nan=float("-inf")) | ||
| assert result == float("-inf") | ||
| assert isinstance(result, float) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could we pre-define
float("inf")andfloat("-inf")so that we don't have to create this object on each call?