Skip to content

Normalize locale thousands separators consistently with input - #205

Merged
SethMMorton merged 1 commit into
SethMMorton:mainfrom
kokotatan:fix-normalized-locale-separator
Oct 16, 2026
Merged

SethMMorton merged 1 commit into
SethMMorton:mainfrom
kokotatan:fix-normalized-locale-separator

Conversation

@kokotatan

Copy link
Copy Markdown
Contributor

Combining ns.LOCALENUM with ns.COMPATIBILITYNORMALIZE misorders numbers containing non-breaking-space thousands separators. For example, under fr_FR.UTF-8 on Windows:

>>> values = ['999', '1\u202f000']
>>> natsorted(values, alg=ns.LOCALENUM | ns.COMPATIBILITYNORMALIZE)
['1\u202f000', '999']  # currently

Input is normalized to NFKD before locale numeric processing, which turns U+202F (and U+00A0) into an ordinary space. The thousands-separator regex still looks for the original character, so the key for 1000 becomes ('', 1, ' ', 0).

Normalize the configured thousands separator using the same existing normalization factory before constructing the regex. This produces ('', 1000) and the expected numeric order, preserving the original output strings. It adds no per-input normalization work. With compatibility normalization selected, ordinary spaces and these non-breaking spaces are intentionally equivalent; both can therefore be recognized as grouping separators in these locales.

The documented flags support combining options, including locale numeric handling and NFKD normalization. Tests cover both separator characters, integer/float parsing, and default/compatibility normalization. The locale symbol getters are patched for portable tests; the example above was also reproduced with the actual installed French locale before and after the fix.

Validation on Windows / Python 3.12:

  • Unmodified full suite: 355 passed.
  • New cases before the fix: 4 failed, 4 passed.
  • tox -e py312: 363 passed, 94% overall coverage, plus README/examples and module doctests (15 items) and the how-it-works doctest.
  • tox -e docs, repository-wide Ruff format check, distribution builds and Twine checks: passed.
  • Strict mypy with the Linux platform target: 32 files passed. Native Windows reports one pre-existing unused type: ignore; the same failure was reproduced on the unmodified source.
  • Ruff lint reports 38 pre-existing findings; comparison against unmodified source confirmed identical file/code/message sets. No unrelated lint cleanup is included.

AI assistance: developed with OpenAI Codex and checked through independent Claude Opus and Codex AI reviews. This is not a claim of human review.

@codecov

codecov Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.96%. Comparing base (607eeee) to head (ad0124d).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #205      +/-   ##
==========================================
+ Coverage   97.80%   97.96%   +0.16%     
==========================================
  Files          10       10              
  Lines         637      638       +1     
==========================================
+ Hits          623      625       +2     
+ Misses         14       13       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@SethMMorton
SethMMorton merged commit e9a3a2a into SethMMorton:main Oct 16, 2026
15 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants