Skip to content

Respect secondary digit grouping in locale-aware sorting - #210

Closed
KennyMcSimpson wants to merge 1 commit into
SethMMorton:mainfrom
KennyMcSimpson:fix/locale-secondary-digit-grouping
Closed

KennyMcSimpson wants to merge 1 commit into
SethMMorton:mainfrom
KennyMcSimpson:fix/locale-secondary-digit-grouping

Conversation

@KennyMcSimpson

Copy link
Copy Markdown

Under en_IN.UTF-8, locale-aware sorting currently puts 1,00,000 before 20,000. The locale uses a three-digit rightmost group and two-digit secondary groups ([3, 2, 0]), but the thousands-separator regex only recognizes three-digit groups. It leaves part of the punctuation in the numeric key.

Read the primary and secondary grouping sizes from the standard-library or ICU backend and recognize secondary groups followed by another separator. Keep the existing three-digit grouping behavior and decimal guards. The lookahead stays bounded rather than scanning the remaining grouped number for every separator.

Tests cover integer/float sorting through LOCALENUM and LOCALE, strings containing grouped numbers, signs/exponents, compatibility with three-digit groups, and metadata fallbacks in both locale backends. The real en_IN integration cases skip only when that locale is unavailable; the transform and backend cases are portable.

Validation on Windows:

  • Original full suite: 363 passed. New behavior cases against original source: 10 failed, 4 controls passed.
  • Python 3.10 without fastnumbers and Python 3.13 with fastnumbers 5.2.0: 390 tests passed each, plus 15 README/module doctests and the how-it-works doctest on each version.
  • 10,080 key comparisons against the standard-library locale formatter in en_IN, hi_IN, en_US, de_DE, and fr_FR; grouped integer/float sorting matches locale.atoi/atof.
  • Formatting, strict mypy with the Linux platform target, strict documentation build, distribution builds, and Twine checks passed.
  • Ruff 0.17.2 reports the same 39 existing findings before and after; excluding those existing rule categories passes. Native Windows mypy has the same existing unused-ignore at natsort.py:695 in both versions.

Native PyICU could not be installed in this Windows environment, so its new metadata paths were checked with stubs rather than a native ICU run. The APIs were also checked in the official PyICU 1.0 and 2.17 sources. The repository's existing Ubuntu extras job covers native ICU when CI runs.

AI assistance: prepared with OpenAI Codex. The checks above were run locally; this does not claim human review or successful upstream CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant