fix: preserve paragraph breaks as sentence boundaries in normalizer - #525
Merged
Conversation
The normalizer replaced all newlines with spaces unconditionally, merging paragraph-separated sections (headings, chapter titles, body text) into one continuous phrase with no prosodic pause. Convert double-newline paragraph breaks to a period before the newline-to-space pass: if the preceding non-whitespace character is already sentence-ending punctuation the period is omitted to avoid duplication. Single newlines continue to become spaces. Add tests covering the exact failure case from the issue report, no- double-punctuation, single newline, and multiple consecutive blank lines. Fixes remsky#519 Signed-off-by: Christian-Sidak <61099993+Christian-Sidak@users.noreply.github.com>
Owner
|
Glad to merge in for the stated case fix, though I think to fully close #519 , the chunker needs to split at blank lines too for refactor/normalization-classes is a more general fix in the same line if you spot anything |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
\n\nparagraph breaks) with spaces, causing headings and body text to be merged into one continuous phrase with no prosodic pause..,!,?), no extra period is inserted, avoiding double punctuation.Failure case from the issue:
Partial resolution of #519
Test plan
test_paragraph_breaks_preserved_as_sentence_boundary: exact failure case from the issue reporttest_paragraph_break_no_double_punctuation: paragraph already ends with.,!, or?test_single_newline_becomes_space: soft line breaks still collapse to a spacetest_multiple_blank_lines_treated_as_paragraph_break: three or more consecutive newlines produce exactly one period