Fix tokenization of escaped backslashes in SQL string literals - #837
Open
RamiNoodle733 wants to merge 2 commits into
Open
RamiNoodle733 wants to merge 2 commits into
RamiNoodle733 wants to merge 2 commits into
Conversation
…ndialbrecht#814) The regex pattern for matching single-quoted strings did not handle escaped backslashes (\) properly. This caused strings like '\', '\' to be incorrectly tokenized as a single string with a comma instead of two separate string literals. Changes: - Add \\ to the string literal patterns in SQL_REGEX to match escaped backslashes as valid content within string literals - Add test case for escaped backslash tokenization Fixes andialbrecht#814
There was a problem hiding this comment.
Pull request overview
This PR fixes a lexer bug in sqlparse where SQL single-quoted string literals containing escaped backslashes (e.g. '\\') could incorrectly consume the closing quote and merge subsequent tokens (comma, whitespace, next string). It addresses #814 by updating the string-literal regex so pairs of backslashes are consumed as content, preventing the closing quote from being misinterpreted as escaped.
Changes:
- Extend the
String.SingleandString.Symbolregex patterns to recognize escaped backslashes (\\\\) within quoted strings. - Add a regression test ensuring
SELECT '\\', '\\'tokenizes into two separate string tokens (with punctuation/whitespace between).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
sqlparse/keywords.py |
Updates lexer regex patterns for quoted strings to correctly consume escaped backslashes. |
tests/test_tokenize.py |
Adds a regression test covering correct tokenization of single-quoted strings containing escaped backslashes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #814
Problem
Escaped backslashes in SQL string literals were not being tokenized correctly. For example, the query:
currently results in incorrect tokenization where the two string literals are merged together with the comma.
Root Cause
The regex pattern for matching single-quoted strings did not handle escaped backslashes (
\\) as a valid sequence within the string. When a backslash followed by a quote was encountered, it was interpreted as an escape sequence, but standalone escaped backslashes weren't properly consumed.Solution
Added
\\\\to the string literal patterns to match escaped backslashes (\\) as valid content within string literals. This ensures that:'\\'is tokenized as a single string containing one backslash'\\', '\\'is correctly tokenized as two separate string literalsChanges
sqlparse/keywords.py: Added\\\\toString.SingleandString.Symbolpatternstests/test_tokenize.pyfor escaped backslash tokenizationAll existing tests continue to pass.