Skip to content

[MINOR][CORE] Avoid O(n^2) stack walk in CallerInfo.inBloomFilterStatFunctionCall#12477

Open
LuciferYang wants to merge 1 commit into
apache:mainfrom
LuciferYang:gluten-callerinfo-bloomfilter-scan
Open

[MINOR][CORE] Avoid O(n^2) stack walk in CallerInfo.inBloomFilterStatFunctionCall#12477
LuciferYang wants to merge 1 commit into
apache:mainfrom
LuciferYang:gluten-callerinfo-bloomfilter-scan

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

CallerInfo.inBloomFilterStatFunctionCall used a nested stack.exists inside the outer predicate:

stack.exists(
  _.getClassName.equals("org.apache.spark.sql.DataFrameStatFunctions")
    && stack.exists(_.getMethodName.equals("bloomFilter")))

The inner stack.exists does not depend on the outer element and re-walks the whole stack for every candidate frame, making the check O(n^2) in stack depth. Split it into two independent single-pass checks that combine with &&; the resulting predicate is "the stack contains both a DataFrameStatFunctions frame and a bloomFilter frame," identical to the original.

Why are the changes needed?

CallerInfo.create() is invoked once per columnar rule application, so this predicate runs on every planned query. The change is a straight refactor with no behavior difference; the value is that the stack scan drops from quadratic to linear.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

mvn -pl gluten-core -Pspark-3.5 spotless:check and compile pass. No behavior change, no new test.

Copilot AI review requested due to automatic review settings July 8, 2026 13:28
@github-actions github-actions Bot added the CORE works for Gluten Core label Jul 8, 2026
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors CallerInfo.inBloomFilterStatFunctionCall in gluten-core to avoid a nested stack.exists that caused an O(n²) walk over the thread stack trace, replacing it with two independent O(n) existence checks while preserving the original (stack-level) semantics.

Changes:

  • Replaced nested stack.exists usage with two independent exists checks combined via &&.
  • Added an in-code comment explaining the prior complexity issue and why the refactor is semantically equivalent.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@LuciferYang

Copy link
Copy Markdown
Contributor Author

cc @jackylee-ch , a minor fix

@jackylee-ch jackylee-ch left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CORE works for Gluten Core

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants