Skip to content

Build: Fix Spark publication and CI configuration #18042

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kevinjqliu merged 3 commits into from
Sep 12, 2026
Merged

Build: Fix Spark publication and CI configuration #18042

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
baa09f2
Build: Fix Spark publication and CI configuration
manuzhang Sep 10, 2026
029a0ec
Build: Remove outdated Spark Java 21 support comment
manuzhang Sep 11, 2026
aed92f5
Build: Restore binary staging script
manuzhang Sep 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/delta-conversion-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ on:
- '.github/workflows/flink-ci.yml'
- '.github/workflows/hive-ci.yml'
- '.github/workflows/java-ci.yml'
- '.github/workflows/jmh-benchmarks-ci.yml'
- '.github/workflows/jmh-benchmarks.yml'
- '.github/workflows/kafka-connect-ci.yml'
- '.github/workflows/cve-scan.yml'
- '.github/workflows/labeler.yml'
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/flink-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ on:
- '.github/workflows/docs-ci.yml'
- '.github/workflows/hive-ci.yml'
- '.github/workflows/java-ci.yml'
- '.github/workflows/jmh-benchmarks-ci.yml'
- '.github/workflows/jmh-benchmarks.yml'
- '.github/workflows/kafka-connect-ci.yml'
- '.github/workflows/cve-scan.yml'
- '.github/workflows/labeler.yml'
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/hive-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ on:
- '.github/workflows/docs-ci.yml'
- '.github/workflows/flink-ci.yml'
- '.github/workflows/java-ci.yml'
- '.github/workflows/jmh-benchmarks-ci.yml'
- '.github/workflows/jmh-benchmarks.yml'
- '.github/workflows/kafka-connect-ci.yml'
- '.github/workflows/cve-scan.yml'
- '.github/workflows/labeler.yml'
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/java-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ on:
- '.github/workflows/docs-ci.yml'
- '.github/workflows/flink-ci.yml'
- '.github/workflows/hive-ci.yml'
- '.github/workflows/jmh-benchmarks-ci.yml'
- '.github/workflows/jmh-benchmarks.yml'
- '.github/workflows/kafka-connect-ci.yml'
- '.github/workflows/cve-scan.yml'
- '.github/workflows/labeler.yml'
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/kafka-connect-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ on:
- '.github/workflows/flink-ci.yml'
- '.github/workflows/hive-ci.yml'
- '.github/workflows/java-ci.yml'
- '.github/workflows/jmh-benchmarks-ci.yml'
- '.github/workflows/jmh-benchmarks.yml'
- '.github/workflows/labeler.yml'
- '.github/workflows/license-check.yml'
- '.github/workflows/open-api.yml'
Expand Down
6 changes: 5 additions & 1 deletion .github/workflows/publish-snapshot.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,4 +53,8 @@ jobs:
run: |
./gradlew printVersion
./gradlew -DallModules publishApachePublicationToMavenRepository -PmavenUser="$NEXUS_USER" -PmavenPassword="$NEXUS_PW"
./gradlew -DflinkVersions= -DsparkVersions=3.5,4.0 -DscalaVersion=2.13 -DkafkaVersions=3 publishApachePublicationToMavenRepository -PmavenUser="$NEXUS_USER" -PmavenPassword="$NEXUS_PW"
./gradlew -DflinkVersions= -DsparkVersions=3.5 -DscalaVersion=2.13 -DkafkaVersions= \

@kevinjqliu kevinjqliu Sep 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, so the first ./gradlew -DallModules command publishes all the packages and uses the default 2.12 scala.

Spark 3.5 is the odd one out, it needs to publish package for both 2.12 and 2.13

iceberg/build.gradle

Lines 1257 to 1263 in cff60bc

// The Iceberg-Build builds for only one Scala version at a time, so the BOM would also
// only contain artifacts for that single Scala version. The following code ensures that
// the BOM references the artifacts for all Scala versions.
def sparkScalaPattern = ~"(.*)-([0-9][.][0-9]+)_([0-9][.][0-9]+)"
def sparkScalaVersions = [
"3.5": ["2.12", "2.13"],
]

So this second command publishes explicitly for Spark 3.5 with 2.13

@manuzhang manuzhang Sep 12, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first command publishes with default 2.12 for 3.5 and 2.13 for 4+.

:iceberg-spark:iceberg-spark-3.5_2.13:publishApachePublicationToMavenRepository \
:iceberg-spark:iceberg-spark-extensions-3.5_2.13:publishApachePublicationToMavenRepository \
:iceberg-spark:iceberg-spark-runtime-3.5_2.13:publishApachePublicationToMavenRepository \
-PmavenUser="$NEXUS_USER" -PmavenPassword="$NEXUS_PW"
6 changes: 3 additions & 3 deletions .github/workflows/spark-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ on:
- '.github/workflows/flink-ci.yml'
- '.github/workflows/hive-ci.yml'
- '.github/workflows/java-ci.yml'
- '.github/workflows/jmh-benchmarks-ci.yml'
- '.github/workflows/jmh-benchmarks.yml'
- '.github/workflows/kafka-connect-ci.yml'
- '.github/workflows/cve-scan.yml'
- '.github/workflows/labeler.yml'
Expand Down Expand Up @@ -81,6 +81,7 @@ jobs:
# Keep matrix <= 20 jobs; exceeding it queues a second wave and slows CI.
max-parallel: 20
matrix:
# Spark 3.5 is the first version not failing on Java 21 (https://issues.apache.org/jira/browse/SPARK-42369)
jvm: [17, 21]
event_name: ['${{ github.event_name }}']
spark: ['3.5', '4.0', '4.1', '4.2']
Expand All @@ -89,8 +90,7 @@ jobs:
# so they run as concurrent jobs rather than serially in one job.
tests: [core, extensions]
exclude:
# Spark 3.5 is the first version not failing on Java 21 (https://issues.apache.org/jira/browse/SPARK-42369)
# Full Java 21 support is coming in Spark 4 (https://issues.apache.org/jira/browse/SPARK-43831)
# Spark 4.0+ only supports Scala 2.13.
- spark: '4.0'
scala: '2.12'
- spark: '4.1'
Expand Down
3 changes: 3 additions & 0 deletions build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1258,8 +1258,11 @@ project(':iceberg-bom') {
// only contain artifacts for that single Scala version. The following code ensures that
// the BOM references the artifacts for all Scala versions.
def sparkScalaPattern = ~"(.*)-([0-9][.][0-9]+)_([0-9][.][0-9]+)"
// Only include Spark versions published in binary releases.
def sparkScalaVersions = [
"3.5": ["2.12", "2.13"],
"4.0": ["2.13"],
"4.1": ["2.13"],
Comment on lines +1264 to +1265

@kevinjqliu kevinjqliu Sep 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a separate BOM issue.

]
rootProject.allprojects.forEach {
// Do not include ':iceberg-spark', the bom itself and the root project.
Expand Down
Loading