8000 [WIP][SPARK-54276][BUILD] Bump Hadoop 3.4.3 RC0 by pan3793 · Pull Request #54029 · apache/spark · GitHub
[go: up one dir, main page]
Skip to content

[WIP][SPARK-54276][BUILD] Bump Hadoop 3.4.3 RC0 #54029

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
pan3793 wants to merge 7 commits into
base: master
Choose a base branch
from
Open

[WIP][SPARK-54276][BUILD] Bump Hadoop 3.4.3 RC0 #54029

pan3793 wants to merge 7 commits into from
+63 −31

Conversation

@pan3793
Copy link
Member
@pan3793 pan3793 commented Jan 28, 2026 •
edited
Loading

What changes were proposed in this pull request?

https://lists.apache.org/thread/npwwpj45wnt804xmoqgth0s6fhydhxfb

Why are the changes needed?

This release includes HADOOP-19212, which makes UGI work with Java 25.

CHANGELOG: https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.3-RC0/CHANGELOG.md

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass CI. Also verified spark-sql can successfully bootstrap on JDK 25 now

$ java -version
openjdk version "25.0.1" 2025-10-21 LTS
OpenJDK Runtime Environment Temurin-25.0.1+8 (build 25.0.1+8-LTS)
OpenJDK 64-Bit Server VM Temurin-25.0.1+8 (build 25.0.1+8-LTS, mixed mode, sharing)

$ build/sbt -Phive,hive-thriftserver clean package

$ SPARK_PREPEND_CLASSES=true bin/spark-sql
NOTE: SPARK_PREPEND_CLASSES is set, placing locally compiled Spark classes ahead of assembly.
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: package sun.security.action not in java.base
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
26/01/28 17:23:22 WARN Utils: Your hostname, H27212-MAC-01.local, resolves to a loopback address: 127.0.0.1; using 10.242.159.140 instead (on interface en0)
26/01/28 17:23:22 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
26/01/28 17:23:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::arrayBaseOffset has been called by org.apache.spark.unsafe.Platform (file:/Users/chengpan/Projects/apache-spark/common/unsafe/target/scala-2.13/classes/)
WARNING: Please consider reporting this to the maintainers of class org.apache.spark.unsafe.Platform
WARNING: sun.misc.Unsafe::arrayBaseOffset will be removed in a future release
26/01/28 17:23:27 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
26/01/28 17:23:27 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore chengpan@127.0.0.1
26/01/28 17:23:27 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
Spark Web UI available at http://10.242.159.140:4040
Spark master: local[*], Application Id: local-1769592205115
spark-sql (default)> select version();
4.2.0 14557582199659d838bbaa7d7b182e5d92c3b907
Time taken: 1.376 seconds, Fetched 1 row(s)
spark-sql (default)>

Was this patch authored or co-authored using generative AI tooling?

No.

@github-actions
Copy link
github-actions bot commented Jan 28, 2026 •
edited
Loading

JIRA Issue Information

=== Sub-task SPARK-54276 ===
Summary: Upgrade Hadoop to 3.4.3
Assignee: None
Status: Open
Affected: ["4.2.0"]

This comment was automatically generated by GitHub Actions

@github-actions github-actions bot added the CORE label Jan 28, 2026
@dongjoon-hyun
Copy link
Member
dongjoon-hyun commented Jan 28, 2026

Nice! Thank you, @pan3793 .

@pan3793
Copy link
Member Author
pan3793 commented Jan 28, 2026 •
edited
Loading

@steveloughran, seems not lucky, there are no classes in
- hadoop-client-api
- hadoop-client-runtime
- hadoop-client-minicluster

this is actually caused by my local maven repo dirty cache, sorry for making noise, the jars in the staging repo are good.

@steveloughran
Copy link
Contributor
steveloughran commented Jan 28, 2026

@pan3793 sometimes it's good to rm -r all of ~/m2/repository/org/apache/hadoop (or any other project you actively work on). Saves disk space, even if your next few builds are slow.

@steveloughran
Copy link
Contributor
steveloughran commented Jan 28, 2026

@pan3793 thanks for testing this.
@dongjoon-hyun anything you can do to help test would be good too -really hard a hard time getting bits of the rc out. FWIW the maven artifacts are being built on a raspberry pi as that worked more reliably network-wise than EC2 VMs within the cloudera vpn

@pan3793
Copy link
Member Author
pan3793 commented Jan 28, 2026

@steveloughran, thanks for tips, yes, I fixed it by rm -r ~/.m2/repository/org/apache/hadoop/**/3.4.3/.

For integration tests, I don't see any issue with default JDK 17, and I'm trying with JDK 25, so far, no issues are related to Hadoop.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

BUILD CORE DOCS KUBERNETES SQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@pan3793 @dongjoon-hyun @steveloughran
0