8000 HADOOP-19315. Upgrade Apache Avro to 1.11.4 (#7128) by steveloughran · Pull Request #7615 · apache/hadoop · GitHub
[go: up one dir, main page]
Skip to content

HADOOP-19315. Upgrade Apache Avro to 1.11.4 (#7128) #7615

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Apr 21, 2025
Merged

HADOOP-19315. Upgrade Apache Avro to 1.11.4 (#7128) #7615

merged 1 commit into from
Apr 21, 2025

Conversation

steveloughran
Copy link
Contributor
@steveloughran steveloughran commented Apr 15, 2025 •
edited
Loading

HADOOP-19315

  • All field access is now via setter/getter methods
  • To use Avro to marshal Serializable objects, the packages they are in must be declared in the system property "org.apache.avro.SERIALIZABLE_PACKAGES"

This is required to address

This change is not backwards compatible.

Contributed by Dominik Diedrich

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@dom93dd @steveloughran
HADOOP-19315. Upgrade Apache Avro to 1.11.4 (apache#7128)
fb0a65e 
* All field access is now via setter/getter methods
* To use Avro to marshal Serializable objects,
  the packages they are in must be declared in the system property
  "org.apache.avro.SERIALIZABLE_PACKAGES"

This is required to address
- CVE-2024-47561
- CVE-2023-39410

This change is not backwards compatible.

Contributed by Dominik Diedrich
@steveloughran
Copy link
Contributor Author

This is #7128 for branch-3.4; even though it means that 3.4.2 is no longer binary compatible with code running avro 1.9.4 -it is not safe for people to be doing that.

steveloughran
steveloughran commented Apr 15, 2025
View reviewed changes
Copy link
Contributor Author
@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 pending a successful yetus run

ahmarsuhail
ahmarsuhail approved these changes Apr 16, 2025
View reviewed changes
Copy link
Contributor
@ahmarsuhail ahmarsuhail left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 11m 43s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+0 🆗 xmllint 0m 1s xmllint was not available.
+0 🆗 shelldocs 0m 1s Shelldocs was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 5 new or modified test files.
_ branch-3.4 Compile Tests _
+0 🆗 mvndep 2m 5s Maven dependency ordering for branch
+1 💚 mvninstall 33m 18s branch-3.4 passed
+1 💚 compile 16m 55s branch-3.4 passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 compile 16m 41s branch-3.4 passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 checkstyle 4m 46s branch-3.4 passed
+1 💚 mvnsite 21m 16s branch-3.4 passed
+1 💚 javadoc 8m 40s branch-3.4 passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 7m 41s branch-3.4 passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+0 🆗 spotbugs 0m 19s branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
-1 ❌ spotbugs 30m 10s /branch-spotbugs-root-warnings.html root in branch-3.4 has 1 extant spotbugs warnings.
+1 💚 shadedclient 35m 22s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 33s Maven dependency ordering for patch
+1 💚 mvninstall 33m 28s the patch passed
+1 💚 compile 16m 24s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javac 16m 24s the patch passed
+1 💚 compile 16m 36s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 javac 16m 36s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 4m 42s the patch passed
+1 💚 mvnsite 18m 6s the patch passed
+1 💚 shellcheck 0m 0s No new issues.
+1 💚 javadoc 8m 25s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 7m 36s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+0 🆗 spotbugs 0m 18s hadoop-project has no data from spotbugs
+1 💚 shadedclient 67m 20s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 823m 58s /patch-unit-root.txt root in the patch passed.
+1 💚 asflicense 1m 41s The patch does not generate ASF License warnings.
1203m 6s
Reason Tests
Failed junit tests hadoop.fs.contract.router.TestRouterHDFSContractRootDirectory
hadoop.fs.contract.router.TestRouterHDFSContractAppend
hadoop.fs.contract.router.TestRouterHDFSContractMkdirSecure
hadoop.fs.contract.router.web.TestRouterWebHDFSContractConcat
hadoop.fs.contract.router.web.TestRouterWebHDFSContractRename
hadoop.fs.contract.router.TestRouterHDFSContractConcat
hadoop.fs.contract.router.web.TestRouterWebHDFSContractMkdir
hadoop.fs.contract.router.web.TestRouterWebHDFSContractOpen
hadoop.fs.contract.router.TestRouterHDFSContractSeekSecure
hadoop.fs.contract.router.TestRouterHDFSContractDelete
hadoop.fs.contract.router.TestRouterHDFSContractMkdir
hadoop.fs.contract.router.web.TestRouterWebHDFSContractDelete
hadoop.fs.contract.router.web.TestRouterWebHDFSContractRootDirectory
hadoop.fs.contract.router.TestRouterHDFSContractDelegationToken
hadoop.fs.contract.router.TestRouterHDFSContractGetFileStatus
hadoop.fs.contract.router.TestRouterHDFSContractGetFileStatusSecure
hadoop.fs.contract.router.TestRouterHDFSContractCreate
hadoop.fs.contract.router.TestRouterHDFSContractCreateSecure
hadoop.fs.contract.router.TestRouterHDFSContractOpen
hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate
hadoop.fs.contract.router.TestRouterHDFSContractAppendSecure
hadoop.fs.contract.router.TestRouterHDFSContractConcatSecure
hadoop.fs.contract.router.TestRouterHDFSContractSetTimes
hadoop.fs.contract.router.web.TestRouterWebHDFSContractSeek
hadoop.fs.contract.router.web.TestRouterWebHDFSContractAppend
Subsystem Report/Notes
Docker ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7615/1/artifact/out/Dockerfile
GITHUB PR #7615
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint shellcheck shelldocs
uname Linux a18699da8c77 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.4 / fb0a65e
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7615/1/testReport/
Max. process+thread count 3886 (vs. ulimit of 5500)
modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-rumen . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7615/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@steveloughran
Copy link
Contributor Author

failures are all OOM; ignoring.

I am worried about all these failures, FWIW, along with yarn flakiness.

Maybe hdfs tests are leaking miniclusters...we could use the leak detection stuff added recently (the callback when finalize() is called on an open stream) to pick these up

@steveloughran steveloughran merged commit c239f61 into apache:branch-3.4 Apr 21, 2025
0 of 4 checks passed
@slfan1989
Copy link
Contributor

failures are all OOM; ignoring.

I am worried about all these failures, FWIW, along with yarn flakiness.

Maybe hdfs tests are leaking miniclusters...we could use the leak detection stuff added recently (the callback when finalize() is called on an open stream) to pick these up

@steveloughran This week, I will focus on resolving the YARN flakiness test and actively work on upgrading Junit5.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ahmarsuhail ahmarsuhail ahmarsuhail approved these changes

@pan3793 pan3793 pan3793 approved these changes

@slfan1989 slfan1989 slfan1989 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@steveloughran @hadoop-yetus @slfan1989 @ahmarsuhail @pan3793 @dom93dd
0