[Do not merge] HADOOP-19461. JVM GC Metrics supports Generational ZGC pause time #7406

chaijunjie0101 · 2025-02-19T14:12:30Z

Description of PR

https://issues.apache.org/jira/browse/HADOOP-19461
Add 4 metrics to support Generational ZGC
MinorGcCount
MajorGcCount
MinorGcTimeMillis
MajorGcTimeMillis
so we could focous minor or major GC from Hadoop jvm jmx

Test on my cluster(first verion, not include checkstyle fix):

hadoop-yetus · 2025-02-19T15:15:44Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	8m 10s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
-1 ❌	mvninstall	0m 21s	/branch-mvninstall-root.txt	root in trunk failed.
-1 ❌	compile	0m 22s	/branch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt	root in trunk failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌	compile	0m 22s	/branch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06.txt	root in trunk failed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06.
-0 ⚠️	checkstyle	0m 20s	/buildtool-branch-checkstyle-hadoop-common-project_hadoop-common.txt	The patch fails to run checkstyle in hadoop-common
-1 ❌	mvnsite	1m 26s	/branch-mvnsite-hadoop-common-project_hadoop-common.txt	hadoop-common in trunk failed.
-1 ❌	javadoc	0m 22s	/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt	hadoop-common in trunk failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌	javadoc	0m 22s	/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06.txt	hadoop-common in trunk failed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06.
-1 ❌	spotbugs	0m 14s	/branch-spotbugs-hadoop-common-project_hadoop-common.txt	hadoop-common in trunk failed.
+1 💚	shadedclient	3m 28s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
-1 ❌	mvninstall	0m 9s	/patch-mvninstall-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
-1 ❌	compile	0m 48s	/patch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt	root in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌	javac	0m 48s	/patch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt	root in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌	compile	0m 19s	/patch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06.txt	root in the patch failed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06.
-1 ❌	javac	0m 19s	/patch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06.txt	root in the patch failed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06.
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 20s	/buildtool-patch-checkstyle-hadoop-common-project_hadoop-common.txt	The patch fails to run checkstyle in hadoop-common
-1 ❌	mvnsite	0m 56s	/patch-mvnsite-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
+1 💚	javadoc	0m 37s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 21s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
-1 ❌	spotbugs	0m 9s	/patch-spotbugs-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
+1 💚	shadedclient	26m 19s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	0m 11s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
+1 💚	asflicense	0m 26s		The patch does not generate ASF License warnings.
		42m 18s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7406/2/artifact/out/Dockerfile
GITHUB PR	#7406
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux c94bbf019c14 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `404fcfc`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7406/2/testReport/
Max. process+thread count	545 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7406/2/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-02-19T16:16:14Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	24m 8s		trunk passed
+1 💚	compile	9m 29s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	8m 9s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	0m 44s		trunk passed
+1 💚	mvnsite	0m 57s		trunk passed
+1 💚	javadoc	0m 46s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 31s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 31s		trunk passed
+1 💚	shadedclient	21m 19s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 30s		the patch passed
+1 💚	compile	7m 45s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	7m 45s		the patch passed
+1 💚	compile	7m 19s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	7m 19s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 35s		the patch passed
+1 💚	mvnsite	0m 57s		the patch passed
+1 💚	javadoc	0m 41s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 33s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
-1 ❌	spotbugs	1m 34s	/new-spotbugs-hadoop-common-project_hadoop-common.html	hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	21m 18s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	12m 8s		hadoop-common in the patch passed.
+1 💚	asflicense	0m 40s		The patch does not generate ASF License warnings.
		122m 43s

Reason	Tests
SpotBugs	module:hadoop-common-project/hadoop-common
	Null passed for non-null parameter of getGcInfo(String) in org.apache.hadoop.metrics2.source.JvmMetrics.getGcUsage(MetricsRecordBuilder) Method invoked at JvmMetrics.java:of getGcInfo(String) in org.apache.hadoop.metrics2.source.JvmMetrics.getGcUsage(MetricsRecordBuilder) Method invoked at JvmMetrics.java:[line 198]

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7406/1/artifact/out/Dockerfile
GITHUB PR	#7406
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 7db574b15f30 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `3c3852e`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7406/1/testReport/
Max. process+thread count	2110 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7406/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-02-19T18:06:23Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	11m 34s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 24s		trunk passed
+1 💚	compile	15m 9s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	13m 27s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	1m 16s		trunk passed
+1 💚	mvnsite	1m 36s		trunk passed
+1 💚	javadoc	1m 16s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 51s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	2m 28s		trunk passed
+1 💚	shadedclient	36m 5s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 54s		the patch passed
+1 💚	compile	14m 24s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	14m 24s		the patch passed
+1 💚	compile	13m 48s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	13m 48s		the patch passed
+1 💚	blanks	0m 1s		The patch has no blanks issues.
+1 💚	checkstyle	1m 12s		the patch passed
+1 💚	mvnsite	1m 33s		the patch passed
+1 💚	javadoc	1m 12s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 54s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
-1 ❌	spotbugs	2m 42s	/new-spotbugs-hadoop-common-project_hadoop-common.html	hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	36m 28s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	14m 54s		hadoop-common in the patch passed.
+1 💚	asflicense	1m 2s		The patch does not generate ASF License warnings.
		209m 9s

Reason	Tests
SpotBugs	module:hadoop-common-project/hadoop-common
	Null passed for non-null parameter of getGcInfo(String) in org.apache.hadoop.metrics2.source.JvmMetrics.getGcUsage(MetricsRecordBuilder) Method invoked at JvmMetrics.java:of getGcInfo(String) in org.apache.hadoop.metrics2.source.JvmMetrics.getGcUsage(MetricsRecordBuilder) Method invoked at JvmMetrics.java:[line 198]

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7406/3/artifact/out/Dockerfile
GITHUB PR	#7406
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 49f469500d5a 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `404fcfc`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7406/3/testReport/
Max. process+thread count	3149 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7406/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-02-19T18:29:43Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	24m 48s		trunk passed
+1 💚	compile	9m 9s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	8m 1s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	0m 41s		trunk passed
+1 💚	mvnsite	0m 57s		trunk passed
+1 💚	javadoc	0m 40s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 27s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 30s		trunk passed
+1 💚	shadedclient	22m 24s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 33s		the patch passed
+1 💚	compile	8m 41s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	8m 41s		the patch passed
+1 💚	compile	8m 8s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	8m 8s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 37s		the patch passed
+1 💚	mvnsite	0m 54s		the patch passed
+1 💚	javadoc	0m 38s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 28s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 30s		the patch passed
+1 💚	shadedclient	22m 39s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	12m 24s		hadoop-common in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		126m 40s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7406/4/artifact/out/Dockerfile
GITHUB PR	#7406
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux cb3e354d7ab8 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `114e0b1`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7406/4/testReport/
Max. process+thread count	3148 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7406/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-02-19T18:32:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	24m 47s		trunk passed
+1 💚	compile	9m 5s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	7m 52s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	0m 38s		trunk passed
+1 💚	mvnsite	0m 58s		trunk passed
+1 💚	javadoc	0m 41s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 26s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 25s		trunk passed
+1 💚	shadedclient	22m 45s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 46s		the patch passed
+1 💚	compile	8m 44s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	8m 44s		the patch passed
+1 💚	compile	7m 43s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	7m 43s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 32s		the patch passed
+1 💚	mvnsite	0m 53s		the patch passed
+1 💚	javadoc	0m 37s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 27s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 25s		the patch passed
+1 💚	shadedclient	22m 24s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	12m 34s		hadoop-common in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		126m 27s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7406/5/artifact/out/Dockerfile
GITHUB PR	#7406
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux d96ec869baa2 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `272ff74`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7406/5/testReport/
Max. process+thread count	3149 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7406/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2025-02-20T04:45:13Z

@pan3793 Could you please review this PR? Thank you very much!

chaijunjie0101 · 2025-02-20T06:48:43Z

@pan3793 Could you please review this PR? Thank you very much!

thank you all, please review this PR, if anyone have time

pan3793 · 2025-02-24T13:20:31Z

@chaijunjie0101 Could you provide the JVM info of your tested cluster? e.g. the output of java -version

pan3793

Sorry, the PR title is misleading, I think the current trunk code could report "Total GC Count" and "Total GC time in milliseconds" correctly for Generational ZGC without this patch, and the implementation does not look correct to me.

Minor and Major GC are quite common concepts for GC, not specific to ZGC, for example, the Spark docs say:

.MinorGCCount - Total minor GC count. For example, the garbage collector is one of Copy, PS Scavenge, ParNew, G1 Young Generation and so on.

.MajorGCCount - Total major GC count. For example, the garbage collector is one of MarkSweepCompact, PS MarkSweep, ConcurrentMarkSweep, G1 Old Generation and so on.

chaijunjie0101 · 2025-02-25T01:55:12Z

Sorry, the PR title is misleading, I think the current trunk code could report "Total GC Count" and "Total GC time in milliseconds" correctly for Generational ZGC without this patch, and the implementation does not look correct to me.

Minor and Major GC are quite common concepts for GC, not specific to ZGC, for example, the Spark docs say:

.MinorGCCount - Total minor GC count. For example, the garbage collector is one of Copy, PS Scavenge, ParNew, G1 Young Generation and so on.
.MajorGCCount - Total major GC count. For example, the garbage collector is one of MarkSweepCompact, PS MarkSweep, ConcurrentMarkSweep, G1 Old Generation and so on.

hi, thanks reviwing~
I used JDK 21(https://gitee.com/openeuler/bishengjdk-21), you are right, but for different GC, has different bean name for minor/Major GC, I just found Generational ZGC use MajorGC/MinorGC directly, but for example CMS GC:

jmx["java.lang:type=GarbageCollector,name=ParNew","CollectionCount"]
jmx["java.lang:type=GarbageCollector,name=ParNew","CollectionTime"]
jmx["java.lang:type=GarbageCollector,name=ConcurrentMarkSweep","CollectionCount"]
jmx["java.lang:type=GarbageCollector,name=ConcurrentMarkSweep","CollectionTime"]

ParNew likely Minor GC, ConcurrentMarkSweep likely Major GC, but I do not why they not use same bean name like "MajorGC", maybe some diiferent here?

when I test on JDK21, I just want get a more refined GC time to adjust Youg/Old memory size, since ZGC provides them(Major/Minor), we could fetch them...

you means we no need it, or need to support all GC's "MajorGC/MinorGC", but it may be difficult, the JDK not define a common metric for them?

pan3793 · 2025-02-25T03:37:49Z

you means ...

Don't do things that may confuse users.

your current PR title "JVM GC Metrics supports Generational ZGC pause time" makes me think that the current trunk code does not support "Generational ZGC pause time", it is incorrect, a better title might be "JVM GC Metrics supports fine-grained metrics for Generational ZGC"
a generic metrics name deserves a generic implementation. For example, both micrometer (which is used by Spring Boot) and Spark classify the JVM GC metrics into a few categories, for all common GC algorithms. if this is hard for you, please make the metrics name accurate, for example, "ZGCMajorPausesTimeMillis"

chaijunjie0101 · 2025-02-25T03:51:22Z

you means ...

Don't do things that may confuse users.

your current PR title "JVM GC Metrics supports Generational ZGC pause time" makes me think that the current trunk code does not support "Generational ZGC pause time", it is incorrect, a better title might be "JVM GC Metrics supports fine-grained metrics for Generational ZGC"

a generic metrics name deserves a generic implementation. For example, both micrometer (which is used by Spring Boot) and Spark classify the JVM GC metrics into a few categories, for all common GC algorithms. if this is hard for you, please make the metrics name accurate, for example, "ZGCMajorPausesTimeMillis"

thank you give me these examples, I will try to combine and support all GCs

github-actions bot added trunk Common labels Feb 19, 2025

chaijunjie0101 force-pushed the HADOOP-19461 branch 2 times, most recently from 468ffe9 to 404fcfc Compare February 19, 2025 14:35

chaijunjie0101 force-pushed the HADOOP-19461 branch from 404fcfc to 114e0b1 Compare February 19, 2025 16:22

HADOOP-19461. JVM GC Metrics supports Generational ZGC pause time

272ff74

chaijunjie0101 force-pushed the HADOOP-19461 branch from 114e0b1 to 272ff74 Compare February 19, 2025 16:24

zeekling approved these changes Feb 22, 2025

View reviewed changes

pan3793 suggested changes Feb 24, 2025

View reviewed changes

chaijunjie0101 changed the title ~~HADOOP-19461. JVM GC Metrics supports Generational ZGC pause time~~ [Do not merge] HADOOP-19461. JVM GC Metrics supports Generational ZGC pause time Feb 25, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Do not merge] HADOOP-19461. JVM GC Metrics supports Generational ZGC pause time #7406

[Do not merge] HADOOP-19461. JVM GC Metrics supports Generational ZGC pause time #7406

[Do not merge] HADOOP-19461. JVM GC Metrics supports Generational ZGC pause time #7406

Are you sure you want to change the base?

[Do not merge] HADOOP-19461. JVM GC Metrics supports Generational ZGC pause time #7406

Conversation

Description of PR

Choose a reason for hiding this comment