draft: Parallelize petclinic & insecure-bank benchmarks #8982

ddyurchenko · 2025-06-13T15:17:55Z

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

This reverts commit a8c774e

pr-commenter · 2025-06-17T17:44:19Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	ddyurchenko/parallelize-load-tests
git_commit_date	1750176286	1750240216
git_commit_sha	`72f45a8`	`407b1a2`
release_version	1.50.0-SNAPSHOT~72f45a8563	1.50.0-SNAPSHOT~407b1a2cd9

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1750242499	1750242499
ci_job_id	987651333	987651333
ci_pipeline_id	68085722	68085722
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-jp348zax-project-304-concurrent-0-cxxuwfm9 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-jp348zax-project-304-concurrent-0-cxxuwfm9 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 58 metrics, 13 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~407b1a2cd9, baseline=1.50.0-SNAPSHOT~72f45a8563

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.026 s) : 0, 1026115
Total [baseline] (8.603 s) : 0, 8603364
Agent [candidate] (1.029 s) : 0, 1029072
Total [candidate] (8.542 s) : 0, 8541940
section iast
Agent [baseline] (1.154 s) : 0, 1153706
Total [baseline] (9.217 s) : 0, 9216700
Agent [candidate] (1.154 s) : 0, 1153916
Total [candidate] (9.188 s) : 0, 9188022
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.162 s) : 0, 1161903
Total [baseline] (9.185 s) : 0, 9184759
Agent [candidate] (1.165 s) : 0, 1165030
Total [candidate] (9.223 s) : 0, 9223322
section iast_TELEMETRY_OFF
Agent [baseline] (1.157 s) : 0, 1156676
Total [baseline] (9.263 s) : 0, 9263254
Agent [candidate] (1.155 s) : 0, 1155487
Total [candidate] (9.212 s) : 0, 9211787

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.026 s	-
Agent	iast	1.154 s	127.591 ms (12.4%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.162 s	135.788 ms (13.2%)
Agent	iast_TELEMETRY_OFF	1.157 s	130.561 ms (12.7%)
Total	tracing	8.603 s	-
Total	iast	9.217 s	613.336 ms (7.1%)
Total	iast_HARDCODED_SECRET_DISABLED	9.185 s	581.395 ms (6.8%)
Total	iast_TELEMETRY_OFF	9.263 s	659.89 ms (7.7%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.029 s	-
Agent	iast	1.154 s	124.845 ms (12.1%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.165 s	135.958 ms (13.2%)
Agent	iast_TELEMETRY_OFF	1.155 s	126.416 ms (12.3%)
Total	tracing	8.542 s	-
Total	iast	9.188 s	646.082 ms (7.6%)
Total	iast_HARDCODED_SECRET_DISABLED	9.223 s	681.382 ms (8.0%)
Total	iast_TELEMETRY_OFF	9.212 s	669.847 ms (7.8%)

gantt
    title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~407b1a2cd9, baseline=1.50.0-SNAPSHOT~72f45a8563

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (683.329 ms) : 0, 683329
BytebuddyAgent [candidate] (685.194 ms) : 0, 685194
GlobalTracer [baseline] (240.457 ms) : 0, 240457
GlobalTracer [candidate] (241.666 ms) : 0, 241666
AppSec [baseline] (59.807 ms) : 0, 59807
AppSec [candidate] (59.931 ms) : 0, 59931
Debugger [baseline] (6.163 ms) : 0, 6163
Debugger [candidate] (6.095 ms) : 0, 6095
Remote Config [baseline] (749.264 µs) : 0, 749
Remote Config [candidate] (751.149 µs) : 0, 751
Telemetry [baseline] (12.153 ms) : 0, 12153
Telemetry [candidate] (11.939 ms) : 0, 11939
section iast
BytebuddyAgent [baseline] (803.548 ms) : 0, 803548
BytebuddyAgent [candidate] (803.45 ms) : 0, 803450
GlobalTracer [baseline] (231.048 ms) : 0, 231048
GlobalTracer [candidate] (231.068 ms) : 0, 231068
AppSec [baseline] (53.809 ms) : 0, 53809
AppSec [candidate] (54.951 ms) : 0, 54951
Debugger [baseline] (5.945 ms) : 0, 5945
Debugger [candidate] (5.917 ms) : 0, 5917
Remote Config [baseline] (579.632 µs) : 0, 580
Remote Config [candidate] (582.958 µs) : 0, 583
Telemetry [baseline] (7.865 ms) : 0, 7865
Telemetry [candidate] (7.836 ms) : 0, 7836
IAST [baseline] (27.436 ms) : 0, 27436
IAST [candidate] (26.707 ms) : 0, 26707
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (808.45 ms) : 0, 808450
BytebuddyAgent [candidate] (810.841 ms) : 0, 810841
GlobalTracer [baseline] (232.867 ms) : 0, 232867
GlobalTracer [candidate] (233.944 ms) : 0, 233944
AppSec [baseline] (55.189 ms) : 0, 55189
AppSec [candidate] (57.325 ms) : 0, 57325
Debugger [baseline] (6.061 ms) : 0, 6061
Debugger [candidate] (6.013 ms) : 0, 6013
Remote Config [baseline] (598.386 µs) : 0, 598
Remote Config [candidate] (581.622 µs) : 0, 582
Telemetry [baseline] (8.043 ms) : 0, 8043
Telemetry [candidate] (7.962 ms) : 0, 7962
IAST [baseline] (27.173 ms) : 0, 27173
IAST [candidate] (24.802 ms) : 0, 24802
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (803.762 ms) : 0, 803762
BytebuddyAgent [candidate] (803.342 ms) : 0, 803342
GlobalTracer [baseline] (233.132 ms) : 0, 233132
GlobalTracer [candidate] (232.628 ms) : 0, 232628
AppSec [baseline] (52.724 ms) : 0, 52724
AppSec [candidate] (53.298 ms) : 0, 53298
Debugger [baseline] (6.102 ms) : 0, 6102
Debugger [candidate] (6.109 ms) : 0, 6109
Remote Config [baseline] (612.16 µs) : 0, 612
Remote Config [candidate] (601.987 µs) : 0, 602
Telemetry [baseline] (7.917 ms) : 0, 7917
Telemetry [candidate] (7.958 ms) : 0, 7958
IAST [baseline] (29.033 ms) : 0, 29033
IAST [candidate] (28.007 ms) : 0, 28007

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~407b1a2cd9, baseline=1.50.0-SNAPSHOT~72f45a8563

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.026 s) : 0, 1025727
Total [baseline] (10.698 s) : 0, 10697889
Agent [candidate] (1.028 s) : 0, 1027912
Total [candidate] (10.712 s) : 0, 10711565
section appsec
Agent [baseline] (1.179 s) : 0, 1178957
Total [baseline] (10.73 s) : 0, 10729855
Agent [candidate] (1.18 s) : 0, 1179749
Total [candidate] (10.755 s) : 0, 10754921
section iast
Agent [baseline] (1.16 s) : 0, 1159556
Total [baseline] (10.92 s) : 0, 10920491
Agent [candidate] (1.162 s) : 0, 1162292
Total [candidate] (10.894 s) : 0, 10893959
section profiling
Agent [baseline] (1.274 s) : 0, 1273770
Total [baseline] (11.027 s) : 0, 11027460
Agent [candidate] (1.276 s) : 0, 1275941
Total [candidate] (10.953 s) : 0, 10953177

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.026 s	-
Agent	appsec	1.179 s	153.23 ms (14.9%)
Agent	iast	1.16 s	133.829 ms (13.0%)
Agent	profiling	1.274 s	248.042 ms (24.2%)
Total	tracing	10.698 s	-
Total	appsec	10.73 s	31.966 ms (0.3%)
Total	iast	10.92 s	222.602 ms (2.1%)
Total	profiling	11.027 s	329.571 ms (3.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.028 s	-
Agent	appsec	1.18 s	151.837 ms (14.8%)
Agent	iast	1.162 s	134.38 ms (13.1%)
Agent	profiling	1.276 s	248.029 ms (24.1%)
Total	tracing	10.712 s	-
Total	appsec	10.755 s	43.356 ms (0.4%)
Total	iast	10.894 s	182.394 ms (1.7%)
Total	profiling	10.953 s	241.612 ms (2.3%)

gantt
    title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~407b1a2cd9, baseline=1.50.0-SNAPSHOT~72f45a8563

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (682.997 ms) : 0, 682997
BytebuddyAgent [candidate] (684.115 ms) : 0, 684115
GlobalTracer [baseline] (240.554 ms) : 0, 240554
GlobalTracer [candidate] (241.132 ms) : 0, 241132
AppSec [baseline] (58.384 ms) : 0, 58384
AppSec [candidate] (60.386 ms) : 0, 60386
Debugger [baseline] (8.344 ms) : 0, 8344
Debugger [candidate] (6.111 ms) : 0, 6111
Remote Config [baseline] (737.987 µs) : 0, 738
Remote Config [candidate] (745.506 µs) : 0, 746
Telemetry [baseline] (11.35 ms) : 0, 11350
Telemetry [candidate] (12.051 ms) : 0, 12051
section appsec
BytebuddyAgent [baseline] (706.048 ms) : 0, 706048
BytebuddyAgent [candidate] (706.393 ms) : 0, 706393
GlobalTracer [baseline] (234.873 ms) : 0, 234873
GlobalTracer [candidate] (234.873 ms) : 0, 234873
IAST [baseline] (21.732 ms) : 0, 21732
IAST [candidate] (21.79 ms) : 0, 21790
AppSec [baseline] (179.086 ms) : 0, 179086
AppSec [candidate] (179.32 ms) : 0, 179320
Debugger [baseline] (5.904 ms) : 0, 5904
Debugger [candidate] (5.905 ms) : 0, 5905
Remote Config [baseline] (613.069 µs) : 0, 613
Remote Config [candidate] (610.86 µs) : 0, 611
Telemetry [baseline] (7.258 ms) : 0, 7258
Telemetry [candidate] (7.338 ms) : 0, 7338
section iast
BytebuddyAgent [baseline] (807.544 ms) : 0, 807544
BytebuddyAgent [candidate] (809.305 ms) : 0, 809305
GlobalTracer [baseline] (232.1 ms) : 0, 232100
GlobalTracer [candidate] (232.548 ms) : 0, 232548
IAST [baseline] (29.331 ms) : 0, 29331
IAST [candidate] (27.753 ms) : 0, 27753
AppSec [baseline] (51.697 ms) : 0, 51697
AppSec [candidate] (53.443 ms) : 0, 53443
Debugger [baseline] (6.014 ms) : 0, 6014
Debugger [candidate] (6.081 ms) : 0, 6081
Remote Config [baseline] (584.446 µs) : 0, 584
Remote Config [candidate] (591.148 µs) : 0, 591
Telemetry [baseline] (7.984 ms) : 0, 7984
Telemetry [candidate] (8.043 ms) : 0, 8043
section profiling
BytebuddyAgent [baseline] (675.62 ms) : 0, 675620
BytebuddyAgent [candidate] (675.193 ms) : 0, 675193
GlobalTracer [baseline] (361.326 ms) : 0, 361326
GlobalTracer [candidate] (360.947 ms) : 0, 360947
AppSec [baseline] (64.448 ms) : 0, 64448
AppSec [candidate] (64.817 ms) : 0, 64817
Debugger [baseline] 
9E88
(6.148 ms) : 0, 6148
Debugger [candidate] (6.191 ms) : 0, 6191
Remote Config [baseline] (646.522 µs) : 0, 647
Remote Config [candidate] (666.492 µs) : 0, 666
Telemetry [baseline] (8.2 ms) : 0, 8200
Telemetry [candidate] (8.247 ms) : 0, 8247
ProfilingAgent [baseline] (106.574 ms) : 0, 106574
ProfilingAgent [candidate] (108.986 ms) : 0, 108986
Profiling [baseline] (106.599 ms) : 0, 106599
Profiling [candidate] (109.011 ms) : 0, 109011

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	ddyurchenko/parallelize-load-tests
git_commit_date	1750176286	1750240216
git_commit_sha	`72f45a8`	`407b1a2`
release_version	1.50.0-SNAPSHOT~72f45a8563	1.50.0-SNAPSHOT~407b1a2cd9

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1750241857	1750241857
ci_job_id	987651335	987651335
ci_pipeline_id	68085722	68085722
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-2yzfxtfh-project-304-concurrent-0-8j6pppao 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-2yzfxtfh-project-304-concurrent-0-8j6pppao 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 1 performance regressions! Performance is the same for 13 metrics, 15 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:insecure-bank:profiling:high_load	worse [+296.337µs; +654.004µs] or [+2.924%; +6.452%]	unstable [-75.273op/s; +34.585op/s] or [-16.426%; +7.547%]	10.612ms	437.906op/s	10.136ms	458.250op/s
scenario:load:petclinic:profiling:high_load	better [-2.213ms; -1.336ms] or [-4.710%; -2.844%]	unstable [-3.239op/s; +11.014op/s] or [-3.251%; +11.056%]	45.204ms	103.513op/s	46.978ms	99.625op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~407b1a2cd9, baseline=1.50.0-SNAPSHOT~72f45a8563
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.816 ms) : 4757, 4874
.   : milestone, 4816,
iast (10.571 ms) : 10383, 10760
.   : milestone, 10571,
iast_FULL (15.979 ms) : 15666, 16293
.   : milestone, 15979,
iast_GLOBAL (11.262 ms) : 11053, 11471
.   : milestone, 11262,
iast_HARDCODED_SECRET_DISABLED (10.071 ms) : 9891, 10251
.   : milestone, 10071,
iast_INACTIVE (8.829 ms) : 8685, 8973
.   : milestone, 8829,
iast_TELEMETRY_OFF (9.984 ms) : 9819, 10150
.   : milestone, 9984,
profiling (10.136 ms) : 9974, 10298
.   : milestone, 10136,
tracing (8.491 ms) : 8364, 8618
.   : milestone, 8491,
section candidate
no_agent (4.929 ms) : 4867, 4990
.   : milestone, 4929,
iast (10.439 ms) : 10259, 10619
.   : milestone, 10439,
iast_FULL (15.704 ms) : 15390, 16018
.   : milestone, 15704,
iast_GLOBAL (11.648 ms) : 11436, 11860
.   : milestone, 11648,
iast_HARDCODED_SECRET_DISABLED (10.085 ms) : 9911, 10259
.   : milestone, 10085,
iast_INACTIVE (9.01 ms) : 8862, 9158
.   : milestone, 9010,
iast_TELEMETRY_OFF (9.828 ms) : 9670, 9986
.   : milestone, 9828,
profiling (10.612 ms) : 10441, 10782
.   : milestone, 10612,
tracing (8.649 ms) : 8509, 8790
.   : milestone, 8649,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.816 ms [4.757 ms, 4.874 ms]	-
iast	10.571 ms [10.383 ms, 10.76 ms]	5.755 ms (119.5%)
iast_FULL	15.979 ms [15.666 ms, 16.293 ms]	11.164 ms (231.8%)
iast_GLOBAL	11.262 ms [11.053 ms, 11.471 ms]	6.446 ms (133.9%)
iast_HARDCODED_SECRET_DISABLED	10.071 ms [9.891 ms, 10.251 ms]	5.255 ms (109.1%)
iast_INACTIVE	8.829 ms [8.685 ms, 8.973 ms]	4.013 ms (83.3%)
iast_TELEMETRY_OFF	9.984 ms [9.819 ms, 10.15 ms]	5.168 ms (107.3%)
profiling	10.136 ms [9.974 ms, 10.298 ms]	5.321 ms (110.5%)
tracing	8.491 ms [8.364 ms, 8.618 ms]	3.675 ms (76.3%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.929 ms [4.867 ms, 4.99 ms]	-
iast	10.439 ms [10.259 ms, 10.619 ms]	5.511 ms (111.8%)
iast_FULL	15.704 ms [15.39 ms, 16.018 ms]	10.775 ms (218.6%)
iast_GLOBAL	11.648 ms [11.436 ms, 11.86 ms]	6.719 ms (136.3%)
iast_HARDCODED_SECRET_DISABLED	10.085 ms [9.911 ms, 10.259 ms]	5.156 ms (104.6%)
iast_INACTIVE	9.01 ms [8.862 ms, 9.158 ms]	4.081 ms (82.8%)
iast_TELEMETRY_OFF	9.828 ms [9.67 ms, 9.986 ms]	4.899 ms (99.4%)
profiling	10.612 ms [10.441 ms, 10.782 ms]	5.683 ms (115.3%)
tracing	8.649 ms [8.509 ms, 8.79 ms]	3.721 ms (75.5%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~407b1a2cd9, baseline=1.50.0-SNAPSHOT~72f45a8563
    dateFormat X
    axisFormat %s
section baseline
no_agent (36.587 ms) : 36288, 36886
.   : milestone, 36587,
appsec (47.524 ms) : 47110, 47937
.   : milestone, 47524,
code_origins (44.973 ms) : 44601, 45344
.   : milestone, 44973,
iast (43.874 ms) : 43508, 44240
.   : milestone, 43874,
profiling (46.978 ms) : 46555, 47402
.   : milestone, 46978,
tracing (43.8 ms) : 43437, 44162
.   : milestone, 43800,
section candidate
no_agent (36.429 ms) : 36130, 36728
.   : milestone, 36429,
appsec (46.941 ms) : 46531, 47352
.   : milestone, 46941,
code_origins (45.485 ms) : 45090, 45879
.   : milestone, 45485,
iast (44.045 ms) : 43666, 44424
.   : milestone, 44045,
profiling (45.204 ms) : 44814, 45594
.   : milestone, 45204,
tracing (43.21 ms) : 42862, 43558
.   : milestone, 43210,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	36.587 ms [36.288 ms, 36.886 ms]	-
appsec	47.524 ms [47.11 ms, 47.937 ms]	10.937 ms (29.9%)
code_origins	44.973 ms [44.601 ms, 45.344 ms]	8.386 ms (22.9%)
iast	43.874 ms [43.508 ms, 44.24 ms]	7.287 ms (19.9%)
profiling	46.978 ms [46.555 ms, 47.402 ms]	10.392 ms (28.4%)
tracing	43.8 ms [43.437 ms, 44.162 ms]	7.213 ms (19.7%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	36.429 ms [36.13 ms, 36.728 ms]	-
appsec	46.941 ms [46.531 ms, 47.352 ms]	10.512 ms (28.9%)
code_origins	45.485 ms [45.09 ms, 45.879 ms]	9.055 ms (24.9%)
iast	44.045 ms [43.666 ms, 44.424 ms]	7.616 ms (20.9%)
profiling	45.204 ms [44.814 ms, 45.594 ms]	8.775 ms (24.1%)
tracing	43.21 ms [42.862 ms, 43.558 ms]	6.781 ms (18.6%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	ddyurchenko/parallelize-load-tests
git_commit_date	1750176286	1750240216
git_commit_sha	`72f45a8`	`407b1a2`
release_version	1.50.0-SNAPSHOT~72f45a8563	1.50.0-SNAPSHOT~407b1a2cd9

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1750242327	1750242327
ci_job_id	987651338	987651338
ci_pipeline_id	68085722	68085722
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-jp348zax-project-304-concurrent-1-wsi2wrkn 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-jp348zax-project-304-concurrent-1-wsi2wrkn 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~407b1a2cd9, baseline=1.50.0-SNAPSHOT~72f45a8563
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.477 ms) : 1466, 1489
.   : milestone, 1477,
appsec (2.413 ms) : 2364, 2462
.   : milestone, 2413,
iast (2.198 ms) : 2136, 2260
.   : milestone, 2198,
iast_GLOBAL (2.241 ms) : 2178, 2303
.   : milestone, 2241,
profiling (2.478 ms) : 2300, 2656
.   : milestone, 2478,
tracing (2.01 ms) : 1962, 2058
.   : milestone, 2010,
section candidate
no_agent (1.474 ms) : 1462, 1485
.   : milestone, 1474,
appsec (2.423 ms) : 2374, 2473
.   : milestone, 2423,
iast (2.207 ms) : 2145, 2269
.   : milestone, 2207,
iast_GLOBAL (2.236 ms) : 2174, 2299
.   : milestone, 2236,
profiling (2.041 ms) : 1992, 2091
.   : milestone, 2041,
tracing (2.025 ms) : 1976, 2073
.   : milestone, 2025,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.477 ms [1.466 ms, 1.489 ms]	-
appsec	2.413 ms [2.364 ms, 2.462 ms]	936.027 µs (63.4%)
iast	2.198 ms [2.136 ms, 2.26 ms]	720.696 µs (48.8%)
iast_GLOBAL	2.241 ms [2.178 ms, 2.303 ms]	763.456 µs (51.7%)
profiling	2.478 ms [2.3 ms, 2.656 ms]	1.001 ms (67.7%)
tracing	2.01 ms [1.962 ms, 2.058 ms]	532.87 µs (36.1%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.474 ms [1.462 ms, 1.485 ms]	-
appsec	2.423 ms [2.374 ms, 2.473 ms]	949.461 µs (64.4%)
iast	2.207 ms [2.145 ms, 2.269 ms]	732.86 µs (49.7%)
iast_GLOBAL	2.236 ms [2.174 ms, 2.299 ms]	762.394 µs (51.7%)
profiling	2.041 ms [1.992 ms, 2.091 ms]	567.144 µs (38.5%)
tracing	2.025 ms [1.976 ms, 2.073 ms]	550.93 µs (37.4%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~407b1a2cd9, baseline=1.50.0-SNAPSHOT~72f45a8563
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.679 s) : 15679000, 15679000
.   : milestone, 15679000,
appsec (14.674 s) : 14674000, 14674000
.   : milestone, 14674000,
iast (18.492 s) : 18492000, 18492000
.   : milestone, 18492000,
iast_GLOBAL (17.942 s) : 17942000, 17942000
.   : milestone, 17942000,
profiling (15.402 s) : 15402000, 15402000
.   : milestone, 15402000,
tracing (14.859 s) : 14859000, 14859000
.   : milestone, 14859000,
section candidate
no_agent (15.645 s) : 15645000, 15645000
.   : milestone, 15645000,
appsec (15.042 s) : 15042000, 15042000
.   : milestone, 15042000,
iast (18.651 s) : 18651000, 18651000
.   : milestone, 18651000,
iast_GLOBAL (17.927 s) : 17927000, 17927000
.   : milestone, 17927000,
profiling (15.053 s) : 15053000, 15053000
.   : milestone, 15053000,
tracing (14.889 s) : 14889000, 14889000
.   : milestone, 14889000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.679 s [15.679 s, 15.679 s]	-
appsec	14.674 s [14.674 s, 14.674 s]	-1.005 s (-6.4%)
iast	18.492 s [18.492 s, 18.492 s]	2.813 s (17.9%)
iast_GLOBAL	17.942 s [17.942 s, 17.942 s]	2.263 s (14.4%)
profiling	15.402 s [15.402 s, 15.402 s]	-277.0 ms (-1.8%)
tracing	14.859 s [14.859 s, 14.859 s]	-820.0 ms (-5.2%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.645 s [15.645 s, 15.645 s]	-
appsec	15.042 s [15.042 s, 15.042 s]	-603.0 ms (-3.9%)
iast	18.651 s [18.651 s, 18.651 s]	3.006 s (19.2%)
iast_GLOBAL	17.927 s [17.927 s, 17.927 s]	2.282 s (14.6%)
profiling	15.053 s [15.053 s, 15.053 s]	-592.0 ms (-3.8%)
tracing	14.889 s [14.889 s, 14.889 s]	-756.0 ms (-4.8%)

ddyurchenko added 26 commits June 16, 2025 12:24

fix: Add CPU affinity to load benchmarks

b186201

wip: Force trigger Gitlab CI

67461a2

wip: Generalize affinities, export k6 results, change executor

7ab71e3

wip: Save backup of logs

f9a4ed7

wip: Reduce execution time

66d7f08

wip: Tweak duration, add warmup stage, backup logs

70ac693

fix: backup-reports location

d356834

fix: Fix k6 config

2a5c28f

fix: Fix k6 scenario names

587a5c0

fix: Fix k6 scenario names

2f16d6a

fix: Change k6 names format

0000681

fix: Increase warmup time

4f1eb86

fix: Reduce gracefulStop

6cf3cba

fix: Shift start time

037f753

fix: Fix local runs, reduce warmup duration

25f42de

fix: Change debug logs location

< 8000 div class="commit-build-statuses">

57d0b88

tweak: CPU affinities to make sure bottleneck is on k6 side

8efc07e

tweak: Output Java version

6f0e4ea

test: Use constant-arrival-rate executor

b7d0061

tweak: Increase warmup time for petclinic

0fd2e6a

fix: Fix mistake in startTime

f0c5b9a

tweak: Reduce RPS to compensate for loss on profiling petclinic variant

b5bf161

tweak: Parallelize load tests

b1e84e7

tweak: Parallelize insecure-bank, tweak logs storage

ec4fd54

fix: Fix run script location

7ff761e

tweak: Increase iterations count

807da48

ddyurchenko force-pushed the ddyurchenko/parallelize-load-tests branch from 053e7de to 807da48 Compare June 17, 2025 12:09

ddyurchenko added 3 commits June 17, 2025 14:15

fix: Fix paths

5dd26c1

wip: Remove sirun, add verbosity

332b8c6

wip: Fix bash error

c79b0d9

ddyurchenko added 25 commits June 17, 2025 14:39

wip: Fix bash error

a5e489f

wip: Fix bash error

7badabe

wip: Fix bash error

adc00f7

wip: Fix bash error

e03a226

wip: Fix bash error

5a045c5

wip: Add more log lines

87a79bd

wip: Add more log lines

ca7dfb6

wip: Fix REPORTS_DIR

782a347

wip: Fix k6.js

52dab5e

wip: Fix env vars

a8c774e

wip: Fix k6.js

f59664b

wip: Fix k6.js & cleanup

ef92ce5

wip: Reduce verbosity for load apps

868a649

wip: Disable unbound var check

5369d97

wip: Fix pid collection

8c265d5

wip: Fix pid collection

5d6204c

Revert "wip: Fix env vars"

a764311

This reverts commit a8c774e

wip: Separate LOGS_DIR

41e36a2

wip: Reduce verbosity

cd0874a

tweak: k6 config, multiple repetitions

6787311

tweak: multiple repetitions

9a730f7

tweak: Update BP branch, multiple repetitions

c31f6ae

tweak: Tweak RPS for petclinic

67def54

tweak: wait for health check

ee1a7bf

tweak: Tweak RPS

ba7206f

tweak: Increase warmup time

407b1a2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

draft: Parallelize petclinic & insecure-bank benchmarks #8982

draft: Parallelize petclinic & insecure-bank benchmarks #8982

Uh oh!

Uh oh!

Uh oh!

Uh oh!

draft: Parallelize petclinic & insecure-bank benchmarks #8982

Are you sure you want to change the base?

draft: Parallelize petclinic & insecure-bank benchmarks #8982

Uh oh!

Conversation

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

Uh oh!