Add pending traces report in tracer flares #8053

mhlidd · 2024-12-04T01:18:45Z

What Does This Do

APMAPI-906
Creates a TracerFlare Dump in order to dump traces that have been captured but not yet sent to the agent. This PR introduces a new DUMP_DRAIN element in order to properly extract the pending traces and sends such traces to a JsonExporter in order to write them into the TracerFlare as Json. These traces will be displayed from oldest first and are currently limited to 50 traces in a dump.

Motivation

This PR will help with solving customer issues where traces are captured but not sent to the agent because the root span is never finished/written.

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Squash your commits prior merging or merge using GitHub's Squash and merge
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: APMAPI-906

pr-commenter · 2024-12-04T02:00:57Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	mhlidd/pending_traces
git_commit_date	1738176129	1738182647
git_commit_sha	`65e472f`	`b443ddd`
release_version	1.46.0-SNAPSHOT~65e472f132	1.46.0-SNAPSHOT~b443ddd358

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1738185106	1738185106
ci_job_id	786395324	786395324
ci_pipeline_id	54380328	54380328
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-k9ef9rs-project-304-concurrent-0-eog47hem 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-k9ef9rs-project-304-concurrent-0-eog47hem 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 51 metrics, 12 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.46.0-SNAPSHOT~b443ddd358, baseline=1.46.0-SNAPSHOT~65e472f132

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.042 s) : 0, 1041995
Total [baseline] (10.482 s) : 0, 10481904
Agent [candidate] (1.047 s) : 0, 1046822
Total [candidate] (10.498 s) : 0, 10497634
section appsec
Agent [baseline] (1.18 s) : 0, 1180388
Total [baseline] (10.747 s) : 0, 10746902
Agent [candidate] (1.179 s) : 0, 1179456
Total [candidate] (10.762 s) : 0, 10761507
section iast
Agent [baseline] (1.172 s) : 0, 1171739
Total [baseline] (10.946 s) : 0, 10946270
Agent [candidate] (1.169 s) : 0, 1168777
Total [candidate] (10.96 s) : 0, 10960477
section profiling
Agent [baseline] (1.257 s) : 0, 1256880
Total [baseline] (10.853 s) : 0, 10852856
Agent [candidate] (1.259 s) : 0, 1258766
Total [candidate] (10.856 s) : 0, 10855557

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.042 s	-
Agent	appsec	1.18 s	138.393 ms (13.3%)
Agent	iast	1.172 s	129.745 ms (12.5%)
Agent	profiling	1.257 s	214.885 ms (20.6%)
Total	tracing	10.482 s	-
Total	appsec	10.747 s	264.998 ms (2.5%)
Total	iast	10.946 s	464.366 ms (4.4%)
Total	profiling	10.853 s	370.952 ms (3.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.047 s	-
Agent	appsec	1.179 s	132.634 ms (12.7%)
Agent	iast	1.169 s	121.956 ms (11.7%)
Agent	profiling	1.259 s	211.945 ms (20.2%)
Total	tracing	10.498 s	-
Total	appsec	10.762 s	263.873 ms (2.5%)
Total	iast	10.96 s	462.843 ms (4.4%)
Total	profiling	10.856 s	357.923 ms (3.4%)

gantt
    title petclinic - break down per module: candidate=1.46.0-SNAPSHOT~b443ddd358, baseline=1.46.0-SNAPSHOT~65e472f132

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (718.301 ms) : 0, 718301
BytebuddyAgent [candidate] (720.332 ms) : 0, 720332
GlobalTracer [baseline] (240.575 ms) : 0, 240575
GlobalTracer [candidate] (242.468 ms) : 0, 242468
AppSec [baseline] (55.572 ms) : 0, 55572
AppSec [candidate] (55.637 ms) : 0, 55637
Remote Config [baseline] (709.726 µs) : 0, 710
Remote Config [candidate] (729.315 µs) : 0, 729
Telemetry [baseline] (11.55 ms) : 0, 11550
Telemetry [candidate] (12.199 ms) : 0, 12199
section appsec
BytebuddyAgent [baseline] (732.244 ms) : 0, 732244
BytebuddyAgent [candidate] (730.637 ms) : 0, 730637
GlobalTracer [baseline] (237.561 ms) : 0, 237561
GlobalTracer [candidate] (237.214 ms) : 0, 237214
AppSec [baseline] (175.739 ms) : 0, 175739
AppSec [candidate] (176.407 ms) : 0, 176407
Remote Config [baseline] (653.006 µs) : 0, 653
Remote Config [candidate] (656.382 µs) : 0, 656
Telemetry [baseline] (8.291 ms) : 0, 8291
Telemetry [candidate] (8.648 ms) : 0, 8648
IAST [baseline] (21.401 ms) : 0, 21401
IAST [candidate] (21.385 ms) : 0, 21385
section iast
BytebuddyAgent [baseline] (836.119 ms) : 0, 836119
BytebuddyAgent [candidate] (833.454 ms) : 0, 833454
GlobalTracer [baseline] (231.028 ms) : 0, 231028
GlobalTracer [candidate] (231.004 ms) : 0, 231004
AppSec [baseline] (54.042 ms) : 0, 54042
AppSec [candidate] (54.025 ms) : 0, 54025
Remote Config [baseline] (602.63 µs) : 0, 603
Remote Config [candidate] (611.59 µs) : 0, 612
Telemetry [baseline] (8.738 ms) : 0, 8738
Telemetry [candidate] (8.768 ms) : 0, 8768
IAST [baseline] (25.946 ms) : 0, 25946
IAST [candidate] (25.735 ms) : 0, 25735
section profiling
BytebuddyAgent [baseline] (704.412 ms) : 0, 704412
BytebuddyAgent [candidate] (706.129 ms) : 0, 706129
GlobalTracer [baseline] (349.951 ms) : 0, 349951
GlobalTracer [candidate] (351.402 ms) : 0, 351402
AppSec [baseline] (55.339 ms) : 0, 55339
AppSec [candidate] (54.132 ms) : 0, 54132
Remote Config [baseline] (700.984 µs) : 0, 701
Remote Config [candidate] (691.909 µs) : 0, 692
Telemetry [baseline] (8.968 ms) : 0, 8968
Telemetry [candidate] (8.952 ms) : 0, 8952
ProfilingAgent [baseline] (95.286 ms) : 0, 95286
ProfilingAgent [candidate] (95.26 ms) : 0, 95260
Profiling [baseline] (95.309 ms) : 0, 95309
Profiling [candidate] (95.284 ms) : 0, 95284

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.46.0-SNAPSHOT~b443ddd358, baseline=1.46.0-SNAPSHOT~65e472f132

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.045 s) : 0, 1045103
Total [baseline] (8.64 s) : 0, 8639625
Agent [candidate] (1.047 s) : 0, 1046631
Total [candidate] (8.651 s) : 0, 8650927
section iast
Agent [baseline] (1.168 s) : 0, 1167753
Total [baseline] (9.213 s) : 0, 9212517
Agent [candidate] (1.169 s) : 0, 1169341
Total [candidate] (9.22 s) : 0, 9219630
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.185 s) : 0, 1185268
Total [baseline] (9.238 s) : 0, 9238493
Agent [candidate] (1.176 s) : 0, 1176384
Total [candidate] (9.228 s) : 0, 9228245
section iast_TELEMETRY_OFF
Agent [baseline] (1.166 s) : 0, 1165926
Total [baseline] (9.248 s) : 0, 9248155
Agent [candidate] (1.164 s) : 0, 1164436
Total [candidate] (9.277 s) : 0, 9277065

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.045 s	-
Agent	iast	1.168 s	122.65 ms (11.7%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.185 s	140.165 ms (13.4%)
Agent	iast_TELEMETRY_OFF	1.166 s	120.823 ms (11.6%)
Total	tracing	8.64 s	-
Total	iast	9.213 s	572.892 ms (6.6%)
Total	iast_HARDCODED_SECRET_DISABLED	9.238 s	598.868 ms (6.9%)
Total	iast_TELEMETRY_OFF	9.248 s	608.53 ms (7.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.047 s	-
Agent	iast	1.169 s	122.71 ms (11.7%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.176 s	129.753 ms (12.4%)
Agent	iast_TELEMETRY_OFF	1.164 s	117.805 ms (11.3%)
Total	tracing	8.651 s	-
Total	iast	9.22 s	568.703 ms (6.6%)
Total	iast_HARDCODED_SECRET_DISABLED	9.228 s	577.318 ms (6.7%)
Total	iast_TELEMETRY_OFF	9.277 s	626.138 ms (7.2%)

gantt
    title insecure-bank - break down per module: candidate=1.46.0-SNAPSHOT~b443ddd358, baseline=1.46.0-SNAPSHOT~65e472f132

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (717.574 ms) : 0, 717574
BytebuddyAgent [candidate] (719.786 ms) : 0, 719786
GlobalTracer [baseline] (241.96 ms) : 0, 241960
GlobalTracer [candidate] (242.222 ms) : 0, 242222
AppSec [baseline] (55.726 ms) : 0, 55726
AppSec [candidate] (55.767 ms) : 0, 55767
Remote Config [baseline] (1.461 ms) : 0, 1461
Remote Config [candidate] (721.451 µs) : 0, 721
Telemetry [baseline] (13.152 ms) : 0, 13152
Telemetry [candidate] (12.856 ms) : 0, 12856
section iast
BytebuddyAgent [baseline] (832.946 ms) : 0, 832946
BytebuddyAgent [candidate] (833.78 ms) : 0, 833780
GlobalTracer [baseline] (230.775 ms) : 0, 230775
GlobalTracer [candidate] (231.563 ms) : 0, 231563
IAST [baseline] (24.877 ms) : 0, 24877
IAST [candidate] (24.115 ms) : 0, 24115
AppSec [baseline] (54.574 ms) : 0, 54574
AppSec [candidate] (55.352 ms) : 0, 55352
Remote Config [baseline] (610.574 µs) : 0, 611
Remote Config [candidate] (611.917 µs) : 0, 612
Telemetry [baseline] (8.743 ms) : 0, 8743
Telemetry [candidate] (8.624 ms) : 0, 8624
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (846.615 ms) : 0, 846615
BytebuddyAgent [candidate] (839.629 ms) : 0, 839629
GlobalTracer [baseline] (233.106 ms) : 0, 233106
GlobalTracer [candidate] (232.589 ms) : 0, 232589
IAST [baseline] (24.618 ms) : 0, 24618
IAST [candidate] (28.136 ms) : 0, 28136
AppSec [baseline] (56.007 ms) : 0, 56007
AppSec [candidate] (51.353 ms) : 0, 51353
Remote Config [baseline] (634.657 µs) : 0, 635
Remote Config [candidate] (612.754 µs) : 0, 613
Telemetry [baseline] (8.789 ms) : 0, 8789
Telemetry [candidate] (8.704 ms) : 0, 8704
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (831.684 ms) : 0, 831684
BytebuddyAgent [candidate] (830.259 ms) : 0, 830259
GlobalTracer [baseline] (230.25 ms) : 0, 230250
GlobalTracer [candidate] (230.843 ms) : 0, 230843
IAST [baseline] (25.486 ms) : 0, 25486
IAST [candidate] (26.045 ms) : 0, 26045
AppSec [baseline] (53.957 ms) : 0, 53957
AppSec [candidate] (52.94 ms) : 0, 52940
Remote Config [baseline] (613.925 µs) : 0, 614
Remote Config [candidate] (610.595 µs) : 0, 611
Telemetry [baseline] (8.678 ms) : 0, 8678
Telemetry [candidate] (8.556 ms) : 0, 8556

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-01-29T20:41:42	2025-01-29T20:48:45
git_branch	master	mhlidd/pending_traces
git_commit_date	1738176129	1738182647
git_commit_sha	`65e472f`	`b443ddd`
release_version	1.46.0-SNAPSHOT~65e472f132	1.46.0-SNAPSHOT~b443ddd358
start_time	2025-01-29T20:41:28	2025-01-29T20:48:31

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1738184082	1738184082
ci_job_id	786395325	786395325
ci_pipeline_id	54380328	54380328
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-grayk6wr-project-304-concurrent-0-vhzl2jq7 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-grayk6wr-project-304-concurrent-0-vhzl2jq7 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 17 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:petclinic:profiling	better [-94.280µs; -40.325µs] or [-5.971%; -2.554%]	unstable [-436.334op/s; +664.255op/s] or [-14.726%; +22.419%]	1.512ms	3076.923op/s	1.579ms	2962.963op/s

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.46.0-SNAPSHOT~b443ddd358, baseline=1.46.0-SNAPSHOT~65e472f132
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.357 ms) : 1337, 1377
.   : milestone, 1357,
appsec (1.739 ms) : 1716, 1763
.   : milestone, 1739,
appsec_no_iast (1.748 ms) : 1725, 1771
.   : milestone, 1748,
iast (1.497 ms) : 1474, 1521
.   : milestone, 1497,
profiling (1.579 ms) : 1554, 1604
.   : milestone, 1579,
tracing (1.47 ms) : 1445, 1495
.   : milestone, 1470,
section candidate
no_agent (1.348 ms) : 1329, 1367
.   : milestone, 1348,
appsec (1.753 ms) : 1729, 1776
.   : milestone, 1753,
appsec_no_iast (1.75 ms) : 1726, 1774
.   : milestone, 1750,
iast (1.507 ms) : 1484, 1531
.   : milestone, 1507,
profiling (1.512 ms) : 1486, 1537
.   : milestone, 1512,
tracing (1.496 ms) : 1471, 1521
.   : milestone, 1496,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.357 ms [1.337 ms, 1.377 ms]	-
appsec	1.739 ms [1.716 ms, 1.763 ms]	382.285 µs (28.2%)
appsec_no_iast	1.748 ms [1.725 ms, 1.771 ms]	391.009 µs (28.8%)
iast	1.497 ms [1.474 ms, 1.521 ms]	140.443 µs (10.4%)
profiling	1.579 ms [1.554 ms, 1.604 ms]	222.169 µs (16.4%)
tracing	1.47 ms [1.445 ms, 1.495 ms]	112.919 µs (8.3%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.348 ms [1.329 ms, 1.367 ms]	-
appsec	1.753 ms [1.729 ms, 1.776 ms]	404.388 µs (30.0%)
appsec_no_iast	1.75 ms [1.726 ms, 1.774 ms]	401.558 µs (29.8%)
iast	1.507 ms [1.484 ms, 1.531 ms]	159.229 µs (11.8%)
profiling	1.512 ms [1.486 ms, 1.537 ms]	163.534 µs (12.1%)
tracing	1.496 ms [1.471 ms, 1.521 ms]	147.492 µs (10.9%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.46.0-SNAPSHOT~b443ddd358, baseline=1.46.0-SNAPSHOT~65e472f132
    dateFormat X
    axisFormat %s
section baseline
no_agent (373.103 µs) : 353, 393
.   : milestone, 373,
iast (505.228 µs) : 482, 528
.   : milestone, 505,
iast_FULL (738.71 µs) : 716, 762
.   : milestone, 739,
iast_GLOBAL (556.739 µs) : 533, 580
.   : milestone, 557,
iast_HARDCODED_SECRET_DISABLED (501.118 µs) : 478, 524
.   : milestone, 501,
iast_INACTIVE (452.292 µs) : 431, 474
.   : milestone, 452,
iast_TELEMETRY_OFF (498.23 µs) : 475, 522
.   : milestone, 498,
tracing (454.269 µs) : 433, 476
.   : milestone, 454,
section candidate
no_agent (379.618 µs) : 360, 400
.   : milestone, 380,
iast (513.334 µs) : 490, 537
.   : milestone, 513,
iast_FULL (738.599 µs) : 717, 760
.   : milestone, 739,
iast_GLOBAL (557.494 µs) : 533, 582
.   : milestone, 557,
iast_HARDCODED_SECRET_DISABLED (502.657 µs) : 481, 524
.   : milestone, 503,
iast_INACTIVE (451.697 µs) : 431, 473
.   : milestone, 452,
iast_TELEMETRY_OFF (489.075 µs) : 466, 512
.   : milestone, 489,
tracing (443.336 µs) : 423, 464
.   : milestone, 443,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	373.103 µs [353.491 µs, 392.715 µs]	-
iast	505.228 µs [482.057 µs, 528.4 µs]	132.125 µs (35.4%)
iast_FULL	738.71 µs [715.753 µs, 761.667 µs]	365.607 µs (98.0%)
iast_GLOBAL	556.739 µs [533.225 µs, 580.252 µs]	183.636 µs (49.2%)
iast_HARDCODED_SECRET_DISABLED	501.118 µs [478.275 µs, 523.96 µs]	128.015 µs (34.3%)
iast_INACTIVE	452.292 µs [430.977 µs, 473.608 µs]	79.189 µs (21.2%)
iast_TELEMETRY_OFF	498.23 µs [474.592 µs, 521.868 µs]	125.127 µs (33.5%)
tracing	454.269 µs [432.824 µs, 475.715 µs]	81.166 µs (21.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	379.618 µs [359.532 µs, 399.704 µs]	-
iast	513.334 µs [489.596 µs, 537.073 µs]	133.716 µs (35.2%)
iast_FULL	738.599 µs [716.727 µs, 760.47 µs]	358.98 µs (94.6%)
iast_GLOBAL	557.494 µs [533.128 µs, 581.861 µs]	177.876 µs (46.9%)
iast_HARDCODED_SECRET_DISABLED	502.657 µs [481.062 µs, 524.253 µs]	123.039 µs (32.4%)
iast_INACTIVE	451.697 µs [430.647 µs, 472.747 µs]	72.079 µs (19.0%)
iast_TELEMETRY_OFF	489.075 µs [466.14 µs, 512.01 µs]	109.456 µs (28.8%)
tracing	443.336 µs [422.985 µs, 463.687 µs]	63.718 µs (16.8%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	mhlidd/pending_traces
git_commit_date	1738176129	1738182647
git_commit_sha	`65e472f`	`b443ddd`
release_version	1.46.0-SNAPSHOT~65e472f132	1.46.0-SNAPSHOT~b443ddd358

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1738184662	1738184662
ci_job_id	786395326	786395326
ci_pipeline_id	54380328	54380328
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-k9ef9rs-project-304-concurrent-1-wwgcvgi9 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-k9ef9rs-project-304-concurrent-1-wwgcvgi9 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.46.0-SNAPSHOT~b443ddd358, baseline=1.46.0-SNAPSHOT~65e472f132
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.465 ms) : 1453, 1476
.   : milestone, 1465,
appsec (2.342 ms) : 2299, 2386
.   : milestone, 2342,
iast (2.106 ms) : 2051, 2161
.   : milestone, 2106,
iast_GLOBAL (2.149 ms) : 2094, 2205
.   : milestone, 2149,
profiling (1.99 ms) : 1945, 2034
.   : milestone, 1990,
tracing (1.942 ms) : 1899, 1984
.   : milestone, 1942,
section candidate
no_agent (1.47 ms) : 1458, 1482
.   : milestone, 1470,
appsec (2.335 ms) : 2292, 2378
.   : milestone, 2335,
iast (2.095 ms) : 2041, 2150
.   : milestone, 2095,
iast_GLOBAL (2.144 ms) : 2089, 2200
.   : milestone, 2144,
profiling (1.98 ms) : 1936, 2025
.   : milestone, 1980,
tracing (1.928 ms) : 1886, 1970
.   : milestone, 1928,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.465 ms [1.453 ms, 1.476 ms]	-
appsec	2.342 ms [2.299 ms, 2.386 ms]	877.576 µs (59.9%)
iast	2.106 ms [2.051 ms, 2.161 ms]	641.327 µs (43.8%)
iast_GLOBAL	2.149 ms [2.094 ms, 2.205 ms]	684.605 µs (46.7%)
profiling	1.99 ms [1.945 ms, 2.034 ms]	524.984 µs (35.8%)
tracing	1.942 ms [1.899 ms, 1.984 ms]	476.884 µs (32.6%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.47 ms [1.458 ms, 1.482 ms]	-
appsec	2.335 ms [2.292 ms, 2.378 ms]	865.075 µs (58.8%)
iast	2.095 ms [2.041 ms, 2.15 ms]	625.498 µs (42.6%)
iast_GLOBAL	2.144 ms [2.089 ms, 2.2 ms]	674.441 µs (45.9%)
profiling	1.98 ms [1.936 ms, 2.025 ms]	510.175 µs (34.7%)
tracing	1.928 ms [1.886 ms, 1.97 ms]	458.159 µs (31.2%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.46.0-SNAPSHOT~b443ddd358, baseline=1.46.0-SNAPSHOT~65e472f132
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.166 s) : 15166000, 15166000
.   : milestone, 15166000,
appsec (14.992 s) : 14992000, 14992000
.   : milestone, 14992000,
iast (18.39 s) : 18390000, 18390000
.   : milestone, 18390000,
iast_GLOBAL (17.922 s) : 17922000, 17922000
.   : milestone, 17922000,
profiling (15.142 s) : 15142000, 15142000
.   : milestone, 15142000,
tracing (14.728 s) : 14728000, 14728000
.   : milestone, 14728000,
section candidate
no_agent (15.452 s) : 15452000, 15452000
.   : milestone, 15452000,
appsec (14.989 s) : 14989000, 14989000
.   : milestone, 14989000,
iast (18.635 s) : 18635000, 18635000
.   : milestone, 18635000,
iast_GLOBAL (17.978 s) : 17978000, 17978000
.   : milestone, 17978000,
profiling (15.0 s) : 15000000, 15000000
.   : milestone, 15000000,
tracing (15.167 s) : 15167000, 15167000
.   : milestone, 15167000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.166 s [15.166 s, 15.166 s]	-
appsec	14.992 s [14.992 s, 14.992 s]	-174.0 ms (-1.1%)
iast	18.39 s [18.39 s, 18.39 s]	3.224 s (21.3%)
iast_GLOBAL	17.922 s [17.922 s, 17.922 s]	2.756 s (18.2%)
profiling	15.142 s [15.142 s, 15.142 s]	-24.0 ms (-0.2%)
tracing	14.728 s [14.728 s, 14.728 s]	-438.0 ms (-2.9%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.452 s [15.452 s, 15.452 s]	-
appsec	14.989 s [14.989 s, 14.989 s]	-463.0 ms (-3.0%)
iast	18.635 s [18.635 s, 18.635 s]	3.183 s (20.6%)
iast_GLOBAL	17.978 s [17.978 s, 17.978 s]	2.526 s (16.3%)
profiling	15.0 s [15.0 s, 15.0 s]	-452.0 ms (-2.9%)
tracing	15.167 s [15.167 s, 15.167 s]	-285.0 ms (-1.8%)

dd-trace-core/src/main/java/datadog/trace/core/CoreTracer.java

dd-trace-core/src/main/java/datadog/trace/core/flare/TracerFlareService.java

dd-trace-core/src/main/java/datadog/trace/core/util/TracerDump.java

dd-trace-core/src/main/java/datadog/trace/core/CoreTracer.java

dd-trace-core/src/main/java/datadog/trace/core/PendingTrace.java

dd-trace-core/src/main/java/datadog/trace/core/util/TracerDump.java

mcculls · 2024-12-12T12:14:38Z

Have you considered tackling this another way, by adding the reporting to DelayingPendingTraceBuffer ? That has all the unfinished (pending) traces not yet sent to the agent. Rather than maintaining a parallel collection of weak references you could just peek into what's in the buffer and write that out - using some limits to avoid sending a massive flare which may be rejected due to its size.

dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java

PerfectSlayer

Have drain and fill together feels way better! 🙌

Now, about using the collected elements:

You can declare your comparator as a private final Comparator<List<Span>> TRACE_BY_START_DATE field so its meaning will be clear and it will only be allocated once
You will need to figure out how to filter, limit, and map your elements. I would recommend having a look at the Java Stream API
Now you get most of the part in place, try building some tests / triggering your code. You should be able to find some NPE by yourself 😉

dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java

datadog-datadog-prod-us1 · 2025-01-22T21:29:34Z

dd-trace-core/src/main/java/datadog/trace/common/writer/TraceDumpWriter.java

+
+public class TraceDumpWriter implements Writer {
+
+  private StringBuilder dumpText;


🟠 Code Quality Violation

StringBuilder can lead to memory leaks in long lasting classes (...read more)

StringBuffers and StringBuilders have the potential to grow significantly, which could lead to memory leaks if they are retained within objects with extended lifetimes.

datadog-datadog-prod-us1 · 2025-01-22T21:45:06Z

dd-trace-core/src/main/java/datadog/trace/common/writer/TraceDumpJsonExporter.java

+
+public class TraceDumpJsonExporter {
+
+  private StringBuilder dumpText;


🟠 Code Quality Violation

StringBuilder can lead to memory leaks in long lasting classes (...read more)

StringBuffers and StringBuilders have the potential to grow significantly, which could lead to memory leaks if they are retained within objects with extended lifetimes.

dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java

PerfectSlayer

LGTM
cc @mcculls if you would like to check it before it gets merged

dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java

dd-trace-core/src/main/java/datadog/trace/common/writer/TraceDumpJsonExporter.java

dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java

mcculls · 2025-01-29T11:41:44Z

dd-trace-core/src/main/java/datadog/trace/common/writer/TraceDumpJsonExporter.java

+      new Moshi.Builder()
+          .add(DDSpanJsonAdapter.buildFactory(false))
+          .build()
+          .adapter(Types.newParameterizedType(List.class, DDSpan.class));


Any reason we don't use Collection.class here, which would then let us use public void write(final Collection<DDSpan> trace) { or if you want to keep it a bit more specific public void write(final Deque<DDSpan> trace) { and avoid the need to copy span data into a temporary ArrayList just to fit the expected interface?

Initially it was because public void write(final List<DDSpan> trace) has to be implemented regardless to override the writer interface that declares the method. I'm thinking I could override the method with a dummy implementation and overload a new public void write(final Deque<DDSpan> trace) { that has the actual implementation instead.

dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java

mcculls · 2025-01-29T11:50:07Z

dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java

+    private static final class DumpDrain
+        implements MessagePassingQueue.Consumer<Element>, MessagePassingQueue.Supplier<Element> {
+      private static final DumpDrain DUMP_DRAIN = new DumpDrain();
+      private static final List<Element> DATA = new ArrayList<>();


Can you make DATA an instance field?

You could then introduce a method called List<Element> collectTraces() which removes non-pending-trace elements, sorts it by oldest trace, then returns the data. You could also add a reset method to clear the recorded data, which avoids the need to expose the internal collection - although note that would not compact the underlying array.

You might want to consider making the data field volatile - the collectTraces method could then do the following:

store the current data reference in a local variable

assign a new ArrayList() to data - ready for the next dump request

return the local variable, i.e. the old ArrayList with the trace data

The main benefit this would bring is that there's a clear line between the writing and reading sides at the point we assign a new collection to data. It would also allow the data collection to be compacted, because we assign it a fresh ArrayList

I'm not fully sure I understand the point of making data volatile. From what I understand, it essentially allows us to "clear" the data array in collectTraces() by assigning it to a temp variable (whose memory will be cleared once the code exits the scope of the for-loop that calls the method, and instantiate a clean ArrayList for future traces to be stored until the next dump. Is this understanding correct?

datadog-datadog-prod-us1 · 2025-01-29T20:31:49Z

dd-trace-core/src/main/java/datadog/trace/common/writer/TraceDumpJsonExporter.java

+          .add(DDSpanJsonAdapter.buildFactory(false))
+          .build()
+          .adapter(Types.newParameterizedType(Collection.class, DDSpan.class));
+  private StringBuilder dumpText;


🟠 Code Quality Violation

StringBuilder can lead to memory leaks in long lasting classes (...read more)

StringBuffers and StringBuilders have the potential to grow significantly, which could lead to memory leaks if they are retained within objects with extended lifetimes.

PerfectSlayer reviewed Dec 6, 2024

View reviewed changes

mhlidd requested a review from PerfectSlayer December 9, 2024 15:13

PerfectSlayer reviewed Dec 11, 2024

View reviewed changes

PerfectSlayer changed the title ~~adding tracerflare.addtext~~ Add pending traces to tracer flare Dec 11, 2024

PerfectSlayer added type: enhancement comp: telemetry Telemetry labels Dec 11, 2024

mcculls reviewed Dec 12, 2024

View reviewed changes

dd-trace-core/src/main/java/datadog/trace/core/CoreTracer.java Outdated Show resolved Hide resolved

mcculls reviewed Dec 12, 2024

View reviewed changes

dd-trace-core/src/main/java/datadog/trace/core/CoreTracer.java Outdated Show resolved Hide resolved

mcculls reviewed Dec 12, 2024

View reviewed changes

dd-trace-core/src/main/java/datadog/trace/core/PendingTrace.java Outdated Show resolved Hide resolved

mcculls reviewed Dec 12, 2024

View reviewed changes

dd-trace-core/src/main/java/datadog/trace/core/util/TracerDump.java Outdated Show resolved Hide resolved

mhlidd force-pushed the mhlidd/pending_traces branch from 4be2999 to 3e4955a Compare December 16, 2024 21:01

datadog-datadog-prod-us1 bot reviewed Dec 17, 2024

View reviewed changes

PerfectSlayer reviewed Dec 18, 2024

View reviewed changes

mhlidd requested a review from PerfectSlayer December 20, 2024 03:54

PerfectSlayer reviewed Dec 20, 2024

View reviewed changes

PerfectSlayer force-pushed the mhlidd/pending_traces branch from 5119ac7 to 3a8b13c Compare January 21, 2025 09:51

datadog-datadog-prod-us1 bot reviewed Jan 22, 2025

View reviewed changes

mhlidd changed the title ~~Add pending traces to tracer flare~~ Add support for TracerDump in TracerFlare Jan 22, 2025

mhlidd marked this pull request as ready for review January 22, 2025 21:44

mhlidd requested a review from a team as a code owner January 22, 2025 21:44

mhlidd requested review from amarziali, PerfectSlayer and mcculls January 22, 2025 21:44

datadog-datadog-prod-us1 bot reviewed Jan 22, 2025

View reviewed changes

PerfectSlayer reviewed Jan 23, 2025

View reviewed changes

dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java Outdated Show resolved Hide resolved

mhlidd added comp: telemetry Telemetry and removed comp: telemetry Telemetry labels Jan 27, 2025

bm1549 removed the request for review from mcculls January 27, 2025 22:47

mhlidd and others added 12 commits January 28, 2025 09:22

updating test

b3e9048

feat(core): Use prepare for flare to signal dump element

713ccda

feat(core): Update test prevent the span to be written early

0ee90c1

fix(core): Fix tests

bc71dcf

fix(core): Reduce scope

60d08e7

fix(core): Revert drain limit

60bb162

feat(core): Refactor action elements

3382b3d

adding support for json encoding of traces

fbed357

cleanup

1dee7f3

renaming file

af57f32

making TraceDumpJsonExporter a Writer

e9eac71

nit changes

9786fb5

mhlidd force-pushed the mhlidd/pending_traces branch from 4f6a0db to 9786fb5 Compare January 28, 2025 14:25

mhlidd requested a review from PerfectSlayer January 28, 2025 14:25

updating test to match changes

3d0e3f7

mhlidd added type: enhancement and removed type: enhancement labels Jan 28, 2025

PerfectSlayer approved these changes Jan 29, 2025

View reviewed changes

mcculls reviewed Jan 29, 2025

View reviewed changes

dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java Outdated Show resolved Hide resolved

mcculls reviewed Jan 29, 2025

View reviewed changes

dd-trace-core/src/main/java/datadog/trace/common/writer/TraceDumpJsonExporter.java Outdated Show resolved Hide resolved

mcculls reviewed Jan 29, 2025

View reviewed changes

dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java Outdated Show resolved Hide resolved

mcculls reviewed Jan 29, 2025

View reviewed changes

dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java Outdated Show resolved Hide resolved

mcculls reviewed Jan 29, 2025

View reviewed changes

dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java Outdated Show resolved Hide resolved

mcculls reviewed Jan 29, 2025

View reviewed changes

addressing PR comments

b443ddd

mhlidd requested a review from mcculls January 29, 2025 20:31

datadog-datadog-prod-us1 bot reviewed Jan 29, 2025

View reviewed changes

mhlidd added comp: telemetry Telemetry and removed comp: telemetry Telemetry labels Jan 29, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add pending traces report in tracer flares #8053

Add pending traces report in tracer flares #8053

mhlidd commented Dec 4, 2024 •

edited

Loading

pr-commenter bot commented Dec 4, 2024 •

edited

Loading

mcculls commented Dec 12, 2024

PerfectSlayer left a comment

datadog-datadog-prod-us1 bot Jan 22, 2025

datadog-datadog-prod-us1 bot Jan 22, 2025

PerfectSlayer left a comment •

edited

Loading

mcculls Jan 29, 2025

mhlidd Jan 29, 2025

mcculls Jan 29, 2025

mhlidd Jan 29, 2025

datadog-datadog-prod-us1 bot Jan 29, 2025


		public class TraceDumpWriter implements Writer {

		private StringBuilder dumpText;


		public class TraceDumpJsonExporter {

		private StringBuilder dumpText;

Add pending traces report in tracer flares #8053

Are you sure you want to change the base?

Add pending traces report in tracer flares #8053

Conversation

mhlidd commented Dec 4, 2024 • edited Loading

What Does This Do

Motivation

Additional Notes

Contributor Checklist

pr-commenter bot commented Dec 4, 2024 • edited Loading

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

mcculls commented Dec 12, 2024

PerfectSlayer left a comment

Choose a reason for hiding this comment

datadog-datadog-prod-us1 bot Jan 22, 2025

Choose a reason for hiding this comment

🟠 Code Quality Violation

datadog-datadog-prod-us1 bot Jan 22, 2025

Choose a reason for hiding this comment

🟠 Code Quality Violation

PerfectSlayer left a comment • edited Loading

Choose a reason for hiding this comment

mcculls Jan 29, 2025

Choose a reason for hiding this comment

mhlidd Jan 29, 2025

Choose a reason for hiding this comment

mcculls Jan 29, 2025

Choose a reason for hiding this comment

mhlidd Jan 29, 2025

Choose a reason for hiding this comment

datadog-datadog-prod-us1 bot Jan 29, 2025

Choose a reason for hiding this comment

🟠 Code Quality Violation

mhlidd commented Dec 4, 2024 •

edited

Loading

pr-commenter bot commented Dec 4, 2024 •

edited

Loading

PerfectSlayer left a comment •

edited

Loading