add APIs for llm obs sdk #8135

gary-huang · 2024-12-30T15:27:59Z

What Does This Do

Add APIs for LLM obs SDK manual instrumentation

Starting spans

LLMObsSpan span = LLMObs.startLLMSpan(String spanName, String modelName, String modelProvider, @Nullable String mlApp, @Nullable String sessionID)
LLMObsSpan span = LLMObs.startAgentSpan(String spanName, @Nullable String mlApp, @Nullable String sessionID)
LLMObsSpan span = LLMObs.startToolSpan(String spanName, @Nullable String mlApp, @Nullable String sessionID)
LLMObsSpan span = LLMObs.startTaskSpan(String spanName, @Nullable String mlApp, @Nullable String sessionID)
LLMObsSpan span = LLMObs.startWorkflowSpan(String spanName, @Nullable String mlApp, @Nullable String sessionID)

Interacting with spans

span.annotateIO(List<Map<String, Object>> inputData, List<Map<String, Object>> outputData)
span.annotateIO(String inputData, String outputData)
span.setMetadata(Map<String, Object> metadata)
span.setMetrics(Map<String, Number> metrics)
span.setMetric(CharSequence key, int value)
setMetric(CharSequence key, long value)
setMetric(CharSequence key, double value)
span.setTags(Map<String, Object> tags)
span.setTag(String key, String value)
span.setTag(String key, boolean value)
span.setTag(String key, int value)
span.setTag(String key, long value)
span.setTag(String key, double value)
span.setError(boolean error)
span.setErrorMessage(String errorMessage)
span.addThrowable(Throwable throwable)
span.finish()

A sample spring controller using the APIs above

package com.example.restservice;

import datadog.trace.api.appsec.LoginEventCallback;
import datadog.trace.api.civisibility.telemetry.tag.AgentlessLogSubmissionEnabled;
import java.lang.management.ManagementFactory;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.concurrent.atomic.AtomicLong;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

import datadog.trace.api.llmobs.LLMObs;

@RestController
public class GreetingController {

	private static final String template = "Hello, %s, the %dth visitor!";
	private final AtomicLong counter = new AtomicLong();

	private static final Logger logger = LoggerFactory.getLogger(GreetingController.class);

	private String getCurrentSessionID() {
		return ManagementFactory.getRuntimeMXBean().getStartTime() + "-" + counter.get();
	}

	private long taskCall(String input) {
		var currentCount = counter.get();
		Throwable t = null;
		var taskSpan = LLMObs.startTaskSpan("divide_by_zero_task", null, getCurrentSessionID());
		try {
			var l = 100 / 0;
			logger.info("TASK COMPLETE");
		} catch (Exception e) {
			t = e;
		}
		taskSpan.addThrowable(t);
		taskSpan.annotateIO("param input is " + input, "current count is " + currentCount);
		taskSpan.finish();
		logger.info("taskSpan {}", taskSpan);
		return currentCount;
	}

	@GetMapping("/greeting")
	public Greeting greeting(@RequestParam(value = "name", defaultValue = "World") String name) {
		var session = getCurrentSessionID();

		var workflowSpan = LLMObs.startWorkflowSpan("workflow",  null, session);
		var currentCount = taskCall(name);

		var llmObsSpan1 = LLMObs.startLLMSpan("llm1", "gpt4", "openai", "", session);


		var erredLLMSpan = LLMObs.startLLMSpan("llm2-err", "gpt4", "openai", "test-gary-mlapp-override-app", session);


		workflowSpan.setTags(new HashMap<String, Object>(){{put("tag1", 1); put("tag2", "a");}});
		workflowSpan.setMetrics(new HashMap<>(){{put("input_tokens", 100); put("output_tokens", 100l); put("log_prob", 0.1);}});

		llmObsSpan1.annotateIO(
				Arrays.asList(
						LLMObs.LLMMessage.from("user", "user has hit our API endpoint"),
						LLMObs.LLMMessage.from("system", "You are an experienced travel guide. Suggest three vacation destinations for a family looking for sunny weather, outdoor activities and cultural experiences. Each suggestion should be 100-150 words, written in a friendly and conversational tone. Present the suggestions as a numbered list.")
				),
				Arrays.asList(
						LLMObs.LLMMessage.from("assistant", "ok someone visited your API endpoint"
								)))
		);
		llmObsSpan1.setTag("tag1", 1);
		llmObsSpan1.setMetadata(new HashMap<String, Object>(){{put("temperature", "1"); put("max_tokens", 100);}});

		erredLLMSpan.annotateIO("span2 in", "span2 out");
		erredLLMSpan.setTags(new HashMap<String, Object>(){{put("tag1", 1); put("tag2", "a");}});
		erredLLMSpan.setErrorMessage("aaa");

		var resp = String.format(template, name, currentCount);
		workflowSpan.annotateIO(name, resp);

		logger.info("workflow {}", workflowSpan);
		logger.info("llm1 {}", llmObsSpan1);
		logger.info("erredLLMSpan {}", erredLLMSpan);

		workflowSpan.finish();
		llmObsSpan1.finish();
		erredLLMSpan.finish();
		return new Greeting(counter.incrementAndGet(), resp);

	}
}

logging for the spans above

2025-02-13T02:52:24.583-05:00  INFO 53094 --- [nio-8080-exec-1] c.e.restservice.GreetingController       : taskSpan datadog.trace.llmobs.domain.DDLLMObsSpan@7b9b90a2, trace_id=479453473746284206, span_id=2948307615785206470, ml_app=gary-test, service=gary-test, span_kind=task, context=DDSpan [ t_id=479453473746284206, s_id=2948307615785206470, p_id=3544560763018401567 ] trace=gary-test/divide_by_zero_task/divide_by_zero_task *errored* tags={_llmobs_tag.input=param input is gary, _llmobs_tag.ml_app=gary-test, _llmobs_tag.output=current count is 0, _llmobs_tag.session_id=1739433141606-0, env=staging, error.message=/ by zero, error.stack=java.lang.ArithmeticException: / by zero
	at com.example.restservice.GreetingController.taskCall(GreetingController.java:36)
	at com.example.restservice.GreetingController.greeting(GreetingController.java:53)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:255)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:188)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:926)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:831)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:564)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at datadog.trace.instrumentation.springweb6.HandlerMappingResourceNameFilter.doFilterInternal(HandlerMappingResourceNameFilter.java:50)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.springframework.web.filter.ServletRequestPathFilter.doFilter(ServletRequestPathFilter.java:52)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:352)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:268)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:389)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
	at java.base/java.lang.Thread.run(Thread.java:842)
, error.type=java.lang.ArithmeticException, span.kind=task, thread.id=40, thread.name=http-nio-8080-exec-1, version=1.0}
2025-02-13T02:52:24.589-05:00  INFO 53094 --- [nio-8080-exec-1] c.e.restservice.GreetingController       : workflow datadog.trace.llmobs.domain.DDLLMObsSpan@375bdc24, trace_id=479453473746284206, span_id=7264160562704185531, ml_app=gary-test, service=gary-test, span_kind=workflow, context=DDSpan [ t_id=479453473746284206, s_id=7264160562704185531, p_id=3544560763018401567 ] trace=gary-test/workflow/workflow tags={_llmobs_metric.input_tokens=100.0, _llmobs_metric.log_prob=0.1, _llmobs_metric.output_tokens=100.0, _llmobs_tag.input=gary, _llmobs_tag.ml_app=gary-test, _llmobs_tag.output=Hello, gary!, _llmobs_tag.session_id=1739433141606-0, _llmobs_tag.tag1=1, _llmobs_tag.tag2=a, env=staging, span.kind=workflow, thread.id=40, thread.name=http-nio-8080-exec-1, version=1.0}
2025-02-13T02:52:24.589-05:00  INFO 53094 --- [nio-8080-exec-1] c.e.restservice.GreetingController       : llm1 datadog.trace.llmobs.domain.DDLLMObsSpan@65e7b713, trace_id=479453473746284206, span_id=5473581304479074833, ml_app=gary-test, service=gary-test, span_kind=llm, context=DDSpan [ t_id=479453473746284206, s_id=5473581304479074833, p_id=3544560763018401567 ] trace=gary-test/llm1/llm1 tags={_llmobs_tag._llmobs_tag.model_name=gpt4, _llmobs_tag._llmobs_tag.model_provider=openai, _llmobs_tag.metadata={max_tokens=100, temperature=1}, _llmobs_tag.ml_app=gary-test, _llmobs_tag.session_id=1739433141606-0, _llmobs_tag.tag1=1, env=staging, span.kind=llm, thread.id=40, thread.name=http-nio-8080-exec-1, version=1.0}
2025-02-13T02:52:24.589-05:00  INFO 53094 --- [nio-8080-exec-1] c.e.restservice.GreetingController       : erredLLMSpan datadog.trace.llmobs.domain.DDLLMObsSpan@d9d015b, trace_id=479453473746284206, s
10000
pan_id=4239354304348392454, ml_app=test-gary-mlapp-override-app, service=gary-test, span_kind=llm, context=DDSpan [ t_id=479453473746284206, s_id=4239354304348392454, p_id=3544560763018401567 ] trace=gary-test/llm2-err/llm2-err *errored* tags={_llmobs_tag._llmobs_tag.model_name=gpt4, _llmobs_tag._llmobs_tag.model_provider=openai, _llmobs_tag.input=span2 in, _llmobs_tag.ml_app=test-gary-mlapp-override-app, _llmobs_tag.output=span2 out, _llmobs_tag.session_id=1739433141606-0, _llmobs_tag.tag1=1, _llmobs_tag.tag2=a, env=staging, error.message=aaa, span.kind=llm, thread.id=40, thread.name=http-nio-8080-exec-1, version=1.0}
[dd.trace 2025-02-13 02:52:24:589 -0500] [http-nio-8080-exec-1] DEBUG datadog.trace.agent.core.DDSpan - Finished span (PENDING): DDSpan [ t_id=479453473746284206, s_id=7264160562704185531, p_id=3544560763018401567 ] trace=gary-test/workflow/workflow tags={_llmobs_metric.input_tokens=100.0, _llmobs_metric.log_prob=0.1, _llmobs_metric.output_tokens=100.0, _llmobs_tag.input=gary, _llmobs_tag.ml_app=gary-test, _llmobs_tag.output=Hello, gary!, _llmobs_tag.session_id=1739433141606-0, _llmobs_tag.tag1=1, _llmobs_tag.tag2=a, env=staging, span.kind=workflow, thread.id=40, thread.name=http-nio-8080-exec-1, version=1.0}, duration_ns=6872375, forceKeep=false, links=[]

Motivation

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: MLOB-1422

pr-commenter · 2024-12-30T17:22:05Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	gary/add-llm-obs-api
git_commit_date	1745958763	1745965032
git_commit_sha	`e8a2156`	`27177c0`
release_version	1.49.0-SNAPSHOT~e8a21567c5	1.49.0-SNAPSHOT~27177c01ea

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1745967996	1745967996
ci_job_id	918053144	918053144
ci_pipeline_id	63688224	63688224
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-tsd2jdn2-project-304-concurrent-0-laa74cvd 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-tsd2jdn2-project-304-concurrent-0-laa74cvd 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 8 performance improvements and 24 performance regressions! Performance is the same for 34 metrics, 5 unstable metrics.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:startup:insecure-bank:iast:Agent.start	worse [+42.716ms; +50.261ms] or [+3.755%; +4.418%]	1.184s	1.138s
scenario:startup:insecure-bank:iast:BytebuddyAgent	worse [+45.045ms; +50.301ms] or [+5.696%; +6.361%]	838.471ms	790.798ms
scenario:startup:insecure-bank:iast:Debugger	better [-1.758ms; -1.566ms] or [-29.858%; -26.605%]	4.225ms	5.886ms
scenario:startup:insecure-bank:iast:Telemetry	worse [+626.513µs; +910.403µs] or [+7.947%; +11.549%]	8.652ms	7.883ms
scenario:startup:insecure-bank:iast_HARDCODED_SECRET_DISABLED:Agent.start	worse [+47.007ms; +53.114ms] or [+4.142%; +4.680%]	1.185s	1.135s
scenario:startup:insecure-bank:iast_HARDCODED_SECRET_DISABLED:BytebuddyAgent	worse [+48.093ms; +52.591ms] or [+6.098%; +6.669%]	838.976ms	788.634ms
scenario:startup:insecure-bank:iast_HARDCODED_SECRET_DISABLED:Debugger	better [-1.629ms; -1.412ms] or [-28.025%; -24.297%]	4.292ms	5.812ms
scenario:startup:insecure-bank:iast_HARDCODED_SECRET_DISABLED:Telemetry	worse [+637.638µs; +840.784µs] or [+8.038%; +10.599%]	8.672ms	7.933ms
scenario:startup:insecure-bank:iast_TELEMETRY_OFF:Agent.start	worse [+38.333ms; +70.314ms] or [+3.379%; +6.199%]	1.189s	1.134s
scenario:startup:insecure-bank:iast_TELEMETRY_OFF:BytebuddyAgent	worse [+43.852ms; +67.850ms] or [+5.571%; +8.619%]	843.053ms	787.202ms
scenario:startup:insecure-bank:iast_TELEMETRY_OFF:Debugger	better [-1.903ms; -1.715ms] or [-31.971%; -28.816%]	4.143ms	5.952ms
scenario:startup:insecure-bank:iast_TELEMETRY_OFF:Telemetry	worse [+706.153µs; +1160.367µs] or [+9.133%; +15.007%]	8.665ms	7.732ms
scenario:startup:insecure-bank:tracing:Agent.start	worse [+27.575ms; +54.596ms] or [+2.718%; +5.381%]	1.056s	1.015s
scenario:startup:insecure-bank:tracing:BytebuddyAgent	worse [+36.505ms; +52.931ms] or [+5.401%; +7.832%]	720.574ms	675.856ms
scenario:startup:petclinic:appsec:Agent.start	worse [+43.492ms; +67.143ms] or [+3.789%; +5.850%]	1.203s	1.148s
scenario:startup:petclinic:appsec:BytebuddyAgent	worse [+47.957ms; +62.999ms] or [+6.970%; +9.156%]	743.511ms	688.033ms
scenario:startup:petclinic:appsec:Debugger	better [-1.636ms; -1.542ms] or [-28.192%; -26.564%]	4.215ms	5.804ms
scenario:startup:petclinic:appsec:Telemetry	worse [+775.357µs; +887.140µs] or [+10.541%; +12.061%]	8.187ms	7.356ms
scenario:startup:petclinic:iast:Agent.start	worse [+42.809ms; +57.840ms] or [+3.758%; +5.077%]	1.189s	1.139s
scenario:startup:petclinic:iast:BytebuddyAgent	worse [+45.055ms; +52.862ms] or [+5.688%; +6.673%]	841.089ms	792.130ms
scenario:startup:petclinic:iast:Debugger	better [-1.776ms; -1.500ms] or [-30.047%; -25.370%]	4.274ms	5.912ms
scenario:startup:petclinic:iast:Telemetry	worse [+592.581µs; +1089.831µs] or [+7.499%; +13.792%]	8.743ms	7.902ms
scenario:startup:petclinic:profiling:Agent.start	worse [+34.270ms; +44.433ms] or [+2.733%; +3.543%]	1.293s	1.254s
scenario:startup:petclinic:profiling:ProfilingAgent	worse [+4.991ms; +7.024ms] or [+5.213%; +7.336%]	101.744ms	95.737ms
scenario:startup:petclinic:profiling:BytebuddyAgent	worse [+43.906ms; +52.674ms] or [+6.612%; +7.933%]	712.321ms	664.031ms
scenario:startup:petclinic:profiling:GlobalTracer	better [-15.200ms; -11.374ms] or [-4.055%; -3.034%]	361.595ms	374.882ms
scenario:startup:petclinic:profiling:Debugger	better [-1.732ms; -1.603ms] or [-28.511%; -26.389%]	4.408ms	6.075ms
scenario:startup:petclinic:profiling:Telemetry	worse [+724.650µs; +983.075µs] or [+8.899%; +12.073%]	8.997ms	8.143ms
scenario:startup:petclinic:profiling:Profiling	worse [+4.994ms; +7.028ms] or [+5.215%; +7.339%]	101.772ms	95.761ms
scenario:startup:petclinic:tracing:Agent.start	worse [+46.553ms; +57.029ms] or [+4.632%; +5.674%]	1.057s	1.005s
scenario:startup:petclinic:tracing:BytebuddyAgent	worse [+48.527ms; +53.657ms] or [+7.240%; +8.005%]	721.350ms	670.258ms
scenario:startup:petclinic:tracing:Debugger	better [-1.779ms; -1.649ms] or [-29.111%; -26.988%]	4.397ms	6.111ms

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.49.0-SNAPSHOT~27177c01ea, baseline=1.49.0-SNAPSHOT~e8a21567c5

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.015 s) : 0, 1014514
Total [baseline] (8.671 s) : 0, 8670825
Agent [candidate] (1.056 s) : 0, 1055599
Total [candidate] (8.674 s) : 0, 8673956
section iast
Agent [baseline] (1.138 s) : 0, 1137519
Total [baseline] (9.215 s) : 0, 9215467
Agent [candidate] (1.184 s) : 0, 1184007
Total [candidate] (9.257 s) : 0, 9256591
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.135 s) : 0, 1134881
Total [baseline] (9.156 s) : 0, 9155585
Agent [candidate] (1.185 s) : 0, 1184942
Total [candidate] (9.231 s) : 0, 9231127
section iast_TELEMETRY_OFF
Agent [baseline] (1.134 s) : 0, 1134352
Total [baseline] (9.173 s) : 0, 9173070
Agent [candidate] (1.189 s) : 0, 1188675
Total [candidate] (9.274 s) : 0, 9274014

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.015 s	-
Agent	iast	1.138 s	123.005 ms (12.1%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.135 s	120.368 ms (11.9%)
Agent	iast_TELEMETRY_OFF	1.134 s	119.838 ms (11.8%)
Total	tracing	8.671 s	-
Total	iast	9.215 s	544.642 ms (6.3%)
Total	iast_HARDCODED_SECRET_DISABLED	9.156 s	484.76 ms (5.6%)
Total	iast_TELEMETRY_OFF	9.173 s	502.245 ms (5.8%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.056 s	-
Agent	iast	1.184 s	128.408 ms (12.2%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.185 s	129.343 ms (12.3%)
Agent	iast_TELEMETRY_OFF	1.189 s	133.076 ms (12.6%)
Total	tracing	8.674 s	-
Total	iast	9.257 s	582.635 ms (6.7%)
Total	iast_HARDCODED_SECRET_DISABLED	9.231 s	557.171 ms (6.4%)
Total	iast_TELEMETRY_OFF	9.274 s	600.058 ms (6.9%)

gantt
    title insecure-bank - break down per module: candidate=1.49.0-SNAPSHOT~27177c01ea, baseline=1.49.0-SNAPSHOT~e8a21567c5

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (675.856 ms) : 0, 675856
BytebuddyAgent [candidate] (720.574 ms) : 0, 720574
GlobalTracer [baseline] (242.217 ms) : 0, 242217
GlobalTracer [candidate] (239.391 ms) : 0, 239391
AppSec [baseline] (55.392 ms) : 0, 55392
AppSec [candidate] (55.144 ms) : 0, 55144
Debugger [baseline] (7.631 ms) : 0, 7631
Debugger [candidate] (5.082 ms) : 0, 5082
Remote Config [baseline] (729.7 µs) : 0, 730
Remote Config [candidate] (716.048 µs) : 0, 716
Telemetry [baseline] (9.321 ms) : 0, 9321
Telemetry [candidate] (11.395 ms) : 0, 11395
section iast
BytebuddyAgent [baseline] (790.798 ms) : 0, 790798
BytebuddyAgent [candidate] (838.471 ms) : 0, 838471
GlobalTracer [baseline] (229.73 ms) : 0, 229730
GlobalTracer [candidate] (229.935 ms) : 0, 229935
AppSec [baseline] (56.366 ms) : 0, 56366
AppSec [candidate] (55.173 ms) : 0, 55173
Debugger [baseline] (5.886 ms) : 0, 5886
Debugger [candidate] (4.225 ms) : 0, 4225
Remote Config [baseline] (581.171 µs) : 0, 581
Remote Config [candidate] (603.37 µs) : 0, 603
Telemetry [baseline] (7.883 ms) : 0, 7883
Telemetry [candidate] (8.652 ms) : 0, 8652
IAST [baseline] (22.81 ms) : 0, 22810
IAST [candidate] (23.525 ms) : 0, 23525
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (788.634 ms) : 0, 788634
BytebuddyAgent [candidate] (838.976 ms) : 0, 838976
GlobalTracer [baseline] (229.676 ms) : 0, 229676
GlobalTracer [candidate] (229.782 ms) : 0, 229782
AppSec [baseline] (56.058 ms) : 0, 56058
AppSec [candidate] (53.815 ms) : 0, 53815
Debugger [baseline] (5.812 ms) : 0, 5812
Debugger [candidate] (4.292 ms) : 0, 4292
Remote Config [baseline] (597.411 µs) : 0, 597
Remote Config [candidate] (618.091 µs) : 0, 618
Telemetry [baseline] (7.933 ms) : 0, 7933
Telemetry [candidate] (8.672 ms) : 0, 8672
IAST [baseline] (22.724 ms) : 0, 22724
IAST [candidate] (25.309 ms) : 0, 25309
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (787.202 ms) : 0, 787202
BytebuddyAgent [candidate] (843.053 ms) : 0, 843053
GlobalTracer [baseline] (230.629 ms) : 0, 230629
GlobalTracer [candidate] (230.577 ms) : 0, 230577
AppSec [baseline] (56.442 ms) : 0, 56442
AppSec [candidate] (55.727 ms) : 0, 55727
Debugger [baseline] (5.952 ms) : 0, 5952
Debugger [candidate] (4.143 ms) : 0, 4143
Remote Config [baseline] (603.709 µs) : 0, 604
Remote Config [candidate] (621.166 µs) : 0, 621
Telemetry [baseline] (7.732 ms) : 0, 7732
Telemetry [candidate] (8.665 ms) : 0, 8665
IAST [baseline] (22.433 ms) : 0, 22433
IAST [candidate] (22.366 ms) : 0, 22366

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.49.0-SNAPSHOT~27177c01ea, baseline=1.49.0-SNAPSHOT~e8a21567c5

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.005 s) : 0, 1005084
Total [baseline] (10.475 s) : 0, 10474719
Agent [candidate] (1.057 s) : 0, 1056875
Total [candidate] (10.515 s) : 0, 10514548
section appsec
Agent [baseline] (1.148 s) : 0, 1147726
Total [baseline] (10.665 s) : 0, 10664666
Agent [candidate] (1.203 s) : 0, 1203043
Total [candidate] (10.822 s) : 0, 10821973
section iast
Agent [baseline] (1.139 s) : 0, 1139153
Total [baseline] (10.903 s) : 0, 10903280
Agent [candidate] (1.189 s) : 0, 1189477
Total [candidate] (11.097 s) : 0, 11096664
section profiling
Agent [baseline] (1.254 s) : 0, 1253965
Total [baseline] (10.786 s) : 0, 10785955
Agent [candidate] (1.293 s) : 0, 1293316
Total [candidate] (11.016 s) : 0, 11016391

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.005 s	-
Agent	appsec	1.148 s	142.642 ms (14.2%)
Agent	iast	1.139 s	134.069 ms (13.3%)
Agent	profiling	1.254 s	248.881 ms (24.8%)
Total	tracing	10.475 s	-
Total	appsec	10.665 s	189.947 ms (1.8%)
Total	iast	10.903 s	428.561 ms (4.1%)
Total	profiling	10.786 s	311.235 ms (3.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.057 s	-
Agent	appsec	1.203 s	146.168 ms (13.8%)
Agent	iast	1.189 s	132.602 ms (12.5%)
Agent	profiling	1.293 s	236.441 ms (22.4%)
Total	tracing	10.515 s	-
Total	appsec	10.822 s	307.426 ms (2.9%)
Total	iast	11.097 s	582.116 ms (5.5%)
Total	profiling	11.016 s	501.843 ms (4.8%)

gantt
    title petclinic - break down per module: candidate=1.49.0-SNAPSHOT~27177c01ea, baseline=1.49.0-SNAPSHOT~e8a21567c5

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (670.258 ms) : 0, 670258
BytebuddyAgent [candidate] (721.35 ms) : 0, 721350
GlobalTracer [baseline] (239.978 ms) : 0, 239978
GlobalTracer [candidate] (239.79 ms) : 0, 239790
AppSec [baseline] (54.657 ms) : 0, 54657
AppSec [candidate] (55.218 ms) : 0, 55218
Debugger [baseline] (6.111 ms) : 0, 6111
Debugger [candidate] (4.397 ms) : 0, 4397
Remote Config [baseline] (706.643 µs) : 0, 707
Remote Config [candidate] (700.012 µs) : 0, 700
Telemetry [baseline] (10.019 ms) : 0, 10019
Telemetry [candidate] (11.939 ms) : 0, 11939
section appsec
BytebuddyAgent [baseline] (688.033 ms) : 0, 688033
BytebuddyAgent [candidate] (743.511 ms) : 0, 743511
GlobalTracer [baseline] (236.281 ms) : 0, 236281
GlobalTracer [candidate] (236.811 ms) : 0, 236811
AppSec [baseline] (175.41 ms) : 0, 175410
AppSec [candidate] (175.127 ms) : 0, 175127
Debugger [baseline] (5.804 ms) : 0, 5804
Debugger [candidate] (4.215 ms) : 0, 4215
Remote Config [baseline] (623.952 µs) : 0, 624
Remote Config [candidate] (625.416 µs) : 0, 625
Telemetry [baseline] (7.356 ms) : 0, 7356
Telemetry [candidate] (8.187 ms) : 0, 8187
IAST [baseline] (21.682 ms) : 0, 21682
IAST [candidate] (22.004 ms) : 0, 22004
section iast
BytebuddyAgent [baseline] (792.13 ms) : 0, 792130
BytebuddyAgent [candidate] (841.089 ms) : 0, 841089
GlobalTracer [baseline] (230.116 ms) : 0, 230116
GlobalTracer [candidate] (231.896 ms) : 0, 231896
AppSec [baseline] (56.253 ms) : 0, 56253
AppSec [candidate] (56.295 ms) : 0, 56295
Debugger [baseline] (5.912 ms) : 0, 5912
Debugger [candidate] (4.274 ms) : 0, 4274
Remote Config [baseline] (587.776 µs) : 0, 588
Remote Config [candidate] (615.228 µs) : 0, 615
Telemetry [baseline] (7.902 ms) : 0, 7902
Telemetry [candidate] (8.743 ms) : 0, 8743
IAST [baseline] (22.724 ms) : 0, 22724
IAST [candidate] (23.111 ms) : 0, 23111
section profiling
BytebuddyAgent [baseline] (664.031 ms) : 0, 664031
BytebuddyAgent [candidate] (712.321 ms) : 0, 712321
GlobalTracer [baseline] (374.882 ms) : 0, 374882
GlobalTracer [candidate] (361.595 ms) : 0, 361595
AppSec [baseline] (54.127 ms) : 0, 54127
AppSec [candidate] (53.337 ms) : 0, 53337
Debugger [baseline] (6.075 ms) : 0, 6075
Debugger [candidate] (4.408 ms) : 0, 4408
Remote Config [baseline] (644.93 µs) : 0, 645
Remote Config [candidate] (661.296 µs) : 0, 661
Telemetry [baseline] (8.143 ms) : 0, 8143
Telemetry [candidate] (8.997 ms) : 0, 8997
ProfilingAgent [baseline] (95.737 ms) : 0, 95737
ProfilingAgent [candidate] (101.744 ms) : 0, 101744
Profiling [baseline] (95.761 ms) : 0, 95761
Profiling [candidate] (101.772 ms) : 0, 101772

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-04-29T22:36:53	2025-04-29T22:44:36
git_branch	master	gary/add-llm-obs-api
git_commit_date	1745958763	1745965032
git_commit_sha	`e8a2156`	`27177c0`
release_version	1.49.0-SNAPSHOT~e8a21567c5	1.49.0-SNAPSHOT~27177c01ea
start_time	2025-04-29T22:36:38	2025-04-29T22:44:22

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1745967078	1745967078
ci_job_id	918053145	918053145
ci_pipeline_id	63688224	63688224
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-tsd2jdn2-project-304-concurrent-1-5p1cojp2 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-tsd2jdn2-project-304-concurrent-1-5p1cojp2 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 18 unstable metrics.

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.49.0-SNAPSHOT~27177c01ea, baseline=1.49.0-SNAPSHOT~e8a21567c5
    dateFormat X
    axisFormat %s
section baseline
no_agent
10000
 (381.538 µs) : 362, 401
.   : milestone, 382,
iast (524.519 µs) : 502, 547
.   : milestone, 525,
iast_FULL (736.455 µs) : 713, 760
.   : milestone, 736,
iast_GLOBAL (571.371 µs) : 548, 594
.   : milestone, 571,
iast_HARDCODED_SECRET_DISABLED (512.874 µs) : 490, 536
.   : milestone, 513,
iast_INACTIVE (470.395 µs) : 448, 493
.   : milestone, 470,
iast_TELEMETRY_OFF (516.565 µs) : 494, 539
.   : milestone, 517,
tracing (455.345 µs) : 434, 477
.   : milestone, 455,
section candidate
no_agent (376.101 µs) : 356, 396
.   : milestone, 376,
iast (519.522 µs) : 498, 542
.   : milestone, 520,
iast_FULL (729.133 µs) : 707, 751
.   : milestone, 729,
iast_GLOBAL (564.721 µs) : 543, 586
.   : milestone, 565,
iast_HARDCODED_SECRET_DISABLED (512.141 µs) : 491, 534
.   : milestone, 512,
iast_INACTIVE (469.599 µs) : 448, 491
.   : milestone, 470,
iast_TELEMETRY_OFF (502.811 µs) : 481, 525
.   : milestone, 503,
tracing (462.07 µs) : 441, 483
.   : milestone, 462,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	381.538 µs [361.712 µs, 401.365 µs]	-
iast	524.519 µs [502.141 µs, 546.897 µs]	142.981 µs (37.5%)
iast_FULL	736.455 µs [712.889 µs, 760.021 µs]	354.917 µs (93.0%)
iast_GLOBAL	571.371 µs [548.434 µs, 594.308 µs]	189.833 µs (49.8%)
iast_HARDCODED_SECRET_DISABLED	512.874 µs [489.998 µs, 535.75 µs]	131.336 µs (34.4%)
iast_INACTIVE	470.395 µs [448.194 µs, 492.596 µs]	88.857 µs (23.3%)
iast_TELEMETRY_OFF	516.565 µs [493.837 µs, 539.292 µs]	135.026 µs (35.4%)
tracing	455.345 µs [433.771 µs, 476.919 µs]	73.807 µs (19.3%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	376.101 µs [356.319 µs, 395.882 µs]	-
iast	519.522 µs [497.512 µs, 541.533 µs]	143.422 µs (38.1%)
iast_FULL	729.133 µs [707.209 µs, 751.057 µs]	353.032 µs (93.9%)
iast_GLOBAL	564.721 µs [543.074 µs, 586.368 µs]	188.62 µs (50.2%)
iast_HARDCODED_SECRET_DISABLED	512.141 µs [490.557 µs, 533.725 µs]	136.04 µs (36.2%)
iast_INACTIVE	469.599 µs [447.986 µs, 491.212 µs]	93.498 µs (24.9%)
iast_TELEMETRY_OFF	502.811 µs [480.986 µs, 524.637 µs]	126.711 µs (33.7%)
tracing	462.07 µs [440.944 µs, 483.197 µs]	85.97 µs (22.9%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.49.0-SNAPSHOT~27177c01ea, baseline=1.49.0-SNAPSHOT~e8a21567c5
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.358 ms) : 1339, 1378
.   : milestone, 1358,
appsec (1.73 ms) : 1706, 1754
.   : milestone, 1730,
appsec_no_iast (1.735 ms) : 1711, 1759
.   : milestone, 1735,
code_origins (1.645 ms) : 1618, 1672
.   : milestone, 1645,
iast (1.512 ms) : 1487, 1537
.   : milestone, 1512,
profiling (1.542 ms) : 1517, 1566
.   : milestone, 1542,
tracing (1.497 ms) : 1472, 1522
.   : milestone, 1497,
section candidate
no_agent (1.356 ms) : 1336, 1376
.   : milestone, 1356,
appsec (1.743 ms) : 1720, 1767
.   : milestone, 1743,
appsec_no_iast (1.738 ms) : 1715, 1761
.   : milestone, 1738,
code_origins (1.696 ms) : 1668, 1723
.   : milestone, 1696,
iast (1.534 ms) : 1510, 1558
.   : milestone, 1534,
profiling (1.589 ms) : 1565, 1614
.   : milestone, 1589,
tracing (1.522 ms) : 1498, 1547
.   : milestone, 1522,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.358 ms [1.339 ms, 1.378 ms]	-
appsec	1.73 ms [1.706 ms, 1.754 ms]	371.855 µs (27.4%)
appsec_no_iast	1.735 ms [1.711 ms, 1.759 ms]	376.351 µs (27.7%)
code_origins	1.645 ms [1.618 ms, 1.672 ms]	286.644 µs (21.1%)
iast	1.512 ms [1.487 ms, 1.537 ms]	153.774 µs (11.3%)
profiling	1.542 ms [1.517 ms, 1.566 ms]	183.167 µs (13.5%)
tracing	1.497 ms [1.472 ms, 1.522 ms]	138.782 µs (10.2%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.356 ms [1.336 ms, 1.376 ms]	-
appsec	1.743 ms [1.72 ms, 1.767 ms]	387.508 µs (28.6%)
appsec_no_iast	1.738 ms [1.715 ms, 1.761 ms]	382.51 µs (28.2%)
code_origins	1.696 ms [1.668 ms, 1.723 ms]	339.962 µs (25.1%)
iast	1.534 ms [1.51 ms, 1.558 ms]	178.326 µs (13.2%)
profiling	1.589 ms [1.565 ms, 1.614 ms]	233.633 µs (17.2%)
tracing	1.522 ms [1.498 ms, 1.547 ms]	166.539 µs (12.3%)

Dacapo

mcculls · 2025-01-02T20:52:04Z

dd-trace-api/src/main/java/datadog/trace/api/llmobs/LLMObs.java

+  private static LLMObsSpanFactory SPAN_FACTORY = NoOpLLMObsSpanFactory.INSTANCE;
+
+  /**
+   * This a hook for injecting SpanFactory implementation. It should only be used internally by


FYI, internal APIs should be added to the internal-api module.

You can keep the same package structure and they will still be visible to the tracer internals.

ah gotcha, thanks!

github-actions · 2025-01-14T13:24:28Z

Hi! 👋 Looks like you updated a Git Submodule.
If this was not intentional please make sure to:

Update the submodule to the latest commit on the master branch using the git submodule update command,
Check you properly set up your environment for contributing.

github-actions · 2025-01-14T13:27:35Z

Hi! 👋 Looks like you updated a Git Submodule.
If this was not intentional please make sure to:

Update the submodule to the latest commit on the master branch using the git submodule update command,
Check you properly set up your environment for contributing.

github-actions · 2025-04-23T12:04:16Z

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

nayeem-kamal

Overall LGTM. I would just address the comment left by Stuart earlier in the PR. Is there a reason this api isn't in internal-api?

PerfectSlayer

Left few comments about API design.

Can you also add your GitHub team as the owner of this new llmobs package from internal API?

PerfectSlayer · 2025-04-27T06:58:54Z

dd-trace-api/src/main/java/datadog/trace/api/llmobs/LLMObs.java

+      @Nullable String mlApp,
+      @Nullable String sessionID) {
+
+    return SPAN_FACTORY.startLLMSpan(spanName, modelName, modelProvider, mlApp, sessionID);


Suggested change

return SPAN_FACTORY.startLLMSpan(spanName, modelName, modelProvider, mlApp, sessionID);

return SPAN_FACTORY.startLLMSpan(spanName, modelName, modelProvider, mlApp, sessionId);

We usually follow the Java naming convention across the code base using xxxId rather than xxxID (like span, trace, etc...).

PerfectSlayer · 2025-04-27T07:00:23Z

dd-trace-api/src/main/java/datadog/trace/api/llmobs/LLMObs.java

+public class LLMObs {
+  protected LLMObs() {}


Should it a public final class with private constructor if it’s only an helper class?

https://github.com/DataDog/dd-trace-java/pull/8390/files#diff-ee71e1544f39cc6c551488ad74b877a819b780b9d07c2983be13b80161958aa9R1-R10

i extend this class here to allow setting an implemented, non-noop span factory (and in another PR, another factory)

i would say that this is not exactly only a helper class, this is the entry point of the SDK manual API and the calls are registered with the llmobs agent (as seen in #8390)

gary-huang · 2025-04-28T19:58:31Z

Overall LGTM. I would just address the comment left by Stuart earlier in the PR. Is there a reason this api isn't in internal-api?

@nayeem-kamal what Stuart pointed out was a previous function that allows the setting of the span factory - which is meant to be kept internal and out of user's control, my previous implementation had that function in the api package, but that has been moved since https://github.com/DataDog/dd-trace-java/pull/8390/files#diff-ee71e1544f39cc6c551488ad74b877a819b780b9d07c2983be13b80161958aa9R1-R10

these APIs are meant to be for users' to start llm obs spans, so no, they are not meant to be in internal-api

* add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners

* add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * spotless * add APIs for llm obs sdk (#8135) * add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners * rename ID to Id according to java naming conventions * Undo change to integrations-core submodule * fix build gradle * rm empty line * fix test

* add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners

* add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * spotless * add APIs for llm obs sdk (#8135) * add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners * rename ID to Id according to java naming conventions * Undo change to integrations-core submodule * fix build gradle * rm empty line * fix test

* add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners

* add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * spotless * add APIs for llm obs sdk (#8135) * add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners * rename ID to Id according to java naming conventions * Undo change to integrations-core submodule * fix build gradle * rm empty line * fix test

* add APIs for llm obs sdk (#8135) * add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners * implement LLM Obs SDK spans APIs (#8390) * add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * spotless * add APIs for llm obs sdk (#8135) * add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners * rename ID to Id according to java naming conventions * Undo change to integrations-core submodule * fix build gradle * rm empty line * fix test * LLM Obs SDK Mapper (#8372) * add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * impl llmobs agent and llmobs apis * impl llmobs agent * working writer * add support for llm message and tool calls * cleaned up whitespace * resolve merge conflicts * remaining merge conflicts * fix bad method call * fixed llmobs intake creation if llmobs not enabled * removed print statements * added tests for llmobsspanmapper * fixed coverage for tags --------- Co-authored-by: Nayeem Kamal <nayeem.kamal@datadoghq.com> * updated to master submodule * LLM Obs SDK use context API for parent children span linkage (#8711) * add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * impl llmobs agent and llmobs apis * impl llmobs agent * working writer * add support for llm message and tool calls * impl llmobs agent and llmobs apis * use new ctx api to track parent span * cleaned up whitespace * resolve merge conflicts * remaining merge conflicts * fix bad method call * fixed llmobs intake creation if llmobs not enabled * removed print statements * ran spotless * added tests for llmobsspanmapper * fixed coverage for tags --------- Co-authored-by: Nayeem Kamal <nayeem.kamal@datadoghq.com> Co-authored-by: Nayeem Kamal <kamal.nayeem12@gmail.com> * LLM Obs SDK evaluation metrics submission (#8688) * add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * impl llmobs agent and llmobs apis * impl llmobs agent * working writer * add support for llm message and tool calls * impl llmobs agent and llmobs apis * use new ctx api to track parent span * add api for evals * working impl supporting both agentless and agent * handle null tags and default to default ml app if null or empty string provided in the override * cleaned up whitespace * resolve merge conflicts * remaining merge conflicts * fix bad method call * fixed llmobs intake creation if llmobs not enabled * removed print statements * ran spotless * ran spotless * added tests for llmobsspanmapper * fixed coverage for tags --------- Co-authored-by: Nayeem Kamal <nayeem.kamal@datadoghq.com> Co-authored-by: Nayeem Kamal <kamal.nayeem12@gmail.com> * fix CODEOWNERS --------- Co-authored-by: Nayeem Kamal <nayeem.kamal@datadoghq.com> Co-authored-by: Nayeem Kamal <kamal.nayeem12@gmail.com>

gary-huang force-pushed the gary/add-llm-obs-api branch from 84eb184 to 85d0946 Compare December 30, 2024 16:41

gary-huang force-pushed the gary/add-llm-obs-api branch from 85d0946 to dd637fd Compare January 2, 2025 19:08

mcculls reviewed Jan 2, 2025

View reviewed changes

mcculls force-pushed the gary/add-llmobs-configs branch from 14f71b3 to 5d90450 Compare January 7, 2025 09:35

Base automatically changed from gary/add-llmobs-configs to master January 7, 2025 15:36

gary-huang force-pushed the gary/add-llm-obs-api branch from dd637fd to 85debfc Compare January 14, 2025 13:24

gary-huang force-pushed the gary/add-llm-obs-api branch from 85debfc to 98060a9 Compare January 14, 2025 13:27

gary-huang force-pushed the gary/add-llm-obs-api branch from 98060a9 to 62bc9a0 Compare January 14, 2025 13:28

gary-huang force-pushed the gary/add-llm-obs-api branch from 2b23b61 to 08b677e Compare January 30, 2025 21:43

gary-huang force-pushed the gary/add-llm-obs-api branch from 08b677e to 5b12d55 Compare February 12, 2025 06:48

gary-huang force-pushed the gary/add-llm-obs-api branch from 5b12d55 to 77ddec8 Compare February 20, 2025 21:15

gary-huang force-pushed the gary/add-llm-obs-api branch 6 times, most recently from c1f63dc to 6e250a2 Compare March 7, 2025 13:34

gary-huang force-pushed the gary/add-llm-obs-api branch from 6e250a2 to d16b87d Compare March 26, 2025 01:14

gary-huang added 2 commits April 10, 2025 15:09

add APIs for llm obs

5134311

add llm message class to support llm spans

b5c7a0a

gary-huang force-pushed the gary/add-llm-obs-api branch from d16b87d to b5c7a0a Compare April 10, 2025 19:09

gary-huang marked this pull request as ready for review April 23, 2025 12:04

gary-huang requested a review from a team as a code owner April 23, 2025 12:04

gary-huang requested a review from PerfectSlayer April 23, 2025 12:04

gary-huang changed the title ~~add APIs for llm obs~~ add APIs for llm obs sdk Apr 23, 2025

gary-huang added the comp: mlobs ML Observability (LLMObs) label Apr 23, 2025

gary-huang added tag: experimental Experimental changes type: enhancement Enhancements and improvements labels Apr 23, 2025

gary-huang changed the base branch from master to gary/llmobs-sdk-merge April 23, 2025 12:12

gary-huang mentioned this pull request Apr 23, 2025

implement LLM Obs SDK spans APIs #8390

Merged

nayeem-kamal reviewed Apr 25, 2025

View reviewed changes

PerfectSlayer reviewed Apr 27, 2025

View reviewed changes

gary-huang added 2 commits April 29, 2025 16:57

follow java convention of naming Id instead of ID

4126e2e

add codeowners

27177c0

nayeem-kamal approved these changes May 5, 2025

View reviewed changes

gary-huang merged commit 82b8418 into gary/llmobs-sdk-merge May 7, 2025
188 checks passed

gary-huang deleted the gary/add-llm-obs-api branch May 7, 2025 15:08

gary-huang added a commit that referenced this pull request May 14, 2025

add APIs for llm obs sdk (#8135)

2be1b73

* add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners

gary-huang added a commit that referenced this pull request Jun 10, 2025

add APIs for llm obs sdk (#8135)

e2755fe

* add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners

gary-huang added a commit that referenced this pull request Jul 9, 2025

add APIs for llm obs sdk (#8135)

ed68795

* add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add APIs for llm obs sdk #8135

add APIs for llm obs sdk #8135

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

	return SPAN_FACTORY.startLLMSpan(spanName, modelName, modelProvider, mlApp, sessionID);
	return SPAN_FACTORY.startLLMSpan(spanName, modelName, modelProvider, mlApp, sessionId);

add APIs for llm obs sdk #8135

add APIs for llm obs sdk #8135

Uh oh!

Conversation

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!