Tracing
The run view shows where each step spent its time. You get some of that for free, more if you send Argo’s spans to the connector, and links into your own Jaeger or Tempo if you run one.
| Level | What you do | What you get |
|---|---|---|
| One | Install the connector | Each step’s pod wait drawn apart from its working time |
| Two | Turn on the connector’s receiver and point the controller and executor at it | Pod creation, artifact load, artifact save and log save times for each step |
| Three | Name your tracing backend in the connector’s Helm values | A link from the run and from each step into your Jaeger or Tempo |
Level two is enough for most teams. Add level three if your team already works in a tracing backend.
Level one: nothing to configure
Section titled “Level one: nothing to configure”The connector reads when each step’s main container started, so the
run timeline can split pod wait from work. A step that sat
unschedulable for half an hour and then ran for ninety seconds draws as two
segments.
It needs the chart’s status feature, which is on by default, and no spans. If
the pod is gone before the connector reads it, the Pod wait column stays
empty.
Level two: the connector receives the spans
Section titled “Level two: the connector receives the spans”Send Argo’s spans to the connector and each step’s row splits its working time into pod creation, artifact load, artifact save and log save.
Turn the receiver on with Helm. It’s off by default, and it needs
features.status on or the install fails:
helm upgrade connector oci://ghcr.io/dagweave/charts/connector \ --namespace connector-system --reset-then-reuse-values \ --set otlpReceiver.enabled=trueYou get a ClusterIP Service taking OTLP over gRPC only, on port 4317 (change it
with otlpReceiver.port). It’s named after the release, so a release called
connector gives connector-otlp.
Then point both the controller and the executor at it. The controller sends pod creation and the executor sends the rest, so if you only set the controller you get pod creation and nothing else. With the Argo Workflows Helm chart:
controller: extraEnv: - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT value: http://connector-otlp.connector-system.svc:4317 - name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL value: grpcexecutor: env: - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT value: http://connector-otlp.connector-system.svc:4317 - name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL value: grpcWithout that chart, set the same variables on the controller Deployment and in
the executor block of workflow-controller-configmap, as in
Turn on span export. Restart the
controller.
Only the four durations per step go to Dagweave. The raw spans stay in your cluster, there’s no collector to run, and the receiver isn’t reachable from outside.
To limit who can send spans, set otlpReceiver.networkPolicy.enabled and list
the allowed sources in otlpReceiver.networkPolicy.from. Include the workflow
pods if you trace the executor. An empty from fails the install.
Level three: your own Jaeger or Tempo
Section titled “Level three: your own Jaeger or Tempo”Keep sending spans to your own backend and tell the ArgoConnection where it is.
The run gets a Trace link and each step a View trace link. Dagweave
doesn’t collect or store any of it.
The links need two things: the controller exporting spans, and the connection naming the backend.
Turn on span export in Argo Workflows
Section titled “Turn on span export in Argo Workflows”Argo sends no spans until the workflow controller’s environment has an endpoint.
There’s no tracing block in the controller ConfigMap.
Use OTEL_EXPORTER_OTLP_ENDPOINT for every signal or
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT for traces only. The default protocol is
gRPC. Set OTEL_EXPORTER_OTLP_PROTOCOL to http/protobuf if your collector
takes HTTP.
On the controller Deployment:
spec: template: spec: containers: - name: workflow-controller env: - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT value: http://otel-collector.observability:4318 - name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL value: http/protobufOr with the Argo Workflows Helm chart:
controller: extraEnv: - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT value: http://otel-collector.observability:4318 - name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL value: http/protobufRestart the controller. It traces the next workflow it runs.
Dagweave’s links only need the controller’s workflow and node spans. If you
want a step’s trace to show what happened inside it (the init, main and wait
containers, and artifact and log transfers), set the same variables for the
executor too. They go in the executor block of workflow-controller-configmap,
or executor.env in the chart:
executor: env: - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT value: http://otel-collector.observability:4318 - name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL value: http/protobufArgo’s tracing docs list every span. Argo calls tracing beta and may change it, so an Argo upgrade can break these links.
Point a connection at your backend
Section titled “Point a connection at your backend”Set these on the ArgoConnection,
under argoConnection.extraSpec in the connector’s Helm values, then upgrade:
traceBackend:jaegerortempo.traceBackendURL: the root of the UI, so Jaeger itself, or Grafana for Tempo. It has to be anhttporhttpsURL with a host and no query string, fragment or user info, or you get no links.traceBackendDatasource: the UID of the Grafana datasource holding your Tempo traces. Tempo links need it.
The run’s Trace link opens the trace by ID when Argo has written one to the
workflows.argoproj.io/trace-id annotation. Every other link searches on Argo’s
span attributes: name and namespace on the workflow span, plus node_id on
each node span. The controller’s service name is workflows-controller and
the executor’s is argoexec.
Jaeger
Section titled “Jaeger”argoConnection: extraSpec: traceBackend: jaeger traceBackendURL: https://jaeger.dagweave.comSave that as tracing.yaml and apply it:
helm upgrade connector oci://ghcr.io/dagweave/charts/connector \ --namespace connector-system --reset-then-reuse-values -f tracing.yamlA step link, decoded:
https://jaeger.dagweave.com/search ?service=workflows-controller &lookback=7d &tags={"node_id":"hello-world-abc12-3092374","name":"hello-world-abc12","namespace":"argo"}With a trace ID, the run link is https://jaeger.dagweave.com/trace/<trace ID>.
Without one, it’s the same search minus node_id.
Grafana Tempo
Section titled “Grafana Tempo”argoConnection: extraSpec: traceBackend: tempo traceBackendURL: https://grafana.dagweave.com traceBackendDatasource: tempo-prodThe datasource UID is the last part of the URL of the datasource’s settings page in Grafana, not its display name.
A step link opens Grafana Explore. Decoded:
https://grafana.dagweave.com/explore ?schemaVersion=1 &panes={"argo":{"datasource":"tempo-prod","queries":[{"refId":"A", "datasource":{"type":"tempo","uid":"tempo-prod"}, "queryType":"traceql", "query":"{span.node_id=\"hello-world-abc12-3092374\"&&span.name=\"hello-world-abc12\"&&span.namespace=\"argo\"}"}], "range":{"from":"now-7d","to":"now"}}}With a trace ID, the run link opens that trace. Without one, it’s the step search
minus node_id, and you pick the trace from a list.
Where these links can break
Section titled “Where these links can break”- Tempo links use Grafana’s
panesURL format withschemaVersion=1. A Grafana that doesn’t support it opens Explore empty, with no error. - Both search the last seven days. For an older run, widen the time range in your backend.
- Jaeger links search on
service=workflows-controller. If a collector rewritesservice.name, they find nothing. - If clicking a link lands you on Grafana’s home page instead of the trace,
Grafana’s session cookie is set to
SameSite=Strict, so the browser drops it on a link from Dagweave. Setcookie_samesite = laxunder[security]in Grafana’s config. Lax is Grafana’s default.
Writing the URL yourself
Section titled “Writing the URL yourself”For another backend, or a Grafana with different Explore URLs, set
traceURLTemplate (per step) and workflowTraceURLTemplate (per run) under
argoConnection.extraSpec. They override the backend fields. Both take
{namespace} and {name}, and the step one also takes {nodeId}. Values are
URL-escaped.
The easy way to write one: run the search by hand in your backend, copy the address, and put the placeholders where the values were.
When the run view says there is nothing to open
Section titled “When the run view says there is nothing to open”If a run has no trace links, the run view says why.
This cluster isn’t exporting traces. To turn them on, point
OTEL_EXPORTER_OTLP_TRACES_ENDPOINTon Argo’s workflow controller at your collector.
None of the namespace’s recent finished workflows has a trace ID, so the links are hidden. Set the variables from Turn on span export and restart the controller. A namespace with no finished workflow yet gets the links anyway.
Argo 4.1 writes the trace ID even with no endpoint set, so a cluster that exports
nothing can still get links. If one opens an empty search, check the
controller’s OTEL_EXPORTER_OTLP_TRACES_ENDPOINT first.
Trace links aren’t set up for this cluster. Set traceBackend and traceBackendURL under argoConnection.extraSpec in the connector’s Helm values, plus traceBackendDatasource for Tempo, and each step will link to your Jaeger or Grafana.
The connection names no backend, or a field is missing or invalid. Fill in the fields above and reopen the run.
The browser also drops any link that isn’t http or https, including ones from
the URL templates.