Skip to content

Tracing

The run view shows where each step spent its time. You get some of that for free, more if you send Argo’s spans to the connector, and links into your own Jaeger or Tempo if you run one.

Level What you do What you get
One Install the connector Each step’s pod wait drawn apart from its working time
Two Turn on the connector’s receiver and point the controller and executor at it Pod creation, artifact load, artifact save and log save times for each step
Three Name your tracing backend in the connector’s Helm values A link from the run and from each step into your Jaeger or Tempo

Level two is enough for most teams. Add level three if your team already works in a tracing backend.

The connector reads when each step’s main container started, so the run timeline can split pod wait from work. A step that sat unschedulable for half an hour and then ran for ninety seconds draws as two segments.

It needs the chart’s status feature, which is on by default, and no spans. If the pod is gone before the connector reads it, the Pod wait column stays empty.

Level two: the connector receives the spans

Section titled “Level two: the connector receives the spans”

Send Argo’s spans to the connector and each step’s row splits its working time into pod creation, artifact load, artifact save and log save.

Turn the receiver on with Helm. It’s off by default, and it needs features.status on or the install fails:

Terminal window
helm upgrade connector oci://ghcr.io/dagweave/charts/connector \
--namespace connector-system --reset-then-reuse-values \
--set otlpReceiver.enabled=true

You get a ClusterIP Service taking OTLP over gRPC only, on port 4317 (change it with otlpReceiver.port). It’s named after the release, so a release called connector gives connector-otlp.

Then point both the controller and the executor at it. The controller sends pod creation and the executor sends the rest, so if you only set the controller you get pod creation and nothing else. With the Argo Workflows Helm chart:

controller:
extraEnv:
- name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
value: http://connector-otlp.connector-system.svc:4317
- name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
value: grpc
executor:
env:
- name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
value: http://connector-otlp.connector-system.svc:4317
- name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
value: grpc

Without that chart, set the same variables on the controller Deployment and in the executor block of workflow-controller-configmap, as in Turn on span export. Restart the controller.

Only the four durations per step go to Dagweave. The raw spans stay in your cluster, there’s no collector to run, and the receiver isn’t reachable from outside.

To limit who can send spans, set otlpReceiver.networkPolicy.enabled and list the allowed sources in otlpReceiver.networkPolicy.from. Include the workflow pods if you trace the executor. An empty from fails the install.

Keep sending spans to your own backend and tell the ArgoConnection where it is. The run gets a Trace link and each step a View trace link. Dagweave doesn’t collect or store any of it.

The links need two things: the controller exporting spans, and the connection naming the backend.

Argo sends no spans until the workflow controller’s environment has an endpoint. There’s no tracing block in the controller ConfigMap.

Use OTEL_EXPORTER_OTLP_ENDPOINT for every signal or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT for traces only. The default protocol is gRPC. Set OTEL_EXPORTER_OTLP_PROTOCOL to http/protobuf if your collector takes HTTP.

On the controller Deployment:

spec:
template:
spec:
containers:
- name: workflow-controller
env:
- name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
value: http://otel-collector.observability:4318
- name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
value: http/protobuf

Or with the Argo Workflows Helm chart:

controller:
extraEnv:
- name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
value: http://otel-collector.observability:4318
- name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
value: http/protobuf

Restart the controller. It traces the next workflow it runs.

Dagweave’s links only need the controller’s workflow and node spans. If you want a step’s trace to show what happened inside it (the init, main and wait containers, and artifact and log transfers), set the same variables for the executor too. They go in the executor block of workflow-controller-configmap, or executor.env in the chart:

executor:
env:
- name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
value: http://otel-collector.observability:4318
- name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
value: http/protobuf

Argo’s tracing docs list every span. Argo calls tracing beta and may change it, so an Argo upgrade can break these links.

Set these on the ArgoConnection, under argoConnection.extraSpec in the connector’s Helm values, then upgrade:

  • traceBackend: jaeger or tempo.
  • traceBackendURL: the root of the UI, so Jaeger itself, or Grafana for Tempo. It has to be an http or https URL with a host and no query string, fragment or user info, or you get no links.
  • traceBackendDatasource: the UID of the Grafana datasource holding your Tempo traces. Tempo links need it.

The run’s Trace link opens the trace by ID when Argo has written one to the workflows.argoproj.io/trace-id annotation. Every other link searches on Argo’s span attributes: name and namespace on the workflow span, plus node_id on each node span. The controller’s service name is workflows-controller and the executor’s is argoexec.

argoConnection:
extraSpec:
traceBackend: jaeger
traceBackendURL: https://jaeger.dagweave.com

Save that as tracing.yaml and apply it:

Terminal window
helm upgrade connector oci://ghcr.io/dagweave/charts/connector \
--namespace connector-system --reset-then-reuse-values -f tracing.yaml

A step link, decoded:

https://jaeger.dagweave.com/search
?service=workflows-controller
&lookback=7d
&tags={"node_id":"hello-world-abc12-3092374","name":"hello-world-abc12","namespace":"argo"}

With a trace ID, the run link is https://jaeger.dagweave.com/trace/<trace ID>. Without one, it’s the same search minus node_id.

argoConnection:
extraSpec:
traceBackend: tempo
traceBackendURL: https://grafana.dagweave.com
traceBackendDatasource: tempo-prod

The datasource UID is the last part of the URL of the datasource’s settings page in Grafana, not its display name.

A step link opens Grafana Explore. Decoded:

https://grafana.dagweave.com/explore
?schemaVersion=1
&panes={"argo":{"datasource":"tempo-prod","queries":[{"refId":"A",
"datasource":{"type":"tempo","uid":"tempo-prod"},
"queryType":"traceql",
"query":"{span.node_id=\"hello-world-abc12-3092374\"&&span.name=\"hello-world-abc12\"&&span.namespace=\"argo\"}"}],
"range":{"from":"now-7d","to":"now"}}}

With a trace ID, the run link opens that trace. Without one, it’s the step search minus node_id, and you pick the trace from a list.

  • Tempo links use Grafana’s panes URL format with schemaVersion=1. A Grafana that doesn’t support it opens Explore empty, with no error.
  • Both search the last seven days. For an older run, widen the time range in your backend.
  • Jaeger links search on service=workflows-controller. If a collector rewrites service.name, they find nothing.
  • If clicking a link lands you on Grafana’s home page instead of the trace, Grafana’s session cookie is set to SameSite=Strict, so the browser drops it on a link from Dagweave. Set cookie_samesite = lax under [security] in Grafana’s config. Lax is Grafana’s default.

For another backend, or a Grafana with different Explore URLs, set traceURLTemplate (per step) and workflowTraceURLTemplate (per run) under argoConnection.extraSpec. They override the backend fields. Both take {namespace} and {name}, and the step one also takes {nodeId}. Values are URL-escaped.

The easy way to write one: run the search by hand in your backend, copy the address, and put the placeholders where the values were.

When the run view says there is nothing to open

Section titled “When the run view says there is nothing to open”

If a run has no trace links, the run view says why.

This cluster isn’t exporting traces. To turn them on, point OTEL_EXPORTER_OTLP_TRACES_ENDPOINT on Argo’s workflow controller at your collector.

None of the namespace’s recent finished workflows has a trace ID, so the links are hidden. Set the variables from Turn on span export and restart the controller. A namespace with no finished workflow yet gets the links anyway.

Argo 4.1 writes the trace ID even with no endpoint set, so a cluster that exports nothing can still get links. If one opens an empty search, check the controller’s OTEL_EXPORTER_OTLP_TRACES_ENDPOINT first.

Trace links aren’t set up for this cluster. Set traceBackend and traceBackendURL under argoConnection.extraSpec in the connector’s Helm values, plus traceBackendDatasource for Tempo, and each step will link to your Jaeger or Grafana.

The connection names no backend, or a field is missing or invalid. Fill in the fields above and reopen the run.

The browser also drops any link that isn’t http or https, including ones from the URL templates.