Tracing

You do not need Jaeger, Tempo, or any tracing backend to see where a run spent its time. There are three levels here and you can stop at whichever one you like.

	What you do	What you get
Nothing	Install the connector	Waiting drawn apart from working, on every step
Two settings	Point the controller and the executor at the connector	Every step broken into pod creation and artifact transfers
You already run one	Name your backend on the ArgoConnection	All of the above, plus a link into your own Jaeger or Tempo

Most people should stop after the second row. The third is worth it only if your team already lives in a tracing backend and wants to land there from a run.

Level one: nothing to configure

The connector reads each pod’s own start time, so dagweave can tell the wait before a container started apart from the work it did afterwards. A step that sat unschedulable for half an hour and then finished in ninety seconds draws as two segments, not one long bar.

Nothing to switch on. No spans involved, so this works on a cluster that exports no telemetry at all.

Level two: the connector receives the spans

Point span export at the connector instead of at a collector and the run timeline fills in the rest: how long the pod took to create, and how long artifacts took to load and save.

Two environment settings, not one. The controller emits the pod-creation span; the artifact and log spans come from the executor, which is a different process and needs telling separately. Set the export endpoint on both or you get pod creation alone.

Lock waits are not drawn here. Argo times each attempt to take a lock rather than the wait as a whole, so summing them would report milliseconds for a step that queued for ten minutes. The run view names the lock and its holder instead, which is the answer you wanted anyway.

The connector aggregates as spans arrive and sends dagweave a handful of durations per step. Raw spans never leave your cluster. There is no collector to run, no storage to size, and no port opened to the internet: the connector listens on a ClusterIP address that only the controller reaches.

Level three: your own Jaeger or Tempo

If you already run a tracing backend, keep sending spans to it and tell the ArgoConnection where it lives. Every step of a run then carries a View trace link, and the run carries one of its own.

dagweave does not collect or store traces on this path. Your spans go to your own collector and stay there.

Two things have to be true before those links show up. The controller has to be exporting spans, and the ArgoConnection has to name a backend. They are separate jobs, and the run view tells you which one is outstanding.

Turn on span export in Argo Workflows

Argo Workflows exports OpenTelemetry data once you tell it where to send it, and sends nothing until then. That is the sensible default: where a cluster’s telemetry goes is the cluster operator’s call, not an installer’s.

It reads the standard OpenTelemetry environment variables, so the switch sits in the workflow controller’s own environment. There is no tracing block in the controller ConfigMap to find. Set OTEL_EXPORTER_OTLP_ENDPOINT to cover every signal, or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT for traces alone. gRPC is the default protocol; point OTEL_EXPORTER_OTLP_PROTOCOL at http/protobuf if your collector speaks HTTP.

On the controller Deployment:

spec:
  template:
    spec:
      containers:
        - name: workflow-controller
          env:
            - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
              value: http://otel-collector.observability:4318
            - name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
              value: http/protobuf

Through the Argo Workflows Helm chart the same two variables go in controller.extraEnv:

controller:
  extraEnv:
    - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
      value: http://otel-collector.observability:4318
    - name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
      value: http/protobuf

Restart the controller and the next workflow it runs is traced.

That covers the workflow and node spans, which is everything the dagweave links search on. The executor, argoexec, emits the spans underneath them: the init, main and wait containers, and the artifact and log transfers around each one. It reads the same variables from the executor block of workflow-controller-configmap, which is executor.env in the chart. Set them there too and a step’s trace opens as a breakdown rather than a single bar.

executor:
  env:
    - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
      value: http://otel-collector.observability:4318
    - name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
      value: http/protobuf

Argo Workflows’ own tracing docs list every span it emits and the attributes on each one. Argo calls tracing beta and says the shape may change in an incompatible way, so treat everything on this page as tracking a moving target.

Point a connection at your backend

Argo Workflows does not write a trace id back onto the Workflow resource, so neither link can be a lookup by id. Both are a search against your backend, built out of what Argo stamps on its spans: name and namespace on the workflow span, and node_id alongside name, namespace and node_type on each node span. The controller reports itself to your backend under the service name workflows-controller, and the executor under argoexec.

You do not write that search. Name the backend and dagweave writes it, because the encoding is unpleasant enough by hand that a stray character turns every link into an empty result.

Three fields on the ArgoConnection, all optional:

• spec.traceBackend is jaeger or tempo. Which one decides the whole shape of the query, and it cannot be guessed from the address.
• spec.traceBackendURL is the root of the UI: your Jaeger, or your Grafana when the backend is Tempo. It must be an http or https URL with no query string of its own. Anything else is refused and no link is rendered.
• spec.traceBackendDatasource is the Grafana datasource UID holding your Tempo traces. Tempo only. Without it dagweave renders no links at all, because Grafana Explore has no way to say “the Tempo one” and would otherwise run TraceQL against whichever datasource your org defaults to.

Jaeger

apiVersion: connector.dagweave.com/v1alpha1
kind: ArgoConnection
metadata:
  name: prod
spec:
  argoInstanceID: prod
  namespaces: [argo]
  traceBackend: jaeger
  traceBackendURL: https://jaeger.dagweave.com

A step link then comes out as, with the encoding unwrapped for reading:

https://jaeger.dagweave.com/search
  ?service=workflows-controller
  &lookback=7d
  &tags={"node_id":"hello-world-abc12-3092374","name":"hello-world-abc12","namespace":"argo"}

The whole-run link is the same search without node_id. Jaeger takes its tag filter as a JSON object in a single parameter, which is why the real URL is a wall of %22.

Grafana Tempo

apiVersion: connector.dagweave.com/v1alpha1
kind: ArgoConnection
metadata:
  name: prod
spec:
  argoInstanceID: prod
  namespaces: [argo]
  traceBackend: tempo
  traceBackendURL: https://grafana.dagweave.com
  traceBackendDatasource: tempo-prod

The datasource UID is the last path segment of your datasource’s settings page in Grafana, not its display name.

Tempo is read through Grafana Explore, which carries the whole pane as JSON in one parameter. Unwrapped, a step link is:

https://grafana.dagweave.com/explore
  ?schemaVersion=1
  &panes={"argo":{"datasource":"tempo-prod","queries":[{"refId":"A",
          "datasource":{"type":"tempo","uid":"tempo-prod"},
          "queryType":"traceql",
          "query":"{span.node_id=\"hello-world-abc12-3092374\"&&span.name=\"hello-world-abc12\"&&span.namespace=\"argo\"}"}],
          "range":{"from":"now-7d","to":"now"}}}

Every term is scoped with span., including the workflow name. Bare name in TraceQL means the span’s own name, and Argo happens to call one of its attributes name as well.

Where these links are fragile

Jaeger’s search parameters have been stable for years and the shape above should hold across Jaeger 1.x and Jaeger 2. Grafana’s is not stable in the same way. The panes parameter and schemaVersion=1 arrived in Grafana 10.1, replacing an older left parameter, and that is what these links target. On Grafana 9 or earlier they will not open the query. If a future Grafana changes the format again, the symptom is an Explore page that opens empty rather than an error.

Both links look back seven days. Jaeger’s own default is one hour, which would miss yesterday’s run and read as a broken link; seven days is a guess at something wider than most people’s trace retention. A run older than that opens an empty result, and widening the range in the backend’s own UI finds it.

One more thing that can silently empty a search: service=workflows-controller is the name Argo’s controller reports itself under. If a collector in front of your backend rewrites service.name, Jaeger links find nothing until it stops.

Writing the URL yourself

spec.traceURLTemplate and spec.workflowTraceURLTemplate are still there and still win over anything above. They take {namespace} and {name}, and the per-step one also takes {nodeId}. dagweave substitutes URL-escaped values into them without touching the rest. Reach for them if your backend is neither of the two above, or if your Grafana builds Explore URLs differently: run the query there by hand, copy the address bar, and put {nodeId} back where the node id was.

When the run view says there is nothing to open

The run view names the missing half rather than showing an empty space. There are two messages, and which one you get says which of the jobs above is still open.

Argo is not exporting traces on this cluster, so there is nothing here to open. The workflow controller starts tracing once OTEL_EXPORTER_OTLP_TRACES_ENDPOINT in its environment points at your collector.

The connector works this out from finished workflows. One that carries a trace id means spans are being emitted; finished workflows without one mean the controller is not exporting. While that verdict stands, dagweave withholds both links even if you have named a backend, because a search of a backend holding nothing for that run reads as a broken link rather than as tracing being off. Set the environment variables above and the links come back on the next run.

Until a run has finished there is nothing to judge from, and dagweave renders the links for whatever backend you named rather than claiming your cluster is not tracing.

This cluster connection does not say where to open a trace. Set traceBackend and traceBackendURL on its ArgoConnection, and traceBackendDatasource as well for Tempo. Every step below then links into your own Jaeger or Grafana.

Spans are reaching your collector and nothing on the connection says where to open them. Or the backend is named and something it needs is missing, which comes to the same thing. Fill in the fields above for that instance and reopen the run.

The backend URL is authored in your cluster and rendered as a link in someone’s browser, so dagweave allows only http and https. Anything else is dropped before it is stored, and dropped again in the browser.