Skip to content

Run timeline

When a run took longer than it should, the timeline tells you which step held it up and why.

Open a run from the Runs screen, or click Logs on the live run panel. The Steps list is the timeline: one bar per step, all on one clock from the run’s start to its finish. Each row shows the phase, the bar, how long the step took and how long it waited for its pod. You don’t need to turn anything on.

A run takes longer than its steps. Argo needs time to start, to move from one step to the next, and to wrap up. A thin strip along the top shows those gaps, and faint bands behind every step show them too:

  • Starting, before the first step.
  • Between steps, when no step was running.
  • Finishing, after the last step.

A finished run adds them up after its duration, for example Took 1m25s · 57s running steps · 24s between steps · 4s finishing. With Argo’s defaults, expect up to about ten seconds between two steps in a row.

Argo’s step duration includes waiting for a pod, so 20 minutes waiting for a GPU node looks the same as 20 minutes of work. The timeline splits each bar where the main container started. The shaded start of the bar is the pod wait, and the Pod wait column gives it, for example 3m12s, or 0s if the container started straight away.

Open the row to split the wait into:

  • Creating the pod
  • Finding a node. A long one means the cluster had no room.
  • Starting containers: pulling images, and Argo’s init container loading input artifacts.

If the pod was never scheduled, the whole bar is pod wait.

Steps that ran no pod of their own read No pod: step groups, retry parents, suspend steps and cached results. Dagweave keeps these times after Kubernetes deletes the pod.

With more than one bar, sort by:

  • Run order, when each step started.
  • Duration, longest first. On a wide DAG, that’s the quickest way to find what held the run up.

Bars stay at the same place on the clock either way. You can also filter by step name.

If the longest step is mostly pod wait, faster code won’t help. Look at what it’s waiting for: a node with enough CPU, memory or GPUs, or one that matches its selector and tolerations.

Everything Argo reports except the workflow itself:

  • Each retry attempt. A step that passed on its third try has three rows.
  • Each item of a fan-out (withItems, withParam, withSequence). A thousand items give a thousand rows, ungrouped.
  • Steps in a nested workflow, in the same flat list.
  • Argo’s group nodes, like the parent of a set of retries. They read No pod.

A step with no start time sits below the rest with no bar and reads no timing reported.

Some things show on a closed row: a lock or node wait, and a failed step’s exit code and Argo’s message. Click the row to add:

  • requested, limited and peak CPU and memory, and a new size for the step when Dagweave has one (see Right-size a step)
  • a View trace link, if tracing is set up
  • the step’s logs

A failed run opens with the failing step already expanded. See Runs and logs for where logs come from and how many lines you get.

An open step shows its requested, limited and peak CPU and memory, for example mem 1Gi req/2Gi lim/140Mi peak. The peak is the largest of every sample the connector took while the step ran, about one every 30 seconds. A flag marks a step that’s out of line:

  • near limit: the peak reached 90% of a limit. Past a memory limit the step is killed. Past a CPU limit it’s throttled.
  • over-provisioned: the peak stayed under 30% of the request. The cluster holds the rest of the request for a step that never uses it.

One sample says little. A step that peaks at 900Mi while it loads its data and ends at 100Mi would look over-provisioned from its last sample, and a request cut to match would get it killed for running out of memory on the next run. So Dagweave sizes from the peak and nothing else.

Connectors older than 0.14.0 send only the last sample, which Dagweave shows as last seen. That’s still enough to say a step is near its limit, because the peak was at least that high. It isn’t enough to call a step over-provisioned or to recommend a size. Upgrade the connector to get both.

Once a workflow has three or more successful runs that report peaks, a finished step gets a recommendation for its template, from the last five of those runs:

  • Memory request: the peak plus 30% headroom, rounded up to a multiple of 16Mi, and at least 32Mi.
  • CPU request: the peak plus 20%, rounded up to 10m, and at least 10m.
  • A step near its limit: the limit raised to the peak plus 50%, rounded the same way.

A lower request is only suggested when it’s at most 70% of the current one, since a smaller cut isn’t worth a change. When several steps run the same template, such as the items of a fan-out, the largest peak counts. Lowering a request leaves the limit alone, and raising a limit leaves the request alone. Test runs don’t count.

The step shows a button such as Set memory request to 192Mi, with the basis under it: Peak 140Mi over the last 5 runs, plus 30% headroom.

That depends on where the template lives.

  • A workflow saved in Dagweave gets a new version with the new value, and the next run uses it. If the workflow is open on a canvas, that canvas’s next save is refused and tells you the workflow changed.
  • A workflow linked to Git gets a pull request on its file. Nothing changes in Dagweave or on the cluster until it’s merged.
  • A step that calls a WorkflowTemplate changes that WorkflowTemplate instead, saved or through a pull request by the same rules. Every workflow that calls it gets the new value. If Dagweave has no copy of the WorkflowTemplate, you get YAML to paste instead.
  • An adopted run, one Dagweave didn’t start, has no button. You get a YAML snippet naming the template and its resources, to paste wherever that template is defined. Adopted runs count as the same workflow when Argo named them from the same prefix, on the same cluster and namespace.

If you can’t edit the workflow, the button is greyed out and says why. Each change is in the audit log as workflow.rightsize. Over the API the same change is POST /api/v1/workflows/{id}/resources, and over MCP it’s dagweave_set_step_resources.

A running step’s bar grows every second, and the clock runs to now. Time since the last step ended counts as Between steps until the run finishes.

If a bar keeps growing and nothing’s happening, see stuck runs.

If the connector receives Argo’s OpenTelemetry spans, the working part of each bar splits into pod creation, artifact loading, artifact saving and log saving. What’s left is usually your own command. See Tracing.

To follow a trace into your own Jaeger or Tempo, use View trace on a step or Trace in the run’s header. Those links need:

  • the workflow controller exporting spans, with OTEL_EXPORTER_OTLP_TRACES_ENDPOINT in its environment pointing at your collector
  • traceBackend and traceBackendURL under argoConnection.extraSpec in the connector’s Helm values, plus traceBackendDatasource for Tempo

If either is missing, the run view says which.

Dagweave only reads the main container: its resources, start time and logs.

Argo Workflows 4.1 has a beta init-less layout, off by default, where a supervisor container runs beside main and does the jobs of init and wait. A cluster operator turns it on with initlessPod.enabled: true in the workflow controller ConfigMap. It needs Kubernetes image volumes: the ImageVolume feature gate on 1.33 to 1.35, on by default from 1.36.

It moves input artifact loading. On the default layout, main starts after loading, so loading counts as pod wait. On the init-less layout, main is already running, so loading counts as work. Don’t compare pod wait across the two.