maborak.com~/blog/argo-cd-3-6-upgrade-daysoftware & infrastructure engineering
MaborakTechnologies
Contact
Back to all articles

Argo CD 3.6 Upgrade Day: Synced Is Not Always Synced

Overview

On Argo CD 3.6.1, a failed server-side dry run turns into a Synced badge, so OutOfSync alerts and automated sync never act. We reproduced it three ways and on a 50-app upgrade.

By the Maborak engineering team · · 29 min read

When the server-side dry run fails on Argo CD 3.6.1, the application shows an error for a few minutes and then reports Synced, while the cluster still differs from Git. It happened in all 15 of our trials across three causes: an admission policy, an immutable Job field and a validating webhook that timed out, five trials each on one application per cause. On 3.5.7 with default settings the same setups stayed OutOfSync in all 19 trials. While an application reads Synced, an OutOfSync notification has nothing to fire on and automated sync, by its documented rules, does not run; we watched one application per version for 20 minutes and saw exactly that.

We also upgraded a 50-application fleet from 3.5.7 to 3.6.1 three times. Each time, none of the 10 applications whose change the cluster rejects was still OutOfSync by the end of the cycle; they read Synced or Unknown. How long that took varied from 3 to almost 14 minutes. Rolling back brought the honest status back in about three minutes.

Checked 8 November 2026.

How we tested

This follows our check of the 3.6 release claims, where we first saw the false Synced with an admission policy. The same lab ran it: kind v0.33.0 with Kubernetes v1.37.0 on an arm64 Mac, Argo CD 3.5.7 and 3.6.1 from their release manifests, Gitea for Git and registry:2 for OCI artifacts. Every application, namespace and webhook below is named maborak-*; they are ours. Before each planned run we wrote down what we would measure and how we would count it; runs added later are marked as such and Limits lists every change to the plan. Every number is computed by script from raw output that sits in a collapsed block beside each result. An AI coding agent built the lab, ran it and drafted this post; no model judged any result.

The cause is a default that changed in 3.6. PR #29103 makes every application with ServerSideApply=true compute its diff with Server-Side Diff, a dry-run apply against the API server. [2] [3] When that dry run fails, Argo CD records a ComparisonError. On the next periodic reconcile the error is gone and the status is Synced; our reading is that this reconcile reuses a cached comparison instead of repeating the failed dry run, but we did not trace it in the code. Two observations fit that reading: the controller logs these reconciles as “comparison expired” refreshes, and during our upgrade drill the periodic reconciles on 3.6.1 sent no dry-run requests at all. 3.5.7 shows the same flip if you switch Server-Side Diff on yourself (3 of 3 trials in the first post), so the same behaviour already exists in 3.5.7’s Server-Side Diff. What is new is that it now applies to every SSA application by default.

Any failed dry run does it

We gave each case its own SSA application with a real difference between Git and the cluster, forced a hard refresh, then waited for the next periodic reconcile without touching anything.

What makes the dry run fail 3.6.1 after hard refresh 3.6.1 after the next reconcile 3.5.7 after the next reconcile
Admission policy rejects the change (first post) ComparisonError, 5 of 5 Synced, 5 of 5 OutOfSync, 5 of 5
Immutable field: a Job’s pod template changed in Git ComparisonError, 5 of 5 Synced, 5 of 5 OutOfSync, 3 of 3
Validating webhook times out (2 s limit, answers after 10 s) ComparisonError, 5 of 5 Synced, 5 of 5 OutOfSync, 3 of 3
Control: custom resource whose CRD is missing (planned as a failure mode; the dry run did not fail) OutOfSync, 5 of 5 OutOfSync, 5 of 5 OutOfSync, 3 of 3
Control: plain drift, no policy (first post) OutOfSync, 5 of 5 OutOfSync, 5 of 5 OutOfSync, 5 of 5

Each failure mode flipped in 5 of 5 trials (Wilson 95% interval 57–100% for each); the two controls never flipped (0 of 10), nor did any of the 19 trials on 3.5.7 with default settings. Every trial repeats a hard refresh on the same application, so the intervals describe how reliably it happens on that application, not how common it is across clusters. We verified the drift at the end rather than in every trial: the Job still runs migration v1, the ConfigMaps still hold k: live, the applications’ sync history is empty, and only kubectl-create ever wrote those objects. Each reconcile we counted appears in the controller log as a “comparison expired” refresh, its two-minute periodic check. Many teams will meet the immutable-field case: a Job or a StatefulSet field changed in Git, which Kubernetes refuses to update in place.

Per-trial data: status after a hard refresh and after the next periodic reconcile (47 trials, both posts)
source version app trial after_hard after_periodic periodic_at
first post 357 maborak-drift 1 OutOfSync/none OutOfSync/none 2026-11-07T23:43:03Z
first post 357 maborak-drift 2 OutOfSync/none OutOfSync/none 2026-11-07T23:47:13Z
first post 357 maborak-drift 3 OutOfSync/none OutOfSync/none 2026-11-07T23:51:14Z
first post 357 maborak-drift 4 OutOfSync/none OutOfSync/none 2026-11-07T23:55:08Z
first post 357 maborak-drift 5 OutOfSync/none OutOfSync/none 2026-11-07T23:57:20Z
this post 357 maborak-immutable 1 OutOfSync/none OutOfSync/none 2026-11-08T00:36:41Z
this post 357 maborak-immutable 2 OutOfSync/none OutOfSync/none 2026-11-08T00:41:15Z
this post 357 maborak-immutable 3 OutOfSync/none OutOfSync/none 2026-11-08T00:44:57Z
this post 357 maborak-nocrd 1 OutOfSync/none OutOfSync/none 2026-11-08T00:37:11Z
this post 357 maborak-nocrd 2 OutOfSync/none OutOfSync/none 2026-11-08T00:41:16Z
this post 357 maborak-nocrd 3 OutOfSync/none OutOfSync/none 2026-11-08T00:45:03Z
first post 357 maborak-policy-check 1 OutOfSync/none OutOfSync/none 2026-11-07T23:42:54Z
first post 357 maborak-policy-check 2 OutOfSync/none OutOfSync/none 2026-11-07T23:45:18Z
first post 357 maborak-policy-check 3 OutOfSync/none OutOfSync/none 2026-11-07T23:48:44Z
first post 357 maborak-policy-check 4 OutOfSync/none OutOfSync/none 2026-11-07T23:51:02Z
first post 357 maborak-policy-check 5 OutOfSync/none OutOfSync/none 2026-11-07T23:54:33Z
first post 357 maborak-policy-ssd 1 Unknown/ComparisonError Synced/none 2026-11-08T00:02:46Z
first post 357 maborak-policy-ssd 2 Unknown/ComparisonError Synced/none 2026-11-08T00:04:51Z
first post 357 maborak-policy-ssd 3 Unknown/ComparisonError Synced/none 2026-11-08T00:07:13Z
this post 357 maborak-slowhook 1 OutOfSync/none OutOfSync/none 2026-11-08T00:37:00Z
this post 357 maborak-slowhook 2 OutOfSync/none OutOfSync/none 2026-11-08T00:41:02Z
this post 357 maborak-slowhook 3 OutOfSync/none OutOfSync/none 2026-11-08T00:43:05Z
first post 361 maborak-drift 1 OutOfSync/none OutOfSync/none 2026-11-07T23:43:07Z
first post 361 maborak-drift 2 OutOfSync/none OutOfSync/none 2026-11-07T23:47:13Z
first post 361 maborak-drift 3 OutOfSync/none OutOfSync/none 2026-11-07T23:49:17Z
first post 361 maborak-drift 4 OutOfSync/none OutOfSync/none 2026-11-07T23:53:08Z
first post 361 maborak-drift 5 OutOfSync/none OutOfSync/none 2026-11-07T23:56:59Z
this post 361 maborak-immutable 1 Unknown/ComparisonError Synced/none 2026-11-08T00:37:05Z
this post 361 maborak-immutable 2 Unknown/ComparisonError Synced/none 2026-11-08T00:40:58Z
this post 361 maborak-immutable 3 Unknown/ComparisonError Synced/none 2026-11-08T00:45:08Z
this post 361 maborak-immutable 4 Unknown/ComparisonError Synced/none 2026-11-08T00:47:14Z
this post 361 maborak-immutable 5 Unknown/ComparisonError Synced/none 2026-11-08T00:49:19Z
this post 361 maborak-nocrd 1 OutOfSync/none OutOfSync/none 2026-11-08T00:37:00Z
this post 361 maborak-nocrd 2 OutOfSync/none OutOfSync/none 2026-11-08T00:39:13Z
this post 361 maborak-nocrd 3 OutOfSync/none OutOfSync/none 2026-11-08T00:42:33Z
this post 361 maborak-nocrd 4 OutOfSync/none OutOfSync/none 2026-11-08T00:44:56Z
this post 361 maborak-nocrd 5 OutOfSync/none OutOfSync/none 2026-11-08T00:49:19Z
first post 361 maborak-policy-check 1 Unknown/ComparisonError Synced/none 2026-11-07T23:42:52Z
first post 361 maborak-policy-check 2 Unknown/ComparisonError Synced/none 2026-11-07T23:45:03Z
first post 361 maborak-policy-check 3 Unknown/ComparisonError Synced/none 2026-11-07T23:48:27Z
first post 361 maborak-policy-check 4 Unknown/ComparisonError Synced/none 2026-11-07T23:51:12Z
first post 361 maborak-policy-check 5 Unknown/ComparisonError Synced/none 2026-11-07T23:53:17Z
this post 361 maborak-slowhook 1 Unknown/ComparisonError Synced/none 2026-11-08T00:37:13Z
this post 361 maborak-slowhook 2 Unknown/ComparisonError Synced/none 2026-11-08T00:41:08Z
this post 361 maborak-slowhook 3 Unknown/ComparisonError Synced/none 2026-11-08T00:44:26Z
this post 361 maborak-slowhook 4 Unknown/ComparisonError Synced/none 2026-11-08T00:47:06Z
this post 361 maborak-slowhook 5 Unknown/ComparisonError Synced/none 2026-11-08T00:50:31Z
Raw output: end-state check of the live objects and controller log lines 
## End-state check: live objects vs Git, 2026-11-08T02:39:31Z
### 357
maborak-immutable Job command (Git: echo maborak migration v2): ["sh","-c","echo maborak migration v1"] managers: kubectl-create@2026-11-08T00:32:17Z kube-controller-manager@2026-11-08T00:32:20Z 
maborak-slowhook ConfigMap data (Git: k=from-git): {"k":"live"} managers: kubectl-create@2026-11-08T00:32:17Z 
maborak-policy-check ConfigMap (Git: k=from-git + label maborak-reject): data={"k":"live"} labels=
maborak-immutable: sync history entries=0
maborak-slowhook: sync history entries=0
maborak-policy-check: sync history entries=0
controller log lines 'comparison expired' for trial apps (last 4h): 213
example: {"app-namespace":"argocd","application":"maborak-immutable","level":"info","msg":"Refreshing app status (comparison expired, requesting refresh. reconciledAt: 2026-11-08 00:33:10 +0000 UTC, expiry: 2m0s), level (2)","project":"def
### 361
maborak-immutable Job command (Git: echo maborak migration v2): ["sh","-c","echo maborak migration v1"] managers: kubectl-create@2026-11-08T00:32:25Z kube-controller-manager@2026-11-08T00:32:28Z 
maborak-slowhook ConfigMap data (Git: k=from-git): {"k":"live"} managers: kubectl-create@2026-11-08T00:32:25Z 
maborak-policy-check ConfigMap (Git: k=from-git + label maborak-reject): data={"k":"live"} labels=
maborak-immutable: sync history entries=0
maborak-slowhook: sync history entries=0
maborak-policy-check: sync history entries=0
controller log lines 'comparison expired' for trial apps (last 4h): 159
example: {"app-namespace":"argocd","application":"maborak-immutable","level":"info","msg":"Refreshing app status (comparison expired, requesting refresh. reconciledAt: 2026-11-08 00:33:10 +0000 UTC, expiry: 2m0s), level (2)","project":"def

Argo CD 3.6.1 Application conditions for maborak-immutable: ComparisonError, error running server side apply in dryrun mode for resource Job/maborak-migrate: Job.batch “maborak-migrate” is invalid: spec.template … field is immutable, showing command echo maborak migration v1 live and v2 in Git.

Right after a hard refresh: the immutable Job field fails the dry run. A few minutes later this application reads Synced. Captured on our 3.6.1 instance on 8 November 2026, 01:19 UTC.

Argo CD 3.6.1 Application conditions for maborak-slowhook: ComparisonError, failed calling webhook “slow.maborak.com”: failed to call webhook: Post “https://maborak-slow-webhook.maborak-webhook.svc:443/validate?timeout=2s”: context deadline exceeded.

A webhook that does not answer within its timeout fails the dry run the same way. Captured on our 3.6.1 instance on 8 November 2026, 01:20 UTC.

What the false Synced turns off

A wrong badge is bad enough. We wanted to know what else trusts it, so we subscribed one policy-blocked application on each version to three notification triggers (OutOfSync, Unknown, Synced) sent to a small receiver in the cluster, and gave a second one automated sync with selfHeal: true. After one hard refresh we left both alone for 20 minutes.

3.5.7 3.6.1
Status during the 20 minutes OutOfSync throughout Unknown for 151 s, then Synced for the remaining 17.5 min
Notification when the subscription started (27 s before the watch) OutOfSync Synced
Notifications during the 20 minutes none Unknown, then Synced
Automated sync operations 1, at creation 10 s before the watch; failed with SyncError 0

The 3.6.1 application never read OutOfSync, so the OutOfSync trigger had nothing to fire on, and it sent a Synced message twice while the cluster still differed from Git. 3.5.7 alerted when the subscription started and kept showing OutOfSync, with a red “Sync failed” from its first automated sync. Neither version tried again inside the window, which matches the documentation: automated sync runs only when an application is OutOfSync, and it does not retry a failed sync of the same commit. [9] We did not test the other self-heal case, where someone edits the live object of an application that was in sync. This is one application per version over 20 minutes, a case study rather than a rate.

Raw output: status changes and sync operations during the 20-minute watch 
357 maborak-policy-check: 120 samples every 10 s
  t=   0s  OutOfSync
  sync operations started: 0 
357 maborak-selfheal: 120 samples every 10 s
  t=   0s  OutOfSync
  t= 302s  OutOfSync / SyncError
  sync operations started: 1 (2026-11-08T00:46:17Z)
361 maborak-policy-check: 120 samples every 10 s
  t=   0s  Unknown / ComparisonError
  t= 151s  Synced
  sync operations started: 0 
361 maborak-selfheal: 120 samples every 10 s
  t=   0s  Unknown / ComparisonError
  t= 151s  Synced
  sync operations started: 0
Raw output: every notification the two receivers logged (both versions) 
== 357 receiver
{"received": "2026-11-08T00:46:00.679561Z", "body": "{\"app\":\"maborak-policy-check\",\"sync\":\"OutOfSync\",\"revision\":\"a8228cd32e71bf054e06c8e1dbb6b382037e1ade\"}"}
== 361 receiver
{"received": "2026-11-08T00:46:01.246408Z", "body": "{\"app\":\"maborak-policy-check\",\"sync\":\"Synced\",\"revision\":\"a8228cd32e71bf054e06c8e1dbb6b382037e1ade\"}"}
{"received": "2026-11-08T00:46:28.446553Z", "body": "{\"app\":\"maborak-policy-check\",\"sync\":\"Unknown\",\"revision\":\"a8228cd32e71bf054e06c8e1dbb6b382037e1ade\"}"}
{"received": "2026-11-08T00:48:53.187639Z", "body": "{\"app\":\"maborak-policy-check\",\"sync\":\"Synced\",\"revision\":\"a8228cd32e71bf054e06c8e1dbb6b382037e1ade\"}"}
{"received": "2026-11-08T01:32:58.339397Z", "body": "{\"app\":\"maborak-policy-check\",\"sync\":\"Unknown\",\"revision\":\"a8228cd32e71bf054e06c8e1dbb6b382037e1ade\"}"}
{"received": "2026-11-08T01:37:03.654106Z", "body": "{\"app\":\"maborak-policy-check\",\"sync\":\"Synced\",\"revision\":\"a8228cd32e71bf054e06c8e1dbb6b382037e1ade\"}"}
{"received": "2026-11-08T02:20:54.202517Z", "body": "{\"app\":\"maborak-immutable\",\"sync\":\"Unknown\",\"revision\":\"a8228cd32e71bf054e06c8e1dbb6b382037e1ade\"}"}
{"received": "2026-11-08T02:34:18.054465Z", "body": "{\"app\":\"maborak-policy-check\",\"sync\":\"Unknown\",\"revision\":\"a8228cd32e71bf054e06c8e1dbb6b382037e1ade\"}"}
{"received": "2026-11-08T02:36:30.100089Z", "body": "{\"app\":\"maborak-immutable\",\"sync\":\"Unknown\",\"revision\":\"a8228cd32e71bf054e06c8e1dbb6b382037e1ade\"}"}
{"received": "2026-11-08T02:36:48.642081Z", "body": "{\"app\":\"maborak-policy-check\",\"sync\":\"Synced\",\"revision\":\"a8228cd32e71bf054e06c8e1dbb6b382037e1ade\"}"}

Argo CD 3.5.7 application maborak-selfheal: Sync status OutOfSync from main (a8228cd), auto sync enabled, last sync result “Sync failed” to a8228cd; tree shows ConfigMap maborak-guarded out of sync.

3.5.7: the first automated sync failed, and the application says so. Captured on our 3.5.7 instance on 8 November 2026, 01:19 UTC.

Argo CD 3.6.1 application maborak-selfheal: App health Healthy, Sync status Synced to main (a8228cd), auto sync enabled; ConfigMap maborak-guarded shown synced, although the live object differs from Git.

3.6.1, same Git commit and cluster: Synced, Healthy, and nothing to retry. Captured on our 3.6.1 instance on 8 November 2026, 01:19 UTC.

Upgrade day on 50 applications

To see what an upgrade looks like on a dashboard, we built a fleet on a fresh 3.5.7 cluster: 40 SSA applications with 10 ConfigMaps each, and 10 SSA applications whose Git manifest an admission policy rejects, 410 managed resources in all. After 10 minutes at steady state we applied the 3.6.1 install manifest in place, watched for 15 minutes, applied the 3.5.7 manifest again, watched for 15 minutes, and did that three times. A script sampled the fleet every 30 seconds. This was planned as a measurement of a short burst; it turned out to be neither short nor a burst.

Cycle First blocked app leaves OutOfSync No blocked app OutOfSync from Blocked apps at the end Dry-run requests (410 reached at) Correct status back after rollback
Upgrade 1 183 s 819 s 6 Synced, 4 Unknown 410 (819 s) 182 s
Upgrade 2 31 s 183 s 10 Synced 410 (213 s) 183 s
Upgrade 3 153 s 798 s (no samples between 153 and 768 s) 7 Synced, 3 Unknown 410 (889 s) 183 s

By the end of every cycle no blocked application was OutOfSync (3 of 3 cycles, Wilson 44–100%); 6, 10 and 7 of them read Synced and the rest Unknown. The speed varied a lot. In upgrade 1, six blocked applications stayed honestly OutOfSync on 3.6.1 for about seven minutes while the dry-run counter sat at 174, before a second wave hid them; we have not explained that pause. Upgrade 2 hid everything within three minutes. The counter reached 410 each time, which equals the managed resources (40 × 10 + 10 × 1); reading it as one dry run per resource is our inference, since the counter counts every dry run the API server receives. Rolling back to 3.5.7 added no dry-run requests and restored the 10 OutOfSync applications after 182–183 seconds every time, close to the controller’s two-minute reconcile interval, so a different timeout.reconciliation would change it. Our status counts are fleet totals: the Synced and Unknown split among the blocked applications assumes the 40 clean ones stayed Synced, as they did at every baseline and rollback sample. The three cycles ran one after another on one cluster while other lab runs loaded the same machine, so read them as one drill repeated, not three independent samples.

Analysis output: per-phase summary 
baseline    samples= 20 dry_run_in_phase=   0 first_outofsync<10_at=Nones end: synced=40 outofsync=10 unknown=0 max_unknown=0 max_synced=40 
upgrade-1   samples= 30 dry_run_in_phase= 410 first_outofsync<10_at=183s end: synced=46 outofsync=0 unknown=4 max_unknown=5 max_synced=46 
rollback-1  samples= 30 dry_run_in_phase=   0 first_outofsync<10_at=1s end: synced=40 outofsync=10 unknown=0 max_unknown=4 max_synced=46 10_outofsync_again_at=182s
upgrade-2   samples= 30 dry_run_in_phase= 410 first_outofsync<10_at=31s end: synced=50 outofsync=0 unknown=0 max_unknown=7 max_synced=50 
rollback-2  samples= 30 dry_run_in_phase=   0 first_outofsync<10_at=1s end: synced=40 outofsync=10 unknown=0 max_unknown=0 max_synced=50 10_outofsync_again_at=183s
upgrade-3   samples= 11 dry_run_in_phase= 410 first_outofsync<10_at=153s end: synced=47 outofsync=0 unknown=3 max_unknown=8 max_synced=47 
rollback-3  samples= 30 dry_run_in_phase=   0 first_outofsync<10_at=2s end: synced=40 outofsync=10 unknown=0 max_unknown=3 max_synced=47 10_outofsync_again_at=183s
Per-sample data: the drill, every 30 s (phase, time, controller version, dry-run counter, apps per status) 
phase,utc,elapsed_s,controller_image,dry_run_total,synced,outofsync,unknown,comparison_errors
baseline,2026-11-08T00:37:06Z,1,v3.5.7,0,40,10,0,0
baseline,2026-11-08T00:37:36Z,31,v3.5.7,0,40,10,0,0
baseline,2026-11-08T00:38:06Z,61,v3.5.7,0,40,10,0,0
baseline,2026-11-08T00:38:37Z,92,v3.5.7,0,40,10,0,0
baseline,2026-11-08T00:39:07Z,122,v3.5.7,0,40,10,0,0
baseline,2026-11-08T00:39:37Z,152,v3.5.7,0,40,10,0,0
baseline,2026-11-08T00:40:07Z,182,v3.5.7,0,40,10,0,0
baseline,2026-11-08T00:40:38Z,213,v3.5.7,0,40,10,0,0
baseline,2026-11-08T00:41:08Z,243,v3.5.7,0,40,10,0,0
baseline,2026-11-08T00:41:38Z,273,v3.5.7,0,40,10,0,0
baseline,2026-11-08T00:42:08Z,303,v3.5.7,0,40,10,0,0
baseline,2026-11-08T00:42:39Z,334,v3.5.7,0,40,10,0,0
baseline,2026-11-08T00:43:09Z,364,v3.5.7,0,40,10,0,0
baseline,2026-11-08T00:43:40Z,395,v3.5.7,0,40,10,0,0
baseline,2026-11-08T00:44:10Z,425,v3.5.7,0,40,10,0,0
baseline,2026-11-08T00:44:40Z,455,v3.5.7,0,40,10,0,0
baseline,2026-11-08T00:45:10Z,485,v3.5.7,0,40,10,0,0
baseline,2026-11-08T00:45:41Z,516,v3.5.7,0,40,10,0,0
baseline,2026-11-08T00:46:11Z,546,v3.5.7,0,40,10,0,0
baseline,2026-11-08T00:46:41Z,576,v3.5.7,0,40,10,0,0
upgrade-1,2026-11-08T00:47:12Z,1,v3.5.7,0,40,10,0,0
upgrade-1,2026-11-08T00:47:43Z,32,v3.6.1,0,40,10,0,0
upgrade-1,2026-11-08T00:48:13Z,62,v3.6.1,0,40,10,0,0
upgrade-1,2026-11-08T00:48:43Z,92,v3.6.1,0,40,10,0,0
upgrade-1,2026-11-08T00:49:13Z,122,v3.6.1,0,40,10,0,0
upgrade-1,2026-11-08T00:49:44Z,153,v3.6.1,0,40,10,0,0
upgrade-1,2026-11-08T00:50:14Z,183,v3.6.1,83,40,7,3,3
upgrade-1,2026-11-08T00:50:44Z,213,v3.6.1,174,40,6,4,4
upgrade-1,2026-11-08T00:51:15Z,244,v3.6.1,174,40,6,4,4
upgrade-1,2026-11-08T00:51:45Z,274,v3.6.1,174,40,6,4,4
upgrade-1,2026-11-08T00:52:15Z,304,v3.6.1,174,40,6,4,4
upgrade-1,2026-11-08T00:52:45Z,334,v3.6.1,174,44,6,0,0
upgrade-1,2026-11-08T00:53:16Z,365,v3.6.1,174,44,6,0,0
upgrade-1,2026-11-08T00:53:46Z,395,v3.6.1,174,44,6,0,0
upgrade-1,2026-11-08T00:54:16Z,425,v3.6.1,174,44,6,0,0
upgrade-1,2026-11-08T00:54:46Z,455,v3.6.1,174,44,6,0,0
upgrade-1,2026-11-08T00:55:17Z,486,v3.6.1,174,44,6,0,0
upgrade-1,2026-11-08T00:55:47Z,516,v3.6.1,174,44,6,0,0
upgrade-1,2026-11-08T00:56:17Z,546,v3.6.1,174,44,6,0,0
upgrade-1,2026-11-08T00:56:48Z,577,v3.6.1,174,44,6,0,0
upgrade-1,2026-11-08T00:57:18Z,607,v3.6.1,174,44,6,0,0
upgrade-1,2026-11-08T00:57:48Z,637,v3.6.1,174,44,6,0,0
upgrade-1,2026-11-08T00:58:18Z,667,v3.6.1,246,44,4,2,2
upgrade-1,2026-11-08T00:58:49Z,698,v3.6.1,306,44,4,2,2
upgrade-1,2026-11-08T00:59:19Z,728,v3.6.1,306,44,4,2,2
upgrade-1,2026-11-08T00:59:49Z,758,v3.6.1,337,44,3,3,3
upgrade-1,2026-11-08T01:00:20Z,789,v3.6.1,389,44,1,5,5
upgrade-1,2026-11-08T01:00:50Z,819,v3.6.1,410,46,0,4,4
upgrade-1,2026-11-08T01:01:20Z,849,v3.6.1,410,46,0,4,4
upgrade-1,2026-11-08T01:01:51Z,880,v3.6.1,410,46,0,4,4
rollback-1,2026-11-08T01:02:22Z,1,v3.6.1,410,46,0,4,4
rollback-1,2026-11-08T01:02:52Z,31,v3.5.7,410,43,7,0,0
rollback-1,2026-11-08T01:03:22Z,61,v3.5.7,410,43,7,0,0
rollback-1,2026-11-08T01:03:52Z,91,v3.5.7,410,43,7,0,0
rollback-1,2026-11-08T01:04:23Z,122,v3.5.7,410,43,7,0,0
rollback-1,2026-11-08T01:04:53Z,152,v3.5.7,410,42,8,0,0
rollback-1,2026-11-08T01:05:23Z,182,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:05:54Z,213,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:06:24Z,243,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:06:54Z,273,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:07:24Z,303,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:07:55Z,334,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:08:25Z,364,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:08:55Z,394,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:09:25Z,424,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:09:56Z,455,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:10:26Z,485,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:10:56Z,515,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:11:27Z,546,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:11:57Z,576,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:12:27Z,606,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:12:57Z,636,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:13:28Z,667,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:13:58Z,697,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:14:28Z,727,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:14:58Z,757,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:15:29Z,788,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:15:59Z,818,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:16:29Z,848,v3.5.7,410,40,10,0,0
rollback-1,2026-11-08T01:17:00Z,879,v3.5.7,410,40,10,0,0
upgrade-2,2026-11-08T01:17:31Z,1,v3.5.7,410,40,10,0,0
upgrade-2,2026-11-08T01:18:01Z,31,v3.6.1,554,40,6,4,4
upgrade-2,2026-11-08T01:18:31Z,61,v3.6.1,554,40,6,4,4
upgrade-2,2026-11-08T01:19:02Z,92,v3.6.1,554,40,6,4,4
upgrade-2,2026-11-08T01:19:32Z,122,v3.6.1,554,40,6,4,4
upgrade-2,2026-11-08T01:20:02Z,152,v3.6.1,675,41,5,4,5
upgrade-2,2026-11-08T01:20:33Z,183,v3.6.1,790,43,0,7,7
upgrade-2,2026-11-08T01:21:03Z,213,v3.6.1,820,44,0,6,6
upgrade-2,2026-11-08T01:21:33Z,243,v3.6.1,820,44,0,6,6
upgrade-2,2026-11-08T01:22:04Z,274,v3.6.1,820,44,0,6,6
upgrade-2,2026-11-08T01:22:34Z,304,v3.6.1,820,47,0,3,3
upgrade-2,2026-11-08T01:23:04Z,334,v3.6.1,820,47,0,3,3
upgrade-2,2026-11-08T01:23:34Z,364,v3.6.1,820,47,0,3,3
upgrade-2,2026-11-08T01:24:05Z,395,v3.6.1,820,48,0,2,2
upgrade-2,2026-11-08T01:24:35Z,425,v3.6.1,820,50,0,0,0
upgrade-2,2026-11-08T01:25:05Z,455,v3.6.1,820,50,0,0,0
upgrade-2,2026-11-08T01:25:35Z,485,v3.6.1,820,50,0,0,0
upgrade-2,2026-11-08T01:26:06Z,516,v3.6.1,820,50,0,0,0
upgrade-2,2026-11-08T01:26:36Z,546,v3.6.1,820,50,0,0,0
upgrade-2,2026-11-08T01:27:06Z,576,v3.6.1,820,50,0,0,0
upgrade-2,2026-11-08T01:27:37Z,607,v3.6.1,820,50,0,0,0
upgrade-2,2026-11-08T01:28:07Z,637,v3.6.1,820,50,0,0,0
upgrade-2,2026-11-08T01:28:37Z,667,v3.6.1,820,50,0,0,0
upgrade-2,2026-11-08T01:29:07Z,697,v3.6.1,820,50,0,0,0
upgrade-2,2026-11-08T01:29:38Z,728,v3.6.1,820,50,0,0,0
upgrade-2,2026-11-08T01:30:08Z,758,v3.6.1,820,50,0,0,0
upgrade-2,2026-11-08T01:30:38Z,788,v3.6.1,820,50,0,0,0
upgrade-2,2026-11-08T01:31:09Z,819,v3.6.1,820,50,0,0,0
upgrade-2,2026-11-08T01:31:39Z,849,v3.6.1,820,50,0,0,0
upgrade-2,2026-11-08T01:32:09Z,879,v3.6.1,820,50,0,0,0
rollback-2,2026-11-08T01:32:40Z,1,v3.6.1,820,50,0,0,0
rollback-2,2026-11-08T01:33:11Z,32,v3.5.7,820,47,3,0,0
rollback-2,2026-11-08T01:33:41Z,62,v3.5.7,820,47,3,0,0
rollback-2,2026-11-08T01:34:11Z,92,v3.5.7,820,47,3,0,0
rollback-2,2026-11-08T01:34:41Z,122,v3.5.7,820,47,3,0,0
rollback-2,2026-11-08T01:35:12Z,153,v3.5.7,820,44,6,0,0
rollback-2,2026-11-08T01:35:42Z,183,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:36:12Z,213,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:36:43Z,244,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:37:13Z,274,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:37:43Z,304,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:38:13Z,334,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:38:44Z,365,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:39:14Z,395,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:39:44Z,425,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:40:14Z,455,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:40:45Z,486,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:41:15Z,516,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:41:45Z,546,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:42:16Z,577,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:42:46Z,607,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:43:16Z,637,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:43:46Z,667,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:44:17Z,698,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:44:47Z,728,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:45:17Z,758,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:45:48Z,789,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:46:18Z,819,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:46:48Z,849,v3.5.7,820,40,10,0,0
rollback-2,2026-11-08T01:47:18Z,879,v3.5.7,820,40,10,0,0
upgrade-3,2026-11-08T01:47:50Z,2,v3.6.1,820,40,10,0,0
upgrade-3,2026-11-08T01:48:20Z,32,v3.6.1,820,40,10,0,0
upgrade-3,2026-11-08T01:48:51Z,63,v3.6.1,820,40,10,0,0
upgrade-3,2026-11-08T01:49:21Z,93,v3.6.1,820,40,10,0,0
upgrade-3,2026-11-08T01:49:51Z,123,v3.6.1,820,40,10,0,0
upgrade-3,2026-11-08T01:50:21Z,153,v3.6.1,1014,40,6,4,4
upgrade-3,2026-11-08T02:00:36Z,768,v3.6.1,1179,41,1,8,8
upgrade-3,2026-11-08T02:01:06Z,798,v3.6.1,1200,43,0,7,7
upgrade-3,2026-11-08T02:01:37Z,829,v3.6.1,1200,43,0,7,7
upgrade-3,2026-11-08T02:02:07Z,859,v3.6.1,1210,43,0,7,7
upgrade-3,2026-11-08T02:02:37Z,889,v3.6.1,1230,47,0,3,3
rollback-3,2026-11-08T02:03:09Z,2,v3.6.1,1230,47,0,3,3
rollback-3,2026-11-08T02:03:39Z,32,v3.5.7,1230,44,6,0,0
rollback-3,2026-11-08T02:04:09Z,62,v3.5.7,1230,44,6,0,0
rollback-3,2026-11-08T02:04:39Z,92,v3.5.7,1230,44,6,0,0
rollback-3,2026-11-08T02:05:10Z,123,v3.5.7,1230,44,6,0,0
rollback-3,2026-11-08T02:05:40Z,153,v3.5.7,1230,43,7,0,0
rollback-3,2026-11-08T02:06:10Z,183,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:06:41Z,214,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:07:11Z,244,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:07:41Z,274,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:08:11Z,304,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:08:42Z,335,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:09:12Z,365,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:09:42Z,395,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:10:13Z,426,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:10:43Z,456,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:11:13Z,486,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:11:43Z,516,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:12:14Z,547,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:12:44Z,577,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:13:14Z,607,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:13:45Z,638,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:14:15Z,668,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:14:45Z,698,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:15:15Z,728,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:15:46Z,759,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:16:16Z,789,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:16:46Z,819,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:17:16Z,849,v3.5.7,1230,40,10,0,0
rollback-3,2026-11-08T02:17:47Z,880,v3.5.7,1230,40,10,0,0

Argo CD 3.6.1 Applications list filtered to label fleet=blocked: maborak-blocked-01 to maborak-blocked-09 visible, all Healthy and Synced; health filter shows Healthy 10.

Ten minutes into the second upgrade: all ten applications whose change the cluster rejects read Synced. On 3.5.7 they were OutOfSync. Captured on our drill cluster on 8 November 2026, 01:27 UTC.

What to do before you upgrade

  • Find the applications that can hit this. They have ServerSideApply=true and a manifest the cluster may refuse: admission policies (ValidatingAdmissionPolicy, Kyverno, Gatekeeper), fields Kubernetes will not change in place (Job templates, StatefulSet selectors and volume claim templates), or webhooks that can time out.
  • Opt those applications out until you have checked them: argocd.argoproj.io/compare-options: ServerSideDiff=false brings back the 3.5 behaviour per application. [3] We tested the opt-out only with the admission policy (5 of 5 refreshes OutOfSync in the first post), not with the immutable field or the webhook.
  • Alert on the ComparisonError condition as well as the badge. We added a trigger with when: app.status.conditions != nil && any(app.status.conditions, {.type == 'ComparisonError'}) to the immutable-Job application, after writing the rest of this advice. In that single trial a message arrived one second after a hard refresh; it did not name its trigger, but the application was subscribed only to this one and to an OutOfSync trigger, which never fired. The application read Synced 3.7 minutes later. In the drill the error also appeared without any manual refresh, after the controller restarted, so the condition can show up on its own; we did not measure how often or for how long.
  • Hard refresh before you trust Synced on those applications after the upgrade.
  • Rollback works. In our drill, going back to 3.5.7 restored correct statuses in about three minutes and sent no dry-run requests.
Raw output: ComparisonError notification trigger test 
## T1 condition trigger test, hard refresh at 2026-11-08T02:20:53Z
trigger: when: app.status.conditions != nil && any(app.status.conditions, {.type == 'ComparisonError'})
### receiver messages for maborak-immutable since t0
{"received": "2026-11-08T02:20:54.202517Z", "body": "{\"app\":\"maborak-immutable\",\"sync\":\"Unknown\",\"revision\":\"a8228cd32e71bf054e06c8e1dbb6b382037e1ade\"}"}
### app status now
{"sync":"Synced","cond":[],"reconciledAt":"2026-11-08T02:24:33Z"}
### notification controller log lines
{"level":"info","msg":"Trigger 'on-maborak-out-of-sync' FAILED | revision:  | templates: [maborak-status]","resource":"argocd/maborak-immutable","time":"2026-11-08T02:24:33Z"}
{"level":"info","msg":"Trigger 'on-maborak-comparison-error' FAILED | revision:  | templates: [maborak-status]","resource":"argocd/maborak-immutable","time":"2026-11-08T02:24:33Z"}
{"level":"info","msg":"Trigger 'on-maborak-out-of-sync' FAILED | revision:  | templates: [maborak-status]","resource":"argocd/maborak-immutable","time":"2026-11-08T02:24:33Z"}
{"level":"info","msg":"Trigger 'on-maborak-comparison-error' FAILED | revision:  | templates: [maborak-status]","resource":"argocd/maborak-immutable","time":"2026-11-08T02:24:45Z"}
{"level":"info","msg":"Trigger 'on-maborak-out-of-sync' FAILED | revision:  | templates: [maborak-status]","resource":"argocd/maborak-immutable","time":"2026-11-08T02:24:45Z"}

Rollouts, restores and the smaller checks

Progressive sync with real webhooks. The first post measured the new 30-second wait with manual refreshes. With a Gitea webhook pointed at Argo CD and no manual refresh, step 1 of a three-step rollout deployed 3–4 seconds after the push on 3.5.7, 33–34 seconds after it on 3.6.1, and 2–5 seconds after it on 3.6.1 with --refresh-grace-period-seconds=0, five pushes each. [4] The arms ran one after another while other lab runs used the same machine, and our clock starts when git push returns; the 30-second gap is far larger than either. If you deploy on every merge, that is half a minute more per rollout unless you lower the flag.

Per-trial data: seconds from push to deploy per step, webhook-driven (15 pushes)
arm trial push_utc rev dev_s staging_s prod_s
v357 1 2026-11-08T01:09:03Z 0c5bfca 3 4 5
v357 2 2026-11-08T01:09:30Z 773de68 4 4 5
v357 3 2026-11-08T01:09:56Z cc7c9e8 3 4 5
v357 4 2026-11-08T01:10:23Z ba4360f 3 4 5
v357 5 2026-11-08T01:10:50Z 30344d6 3 4 5
v361-grace30 1 2026-11-08T01:11:16Z 395897a 34 35 36
v361-grace30 2 2026-11-08T01:12:13Z fb7d573 34 35 36
v361-grace30 3 2026-11-08T01:13:11Z aac0774 33 34 35
v361-grace30 4 2026-11-08T01:14:08Z 6a0a956 34 35 36
v361-grace30 5 2026-11-08T01:15:05Z d7c3ae3 34 35 36
v361-grace0 1 2026-11-08T01:16:24Z 20e50da 4 5 6
v361-grace0 2 2026-11-08T01:16:51Z 5a994cd 2 3 4
v361-grace0 3 2026-11-08T01:17:15Z 568b2aa 3 4 5
v361-grace0 4 2026-11-08T01:17:42Z 7b43b94 3 4 5
v361-grace0 5 2026-11-08T01:18:08Z f327fcc 5 5 6

Restoring from argocd admin export. The disaster-recovery page says the command exports “all Argo CD data”. [5] We exported our 3.6.1 instance with --strip-status and applied it to a fresh 3.6.1 cluster with kubectl apply -n argocd. All 19 applications, 5 ApplicationSets, both projects, the repository credentials, argocd-cm, argocd-rbac-cm and argocd-secret came across. argocd-cmd-params-cm did not. The export code writes argocd-cm, argocd-rbac-cm, the known-hosts and TLS ConfigMaps, the Argo CD secrets, projects, applications and ApplicationSets; it reads argocd-cmd-params-cm only to find application namespaces and does not export it, nor argocd-notifications-cm. [6] On the restored cluster an ApplicationSet that the source refuses to roll out (Degraded, no steps) generated two applications. When we copied argocd-cmd-params-cm from the source and restarted the components, that ApplicationSet turned Degraded with “No steps defined for rollout”, as on the source, which confirms the missing settings as the cause; the two applications it had already generated stayed. Many restored applications also show OutOfSync or Missing in the raw output because we restored Argo CD’s objects, not the workloads they had deployed; that part is expected. Back up argocd-cmd-params-cm and argocd-notifications-cm yourself.

Raw output: restore drill (apply output, cut at 40 lines when saved, and application status on both clusters) 
## T4 restore drill, 2026-11-08T00:45:03Z
### kubectl apply -n argocd -f export.yaml (3.6.1 export, --strip-status) into a fresh 3.6.1 cluster
   1   Warning: resource secrets/argocd-secret is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
   1   Warning: resource configmaps/argocd-tls-certs-cm is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
   1   Warning: resource configmaps/argocd-ssh-known-hosts-cm is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
   1   Warning: resource configmaps/argocd-rbac-cm is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
   1   Warning: resource configmaps/argocd-cm is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
   1   Warning: resource appprojects/default is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
   1   Warning: metadata.finalizers: "resources-finalizer.argocd.argoproj.io": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers
   1   secret/repo-3957700771 created
   1   secret/repo-2003866583 created
   1   secret/gitea-write created
   1   secret/argocd-secret configured
   1   configmap/argocd-tls-certs-cm configured
   1   configmap/argocd-ssh-known-hosts-cm configured
   1   configmap/argocd-rbac-cm configured
   1   configmap/argocd-cm configured
   1   appproject.argoproj.io/imp created
   1   appproject.argoproj.io/default configured
   1   applicationset.argoproj.io/maborak-teams created
   1   applicationset.argoproj.io/maborak-rollout created
   1   applicationset.argoproj.io/maborak-platform created
   1   applicationset.argoproj.io/maborak-no-steps created
   1   applicationset.argoproj.io/maborak-clusters created
   1   application.argoproj.io/maborak-web created
   1   application.argoproj.io/maborak-team-search created
   1   application.argoproj.io/maborak-team-payments created
   1   application.argoproj.io/maborak-slowhook created
   1   application.argoproj.io/maborak-rollout-staging created
   1   application.argoproj.io/maborak-rollout-prod created
   1   application.argoproj.io/maborak-rollout-dev created
   1   application.argoproj.io/maborak-policy-check created
   1   application.argoproj.io/maborak-nocrd created
   1   application.argoproj.io/maborak-monitoring created
   1   application.argoproj.io/maborak-ingress created
   1   application.argoproj.io/maborak-immutable created
   1   application.argoproj.io/maborak-hydrated created
   1   application.argoproj.io/maborak-history created
   1   application.argoproj.io/maborak-held created
   1   application.argoproj.io/maborak-env-staging created
   1   application.argoproj.io/maborak-env-production created
   1   application.argoproj.io/maborak-drift created
### after 10 min (2026-11-08T00:53:27Z): restore cluster vs source
## argo361
maborak-api Synced Healthy 
maborak-drift OutOfSync Healthy 
maborak-env-production Synced Healthy 
maborak-env-staging Synced Healthy 
maborak-held Synced Progressing 
maborak-history Synced Healthy 
maborak-hydrated Synced Healthy 
maborak-immutable Synced Healthy 
maborak-ingress Synced Healthy 
maborak-monitoring Synced Healthy 
maborak-nocrd OutOfSync Healthy 
maborak-policy-check Synced Healthy 
maborak-rollout-dev Synced Healthy 
maborak-rollout-prod Synced Healthy 
maborak-rollout-staging Synced Healthy 
maborak-selfheal Synced Healthy 
maborak-slowhook Synced Healthy 
maborak-team-payments Synced Healthy 
maborak-team-search Synced Healthy 
maborak-web Synced Healthy 
## argorestore
maborak-api Synced Healthy 
maborak-drift OutOfSync Missing 
maborak-env-production Synced Healthy 
maborak-env-staging Synced Healthy 
maborak-held Synced Healthy 
maborak-history OutOfSync Missing 
maborak-hydrated Synced Healthy 
maborak-immutable OutOfSync Missing 
maborak-ingress Synced Healthy 
maborak-monitoring Synced Healthy 
maborak-no-steps-dev OutOfSync Missing 
maborak-no-steps-prod OutOfSync Missing 
maborak-nocrd OutOfSync Missing 
maborak-policy-check OutOfSync Missing 
maborak-rollout-dev OutOfSync Missing 
maborak-rollout-prod OutOfSync Missing 
maborak-rollout-staging OutOfSync Missing 
maborak-slowhook OutOfSync Missing 
maborak-team-payments Synced Healthy 
maborak-team-search Synced Healthy 
maborak-web Synced Healthy 
## argocd-cmd-params-cm in restore cluster

## source
{"applicationsetcontroller.enable.progressive.syncs":"true","hydrator.enabled":"true","otlp.address":"otel-collector.argocd:4317"}
## hydrated app in restore
{"c":[],"h":null}
Raw output: positive control, argocd-cmd-params-cm copied to the restored cluster 
## Positive control: copy argocd-cmd-params-cm from the source to the restored cluster, 2026-11-08T02:39:48Z
### before: maborak-no-steps appset health and generated apps
{"health":{"message":"All applications have been generated successfully","status":"Healthy"}}
maborak-no-steps-dev OutOfSync Missing
maborak-no-steps-prod OutOfSync Missing
patched with: {"data":{"applicationsetcontroller.enable.progressive.syncs":"true","hydrator.enabled":"true","otlp.address":"otel-collector.argocd:4317"}}
### after 90 s
{"health":{"message":"No steps defined for rollout","status":"Degraded"},"conditions":[{"type":"ErrorOccurred","reason":"ApplicationSetRolloutError","message":"No steps defined for rollout"},{"type":"ParametersGenerated","reason":"ParametersGenerated","message":"All parameters have been generated successfully"},{"type":"ResourcesUpToDate","reason":"ErrorOccurred","message":"No steps defined for rollout"}]}
maborak-no-steps-dev OutOfSync Missing
maborak-no-steps-prod OutOfSync Missing
maborak-rollout-dev OutOfSync Missing
maborak-rollout-prod OutOfSync Missing
maborak-rollout-staging OutOfSync Missing

Rollback restrictions. The post says a rollback is limited to the application’s history, can only override dryRun and prune, and cannot sync part of an application. All three held: an ID that is not in the history is refused (does not have deployment id '99' in history), and an extra resources field in the API request is dropped without an error, so the rollback covers the whole application. The CLI has --prune but no --dry-run; only the API takes dryRun.

Raw output: rollback restrictions 
## T4 rollback restrictions (3.6.1), 2026-11-08T00:39:01Z
application 'maborak-history' created
### history
SOURCE  http://argolab-gitea:3000/lab/apps.git
ID      DATE                           REVISION
0       2026-11-08 00:39:02 +0000 UTC   (35b4397)
1       2026-11-08 00:39:02 +0000 UTC   (a8228cd)
### rollback CLI flags
      --prune                  Allow deleting unexpected resources
      --timeout uint           Time out after this many seconds
      --argocd-context string           The name of the Argo-CD server context to use
      --auth-token string               Authentication token; set this or the ARGOCD_AUTH_TOKEN environment variable
      --client-crt string               Client certificate file
      --client-crt-key string           Client certificate key file
      --config string                   Path to Argo CD config (default "<local path>")
      --controller-name string          Name of the Argo CD Application controller; set this or the ARGOCD_APPLICATION_CONTROLLER_NAME environment variable when the controller's name label differs from the default, for example when installing via the Helm chart (default "argocd-application-controller")
      --core                            If set to true then CLI talks directly to Kubernetes instead of talking to Argo CD API server
      --grpc-web                        Enables gRPC-web protocol. Useful if Argo CD server is behind proxy which does not support HTTP2.
      --grpc-web-root-path string       Enables gRPC-web protocol. Useful if Argo CD server is behind proxy which does not support HTTP2. Set web root.
      --http-retry-max int              Maximum number of retries to establish http connection to Argo CD server
      --insecure                        Skip server certificate and domain verification
      --kube-context string             Directs the command to the given kube-context
      --logformat string                Set the logging format. One of: json|text (default "json")
      --loglevel string                 Set the logging level. One of: debug|info|warn|error (default "info")
      --plaintext                       Disable TLS
      --port-forward                    Connect to a random argocd-server port using port forwarding
      --port-forward-namespace string   Namespace name which should be used for port forwarding
      --prompts-enabled                 Force optional interactive prompts to be enabled or disabled, overriding local configuration. If not specified, the local configuration value will be used, which is false by default.
      --redis-compress string           Enable this if the application controller is configured with redis compression enabled. (possible values: gzip, none) (default "gzip")
      --redis-haproxy-name string       Name of the Redis HA Proxy; set this or the ARGOCD_REDIS_HAPROXY_NAME environment variable when the HA Proxy's name label differs from the default, for example when installing via the Helm chart (default "argocd-redis-ha-haproxy")
      --redis-name string               Name of the Redis deployment; set this or the ARGOCD_REDIS_NAME environment variable when the Redis's name label differs from the default, for example when installing via the Helm chart (default "argocd-redis")
      --repo-server-name string         Name of the Argo CD Repo server; set this or the ARGOCD_REPO_SERVER_NAME environment variable when the server's name label differs from the default, for example when installing via the Helm chart (default "argocd-repo-server")
      --server string                   Argo CD server address
      --server-crt string               Server certificate file
      --server-name string              Name of the Argo CD API server; set this or the ARGOCD_SERVER_NAME environment variable when the server's name label differs from the default, for example when installing via the Helm chart (default "argocd-server")
### rollback to an ID not in history (99)
{"level":"fatal","msg":"application 'maborak-history' does not have deployment id '99' in history","time":"2026-11-08T00:39:02Z"}
### rollback API with an extra field (resources) - is it accepted/ignored?
Invalid content type

### rollback to ID 0 with --dry-run
Error: unknown flag: --dry-run
### rollback API: dryRun=true plus an unknown field (resources)
jq: parse error: Invalid numeric literal at line 1, column 7

### rollback API: dryRun=true only
jq: parse error: Invalid numeric literal at line 1, column 7

### rollback API (POST): dryRun=true plus a field the request does not define (resources)
{"error":null,"message":null,"sync":{"revision":"35b43971d677a76cd49692ee5cc67caac0c1a0ed","dryRun":true,"syncStrategy":{"apply":{}},"source":{"repoURL":"http://argolab-gitea:3000/lab/apps.git","path":"maborak/rollout"},"syncOptions":["CreateNamespace=true"]}}
### rollback API (POST): dryRun=true only
{"error":null,"message":null,"sync":{"revision":"35b43971d677a76cd49692ee5cc67caac0c1a0ed","dryRun":true,"syncStrategy":{"apply":{}},"source":{"repoURL":"http://argolab-gitea:3000/lab/apps.git","path":"maborak/rollout"},"syncOptions":["CreateNamespace=true"]}}

OCI files generator. Two YAML files in an OCI artifact, maborak/configs:v2.1.0, produced maborak-env-staging (2 replicas, region lab-a) and maborak-env-production (5 replicas, lab-b) with those values in their annotations.

Raw output: OCI files generator 
## T4 OCI files generator, 2026-11-08T00:38:33Z
maborak-env-production ns=maborak-production replicas=5 region=lab-b sync=Synced
maborak-env-staging ns=maborak-staging replicas=2 region=lab-a sync=Synced
[{"type":"ErrorOccurred","status":"False","message":"All applications have been generated successfully"},{"type":"ParametersGenerated","status":"True","message":"All parameters have been generated successfully"},{"type":"ResourcesUpToDate","status":"True","message":"All applications have been generated successfully"}]

Terminal and diff views are lazy-loaded. Opening a pod terminal on 3.6.1 fetched 404,626 bytes of JavaScript at that moment, including a pod-terminal chunk; opening a diff fetched 106,196 bytes including app-resources-diff. On 3.5.7 both were already inside the 3.48 MB main.js.

Raw output: JavaScript fetched when opening the terminal and the diff, 3.5.7 and 3.6.1 
T4-lazy 357 app tree loaded: new_js_files=2 bytes=3478669 ["main.2cde79b47a6b020c2119.js:3478669","extensions.js:0"]
diff click fail page.click: Timeout 30000ms exceeded.
T4-lazy 357 -> diff panel: new_js_files=0 bytes=0 []
T4-lazy 357 -> pod panel (summary): new_js_files=3 bytes=3203599 ["4614.785b00342aa638828e0d.chunk.js:2998345","1817.35bc2595ef84f99b2501.chunk.js:3668","editor.worker.js:201586"]
T4-lazy 357 -> terminal tab: new_js_files=0 bytes=0 []
T4-lazy 361 app tree loaded: new_js_files=17 bytes=3532328 ["main.c25be9770a9983c81271.js:206427","runtime.d265cbbcc504ea212795.js:7459","vendors.1a64bfbed4f625c48d2f.js:1431764","extensions.js:0","8532.875f96ccb46f7f4e065a.chunk.js:18958","1864.0a62e980d3a4535627c6.chunk.js:11058","4720.569c27eb7e1ae91fe4d2.chunk.js:59761","1391.8d4d757f0ee82837c27f.chunk.js:218165","login.a327baee153060901109.chunk.js:13050","9394.098318ad28f8f5cffa40.chunk.js:479797","applications.c0b51ee668346620f6a8.chunk.js:149075","1864.0a62e980d3a4535627c6.chunk.js:11058","4720.569c27eb7e1ae91fe4d2.chunk.js:59761","1391.8d4d757f0ee82837c27f.chunk.js:218165","8532.875f96ccb46f7f4e065a.chunk.js:18958","applications.c0b51ee668346620f6a8.chunk.js:149075","9394.098318ad28f8f5cffa40.chunk.js:479797"]
diff click fail page.click: Timeout 30000ms exceeded.
T4-lazy 361 -> diff panel: new_js_files=0 bytes=0 []
T4-lazy 361 -> pod panel (summary): new_js_files=3 bytes=3202923 ["5857.516678390fb517b0025c.chunk.js:2997667","1817.35bc2595ef84f99b2501.chunk.js:3668","editor.worker.js:201588"]
T4-lazy 361 -> terminal tab: new_js_files=2 bytes=404626 ["523.90db0e5a5d6843067aa4.chunk.js:398382","pod-terminal.993a2e062ec614c094e8.chunk.js:6244"]
T4-lazy 357 drift app tree loaded: new_js_files=2 bytes=3478669 ["main.2cde79b47a6b020c2119.js:3478669","extensions.js:0"]
T4-lazy 357 -> diff panel: new_js_files=0 bytes=0 []
T4-lazy 361 drift app tree loaded: new_js_files=17 bytes=3532328 ["main.c25be9770a9983c81271.js:206427","extensions.js:0","runtime.d265cbbcc504ea212795.js:7459","vendors.1a64bfbed4f625c48d2f.js:1431764","1864.0a62e980d3a4535627c6.chunk.js:11058","4720.569c27eb7e1ae91fe4d2.chunk.js:59761","1391.8d4d757f0ee82837c27f.chunk.js:218165","8532.875f96ccb46f7f4e065a.chunk.js:18958","applications.c0b51ee668346620f6a8.chunk.js:149075","9394.098318ad28f8f5cffa40.chunk.js:479797","login.a327baee153060901109.chunk.js:13050","1391.8d4d757f0ee82837c27f.chunk.js:218165","1864.0a62e980d3a4535627c6.chunk.js:11058","4720.569c27eb7e1ae91fe4d2.chunk.js:59761","8532.875f96ccb46f7f4e065a.chunk.js:18958","9394.098318ad28f8f5cffa40.chunk.js:479797","applications.c0b51ee668346620f6a8.chunk.js:149075"]
T4-lazy 361 -> diff panel: new_js_files=2 bytes=106196 ["7483.44ac082d3b24ed2db702.chunk.js:96684","app-resources-diff.d4e0bd1e685791088c94.chunk.js:9512"]

The CPU cost of manifest compression. The announcement puts it at about 2.5% CPU. [8] On the 5,000-ConfigMap cluster from the first post we restarted the controller five times with compression off and five times on, and measured its CPU over ten hard refreshes. Off took a median of 46.7 CPU-seconds, on 48.4: 3.6% more, with a 95% interval of 0.99 to 1.05 times and Mann-Whitney p=0.095. Idle CPU did not differ (3.16 and 3.21 seconds over five minutes). That leans the same way as the announcement without confirming it. The run always measured compression off before on, overlapped other lab runs and had periodic reconciliation turned off on that cluster, so read it as “no large cost”, not as a measured 3.6%.

Analysis output and per-start data: controller CPU, compression off and on (5 starts each) 
refresh_cpu_s_10x: off n=5 [46.44, 46.59, 46.73, 47.86, 48.06] median 46.73 | on n=5 [47.32, 47.61, 48.39, 48.67, 49.02] median 48.39 | ratio 1.036 CI95 [0.991,1.049] MWU p=0.095
idle_cpu_s_300s: off n=5 [2.96, 3.1, 3.16, 3.64, 3.66] median 3.16 | on n=5 [3.14, 3.18, 3.21, 3.37, 3.41] median 3.21 | ratio 1.016 CI95 [0.869,1.100] MWU p=0.84

arm,start,idle_cpu_s_300s,refresh_cpu_s_10x,refresh_window_s,cache_objects
off,1,3.64,46.73,206,5406
on,1,3.21,47.32,205,5406
off,2,3.10,46.59,206,5406
on,2,3.41,49.02,206,5406
off,3,2.96,47.86,206,5406
on,3,3.37,48.39,206,5406
off,4,3.66,48.06,206,5406
on,4,3.18,47.61,206,5406
off,5,3.16,46.44,206,5406
on,5,3.14,48.67,206,5406

Limits

  • One host, small scale. All clusters ran in one Colima VM on one arm64 Mac, sharing one Gitea and one registry. Timings and request counts compare versions; they are not capacity numbers.
  • Several runs shared the machine. While the failure-mode trials ran, we also patched argocd-cm, set up notifications and created the self-heal application on the same clusters; the upgrade drill overlapped the webhook and CPU runs. The controller logs show the reconciles we counted as its two-minute “comparison expired” refreshes, not as side effects of that work.
  • Small n, one application per case. Five trials per failure mode on 3.6.1 and three on 3.5.7, each repeating a hard refresh on the same application; one application per version in the 20-minute watch; one fleet and three cycles in the drill; one trial for the notification trigger.
  • Changes to our written plan. We checked that the live objects differed from Git at the end, not in every trial. The missing-CRD case was planned as a failure mode and became a control when its dry run did not fail. For the drill we planned per-minute rates and medians and report time to each milestone instead; we also expected a short burst of dry runs, which it did not show, and its third cycle has no samples between 153 and 768 seconds, when another run was loading the machine. We added the notification-trigger trial and the restore control after the review of this draft. The CPU test used a 5-minute idle window instead of 10, always ran compression off before on, and ran on a cluster with periodic reconciliation turned off.
  • Not tested. Kyverno and Gatekeeper themselves (we used a ValidatingAdmissionPolicy and our own webhook), StatefulSets, a hand edit of an application that was in sync, the opt-out for the immutable and webhook cases, HA installs, the Helm chart upgrade path and any production-size fleet.
  • The writer ran the lab. The same AI agent designed the tests, ran them and wrote this post. Scoring is by script, and a separate agent reviewed the methods; its points and our answers are in our research notes.

Sources

  1. “Argo CD 3.6 in a Lab: The Release Post’s 32 Claims, Checked”, Maborak, 7 November 2026. https://www.maborak.com/blog/argo-cd-3-6-claims-checked. The first false-Synced result and the lab setup.
  2. PR #29103 “Default to SSD with SSA and remove SMD”, merged 9 September 2026. https://github.com/argoproj/argo-cd/pull/29103 (opens in a new tab). Server-Side Diff becomes the default for SSA applications.
  3. “v3.5 to 3.6” upgrade guide at v3.6.1, docs/operator-manual/upgrading/3.5-3.6.md. https://github.com/argoproj/argo-cd/blob/v3.6.1/docs/operator-manual/upgrading/3.5-3.6.md (opens in a new tab). “No action is required”, the opt-out annotation.
  4. argocd-applicationset-controller command reference at v3.6.1. https://github.com/argoproj/argo-cd/blob/v3.6.1/docs/operator-manual/server-commands/argocd-applicationset-controller.md (opens in a new tab). --refresh-grace-period-seconds, default 30.
  5. Disaster recovery documentation at v3.6.1, docs/operator-manual/disaster_recovery.md. https://github.com/argoproj/argo-cd/blob/v3.6.1/docs/operator-manual/disaster_recovery.md (opens in a new tab). “Import and export all Argo CD data”, --strip-status.
  6. cmd/argocd/commands/admin/backup.go at v3.6.1. https://github.com/argoproj/argo-cd/blob/v3.6.1/cmd/argocd/commands/admin/backup.go (opens in a new tab). Which ConfigMaps the export writes, and that it reads argocd-cmd-params-cm only for application namespaces.
  7. Issue #29348 “ComparisonError when using ServerSideDiff=true”, opened 24 August 2026. https://github.com/argoproj/argo-cd/issues/29348 (opens in a new tab). The immutable-field error with Server-Side Diff; it does not describe the later Synced.
  8. PR #28600 “configurable serialization and compression for cached manifests”, merged 10 September 2026. https://github.com/argoproj/argo-cd/pull/28600 (opens in a new tab). The compression settings.
  9. “Automated Sync Policy” documentation at v3.6.1, docs/user-guide/auto_sync.md. https://github.com/argoproj/argo-cd/blob/v3.6.1/docs/user-guide/auto_sync.md (opens in a new tab). Automated sync runs only when an application is OutOfSync and does not retry a failed sync of the same commit.

Topics: kubernetes, sre, migration

Back to all articles

Technologies we work with

  • Argo CD
  • Kubernetes
  • Docker
  • Podman
  • Ansible
  • Jenkins
  • Chef
  • Proxmox VE
  • Google Cloud
  • Linux
  • Red Hat
  • CentOS
  • NGINX
  • Apache
  • Traefik
  • Prometheus
  • Grafana
  • PostgreSQL
  • MySQL
  • MariaDB
  • Redis
  • etcd
  • Qdrant
  • Elasticsearch
  • Logstash
  • Kibana
  • PagerDuty
  • Go
  • Python
  • FastAPI
  • PHP
  • TypeScript
  • React