maborak.com~/blog/argo-cd-3-6-claims-checkedsoftware & infrastructure engineering
MaborakTechnologies
Contact
Back to all articles

Argo CD 3.6 in a Lab: The Release Post's 32 Claims, Checked

Overview

We ran Argo CD 3.5.7 and 3.6.1 side by side, tested all 32 claims in the 3.6 announcement and screenshot every new feature. Most hold; the SSA diff change is missing.

By the Maborak engineering team · · 45 min read

We tested all 32 claims in the Argo CD 3.6 announcement [1] on two kind clusters, one running 3.5.7 and one running 3.6.1. Twenty-six behaved as described, eight of them with a gap. Filtering in the UI got a third cheaper, as promised; search did not. Compression halved the controller’s cache in our lab, once switched on.

Seven details in the post do not match the software: three links (two to the wrong pull request, one broken), the sampling variable’s name, the count of new health checks (55, not about 20), a filter panel on the Projects page that does not exist, and the rollout condition names. Three and a half of its seven “notable bug fixes” already shipped in 3.5 patch releases.

The change most likely to surprise someone upgrading is not in the post. Applications with ServerSideApply=true now compute their diff with a dry-run apply against the API server. In our lab every manual refresh cost one dry-run request per managed resource, and an admission policy that used to fail the sync now failed the diff. After that failure, the next periodic reconcile reported the application Synced while the cluster still differed from Git, in 5 of 5 trials. [3] [4]

Checked 7 November 2026.

How we checked

Argo CD 3.6.0 was released on 3 November 2026 and 3.6.1, with two UI fixes, the next day. [2] The announcement’s URL still says “rc1”, but its page metadata shows it was first published on 16 September and last modified on 4 November, and the text now describes the final release. [1]

The lab ran kind v0.33.0 with Kubernetes v1.37.0 on an arm64 Mac. Both Argo CD versions were installed from their release install.yaml, with a Gitea container as the Git server and registry:2 for OCI artifacts. UI checks used Playwright with Chrome for Testing 156. Every application, namespace, ApplicationSet, OCI artifact and finalizer in the screenshots carries the maborak prefix: they are objects we created and clicked through ourselves, not images from the announcement. Where a claim is a count or a backport, we checked the Git tags. Every number below is computed by script from raw output, and the raw output and per-trial data sit in collapsed blocks beside each result. An AI coding agent built the lab, chose the follow-up runs and drafted this post; no model judged any result, and a separate agent reviewed the methods before publication. Each result says whether its analysis was planned before the data or chosen afterwards, and Limits lists them. If you are still below 3.5.6, read our October security post first.

The change the post leaves out: SSA now means Server-Side Diff

Before 3.6, an application with the ServerSideApply=true sync option compared Git with the cluster using Structured-Merge Diff, a local approximation. PR #29103, merged on 9 September 2026, removes that strategy: SSA now implies Server-Side Diff, which asks the API server to dry-run the apply. [4] The upgrade guide files this under “Behavioral Improvements” and says “No action is required”. [3]

We tested what “no action” means with a ValidatingAdmissionPolicy that rejects ConfigMaps carrying a lab-reject label (maborak-reject in the screenshots, which were taken later with renamed objects). A live ConfigMap existed without the label, and Git added it. Each arm got five hard refreshes and one sync attempt, with impersonation off.

Version Application status Where the policy error appears
3.5.7 OutOfSync (5 of 5 refreshes) only when you sync
3.6.1 Unknown, ComparisonError on every refresh (5 of 5): forbidden: ValidatingAdmissionPolicy 'lab-reject-label' ... denied request, and again at sync
3.6.1 with ServerSideDiff=false OutOfSync (5 of 5 refreshes) only when you sync

Argo CD 3.5.7 application maborak-policy-check: App health Healthy, Sync status OutOfSync from main (cacaca8), auto sync not enabled; tree shows ConfigMap maborak-guarded out of sync.

3.5.7: the policy-blocked change is plain OutOfSync. Captured on our 3.5.7 instance on 7 November 2026, 23:59 UTC.

Argo CD 3.6.1 application maborak-policy-check: App health Healthy, Sync status Unknown at main (cacaca8), App conditions 1 Error.

3.6.1, same Git commit and cluster state: Unknown with one error. Captured on our 3.6.1 instance on 7 November 2026, 23:59 UTC.

Argo CD 3.6.1 Application conditions panel: ComparisonError, Failed to compare desired state to live state: failed to calculate diff: error calculating server side diff: serverSideDiff error: error running server side apply in dryrun mode for resource ConfigMap/maborak-guarded: configmaps “maborak-guarded” is forbidden: ValidatingAdmissionPolicy ‘lab-reject-label’ with binding ‘lab-reject-label’ denied request: Maborak policy: ConfigMaps may not carry the maborak-reject label.

The error behind it: the admission policy rejected Argo CD’s dry run. Captured on our 3.6.1 instance on 7 November 2026, 23:59 UTC.

Seeing the rejection before a sync is useful. A dashboard full of Unknown the morning after an upgrade is less so, if your clusters run policy engines that reject manifests teams have been syncing around.

The error does not stay

We then watched the same application without touching it. After each hard refresh it showed the error; after the next periodic reconcile, about three minutes later, it showed Synced with no condition, although the live ConfigMap still had data: {k: live} and no label while Git has k: from-git and the label.

Application After hard refresh After the next periodic reconcile
3.6.1, policy blocks the change Unknown, ComparisonError (5 of 5) Synced, no condition (5 of 5)
3.6.1, same drift, no policy OutOfSync (5 of 5) OutOfSync (5 of 5)
3.5.7, policy blocks the change OutOfSync (5 of 5) OutOfSync (5 of 5)
3.5.7 with ServerSideDiff=true, policy blocks the change Unknown, ComparisonError (3 of 3) Synced (3 of 3)

Argo CD 3.6.1 application maborak-policy-check showing App health Healthy and Sync status Synced to main (cacaca8), with ConfigMap maborak-guarded marked synced, although the live object differs from Git.

The false Synced, a few minutes after the error. At this moment kubectl get configmap maborak-guarded returned data: {"k":"live"} and no labels. Captured on our 3.6.1 instance on 8 November 2026, 00:03 UTC.

A drifted application with no policy stays OutOfSync, so the false Synced appears only after the server-side dry run has failed. 3.5.7 shows the same flip once Server-Side Diff is switched on, so this is not new code; what 3.6 changes is that every ServerSideApply=true application now takes this path by default. Five of five against zero of five gives Fisher’s exact p=0.008; with five trials per arm, the Wilson 95% interval for the flip rate is 57–100%. This was found while preparing screenshots, after the review, so treat it as an exploratory result. We did not find a matching upstream issue; the closest, #29348, reports the error itself, not the flip. If you run admission policies, do not trust a Synced badge on an SSA application until a hard refresh agrees.

Per-trial data: status after a hard refresh and after the next periodic reconcile (23 trials)
version app trial after_hard after_periodic periodic_at
357 maborak-drift 1 OutOfSync/none OutOfSync/none 2026-11-07T23:43:03Z
357 maborak-drift 2 OutOfSync/none OutOfSync/none 2026-11-07T23:47:13Z
357 maborak-drift 3 OutOfSync/none OutOfSync/none 2026-11-07T23:51:14Z
357 maborak-drift 4 OutOfSync/none OutOfSync/none 2026-11-07T23:55:08Z
357 maborak-drift 5 OutOfSync/none OutOfSync/none 2026-11-07T23:57:20Z
357 maborak-policy-check 1 OutOfSync/none OutOfSync/none 2026-11-07T23:42:54Z
357 maborak-policy-check 2 OutOfSync/none OutOfSync/none 2026-11-07T23:45:18Z
357 maborak-policy-check 3 OutOfSync/none OutOfSync/none 2026-11-07T23:48:44Z
357 maborak-policy-check 4 OutOfSync/none OutOfSync/none 2026-11-07T23:51:02Z
357 maborak-policy-check 5 OutOfSync/none OutOfSync/none 2026-11-07T23:54:33Z
357 maborak-policy-ssd 1 Unknown/ComparisonError Synced/none 2026-11-08T00:02:46Z
357 maborak-policy-ssd 2 Unknown/ComparisonError Synced/none 2026-11-08T00:04:51Z
357 maborak-policy-ssd 3 Unknown/ComparisonError Synced/none 2026-11-08T00:07:13Z
361 maborak-drift 1 OutOfSync/none OutOfSync/none 2026-11-07T23:43:07Z
361 maborak-drift 2 OutOfSync/none OutOfSync/none 2026-11-07T23:47:13Z
361 maborak-drift 3 OutOfSync/none OutOfSync/none 2026-11-07T23:49:17Z
361 maborak-drift 4 OutOfSync/none OutOfSync/none 2026-11-07T23:53:08Z
361 maborak-drift 5 OutOfSync/none OutOfSync/none 2026-11-07T23:56:59Z
361 maborak-policy-check 1 Unknown/ComparisonError Synced/none 2026-11-07T23:42:52Z
361 maborak-policy-check 2 Unknown/ComparisonError Synced/none 2026-11-07T23:45:03Z
361 maborak-policy-check 3 Unknown/ComparisonError Synced/none 2026-11-07T23:48:27Z
361 maborak-policy-check 4 Unknown/ComparisonError Synced/none 2026-11-07T23:51:12Z
361 maborak-policy-check 5 Unknown/ComparisonError Synced/none 2026-11-07T23:53:17Z

We counted the API server’s apiserver_request_total{dry_run="All"} around single refreshes, with no other SSA application on the cluster. A 10-resource SSA application on 3.6.1 added exactly 10 dry-run requests per refresh, for 5 of 5 normal refreshes and 5 of 5 hard refreshes; a 3-resource application added 3. On 3.5.7 the same application added none. A periodic reconcile on 3.6.1 (its reconciledAt advanced during a 240-second idle window) added none either, because the diff result is cached until something changes. So the cost is one dry-run per managed resource for every refresh someone or something requests: a webhook, the UI button, argocd app get --refresh, or a new commit.

Raw output: dry-run requests per refresh, 10-resource and 3-resource applications 
## SSD10 per-refresh dry-run deltas, 2026-11-07T21:43:25Z
### 357 SSA/SSD apps: ssa10 sync=Synced
357 normal refresh deltas: 0 0 0 0 0
357 hard refresh deltas: 0 0 0 0 0
357 idle 240 s: delta=0 reconciledAt before=2026-11-07T21:44:23Z after=2026-11-07T21:47:43Z
### 361 SSA/SSD apps: ssa10 sync=Synced
361 normal refresh deltas: 10 10 10 10 10
361 hard refresh deltas: 10 10 10 10 10
361 idle 240 s: delta=0 reconciledAt before=2026-11-07T21:49:26Z after=2026-11-07T21:51:43Z
## SSD apiserver dry-run requests per hard refresh of app ssa-plain (3 resources, ServerSideApply=true), 2026-11-07T20:20:11Z
dry_run=All verbs on 361:    4  APPLY;   4  POST;
357: dry-run requests before=2 after=2 delta=0 over 10 hard refreshes
361: dry-run requests before=13 after=43 delta=30 over 10 hard refreshes
361: normal (non-hard) refreshes x10: dry-run delta=30
361: idle 200 s (periodic reconcile only): dry-run delta=0
Raw output: admission policy runs, five hard refreshes per arm and sync attempts 
## SSD default with ServerSideApply=true (not in the post), 2026-11-07T20:17:37Z
setup: VAP denies ConfigMaps labelled lab-reject in vap-ns; live ConfigMap vap-ns/guarded exists without the label; Git adds the label
### 357 ssa-vap
{"sync":"OutOfSync","conditions":[]}
### 357 ssa-vap-optout
{"sync":"OutOfSync","conditions":[]}
### 361 ssa-vap
{"sync":"Unknown","conditions":[{"type":"ComparisonError","message":"Failed to compare desired state to live state: StateDiffs error: diffConfig can not be nil"},{"type":"UnknownError","message":"no matching service account found for destination server https://kubernetes.default.svc and namespace vap-ns"}]}
### 361 ssa-vap-optout
{"sync":"OutOfSync","conditions":[]}
### impersonation disabled, 2026-11-07T20:18:58Z
### 357 ssa-vap
{"sync":"OutOfSync","conditions":[]}
### 357 ssa-vap-optout
{"sync":"OutOfSync","conditions":[]}
### 361 ssa-vap
{"sync":"Unknown","conditions":[{"type":"ComparisonError","message":"Failed to compare desired state to live state: failed to calculate diff: error calculating server side diff: serverSideDiff error: error running server side apply in dryrun mode for resource ConfigMap/guarded: configmaps \"guarded\" is forbidden: ValidatingAdmissionPolicy 'lab-reject-label' with bindi"}]}
### 361 ssa-vap-optout
{"sync":"OutOfSync","conditions":[]}
### sync attempt 3.5.7 ssa-vap
TIMESTAMP                  GROUP        KIND   NAMESPACE                  NAME    STATUS    HEALTH        HOOK  MESSAGE
2026-11-07T20:18:58+00:00          ConfigMap      vap-ns               guarded  OutOfSync  Healthy              configmaps "guarded" is forbidden: ValidatingAdmissionPolicy 'lab-reject-label' with binding 'lab-reject-label' denied request: lab policy: ConfigMaps may not carry the lab-reject label
Message:            one or more objects failed to apply, reason: configmaps "guarded" is forbidden: ValidatingAdmissionPolicy 'lab-reject-label' with binding 'lab-reject-label' denied request: lab policy: ConfigMaps may not carry the lab-reject label
## VAP5, impersonation off, 2026-11-07T21:54:04Z
357 ssa-vap statuses: OutOfSync/none OutOfSync/none OutOfSync/none OutOfSync/none OutOfSync/none
357 ssa-vap sync attempt: denied request: lab policy: ConfigMaps may not carry the lab-reject label
361 ssa-vap statuses: Unknown/ComparisonError Unknown/ComparisonError Unknown/ComparisonError Unknown/ComparisonError Unknown/ComparisonError
361 ssa-vap sync attempt: denied request: lab policy: ConfigMaps may not carry the lab-reject label
361 ssa-vap-optout statuses: OutOfSync/none OutOfSync/none OutOfSync/none OutOfSync/none OutOfSync/none
361 ssa-vap-optout sync attempt: denied request: lab policy: ConfigMaps may not carry the lab-reject label

The opt-out works per application:

metadata:
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=false

Impersonation interacts with this. The post lists PR #28821, “server-side diff respects impersonation”, as a notable fix, and we reproduced it: with a project mapped to a service account that has no RBAC, 3.5.7 reported OutOfSync with no error, while 3.6.1 reported configmaps "tiny" is forbidden: User "system:serviceaccount:imp-ns:limited" cannot patch resource "configmaps". [6] Since that account cannot patch anything, a successful dry run on 3.5.7 must have used other credentials, the controller’s own (an inference; we did not capture an audit log). With impersonation enabled and no service account mapped for a destination, an SSA application on 3.6.1 showed StateDiffs error: diffConfig can not be nil; on 3.5.7 it showed OutOfSync. If you enable impersonation, map every destination before upgrading.

Raw output: Server-Side Diff with impersonation (#28821), 3.5.7 and 3.6.1 
## C32 #28821 Server-Side Diff and impersonation, 2026-11-07T20:16:28Z
setup: impersonation enabled; project imp maps imp-ns to SA imp-ns:limited (no RBAC); app imp-ssd uses ServerSideDiff=true + ServerSideApply=true
### 357
{"sync":"OutOfSync","conditions":[]}
### 361
{"sync":"OutOfSync","conditions":[]}
### can SA limited create configmaps in imp-ns?
no
### after creating live ConfigMap imp-ns/tiny (data k=live-value) so the diff needs a dry-run, 2026-11-07T20:16:52Z
### 357
{"sync":"OutOfSync","conditions":[]}
### 361
{"sync":"Unknown","conditions":[{"type":"ComparisonError","message":"Failed to compare desired state to live state: failed to calculate diff: error calculating server side diff: serverSideDiff error: error running server side apply in dryrun mode for resource ConfigMap/tiny: configmaps \"tiny\" is forbidden: User \"system:serviceaccount:imp-ns:limited\" cannot patch resource \"configmaps\" in API group \"\" in the namespace \"imp-ns\""}]}

What is new in 3.6, screen by screen

All screenshots in this section come from our 3.6.1 instance, captured with Playwright at three times device scale on 7 and 8 November 2026. Where 3.5.7 lacks the feature we say what it shows instead.

Resources page

A new top-level Resources page lists every managed resource across all applications, with health, sync status, cluster and owning application, and the same filters as the application list. Searching maborak below returns our demo objects across eight applications; a Deployment does not drag its ReplicaSets and Pods along, as the post says. Search, filters, the summary view and the open resource panel are all written to the URL, so a filtered view can be shared as a link. A sidebar filter survived navigating away and back; the search text did not. With RBAC limited to one project, our test user saw only that project’s resources. 3.5.7 has no such page; the route renders blank. It can be switched off with ui.view.resources.disabled: "true".

Raw output: Resources page with an RBAC-limited user 
C1 RBAC user=admin rows for search imp-: ["ConfigMap imp-ns tiny"] apps visible via API: 1019
C1 RBAC user=dev rows for search imp-: [] apps visible via API: 1018

Argo CD 3.6.1 Resources page with search maborak: rows for Service and Deployment maborak-api, ConfigMaps maborak-drift, maborak-guarded, maborak-held-config, Service maborak-held-svc, ConfigMaps maborak-ingress, maborak-monitoring and maborak-release, each with namespace, cluster in-cluster, owning application and Healthy plus Synced or OutOfSync; health filter shows Healthy 19.

Resources page filtered to our maborak objects. Captured on our 3.6.1 instance on 7 November 2026, 23:58 UTC.

Settings: search, filters and the Advanced page

Repositories, Clusters, Projects, Accounts, GPG Keys and Certificates gained a search box whose text goes into the URL. All of them except Projects also gained a filter sidebar with counts, although the post includes Projects. A new Settings → Advanced page shows the instance configuration as a form and, on the JSON tab, as raw JSON, including our Maborak Lab IdP connector.

Argo CD 3.6.1 Settings Advanced page, JSON tab: url https://localhost:8361, dexConfig with connector name Maborak Lab IdP, type mockCallback, id mock, appLabelKey app.kubernetes.io/instance and resourceOverrides including the ConfigMap health.lua.

The new Advanced page, JSON tab. Captured on our 3.6.1 instance on 7 November 2026, 23:58 UTC.

The .* button next to the search box turns the query into a regular expression, in the Applications and ApplicationSets lists and on the Resources page. ^maborak-(web|api)$ matches nothing as plain text and exactly two applications as a regex.

Argo CD 3.6.1 Applications list with regex search ^maborak-(web|api)$ enabled, showing two rows: maborak-api and maborak-web, both default project, label team=maborak, Healthy and Synced.

Regex search on. Captured on our 3.6.1 instance on 7 November 2026, 23:58 UTC.

Faster lists: virtual scrolling and lazy loading

These two do not show in a screenshot, so we counted. With “Items per page: all” and 1,011 applications, 3.5.7 rendered 1,011 tiles and 60,349 DOM nodes; 3.6.1 rendered 33 tiles and 3,469 nodes. Virtualization starts above 50 matching applications (VIRTUAL_THRESHOLD = 50): a regex matching exactly 50 rendered 50 rows, one matching 51 rendered 30.

Raw output: rendered tiles, rows and DOM nodes with "all" selected 
C8 357 tiles load1: {"tiles":1011,"rows":0,"domNodes":60349} ms_to_first_render_after_click=773
C8 357 tiles load2: {"tiles":1011,"rows":0,"domNodes":60349} ms_to_first_render_after_click=297
C8 357 tiles load3: {"tiles":1011,"rows":0,"domNodes":60349} ms_to_first_render_after_click=243
C8 357 list load1: {"tiles":0,"rows":1011,"domNodes":53219} ms_to_first_render_after_click=232
C8 357 list load2: {"tiles":0,"rows":1011,"domNodes":53219} ms_to_first_render_after_click=241
C8 357 list load3: {"tiles":0,"rows":1011,"domNodes":53219} ms_to_first_render_after_click=254
C8 361 tiles load1: {"tiles":33,"rows":0,"domNodes":3464} ms_to_first_render_after_click=67
C8 361 tiles load2: {"tiles":33,"rows":0,"domNodes":3469} ms_to_first_render_after_click=27
C8 361 tiles load3: {"tiles":33,"rows":0,"domNodes":3469} ms_to_first_render_after_click=48
C8 361 list load1: {"tiles":0,"rows":29,"domNodes":3096} ms_to_first_render_after_click=32
C8 361 list load2: {"tiles":0,"rows":29,"domNodes":3096} ms_to_first_render_after_click=28
C8 361 list load3: {"tiles":0,"rows":29,"domNodes":3096} ms_to_first_render_after_click=46
C8 threshold 50 apps (^load-00([0-4][0-9]|50)$): rendered rows=50
C8 threshold 51 apps (^load-00([0-4][0-9]|5[01])$): rendered rows=30
ERR page.click: Timeout 30000ms exceeded. | Call log: |   - waiting for locator('[qe-id="paginate-items-per-page-applications-list"]') |     - locator resolved to <div class="argo-dropdown__anchor" qe-id="paginate-items-per-page-applications-list">…</div>

Lazy loading shows up in the network panel, though not for every view the post lists. 3.5.7 downloads one 3,478,669-byte main.js before anything renders; it already loaded the Monaco editor separately when the log viewer opened. 3.6.1 downloads 2,600,195 bytes in 12 files, then fetches a 251,239-byte settings bundle the first time you open Settings and a pod-logs chunk for the log viewer. Opening an application’s tree view fetched nothing new, and we did not test the terminal or the diff view. Applications and ApplicationSets tiles, rows and sidebar items are now real <a href> links (3.5.7 used <div>), so middle-click and “open in new tab” work.

Raw output: JavaScript files fetched per navigation, 3.5.7 and 3.6.1 
C7 357 initial load of /user-info: new_js_files=2 decoded_bytes=3478669 ["main.2cde79b47a6b020c2119.js:3478669","extensions.js:0"]
C7 357 -> Applications list: new_js_files=0 decoded_bytes=0 []
C7 357 -> app tree: new_js_files=0 decoded_bytes=0 []
C7 357 -> pod logs tab: new_js_files=3 decoded_bytes=3203599 ["4614.785b00342aa638828e0d.chunk.js:2998345","1817.35bc2595ef84f99b2501.chunk.js:3668","editor.worker.js:201586"]
C7 357 -> Settings: new_js_files=0 decoded_bytes=0 []
C7 357 -> ApplicationSets: new_js_files=0 decoded_bytes=0 []
C7 361 initial load of /user-info: new_js_files=12 decoded_bytes=2600195 ["main.c25be9770a9983c81271.js:206427","runtime.d265cbbcc504ea212795.js:7459","extensions.js:0","vendors.1a64bfbed4f625c48d2f.js:1431764","1864.0a62e980d3a4535627c6.chunk.js:11058","1391.8d4d757f0ee82837c27f.chunk.js:218165","4720.569c27eb7e1ae91fe4d2.chunk.js:59761","8532.875f96ccb46f7f4e065a.chunk.js:18958","login.a327baee153060901109.chunk.js:13050","9394.098318ad28f8f5cffa40.chunk.js:479797","applications.c0b51ee668346620f6a8.chunk.js:149075","user-info.4169aa64d197b445b669.chunk.js:4681"]
C7 361 -> Applications list: new_js_files=0 decoded_bytes=0 []
C7 361 -> app tree: new_js_files=0 decoded_bytes=0 []
C7 361 -> pod logs tab: new_js_files=5 decoded_bytes=3261661 ["5857.516678390fb517b0025c.chunk.js:2997667","1817.35bc2595ef84f99b2501.chunk.js:3668","editor.worker.js:201588","7689.78a782fc9d00f3625ea0.chunk.js:21343","pod-logs.3729c09629c850c29dc8.chunk.js:37395"]
C7 361 -> Settings: new_js_files=2 decoded_bytes=251239 ["7973.856aeb69764db7e07fe4.chunk.js:62301","settings.83e13d827d143f607420.chunk.js:188938"]
C7 361 -> ApplicationSets: new_js_files=0 decoded_bytes=0 []

Log viewer and events

The pod log viewer has a Clear displayed logs button (the eraser). It empties the view; new lines keep streaming in. Event lists gained a Component column with the reporting component, here default-scheduler and kubelet. 3.5.7 has neither.

Argo CD 3.6.1 pod log viewer for pod maborak-web with tooltip Clear displayed logs over the eraser button; log lines read maborak-web: verified by Maborak lab, request N.

Our maborak-web pod writes “verified by Maborak lab” on every line. Captured on our 3.6.1 instance on 7 November 2026, 23:58 UTC.

Argo CD 3.6.1 events table for a maborak-web pod with columns Reason, Message, Component, Count, First occurred, Last occurred; rows Scheduled from default-scheduler and Pulled from kubelet.

The new Component column. Captured on our 3.6.1 instance on 7 November 2026, 23:59 UTC.

SSO button label

ui.loginButtonText in argocd-cm replaces the SSO button text. With the same setting, 3.5.7 kept its default “LOG IN VIA MABORAK LAB IDP”; 3.6.1 shows our text. The response also carries X-Robots-Tag: noindex, nofollow, so a public instance asks search engines not to index it.

Raw output: response headers, 3.5.7 and 3.6.1 
## C10 X-Robots-Tag, 2026-11-07T19:05:52Z
### port 8357
HTTP/1.1 200 OK
HTTP/1.1 200 OK
### port 8361
HTTP/1.1 200 OK
X-Robots-Tag: noindex, nofollow
HTTP/1.1 200 OK

Argo CD 3.6.1 login form with an SSO button reading SIGN IN WITH MABORAK SSO above the username and password fields.

Our button text from ui.loginButtonText: "Sign in with Maborak SSO". Captured on our 3.6.1 instance on 8 November 2026, 00:00 UTC.

Health: deletion messages and the resource that caused it

A resource stuck on a finalizer used to read only “Pending deletion”. 3.6.1 lists the finalizers, and a Lua health check can return a deletionMessage that explains them. We deleted a Service and a ConfigMap that carry our maborak.com/hold finalizer: the Service reads “Pending deletion; blocked by finalizers: maborak.com/hold”, and the ConfigMap shows the message from our Lua check. 3.5.7 shows “Pending deletion” for both. The application header also names the resource behind its health (“Caused by Service:maborak-held/maborak-held-svc”), a 3.6 change (#28455) the post does not mention.

Raw output: health of resources pending deletion, 3.5.7 and 3.6.1 
## C30 deletion health (Service with finalizer), 2026-11-07T19:07:45Z
### 357
{"kind":"Service","name":"held-svc","health":null}
### 361
{"kind":"Service","name":"held-svc","health":null}
### 357
### 361
### resource-tree health, 2026-11-07T19:08:12Z
### 357
{"kind":"ConfigMap","name":"held","health":{"status":"Progressing","message":"Pending deletion"}}
{"kind":"Service","name":"held-svc","health":{"status":"Progressing","message":"Pending deletion"}}
### 361
{"kind":"ConfigMap","name":"held","health":{"status":"Progressing","message":"Pending deletion; blocked by finalizers: lab.maborak.com/hold"}}
{"kind":"Service","name":"held-svc","health":{"status":"Progressing","message":"Pending deletion; blocked by finalizers: lab.maborak.com/hold"}}
### custom Lua deletionMessage, 2026-11-07T19:09:11Z
### 357
{"kind":"ConfigMap","name":"held","health":{"status":"Progressing","message":"Pending deletion"}}
{"kind":"Service","name":"held-svc","health":{"status":"Progressing","message":"Pending deletion"}}
### 361
{"kind":"ConfigMap","name":"held","health":{"status":"Degraded","message":"lab.maborak.com/hold: lab finalizer; removing it by hand skips the cleanup step."}}
{"kind":"Service","name":"held-svc","health":{"status":"Progressing","message":"Pending deletion; blocked by finalizers: lab.maborak.com/hold"}}

Argo CD 3.6.1 application maborak-held: App health Progressing, Caused by Service:maborak-held/maborak-held-svc; tree with ConfigMap maborak-held-config and Service maborak-held-svc, both progressing.

The health panel names the resource that caused it. Captured on our 3.6.1 instance on 7 November 2026, 23:59 UTC.

Argo CD 3.6.1 resource panel for ConfigMap maborak-held-config: Status Synced, Health Progressing, Health details: maborak.com/hold: Maborak lab finalizer. The Maborak cleanup job removes it after archiving the config; do not remove it by hand.

Our custom deletionMessage from a Lua health check. Captured on our 3.6.1 instance on 7 November 2026, 23:59 UTC.

Source hydrator

The hydrator renders manifests from a “dry” branch and commits them to an environment branch. In the application header you see the hydration state with both commits, and the new hydrator_status label on argocd_app_info reports the same state to Prometheus (hydrator_status="Hydrated" for this application).

Raw output: argocd_app_info labels, 3.5.7 and 3.6.1 
## C28 argocd_app_info hydrator_status, 2026-11-07T19:34:35Z
### 361 hydrated app
argocd_app_info{autosync_enabled="false",dest_namespace="hydrated",dest_server="https://kubernetes.default.svc",health_status="Missing",hydrator_status="Hydrated",name="hydrated",namespace="argocd",operation="",phase="",project="default",repo="http://argolab-gitea:3000/lab/apps",sync_status="OutOfSync"} 1
argocd_app_info{autosync_enabled="true",dest_namespace="plain",dest_server="https://kubernetes.default.svc",health_status="Healthy",hydrator_status="",name="plain",namespace="argocd",operation="",phase="Succeeded",project="default",repo="http://argolab-gitea:3000/lab/apps",sync_status="Synced"} 1
### 357 plain app (label set)
argocd_app_info{autosync_enabled="true",dest_namespace="plain",dest_server="https://kubernetes.default.svc",health_status="Healthy",name="plain",namespace="argocd",operation="",project="default",repo="http://argolab-gitea:3000/lab/apps",sync_status="Synced"} 1

Argo CD 3.6.1 application maborak-hydrated: App health Healthy, Source hydrator Hydrated from main (cacaca8) to environments/maborak-prod (0d396ee), Sync status Synced to environments/maborak-prod; tree with maborak-web-config, Service and Deployment maborak-web.

A hydrated application, writing to environments/maborak-prod. Captured on our 3.6.1 instance on 7 November 2026, 23:59 UTC.

“20–30% less rendering work”

The post credits the React Compiler with “roughly 20–30% less rendering work” when filtering and searching large installations. We measured page script time in Chrome while toggling a health filter on and off, and while typing a search and clearing it, on both versions with about 1,000 applications in list view, 20 per page. The filter matched 1,003 applications on 3.5.7 and 1,005 on 3.6.1.

The first, pre-registered run was noisy. Background reconciliation of the 1,000 applications kept each UI busy with its own cluster’s updates, at different moments on each version. Its unpaired comparison favoured 3.6.1 for filtering (median 75.2 ms against 50.1 ms, Mann-Whitney p=0.008), but the 95% intervals of both ratios included no change. We reran with periodic reconciliation off on both clusters, three fresh browser sessions per version, and an idle cycle subtracted from each session (idle cost about 0.3 ms):

Action 3.5.7, three sessions 3.6.1, three sessions Ratio of medians
Toggle a health filter on and off 48.3, 49.0, 49.7 ms 34.6, 28.9, 31.2 ms 0.64
Type a search and clear it 47.4, 45.7, 48.4 ms 45.6, 51.6, 49.3 ms 1.04

Filtering got about a third cheaper, and every 3.6.1 session beat every 3.5.7 session. Search did not change in the quiet run. Three sessions per version give a direction rather than a precise size, and the comparison is “3.6.1 UI against 3.5.7 UI”: the React Compiler is not the only thing that changed between them.

Analysis output: rendering, pre-registered run and quiet rerun 
filter ScriptDuration_ms        n=20/20 median 3.5.7=  75.20 3.6.1=  50.12 ratio=0.666 CI95=[0.291,1.372] change=-33.4% MWU p=0.0077
filter TaskDuration_ms          n=20/20 median 3.5.7= 111.18 3.6.1=  78.08 ratio=0.702 CI95=[0.408,1.250] change=-29.8% MWU p=0.0098
filter LayoutDuration_ms        n=20/20 median 3.5.7=   2.96 3.6.1=   1.82 ratio=0.615 CI95=[0.432,0.919] change=-38.5% MWU p=0.00012
filter RecalcStyleDuration_ms   n=20/20 median 3.5.7=   2.93 3.6.1=   1.40 ratio=0.479 CI95=[0.398,0.617] change=-52.1% MWU p=9.1e-08
search ScriptDuration_ms        n=20/20 median 3.5.7=  49.45 3.6.1=  43.27 ratio=0.875 CI95=[0.461,1.154] change=-12.5% MWU p=0.036
search TaskDuration_ms          n=20/20 median 3.5.7= 111.06 3.6.1=  90.95 ratio=0.819 CI95=[0.626,1.120] change=-18.1% MWU p=0.044
search LayoutDuration_ms        n=20/20 median 3.5.7=   9.23 3.6.1=   8.84 ratio=0.957 CI95=[0.875,1.024] change=-4.3% MWU p=0.35
search RecalcStyleDuration_ms   n=20/20 median 3.5.7=   8.43 3.6.1=   7.21 ratio=0.856 CI95=[0.829,0.928] change=-14.4% MWU p=7.5e-06
filter ScriptDuration_ms  pairs=20 median paired ratio=0.637 CI95=[0.594,0.715] 3.6.1 lower in 18/20 pairs, Wilcoxon p=3.6e-05
filter TaskDuration_ms    pairs=20 median paired ratio=0.717 CI95=[0.657,0.807] 3.6.1 lower in 17/20 pairs, Wilcoxon p=6.3e-05
search ScriptDuration_ms  pairs=20 median paired ratio=0.854 CI95=[0.708,0.932] 3.6.1 lower in 15/20 pairs, Wilcoxon p=0.001
search TaskDuration_ms    pairs=20 median paired ratio=0.787 CI95=[0.740,0.970] 3.6.1 lower in 15/20 pairs, Wilcoxon p=0.012
filter 357 idle-corrected session medians: [48.3, 49.0, 49.7]  (idle medians: [0.3, 0.3, 0.3])
filter 361 idle-corrected session medians: [34.6, 28.9, 31.2]  (idle medians: [0.3, 0.4, 0.3])
filter ratio of medians of session medians 3.6.1/3.5.7 = 0.636; every 3.6.1 session below every 3.5.7 session: True
search 357 idle-corrected session medians: [47.4, 45.7, 48.4]  (idle medians: [0.3, 0.3, 0.3])
search 361 idle-corrected session medians: [45.6, 51.6, 49.3]  (idle medians: [0.3, 0.4, 0.3])
search ratio of medians of session medians 3.6.1/3.5.7 = 1.040; every 3.6.1 session below every 3.5.7 session: False
Per-cycle data: quiet rerun, script and task time per cycle (216 cycles) 
version,session,action,cycle,ScriptDuration_ms,TaskDuration_ms
357,1,filter,1,51.91,84.04
357,1,search,1,45.70,106.85
357,1,idle,1,0.27,8.02
357,1,filter,2,53.78,92.40
357,1,search,2,46.69,113.96
357,1,idle,2,0.21,7.75
357,1,filter,3,43.93,72.69
357,1,search,3,45.75,113.34
357,1,idle,3,0.39,10.44
357,1,filter,4,48.66,80.52
357,1,search,4,54.78,120.93
357,1,idle,4,0.23,7.60
357,1,filter,5,48.44,92.59
357,1,search,5,48.08,110.51
357,1,idle,5,0.43,9.01
357,1,filter,6,46.75,88.70
357,1,search,6,50.55,122.16
357,1,idle,6,0.27,10.70
357,1,filter,7,53.88,96.03
357,1,search,7,45.35,117.99
357,1,idle,7,0.37,8.85
357,1,filter,8,44.89,83.47
357,1,search,8,50.15,114.29
357,1,idle,8,0.31,9.88
357,1,filter,9,49.39,94.24
357,1,search,9,55.19,125.52
357,1,idle,9,0.29,7.97
357,1,filter,10,54.09,100.83
357,1,search,10,47.37,112.60
357,1,idle,10,0.26,6.87
357,1,filter,11,47.47,83.08
357,1,search,11,45.38,113.37
357,1,idle,11,0.27,7.73
357,1,filter,12,51.88,86.73
357,1,search,12,43.31,102.92
357,1,idle,12,0.24,8.66
361,1,filter,1,29.13,51.73
361,1,search,1,46.96,114.23
361,1,idle,1,0.44,2.06
361,1,filter,2,29.68,62.33
361,1,search,2,51.96,128.61
361,1,idle,2,0.38,2.26
361,1,filter,3,35.28,70.31
361,1,search,3,41.06,109.79
361,1,idle,3,0.33,1.81
361,1,filter,4,28.16,60.90
361,1,search,4,43.41,107.12
361,1,idle,4,0.35,1.98
361,1,filter,5,36.60,90.42
361,1,search,5,57.15,122.17
361,1,idle,5,0.34,2.03
361,1,filter,6,40.12,85.17
361,1,search,6,49.79,113.64
361,1,idle,6,0.32,3.74
361,1,filter,7,35.35,79.25
361,1,search,7,43.75,110.10
361,1,idle,7,0.30,2.99
361,1,filter,8,25.36,72.59
361,1,search,8,55.61,122.64
361,1,idle,8,0.40,2.09
361,1,filter,9,28.48,62.91
361,1,search,9,45.41,104.37
361,1,idle,9,0.37,2.16
361,1,filter,10,35.15,84.89
361,1,search,10,37.95,96.65
361,1,idle,10,0.35,1.96
361,1,filter,11,26.44,71.69
361,1,search,11,52.81,117.57
361,1,idle,11,0.38,1.94
361,1,filter,12,34.69,79.00
361,1,search,12,46.44,104.40
361,1,idle,12,0.37,2.33
361,2,filter,1,35.15,71.80
361,2,search,1,50.09,118.94
361,2,idle,1,0.38,2.13
361,2,filter,2,30.82,71.07
361,2,search,2,54.18,127.57
361,2,idle,2,0.38,2.03
361,2,filter,3,30.35,60.60
361,2,search,3,53.56,129.72
361,2,idle,3,0.39,1.92
361,2,filter,4,27.58,57.26
361,2,search,4,50.43,119.87
361,2,idle,4,0.34,1.74
361,2,filter,5,28.82,83.48
361,2,search,5,56.17,118.88
361,2,idle,5,0.40,2.13
361,2,filter,6,38.42,78.55
361,2,search,6,46.39,107.51
361,2,idle,6,0.48,5.54
361,2,filter,7,30.92,71.84
361,2,search,7,49.69,120.15
361,2,idle,7,0.44,2.33
361,2,filter,8,27.06,75.48
361,2,search,8,58.34,123.58
361,2,idle,8,0.34,1.93
361,2,filter,9,36.80,80.83
361,2,search,9,58.24,124.71
361,2,idle,9,0.34,2.27
361,2,filter,10,28.22,75.62
361,2,search,10,41.73,111.81
361,2,idle,10,0.33,1.97
361,2,filter,11,27.30,80.45
361,2,search,11,55.38,124.53
361,2,idle,11,0.42,2.54
361,2,filter,12,29.68,67.05
361,2,search,12,43.59,101.39
361,2,idle,12,0.34,2.20
357,2,filter,1,53.84,96.33
357,2,search,1,54.31,117.47
357,2,idle,1,0.28,9.02
357,2,filter,2,49.51,89.10
357,2,search,2,57.02,127.17
357,2,idle,2,0.33,7.96
357,2,filter,3,47.88,79.84
357,2,search,3,47.38,114.86
357,2,idle,3,0.23,8.18
357,2,filter,4,48.55,81.65
357,2,search,4,46.41,115.97
357,2,idle,4,0.44,10.03
357,2,filter,5,48.03,86.77
357,2,search,5,46.26,110.12
357,2,idle,5,0.22,8.29
357,2,filter,6,53.88,100.32
357,2,search,6,45.48,104.84
357,2,idle,6,0.28,11.19
357,2,filter,7,49.97,91.83
357,2,search,7,43.36,111.07
357,2,idle,7,0.28,8.13
357,2,filter,8,55.39,97.17
357,2,search,8,45.99,111.39
357,2,idle,8,0.22,8.23
357,2,filter,9,55.29,99.35
357,2,search,9,39.47,97.60
357,2,idle,9,0.28,7.65
357,2,filter,10,55.26,102.26
357,2,search,10,54.06,116.55
357,2,idle,10,0.28,7.48
357,2,filter,11,44.07,80.83
357,2,search,11,45.96,114.48
357,2,idle,11,0.23,7.20
357,2,filter,12,46.26,82.78
357,2,search,12,41.23,105.13
357,2,idle,12,0.27,7.99
357,3,filter,1,51.91,87.82
357,3,search,1,48.69,108.55
357,3,idle,1,0.26,8.25
357,3,filter,2,56.71,102.80
357,3,search,2,58.70,130.70
357,3,idle,2,0.29,8.93
357,3,filter,3,50.59,83.35
357,3,search,3,48.99,117.00
357,3,idle,3,0.29,8.98
357,3,filter,4,59.93,105.04
357,3,search,4,56.05,127.25
357,3,idle,4,0.26,7.21
357,3,filter,5,51.84,85.18
357,3,search,5,48.88,110.84
357,3,idle,5,0.27,9.21
357,3,filter,6,46.71,90.92
357,3,search,6,47.90,105.72
357,3,idle,6,0.22,9.44
357,3,filter,7,54.76,99.46
357,3,search,7,56.78,127.95
357,3,idle,7,0.40,8.91
357,3,filter,8,48.56,85.90
357,3,search,8,43.89,116.65
357,3,idle,8,0.22,6.91
357,3,filter,9,49.30,88.89
357,3,search,9,41.87,104.57
357,3,idle,9,0.26,7.75
357,3,filter,10,49.32,95.54
357,3,search,10,46.37,109.57
357,3,idle,10,0.22,5.95
357,3,filter,11,48.40,96.96
357,3,search,11,48.53,114.73
357,3,idle,11,0.26,9.10
357,3,filter,12,55.88,94.66
357,3,search,12,50.67,125.25
357,3,idle,12,0.30,10.38
361,3,filter,1,35.90,72.85
361,3,search,1,58.12,133.73
361,3,idle,1,0.37,1.93
361,3,filter,2,34.82,68.28
361,3,search,2,43.18,111.67
361,3,idle,2,0.38,2.09
361,3,filter,3,32.63,70.39
361,3,search,3,47.75,117.28
361,3,idle,3,0.55,2.25
361,3,filter,4,35.46,74.11
361,3,search,4,41.26,112.04
361,3,idle,4,0.38,1.79
361,3,filter,5,28.13,58.51
361,3,search,5,50.09,117.02
361,3,idle,5,0.35,2.12
361,3,filter,6,31.73,92.78
361,3,search,6,53.91,116.54
361,3,idle,6,0.33,4.81
361,3,filter,7,31.31,71.90
361,3,search,7,54.95,117.66
361,3,idle,7,0.35,1.85
361,3,filter,8,28.52,70.46
361,3,search,8,49.29,113.60
361,3,idle,8,0.31,1.77
361,3,filter,9,25.79,86.27
361,3,search,9,58.54,128.84
361,3,idle,9,0.28,1.81
361,3,filter,10,37.55,82.55
361,3,search,10,45.14,105.71
361,3,idle,10,0.57,2.67
361,3,filter,11,33.24,80.17
361,3,search,11,42.41,105.64
361,3,idle,11,0.33,2.23
361,3,filter,12,26.61,79.70
361,3,search,12,55.27,119.34
361,3,idle,12,0.48,2.27
Per-cycle data: first, pre-registered run (88 cycles) 
version,action,cycle,ScriptDuration_ms,TaskDuration_ms,LayoutDuration_ms,RecalcStyleDuration_ms
357,filter,1,49.36,82.90,2.77,3.39
361,filter,1,33.25,69.11,2.07,2.50
361,filter,2,38.43,57.67,1.75,1.43
357,filter,2,47.35,72.68,2.12,2.13
357,filter,3,42.95,59.60,2.08,1.83
361,filter,3,42.37,63.04,1.80,1.37
361,filter,4,63.22,100.33,2.25,1.62
357,filter,4,98.21,139.69,3.59,3.16
357,filter,5,110.33,142.30,3.51,3.07
361,filter,5,84.47,131.87,2.69,1.84
361,filter,6,87.15,127.05,2.54,1.71
357,filter,6,104.14,144.60,3.35,3.46
357,filter,7,97.84,136.24,3.72,3.18
361,filter,7,71.12,101.03,2.20,1.58
361,filter,8,74.37,105.29,2.00,1.51
357,filter,8,121.16,158.34,3.91,3.70
357,filter,9,120.78,163.89,3.83,3.87
361,filter,9,84.84,126.56,2.65,1.91
361,filter,10,80.23,113.78,2.49,1.67
357,filter,10,129.85,172.00,3.74,3.42
357,filter,11,127.54,168.91,4.19,3.53
361,filter,11,85.79,122.09,2.36,1.66
361,filter,12,39.98,65.25,1.60,1.29
357,filter,12,129.54,174.41,3.88,3.46
357,filter,13,135.96,182.07,3.92,3.69
361,filter,13,29.28,51.25,1.39,1.23
361,filter,14,28.84,58.87,1.27,1.29
357,filter,14,47.38,82.36,2.21,2.53
357,filter,15,51.88,78.41,2.57,2.52
361,filter,15,29.17,48.26,1.65,1.29
361,filter,16,27.18,53.65,1.42,1.23
357,filter,16,41.15,63.74,1.99,2.15
357,filter,17,49.75,74.85,2.29,2.43
361,filter,17,28.79,48.83,1.32,1.28
361,filter,18,30.90,60.01,1.51,1.38
357,filter,18,49.08,86.12,2.12,2.24
357,filter,19,45.89,73.25,2.28,2.51
361,filter,19,26.14,40.58,1.11,1.12
361,filter,20,25.82,39.94,1.13,1.11
357,filter,20,52.56,79.66,2.25,2.80
357,filter,21,49.57,81.77,2.04,2.33
361,filter,21,57.87,90.91,1.84,1.54
361,filter,22,68.04,102.01,1.95,1.43
357,filter,22,50.45,80.60,2.26,2.32
357,search,1,46.11,110.31,14.59,13.25
361,search,1,90.20,167.51,15.80,13.71
361,search,2,63.27,106.03,8.49,7.19
357,search,2,97.86,152.45,9.69,8.40
357,search,3,109.51,167.31,9.46,8.80
361,search,3,77.00,130.91,8.91,7.37
361,search,4,63.20,106.22,8.18,7.24
357,search,4,101.68,158.03,9.40,8.52
357,search,5,88.68,141.13,9.13,8.52
361,search,5,91.01,140.09,8.45,7.10
361,search,6,69.00,111.83,8.07,7.06
357,search,6,96.73,150.28,9.35,8.52
357,search,7,123.88,178.10,9.16,8.69
361,search,7,60.94,106.79,7.88,7.01
361,search,8,73.44,115.97,8.26,7.18
357,search,8,87.66,140.37,9.24,8.42
357,search,9,88.03,139.92,9.13,8.44
361,search,9,40.70,82.32,7.40,6.66
361,search,10,32.74,67.63,7.20,6.62
357,search,10,87.29,140.73,9.23,8.64
357,search,11,87.18,139.02,9.12,8.40
361,search,11,36.54,88.27,9.22,7.88
361,search,12,43.31,87.91,11.01,8.38
357,search,12,41.96,92.73,7.80,8.04
357,search,13,41.29,85.86,7.80,8.21
361,search,13,49.50,114.24,10.67,7.96
361,search,14,48.13,93.42,9.37,7.92
357,search,14,51.04,114.72,9.94,9.16
357,search,15,42.83,84.95,10.11,8.78
361,search,15,43.23,105.22,10.74,8.17
361,search,16,42.59,84.94,10.30,7.71
357,search,16,47.85,107.39,9.55,9.65
357,search,17,42.71,79.59,9.46,7.42
361,search,17,43.55,118.81,10.84,8.31
361,search,18,36.40,72.24,7.48,6.86
357,search,18,42.84,98.27,9.29,8.04
357,search,19,41.00,81.08,8.35,7.60
361,search,19,37.77,88.49,9.45,7.19
361,search,20,36.57,72.70,7.63,6.93
357,search,20,42.91,94.23,9.12,8.21
357,search,21,41.38,77.80,9.66,7.41
361,search,21,37.25,87.19,9.32,7.52
361,search,22,36.68,71.60,8.76,6.94
357,search,22,42.89,95.40,9.20,8.32

Memory: compression halves the cache, if you turn it on

The post’s headline memory claim, 3.27 GB down to under 1 GB, comes from a benchmark in PR #28600, which the post mislinks as #29570. [5] The feature is off by default: nothing changes until you set resource.manifest.compression.enabled: "true" in argocd-cm.

We tested it on a third cluster running only 3.6.1, with one application managing 5,000 ConfigMaps of about 2.3 KB each, and restarted the controller five times with compression off and five times on. Our registered measurement, the controller’s working set and Go heap sampled over a minute, found no difference: median working set 858 MiB off and 936 MiB on (Mann-Whitney p=0.69). Those readings were dominated by memory from reconciling 5,000 resources that the garbage collector had not freed yet, not by the cache.

A follow-up, chosen after that result, turned off periodic reconciliation and forced a garbage collection before each reading (/debug/pprof/heap?gc=1). It measures the memory the controller keeps, which is where the cache lives:

Compression Live heap after garbage collection, five controller starts each
off 90.6–90.7 MiB
on 45.1–45.3 MiB

Compression halved the controller’s live heap at this load (ratio 0.50, Mann-Whitney p=0.008, the smallest p possible with five starts per arm). That is 45 MiB for 5,000 resources, about 9.5 KB per resource. It is not the same quantity as the post’s 70% of total controller memory, and we did not measure the CPU cost. On a controller whose memory is dominated by reconciliation, as in our first run, the saving may not show in the container’s memory graph.

Analysis output: memory, registered run and live-heap follow-up 
off start 1: working_set=1185 MiB heap_inuse=912 MiB cpu_s_over_window=0.10 samples=7
off start 2: working_set=1079 MiB heap_inuse=735 MiB cpu_s_over_window=4.27 samples=7
off start 3: working_set=858 MiB heap_inuse=223 MiB cpu_s_over_window=0.11 samples=7
off start 4: working_set=828 MiB heap_inuse=228 MiB cpu_s_over_window=0.09 samples=7
off start 5: working_set=690 MiB heap_inuse=229 MiB cpu_s_over_window=4.42 samples=7
on start 1: working_set=935 MiB heap_inuse=174 MiB cpu_s_over_window=0.17 samples=7
on start 2: working_set=622 MiB heap_inuse=173 MiB cpu_s_over_window=0.09 samples=7
on start 3: working_set=1164 MiB heap_inuse=762 MiB cpu_s_over_window=0.09 samples=7
on start 4: working_set=525 MiB heap_inuse=171 MiB cpu_s_over_window=0.08 samples=7
on start 5: working_set=1027 MiB heap_inuse=872 MiB cpu_s_over_window=4.54 samples=7
excluded: none
working_set: n=5/5 median off=858.4 on=935.5 ratio=1.090 CI95=[0.525,1.406] change=+9.0% MWU p=0.69
heap_inuse: n=5/5 median off=228.9 on=174.2 ratio=0.761 CI95=[0.191,3.809] change=-23.9% MWU p=0.42
cpu_seconds: n=5/5 median off=0.1 on=0.1 ratio=0.818 CI95=[0.020,41.273] change=-18.2% MWU p=0.6
working_set: n=4/5 median off=843.2 on=935.5 ratio=1.109 CI95=[0.576,1.406] change=+10.9% MWU p=1
heap_inuse: n=4/5 median off=228.6 on=174.2 ratio=0.762 CI95=[0.237,3.814] change=-23.8% MWU p=0.73
cpu_seconds: n=4/5 median off=2.2 on=0.1 ratio=0.041 CI95=[0.020,2.073] change=-95.9% MWU p=0.62
off start 1: live heap 90.6 MiB (samples 5, cache 5406)
off start 2: live heap 90.6 MiB (samples 5, cache 5406)
off start 3: live heap 90.7 MiB (samples 5, cache 5406)
off start 4: live heap 90.6 MiB (samples 5, cache 5406)
off start 5: live heap 90.6 MiB (samples 5, cache 5406)
on start 1: live heap 45.3 MiB (samples 5, cache 5406)
on start 2: live heap 45.2 MiB (samples 5, cache 5406)
on start 3: live heap 45.3 MiB (samples 5, cache 5406)
on start 4: live heap 45.3 MiB (samples 5, cache 5406)
on start 5: live heap 45.1 MiB (samples 5, cache 5406)
excluded: none
live heap after GC: off median 90.6 MiB (range 90.6-90.7), on median 45.3 MiB (range 45.1-45.3); ratio 0.500 CI95 [0.498,0.500]; diff 45.3 MiB; MWU p=0.00794
Per-sample data: live heap after forced garbage collection (50 samples) 
arm,start,sample,utc,heap_alloc_after_gc_bytes,heap_inuse_after_gc_bytes,cache_objects,app_sync
off,1,1,2026-11-07T21:23:01Z,93926904,122748928,5406,Synced
off,1,2,2026-11-07T21:32:38Z,95014616,123551744,5406,Synced
off,1,3,2026-11-07T21:32:48Z,95008800,123494400,5406,Synced
off,1,4,2026-11-07T21:32:59Z,95016376,123543552,5406,Synced
off,1,5,2026-11-07T21:33:09Z,95056064,123535360,5406,Synced
on,1,1,2026-11-07T21:39:43Z,48687304,67149824,5406,Synced
on,1,2,2026-11-07T21:39:53Z,47504960,65888256,5406,Synced
on,1,3,2026-11-07T21:40:04Z,47506232,65871872,5406,Synced
on,1,4,2026-11-07T21:40:14Z,47502232,65830912,5406,Synced
on,1,5,2026-11-07T21:40:25Z,47541488,65822720,5406,Synced
off,2,1,2026-11-07T21:46:59Z,98497160,126861312,5406,Synced
off,2,2,2026-11-07T21:47:09Z,95004064,123215872,5406,Synced
off,2,3,2026-11-07T21:47:20Z,95008264,123207680,5406,Synced
off,2,4,2026-11-07T21:47:30Z,95045840,123232256,5406,Synced
off,2,5,2026-11-07T21:47:40Z,95038848,123125760,5406,Synced
on,2,1,2026-11-07T21:54:16Z,46345448,64667648,5406,Synced
on,2,2,2026-11-07T21:54:26Z,47415840,65650688,5406,Synced
on,2,3,2026-11-07T21:54:36Z,47415912,65634304,5406,Synced
on,2,4,2026-11-07T21:54:47Z,47415360,65626112,5406,Synced
on,2,5,2026-11-07T21:54:57Z,47455472,65617920,5406,Synced
off,3,1,2026-11-07T22:01:33Z,97436448,127451136,5406,Synced
off,3,2,2026-11-07T22:01:43Z,95036648,124682240,5406,Synced
off,3,3,2026-11-07T22:01:53Z,95034360,124674048,5406,Synced
off,3,4,2026-11-07T22:02:04Z,95077816,124682240,5406,Synced
off,3,5,2026-11-07T22:02:14Z,95111056,124690432,5406,Synced
on,3,1,2026-11-07T22:08:48Z,46401304,65175552,5406,Synced
on,3,2,2026-11-07T22:08:58Z,47528640,66281472,5406,Synced
on,3,3,2026-11-07T22:09:08Z,47486688,66232320,5406,Synced
on,3,4,2026-11-07T22:09:19Z,47494888,66273280,5406,Synced
on,3,5,2026-11-07T22:09:29Z,47523528,66281472,5406,Synced
off,4,1,2026-11-07T22:16:04Z,93917056,123666432,5406,Synced
off,4,2,2026-11-07T22:16:15Z,95006248,124657664,5406,Synced
off,4,3,2026-11-07T22:16:25Z,95001176,124665856,5406,Synced
off,4,4,2026-11-07T22:16:35Z,95008632,124657664,5406,Synced
off,4,5,2026-11-07T22:16:46Z,95044808,124674048,5406,Synced
on,4,1,2026-11-07T22:23:21Z,49860840,67936256,5406,Synced
on,4,2,2026-11-07T22:23:31Z,47497048,65413120,5406,Synced
on,4,3,2026-11-07T22:23:42Z,47539024,65404928,5406,Synced
on,4,4,2026-11-07T22:23:52Z,47493384,65347584,5406,Synced
on,4,5,2026-11-07T22:24:03Z,47572344,65355776,5406,Synced
off,5,1,2026-11-07T22:30:38Z,93927288,123625472,5406,Synced
off,5,2,2026-11-07T22:30:48Z,95013456,124370944,5406,Synced
off,5,3,2026-11-07T22:30:59Z,95010912,124362752,5406,Synced
off,5,4,2026-11-07T22:31:09Z,95005544,124362752,5406,Synced
off,5,5,2026-11-07T22:31:19Z,95053656,124379136,5406,Synced
on,5,1,2026-11-07T22:37:54Z,46200208,64110592,5406,Synced
on,5,2,2026-11-07T22:38:04Z,47326936,65134592,5406,Synced
on,5,3,2026-11-07T22:38:14Z,47307000,65085440,5406,Synced
on,5,4,2026-11-07T22:38:25Z,47306768,65085440,5406,Synced
on,5,5,2026-11-07T22:38:36Z,47342576,65085440,5406,Synced
Per-sample data: registered run, working set and heap in use (70 samples) 
arm,start,sample,utc,working_set_bytes,heap_inuse_bytes,cpu_seconds_total,cache_objects,app_sync
off,1,1,2026-11-07T20:18:42Z,1.24286976e+09,9.51369728e+08,12.39,5397,Synced
off,1,2,2026-11-07T20:18:52Z,1.24286976e+09,9.53720832e+08,12.41,5397,Synced
off,1,3,2026-11-07T20:19:02Z,1.24286976e+09,9.53950208e+08,12.42,5397,Synced
off,1,4,2026-11-07T20:19:12Z,1.24286976e+09,9.566208e+08,12.43,5397,Synced
off,1,5,2026-11-07T20:19:23Z,1.243000832e+09,9.56850176e+08,12.45,5397,Synced
off,1,6,2026-11-07T20:19:33Z,1.243000832e+09,9.58447616e+08,12.47,5397,Synced
off,1,7,2026-11-07T20:19:43Z,1.243131904e+09,9.59848448e+08,12.49,5397,Synced
on,1,1,2026-11-07T20:24:12Z,9.8076672e+08,1.81395456e+08,8.49,5398,Synced
on,1,2,2026-11-07T20:24:23Z,9.80905984e+08,1.82673408e+08,8.52,5398,Synced
on,1,3,2026-11-07T20:24:33Z,9.8093056e+08,1.85106432e+08,8.53,5398,Synced
on,1,4,2026-11-07T20:24:43Z,9.80975616e+08,1.86458112e+08,8.54,5398,Synced
on,1,5,2026-11-07T20:24:53Z,9.80979712e+08,1.87670528e+08,8.56,5398,Synced
on,1,6,2026-11-07T20:25:04Z,9.80979712e+08,1.7547264e+08,8.65,5398,Synced
on,1,7,2026-11-07T20:25:14Z,2.292736e+08,1.76693248e+08,8.66,5398,Synced
off,2,1,2026-11-07T20:29:48Z,1.13139712e+09,2.36331008e+08,12.61,5399,Synced
off,2,2,2026-11-07T20:29:58Z,6.97073664e+08,2.36494848e+08,12.62,5399,Synced
off,2,3,2026-11-07T20:30:08Z,6.97073664e+08,5.50199296e+08,14.44,5399,Synced
off,2,4,2026-11-07T20:30:19Z,7.35473664e+08,7.702528e+08,16.83,5399,Synced
off,2,5,2026-11-07T20:30:29Z,1.255038976e+09,7.72972544e+08,16.85,5399,Synced
off,2,6,2026-11-07T20:30:39Z,1.255038976e+09,7.74356992e+08,16.86,5399,Synced
off,2,7,2026-11-07T20:30:50Z,1.255038976e+09,7.76716288e+08,16.88,5399,Synced
on,2,1,2026-11-07T20:36:10Z,6.52005376e+08,1.77012736e+08,8.85,5400,Synced
on,2,2,2026-11-07T20:36:21Z,6.52005376e+08,1.78274304e+08,8.86,5400,Synced
on,2,3,2026-11-07T20:36:31Z,6.52038144e+08,1.80641792e+08,8.88,5400,Synced
on,2,4,2026-11-07T20:36:42Z,6.52038144e+08,1.81854208e+08,8.89,5400,Synced
on,2,5,2026-11-07T20:36:52Z,6.52038144e+08,1.82099968e+08,8.91,5400,Synced
on,2,6,2026-11-07T20:37:02Z,6.52161024e+08,1.84672256e+08,8.92,5400,Synced
on,2,7,2026-11-07T20:37:12Z,6.52161024e+08,1.87219968e+08,8.94,5400,Synced
off,3,1,2026-11-07T20:41:57Z,1.193869312e+09,2.29187584e+08,13.03,5401,Synced
off,3,2,2026-11-07T20:42:07Z,9.00059136e+08,2.31555072e+08,13.04,5401,Synced
off,3,3,2026-11-07T20:42:18Z,9.00059136e+08,2.32890368e+08,13.06,5401,Synced
off,3,4,2026-11-07T20:42:28Z,9.00059136e+08,2.34266624e+08,13.08,5401,Synced
off,3,5,2026-11-07T20:42:38Z,9.00059136e+08,2.36609536e+08,13.1,5401,Synced
off,3,6,2026-11-07T20:42:48Z,9.00059136e+08,2.3916544e+08,13.12,5401,Synced
off,3,7,2026-11-07T20:42:59Z,9.00059136e+08,2.39403008e+08,13.14,5401,Synced
on,3,1,2026-11-07T20:48:10Z,6.80771584e+08,7.94976256e+08,18.83,5402,Synced
on,3,2,2026-11-07T20:48:21Z,6.80771584e+08,7.96532736e+08,18.85,5402,Synced
on,3,3,2026-11-07T20:48:31Z,1.218936832e+09,7.9896576e+08,18.87,5402,Synced
on,3,4,2026-11-07T20:48:41Z,1.221025792e+09,7.99178752e+08,18.88,5402,Synced
on,3,5,2026-11-07T20:48:52Z,1.221025792e+09,8.0203776e+08,18.89,5402,Synced
on,3,6,2026-11-07T20:49:02Z,1.221025792e+09,8.034304e+08,18.91,5402,Synced
on,3,7,2026-11-07T20:49:12Z,1.221025792e+09,8.03766272e+08,18.92,5402,Synced
off,4,1,2026-11-07T20:54:07Z,8.68265984e+08,2.34397696e+08,8.58,5403,Synced
off,4,2,2026-11-07T20:54:17Z,8.68265984e+08,2.3560192e+08,8.6,5403,Synced
off,4,3,2026-11-07T20:54:28Z,8.68265984e+08,2.37985792e+08,8.62,5403,Synced
off,4,4,2026-11-07T20:54:38Z,8.68265984e+08,2.3937024e+08,8.63,5403,Synced
off,4,5,2026-11-07T20:54:48Z,8.68278272e+08,2.4174592e+08,8.65,5403,Synced
off,4,6,2026-11-07T20:54:59Z,8.68282368e+08,2.41860608e+08,8.66,5403,Synced
off,4,7,2026-11-07T20:55:09Z,8.68282368e+08,2.4305664e+08,8.67,5403,Synced
on,4,1,2026-11-07T21:00:10Z,5.50518784e+08,1.74792704e+08,13.58,5404,Synced
on,4,2,2026-11-07T21:00:20Z,5.50518784e+08,1.77143808e+08,13.59,5404,Synced
on,4,3,2026-11-07T21:00:31Z,5.50518784e+08,1.7948672e+08,13.62,5404,Synced
on,4,4,2026-11-07T21:00:41Z,5.50518784e+08,1.79658752e+08,13.63,5404,Synced
on,4,5,2026-11-07T21:00:52Z,5.50518784e+08,1.80994048e+08,13.64,5404,Synced
on,4,6,2026-11-07T21:01:02Z,5.50518784e+08,1.83468032e+08,13.65,5404,Synced
on,4,7,2026-11-07T21:01:12Z,5.50518784e+08,1.83713792e+08,13.66,5404,Synced
off,5,1,2026-11-07T21:05:35Z,1.148686336e+09,2.35012096e+08,12.5,5405,Synced
off,5,2,2026-11-07T21:05:45Z,7.23697664e+08,2.36249088e+08,12.51,5405,Synced
off,5,3,2026-11-07T21:05:56Z,7.23697664e+08,2.37600768e+08,12.53,5405,Synced
off,5,4,2026-11-07T21:06:06Z,7.23697664e+08,2.40001024e+08,12.54,5405,Synced
off,5,5,2026-11-07T21:06:16Z,7.23697664e+08,7.636992e+08,16.89,5405,Synced
off,5,6,2026-11-07T21:06:27Z,1.29941504e+09,7.66263296e+08,16.91,5405,Synced
off,5,7,2026-11-07T21:06:37Z,1.29941504e+09,7.67557632e+08,16.92,5405,Synced
on,5,1,2026-11-07T21:11:26Z,1.204215808e+09,1.77733632e+08,9.09,5406,Synced
on,5,2,2026-11-07T21:11:36Z,7.98937088e+08,1.80027392e+08,9.11,5406,Synced
on,5,3,2026-11-07T21:11:46Z,7.98941184e+08,3.84671744e+08,11.82,5406,Synced
on,5,4,2026-11-07T21:11:57Z,1.076989952e+09,9.1418624e+08,13.58,5406,Synced
on,5,5,2026-11-07T21:12:07Z,1.076989952e+09,9.15644416e+08,13.6,5406,Synced
on,5,6,2026-11-07T21:12:17Z,1.077796864e+09,9.18011904e+08,13.61,5406,Synced
on,5,7,2026-11-07T21:12:28Z,1.077796864e+09,9.18298624e+08,13.63,5406,Synced

The post’s other memory items (partial JSON unmarshalling, fewer application copies, counting applications per cluster in one pass) are merged in 3.6.0. We checked them in the source tree, not in the lab.

ApplicationSets

The OCI generator reads directories and files from an OCI artifact instead of a Git repository. We pushed maborak/platform:v1.0.0 with three directories to a local registry and excluded platform/maborak-legacy: the ApplicationSet created maborak-ingress and maborak-monitoring, both Synced. 3.5.7 rejects the manifest with unknown field "spec.generators[0].oci".

Argo CD 3.6.1 ApplicationSet maborak-platform: health Healthy, All applications have been generated successfully, 3 Info conditions; tree with generated applications maborak-ingress and maborak-monitoring, both healthy and synced.

An ApplicationSet fed by an OCI artifact. Captured on our 3.6.1 instance on 7 November 2026, 23:59 UTC. With tpl, a value read from a Git file, maborak team {{ .team }} verified, rendered as maborak team payments verified in the generated application’s annotation; 3.5.7 fails with function "tpl" not defined. Of the new workqueue flags, --wq-maxdelay defaults to 16m40s, the “roughly 16 minutes” in the post.

Raw output: `tpl` rendering and ApplicationSet controller flags 
## C17 tpl, 2026-11-07T19:11:57Z
### 357 apps
### 357 appset conditions
{"type":"ErrorOccurred","status":"True","reason":"RenderTemplateParamsError","message":"failed to parse template {{ tpl .greeting . }}: template: base:1: function \"tpl\" not defined"}
{"type":"ParametersGenerated","status":"False","reason":"ErrorOccurred","message":"failed to parse template {{ tpl .greeting . }}: template: base:1: function \"tpl\" not defined"}
{"type":"ResourcesUpToDate","status":"False","reason":"ErrorOccurred","message":"failed to parse template {{ tpl .greeting . }}: template: base:1: function \"tpl\" not defined"}
### 361 apps
tpl-team-a greeting=hello from team-a
tpl-team-b greeting=hello from team-b
### 361 appset conditions
{"type":"ErrorOccurred","status":"False","reason":"ApplicationSetUpToDate","message":"All applications have been generated successfully"}
{"type":"ParametersGenerated","status":"True","reason":"ParametersGenerated","message":"All parameters have been generated successfully"}
{"type":"ResourcesUpToDate","status":"True","reason":"ApplicationSetUpToDate","message":"All applications have been generated successfully"}
## C18 appset controller flags, 2026-11-07T19:10:41Z
### 357
(no wq-* or refresh-grace flags)
### 361
      --refresh-grace-period-seconds int          The minimum grace period before a progressive sync may start refreshing outdated applications (default 30)
      --wq-backoff-factor float                   Set Workqueue Per Item Rate Limiter Backoff Factor, default is 1.5 (default 1.5)
      --wq-basedelay duration                     Set Workqueue Per Item Rate Limiter Base Delay duration, default 1ms (default 1ms)
      --wq-bucket-qps float                       Set Workqueue Rate Limiter Bucket QPS, default set to MaxFloat64 which disables the bucket limiter (default 1.7976931348623157e+308)
      --wq-bucket-size int                        Set Workqueue Rate Limiter Bucket Size, default 500 (default 500)
      --wq-cooldown duration                      Set Workqueue Per Item Rate Limiter Cooldown duration, default 0 (per item rate limiter disabled)
      --wq-maxdelay duration                      Set Workqueue Per Item Rate Limiter Max Delay duration, default 16m40s (matches controller-runtime default) (default 16m40s)
Raw output: OCI generator, 3.5.7 and 3.6.1 
## C13 OCI generator, 2026-11-07T19:19:49Z
### 357 apply
Error from server (BadRequest): error when creating "manifests/appset-oci.yaml": ApplicationSet in version "v1alpha1" cannot be handled as a ApplicationSet: strict decoding error: unknown field "spec.generators[0].oci"
### 361 apply
applicationset.argoproj.io/oci-platform created
### 357 apps
### 361 apps
oci-alpha sync=Synced health=Healthy
oci-beta sync=Synced health=Healthy
[{"type":"ErrorOccurred","status":"False","message":"All applications have been generated successfully"},{"type":"ParametersGenerated","status":"True","message":"All parameters have been generated successfully"},{"type":"ResourcesUpToDate","status":"True","message":"All applications have been generated successfully"}]

Progressive sync now refreshes the applications itself before it moves. In three trials we pushed a change and refreshed only rolling-dev; on 3.6.1 the ApplicationSet controller refreshed the other two (its new argocd_appset_app_refresh_total counter rose by 2 each time), and all three were on the new revision within 45 seconds. On 3.5.7, one of the two was still on the old revision after 45 seconds in 2 of the 3 trials.

Raw output: progressive sync refresh mechanism, three trials per version 
357 trial 1: annotations_seen_on_unrefreshed_apps=[] app_refresh_total 0->0 at45s_revision_current: dev=true,prod=true,staging=true
357 trial 2: annotations_seen_on_unrefreshed_apps=[] app_refresh_total 0->0 at45s_revision_current: dev=true,prod=true,staging=false
357 trial 3: annotations_seen_on_unrefreshed_apps=[] app_refresh_total 0->0 at45s_revision_current: dev=true,prod=false,staging=true
361 trial 1: annotations_seen_on_unrefreshed_apps=[] app_refresh_total 2->4 at45s_revision_current: dev=true,prod=true,staging=true
361 trial 2: annotations_seen_on_unrefreshed_apps=[] app_refresh_total 5->7 at45s_revision_current: dev=true,prod=true,staging=true
361 trial 3: annotations_seen_on_unrefreshed_apps=[] app_refresh_total 8->10 at45s_revision_current: dev=true,prod=true,staging=true

The same change adds a wait. With a push followed by a refresh of all three applications, step 1 was deployed 0–1 seconds after the push on 3.5.7 and 30–31 seconds after it on 3.6.1, in 5 of 5 trials each (Wilson 95% interval 57–100%). The new --refresh-grace-period-seconds flag defaults to 30; setting it to 0 brought 3.6.1 back to 0–1 seconds. Later steps followed within 2 seconds in every arm. [9] The post presents the flag as a way to give webhooks time; it does not say the default adds about 30 seconds before the first step of a rollout. We measured one three-step ApplicationSet; a larger one may wait for longer reconciles.

Per-trial data: seconds from push to deploy per step (15 trials)
arm trial push_utc rev dev_s staging_s prod_s
v357 1 2026-11-07T21:16:47Z e66ed01 1 1 2
v357 2 2026-11-07T21:17:10Z c5e6bb8 0 1 2
v357 3 2026-11-07T21:17:32Z a0c9926 1 2 3
v357 4 2026-11-07T21:17:57Z a3f968f 1 2 3
v357 5 2026-11-07T21:18:22Z 9343ea4 1 2 3
v361-grace30 1 2026-11-07T21:18:47Z b8e2c57 31 32 33
v361-grace30 2 2026-11-07T21:19:41Z 4094976 30 31 32
v361-grace30 3 2026-11-07T21:20:35Z 7342c6e 30 31 32
v361-grace30 4 2026-11-07T21:21:29Z c50da64 30 31 32
v361-grace30 5 2026-11-07T21:22:23Z c5fff87 30 31 32
v361-grace0 1 2026-11-07T21:33:29Z 296e87b 1 1 2
v361-grace0 2 2026-11-07T21:33:52Z 7434a51 0 1 2
v361-grace0 3 2026-11-07T21:34:15Z 64d698b 0 1 2
v361-grace0 4 2026-11-07T21:34:38Z 9835a85 0 1 2
v361-grace0 5 2026-11-07T21:35:00Z 6cebcab 1 1 2

The rollout conditions fix a real ambiguity. A RollingSync with no steps reports Healthy and “ApplicationSet Rollout has completed” on 3.5.7, while its applications are never synced. On 3.6.1 the ApplicationSet is Degraded with reason ApplicationSetRolloutError and “No steps defined for rollout”.

Argo CD 3.5.7 ApplicationSet maborak-no-steps: health Healthy, ApplicationSet Rollout has completed, 4 Info; tree shows maborak-no-steps-dev and maborak-no-steps-prod both missing and out of sync.

3.5.7: “completed”, while both applications were never synced. Captured on our 3.5.7 instance on 7 November 2026, 23:59 UTC.

Argo CD 3.6.1 ApplicationSet maborak-no-steps: health Degraded, No steps defined for rollout, conditions 1 Info and 2 Errors; no applications generated.

3.6.1: the same ApplicationSet is Degraded and says why. Captured on our 3.6.1 instance on 7 November 2026, 23:59 UTC. The post’s names are Go constants: on the object, the new condition type is InvalidRolloutConfig and its reason is ApplicationSetInvalidRolloutConfig, which we triggered with maxUpdate: "lots". [12]

Raw output: rollout conditions, 3.5.7 and 3.6.1 
## C16 RollingSync with no steps, 2026-11-07T19:11:57Z
### 357
{"health":{"message":"All applications have been generated successfully","status":"Healthy"},"conditions":[{"type":"ErrorOccurred","status":"False","reason":"ApplicationSetUpToDate","message":"All applications have been generated successfully"},{"type":"ParametersGenerated","status":"True","reason":"ParametersGenerated","message":"Successfully generated parameters for all Applications"},{"type":"ResourcesUpToDate","status":"True","reason":"ApplicationSetUpToDate","message":"All applications have been generated successfully"},{"type":"RolloutProgressing","status":"False","reason":"ApplicationSetRolloutComplete","message":"ApplicationSet Rollout has completed"}]}
### 361
{"health":{"message":"No steps defined for rollout","status":"Degraded"},"conditions":[{"type":"ErrorOccurred","status":"True","reason":"ApplicationSetRolloutError","message":"No steps defined for rollout"},{"type":"ParametersGenerated","status":"True","reason":"ParametersGenerated","message":"All parameters have been generated successfully"},{"type":"ResourcesUpToDate","status":"False","reason":"ErrorOccurred","message":"No steps defined for rollout"}]}
## C16 invalid rollout configs, 2026-11-07T19:27:19Z
### 357 badmax
{"health":{"message":"All applications have been generated successfully","status":"Healthy"},"conditions":[{"type":"ErrorOccurred","status":"False","reason":"ApplicationSetUpToDate","message":"All applications have been generated successfully"},{"type":"ResourcesUpToDate","status":"True","reason":"ApplicationSetUpToDate","message":"All applications have been generated successfully"},{"type":"RolloutProgressing","status":"False","reason":"ApplicationSetRolloutComplete","message":"ApplicationSet Rollout has completed"}]}
### 357 unmatched
{"health":{"message":"All applications have been generated successfully","status":"Healthy"},"conditions":[{"type":"ErrorOccurred","status":"False","reason":"ApplicationSetUpToDate","message":"All applications have been generated successfully"},{"type":"ResourcesUpToDate","status":"True","reason":"ApplicationSetUpToDate","message":"All applications have been generated successfully"},{"type":"RolloutProgressing","status":"False","reason":"ApplicationSetRolloutComplete","message":"ApplicationSet Rollout has completed"}]}
### 361 badmax
{"health":{"message":"All applications have been generated successfully","status":"Healthy"},"conditions":[{"type":"ErrorOccurred","status":"False","reason":"ApplicationSetUpToDate","message":"All applications have been generated successfully"},{"type":"InvalidRolloutConfig","status":"True","reason":"ApplicationSetInvalidRolloutConfig","message":"Step 1 has an invalid maxUpdate value: lots"},{"type":"ResourcesUpToDate","status":"True","reason":"ApplicationSetUpToDate","message":"All applications have been generated successfully"},{"type":"RolloutProgressing","status":"False","reason":"ApplicationSetRolloutComplete","message":"ApplicationSet Rollout has completed"}]}
### 361 unmatched
{"health":{"message":"All applications have been generated successfully","status":"Healthy"},"conditions":[{"type":"ErrorOccurred","status":"False","reason":"ApplicationSetUpToDate","message":"All applications have been generated successfully"},{"type":"InvalidRolloutConfig","status":"False","reason":"ApplicationSetValidRolloutConfig","message":"Rolling Sync Configured correctly"},{"type":"ResourcesUpToDate","status":"True","reason":"ApplicationSetUpToDate","message":"All applications have been generated successfully"},{"type":"RolloutProgressing","status":"False","reason":"ApplicationSetRolloutComplete","message":"ApplicationSet Rollout has completed"}]}

All four new progressive sync metrics appeared on 3.6.1. argocd_appset_app_refresh_total shows up only once the controller has refreshed something itself, which never happened while we refreshed every application by hand.

Controller and CLI

  • Rollback permission. We tested the permission, not every restriction the post lists. By default, sync permission still allows a rollback. With server.rbac.rollback.enforce.enable: "true", a user without the new rollback action got PermissionDenied, and the same user with it succeeded. Rolling back an auto-sync application failed with FailedPrecondition. 3.5.7 rejects rollback as an RBAC action.
  • Label selectors. With resource.selectors set to !lab-noise for ConfigMaps, the 3.6.1 controller cache dropped from 3,452 to 453 objects, which accounts for our 3,000 labelled noise ConfigMaps. 3.5.7 ignored the key.
  • Custom file extensions. With disableExtensionFilter: true, a file named one.manifest rendered. With the filter on, include: '*.manifest' rendered nothing.
  • --server-proxy-url. The CLI created its RBAC objects over a direct connection; the server then failed with proxyconnect tcp: dial tcp: lookup server-proxy.lab ... no such host, so it used the proxy we passed. Adding --proxy-url sent the CLI’s own call through that second proxy.
  • admin export --strip-status. The export lost all status blocks, and kubectl apply -n argocd re-created two deleted Applications from it. The objects carry no namespace, so pass -n.
  • Deletion messages. A Service stuck on a finalizer reads “Pending deletion” on 3.5.7 and “Pending deletion; blocked by finalizers: lab.maborak.com/hold” on 3.6.1. A Lua deletionMessage replaced both the status and the message.
  • Tracing. 3.6.1 emits controller.Refresh, CompareAppState, sync.* and reposerver.GenerateManifest spans, and 59 of our traces ran from the controller into the repo server; 3.5.7 sent only gRPC spans. The post names the sampling variable <COMPONENT_NAME>_OTLP_SAMPLE_RATIO. Read literally, APPLICATION_CONTROLLER_OTLP_SAMPLE_RATIO=0 is ignored; the documented key otlp.sample.ratio, which sets ARGOCD_APPLICATION_CONTROLLER_OTLP_SAMPLE_RATIO, stopped every controller span. [8]
  • Hydrator metric. argocd_app_info gained hydrator_status="Hydrated" for a hydrated application.
  • CLI interrupts. The post says interrupted commands “could hang more than a minute”. Against an API server that accepts connections and never answers, both versions exit at once on Ctrl-C: 3.5.7 is simply killed by the signal. The hang appears in --core mode, where the CLI runs an API server in-process. We found that by looking further after the planned test showed no hang, so treat it as exploratory. After Ctrl-C, argocd app wait --core on 3.5.7 was still running at 150 seconds, when our script killed it, in 10 of 10 trials (Wilson 95% interval 72–100%), and argocd app logs --follow --core in 5 of 5. 3.6.1 exited within 0.01 seconds in all 15 (Fisher’s exact test, 15 of 15 against 0 of 15: p < 0.0001). Both CLIs ran against the same cluster. On 3.6.1, app wait reports timed out (600s) waiting for app when interrupted, which is misleading. Two earlier attempts were discarded because the CLI talked to the wrong cluster; see Limits. [7]
Raw output: rollback permission matrix 
## C22 rollback RBAC action (3.6.1), 2026-11-07T19:09:40Z
policy: dev has applications get+sync on default/*, no rollback
### A: server.rbac.rollback.enforce.enable unset (default)
yes
       ConfigMap  rb         rb    OutOfSync  Healthy        configmap/rb configured
### B: enforce=true, no rollback grant
no
{"level":"fatal","msg":"rpc error: code = PermissionDenied desc = permission denied","time":"2026-11-07T19:09:48Z"}
### C: enforce=true, rollback granted
yes
       ConfigMap  rb         rb    Synced  Healthy        configmap/rb configured
### D: enforce=true, rollback granted, but app has automated sync
{"level":"fatal","msg":"rpc error: code = FailedPrecondition desc = rollback cannot be initiated when auto-sync is enabled","time":"2026-11-07T19:09:56Z"}
### E: 3.5.7 CLI can-i rollback
{"level":"fatal","msg":"rpc error: code = InvalidArgument desc = [get create update delete sync override action invoke] does not contain rollback","time":"2026-11-07T19:09:57Z"}
Raw output: controller cache with `resource.selectors` 
## C23 resource.selectors, 2026-11-07T19:21:32Z
3000 ConfigMaps labelled lab-noise=1 in ns noise
### 357 before: {"resourcesCount":3437,"apisCount":60}
### 361 before: {"resourcesCount":3452,"apisCount":60}
### 357 after resource.selectors '!lab-noise' on ConfigMap: {"resourcesCount":3438,"apisCount":60}
### 361 after resource.selectors '!lab-noise' on ConfigMap: {"resourcesCount":453,"apisCount":60}
Raw output: custom file extensions 
## C26 custom extensions, 2026-11-07T19:05:21Z
### 3.6.1: disableExtensionFilter=true include={*.manifest,*.yaml}
application 'ext-on' created
### 3.6.1: filter on (default) include=*.manifest
application 'ext-off' created
### 3.5.7 CLI flag
Error: unknown flag: --directory-disable-extension-filter
### a361 manifests ext-on
  source: one.manifest
  name: from-manifest-ext
  source: two.yaml
  name: from-yaml-ext
### a361 manifests ext-off
### stored spec ext-on (3.6.1)
{"disableExtensionFilter":true,"include":"{*.manifest,*.yaml}"}
### CRD field present? 3.5.7 / 3.6.1
0
15
Raw output: `--server-proxy-url` 
## C19 cluster add --server-proxy-url, 2026-11-07T19:18:21Z
### 3.5.7 CLI
Error: unknown flag: --server-proxy-url
### 3.6.1 CLI: --server-proxy-url only (CLI connects directly)
{"level":"info","msg":"Created bearer token secret \"argocd-manager-long-lived-token\" for ServiceAccount \"argocd-manager\"","time":"2026-11-07T19:18:21Z"}
{"level":"fatal","msg":"rpc error: code = Unknown desc = error getting server version: failed to get server version: Get \"https://127.0.0.1:50093/version?timeout=32s\": proxyconnect tcp: dial tcp: lookup server-proxy.lab on 10.96.0.10:53: no such host","time":"2026-11-07T19:18:22Z"}
### stored cluster secret
### 3.6.1 CLI: --proxy-url to an unreachable proxy (CLI side) with a different --server-proxy-url
{"level":"fatal","msg":"failed to create service account \"argocd-manager\" in namespace \"kube-system\": Post \"https://127.0.0.1:50093/api/v1/namespaces/kube-system/serviceaccounts\": proxyconnect tcp: dial tcp 127.0.0.1:9: connect: connection refused","time":"2026-11-07T19:18:22Z"}
Raw output: `admin export --strip-status` 
## C20 admin export --strip-status (rerun with --context), 2026-11-07T19:06:30Z
### 3.5.7 CLI
Error: unknown flag: --strip-status
### Applications exported: 3
### top-level status: blocks full=4 stripped=0
   18437 out/export-full.yaml
   14232 out/export-strip.yaml
   32669 total
### delete Applications, re-create from stripped export with kubectl apply -n argocd
application.argoproj.io "ext-off" deleted from argocd namespace
application.argoproj.io "ext-on" deleted from argocd namespace
application.argoproj.io/ext-off created
application.argoproj.io/ext-on created
Warning: resource applications/plain is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
application.argoproj.io/plain configured
same Application set restored
Raw output: sampling variable names 
## C27 sample-ratio env var name, 2026-11-07T19:27:30Z
### arm=post-name since=2026-11-07T19:27:42Z env=[{"name":"ARGOCD_APPLICATION_CONTROLLER_OTLP_SAMPLE_RATIO","valueFrom":{"configMapKeyRef":{"key":"otlp.sample.ratio","name":"argocd-cmd-params-cm","optional":true}}},{"name":"APPLICATION_CONTROLLER_OTLP_SAMPLE_RATIO","value":"0"}]
    3 | argocd-controller | controller.CompareAppState
    3 | argocd-controller | controller.Refresh
### arm=doc-key-ratio0 since=2026-11-07T19:29:25Z env=[{"name":"ARGOCD_APPLICATION_CONTROLLER_OTLP_SAMPLE_RATIO","valueFrom":{"configMapKeyRef":{"key":"otlp.sample.ratio","name":"argocd-cmd-params-cm","optional":true}}}]
   0 controller spans
### arm=unset since=2026-11-07T19:31:09Z env=[{"name":"ARGOCD_APPLICATION_CONTROLLER_OTLP_SAMPLE_RATIO","valueFrom":{"configMapKeyRef":{"key":"otlp.sample.ratio","name":"argocd-cmd-params-cm","optional":true}}}]
    3 | argocd-controller | controller.CompareAppState
    3 | argocd-controller | controller.Refresh
Per-trial data: CLI exit after Ctrl-C (42 trials)
label cli trial exit_code seconds last_stderr
server-silent-tcp argocd-3.5.7 1 -2 0.01
server-silent-tcp argocd-3.6.1 1 1 0.00
server-silent-tcp argocd-3.5.7 2 -2 0.00
server-silent-tcp argocd-3.6.1 2 1 0.01
server-silent-tcp argocd-3.5.7 3 -2 0.00
server-silent-tcp argocd-3.6.1 3 1 0.00
server-silent-tcp argocd-3.5.7 4 -2 0.00
server-silent-tcp argocd-3.6.1 4 1 0.00
server-silent-tcp argocd-3.5.7 5 -2 0.00
server-silent-tcp argocd-3.6.1 5 1 0.00
server-silent-tcp argocd-3.5.7 6 -2 0.00
server-silent-tcp argocd-3.6.1 6 1 0.01
server-silent-tcp argocd-3.5.7 7 -2 0.00
server-silent-tcp argocd-3.6.1 7 1 0.00
server-silent-tcp argocd-3.5.7 8 -2 0.00
server-silent-tcp argocd-3.6.1 8 1 0.01
server-silent-tcp argocd-3.5.7 9 -2 0.01
server-silent-tcp argocd-3.6.1 9 1 0.00
server-silent-tcp argocd-3.5.7 10 -2 0.01
server-silent-tcp argocd-3.6.1 10 1 0.00
core-app-wait argocd-3.5.7 1 killed 150.00
core-app-wait argocd-3.6.1 1 20 0.01 {“level”:“fatal”
core-app-wait argocd-3.5.7 2 killed 150.00 {“level”:“error”
core-app-wait argocd-3.5.7 3 killed 150.00
core-app-wait argocd-3.6.1 3 20 0.01 {“level”:“fatal”
core-app-wait argocd-3.5.7 4 killed 150.00 {“level”:“error”
core-app-wait argocd-3.6.1 4 20 0.01 {“level”:“fatal”
core-app-wait argocd-3.5.7 5 killed 150.00 {“level”:“error”
core-app-wait argocd-3.5.7 6 killed 150.00 {“level”:“error”
core-app-wait argocd-3.5.7 7 killed 150.00
core-app-wait argocd-3.6.1 7 20 0.01 {“level”:“fatal”
core-app-wait argocd-3.5.7 8 killed 150.00
core-app-wait argocd-3.6.1 8 20 0.01 {“level”:“fatal”
core-app-wait argocd-3.5.7 9 killed 150.00
core-app-wait argocd-3.6.1 9 20 0.00 {“level”:“fatal”
core-app-wait argocd-3.5.7 10 killed 150.00
core-app-wait argocd-3.6.1 10 20 0.01 {“level”:“fatal”
core-app-logs-follow argocd-3.5.7 1 killed 150.00 {“level”:“error”
core-app-logs-follow argocd-3.5.7 2 killed 150.00 {“level”:“error”
core-app-logs-follow argocd-3.5.7 3 killed 150.00 {“level”:“error”
core-app-logs-follow argocd-3.5.7 4 killed 150.00 {“level”:“error”
core-app-logs-follow argocd-3.5.7 5 killed 150.00 {“level”:“error”
Discarded batch: CLI trials that reached the wrong cluster 
label,cli,trial,exit_code,seconds,last_stderr
server-silent-tcp,argocd-3.5.7,1,-2,0.01,
server-silent-tcp,argocd-3.6.1,1,1,0.00,
server-silent-tcp,argocd-3.5.7,2,-2,0.00,
server-silent-tcp,argocd-3.6.1,2,1,0.01,
server-silent-tcp,argocd-3.5.7,3,-2,0.00,
server-silent-tcp,argocd-3.6.1,3,1,0.00,
server-silent-tcp,argocd-3.5.7,4,-2,0.00,
server-silent-tcp,argocd-3.6.1,4,1,0.00,
server-silent-tcp,argocd-3.5.7,5,-2,0.00,
server-silent-tcp,argocd-3.6.1,5,1,0.00,
server-silent-tcp,argocd-3.5.7,6,-2,0.00,
server-silent-tcp,argocd-3.6.1,6,1,0.01,
server-silent-tcp,argocd-3.5.7,7,-2,0.00,
server-silent-tcp,argocd-3.6.1,7,1,0.00,
server-silent-tcp,argocd-3.5.7,8,-2,0.00,
server-silent-tcp,argocd-3.6.1,8,1,0.01,
server-silent-tcp,argocd-3.5.7,9,-2,0.01,
server-silent-tcp,argocd-3.6.1,9,1,0.00,
server-silent-tcp,argocd-3.5.7,10,-2,0.01,
server-silent-tcp,argocd-3.6.1,10,1,0.00,
core-app-wait,argocd-3.5.7,1,killed,150.00,{"level":"error","msg":"Error shutting down http server: close tcp 127.0.0.1:64772: use of closed network connection","t
core-app-wait,argocd-3.6.1,1,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:06:13Z"}
core-app-wait,argocd-3.5.7,2,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:06:23Z"}
core-app-wait,argocd-3.6.1,2,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:06:33Z"}
core-app-wait,argocd-3.5.7,3,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:06:43Z"}
core-app-wait,argocd-3.6.1,3,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:06:54Z"}
core-app-wait,argocd-3.5.7,4,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:07:03Z"}
core-app-wait,argocd-3.6.1,4,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:07:13Z"}
core-app-wait,argocd-3.5.7,5,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:07:24Z"}
core-app-wait,argocd-3.6.1,5,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:07:34Z"}
core-app-wait,argocd-3.5.7,6,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:07:44Z"}
core-app-wait,argocd-3.6.1,6,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:07:54Z"}
core-app-wait,argocd-3.5.7,7,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:08:04Z"}
core-app-wait,argocd-3.6.1,7,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:08:14Z"}
core-app-wait,argocd-3.5.7,8,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:08:24Z"}
core-app-wait,argocd-3.6.1,8,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:08:34Z"}
core-app-wait,argocd-3.5.7,9,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:08:44Z"}
core-app-wait,argocd-3.6.1,9,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:08:54Z"}
core-app-wait,argocd-3.5.7,10,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:09:04Z"}
core-app-wait,argocd-3.6.1,10,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:09:14Z"}
core-app-logs-follow,argocd-3.5.7,1,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:09:24Z"}
core-app-logs-follow,argocd-3.6.1,1,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:09:34Z"}
core-app-logs-follow,argocd-3.5.7,2,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:09:44Z"}
core-app-logs-follow,argocd-3.6.1,2,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:09:54Z"}
core-app-logs-follow,argocd-3.5.7,3,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:10:04Z"}
core-app-logs-follow,argocd-3.6.1,3,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:10:14Z"}
core-app-logs-follow,argocd-3.5.7,4,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:10:24Z"}
core-app-logs-follow,argocd-3.6.1,4,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:10:34Z"}
core-app-logs-follow,argocd-3.5.7,5,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:10:44Z"}
core-app-logs-follow,argocd-3.6.1,5,20,0.00,{"level":"fatal","msg":"configmap \"argocd-cm\" not found","time":"2026-11-07T20:10:54Z"}

The tracing change is the easiest to see. We sent the controller’s spans to Jaeger; one refresh of maborak-web is a single trace that runs from controller.Refresh through manifest generation in the repo server to the diff and the status update:

Jaeger UI trace argocd-controller: controller.Refresh, 12.82 ms, 2 services, depth 7, 11 spans: controller.CompareAppState, controller.GetRepoObjs, repository call, argocd-repo-server spans, controller.diff, controller.autoSync, controller.persistAppStatus, controller.handleRefreshAnnotations.

One reconcile of maborak-web as a trace across the controller and the repo server. Captured from Jaeger on 8 November 2026, 00:00 UTC.

Raw output: span names per service after one hard refresh, 3.5.7 and 3.6.1 
## C27 OTel span names by service (collector up ~3 min, includes one hard refresh of app plain), 2026-11-07T19:24:59Z
### 357 (count | service | span name)
   51 | argocd-controller | repository.RepoServerService/GenerateManifest
    4 | argocd-repo-server | grpc.health.v1.Health/Check
    3 | argocd-server | server.ArgoCDServer.Authenticate
    3 | argocd-server | server.ArgoCDServer.getClaims
    3 | argocd-server | session.SessionManager.VerifyToken
    1 | argocd-repo-server | repository.RepoServerService/GenerateManifest
    1 | argocd-repo-server | repository.RepoServerService/GetAppDetails
    1 | argocd-server | application.ApplicationService/Get
    1 | argocd-server | cluster.SettingsService/Get
    1 | argocd-server | project.ProjectService/Get
    1 | argocd-server | repository.RepoServerService/GetAppDetails
### 361 (count | service | span name)
  198 | argocd-controller | controller.CompareAppState
  198 | argocd-controller | controller.GetRepoObjs
  198 | argocd-controller | controller.GetRepoObjs.source
  198 | argocd-controller | controller.diff
  198 | argocd-controller | repository.RepoServerService/GenerateManifest
  176 | argocd-controller | sync.apply
  176 | argocd-controller | sync.runTasks
  109 | argocd-controller | controller.Refresh
  109 | argocd-controller | controller.autoSync
  109 | argocd-controller | controller.handleRefreshAnnotations
  109 | argocd-controller | controller.persistAppStatus
  107 | argocd-repo-server | reposerver.GenerateManifest
  107 | argocd-repo-server | repository.RepoServerService/GenerateManifest
   89 | argocd-controller | sync.getSyncTasks
   88 | argocd-controller | controller.Operation
   88 | argocd-controller | controller.SyncAppState
   88 | argocd-controller | sync.Sync
   12 | argocd-server | server.ArgoCDServer.Authenticate
   12 | argocd-server | server.ArgoCDServer.getClaims
   12 | argocd-server | session.SessionManager.VerifyToken
    7 | argocd-repo-server | reposerver.GenerateManifests
    4 | argocd-repo-server | grpc.health.v1.Health/Check
    4 | argocd-repo-server | reposerver.checkoutRevision
    4 | argocd-server | application.ApplicationService/Get
    4 | argocd-server | cluster.SettingsService/Get
    4 | argocd-server | project.ProjectService/Get
    1 | argocd-repo-server | repository.RepoServerService/GetAppDetails
    1 | argocd-repo-server | repository.RepoServerService/GetGitFiles
    1 | argocd-repo-server | repository.RepoServerService/GetOciDirectories
    1 | argocd-server | repository.RepoServerService/GetAppDetails
traces=214 with controller.Refresh=109 spanning controller+repo-server=59
example trace b35948db5a22a90a201b018a5b197966
   argocd-repo-server | reposerver.checkoutRevision
   argocd-repo-server | reposerver.GenerateManifests
   argocd-repo-server | reposerver.GenerateManifest
   argocd-repo-server | repository.RepoServerService/GenerateManifest
   argocd-controller | repository.RepoServerService/GenerateManifest
   argocd-controller | controller.GetRepoObjs.source
   argocd-controller | controller.GetRepoObjs
   argocd-controller | controller.diff
   argocd-controller | controller.CompareAppState
   argocd-controller | controller.autoSync
   argocd-controller | controller.persistAppStatus
   argocd-controller | controller.handleRefreshAnnotations
   argocd-controller | controller.Refresh

Counts and bug fixes

The post promises “about 20 new custom healthchecks”. At v3.5.7 the resource_customizations tree has 238 health.lua files; at v3.6.0 it has 293, and 55 of the 3.6.0 files do not exist at v3.5.7. [10] It says “more than 170 bugfixes”: the v3.6.0 release notes list 206 entries under Bug fixes, 190 without CI, docs and flaky-test fixes. [2]

Three of the seven “notable” fixes were backported before 3.6 shipped: the OIDC refresh fix (#27777) is in v3.5.0, #29070 in v3.5.2 and #28771 in v3.5.3. The repository-normalization item covers two fixes, and one of them (#28904) is also in v3.5.2. [11] On a current 3.5 patch release you already have those. New in 3.6 are #28414 (lost refresh requests), #28821 (impersonation), #28518 (oci:// without credentials) and #28619 (fewer repeated sync operations). Of these, we reproduced only #28821. #28619 was not tested; in progressive sync rollouts we saw six to ten sync operations per application within two seconds on both versions, which is a different code path.

All 32 claims

# Claim What we found
1 Resources Explorer Works, RBAC included; search is not kept in view preferences
2 Settings list search, filters, sort Works; no filter sidebar on Projects
3 Advanced settings page Works
4 Open rows in a new tab Works (<a href>)
5 Regex search Works
6 20–30% less rendering work Filtering about a third less script time; search unchanged
7 Lazy loading Works for routes and Settings; tree view not split; terminal and diff not tested
8 Virtual scrolling above 50 apps Works; threshold confirmed
9 Clear logs Works
10 X-Robots-Tag Works
11 ui.loginButtonText Works
12 Event component column Works
13 OCI generator Works (directory generator, one exclude)
14 Progressive sync refreshes apps first Works; default adds about 30 s before step 1
15 Four progressive sync metrics Works
16 New rollout conditions Works; names differ on the object
17 tpl in go-templates Works
18 Workqueue flags, ~16 min cap Works
19 --server-proxy-url Works
20 --strip-status Works
21 CLI interrupt handling Works; the hang was in --core mode
22 rollback RBAC action Works (permission; auto-sync refusal)
23 resource.selectors Works
24 Manifest compression, ~70% Live heap halved in a follow-up run; registered working-set test showed no difference; opt-in; post links wrong PR
25 Three CPU and memory changes In source
26 Custom file extensions Works
27 Spans and sample ratio Works; variable name wrong in post
28 hydrator_status label Works
29 About 20 new health checks 55 new health.lua files
30 Deletion messages Works
31 More than 170 bug fixes 206
32 Seven notable fixes Three and a half already in 3.5.x; #28821 reproduced; others not tested

Limits

  • One host, small scale. Everything ran on one arm64 Mac: three kind clusters in one Colima VM, sharing one Gitea and one registry. Timings and memory are relative comparisons at lab scale, not capacity numbers.
  • Planned and unplanned analyses. The rendering and memory protocols were written down before their first runs. The quiet rendering rerun, the live-heap memory run, the --core interrupt tests, the 10-resource dry-run count and the progressive sync mechanism check were added afterwards, after a first result or a method review; they are labelled where they appear.
  • Discarded runs. Two batches of CLI interrupt trials were discarded because the CLI talked to the wrong cluster (configmap "argocd-cm" not found, and pods ... not found with the command exiting before Ctrl-C). The first batch is shown in a collapsed block under the CLI results; the second was overwritten when the rerun started. A first progressive sync timing run overlapped controller restarts for other checks; its numbers matched the clean rerun, which is the one reported.
  • Partial tests of compound claims. Where the post lists several behaviours in one claim (rollback restrictions, OCI include and exclude patterns, lazy-loaded views), the table says which part we tested.
  • Screenshots and measurements are separate. The maborak-* objects were created after the measured runs, on the same clusters with the lab objects removed, to show each feature. The false-Synced result was found during that step, after the method review.
  • The writer ran the lab. The same AI agent designed the checks, ran them and wrote this post. Scoring is by script, but the choice of which follow-ups to run was the agent’s.

Sources

  1. “Argo CD v3.6 Less Memory, Faster UI, Better Rollout Visibility”, Eugene Doudine, Argo Project blog. Published 16 September 2026, modified 4 November 2026. https://blog.argoproj.io/argo-cd-v3-6-rc1-is-here-c0b562dc78d1 (opens in a new tab). The claims we tested.
  2. Argo CD releases v3.6.0 (3 November 2026) and v3.6.1 (4 November 2026), argoproj/argo-cd on GitHub. https://github.com/argoproj/argo-cd/releases/tag/v3.6.0 (opens in a new tab) and https://github.com/argoproj/argo-cd/releases/tag/v3.6.1 (opens in a new tab). Release dates, changelog, bug fix count.
  3. “v3.5 to 3.6” upgrade guide, docs/operator-manual/upgrading/3.5-3.6.md at tag v3.6.1. https://github.com/argoproj/argo-cd/blob/v3.6.1/docs/operator-manual/upgrading/3.5-3.6.md (opens in a new tab). SSA implies SSD, extra API server load, OTel changes.
  4. PR #29103 “Default to SSD with SSA and remove SMD”, merged 9 September 2026, closes issue #24134. https://github.com/argoproj/argo-cd/pull/29103 (opens in a new tab). The diff strategy change.
  5. PR #28600 “configurable serialization and compression for cached manifests”, merged 10 September 2026. https://github.com/argoproj/argo-cd/pull/28600 (opens in a new tab). Configuration keys and the 3.27 GB benchmark.
  6. PR #28821 “Server-Side diff should respect impersonation”. https://github.com/argoproj/argo-cd/pull/28821 (opens in a new tab). The impersonation fix.
  7. Issue #28732 “argocd CLI does not propagate SIGINT/SIGTERM”, opened 15 July 2026, and PRs #28734 and #29226. https://github.com/argoproj/argo-cd/issues/28732 (opens in a new tab). CLI signal handling.
  8. Observability documentation, docs/operator-manual/observability.md at v3.6.1. https://github.com/argoproj/argo-cd/blob/v3.6.1/docs/operator-manual/observability.md (opens in a new tab). otlp.sample.ratio and ARGOCD_<COMPONENT>_OTLP_* names.
  9. argocd-applicationset-controller command reference at v3.6.1. https://github.com/argoproj/argo-cd/blob/v3.6.1/docs/operator-manual/server-commands/argocd-applicationset-controller.md (opens in a new tab). --refresh-grace-period-seconds default 30, --wq-* flags.
  10. Source tree at v3.5.7 and v3.6.1: resource_customizations/**/health.lua, ui/src/app/applications/components/applications-list/virtual-scroll.ts, util/settings/settings.go. https://github.com/argoproj/argo-cd/tree/v3.6.1 (opens in a new tab). Health check count, virtual scroll threshold, settings keys.
  11. Backports on release-3.5: #28689 (#27777), #29195 (#29070), #29399 (#28904), #29486 (#28771). https://github.com/argoproj/argo-cd/commits/release-3.5 (opens in a new tab). Which “notable” fixes are already in 3.5.x.
  12. PR #27820 rollout conditions, and pkg/apis/application/v1alpha1/applicationset_types.go at v3.6.1. https://github.com/argoproj/argo-cd/pull/27820 (opens in a new tab). Condition type and reason names.

Topics: kubernetes, sre, migration

Back to all articles

Technologies we work with

  • Argo CD
  • Kubernetes
  • Docker
  • Podman
  • Ansible
  • Jenkins
  • Chef
  • Proxmox VE
  • Google Cloud
  • Linux
  • Red Hat
  • CentOS
  • NGINX
  • Apache
  • Traefik
  • Prometheus
  • Grafana
  • PostgreSQL
  • MySQL
  • MariaDB
  • Redis
  • etcd
  • Qdrant
  • Elasticsearch
  • Logstash
  • Kibana
  • PagerDuty
  • Go
  • Python
  • FastAPI
  • PHP
  • TypeScript
  • React