Prometheus 3 changes queries, not just the binary

The 3.0 migration guide is a list of silent behaviour changes: regex newlines, left-open ranges, normalized le labels and strict scrape content types.

By the Maborak engineering team · 2026-10-29 · 4 min read

A Prometheus major upgrade is usually treated as a binary replacement: new image, same configuration, same dashboards. Version 3 changes query semantics and scrape behaviour in ways that produce wrong values or no data rather than an error, so the dashboards keep rendering and the numbers quietly move.

Checked 29 October 2026. The behaviours below are from the Prometheus 3.0 migration guide. The guide covers 2.x to 3.0 and newer; the line numbers and examples are from that page.

Background

Prometheus 2.x accumulated feature flags for behaviour that later became default: the @ modifier, negative offsets, the service-discovery manager, external-label expansion. Version 3 folds those into the default behaviour and removes the flags. The same release adds native histograms as a stable feature and moves the TSDB format slightly, which limits downgrade.

Mechanics

The changes that break silently:

Regular expressions match newlines. In PromQL, . now matches \n. A matcher like .* that used to stop at a line boundary now crosses it. To keep the old behaviour, replace . with [^\n].
Range selectors are left-open. [5m] used to include a sample exactly on the left boundary. It no longer does. Subqueries are the common failure: foo[1m:1m] can return one point instead of two, so rate() returns no data. Widen the window (foo[2m:1m]) rather than guessing.
le and quantile values are normalized. A classic histogram ingested as le="1" becomes le="1.0". Alerts and recording rules that match le="1" stop matching.
Scrapes require a recognised Content-Type. Version 2 fell back to the text format when the header was missing or unparsable. Version 3 fails the scrape; fallback_scrape_protocol restores the old fallback for a specific job.
Remote write defaults to HTTP/1.1. http_config.enable_http2 is now false. A remote-write endpoint that relied on HTTP/2 needs the option set explicitly.
holt_winters is renamed to double_exponential_smoothing, and it is behind promql-experimental-functions.
Metric names accept UTF-8. Names that were previously rejected now pass validation. metric_name_validation_scheme: legacy restores the old scheme globally or per scrape.
Alertmanager v1 is removed. Prometheus 3 requires Alertmanager 0.16 or newer and api_version: v2.

Impact

The dangerous changes are the ones with no error path: a dashboard that shows a slightly different rate, an alert whose le matcher never fires again, a scrape target that now fails and looks like a target outage. A test instance running the same rules and queries against the same data is the only cheap way to see them.

Probe

promtool check config /etc/prometheus/prometheus.yml

# Removed flags still passed to the old binary.
grep -REn -- '--enable-feature=.*(promql-at-modifier|promql-negative-offset|new-service-discovery-manager|expand-external-labels|no-default-scrape-port|remote-write-receiver|auto-gomemlimit|auto-gomaxprocs)' /etc/prometheus /etc/default 2>/dev/null

# Integer le/quantile matchers that will stop matching after normalization.
grep -REn 'le="[0-9]+"' /etc/prometheus/rules /etc/prometheus/*.yml 2>/dev/null

# Renamed function still in use.
grep -REn '\bholt_winters\b' /etc/prometheus 2>/dev/null

Then run the same rules against a 3.x test instance pointed at a copy of the TSDB, and compare the query results for the dashboards that matter. The promtool check validates syntax; it does not validate semantics.

Mitigation

Upgrade to 2.55 first if the current install is older. It writes the TSDB format that 3.x can read, and it is the last version you can downgrade to without losing data.
Change integer le/quantile references to their float form rather than relabelling. Relabelling is a per-metric workaround that hides the change.
Widen subquery windows where a rate needs at least two points.
Set fallback_scrape_protocol only for targets that cannot fix their Content-Type. It is a compatibility switch, not a fix.
Run Alertmanager 0.16 or newer before Prometheus 3, and switch the configuration to api_version: v2.

Verification

promtool check rules /etc/prometheus/rules/*.yml
curl -s localhost:9090/-/healthy
curl -s 'localhost:9090/api/v1/query?query=histogram_quantile(0.9,sum(rate(http_request_duration_seconds_bucket[5m]))by(le))' | head

Compare one known alert and one rate panel with the pre-upgrade result on the same range. A successful health check only says the process started.

Sources

Prometheus 3.0 migration guide (checked 29 October 2026).
Prometheus stability promise.
promtool documentation.

Topics: observability, sre

Back to all articles