Back to overview
Resolved

Elevated latency and timeouts on the Ampersand-hosted workspace API (back-filled)

Aug 21, 2026 at 9:02pm UTC
Affected services
Ampersand-hosted workspaces — API

Resolved
Aug 21, 2026 at 9:02pm UTC

Back-filled from our internal incident record before this status page existed; timestamps are the internal ones.

2026-08-21 21:02 UTC — Detected. Our liveness probe began timing out; the API writer database sat at 100% CPU. Root cause: the org-scoped health check ran an unbounded aggregate query against the primary database on every tick and became its heaviest statement. Capture ingest and Heimdall reads were slow or intermittently timing out for workspaces hosted by Ampersand. No data was lost.

2026-08-22 02:42 UTC — Fix 1 deployed: the health check now runs on the read replica with a strict statement timeout and a 60-second per-org cache.

2026-08-22 02:47 UTC — Probe latency back to normal (2.0s → 0.47s); alarm cleared.

2026-08-22 04:05 UTC — Fix 2 deployed: the expensive 7-day row count is now best-effort and never blocks the probe.

Customer-deployed (BYOC) workspaces were not affected.