Commercial Customers: Influx Commercial Cloud API

Resolved in 3 hours

Incident timeline

It looks like bucket reads are coming back from the Commercial Influx cloud. Successful API checks from (UTC) Sep 7 21:23:30 onwards: Sep 7 21:24:25 dsf3-b001 haproxy[3233815]: 2a03:b0c0:2:d0::100b:8001:41226 [07/Sep/2023:21:24:25.177] webrouter-vip~ ingest5/influxcloud5 0/0/50/44/94 200 We are not seeing any further HTTP 5xx at this time. Closing.
2023-09-07 21:24:00 UTC - 23 days
Problem detected
Summary: Our monitoring highlighted gateway timeouts from the Influx *Commercial* cloud API (not the free OSS tier) in relation to the main /api/v2 endpoint. Agents look to be able to intermittently write to their respective buckets (agents will continue to buffer data they can not write, and no data is dropped). Issue is affecting the web application and read requests for bucket endpoints (inc. our Influx Cloud operator UI). The top-level API is returning HTTP 504's, and the write endpoint is intermittently reporting normal HTTP 204's. Investigating further. Impact (Commercial Customers *only*): Web application is not returning data for commercial customers, reads are also failing. Agent writes are intermittent. Began (UTC): Sep 7 18:18:10 dsf3-b001 haproxy[3233815]: 2a03:b0c0:2:d0::100b:8001:37636 [07/Sep/2023:18:17:57.304] webrouter-vip~ ingest5/influxcloud5 0/0/50/13095/13145 502 398 - - ---- 11/11/2/2/0 0/0 "GET /api/v2 HTTP/1.1" Sep 7 18:23:16 dsf3-b001 haproxy[3233815]: ::ffff: [07/Sep/2023:18:23:06.204] webrouter-vip~ ingest5/influxcloud5 0/0/49/10506/10555 504 413 - - ---- 26/26/9/9/0 0/0 "GET /api/v2 HTTP/1.1" Vendor: Influx status pages do not reflect any issues at this time, but we're checking with the vendor on their Slack.
2023-09-07 18:18:00 UTC - 23 days