Problem detected
Investigating
Summary:
Our monitoring highlighted gateway timeouts from the Influx *Commercial* cloud API (not the free OSS tier) in relation to the main /api/v2 endpoint. Agents look to be able to intermittently write to their respective buckets (agents will continue to buffer data they can not write, and no data is dropped). Issue is affecting the web application and read requests for bucket endpoints (inc. our Influx Cloud operator UI). The top-level API is returning HTTP 504's, and the write endpoint is intermittently reporting normal HTTP 204's. Investigating further.
Impact (Commercial Customers *only*):
Web application is not returning data for commercial customers, reads are also failing. Agent writes are intermittent.
Began (UTC):
Sep 7 18:18:10 dsf3-b001 haproxy[3233815]: 2a03:b0c0:2:d0::100b:8001:37636 [07/Sep/2023:18:17:57.304] webrouter-vip~ ingest5/influxcloud5 0/0/50/13095/13145 502 398 - - ---- 11/11/2/2/0 0/0 "GET /api/v2 HTTP/1.1"
Sep 7 18:23:16 dsf3-b001 haproxy[3233815]: ::ffff:178.62.233.176:59430 [07/Sep/2023:18:23:06.204] webrouter-vip~ ingest5/influxcloud5 0/0/49/10506/10555 504 413 - - ---- 26/26/9/9/0 0/0 "GET /api/v2 HTTP/1.1"
Vendor:
Influx status pages https://status.influxdata.com/ do not reflect any issues at this time, but we're checking with the vendor on their Slack.
2023-09-07 18:18:00 UTC
-
23 days