Time Series DB Timeouts: Intermittent Latency and Connectivity from Netherlands to Germany
Marking as resolved as not seeing any further drops from application and network level internal monitoring. (App - Honeybadger, App/network bidirectional - monit).
2022-12-06 14:00:55 UTC
We will keep this ticket open for the next 24 hours to monitor.
2022-12-05 21:38:40 UTC
Free user plans are intermittently affected by time-series database timeouts due to Internet transit issues between our providers. Paid users with buckets on the Influx commercial cloud are unaffected.
There are intermittent transport issues and latency between DigitalOcean Amsterdam, NL (webserver) and Hetzner FSN1 Falkenstein, DE (time-series DB server), with the result that we are exceeding some Influx "open" NET::Timeout for queries. We are investigating but neither vendor's status pages have issues so it might be an as of yet highlighted path or AS. This is manifesting as Influx DB timeouts as the network socket open threshold for the API client is set at 10s and the READ timeout is 20s.
Note: Nothing reported for DigitalOcean: DC AMS3 https://status.digitalocean.com/ and only non-outage related electricity upgrades ???? for Hetzner: FSN1 https://status.hetzner.com/