Marking as resolved. Continuing to investigate and to identify/confirm for root cause for mitigations to be put in place.

Investigating aggressive vulnerability scanning from the Internet which may have caused Puma/Rack to stop responding. NGINX was serving throughout. Rails continued to process background jobs, but Puma/Rack seemed to stop serving content. HoneyBadger alerts and the Rails production log show non-platform vulnerability-specific scanning which is a common occurrence on the Internet (we encounter it constantly) so Puma/Rack failing to continue serving is unusual. Continuing to investigate.

The problem was fixed with a hard Puma restart (had to manually kill processes). Investigating.

Marking as resolved while continuing to investigate.

The main PanSift application running Ruby on Rails stopped responding at 2023-04-22 03:29:46 UTC. A restart of the Rails service fixed the issue at 2023-04-22 07:46:04 UTC. We are investigating now. We apologise for the 4h 16m 17s outage. No agent data was affected or lost, as the ingestion tier is separate from the web tier. Only the ability to view data was affected.