Resolved -
Google marked the incident fully recovered. Our health-checks confirmed service stability. All Nonli components are now operating normally.
Jun 13, 02:28 UTC
Identified -
Google has now published an advisory on its Cloud Service Health Dashboard showing “Service Disruption” for several core products including Compute Engine, Google Kubernetes Engine and more across multiple regions. This confirms that the outage affecting Nonli is rooted in an upstream Google Cloud Platform incident. https://status.cloud.google.com/
Jun 12, 18:54 UTC
Investigating -
Impact All Nonli services are currently unreachable. Users may encounter time-outs, 503 Service Unavailable errors, or blank pages when trying to use the application or its API. Background jobs and webhooks are also failing.
Preliminary Root Cause Our infrastructure runs on Google Cloud Platform (GCP). External observers (e.g. Downdetector https://downdetector.fr/statut/google/) are reporting widespread issues with Google services, suggesting an upstream outage. Google’s public status dashboard has not yet acknowledged an incident.
Our Actions Time (CEST) Action 20:10 Incident declared – all on-call engineers paged. 20:15 Attempted to open a Sev-1 case via Google Cloud Support. The support portal is currently unreachable. 20:18 Retrying escalation through alternative channels: phone hotline and account-manager email. 20:22 Activated internal failover playbook; standby replicas in other regions cannot take traffic until GCP networking stabilises. 20:25 Increased monitoring frequency and set up automatic status-page updates every 30 minutes.
Next Update We will post an update no later than 21:00 CEST (19:00 UTC), or sooner if we receive material information from Google.
Jun 12, 18:29 UTC