This incident has been resolved, nothing new happened since the last update. A detailed communication is being prepared by our customer facing teams.
Posted Jul 24, 2024 - 16:31 CEST
Monitoring
The OVHcloud API and control plane is back online, which allowed the OVH support team to fix their DDoS protection mechanism. FRA1 is now fully reacheable and operational. We are seeing a nominal volume of incoming events, our API error rates are low and our external TCP probes are reporting no errors. Our team is still actively monitoring the situation to ensure FRA1 remains stable.
We are closely working with OVHcloud to produce a post-mortem and ensure their DDoS protection does not erroneously trigger again.
Posted Jul 24, 2024 - 12:56 CEST
Update
OVHcloud is having a parallel incident which made their administration console and API unavailable. This means that both OVH support and Sekoia.io incident handlers are unable to perform any changes to our FRA1 infrastructure.
We are waiting for the OVH control plane to get back online for them to disable the faulty DDoS protection rules.
In the meantime, our team is looking for workarounds.
Posted Jul 24, 2024 - 12:10 CEST
Identified
Our cloud provider for FRA1 (OVHcloud) has allegedly flagged our cloud load balancer traffic as a DDoS attempt, resulting in a complete loss of access for the web application, APIs, and intakes. Their DDoS protection is currently flapping on and off, resulting in an unstable access to FRA1. We are working with our OVH Technical Account Manager to resolve this as soon as possible.
Posted Jul 24, 2024 - 10:51 CEST
Investigating
We are currently facing a major outage due to a problem on our cloud provider side, impacting our reverse proxies. Our team is getting in touch with them to get more information and try to fix this.
Posted Jul 24, 2024 - 10:32 CEST
This incident affected: FRA1 - XDR (Ingestion, Threat Intelligence for research & triage, Automation, Event storage, Detection, Hunting, Case management, Web application) and FRA1 - CTI (Search, API consumption, TAXII consumption, MISP consumption, Enrichers, Web application).