We have completed critical changes to our event processing pipeline to address and durably fix the underlying issue that was affecting metrics performance. During this urgent maintenance, we had to pause the processing of events to stabilize the system and ensure no events were lost.
From 10 AM to 5 PM today, a small percentage of events were enqueued without our monitoring system notifying, which is the initial cause of this maintenance. They are now being processed. As a result, you may experience significant delays for this subset of events. Rest assured, no events were lost during this process.
This incident will be closed when we are back in real-time.
We appreciate your patience and understanding. If you have any questions or concerns, please reach out to our support team. A detailed post-mortem will be made available in the following days.
Posted Jun 19, 2024 - 22:08 CEST
Update
Our services are not all up and the platform is not stable. Ingestion is still working, we are receiving events. We are still accumulating lag on event processing.
Posted Jun 19, 2024 - 19:08 CEST
Update
Services are coming up again. The platform is starting to consume events that are in queue.
Posted Jun 19, 2024 - 18:16 CEST
Identified
Our team is currently performing a maintenance operation on the platform to fix a recurring metric problem. We are taking lag on event processing and storage, but we are still ingesting events. We'll keep you updated.
Posted Jun 19, 2024 - 17:45 CEST
This incident affected: FRA1 - XDR (Event storage, Detection).