Resolved -
The entire backlog has been re-processed. Systems are back in a healthy state.
May 27, 08:00 EDT
Update -
We are continuing to work through the backlog of re-processing the in-flight TM2 requests.
May 27, 07:21 EDT
Monitoring -
We have worked with our infrastructure provider to fix connectivity issues, and all systems are recovering. We are working on re-processing in-flight TM2 requests during the outage.
May 27, 06:19 EDT
Update -
We are working on requeuing any outstanding transfers submitted during the incident
May 27, 05:40 EDT
Update -
We failed over TM2, and all new requests should proceed as normal. We are actively working on requeueing pending requests posted during the outage.
May 27, 04:45 EDT
Update -
We are continuing to work on a fix for this issue.
May 27, 04:27 EDT
Identified -
We have validated the root cause (web tier egress IPs were auto-blocklisted for malicious activity), and we are continuing to failover to AWS unblock new requests.
May 27, 04:26 EDT
Update -
We are continuing to look into the issue and pursuing a few mitigation options.
May 27, 04:00 EDT
Update -
We are continuing to investigate network connectivity issues between our web tier and confluent cloud. We have identified a potential cause, and currently working on failover
May 27, 03:56 EDT
Update -
We are continuing to investigate this issue.
May 27, 03:23 EDT
Investigating -
We an investigating an issue with an upstream infrastructure provider, and working on failover options.
May 27, 03:21 EDT