Ally - US-EAST-1 - Degraded Performance
Incident Report for Blackboard Inc.
Postmortem

On February 1st, 2022, our team was notified of delays related to Ally's processing of new content for Blackboard Learn clients in the US-EAST-1 region. 

The Learn telemetry service responsible for sending events to Ally became unresponsive and a backlog of events accumulated preventing processing of new content.  

In the short term our teams will be adding additional monitoring to our systems to ensure we are able to more quickly respond to similar types of events as part of an upcoming Ally release. 

Longer term our teams will be looking to improve the supporting telemetry infrastructure to better support Blackboard Learn clients. 

We realize the impact of the service disruption and apologize for the impact this may have created at your institutions.

Posted Feb 07, 2022 - 15:58 EST

Resolved
This has been resolved, the backlog of events have been processed. New events are being processed in real time.
Posted Feb 03, 2022 - 08:18 EST
Monitoring
Processing of new content has been restored with some delays. We anticipate these queued items to be processed by February 4th, 2022.
Posted Feb 02, 2022 - 10:55 EST
Identified
Processing of new content has been restored with some delays. Our teams continue to work on restoring services to normal operations.
Posted Feb 01, 2022 - 17:05 EST
Investigating
We are currently investigating an issue with Ally in the Americas (excluding Canada) that is preventing the processing of new content added to Learn courses. All other Ally capabilities remain functional. Please follow status.blackboard.com for updates.
Posted Feb 01, 2022 - 14:22 EST
This incident affected: Americas (Ally - North America).