Blackboard Learn - Global DNS Issue
Incident Report for Blackboard Inc.
Postmortem

4/27/2023 Update: The changes that were being planned for 4/28 have been postponed due to feedback from our client community.

Anthology has implemented additional measures to prevent and minimize future occurrences of the events that took place on 4/20 and 4/25. We continue to evaluate additional enhancements to our infrastructure and will communicate any additional enhancements or changes via our standard channel.

Please see Article 77581 for the Incident Report.
https://blackboard.secure.force.com/btbb_articleview?id=kAE5Y000000CaojWAC

Posted Apr 27, 2023 - 11:58 EDT

Resolved
This update is to let you know that the DNS changes that occurred yesterday have been rolled back and any issues related to this change are now resolved, we are currently working with our vendors to ensure that such incidents do not happen in the future.

While we are confident that the issue has been resolved, we understand that some users may still experience problems. This is because the DNS server propagation process may not have completed for those who are connected to certain Internet Service Providers (ISPs). However, please be assured that this issue will resolve itself over time as the Time to Live (TTL) on the ISP DNS servers expires and the servers update accordingly.

If you or your team receive an influx of end-user complaints, please do not hesitate to open a case.

4/27/2023 Update:
The changes that were being planned for 4/28 have been postponed due to feedback from our client community.

Anthology has implemented additional measures to prevent and minimize future occurrences of the events that took place on 4/20 and 4/25. We continue to evaluate additional enhancements to our infrastructure and will communicate any additional enhancements or changes via our standard channel.
Posted Apr 26, 2023 - 09:48 EDT
Update
Service to the majority of sites has been restored. Our Engineers are actively engaging with vendors on the Permanent fix and ensuring the continuity of our services without any Interruptions. Please contact client support if you are still seeing any issues.
Posted Apr 26, 2023 - 09:12 EDT
Update
We are continuing to work on a fix for this issue.
Posted Apr 26, 2023 - 09:11 EDT
Update
We are continuing to work on a fix for this issue.
Posted Apr 26, 2023 - 09:06 EDT
Update
Service to the majority of sites has been restored. Our Engineers are actively engaging with vendors on the Permanent fix and ensuring the continuity of our services without any Interruptions. Please contact client support if you are still seeing any issues.
Posted Apr 25, 2023 - 21:32 EDT
Update
We are seeing most sites return to normal functionality as their DNS refreshes. We are actively engaged with our vendors to find opportunities to expedite a complete resolution.
Posted Apr 25, 2023 - 15:22 EDT
Update
We believe the DNS issue has been resolved. Blackboard technical team are advising the following steps if users are still experiencing issues with DNS

Run the Flush DNS command on DNS Servers
(IF THAT DOESN'T WORK) Reduce the TTL on DNS Servers
(IF THAT DOESN'T WORK) Recommend individual users to run the Flush DNS command in their local machines.

If Issue still persist Please contact client support for any assistance.
Posted Apr 25, 2023 - 09:33 EDT
Update
Issue still persists with some of the clients. We are continuing to work on the fix for the issue. Please contact client support for any assistance.
Posted Apr 25, 2023 - 07:33 EDT
Identified
Several of the affected systems have recovered and work continues with the remaining environments that still are experiencing performance degradation. Please contact client support if you are still seeing any issues.
Posted Apr 25, 2023 - 04:32 EDT
Investigating
Blackboard Learn Sites are Intermittent at the moment. Teams are currently Investigating the issue.
Posted Apr 25, 2023 - 03:14 EDT
This incident affected: Global (Learn - SaaS - Global) and Americas (Learn - SaaS (US)).