india employmentnews

Microsoft: This is why chaos erupted across 74 countries—the major reason behind the service outages has been revealed..

 | 
cxcx

The global disruption to Microsoft’s email and other cloud services last week was caused by several technical factors. The company reported that, at the peak of the outage, more than 80% of its networking gateway servers stopped functioning. The root cause was a bug hidden within a software change implemented in July—a change that had operated without any issues for over three months.

According to Microsoft, the bug was triggered by a combination of three specific conditions, leading to a rapid failure of the servers. Charles Lamanna, an Executive Vice President at Microsoft, shared these details on the social media platform X on Monday, while also apologizing to customers for the service disruption.

**Bug originated in a July software change**
According to Lamanna, the primary cause of the outage was a flaw in a software update designed for the networking gateway servers. Under certain load conditions, this bug placed excessive pressure on the servers, causing them to stop responding.

This change was implemented in July and remained present on servers worldwide for more than three months without causing any issues. The flaw also went undetected during the company’s load testing, as those tests were conducted on older, lower-capacity hardware—whereas the bug had a more severe impact on newer servers equipped with a higher number of processing cores.

**Teams traffic increased pressure on servers**
A second factor contributing to the outage was a shift in Microsoft Teams traffic. Approximately 15 hours before the issues began, Teams traffic was moved from an alternative networking gateway to the primary gateway. This same gateway also supports Exchange Online, Microsoft’s cloud-based email service.

Following this shift, traffic on the primary gateway increased, placing a heavier workload on the servers. The company noted that such traffic shifts are a standard part of its operational procedures. A third reason was that, due to an unrelated technical issue that occurred a day before the outage, approximately 14% of the gateway's server capacity was already unavailable. Consequently, the combination of increased traffic and reduced server capacity triggered the software bug from July.

More than 80% of servers became unresponsive.
When some servers stopped responding, Microsoft’s traffic management system began redirecting user requests to other active servers. Typically, this mechanism helps keep services running in the event of a server or network failure.

However, in this instance, the situation backfired. Traffic that was previously distributed across many servers began concentrating on a smaller number of servers. This increased the load on the remaining servers, causing them to stop responding one by one—a chain reaction that spread the issue from server to server.
Lamanna explained that the disruption spread throughout the entire server network in less than two hours. At the peak of the outage, more than 80% of the servers had become unresponsive. While safeguards are generally in place to isolate different parts of the gateway from one another, a separate flaw rendered these protections ineffective, allowing the problem to spread globally.

Teams and OneDrive were restored first; Outlook took longer
According to Microsoft, services such as Teams, OneDrive, SharePoint, and Office apps were successfully migrated away from the affected gateway to other systems, allowing for a relatively quick restoration. For much of the duration of the outage, Teams chat and meeting features remained functional.

Exchange Online, however, could not be shifted to another gateway in the same manner. This is why it took longer to restore Outlook email services. As soon as the servers were restarted, they were overwhelmed by a backlog of emails and other requests, causing them to stop responding once again.

The company’s automated recovery system was not equipped to handle a failure of such magnitude. Furthermore, the bug in the software had been present for several months, rendering standard, immediate remediation methods ineffective.
Ultimately, engineers developed an update to revert the software code to a previous version. This involved removing individual server groups from the network, applying the necessary changes, and then restarting them. The company plans to release a detailed review of the incident later this week.

Airlines and financial companies in India were also affected.

The outage, which began late Thursday night, impacted business operations across several countries, including India. In India, airlines such as IndiGo, Air India, and Akasa Air reported difficulties with their email-based customer support, which hindered their ability to receive or respond to certain passenger inquiries and requests.

To maintain communication with customers via email, Air India temporarily routed its email services through an alternative channel. However, there were no reports of any impact on flight operations.

Some companies in the financial services sector also faced challenges in accessing research reports and communicating with clients. During the outage, users were unable to open their mailboxes on both the web and desktop versions of Outlook. In many instances, there were also delays in the delivery of emails.

DownDetector, a website that tracks service outages, recorded over 130,000 complaints regarding the issues with Outlook. The problem was reported across 72 to 74 countries worldwide.


Disclaimer: This content has been sourced and edited from Amar Ujala. While we modified it for clarity and presentation, the original content belongs to its respective authors and website. We do not claim ownership of the content.