Outages everywhere!
Fall is here, and it seems as though it’s not just the leaves, but the communication and collaboration services that are dropping left and right lately. One of the most significant outages to hit the Comms environment comes from Cisco, a leader in cloud and on-premises communication solutions.
At the time of writing (28th of September), some Cisco WebEx users are still facing issues with their connections, days after the service started to experience problems on the 25th of September. The WebEx outage has been so significant, that company CEO Chuck Robbins jumped onto his Twitter account to apologize for the inconvenience, saying:
“The @WebEx outage today is unacceptable.”
Chuck reassured users on the 25th that Cisco engineers were already rushing to restore their WebEx solutions and ensure that an outage of such magnitude wouldn’t happen again.
The Scope of the Cisco WebEx Outage
Often, when cloud-based communication services go down, they’re back up and running again within the same day. The flexible nature of the cloud makes it easier for engineers to find issues and roll out solutions rapidly. Unfortunately, Cisco seems to be having some serious trouble getting their services back online.
Cisco first began identifying problems on the 24th of September at 9:44 P.M ET. The status website for Cisco stated that WebEx Teams are “currently impacted by an ongoing service outage.” Early Tuesday morning, the complaints started to roll in from customers who still couldn’t access their Teams, Meeting, Calling, Care and Context solutions from Cisco.
Now, three days on, Cisco has officially reported that WebEx Meetings has been restored to “fully operational” status. However, no details are available yet as to why the system shut down in the first place. All we know is that engineers have fixed a problematic behaviour in the WebEx infrastructure.
Additionally, core messaging capabilities are now back up-and-running, with access to full messaging history for users. However, you can only access your message history if you have previously stored messages on your device. Additional message history options may not be available from the Cisco cloud services at this stage, and you won’t be able to flag any messages, see user presence, or access the “one button to push” or “ability to bind a space” feature.
Rebuilding WebEx one Feature at a Time
It seems that Cisco has been forced to resort to rolling fixes out one at a time, rather than being able to simply flip a switch and get WebEx back on track. On the 27th of September, Cisco announced that “complete service restoration for messaging is an ongoing effort.”
At the time of writing, WebEx Calling device activation is now back up and running, and Cisco has also reported that WebEx Desktop calling isn’t affected by the incident. Hybrid call services became available on the 27th of September at 22:48 UTC. An hour later, Cisco also announced that the Care Assistant Bot for Cisco WebEx solutions is now operational. Other fixes rolled out up to now include:
27th of September:
- Hybrid call services available
- Work continues into the restoration of Messaging features
- Calendar services are “partially” restored
- Control hub services are restored
28th of September:
- Call Service Connect and Call Serviced Aware are restore
- Engineering tests a fix for the current webhook issue
- Download, install, and authentication access is available
- Monitoring for Call Service Connect and Call Service Aware is now fully operational
Currently, engineering teams are continuing to work on executing additional fixes for the WebEx system. The team is also working on implementing a webhook fix deployment plan. Cisco has announced that once this fix is deployed, the Dev API and integration restoration process will be complete.
You can check out the full list of updates to the Cisco WebEx service status.
Cisco Isn’t the Only Company Facing Problems
Of course, Cisco isn’t the only communications vendor that has seen problems this month. Recently, Microsoft Teams posted an outage at their data centre in Texas, which left some users unable to access Office documents within Teams on September 4th.
More significantly, Slack has seen a massive selection of outages in the last few months, with a three-and-a-half-hour disruption on the 27th of June, a 3-hour outage on the 17th of July, and disrupted services throughout August.
However, while outage issues may not be rare across cloud service providers, it’s difficult to find a vendor that’s experienced an issue quite as severe as the one that Cisco is facing right now. Cisco’s outage makes all the other problems reported by vendors like Slack and Microsoft Teams seem tiny in comparison. After all, it’s taking several days to get all the various WebEx solutions that Cisco provides back up and running.
Though it’s evident that Cisco is doing everything they can to support their users and get their solution back into working order, the company’s reputation is already taking a significant hit on Twitter, were countless users lamented their inability to use Meeting services for 22 hours.
At this point, we have no idea what’s caused such a severe problem within the Cisco WebEx environment. However, we’ll try to keep you updated with the latest developments as they happen.
Have any of these Team Collaboration outages affected you or your business? Let us know in the comments below.
