Complex Drive will be performing a maintenance window on Friday, March 21st at 11:59pm. This 2 hour window will last until 2:00am on Saturday March 22nd. During this maintenance event, customers will experience brief periods of latency and intermittent loss of connectivity during the 2 hour window. The purpose of this window is to perform a number of critical upgrades which will enhance overall performance and redundancy.

UPDATE:
The maintenance window scheduled for March 21/22 has been completed on schedule. The focus of this window was three fold:

1.Resolve all CEF / Layer 2 routing issues 100%. – COMPLETED
2.Ensure availability of redundant network components – COMPLETED
3.Improve overall network performance – COMPLETED

Complex Drive has concluded after considerable testing that the CEF / Layer 2 routing issues have been resolved 100%. The Network is running at peak performance levels and we anticipate zero issues moving forward. Customers should now experience better overall performance in addition to having extremely reliable and redundant IP services.

We thank you for your patience during this past week, and for your continued patronage.

Complex Drive experienced a Network Issue this morning. At this time, all services are stable. More details will be provided as soon as they are available.

UPDATE: During preparation for tomorrow’s maintenance window, our Networking staff discovered additional Layer 2 vulnerabilities which exhibited characteristics similar to those which caused problems this past week.

In the interest of our customers, the decision was made at 12:56am to mitigate the issues immediately as opposed to waiting until the preplanned maintenance window late Friday night. The line of thought was that if these issues manifested a problem during the Friday business hours it would cause a considerably larger impact on our customer base.

The issues have been resolved and with the final changes being made during tonight’s upcoming maintenance window, the Layer 2 / CEF routing issues should be 100% contained.

We apologize for the early morning changes, but the critical nature of the issue required action and we acted with the best interest of our customers in mind.

Complex Drive experienced a Network Issue this afternoon. At this time, all services are stable. A full update regarding this issue has been sent via email to the primary contact on each account. If you are experiencing any issues at this time, please contact our 24/7/365 Network Operation Center.

At 8:51am Complex Drive’s network began experiencing significant routing and BGP issues. The problems continued until 10:10am, during which time customers experienced a variety of connectivity problems ranging from DNS to direct Internet connectivity issues.

We apologize for the outage and any inconvenience it may have caused. A full scale inquiry is being conducted and further details will be provided within the next 2 hours.

If you are still experiencing connectivity issues, please call our 24/7/365 Support Center or open a trouble ticket.

The following information is data that has been compiled and it being provided as our 2 hour post incident update. Complex Drive management and networking teams are working to evaluate this mornings connectivity issues in an effort to adequately explain and prevent further occurrences. Below is our 2-hour post incident update. The information provided is based on “first response” review of the situation. Another update will be provided before close of business today with more a more in-depth and definitive review of the situation.

*****************************************

Broadcast storm of undetermined origin caused link flapping which in turn caused HSRP and spanning tree failures. The broadcast storm apparently began in C2 data center, disrupting traffic on key corporate vlans as well as hosted servers. The C2 core router’s CPU became overloaded and inter-data center links were non-responsive, causing STP recalculations and HSRP failures. Key corporate infrastructure became inaccessible as multiple routers attempted to take over (or relinquish) gateway IPs as spanning tree calculated switching paths appeared and disappeared. The C2 router shares switching infrastructure with the C3 core and the initial state of the data center interconnections had most traffic passing through the C5 data center. The broadcast storm cascaded through both the primary and backup C5 distribution networks, leaving access switches with no egress. The broadcast storm propagated through the shared switching infrastructure of the C3 data center facility. Both primary and redundant customer colocation access routers were affected and the storm propagated to the customer access switches. As a result, many customer access devices (in the colocation cabinets) were left in a non-functioning state and required a reboot to restore services. Cisco engineers are on site to determine the root cause of the issue. In the interim we have taken the steps to deploy additional equipment and to remove certain HSRP and redundant switch paths to reduce the severity of link flapping in 100% resolution is proven.

*****************************************

As stated above, all resources are focused on this critical situation and Complex Drive is leveraging other business relationships to have third party visibility and oversight into the issue resolution process. We thank you for your continued patronage as we work to provide 100% resolution to this issue.

At approximately 11:30am today, one of Complex Drive’s core routers began operating in a degraded mode, resulting in HSRP issues within specific corporate VLANS. Some customers experienced caching DNS issues, intermittent control panel access, and other minor issues with internal resources such as bandwidth graphing servers were reported. After the router was reset, Complex Drive’s customer network was restored to full functionality.

The routing advertisements caused by the router reset uncovered a routing issue in the Cox subscriber network. This was immediately escalated to the Cox engineering team and when they could not resolve the issue, the Complex Drive network operations team made the decision to close down the Cox transit connection. This action resolved all Cox subscriber access issues.

At this time the network is fully operational, with the Cox circuit disconnected until the connectivity issue is resolved. A window will be scheduled with notice to customers in order to bring the Cox connection back online.

We will provide further information regarding this incident as soon as it is available. Any customers who have questions or concerns should contact our customer service department to open a work order.

We apologize for any inconvenience.