Network Status
Alerts for 2007
Network Maintenance Window - Standard Window
During the maintenance window, electrical power for colocation customers connected to B and B1 power circuits will be shifted to a redundant power source while upgrades are made to allow for new air conditioning units to be placed on generator protected power.
Complex Drive network, management, and NOC staff will be present to manage this event. If you have any questions, please contact the Network Operation Center, or open a case through your online account manager.
Network Maintenance Window - Standard Window
Complex Drive’s Network Engineering team will be performing basic maintenance on our Core routing infrastructure. This work will require only 10 – 15 minutes. At the completion of the window, BGP tables will need to be recalculated. During this time, customer may experience momentary slowness of IP connectivity.
Brush Fire Update
As of 4:45pm this afternoon, Complex Drive is still under no physical threat from the San Diego wildfires. Although San Diego officials report that the SDGE power grid is under great strain, Kearny Mesa’s power supply has been stable for the past 48 hours with the exception of very minor fluctuations in voltage, that are immediately filtered out by our UPS systems. With the physical threat of fire all but completely gone and the power situation under control, the current concern surrounds our bandwidth connections. Late last night and twice during the business day today Level 3 and Cox Communications each have experienced network issues caused by the wildfire. These issues occur out on the public Internet and affect anyone with traffic that traverses the troubled areas of the respective networks. As soon as these issues are identified, Complex Drive network engineers take one of two actions. Routing is either adjusted to avoid the affected areas or the upstream connection to the affected carrier is cut altogether. We are keeping a close eye on the health of all of our upstream network providers and are doing everything in our power to ensure that our customers traffic flows freely to the intended destination.
Further updates will be made within 24 hours. Should you have any questions or concerns, please feel free to open a case or contact our customer services departments directly.
———————————————————————————————-
FIRE UPDATE, October 23rd, 2007 – 10:30am This morning the situation at our Kearny Mesa campus as it pertains to the San Diego wildfires is very much the same as it was yesterday. We are not in the line of fire and do not anticipate any threat of fire within the next 24 hours. With the weather changes occurring slowly (for the better), it is more than likely that Keary Mesa will not be threatened in any way. Even without the threat of fire, the SDGE power grid is a concern. Yesterday, Kearny Mesa lost power from 2:08pm and 2:23pm between Ruffin Road and the 163. All of Complex Drive’s 5 datacenters were transferred onto our emergency power systems without incident. Last night at 2:03am there was another brief power incident that was handled completely by the UPS systems. Fuel trucks arrived yesterday within 32 minutes of our call and re-filled all fuel tanks. Complex Drive can operate on our current fuel capacity for more than 24 hours, and additional fuel trucks are standing by should the need arise. Complex Drive has supported multiple “emergency move in” customers since yesterday, for San Diego companies that have decided that Kearny Mesa, and the protection of the colocation facilities is the place they want to have their critical business infrastructure hosted from now on.
We continue to keep everyone in Southern California in our thoughts. Further information will be provided every 24 hours unless more frequent updates are required.
———————————————————————————————-
This message is being provided as an update to Complex Drive customers concerned about the wildfires burning within San Diego county. As of the time of this message, 10:30am on Monday, October 22nd, the Complex Drive facility is operating at 100% and is in little to no danger.
Senior Management staff have been in communication since early this weekend when the fires began and our staffing schedules have been adjusted to compensate for individuals whose homes and families are being threatened by the fires.
Complex Drive is fortunate to be located in a central area of Kearny Mesa which is unlikely to be affected by wildfires of any kind. We have large parking lots free of vegetative materials surrounding our buildings, and our facilities teams have been clearing the roof areas of all debris throughout the weekend and today.
Our generators are fully fueled and our redundant UPS systems are fully operational. These systems stand ready to provide emergency power should SDGE have problems related to the fire or otherwise.
Rest assured, the team at Complex Drive is keeping a close eye on the situation to ensure that your equipment and connectivity are safe and available.
Our thoughts are with those whose homes are threatened. Should our staff be able to provide any assistance during this time, please don’t hesitate to ask. We have already helped a number of customers place emergency messages on their websites, and we would be more than happy to help anyone else who requires assistance.
Best Regards,
The Team at Complex Drive
Data Center C3: New Power Circuits
During this maintenance window we will be migrating some of our switching infrastructure to the new redundant power system. As switches are repowered onto the new circuits, customers may see an interruption of connectivity ranging from 30 seconds to several minutes.
SUCCESS
"A Circuit" power has been restored. The maintenance window has been concluded successfully.
Purpose
To migrate all Panel A electrical circuits throughout the datacenter onto a new redundant electrical system. By doing so, Complex Drive will be adding unmatched power redundancy to our facilities. Customers who have both A & B circuits assigned to them will now be supported by 2 completely independent electrical systems.
Details
On September 14th / 15th, Complex Drive will be performing major modifications to our electrical power infrastructure in support of the new datacenter expansion and facility upgrade. The window will occur during our standard maintenance timeframe. The window will begin at 11:59pm Friday September 14th and will end on Saturday morning, September 15th, at 1:59am.
During this maintenance window, Complex Drive engineering and facility teams will be disconnecting the main electrical feeds that provide UPS protected electricity to nearly all power circuits fed from the “A” side of the facilities internal distribution system. (A1 – A7) Note: A8 circuits are already protected by their own UPS and are not affected at this time. Once disconnected, these main feeds will be moved and reconnected to the new electrical grid that is supported by the new UPS, new ATS, new Generator, and new SDGE circuit. This migration is scheduled to take 30 minutes or less.
Customers connected to circuits designated with UPSA1 – UPSA7 will need to power down equipment connected to these circuits during the “Service Impacting Window” timeframe. Customers have been asked to power down equipment prior to 12:15am on September 15th. Power will be restored by 12:45am or earlier (30 minute or less window) and notification of the successful move will be provided via onsite personnel, via case updates, and via the Complex Drive website.
Due to the important nature of this maintenance window, we are requesting that customers provide feedback, instructions, etc through the case systems.
Network Maintenance Window
On Friday, July 27th, Complex Drive’s Networking team will be performing a number of upgrades to our core routing infrastructure. These upgrades are being performed to meet the ever-growing demands of our production network and in anticipation of a new AT&T OC-48 (2.4 Gbps) upstream connection.
The window is scheduled to begin at 11:59pm PST and will last until 1:30am PST. During the window, customers may experience intermittent periods of latency and brief loss of connectivity as routing tables are adjusted and upgrades are performed.
We thank you for your patience as this work is completed.
Level3 IP Service Outage
At 8:21am this morning one of the primary upstream carriers (Level3) experienced a complete 100% outage of IP service to our location. Traffic has been re-routed to alternate carriers are we are currently working to re-establish this connection. We will provide further updates as soon as they are available.
UPDATE: Level3 has informed us that they have Switching Fabric Card failure locally that is affecting a number of customers. Normal repair time is 2 hours for this type of outage. We are contacting them every 15 minutes to receive status updates and expect to have the problem resolved within the next 1 – 2 hours. Thank you for your patience. If you have further questions or concerns, please open a case through your customer portal or call our customer service department.
RESOLVED: As of 11:27am, Level3 has repaired the damaged portion of this network and service has been restored. Complex Drive Network Status has been restored to 100%.
Network Issues related to DDoS Attack
At 10:30am on Monday July 16th, Complex Drive began taking the final steps to return network operations to a normal status. One of these final pieces involved removing “null-route” blocks with our upstream carriers. Typically, these “null-routes” are left in place for 24 hours following a DDoS attack. In this case, these blocks has been in place for over 48 hours. Complex Drive’s / Complex Drive’s network service team made the decision to remove these blocks one at a time. Contact was made with our first upstream carrier and the IP blocks were removed. Around 11:42am, we began to see signs that the attacker was still mounting an assault. This caused additional latency on the network for periods of 10 – 15 minutes for some customers depending on location.
As of this moment, the IP blocks are still in place and will remain that way until additional security precautions can be taken. We apologize for any inconvenience this may have caused.
Massive DDoS Attack
On Saturday afternoon July 14th, 2007 a customer located within the Complex Drive network was targeted by a massive DDoS (Distributed Denial of Service) attack. This attack reached magnitudes of 2Gbps at times and was carried out in a very sophisticated manor. This was one of the largest attacks seen in our company history.
The Complex Drive networking team was onsite within minutes of the initial report. The first priority was to determine the target of the attack. Once this was done, the target was taken offline and security measures were implemented to try and stop the spread of the attack. After all steps had been taken locally to stem the attack, the Networking team contacted Complex Drive’s upstream carriers and enlisted their assistance to “black-hole” the offending IP addresses from which the attack was believed to have been originated.
Even with these measures in place the attack still caused issues on the network. Complex Drive engineers adjusted traffic routes and even removed entire upstream paths in an effort to stem the residual effects of the attack.
In the early morning hours of July 15th, the attack was considered under control but lingering effects could still be felt depending on where specific internet traffic was destined. Complex Drive network engineers continue their work today to resolve all remaining issues and to return network performance to 100%. Minor BGP recalculations can be expected during the daytime hours Sunday as upstream connection and routes are brought back online.
For those that have experienced them, DDoS attacks can be one of the most difficult networking issues to deal with. Complex Drive has been making modifications to our network to allow us the ability to deal with these issue more effectively for the past 60 days. These changes include:
1. Deploying our dual Proventia G2000 series IDS/IPS devices to cover a larger portion of our network. When complete, we will be able to provide more protection and enhanced defense against future DDoS attacks.
2. Only 10 days ago we purchased an additional OC48 of traffic from AT&T. This connection will be live within the next 60 days and will add even greater connectivity to our current BGP blend.
3. We are also upgrading other upstream connections to OC48 size so that we are able to handle any eventuality that may arise in the future.
With these changes in place, Complex Drive’s network will be more robust than ever before.
We appreciate your patience as we worked through this issue. Should you still have any concerns or issues, please open a case with specific details and our on-site networking team will attend to your requests.
Required Power Maintenance Window
SDGE, Ridgeway Electric, and Complex Drive Staff electricians will be performing maintenance and upgrades on Complex Drive’s electrical power infrastructure.
This maintenance window will last between 10:00pm Friday March 30th and 2:15am Saturday March 31st. During this window, main power for the facility will be switched onto our Generators in order to provide un-interrupted service. Upon completion of the window, power will be returned to our standard UPS + Generator protected grid.
Emergency SDGE Power Issue - C3 & C5 Datacenters
At 13:15 this afternoon a large power switch feeding five electrical transformers to and around our business park failed due to a short on one phase of SDGE’s 2000V main lines. Two (2) of our five (5) data centers were transferred automatically to alternate power flawlessly, permitting uninterrupted supply to these data centers. The two data centers affected are C5 Server Center and C3 Colo Center. Both are operating in normal range and fully staffed to handle any additional or further issue escalation.
The San Diego SDGE “SWAT” team arrived onsite at 14:00 and are currently attempting to isolate the issue. While at this time SDGE does not know the root cause, they believe it could be anywhere from 1 – 3 hours to resolve. As we receive word on exact timelines, we will keep you posted.
Complex Drive’s entire senior management staff is onsite managing this event. Network Operations teams are onsite monitoring connectivity and internal systems. Facilities engineers are monitoring power systems, and Customer Support Centers are ready and available to take on any customer needs.
UPDATE:
As of 17:30, power has been restored to the C2 and C5 dedicated hosting facilities. The issue has been isolated by SDGE to be a faulty ground wire connecting the main power circuits to this business parks distribution equipment.
During this outage, priority fueling contracts were utilized to ensure that fuel was in ready supply. Over 800 gallons of fuel arrived on two separate transport vehicles within 50 minutes of our initial call.
Power redundancy systems all worked flawlessly and ZERO customer downtime was experienced for our dedicated hosting customers!
SDGE is still replacing equipment that provides service to C3, our Colocation facility. Our teams continue to monitor all critical systems to ensure uninterrupted operation until all systems are returned to normal working order.
At this time, all Complex Drive dedicated server facilities are back online and operating at 100%.
Level3 OC-192 Emergency Maintenance
Level3 did emergency maintenance (Tkt# 1798870) on the OC-192 SONET Link provides a portion of our IP transit. The circuit was generating critical errors and the fibers were literally removed and cleaned.
Since this event occurred in the middle of the Level3 network (and not adjacent to our network), the BGP routing session was not affected and the Level3 router was still advertising its routes to our routers. This meant that sites using routes from our other transit providers were operating normally, while many of those routing through the Level3 network were not accessible.
This effect was compounded by the fact that other networks also prefer Level3 to reach our network; their BGP routers were telling them that the Level3 network was intact.
The only way to force failover was to cut our connection to Level3, then re-route traffic over our remaining transit links.
This issue appears to be completely resolved and Level3 does not anticipate further maintenance being required on this circuit.
C3 Datacenter Expansion & Upgrade
We are pleased to announce that on October 26th, 2006, construction began on the expansion and upgrade of the C3 datacenter (our Colocation facility). This expansion and upgrade will entail 2+ months of work as our existing raised floor, gas based fire suppression protected colocation facility is expanded to house more than 100 additional customer cabinets. In addition to the additional customer space – brand new, completely independent power infrastructure is being installed to support new and existing customer requirements.
A unique approach is being taken with this expansion and upgrade. Instead of simply increasing capacity to existing power infrastructure, a completely new UPS system, Generator, and electrical power grid are being installed to support the additional space. In doing so, the C3 facility will be one of only a hand full of datacenters throughout the U.S. capable of offering true, 100% power system redundancy to our customers. Existing C3 customers will be able to purchase power feeds from both the existing power grid, and from the new completely independent power system and source, and new customers will be able to elect the same reassuring levels of power redundancy.
Further updates will be made available when available and as required by our construction schedule. Should you have any questions or if you have inquiries about the expansion or additional power redundancy needs, please feel free to contact Chris Orlando, VP Sales & Marketing at 888.221.5902.
