Network Status
Alerts for 2006
Upstream Routing Issues
At 18:15 this evening 1 of our primary upstream providers appeared to be experiencing routing issues within their network. The issue affected a small number of customers on our network, while others remained unaffected. We are working with our upstream to identify the issue. As of this moment, we have disconnected this provider until we can be guaranteed the problem will not resurface. More information will be provided on this page as soon as it is available.
C3, C5 Datacenter network upgrades
On Friday, September 15 at 11:00pm, the Systems and Networking division will be installing an additional router in the C3 datacenter. This router is being installed to add redundancy for the colocation facility. In addition, core routing equipment in the C5 datacenter will be replaced with a new chassis and dual routing engines.
Datacenters C3, C5: Router Upgrade
The Systems and Networking Division will be upgrading two core network routers on Friday, September 1st at 23:00 PDT. New redundant routing modules will be installed in core routers in the C3 and C5 data centers.
iBizPanel Redundant Systems Test
As part of our ongoing effort to ensure the highest level of service for our customers, we will be performing a series of tests on the iBizPanel. These tests will ensure that newly installed iBizPanel redundant systems are fully functional. We will begin these tests tomorrow (8/30/06) at 12:00am EST with a controlled failover to the new redundant systems. We expect no down time during this process. Being that a seamless cutover is the primary objective for this test we will be monitoring the system very closely. If problems do arise we will restore normal operations on the primary systems within minutes. After a complete 24 hour test of the secondary systems, we will return services to the primary systems the following evening (8/31/06) at 12:00am EST.
Thank you for your continued patronage.
Datacenter : C2, C3, C4, C5 - Emergency Power Transformer Replacement - Emergency Maintenance Window
***THIS WORK HAS BEEN COMPLETED AS OF 9:28pm ALL SERVICES ARE AT 100% ***
SDGE has reported a problem with 2 of the primary electrical transformers that feed power into the industrical business park where our datacenters are located. This problem is of a critical nature, and SDGE has insisted that these primary transformers be replaced immediately. SDGE will be onsite from 5pm until 12am this evening to perform these emergency replacements.
During this maintenance window, we will be running C3 (our carrier grade Colocation facility located at 8913 Complex Drive) on generator power to ensure continuous operations. Shared Hosting and Dedicated Server customers in C2, C3, and C4 are being notified as a precaution, since the transformers being replaced feed power to the entire business park.
While we do not expect any downtime whatsoever during this emergency maintenance window, our facilities will be fully staffed during this window, and senior level personnel will be onsite 100% of the time managing this event.
SDGE will be onsite at 5pm. They expect to start work at 5:30pm and expect to complete their work by 11:30pm. We will info you as soon as the work has been completed by updating the network-status page on our website, and by sending a follow up email message.
Datacenters C2, C3, C5: Link Capacity being added
Network capacity is being added between the C2, C3, and C5 datacenter facilities to increase inter-facility redundancy to our transit IP carriers. Additional Gigabit links will be installed by the Network Engineering Division on the evening of 8/18/2006 at 23:00 PDT (11:00pm).
Datacenter C1,C2,C3,C4,C5 : Link Capacity being added
Bandwidth usage between all 5 datacenter facilities has increased tremendously over recent months. On the evenings of 8/1/2006, at 9:30pm, network engineers will be increasing inter-facility capacity between all 5 datacenters. Additional Gigabit links will be added to improve throughput between facilities and to our primary IP carriers located throughout the different facilities.
Network Infrastructure Upgrade Advisory
Complex Drive will be restructuring and upgrading the three primary data center networks (C2, C3, and C5) to increase fault tolerance and overall network performance. This action includes redistribution of core support services such as DNS and business automation systems. Although our facilities currently have redundant UPS and emergency power generators in place, we are taking these additional measures to minimize the risk of a single power grid outage disrupting service across multiple data centers.
The restructuring project will require several non-standard maintenance windows since we will be adding and moving fiber optic links between our data centers and transit (backbone) carriers. Every effort will be made to minimize the impact of these actions on network services, but some customers may experience very brief intervals of interrupted or re-routed traffic as new links and network equipment are brought online. Whenever possible, we will perform these upgrades outside of standard working hours.
During the course of normal network operations there are small network reconfigurations, such as replacement of network equipment or the reconfiguration of core network equipment to optimize traffic flow. These changes are usually undetectable and transient in nature. Due to the larger scale of this infrastructure reconfiguration, there will be an increased chance that it could have a noticeable ñ but transient ñ impact on network services.
Here is a summary list of planned (and in some cases, completed) activities that will be completed for the network upgrade:
- Installation of core routing and switching capacity in C5 data center. (Completed)
- Installation of fiber capacity between C2 and C5 data centers. (Completed)
- Installation of additional fiber capacity between C2 and C3 data centers. (Completed)
- Upgrade of colocation network and switching equipment in C3 facility. (Completed)
- Relocation of transit carrier (Cogent) from our C1 data center to our C2 data center. This will occur at 11:00pm Monday, July 10th.
- Reconfiguration of C2 distribution switching network to support additional core router and network segmentation. (Ongoing, various times)
- Transfer of transit service (Level3) from our C1 data center to our newest data center (C5). This move will take place during our standard maintenance window on Wednesday, July 12th, at 11:59pm.
Data Center C3: Scheduled Network Upgrade
Complex Drive will be installing core network and switching capability in the C3 colocation Data Center.
Data Centers C2 and C3: Fiber Upgrade
Complex Drive will be installing fiber optic capacity between Data Centers C2 and C3.
