Methodology

The project was designed to take place in four distinct phases. This allowed our team to break the project into smaller, more manageable pieces and ensure everything was on schedule.

Phase I - Document and Research

During this first phase of the project, the primary objective was to properly document and research the scope of the project. We began this phase with an on-site visit to United at its Dodge City, KS headquarters. This on-site visit allowed us to familiarize ourselves with United’s equipment and infrastructure. We found that United had a single rack, mounted full of equipment for both its blue and white networks. United splits its subscriber base into two separate networks in order to efficiently distribute the number of users on each network. However, all of the equipment was in the same rack, causing some confusion over which devices operated on what network. Further, the rack contained several devices that were not operational or plugged in and many Ethernet patch cables that were connected on one end, but not the other.

As part of our on-site visit, we made a list of all equipment contained in the rack or connected to devices in the rack. We took note of each piece of equipment’s data connectivity ports its medium. This allowed us to determine the quantity of cables needed to be run in the new racks. We also took note of what type of keyboard, video, and mouse (KVM) connectors each device required, in order to make sure the necessary adapters were purchased for the KVM switch. Finally, we took note of the physical dimensions of each device in order to determine where they would best fit in the new racks.

Once we obtained this information, we organized it into a spreadsheet and then began researching solutions to meet United’s needs. We assigned each group member a component of the project to research and provide a solution for. This delegation proved to be essential when we realized our time constraints and the scale of the project. Assigning tasks allowed for individual pieces of the puzzle to be put together at a time that was convenient for all of us.

Our first solution was to purchase new switches to replace the existing ones United already had in use for the core network. From our initial on-site visit we determined that the new switches would need both fiber optic and copper connectivity. United’s existing core switches were all Nortel Baystack 450 series with 24 10/100 ports and two 100 Mb fiber ports. United also needed a solution that would be easily managed and highly scalable. From this information, and our previous experience, we recommended the purchase of four Cisco Catalyst 3750G 24-port switches to replace United’s existing Baystack switches. This Cisco Catalyst switch provides 24 10/100/1000 ports as well as four Small Form-factor Pluggable (SFP) slots. We also recommended the purchase of six 1000Base-SX SFP adapter modules, which provided the ability to connect devices to the switch via multi-mode fiber optic cables.

Our second solution was to provide a cable management system. Cable management was a problem for United in its existing rack and we wanted to ensure the new racks would have adequate cable management. From our research we found several great options for cable management, but most of them either didn’t fit into the rack properly, or were beyond the scope of our budget. After much research, we finally settled on a solution that utilized existing cable management in the new racks as well as custom-made cable management pieces. Our solution included twenty, seven foot sections of Black Box cable management raceway, Velcro cable ties, and plastic tie wraps. This combination met both our needs and our budgetary restrictions.

The third solution was to establish a cable labeling system. A significant amount of time was dedicated to tracing cables during the first on-site visit, due to a lack of proper documentation. By developing a labeling system a technician can quickly locate a cable and work with it immediately rather than having to spend time tracing it. After discussion with United it was determined that we needed labels that would adhere to the cables properly. Based on this information we recommended a Brady IDXPERT labeling printer with extra strength adhesive vinyl labels.

The fourth solution was an IP-based KVM switch. United needed a way to be able to have remote access to all devices in the new racks. Previously, many devices had been operated with their own dedicated keyboard, monitor, and mouse, which required extra space and a physical trip to the hardware in order to work on it. After more research, we found the DSR8035 KVM switch from Avocent. This device has a built-in web server which allows a technician to access any piece of hardware attached to it from anywhere in the world. The DSR8035 switch provides thirty-two Ethernet ports to connect an Avocent KVM adapter to use any type of device. Instead of using big and bulky KVM cables, the Avocent box uses regular category 5 Ethernet cables to provide connectivity. We recommended the purchase of the DSR8035 switch and twenty KVM adapters.

The total expenditures for the project came in $10,000 under budget at approximately $45,000.

Phase II - Order equipment and prepare

During the second phase of our project we focused on making preparations for the actual data center relocation. Our first task was to work with numerous vendors to make sure equipment that was ordered made it to United in time. This proved to be fairly stressful as we dealt with different vendors to ensure equipment arrived in time. Some items such as the labeling system and some of the cable management pieces were special order items that were not in stock. We worked closely with the vendor to ensure that these items would become available and ship in time for our project.

At this point, we once again split the required tasks into sections and took a separate part of the project. Jason was put in charge of determining the layout of the new rack by creating a Microsoft Visio (a program for visually creating network diagrams) diagram with the correct stencils and then fitting it all together based on power, data cabling, and heat requirements. Nolan was in charge of establishing a schedule for moving equipment in order to mitigate downtime of mission critical items. Doug was in charge of determining the current switch configurations, organizing and cleaning up the configuration, as well as configuring the new switches.

Nolan prepared a schedule based on conversations with United. We classified the servers into three different tiers of service. Servers and not equipment not currently in use or used only for monitoring were grouped into Tier 1. Equipment that could operate with a short down and minimal impact was grouped into Tier 2. This included email servers, spam blocking appliances, web servers, as well as Domain Name System (DNS). Finally, Tier 3 servers and equipment included items that affected the customers’ internet connection, such as core routers, core switches, and authentication servers.

Jason designed the layout of the new racks by creating a network diagram. We utilized some of United’s existing resources and obtained Visio symbols for each piece of equipment that we would be moving. We went through several different variations until we found a solution that allowed everything to fit properly. Jason had to take into account several different factors during this process, including length of cables, weight of equipment, and heat dissipation. The completed diagram turned out to be an essential tool during our move as it allowed us to quickly determine the exact rack position for each piece of equipment we were moving.

Doug worked to create the switch configurations for the new switches. United had some existing documentation on their current switch configurations, but it was out-dated, so existing documentation was combined from our first on-site visit with existing United documentation. The old switch configurations were examined and rewritten to work efficiently on the new Cisco switches. This proved to be a daunting task, but helped to ensure an organized, logical system. Additionally during spring break, Doug was onsite at United for a couple of days and was able to help inventory items and prepare the racks for the move. He worked with the United IT Department to organize the new equipment and test it. He also established the infrastructure requirements for the project and worked with United technicians to make sure proper fiber, copper, and power cables were run to the new racks.

Phase III - Data Center Relocation

The climax of our project came in the third phase. This is when we actually began to physically move the data center as we had been planning and preparing for in the first two phases.

We arrived at United at 8:00 AM on Friday, April 7, 2006. We began by taking inventory all of our new equipment and putting the final touches on a presentation for United’s IT Department. At 10:00 AM, we gave a presentation to the IT department covering our planned schedule for the weekend and updating them on any new changes to the rack layout. We were surprised when we learned during this presentation that they had just received new equipment the day before that needed to be installed in the rack. Although we weren’t expecting it, we were prepared with extra equipment in case we needed it and were able to quickly adapt the rack layout for the new equipment. Once the presentation was finished, we began to prepare the racks for the move. Our first step to prepare the racks was to install the cable management. Following this we installed all power strips onto the same side in order to facilitate data and power separation. Once we had power and cable management mounted, Tier 1 equipment was ready to be moved including:

Two existing ISurf appliances (used to filter web traffic)
One existing SolutionIP server (not yet in use)
One existing BMS500 PPPoE server (not yet in use)
One existing email server (not yet in use)
One existing Optivity box (used to manage Baystack switches)
One new SNMP management box (not yet in use)
One new Cisco Secure Server (not yet in use)
One new IP KVM and slide-out monitor (used to access servers)

We completed the moving of all this equipment by around 8:00 PM that evening. We then proceeded to take a break for dinner and returned around 10:00 PM to clean up and prepare to move a Tier 3 piece of equipment that night during United’s scheduled maintenance window which runs from 12:00 AM to 6:00 AM. We completed this move shortly after midnight and finished for the day.

The next morning we began at 10:00 AM with Saturday’s schedule. Saturday’s schedule was focused on moving Tier 2 equipment, which could sustain brief downtime with minimal service downtime. In order to achieve this goal, temporary patch cables were ran from the old switches to the new equipment locations. Some equipment was in a redundant configuration, allowing some equipment failover while servers were moved. This worked well for DNS servers and spam filtering appliances. Other equipment was shut down properly and then moved as quickly as possible. We tried to minimize downtime for each piece of hardware to thirty minutes, and were successful at doing so.

Equipment that was moved or installed on Saturday:

Existing secondary DNS server (used to provide name resolution)
Existing primary DNS server (used to provide name resolution)
Existing secondary spam filtering server (used to filter junk email)
Existing primary spam filtering server (used to filter junk email)
Existing web server (used to host United’s web sites)
Existing SNMP management box (used to monitor network)
Existing Sniffer appliance (used to capture analyze traffic on the network)
Existing Stormtracker appliance (used to manage DSL modems)

After moving this tier of equipment we proceeded to take a short break before returning at 11:00 PM to prepare to move all the Tier 3 equipment.

When we returned at 11:00 PM we started making final preparations to move Tier 3 equipment. It was essential that this equipment be moved as quickly as possible as it meant downtime for all subscribers until it was operational. We planned to move United’s Blue network first since it had a larger customer base and less Tier 3 equipment. At this time, we had the entire staff of the IT Department on hand to help, as well as technicians from the telephony department to assist with moving fiber, DS-3, and T1 connections. In order to better utilize resources, three groups were formed. The first group was in charge of moving the fiber optic connections; the second was in charge of installing the remaining power and data connections, as well as labeling and organizing these cables. The third was in charge of physically moving all equipment from the old racks to their new locations. By giving assigned tasks to complete, this strategy proved effective to ensure phase completion. Within several hours the Blue network was successfully migrated restoring customer connections. Although we were prepared for minor problems, the Blue network came up with little difficulty.

After relocating the Blue network, we proceeded to move the White network. The White network uses a DS-3, several T1’s, and many fiber optic connections, whereas the Blue network only used fiber optic connections. Due to the increased complexity, the White network required more time to move properly. The equipment relocation went smoothly until we transitioned to the new switches. For unknown reasons, fiber connectivity could not be established. After troubleshooting, it was eventually determined that temporary patch cables would be needed to allow connectivity through the Baystack fiber transceivers. A problem was also encountered with the switch configurations on the White network. Several different VLAN’s had been setup in the switch configuration, but for unknown reasons they hadn’t been saved properly. This issue took a significant amount of time to troubleshoot and rectify. Fortunately, by about 10:00 AM we were able to develop a temporary solution that would allow White network customers to re-establish connectivity. The initial plan was to have finished by 6:00 AM in order to comply with the maintenance window, but the complications encountered set the relocation back longer than anticipated.

Equipment that was moved or installed on Sunday:

Five existing TC fiber transceivers (provides backhauls to other towns)
Two new Cisco 3750G switches (used as core switches for Blue network)
One existing SolutionIP server (used to authenticate Blue network)
One existing Cisco 7200VXR router (used as core router for Blue network)
Two new Cisco 3750G switches (used as core switches for White network)
One existing DS-3 CSU/DSU (used as primary internet connection for United)
Several existing T1’s (used to connect schools to the Internet)
One existing BMS500 PPPoE server (used to authenticate White network)
One existing email server (used as primary email server for United)
One existing Cisco 7200VXR router (used as core router for White network)

he relocation proved to be an overall success. Due to the failure of the fiber optic connections, the Capstone group planned for a follow-up visit to resolve any remaining issues.

Phase IV - Project completion and follow-up

The goals for the fourth phase included another onsite visit to United for resolution of the fiber optic connectivity issues, and to clean and finish organizing the new racks. When we arrived back in Hays from phase three, we spent several days researching and troubleshooting problems and evaluating the project.

The outstanding issues from phase three that were resolved during phase four included:

Insufficient quantity and length of Ethernet patch cables
Cable management solution not fully implemented
IP KVM compatibility issues
Fiber optic adapter incompatibility

After researching these issues, appropriate vendors were contacted to order more cables and exchange the fiber adapters. During this phase the replacement fiber optic adapters were installed and temporary cabling was removed. The firmware on the IP KVM switch was upgraded in order to resolve the compatibility issue. To complete the project all cabling was organized and placed within the cable management system.

At this point, the relocation project was completed. All outstanding issues have been resolved to the satisfaction of United and the Capstone group.