Network Congestion And Productivity
The Bulletproof Network
Filtrona Extrusion implemented a network redesign to guarantee uninterrupted business processes at sites across the United States and Mexico.
"Outsourcing seems like a sexy alternative for a lot of problems, but turning the management of your mission-critical infrastructure over to a third party is pretty scary," says Jeff White, corporate director of IT for Filtrona Extrusion. When Jeff White joined Filtrona Extrusion as corporate IT director, the company was operating a single threaded network with mixed traffic mission-critical and non-mission critical traffic running side-by-side, with no way to separate or prioritize the two. In addition, the company experienced network performance problems and increasing amounts of downtime at its facility in Mexico. Because its manufacturing facilities run 24/7 and do not always have on-site technical resources available when there is a problem, the company needed a better solution.
"You just never know when things are going to happen, from a backhoe digging up a fiber cable or a more significant outage, and these situations were shutting down an entire facility," says White. "Because the network is critical to all areas of our business, we needed to implement a fully redundant network to reconcile problems automatically and ensure we could remain up and running, no matter what the situation."
Filtrona Extrusion is one of the largest manufacturers of extruded plastic profiles, sheets and specialty tubes in the United States. The company manufactures more than 40,000 different products for medical, merchandising, aviation, transportation, traffic, lighting, fencing and custom plastic industries. Its products range from catheter tubes, traffic posts and cones, plastic sheeting for fluorescent lights, air-conditioning ducts in commercial jets, and outdoor furniture.
All of Filtrona Extrusion's business and manufacturing processes rely heavily on its network. The company's most critical business application is an enterprise resource-planning (ERP) system that runs on a centralized server in the corporate headquarters in Atlanta, with a standby server replicated in real time and located at a manufacturing facility in Tacoma, Wash. The ERP system manages sales, manufacturing, purchasing, shipping and receiving, accounting, and quality information for all locations, including additional manufacturing facilities in Illinois, Massachusetts, Pennsylvania, South Carolina and Monterrey, Mexico.
White uses a custom simple network-management protocol system to pull data off the routers directly in order to monitor the circuits and overall network performance.
Since plastic is a commodity product, Filtrona Extrusion must maintain complete visibility of its inventory at all times to ensure purchasing decisions can be made at the most advantageous price. Losing sight of inventory levels for as little as a few hours can prevent the company from taking advantage of spot-buy opportunities or force it to use a more-expensive, higher-grade of plastic than is required to fulfill an order for a customer. Moreover, if the ERP system is down, the company is unable to record quality data during the manufacturing process. Without the quality data, there is no way to certify that the product meets the requirements specified by the customer, and without the certifications, the product cannot ship.
"Our whole planning system depends on inventory," says White. "Without it, we don't know what materials to buy or how much to purchase for existing orders, which creates some very significant business issues that can cost the company tens of millions of dollars."
White's first step after joining the company was to evaluate the effectiveness of his outsourced providers. He needed to understand what the current providers had done, and what they could do moving forward to help him meet the company's strategic objectives.
Carrier Options Explored
"Outsourcing seems like a sexy alternative for a lot of problems, but turning the management of your mission-critical infrastructure over to a third party is pretty scary," he says. "When you depend on your network, you need to trust your provider, and in my experience, most are not as good as they advertise."
White and his IT staff began exploring their options. To create the network he envisioned, White spoke with managed services provider Virtela Communications, which was operating as Filtrona Extrusion's sole access provider. The IT staff also evaluated a host of other providers. During the evaluation process, there were many factors to consider, and while pricing was a concern, the company did not end up going with the least- expensive option.
"Because the carrier manages the routers, most of them wanted to design the network based on a template, and didn't want to work with us to engineer anything creative or non-standard," explains White. "The design we came up with was not what the other carriers were pushing. Early on, Virtela showed us they had the resources and expertise, and they were really the only provider that was willing to work with us and manage the entire infrastructure once it was complete."
For example, White wanted to implement generic routing encapsulation (GRE) tunnels instead of using the more conventional border gateway control (BGP) routing method. With BGP, routers talk to one another and share what routes are available. In theory, if a router goes down, it stops advertising its availability and the other routers adjust and re-route the traffic. There are times, however, when a circuit appears to be up, running and available, but no traffic can pass over it due to errors, latency, fragmentation or other issues.
"I wasn't willing to just accept any design. I needed an absolute bulletproof way to know that when I send that critical traffic out over the network, it's going to get to the other end," White states. "The only way to do that is establish a tunnel from one end to the other, and GRE tunnels solve this problem."
Because it was a completely new network design, Filtrona Extrusion's IT group worked closely with Virtela's engineers to design a "dual-carrier" platform. Virtela did most of the work on the pre-engineering phase and dedicated an engineer to work directly with the IT group over the course of a few months. Together, they designed a managed virtual private network (VPN) with redundant GRE tunnels that used open shortest path first routing between critical hub sites to ensure the automatic re-routing of traffic over the alternate provider link in the event of network congestion or failure on the primary path.
The design also ensured the network would automatically load-balance traffic between the two networks, with the Virtela managed network carrying all mission-critical data traffic, such as ERP, and the secondary provider carrying non-critical traffic such as voice over IP, videoconferencing, e-mail and Internet.
To accommodate the new design, Filtrona Extrusion had to make some infrastructure changes during the planning phase. Virtela began by installing some new circuits and worked with the secondary provider to get their circuits in place. Once all the circuits were in, the partners began a rigorous month-long testing phase, turning up one circuit at a time, site by site, to test the resiliency of the network design.
"Because we had a dedicated Virtela engineer to work with us, the process went very smoothly," says White. "Other than the few seconds we needed to do the final swap, and working out a few minor issues along the way, there were no problems."
As they deployed the network, some minor adjustments were required. One of the unforeseen challenges was that a single router was being used to route traffic over two different circuits. At the remote sites, outbound non-mission-critical data used the secondary network circuit as designed, but when inbound, non-mission-critical data routed back over the primary circuit intended for mission-critical traffic only, it created issues.
Challenges Are Addressed
To solve this problem, the manufacturer used source network address translation over the primary circuit and changed the default route to the secondary circuit. This ensured that traffic that went out from the remote sites to the host location returned over the same circuit.
Filtrona Extrusion also ran into a minor challenge with its videoconferencing application, which had stopped working with the new network. "Because the new design required that packets be sent through the GRE tunnels, we were losing visibility of the diffserv code-point flag on the packet, which tells the MPLS (multiprotocol label switching) network what priority the packet has," explains White. "We weren't sure how to solve the problem, so we talked about taking the video traffic outside of the tunnels and just dropping it directly into the MPLS network, which was less than ideal, but we were going to do it."
Virtela came up with a different solution. Cisco had recently released a service-level specification router update that featured the ability to translate the diffserv flag from the internal packet to the external shell of the tunnel packet, thus solving the problem. Making some minor adjustments to the router configuration allowed Virtela to keep videoconferencing within the tunnel. In addition, if Filtrona Extrusion's non-mission-critical network link went down, the videoconferencing traffic would automatically failover to the primary network.
Virtela's managed VPN service links all eight sites in the United States and Mexico. Virtela manages two of Filtrona Extrusion"s Cisco routers located in Atlanta and Tacoma, and also owns and manages the routers at all other manufacturing locations. In addition, both the Virtela network circuit and the secondary provider circuit terminate in Filtrona Extrusion"s or Virtela's routers, so Virtela is now managing the entire network.
"Typically, different carriers require separate hardware and each provider manages its own, so the customer is forced to coordinate disagreements between them," offers White. "Virtela is saving us from having to manage this ourselves. In addition to managing all the equipment, they also give us access to monitor the routers, as well."
White uses a custom simple network-management protocol system to pull data off the routers directly in order to monitor the circuits and overall network performance. He can also utilize Virtela's Web-based portal, VirtelaView, to monitor network performance statistics or obtain real-time traffic data from the Virtela cloud. White gets regular, proactive notifications from Virtela's global network operations center and the routers directly about events that might be occurring on the network.
System Tested In Mexico
Filtrona Extrusion recently quadrupled the size of its facility in Mexico. During the expansion, the existing circuits were moved from one computer room to another. At the same time, the local provider decided to upgrade the facility from a traditional LAN-line-based circuit to a wireless circuit, without informing Filtrona Extrusion of its plans.
While performing the upgrade, the LAN circuit went down and it took the provider three weeks to get it operational again. White received an e-mail notification from Virtela in the middle of the night informing him of the outage, but with the new fully redundant network in place, no one in the facility ever knew the secondary network went down. In fact, the failover was completely transparent in both directions.
Three weeks later, when the local circuit was back up, the network failed back over to the local provider with no service interruptions. "We did a lot of testing during the installation phase to ensure the network was operating as it should, but the outage in Mexico was great validation. You never want something like that to happen, but when it did, it was very reassuring to see that it had no impact on the business," says White.
Before the fully redundant network was in place, Filtrona Extrusion was experiencing 85-95 percent network uptime, depending on the circuit. Now, the company is consistently experiencing 99.95 percent uptime on all circuits. In addition, because non-critical traffic now runs on another circuit, the bandwidth available for mission-critical traffic has more than doubled, and users have noticed a difference in overall performance.
Filtrona Extrusion still operates with a small IT staff that handles the network-monitoring functions at all eight manufacturing locations during the day and relies on Virtela for after-hours monitoring. "When I came on board, we didn't have this type of network, and there's no way I could have managed this by myself" says White. "We were looking at $250,000 just to hire the staff that it would have required to run the network, not to mention the millions of dollars it was costing us in materials decisions and lost productivity due to network downtime. Now, instead of worrying about network reliability issues, we can focus on more strategic matters."
About the Author
"Bill Dodds is vice president of sales and marketing for Virtela Communications Providing global network solutions. Virtela Communications is a full service managed service provider and network integrator. Virtela delivers award-winning managed network services to many of the world’s largest multinational companies. Contact Virtela for more information about enterprise network pandemic planning .
"