Why Hotter Networks Need A Cooler Core
- Traffic demand seems insatiable, but power and space are in limited supply
- How can core routers grow capacity in the existing resource footprint?
- What are the best practices to optimize power and cooling efficiencies?
Core router cooling and power management are perennial hot topics. And with broadband traffic projections showing continued exponential growth, there’s no relief in the forecast.
While reducing both environmental impact and operating cost are always top of mind, network operators lose sleep over the question “How can we address the virtually insatiable demand for more network capacity, without exceeding the practical limits of space, power and cooling resources available in the NOC?”
Traffic growth sets off a vicious cycle in the Network Operation Center (NOC): higher power consumption to run the additional equipment, more heat generated by the additional equipment, more cooling needed to stay in optimal operating temperature range, and higher power consumption to run the cooling elements.
With steep costs, expanding an existing power and space footprint is the least preferred option to accommodate growing router capacity.
Core router capacity and power efficiency
Core router capacity has been keeping pace roughly with Internet traffic growth. But as scalability and density are pushed ever further, power, cooling and space efficiency have become critical design constraints. Unfortunately, power efficiency gains are falling behind: the power density limit for a single rack currently is around 20 KW per rack. At this rate there will be a significant shortfall in single rack switching capacity in the years ahead (Figure 1).
With higher gains in power efficiency and cooling, enabling higher port densities per rack, service providers are able to scale their core network capacity using significantly less footprint. The end result is significantly reduced operational costs.
There’s good news: with skillful design those efficiency gains are both possible and practical. High performance core routers like the Alcatel-Lucent 7950 XRS are leading the way with a comprehensive approach to designing, managing and conserving power.
Designed for power efficiency
Today, the Alcatel-Lucent 7950 XRS sets the bar high with a benchmark power consumption of only 1 Watt per Gb/s switching capacity (16 Tb/s at 16KW nominal draw). This achievement is made possible by a state-of-the-art design that delivers performance and efficiency without compromising either.
- Faster and more power-efficient silicon, and correspondingly fewer components
- Optimized frequencies and voltages for maximum speed with minimum power consumption
- Intelligent gated power management in the FP3 network processor
- Improved DC to DC power converter technologies
- Line cards powered only when in use
- Optimized load sharing and filtering of power entry modules
Start at the silicon level
Smart power management and cooling starts at the silicon level.
Faster silicon has a range of advantages. For example, on the 7950 XRS, a 400Gb/s forwarding capacity offers a perfect geometry for 10, 40 and 100G port densities, leaving no switching capacity stranded. Its higher performance allows it to more effectively share memory pools for routing and forwarding information than routers using slower silicon. And with fewer components, power consumption is lower and there are fewer potential points of failure.
In an optimized design, the chip set that powers the core router contains built-in logic that will shut-down unused functionalities in order to save power. The power entry modules, as well as system components like line cards that consume power, contain on-board intelligence. This allows for active monitoring and management of both power usage and cooling to maintain safe and energy efficient operating conditions.
Power bus architecture
Some router designs divide a system into different zones with dedicated power supplies for each. An unfortunate drawback is that each zone needs to be protected separately or risk losing the entire router should a critical zone go down.
Instead, a core router that uses a power bus architecture allows power resources to be shared in the most economical manner and improves overall system reliability.
A design based on a single common internal bus allows available power (including spares in an N+M model) to be shared among all system components. Power entry modules with on-board intelligence can keep track of power ratings of individual system components. They can communicate among each other to track the total available power, and potentially power down less critical components in the rare event a system becomes underpowered.
As the core router system grows to keep pace with demand, a bus architecture makes it possible to add power feeds and components simply by plugging in additional power entry modules. And if a power entry module fails, it can be replaced safely without a need to touch or modify any power cables. In the rare instance that multiple power supplies fail, critical systems such as Control Processor Modules, cooling fans and Switching Fabric Modules are protected.
Thermal design for optimal heat dissipation
Thermal design is a critical aspect of any carrier class routing platform, yet it is often taken for granted. That’s an oversight network operators can ill afford.
Heat dissipation determines maximum system capacity and port density. Systems must be designed from the ground up and with a view to where they will be deployed to provide consistent and reliable cooling across all components.
What are the essential thermal design elements to look for?
- Chassis height and slot width designed specifically to create space for air movement
- Air intake area optimized for maximum airflow
- Air guides to distribute air evenly front to back and side to side
- Impedance panels to ensure equal air flow for every slot
- Advanced fan technology to assure quiet and efficient cooling
- “Pull” air flow design to draw air up through all chassis components
- Intake cooling air pulled from the cool aisle side of the chassis
Cool aisle/hot aisle separation
The distinct layout of “cool” and “hot” aisles within the NOC presents an obvious opportunity for optimal thermal design. The cool aisle is used to provide cool air at the cooling inlets of the system, while the hot aisle allows for the hot exhaust to dissipate without impacting the cool aisle air of surrounding equipment. Properly designed routing equipment will maintain a clear separation of the cool and hot aisles; taking in air only from the cool aisle and exhausting hot air only to the hot aisle.
Core routers can be divided in one or more cooling zones, but having more than one cooling zone makes for unnecessary complexity. More fans are required to cool each zone, and there are more air filters to be serviced. Even a full rack core router can be managed with only one cooling zone if designed well.
Pull airflow for effective cooling
A “pull” airflow design is essential for effective and consistent cooling throughout a core router chassis. The system airflow creates a vacuum throughout the system, assuring that all heat is properly dissipated and no hot spots occur. One way to achieve this is with a very large intake plenum with separate columns front to back and side to side. This allows separate and even columns of air to form across all components, with no turbulence. With air guides to help evenly distribute cool air over all system components, efficient cooling can be optimized even further.
The alternative airflow “push” design is one to avoid. Rarely found in high performance telecommunications equipment, it requires an additional large plenum area between the pushing fans and the line cards. The additional plenum allows air streams from the various fans to mix into a single air stream. Without pre-mixing, fan-created turbulence will enter the card cage area causing hot spots, air damming around components, recirculation, and unwanted air currents.
Efficient fan design with active monitoring
A best practice design provides a 1+1 redundant pair of fan trays and, for each of the individual fans, a dedicated controller programmable in granular speed increments. With active control of fan speed, the design can accommodate the different thermal characteristics of individual system components, as well as changing environmental conditions that arise when the system configuration changes. When operational circumstances permit fans to run slower, both power consumption and noise levels are reduced.
Fan operation, speed and power consumption can be monitored through a smart management system, with rules defined to indicate over-heating conditions, and to activate back-ups to avoid air leaks that would break the vacuum.
In NOCs with front to top cooling, further efficiency improvements can be designed in with a vertical exhaust that exits air through the top of the chassis. The top plenum expands the available air space and lowers impedance.
Smart management of power and cooling
Smart power and cooling management is indispensable in next generation core routers. Done right, it allows the network operator to:
- Monitor and assure the supply of sufficient power to operate all system components
- Avoid energy waste by powering down unused system components or circuitry
- Prevent brownouts and mitigate the impact of reduced power availability on system operation
- Assure the system is operating safely at an optimal temperature at all times
- Reduce noise generation
- Minimize operational expense