There are a few technologies to make networking more energy efficient. But a more energy efficient Ethernet would have by far the biggest impact because of its ubiquity.
I visited Broadcom to meet with David Berry, senior marketing manager, and Wael William Diab, technical director, office of the CTO. David and I have talked about the general energy efficiency of networking equipment before. Wael is vice chair of the IEEE 802.3 Working Group
So I focused on finding out what 802.3az is all about. My core technology area started with software engineering and expanded to cover networking and embedded systems. I am by no means an expert in networking technologies but am dangerous enough to know some of the basics. I here confess that while I was summarizing my conversations with these two gentlemen, I studied and researched this subject extensively so I could keep up. If the whole thing is completely over my head, it is easy to let it go, but with some effort, I can understand the subject matter to some extent. Sigh.
Wael Diab and David Berry
The following is an edited version of my questions and their answers.
Q: My understanding is that energy efficient Ethernet (EEE) allocates networking resources when needed and turns them off when not needed. Can you give me a little bit more on EEE?
– – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – Power and Cooling for VoIP and IP Telephony Applications – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – –
A: Traditionally, the most important things about networking were speed and cost. But energy consumption is becoming more important than either of those because opex is beginning to surpass capex. The Ethernet has typically been operated as always-on, even though traffic is not always on. A typical utilization is less than 10% and sometimes even less than 1%. But when traffic is back on, it tends to hit the peak, and the Ethernet has to quickly come back on at 100% capacity. When the PHYsical layer (layer 1 on the OSI model) is on, the layers above (layer 2, data link, and up) are on as well. When there is no traffic, the PHYsical layer (PHY) can be turned off, along with the layers on top of it, creating further energy efficiency. Additionally, we have implemented a way to negotiate when to turn on layer 2. In this way, layer 2 can sleep until absolutely necessary.
Q: Is the ratification of 802.3az done? Is the product based on the standard shipping?
A: The standard is at the final stage, and ratification is expected in September. However, prestandard versions are shipping by vendors, as has happened for other networking products before.
Q: Can this technology be implemented with firmware refresh?
A: Unfortunately, to support firmware refresh, a new PHY is required and, therefore, new equipment. We have a technology called AutoGrEEEn. AutoGrEEEn technology enables a device with a non-EEE MAC to seamlessly transition to EEE capability by implementing control policy assist engines and circuitry inside the PHY device.
EEE requires control for the PHY to be done via in-band signaling over the MAC/PHY interface. This requires a change to both the PHY and MAC silicon. A number of systems have the MAC and PHY as two different pieces of silicon with the MAC often embedded in a switching or controller-type device. These MAC-containing devices have associated drivers and software, and are often multiport devices. So, a transition to EEE may be hampered by additional development that involves replacement of the MAC-containing devices. AutoGrEEEn technology eliminates the need to change the MAC/PHY interface on the MAC silicon, and allows for rapid transition today with legacy non-EEE MAC silicon attached to AutoGrEEEn-enabled PHYs.
Q: Servers are refreshed every three to five years at data centers. How about networking equipment?
A: In general, the networking equipment refresh cycle is probably longer than that of servers. However, it is important to consider networking in conjunction with servers and other IT equipment. Servers get more powerful, with new chips every 18 to 24 months. For example, a newer server with more computing power may require 48 ports rather than 24 ports in the networking equipment to process more loads. Thus, the refresh cycle may not be that much longer than that of servers. In addition, new builds are happening all the time and they can take advantage of any new innovations in energy savings.
Q: I think 802.3az is for 10 Mbps, 100 Mbps, 1 Gbps, and 10 Gbps. Does it apply to 40G and 100G as well?
A: For 40G and 100G, there is another standard called 802.3ba, which was recently ratified.
Both 802.3az (EEE) and 802.3ba (HSE) started about the same time. EEE looked at copper
interfaces that already existed ton be enhanced for energy efficiency. By the way, the
energy efficiency technology we use is called
low power idle . When it is idle, it sends little energy and is very energy efficient.
To achieve this both sides go into a suspended state, effectively turning the PHYs off. To keep both sides synced at the PHYsical layer a refresh signal is occasionally sent. Because the refresh signal has a low duty cycle, it is very energy efficient. To leave the LPI state, either side can send a wake-up signal and the other side reacts to that.
In addition to PHYsical layer savings additional subsystems above the PHY can be turned off. Broadcom championed the development of a technology called layer 2 data link LLDP stateful negotiation for enhanced saving modes in the standard.
Q: Some people insist that saving on the networking side does not make a huge impact on data center power consumption. Some studies indicate that power consumed by networking equipment at a data center is only several percents compared with those of servers and storage equipment. What do you say to that?
A: It is true that the networking equipment by itself does not consume a lot of power from the perspective of a few single ports. However, this energy efficient technology (EEE) for networking enables energy efficiency across the entire data center which typically has 1000’s or more ports. Enabling EEE is like turning off a bunch of leaky faucets. One might not notice the savings at a single point in time but it will certainly show up in the monthly bill. Ethernet has also been proven to be an extremely scalable technology enabling port consolidation during bandwidth migrations cutting total power consumption (1G to 10G for example) while at that same time increasing performance typically by a factor of 10. In a sense, Ethernet networking is playing a much larger role in turning a data center into the next-generation energy efficient one.
Think of this analogy. When you try to establish a high quality of service (QOS), you may want to have the fastest server with the fastest CPUs in it. However, if you study this carefully, you would find out that the fastest CPUs alone are not the answer. It may be established with fast CPUs, memory, disk, and other components combined. Energy consumption at a data center can be considered in the same way. Networking connects IT equipment together and, by activating and deactivating according to needs, networking controls other IT resources (IT equipment). This energy efficient technology in networking should not be considered just for networking equipment but for all the IT resources and, consequently, the entire data center.
After talking with Wael and David, I pondered the following. My discussion with them reminds me of the argument about whether servers or networking are more important in computing. It is true that servers are the center of computing, and we tend to pay attention only to servers. However, networking ties together servers and other equipment, like storage, to create an IT system. Without fast, low-cost, and energy efficient networking equipment, even the fastest and most energy efficient servers could not deliver, alone, the most energy efficient computing for a data center. This means networking equipment needs to be energy efficient. On top of that, networking equipment should be able to allocate resources dynamically, as needed, and without delay, to ensure the energy efficiency of other IT resources.
When the IT energy efficiency discussion was in its infancy a few years ago, it was O.K. to discuss each category of IT equipment in a data center separately to see how much it consumed in order to control its consumption. As IT equipment is being integrated and coming to function as one system, it is time to consider the energy efficiency of the IT system as a whole.
There are a few more technologies for making networking equipment more energy efficient, but I will cover them sometime in the future.