I’ve been asking myself this question for the past several weeks. Seems like I run into a new glassy-eyed IT guy that has heard the words "Fibre Channel over Ethernet" and they’ve been swept off their feet. Sure, I like the sound of running just one network - and Ethernet has proven time and again it is the dominant technology. So I started to look behind the curtain at FCoE to see what is really there.

Colleagues that have sat through briefings tell tales of product lines that will unify the data center, cure cancer and stop world hunger! After taking a closer look the best analogy that comes to mind is Dorothy looking behind the curtain of the Wizard of Oz. The grandness of vision at first glance evaporates when you scratch the surface. How disappointing it is to see that there are only two vendors, each with a single switch. I was expecting product lines, not a product. I was expecting broad adoption, choice! Where is the great FCoE vision?

Well it turns out the vision is in the future, more specifically in several IEEE working groups. You see, the only standard ratified so far is the encapsulation of Fibre Channel in Ethernet - it even has its own Ethertype. Isn’t this FCoE many ask? Yes and no.

It is important to realize that putting Fibre Channel into Ethernet packets is just the first part of the solution. There are many other pieces required for Ethernet and FCoE to displace today’s storage network. Fibre Channel is pretty finicky; it mandates ‘lossless Ethernet’. Today’s Ethernet isn’t lossless, and follows its original design as a ‘best efforts’ transport. There is an effort to transform Ethernet into a data center bridging medium, capable of supporting FCoE.

Currently the IEEE has several work groups (e.g. P802.1Qau, P802.1Qbb, P802.1az among others) that are developing the necessary standards (yes that’s a plural) required for Ethernet to be a viable transport layer for Fibre Channel traffic. Within each of these work groups there is a lot of work to be completed. The transformation of Ethernet from "best efforts" to "lossless" is not a small challenge, although vendor briefings I’ve sat through have ignored this. The current estimate of completion is in the latter half of 2011. So should I put an end-to-end data center fabric on my Christmas wish list for 2011?

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		When Less is More: Why Small Companies Should Think Outside the (Red/Yellow) Box for Protecting Endpoints
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

If I did, I think I’d be disappointed. The standards work to be completed will likely have some early draft adopters, but considering the data center is the heart of a company I wouldn’t budget any dollars for this before 2012. Looking at prior standards, there is a gap from when a standard is approved and when standards-based products will likely become available from multiple vendors – providing choice and stability. So given that we’re really looking at two years before FCoE is viable, would I try any products now?

Let’s look at the benefits. Suppose you have today a top of rack switch that has 10GbE Converged Ethernet interfaces and Fibre Channel uplinks. Server traffic destined for various networks would be split at this switch and forwarded to the network it is destined for; Fibre Channel over Ethernet traffic would egress the Fibre Channel interfaces and the data traffic would egress the Ethernet interfaces. FCoE traffic would cease to be FCoE. It would be translated at the very first switch into native Fibre Channel.

Yes, I could reduce the number of interfaces in my servers to a single adapter (likely two for redundancy), but is that enough to take the plunge? Not for my network. I already have Fibre Channel run to my servers and even though we replace servers routinely – especially as we virtualize systems to reduce our footprint – there just isn’t a large enough benefit to reducing what is essentially the first five feet of a storage network.

Instead, we’ve made a decision to migrate to iSCSI storage and reduce the systems using our Fibre Channel SAN. We’re already gaining benefits of a converged network without the risk of integrating incomplete solutions. I won’t risk the religious argument of FCoE vs. iSCSI in this post, but am interested in covering that in the future.

Knowing every site is unique - what are your plans for a converged data and storage network?

Related posts:

1. Discussion on Ethernet Energy Efficiency (802.3az) at BroadCom
2. More Energy Efficiency by IT at Data Centers, Panel Disucssion Proposal
3. Internet Economics: Traffic Trumps Ethics And Content Quality

Realizing the full capability and investment of dynamic re-provisioning, load-balancing, and disaster recovery flexibility offered by today’s virtualization solutions, requires a network infrastructure that will dynamically adapt without human intervention. Intelligent infrastructure automation can solve 3 key network automation challenges in a virtualized data center without sacrificing access controls, prioritization, or compliance controls recognized in static data center environments. These network automation challenges include:

1. Automated virtual machine identification and authentication
2. Automated virtual network provisioning
3. Differentiated access provisioning on shared network interface(s)

The first element of an automated virtual data center solution is authentication. Implementing machine authentication, also known as MAC authorization or MAC bypass, allows the infrastructure to recognize a new forwarding station on an Ethernet port. This intelligence provides insight into the device, capturing necessary requirements for the virtual network access controller. Automated for appropriate network resource allocation and provisioning. The provisioning instructions provided should include the VLAN assignment for the virtual machine, any necessary access control lists (ACLs), and appropriate quality of service (QoS) assignments for application priority delivery. The third key required element is the ability to provide differentiated access provisioning for the multiple virtual machines sharing a single gigabit or 10 gigabit Ethernet interface. Some switches today support individualized virtual sessions for each MAC address that is provisioned. Be sure to ask your vendor if they support this function.

The implementation of a virtual network access control system allows the IT administrator to group virtual machines into supporting business roles. The virtual network access control system ensures that as a virtual machine is provisioned, or re-provisioned, that it receives the correct services it requires. Smarter virtual network access control systems can coordinate multiple resources, solving the challenge of virtual machine mobility; let’s look at a real-world example.

A customer wants to ensure that if his location in Orlando experiences a partial server outage, that the virtual machines that the business is dependent on can be re-hosted at his data center in Tampa. His challenge: the IP address of the virtual machines hosting critical manufacturing applications must not change. As the hypervisor management system rehosts the virtual machine, the network access request from the switch in Tampa matches a location policy in the virtual network access controller. The switch port is provisioned with the correct VLAN, ACLs and QoS settings as it would normally occur, but an additional action provisions the local router to advertise the IP address of the migrated virtual machine at the Tampa facility. The remaining IP address range is still located in Orlando and connectivity is maintained. When the Orlando data center has the capacity to host the virtual machine, the migration is reversed.

	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		Effective Management of Static Analysis Vulnerabilities and Defects
	- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

This scenario would normally require the coordination among several persons in an IT department, along with much communication and manual intervention. The human aspect introduces timing issues and the potential for configuration errors. Automating this process provides for a quick consistent recovery of the business application that is predictable, repeatable and is not dependent on the presence of a human operator.

This type of automation in the virtual data center can solve many challenges, and therefore it is important that the data center networking infrastructure has the core capabilities required to execute precise and dynamic network provisioning.

Related posts:

1. How to Provide Cloud Computing without a Data Center
2. How to Move a Data Center in 35 Days – Week 3: The Power Balancing Act
3. Real Time Protection for Hyper-V

A client acting as a server sounds like it shouldn’t be an awful thing, but it can be if left unchecked. Let’s look at a few examples.

1. A number of viruses use their own mail engine to replicate versus trying to understand what the victim’s computer uses for email


2. Worms that spread across data networks often start a TFTP server on the infected system and use TFTP to copy themselves from system to system


3. Man-in-the-middle (MITM) attacks often use ARP cache poisoning of victim computers to ‘claim’ the MAC address of the router for their own system

Understanding how the network layer is used in each of these attacks can help us build a defensible network layer using a very simple approach.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
   Buyer’s Guide: Sales Force Automation

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

I’ve taken the position that “clients should not be servers”. That sounds simple, but what does this mean? In data networks, there are a number of IP services (represented by protocols on the network) that should only be provided by specific systems located in the data center and managed by the IT organization. If these IP services are accidentally or maliciously attached to the edge of the network, they often disrupt the operation of the network by causing performance, reliability or security problems. A common example is the case of the rogue DHCP server. It could occur on your network for a number of reasons: a contractor that forgot he left a DHCP server running on his laptop from his last demo, an administrative assistance noticing an unplugged cable and attaching the DSL modem to the corporate network, or a person with malicious intent wanting to route traffic on your network through their device. The first two examples result in downtime for the systems affected and IT having to chase down the ghost in the network. The third example may require external assistance.

Rogue DHCP servers present only one common vector we can easily rectify at the network edge. Below are the top protocols for which you should consider policy control at the edge of your network. Each represents a common mistake or exploit vector in the majority of enterprise networks deployed today.

These suggested policies improve security and will harden your network against a variety of threats. Optionally you can create a variety of network profiles based on the person’s or device’s use of the network. These profiles, more accurately described as ‘roles’, allow today’s network manager to better manage resources and manage their threat model. I’ve used the convention “Role Based Network Access Control” to describe this function as it mirrors the concepts in “Role Based Access Controls” published by the US National Institutes of Standards and Technology. You can view their version here: http://csrc.nist.gov/groups/SNS/rbac/

I hope you find this information helpful and would welcome your feedback – did you find this helpful or are you deploying some version of this on your network today? I’d be happy to answer your questions.

Remember, every network is unique. Evaluate your network and consider each policy before deploying it.

Related posts:

1. Justifying IT Security: Managing Risk & Keeping Your Network Secure
2. How VPN Technology and Flexible Policies can protect Employee Health during the H1N1 Pandemic
3. The MPLS Network: A Future-Proof Engine for Voice-Data Convergence

The topic of energy and power manifested itself in other ways in week three of our four-week project. As we began installation of these 35 servers in our new data center, the issue of how to best provide uniform power to the Main Data Frame (MDF) proved to be tricky. If power isn’t evenly distributed across the MDF chassis, the machines won’t power on, and even if they do, there’s a risk of errors. Even, clean distribution of power is an essential element of a successful MDF installation. Because of the changes in floor layout and our plans to virtualize several machines, we couldn’t simply replicate the power plan from the old location. Basically, we were starting from scratch – with one week left!

Our power plan started like this: a 400Amp breaker was supposed to be feeding the MDF, including a 200Amp subpanel that feeds most of the buss ducts. On the buss above the three cabinets are three phases, with each one using up to 80 amps. However, since they are fed off the 200A subpanel that can only draw a max of 160Amps per phase, the 400A that feeds the 200 that feeds the 100 can only draw 320. Confused yet? We were!

Additionally, each power strip had either a 20A or 15A breaker, and each of the junction boxes off of the buss duct also has a 20, or 30 amp breaker. None of the breakers can trip, or we have a mini-disaster on our hands. We were beginning to look at our power plan like it was one of those brain teasers. While we were fairly certain that our plan kept us under all the maximums for consumption, power remained a concern.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Managing Web Infrastructure Storage Cost, Complexity, and Performance
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

As we began installing and powering up equipment in the MDF, we had to re-adjust our original plan due to loading issues. We expected some of this – it’s customary with every build. We had to re-think the use of large power strips, since they couldn’t handle the load as well. We decided to use four smaller ones instead, especially for dense cabinets. We decided that shorter power cords were better for our new space, as the six-foot ones took up too much space in the rack.

But all that changed when the Dell/APC cabinets we ordered showed up and were wider than expected. This meant that the floor layout and power plan needed to be redone quickly, and it also brought into question our use of the shorter power cords. Normally, a change like this wouldn’t be an issue, but when working under a deadline like ours, even small changes seem bigger.

Once our power puzzle was solved, we began virtualization of the 35 total boxes. Our goal was to have them moved out of the Boston co-location facility in 30 minutes. But once again, we hit a snag with performance issues, and after four hours of troubleshooting we discovered a bad blade that needed to be swapped out in order to continue.

By day 20 of the project, we had achieved the following:

1. Successfully virtualized about half the major boxes, including two application servers
2. As a result of virtualization, removed 13 of the servers we had been using in the previous co-location facility
3. Installed fiber switches configured for backup tape drives
4. Resolved Port and IP issues with our VM clusters
5. Planned for the AC installation in the Main Data Frame

Upon completion, we did a quick power audit and see how much power we gained, and discovered a net gain of 25 amps. This was a good sign, and we felt confident enough to start powering up some of the gear, including our SAN switches. However, after adding two more virtualized servers, our estimate showed us at about 8 amps short. This new finding required an additional UPS in the rack and also some unexpected downtime in the project, so the team could install new fiber runs and make more network reconfigs. We also discovered a problem with the conversion of the physical SAP application server to virtual, caused by a Compaq kernel driver that monitored the disk array that no longer exists, since it is virtual now. Time for more work-arounds from the team!

Looking back on it, we learned some best practices on distribution of power through a data center.

If you are going to a colo or hosted data center:

1. Make sure they provide enough power. Many data centers only give you 150-200 watts per sq/ft, which may not be enough for dense environments like virtualization. Newer data centers should be able to supply more power, while older data centers may make you rent more square footage to get enough power for your gear. Cooling and power go hand in hand– so check that, too.
2. Check to make sure they bill for power used. Many times they charge per circuit, which is fine, as long as you realize it and it’s priced accordingly. If you are paying for say four 20 amp circuits at 120volts though and they bill you for 19.2kw (120*20*8) at current market rates, you are probably getting overcharged. Also these facilities rarely let you use more than 80% of the power, so of your 20 amps circuits, you can only use 16amps.

If you are designing your own data center, here are some additional tips:

1. Trace your power all the way back to the street to see where your single points of failure are. You should plan to use redundant circuits versus a single main feed. For example; if you go back to a single 800amp breaker, you risk both your primary and secondary power if that breaker trips. To do it right, you should have separate feeds, generators, UPS and switchgear all the way through the building.

2. Physically separate the two feeds as much as you can (A side on the right, B on the left) and try to bring it back to separate rooms if possible.

3. Don’t forget redundant cooling. If you have power but no cooling you are likely to just overheat all of your gear anyway. Overheating may not cause failures right away, but can cost more over time than the downtime would have.

4. Document, document, document and restrict who can make changes.

5. Test the power redundancy by turning off power at the street and seeing what happens. Ideally a bunch of pagers will go off and that’s it. If not: figure out why.

Related posts:

1. How to Move a Data Center in 35 Days – Week One
2. How To Move a Data Center in 35 days - Week 2
3. Sentilla: Measuring Power Consumption At A Data Center

Weekend Move: First Migrations

Saturday

As part of our migration plan, we knew that we would move systems from our colocation facility to our corporate facility in a number of moves. The first weekend was scheduled with the goal of helping our teams identify any misses in our plans for the larger moves scheduled for the following weeks.

This weekend targeted an initial 35 servers for migration. Personnel that were identified from our previous planning sessions and stakeholders for the systems being relocated were notified prior to any possible interruption of service. Planning really pays off here, as we’re working on a very tight time line.

The 35 servers were a mix of iSCSI storage nodes, VMWare ESX servers and some legacy systems. We started promptly at 5 am by uncabling systems and readying the racks for roll-out.

The first hiccup was that our colocation provider removed power from the wrong racks. This sent some production systems temporarily offline and required our attention to ensure services were restored before we could pull the plug on the systems that we planned to move. The colocation partner was able to restore power and then remove power from the racks we were moving, but this delayed our load time. Finally, at 11 am on Saturday, we had all of the systems loaded on the truck and we were headed on our 30 minute ride north.

Once we arrived in Andover, we unloaded our racks and moved them into our existing data center. The racks were rolled in, powered up and connected to the network by 5 pm. The first order of business at this time was to get an accurate measurement of our power consumption. When we powered up the network chassis and systems, the current draw was much less than what we expected. This provided some wiggle room in our earlier estimates, which was unexpected good news!

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Avoidable Mistakes that Compromise Cooling Performance in
Data Centers and Network Rooms
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Part of the closing activities for day one was to add some 32 bit routes into our routers to allow the IP addresses from the migrated systems to be accessed. This is quite common to defend networks, and is quite handy to allow servers in the same networking space to be in two different locations.

Sunday

The team was split into groups, with team one focusing on the preliminary cabinet layout for the new data center and team two focused on final tasks from yesterday’s move. A key partner, our purchasing department, had been able to expedite eight of the 22 cabinets we had on order – so this helped with our planned migration from the cabinets we brought north the day before.

As our teams were focused on their respective jobs, our network team was able to capture temperature readings from the servers at the colocation facility and compare them to previous readings. We wanted to determine if we were going to have a negative impact in our corporate data center before our additional A/C systems were going to be available. Thankfully, it was still winter during our move so we weren’t faced with systems taxed by a late summer heat.

By Monday, we had executed a partial move that did not impact our key stakeholders. This information was presented to senior management along with the remainder of our move plan. We received executive buy off for the next two weekend moves.

Related posts:

1. How to Move a Data Center in 35 Days – Week One
2. How to Move a Data Center in 35 Days – Week 3: The Power Balancing Act
3. Data Center in a Box Comeback?

Part of the Enterasys IT staff’s annual plan in 2008 was to consolidate our data centers into several strategic locations. The reasons for this consolidation included a need to reduce costs associated with multiple locations, as well as a company-wide campaign to reduce energy consumption. The Enterasys data center site in Boston cost more than $600,000 to maintain, and with another location just 30 miles away in Andover, it no longer made sense. Our original plan was to move the data center to our Andover headquarters by the end of the year. However, anyone in IT knows that all plans have changes, and we received a request to ‘accelerate’ the move. The new plan was to move the entire center in less than 45 days!

We started by engaging our IT operations team. After much discussion, we realized that the space we had available in Andover was much smaller than what we currently had in Boston. The only way to make the move work was to virtualize as many systems as possible to reduce the footprint. But space wasn’t our only roadblock: we also needed to ensure that we would have enough power in the building to support this data center move.

At our weekly staff meeting, we discussed the project with our entire IT staff. That discussion generated the following initial list of questions.

   1. Power

      a. Can we get a 200amp temporary home service from National Grid?

      b. Can we get a power audit to see what we really use for power in Boston?

      c. How much power would we save by switching 21″ CRTs to LCD monitors?

   2. Racks

      a. Do we need new racks, or can we move the existing ones?

      b. Can we fit the Boston servers in existing racks in Andover?

   3. Network

      a. Do we need additional network gear – and can we order and receive it in time to pre-stage in Andover?

      b. How can we split subnets across the WAN?

   4. Backups

      a. Can we import our existing catalog?

      b. Should we keep the existing gear, or replace?

      c. Do we need to have our backup vendor schedule pickups in Andover instead of Boston?

   5. Storage

      a. Do we have time to finish migrating from fiber channel to iSCSI?

      b. Can we get our storage vendor to help us move?

      c. Are we still under maintenance with the storage gear?

   6. Space

      a. Can we free up enough space in the MDF?

      b. Can we retire or consolidate any engineering servers or network gear?

   7. AC

      a. Can we get the new AC units installed and running in time?

      b. Can we get temporary units in place and powered?

      c. How much can we save on cooling costs by opting for virtualization of some of our servers?

   8. Move

      a. Are people available to move in March?

      b. Will Enterasys be willing to move the last month of the quarter?

      c. What trucking company should we use?

      d. Do we need to insure the equipment?

   9. Alternate data center facilities

      a. Get quotes from three other data centers for short term space leases, just in case.

We started by requesting that everyone update their availability in Outlook for the rest of the month. This allowed us to plan the move appropriately. With some juggling of tasks, we were able to assemble a master calendar that didn’t affect key IT projects for the month.

We discussed our power options with our electrical contractors. After consulting with them, we learned that we would, in fact, have sufficient power — until the summer heat kicked in. We also determined that we had another requirement to meet: we would need to run our current AC systems at 100% until the additional AC systems could be added. We sure had our challenges– and not a lot of wiggle room!

The storage team decided on leaving three iSCSI storage nodes with some VMWare servers at the Boston facility, and to replicate the iSCSI nodes from Boston to Andover to make it easier to move the virtual images.

Our applications team reviewed the move plans and made recommendations to start work on the test, mitigation and backup plans. Each application was assigned a priority (A,B,C) which defined how much documentation and support we needed for the move weekend. We also developed a schedule for applications that would be moved each weekend.

 In our next blog entry, we’ll talk about planning for the move, some of the details of laying out the new data center, inventorying the systems and planning the virtualization of our core systems.

With all this work and investment, we also developed a case study with our server and storage partner, Dell.

NetHawk Interactive provides white papers and webcasts about virtualization. Learn more...

Related posts:

1. How To Move a Data Center in 35 days - Week 2
2. How to Move a Data Center in 35 Days – Week 3: The Power Balancing Act
3. Data Center in a Box Comeback?