www.brocade.com
WHITE PAPER
Data Center
What Is an Ethernet Fabric?
Compared to classic hierarchical Ethernet architectures, Ethernet fabrics provide the higher levels of performance, utilization, availability and simplicity required to meet the business needs of data centers today and into the future.
Data center networks rely on Ethernet. Over the decades, Ethernet has evolved as new types of architectures emerged. Today, data center networks carry traffic for a diverse set of applications including client/ server, Web services, unified communications, virtual machines, and storage—each with different traffic patterns and network service requirements. For example, when Ethernet carries block storage traffic, it places stringent demands on the network including lossless packet delivery, deterministic latency and high bandwidth. Combined, these changes drive the next evolutionary step in Ethernet networks, the Ethernet fabric.
Executive Summary In a 2010 Gartner survey of over 1600 CIOs across the globe, participants were asked to identify their top business priorities today and looking 3 to 4 years out. Their responses indicated that the focus today is on improving business processes and cost savings. Then, looking out a few years, improving productivity, driving innovation, gaining competitive advantage, and acquiring new customers increase in priority. These business priorities drive data centers to deploy new applications quickly and efficiently, provide fast and reliable “round-the-clock” access to information, meet or exceed stringent service levels with zero downtime—and all this while driving down costs by maximizing investments. In short, IT must move at the speed of business to capitalize on new opportunities and respond to increasing global competition. High-density, multi-core servers, as well as network, server, and storage virtualization are the technology enablers that address these business needs. Data centers can leverage this set of technologies to pool IT resources and implement cloud architectures that reduce capital and operational expenditures, and at the same time create an infrastructure that rapidly scales and responds to business needs. When data centers leverage these technologies, new networking challenges arise that were not present when applications were tied to physical servers.
2
Core
ISLs
Figure 1. Classic Ethernet network.
Access
Aggregation
Traffic flow
Inactive links
Server rack
This means that the network must evolve. It must move from management of physical ports to flows (virtual server to virtual server or virtual server to virtual storage communication). It must be simpler to operate, more flexible, highly resilient, and much more scalable. These requirements are best met with scalable fabric architectures, while classic Ethernet networks require complex architectures and protocols adding higher levels of complexity and operational costs. Brocade® VCS™ technology is designed to meet these challenges with three main technology pillars: Ethernet fabric, Distributed Intelligence, and Logical Chassis. These pillars enable Brocade VCS architecture to greatly decrease the operational costs of networking by providing a highly reliable, simple, scalable networking infrastructure. Brocade VDX™ 6720 Data Center Switches deliver Brocade VCS technology and will revolutionize the way networks are architected now and in the future.
Characteristics of Ethernet Fabrics One of the pillars of Brocade VCS technology (see Figure 2) is the Ethernet fabric, described in the following sections.
Flatter Classic Ethernet networks are hierarchical with three or more tiers. Traffic has to move up and down a logical tree to flow between server racks, adding latency and creating congestion on Inter-Switch Links (ISLs). Spanning Tree Protocol (STP) prevents loops by allowing only one active path, or (ISL, between any two switches. (In Figure 1, inactive paths are shown as dotted lines.) This means that ISL bandwidth is limited to a single connection, since multiple paths between switches are prohibited. Enhancements to Ethernet tried to overcome this limitation. Link Aggregation Groups (LAGs) were defined so that multiple links between switches were treated as a single connection without forming loops. But, a LAG must be manually configured on each port in the LAG and is not very flexible. Ethernet fabrics prevent loops without using STP. Flatter networks include self-aggregating ISL connections between switches, eliminating manual configuration of LAG ports while providing non-disruptive, scalable bandwidth within the fabric. Ethernet fabrics support any network topology (tree, ring, mesh, or core/edge) and avoid bottlenecks on ISLs as traffic volume grows, since all ISLs are active.
Ethernet Fabrics Compared to classic hierarchical Ethernet architectures, Ethernet fabrics provide higher levels of performance, utilization, availability, and simplicity. They have the following characteristics: • Flatter. Ethernet fabrics are selfaggregating, enabling a flatter network. • Intelligent. Switches in the fabric know about each other and all connected devices. • Scalable. All paths are available for high performance and high reliability. • Efficient. Traffic automatically travels along the shortest path. • Simple. The fabric is managed as a single logical entity.
3
Intelligent Classic Ethernet switches require configuration of each switch port. This includes setting network policies such as QoS, security, VLAN traffic, etc. When only physical servers connected to the network, this model was sufficient. But today, server virtualization requires multiple virtual machines to be configured on each switch port. When a virtual machine migrates either for load balancing or routine maintenance, the port configuration has to move to a new network port or the migration fails. This requires manual configuration. Ethernet fabrics have distributed intelligence, which allows common configuration parameters to be shared by all switch ports in the fabric. In the case of virtual machine migration, the network policies for that virtual machine are known at every switch port so migration does not require any changes to network configuration. In an Ethernet fabric, switches share configuration information, and they also know about each other. When a device connects to an edge port of the fabric, all switches know about that device. As the device sends traffic to other devices, the fabric can identify the shortest loop free path through the fabric and forward frames with the lowest possible latency. New traffic types such as virtual machine migration and storage traffic are latency sensitive. Fabrics ensure this traffic gets to its destination with minimal latency.
Scalable Classic Ethernet allows only one path between switches. Improvements such as link aggregation groups (LAG) allow several physical links to act as a single link. This is manually configured on every port in the LAG and is often inefficient limiting bandwidth. If a new switch is added for more connectivity, it becomes increasingly more complex to manually configure multiple LAG connections. Ethernet fabrics overcome this. When a new switch connects to the fabric, no manual configuration is required for the inter-switch links. The switch joins the fabric and learns about all the other switches in the fabric and the devices connected to the fabric. No manual configuration of policies or special LAG configuration on specific ports is necessary. If multiple inter-switch links are connected between two switches, a logical trunk automatically forms. Traffic is load balanced in hardware so that utilization is near line rate on every link for high efficiency. Should a link in a trunk go off-line, traffic on the remaining links is not affected and incoming frames are automatically distributed on the remaining links without disruption to the devices sending them.
Efficient Classic Ethernet uses STP to define a loop-free path, forming a logical hierarchical switch tree. Even when multiple links are connected for scalability and availability, only one link or LAG can be active. This lowers utilization. When a new link is added or removed, the entire network halts all traffic for tens of seconds to minutes while it configures a new loop-free tree. This is highly disruptive for storage traffic, virtual machine migration, and so on. In the case of storage traffic, traffic disruption could cause a server crash. Ethernet fabrics do not use STP to remove loops. They use link state routing with equal-cost multipath routes, which always take the shortest path through the network. When a link is added or removed in the fabric, traffic on other links continues to flow non-disruptively. Link resiliency is assured and full utilization of all links between switches is automatic when the topology is changed without any manual configuration.
4
Simple Classic Ethernet switches require management. Each switch has to be configured and each port has to be configured for protocols (STP, RSTP, MSTP, LAG, and so on), VLANs, network policies, QoS, and security. As more server racks are added, more switches are added at the top of rack, middle of row or end of row. Each requires configuration and none can share common configuration parameters. Ethernet fabrics share configuration information among all switches in the fabric. When a new switch joins the fabric, it automatically receives common information about devices, network policies, security, and QoS. This simplifies network configuration, reduces mistakes, and reduces operating cost.
Brocade VCS Architecture • VCS Ethernet fabric • VCS Distributed Intelligence • VCS Logical Chassis • VCS Dynamic Services
Brocade VCS Architecture Brocade is the first vendor to deliver products that meet the all the requirements of a true Ethernet fabric. Brocade VCS architecture redesigns traditional Ethernet networks, removing the limitations of classic Ethernet. The Brocade VCS architecture is ideally suited for both private and public cloud computing deployments.
Brocade VCS Technology
Ethernet Fabric
Distributed Intelligence
No STP
Self-forming
Multi-path, deterministic
Arbitrary topology
Auto-healing, non-disruptive
Network aware of all members, devices, VMs
Lossless, low latency Convergence ready
Masterless control, no reconfiguration VAL interaction
Dynamic Services
Figure 2. Brocade VCS architecture. Logical Chassis
Logically flattens and collapses network layers Scale edge and manage as if single switch Auto-configuration Centralized or distributed management, end-to-end
Connectivity over distance, Native Fibre Channel, Security Services, Layer 4 - 7, and so on
Figure 2 shows the key capabilities of the Brocade VCS architecture—capabilities that are responsible for implementing an Ethernet fabric: • Brocade VCS Ethernet fabric in the data plane • Brocade VCS Distributed Intelligence in the control plane • Brocade VCS Logical Chassis in the management plane In addition, Brocade VCS Dynamic Services supports simple scaling of network services such as security, Layer 4-7 application delivery control, extended Layer 2 networks between data centers and native Fibre Channel storage services for converged networks. Combined, these Brocade VCS components transform classic Ethernet networks into Ethernet fabrics.
5
Brocade VCS Ethernet Fabric The Brocade VCS Ethernet fabric simplifies interconnect configuration, provides automatic link failover with no interruption of traffic on unaffected links, and provides plug-and-play fabric scalability. Instead of manually configuring LAG on individual ports on multiple switches, Brocade VCS Ethernet fabrics automatically forms trunks when multiple ISL connections are added between switches. Simply adding another cable increases bandwidth, providing linear scalability of server-to-server and server-to-storage traffic. The flexibility of this topology allows easy specification of oversubscription ratios. For example, High Performance Computing (HPC) workloads may require 1:1 subscription, virtual servers 4:1, and client/ server 10:1 or higher.
Brocade VCS Distributed Intelligence Brocade VCS Distributed Intelligence provides common configuration parameters to every switch in the fabric. For example, when a new switch is added to the fabric, it shares information about other switches and devices. Policies already configured are available to all switch ports. With Automated Migration of Port Profiles (AMPP), virtual servers can migrate between switch ports without risking conflicts with network policies and security settings at edge ports; these settings automatically follow the MAC address of the virtual machine.
Brocade VCS Logical Chassis Instead of managing each switch and its ports independently, the Brocade VCS Logical Chassis service creates a virtual management plane. A Logical Chassis has common configuration parameters for policies and can bind MAC address to policies with AMPP and apply them to all switches in the fabric. The Logical Chassis service creates a single point of management for monitoring all switches, ports and traffic in the fabric. Adding a new switch is as simple as adding a new port blade to a chassis switch. The switch and its ports are uniquely identified, share common configuration parameters, and are monitored from a single management connection. With the Logical Chassis service, management can scale with the network, since adding switches and ports does not add complexity.
6
The Brocade VDX 6720 Family of Data Center Switches The introduction of the Brocade VDX 6720 family of 10 GbE-capable fabric switches delivers a new category of Ethernet switching, the Ethernet fabric switch. The Brocade VDX 6720 includes Brocade VCS architecture, offering revolutionary Ethernet fabrics, Distributed Intelligence, and Logical Chassis technology for the data center. The Brocade VDX 6720 is available in 24- or 60-port rack-mount form factors and can be deployed in 16-, 24-, 40-, 50-, or 60-port configurations with Brocade exclusive Ports-onDemand (POD) licensing. At launch, the Brocade VDX family interoperates with existing Ethernet switching products, providing an evolutionary path from legacy Ethernet networks to Ethernet fabrics. Existing legacy switches do not have to be replaced until the end of their useful life; Brocade VDX 6720 switches can be added at the top of rack or middle or end of row forming a scalable, simple-to-manage Ethernet fabric.
Figure 3. Brocade VDX 6720-24 Data Center Switch (top) and Brocade VDX 6720-60 Data Center Switch (bottom).
For example, several Brocade VDX 6720 switches deployed in a ToR configuration create a single Logical Chassis with a single distributed control plane across multiple racks of servers, delivering compelling reductions in capital and operating costs, while simplifying virtual machine migration. Automated Migration of Port Profiles (AMPP) is hypervisor agnostic, which is an important feature, as most data centers deploy different hypervisor stacks based on application and server requirements. And, since the Brocade VDX 6720 family supports the emerging Ethernet Data Center Bridging (DCB), TRILL, and Fiber Channel over Ethernet (FCoE) standards, the Ethernet fabric is lossless, low latency, and convergence ready.
About Brocade Brocade networking solutions help the world’s leading organizations transition smoothly to a virtualized world where applications and information reside anywhere. This approach is based on the Brocade One™ unified network strategy, which enables a wide range of consolidation, convergence, virtualization, and cloud computing initiatives. Offering an industry-leading family of Ethernet, storage, and converged networking solutions, Brocade helps organizations achieve their most critical business objectives through unmatched simplicity, non-stop networking, application optimization, and investment protection. To ensure a complete solution, Brocade partners with world-class IT companies and provides a full range of education, support, and professional services offerings. Learn more at www.brocade.com.
7
WHITE PAPER
Corporate Headquarters San Jose, CA USA T: +1-408-333-8000
[email protected]
www.brocade.com
European Headquarters Geneva, Switzerland T: +41-22-799-56-40
[email protected]
Asia Pacific Headquarters Singapore T: +65-6538-4700
[email protected]
© 2011 Brocade Communications Systems, Inc. All Rights Reserved. 01/11 GA-WP-1550-01 Brocade, the B-wing symbol, BigIron, DCFM, DCX, Fabric OS, FastIron, IronView, NetIron, SAN Health, ServerIron, TurboIron, and Wingspan are registered trademarks, and Brocade Assurance, Brocade NET Health, Brocade One, Extraordinary Networks, MyBrocade, VCS, and VDX are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries. Other brands, products, or service names mentioned are or may be trademarks or service marks of their respective owners. Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained in this document may require an export license from the United States government.
