128 x 100G - FSW

---

Minipack and F16, Software Alex Eckert, Software Engineer Facebook

Minipack and F16, Software

Software everywhere Management plane Control plane Data plane Platform

Wedge

16 x 40G - RSW

FBOSS Overview Monitoring Daemons

Config Management

FBOSS Agent

Routing Daemon

System Tools & Libraries OpenNSL Linux Kernel

Wedge

OCP Micro Server

Broadcom Trident II

OpenBMC

BMC

Sixpack

128 x 40G - FSW

Wedge 100

32 x 100G - RSW

Backpack

128 x 100G - FSW

Wedge 100S

32 x 100G - RSW

Fabric Aggregator

32 x 100G - RSW N x 100G - FA

Minipack

128 x 100G – FSW, SSW, FA

New challenges (Minipack) ØModular switch, single control plane ØDifferent PIM types, variety of port speeds ØExternal phy support (Broadcom) ØI2C to 128 optics, MDIO, FPGA ØNew microserver (MiniLake), New ASIC (Broadcom TH3) ØCooling 128 optics in only 4 RU

128 x 100G – FSW, SSW, FA

Arista 7368X4

128 x 100G – FSW, SSW, FA

New challenges (FBOSS on Arista) ØFirst time operating FBOSS on non-FB hardware ØDifferent external phy (Credo), Different FPGA ØUnfamiliar microserver, no off-the-shelf UEFI BIOS ØMust manage as either FBOSS or EOS in production ØComplex conversion process b/w EOS / FBOSS

128 x 100G – FSW, SSW, FA

Combinations ØTwo switch models ØModularity ØThree layers of F16 network ØSame binary everywhere

Common abstractions

Facebook Minipack

Arista 7368X4

OCP MiniLake

Arista uServer

Brcm TH3

Brcm TH3

Brcm Gearbox

Credo Gearbox

Edgecore Board (FPGA/PIMs/PSUs/Fans)

Arista Board (FPGA/PIMs/PSUs/Fans)

New microservers!

MiniLake

New microservers!

Arista control module

OCP Open System Firmware

Conversion

FBOSS Overview Monitoring Daemons

Config Management

FBOSS Agent

Routing Daemon

System Tools & Libraries OpenNSL Linux Kernel

Minipack

MiniLake

Tomahawk III

OpenBMC

BMC

Minipack Hardware Components Minipack

MiniLake

Tomahawk III

IOB FPGA

PIM

DOM FPGA External phys

… x8

Transceivers

BMC

Data path Minipack

MiniLake

Tomahawk III

IOB FPGA

PIM

DOM FPGA External phys

… x8

Transceivers

BMC

Management path – I2C Minipack

MiniLake

Tomahawk III

IOB FPGA

PIM

DOM FPGA External phys

… x8

… Transceivers

BMC

Management path – MDIO Minipack

MiniLake

BMC

ASIC

IOB FPGA

PIM DOM FPGA External phys

… x8

Transceivers

Programming External PHYs ASIC

Transceivers

Programming External PHYs ASIC

Transceivers

Programming External PHYs ASIC

Transceivers

State management problem

Switch State SwitchState’ SwitchState VlanMap’ PortMap

Port 1

…

RouteTable Map

VlanMap

Vlan2’ Port N

Vlan 1

Vlan2

ArpTable’ ArpTable

ArpTable

RouteTable 0

FBOSS Agent SwSwitch

HwSwitch Interface

State Update Queue

BcmSwitch

FBOSS Agent SwSwitch

“Platform” HwSwitch Interface

State Update Queue

BcmSwitch

PlatformPort PlatformPort PlatformPort

Platform Configuration ØEach port + speed has unique settings (PlatformPortSettings) ØModel config as a map ØHides complexity from SwSwitch

Testing

Testing strategy

ØTargeted tests in realistic lab environments ØDeploy early ØInvest in automation

Scale challenge

Test all combinations Fabric Aggregator

Minipack + FBOSS

7368X4 + FBOSS

7368X4 + EOS

Spine Switch

Minipack + FBOSS

7368X4 + FBOSS

7368X4 + EOS

Fabric Switch

Minipack + FBOSS

7368X4 + FBOSS

7368X4 + EOS

Deploy early

SwSwitch

Invest in automation HwSwitch Interface

ØAutomate as much as possible of our lab testing ØDevelop targeted ASIC data plane tests

BcmSwitch

OpenNSL

Test specific asic functionality, verify, warm boot ASIC

Utilize common FB testing infra

Takeaways ØMinipack is a powerful modular building block for our networks. ØSoftware support for modularity has its challenges. ØWe were able to overcome these challenges through improved hardware layer abstractions and investing heavily in early deployment and automated testing.

Thank you