1 Ethernet switch device driver model (switchdev)
2 ===============================================
3 Copyright (c) 2014 Jiri Pirko <jiri@resnulli.us>
4 Copyright (c) 2014-2015 Scott Feldman <sfeldma@gmail.com>
7 The Ethernet switch device driver model (switchdev) is an in-kernel driver
8 model for switch devices which offload the forwarding (data) plane from the
11 Figure 1 is a block diagram showing the components of the switchdev model for
12 an example setup using a data-center-class switch ASIC chip. Other setups
13 with SR-IOV or soft switches, such as OVS, are possible.
19 +-------------------------------------------------------------------+
22 +--------------+-------------------------------+
26 +----------------------------------------------+
29 sw1p1 + sw1p3 + sw1p5 + eth1
32 +--+----+----+----+-+--+----+---+ +-----+-----+
33 | Switch driver | | mgmt |
34 | (this document) | | driver |
36 +--------------+----------------+ +-----------+
38 kernel | HW bus (eg PCI)
39 +-------------------------------------------------------------------+
41 +--------------+---+------------+
42 | Switch device (sw1) |
44 | | v offloaded data path | mgmt port
46 +--|----|----+----+----+----+---+
60 #include <linux/netdevice.h>
61 #include <net/switchdev.h>
67 Use "depends NET_SWITCHDEV" in driver's Kconfig to ensure switchdev model
68 support is built for driver.
74 On switchdev driver initialization, the driver will allocate and register a
75 struct net_device (using register_netdev()) for each enumerated physical switch
76 port, called the port netdev. A port netdev is the software representation of
77 the physical port and provides a conduit for control traffic to/from the
78 controller (the kernel) and the network, as well as an anchor point for higher
79 level constructs such as bridges, bonds, VLANs, tunnels, and L3 routers. Using
80 standard netdev tools (iproute2, ethtool, etc), the port netdev can also
81 provide to the user access to the physical properties of the switch port such
82 as PHY link state and I/O statistics.
84 There is (currently) no higher-level kernel object for the switch beyond the
85 port netdevs. All of the switchdev driver ops are netdev ops or switchdev ops.
87 A switch management port is outside the scope of the switchdev driver model.
88 Typically, the management port is not participating in offloaded data plane and
89 is loaded with a different driver, such as a NIC driver, on the management port
95 Udev rules should be used for port netdev naming, using some unique attribute
96 of the port as a key, for example the port MAC address or the port PHYS name.
97 Hard-coding of kernel netdev names within the driver is discouraged; let the
98 kernel pick the default netdev name, and let udev set the final name based on a
101 Using port PHYS name (ndo_get_phys_port_name) for the key is particularly
102 useful for dynamically-named ports where the device names its ports based on
103 external configuration. For example, if a physical 40G port is split logically
104 into 4 10G ports, resulting in 4 port netdevs, the device can give a unique
105 name for each port using port PHYS name. The udev rule would be:
107 SUBSYSTEM=="net", ACTION=="add", DRIVER="<driver>", ATTR{phys_port_name}!="", \
108 NAME="$attr{phys_port_name}"
110 Suggested naming convention is "swXpYsZ", where X is the switch name or ID, Y
111 is the port name or ID, and Z is the sub-port name or ID. For example, sw1p1s0
112 would be sub-port 0 on port 1 on switch 1.
117 The switchdev driver must implement the switchdev op switchdev_port_attr_get
118 for SWITCHDEV_ATTR_PORT_PARENT_ID for each port netdev, returning the same
119 physical ID for each port of a switch. The ID must be unique between switches
120 on the same system. The ID does not need to be unique between switches on
123 The switch ID is used to locate ports on a switch and to know if aggregated
124 ports belong to the same switch.
131 If the switchdev driver (and device) only supports offloading of the default
132 network namespace (netns), the driver should set this feature flag to prevent
133 the port netdev from being moved out of the default netns. A netns-aware
134 driver/device would not set this flag and be responsible for partitioning
135 hardware to preserve netns containment. This means hardware cannot forward
136 traffic from a port in one namespace to another port in another namespace.
141 The port netdevs representing the physical switch ports can be organized into
142 higher-level switching constructs. The default construct is a standalone
143 router port, used to offload L3 forwarding. Two or more ports can be bonded
144 together to form a LAG. Two or more ports (or LAGs) can be bridged to bridge
145 L2 networks. VLANs can be applied to sub-divide L2 networks. L2-over-L3
146 tunnels can be built on ports. These constructs are built using standard Linux
147 tools such as the bridge driver, the bonding/team drivers, and netlink-based
148 tools such as iproute2.
150 The switchdev driver can know a particular port's position in the topology by
151 monitoring NETDEV_CHANGEUPPER notifications. For example, a port moved into a
152 bond will see it's upper master change. If that bond is moved into a bridge,
153 the bond's upper master will change. And so on. The driver will track such
154 movements to know what position a port is in in the overall topology by
155 registering for netdevice events and acting on NETDEV_CHANGEUPPER.
157 L2 Forwarding Offload
158 ---------------------
160 The idea is to offload the L2 data forwarding (switching) path from the kernel
161 to the switchdev device by mirroring bridge FDB entries down to the device. An
162 FDB entry is the {port, MAC, VLAN} tuple forwarding destination.
164 To offloading L2 bridging, the switchdev driver/device should support:
166 - Static FDB entries installed on a bridge port
167 - Notification of learned/forgotten src mac/vlans from device
168 - STP state changes on the port
169 - VLAN flooding of multicast/broadcast and unknown unicast packets
174 The switchdev driver should implement ndo_fdb_add, ndo_fdb_del and ndo_fdb_dump
175 to support static FDB entries installed to the device. Static bridge FDB
176 entries are installed, for example, using iproute2 bridge cmd:
178 bridge fdb add ADDR dev DEV [vlan VID] [self]
180 XXX: what should be done if offloading this rule to hardware fails (for
181 example, due to full capacity in hardware tables) ?
183 Note: by default, the bridge does not filter on VLAN and only bridges untagged
184 traffic. To enable VLAN support, turn on VLAN filtering:
186 echo 1 >/sys/class/net/<bridge>/bridge/vlan_filtering
188 Notification of Learned/Forgotten Source MAC/VLANs
189 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
191 The switch device will learn/forget source MAC address/VLAN on ingress packets
192 and notify the switch driver of the mac/vlan/port tuples. The switch driver,
193 in turn, will notify the bridge driver using the switchdev notifier call:
195 err = call_switchdev_notifiers(val, dev, info);
197 Where val is SWITCHDEV_FDB_ADD when learning and SWITCHDEV_FDB_DEL when
198 forgetting, and info points to a struct switchdev_notifier_fdb_info. On
199 SWITCHDEV_FDB_ADD, the bridge driver will install the FDB entry into the
200 bridge's FDB and mark the entry as NTF_EXT_LEARNED. The iproute2 bridge
201 command will label these entries "offload":
204 52:54:00:12:35:01 dev sw1p1 master br0 permanent
205 00:02:00:00:02:00 dev sw1p1 master br0 offload
206 00:02:00:00:02:00 dev sw1p1 self
207 52:54:00:12:35:02 dev sw1p2 master br0 permanent
208 00:02:00:00:03:00 dev sw1p2 master br0 offload
209 00:02:00:00:03:00 dev sw1p2 self
210 33:33:00:00:00:01 dev eth0 self permanent
211 01:00:5e:00:00:01 dev eth0 self permanent
212 33:33:ff:00:00:00 dev eth0 self permanent
213 01:80:c2:00:00:0e dev eth0 self permanent
214 33:33:00:00:00:01 dev br0 self permanent
215 01:00:5e:00:00:01 dev br0 self permanent
216 33:33:ff:12:35:01 dev br0 self permanent
218 Learning on the port should be disabled on the bridge using the bridge command:
220 bridge link set dev DEV learning off
222 Learning on the device port should be enabled, as well as learning_sync:
224 bridge link set dev DEV learning on self
225 bridge link set dev DEV learning_sync on self
227 Learning_sync attribute enables syncing of the learned/forgotton FDB entry to
228 the bridge's FDB. It's possible, but not optimal, to enable learning on the
229 device port and on the bridge port, and disable learning_sync.
231 To support learning and learning_sync port attributes, the driver implements
232 switchdev op switchdev_port_attr_get/set for SWITCHDEV_ATTR_PORT_BRIDGE_FLAGS.
233 The driver should initialize the attributes to the hardware defaults.
238 There are two FDB ageing models supported: 1) ageing by the device, and 2)
239 ageing by the kernel. Ageing by the device is preferred if many FDB entries
240 are supported. The driver calls call_switchdev_notifiers(SWITCHDEV_FDB_DEL,
241 ...) to age out the FDB entry. In this model, ageing by the kernel should be
242 turned off. XXX: how to turn off ageing in kernel on a per-port basis or
243 otherwise prevent the kernel from ageing out the FDB entry?
245 In the kernel ageing model, the standard bridge ageing mechanism is used to age
246 out stale FDB entries. To keep an FDB entry "alive", the driver should refresh
247 the FDB entry by calling call_switchdev_notifiers(SWITCHDEV_FDB_ADD, ...). The
248 notification will reset the FDB entry's last-used time to now. The driver
249 should rate limit refresh notifications, for example, no more than once a
250 second. If the FDB entry expires, ndo_fdb_del is called to remove entry from
251 the device. XXX: this last part isn't currently correct: ndo_fdb_del isn't
252 called, so the stale entry remains in device...this need to get fixed.
257 XXX: Unimplemented. Need to support FDB flush by bridge driver for port and
258 remove both static and learned FDB entries.
260 STP State Change on Port
261 ^^^^^^^^^^^^^^^^^^^^^^^^
263 Internally or with a third-party STP protocol implementation (e.g. mstpd), the
264 bridge driver maintains the STP state for ports, and will notify the switch
265 driver of STP state change on a port using the switchdev op
266 switchdev_attr_port_set for SWITCHDEV_ATTR_PORT_STP_UPDATE.
268 State is one of BR_STATE_*. The switch driver can use STP state updates to
269 update ingress packet filter list for the port. For example, if port is
270 DISABLED, no packets should pass, but if port moves to BLOCKED, then STP BPDUs
271 and other IEEE 01:80:c2:xx:xx:xx link-local multicast packets can pass.
273 Note that STP BDPUs are untagged and STP state applies to all VLANs on the port
274 so packet filters should be applied consistently across untagged and tagged
280 For a given L2 VLAN domain, the switch device should flood multicast/broadcast
281 and unknown unicast packets to all ports in domain, if allowed by port's
282 current STP state. The switch driver, knowing which ports are within which
283 vlan L2 domain, can program the switch device for flooding. The packet should
284 also be sent to the port netdev for processing by the bridge driver. The
285 bridge should not reflood the packet to the same ports the device flooded.
286 XXX: the mechanism to avoid duplicate flood packets is being discuseed.
288 It is possible for the switch device to not handle flooding and push the
289 packets up to the bridge driver for flooding. This is not ideal as the number
290 of ports scale in the L2 domain as the device is much more efficient at
291 flooding packets that software.
296 XXX: complete this section
302 Offloading L3 routing requires that device be programmed with FIB entries from
303 the kernel, with the device doing the FIB lookup and forwarding. The device
304 does a longest prefix match (LPM) on FIB entries matching route prefix and
305 forwards the packet to the matching FIB entry's nexthop(s) egress ports. To
306 program the device, the switchdev driver is called with add/delete ops for IPv4
307 and IPv6 FIB entries. For IPv4, the driver implements switchdev ops:
309 int (*switchdev_fib_ipv4_add)(struct net_device *dev,
310 __be32 dst, int dst_len,
313 u32 nlflags, u32 tb_id);
315 int (*switchdev_fib_ipv4_del)(struct net_device *dev,
316 __be32 dst, int dst_len,
321 to add/delete IPv4 dst/dest_len prefix on table tb_id. The *fi structure holds
322 details on the route and route's nexthops. *dev is one of the port netdevs
323 mentioned in the routes next hop list. If the output port netdevs referenced
324 in the route's nexthop list don't all have the same switch ID, the driver is
325 not called to add/delete the FIB entry.
327 Routes offloaded to the device are labeled with "offload" in the ip route
331 default via 192.168.0.2 dev eth0
332 11.0.0.0/30 dev sw1p1 proto kernel scope link src 11.0.0.2 offload
333 11.0.0.4/30 via 11.0.0.1 dev sw1p1 proto zebra metric 20 offload
334 11.0.0.8/30 dev sw1p2 proto kernel scope link src 11.0.0.10 offload
335 11.0.0.12/30 via 11.0.0.9 dev sw1p2 proto zebra metric 20 offload
336 12.0.0.2 proto zebra metric 30 offload
337 nexthop via 11.0.0.1 dev sw1p1 weight 1
338 nexthop via 11.0.0.9 dev sw1p2 weight 1
339 12.0.0.3 via 11.0.0.1 dev sw1p1 proto zebra metric 20 offload
340 12.0.0.4 via 11.0.0.9 dev sw1p2 proto zebra metric 20 offload
341 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.15
343 XXX: add/del IPv6 FIB API
348 The FIB entry's nexthop list contains the nexthop tuple (gateway, dev), but for
349 the switch device to forward the packet with the correct dst mac address, the
350 nexthop gateways must be resolved to the neighbor's mac address. Neighbor mac
351 address discovery comes via the ARP (or ND) process and is available via the
352 arp_tbl neighbor table. To resolve the routes nexthop gateways, the driver
353 should trigger the kernel's neighbor resolution process. See the rocker
354 driver's rocker_port_ipv4_resolve() for an example.
356 The driver can monitor for updates to arp_tbl using the netevent notifier
357 NETEVENT_NEIGH_UPDATE. The device can be programmed with resolved nexthops
358 for the routes as arp_tbl updates.