The previous article on this topic generated several questions, which suggested that a more thorough exploration of OneFS source-based routing (SBR) is likely warranted. So here goes…
At its essence, network routing is the process of selecting a path for data traffic, either within a network or traversing multiple networks. The aim is to endure efficient data flow across subnets, while maintaining bandwidth and minimizing congestion. Routers, layer 3 switches, multi-homed system, etc, make routing decisions based on packet header addresses and routing tables, which record the paths packets should take to reach their destinations.
IP packet headers have the following form, with the source and destination addresses located towards the end of the header section, before the packet’s payload.
Routing is typically either static, using manually enter routing statements and rules, or dynamic, via routing protocols such as RIP, OSPF, etc.
While the nomenclature might suggest that OneFS source-based routing would route traffic based on a source IP address, instead SBR actually operates by dynamically creating per-subnet default routes. The gateway is derived from the subnet configuration, and, as such, gateways need to be defined for each subnet.
New cluster deployments running OneFS 9.8 and automatically have SBR enabled, whereas legacy clusters upgrading to 9.8 preserve their existing SBR configuration, whether on or off. While SBR is disabled by default in OneFS 9.7 and earlier releases, it can, if desired, be easily enabled from either the CLI or WebUI.
SBR is configured globally and, as such, is either on or off across the entire cluster and its network pools and subnets. OneFS 9.7 and earlier supports only the IPv4 protocol, whereas OneFS 9.8 and later also accommodate IPv6 subnets.
SBR can be instantly enabled on a PowerScale cluster by running the following CLI command:
# isi network external modify --sbr 1 # isi network external view | grep -i source Source Based Routing: True
Or from the WebUI under Cluster management > Network configuration > Settings:
Similarly, SBR can be disabled as follows:
# isi network external modify --sbr 0 # isi network external view | grep -i source Source Based Routing: False
Under the hood, SBR uses the FreeBSD ‘ipfw’ utility (as does the OneFS firewall) to record and manage its routing rules.
For example, with SBR disabled, querying ipfw on a cluster shows a single ‘any to any’ rule:
# isi network external view | grep -i source Source Based Routing: False # ipfw show 65535 11839927994 7560033188891 allow ip from any to any
By way of contrast, when SBR is enabled, a number of new, higher priority ‘allow’ rules for each NIC and gateway ‘fwd’ rules are added above the ‘any to any’ rule:
# isi network external view | grep -i source Source Based Routing: True # ipfw show 60000 16 33391 allow ip from any to any via lo0 out 60001 0 0 allow ip from any to ff02::1:ff00:0/104 out 60002 116082 112914089 allow ip from any to any via mce0 out 60003 217150 138771611 allow ip from any to any via mce1 out 60004 0 0 allow ip from any to any via ue0 out 60005 0 0 allow ip from any to fe80::/10 out 60006 0 0 allow ip from any to ff02::1 out 62000 0 0 fwd 2620:0:170:7c0f::1 ip from 2620:0:170:7c0f::/64 to not 2620:0:170:7c0f::/64 out 62001 121 94788 fwd 10.30.1.1 ip from 10.30.1.0/22 to not 10.30.1.0/22 out 65535 11842048952 7561181109905 allow ip from any to any
In this example node’s case, on a cluster running OneFS 9.8, there is one IPv4 subnet and one IPv6 subnet:
# isi network subnets list ID Subnet Gateway|Priority Pools SC Service Addrs Firewall Policy ------------------------------------------------------------------------------------------------ groupnet0.subnet0 10.30.1.0/22 10.30.1.1|10 pool0 10.30.1.100-10.30.1.110 default_subnets_policy groupnet0.subnet1 2620:0:170:7c0f::/64 2620:0:170:7c0f::1|20 ipv6pool 2620:0:170:7c0f::4-2620:0:170:7c0f::4 default_subnets_policy ------------------------------------------------------------------------------------------------ Total: 2
So enabling SBR on this cluster results in the creation of a ‘fwd’ rule for each subnet:
# ipfw show | grep fwd 62000 33 2640 fwd 2620:0:170:7c0f::1 ip from 2620:0:170:7c0f::/64 to not 2620:0:170:7c0f::/64 out 62001 145794 140490002 fwd 10.30.1.1 ip from 10.30.1.0/22 to not 10.30.1.0/22 out
Please note that the ‘ipfw’ command should not be used to modify the OneFS routing rules (or firewall table) directly!
By way of a OneFS packet routing example, take the following network topology where three clients, each on separate subnets, are connecting to a PowerScale cluster:
The default gateway is the path for all traffic intended for clients not on the local subnet and not covered by a routing table entry. Utilizing SBR does not negate the need for a default gateway, since SBR effectively overrides the default gateway (but not static routes).
Note that SBR is not simple packet reflection. Instead, it’s the dynamic creation of per-subnet default routes. The router used as the gateway is derived from the FlexNet subnet definitions within the subnet configuration. As such, a gateway needs to be specified for each subnet.
In addition to a gateway address, each subnet also has a defined priority. For example:
Or via the CLI:
# isi network subnets modify groupnet0.subnet1 --gateway 10.30.1.1 --gateway-priority 10
With SBR disabled, the highest priority gateway (ie. the gateway with the lowest reachable value) is used as the default route.
Once SBR is enabled, OneFS examines the FlexNet config for each subnet, and then creates ipfw rules that look at the source IP address from the cluster side and force the next-hop to be the gateway IP defined for the subnet which contains that IP address.
In the previous example with three clients on separate subnets connecting to a cluster, when traffic arrives from a subnet that is unreachable via the default gateway, the following routing rules will be added via ipfw:
The mechanism for adding ipfw rules is stateless, and SBR relies on the source IP address that transmits traffic to the cluster.
A session must be initiated from the source subnet for a corresponding ipfw rule to be created. Also, unless the cluster has received traffic that originated from a subnet has no route to the default gateway, OneFS transmits traffic it originates through the default gateway.
In the next article in this series. We’ll take a look at SBR and its interrelationship with static routes and other OneFS networking components.