This blog was previously published in the SIDN Labs blog on May 25, 2021.
As part of the 2STiC programme, we’ve been evaluating and experimenting with future internet architectures at SIDN Labs. One of the architectures that we are looking at is SCION. SCION stands for Scalability, Control and Isolation on Next-Generation Networks, and is being developed at ETH Zurich, a university in Switzerland, and its spin-off company Anapaya Systems. The goal of SCION is to provide a secure, stable and transparent internet, achieved through secure inter-domain routing and path-aware networking. Previously we gave an introduction to SCION (a recommended read if you’re not familiar with SCION) and showed the technology in action in a video demonstration. In this blog we introduce the high-speed SCION router prototype that we implemented using P4 for the Intel Tofino. Our goal was to evaluate whether we can run a complex protocol such as SCION directly on hardware using the flexibility that is provided by P4, a programming language for networking devices.
SCION
SCION provides path-aware networking. This means that the sender selects the path that the traffic will take through the internet on the level of autonomous systems (ASs). This selected path, called the forwarding path, will be included in every packet. The forwarding path contains so-called hop fields. A hop field contains instructions for one specific AS on how to forward a packet. Also, the hop field is protected using a cryptographic MAC to ensure that only authorised paths are used. As explained in our previous blog, a path can consist of up to three path segments, where each segment is a combination of hop fields. Combined, the hop fields in the different path segments define the path that will be followed. As a result, routers no longer need to store huge tables on where to forward packets with particular IP prefixes. Instead, they only need to look at the current hop field in a packet to determine where to forward it.
P4
P4 is a domain-specific language for packet processing on network devices such as switches and network cards. It allows the addition of support for new protocols to network devices. This has the advantage that it is no longer necessary to wait for vendors to implement new protocols and allows us to easily experiment with new protocols on hardware. P4 can be used to program, for example, software switches, but we used P4 to implement SCION for switches based on the Intel Tofino ASIC.
A key concept in P4 is the use of match-action tables. This allows us to perform a certain action based on a value contained in the packet that we are currently processing. For example, with Ethernet, we match to the destination MAC address contained in the packet. Based on the result of the table lookup, we either send the packet to a specific egress port (if a table entry was found), or we flood the packet to all ports. Matching can be done at line speed in the data plane. However, adding and removing table entries is slower and is done via the control plane.
Implementing SCION
Challenges with cryptography
Based on the SCION protocol specifications, we developed a SCION border router, which is responsible for forwarding packets between neighbouring ASs. Our implementation consists of multiple components: the P4 implementation itself and several control plane components (see Figure 1). When implementing the SCION protocol for the Tofino, the main challenge was the lack of support for cryptographic operations on the Tofino chip. As the hop fields contain a cryptographic MAC, we would need to perform a cryptographic operation for every packet we process. One way to achieve that would be to forward the packets to the control plane for verification. However, that would severely limit the performance, as it is a very slow operation. Luckily, hop fields are used for multiple packets in the basic SCION protocol. This allows us to work around the lack of cryptographic support, by using a table in our implementation that contains all the hop fields that are currently valid. Using this table we can verify the hop fields of incoming packets: if present in the table, the hop field is valid; otherwise it is not and the packet will be dropped. To support this, we modified the SCION control server, which generates the hop fields, to register the generated hop fields at the switch. We also needed to add support for so-called one-hop paths, which are used to communicate with direct neighbours to which no path is yet established. To bootstrap connections, the first packet does not contain a valid hop field for the receiving network and this should be added by the border router. As we cannot generate the hop field in the data plane, we forward incoming packets with a one-hop path to the control plane at the switch. There, a correct hop field is calculated and added to the table, before the updated packet is returned to the data plane and processed as usual. Finally, there is a process that regularly checks for expired hop fields in the table and removes them. We perform the removal of expired hop fields via the control plane as we cannot easily check the timestamp that is included with the hop fields in the data plane.
Figure 1: The control plane of our implementation and its interactions with the data plane
If we take the default settings for SCION beaconing, i.e. beaconing at a five-second interval and a validity time of about six hours, we would need to store around 4,250 hop fields per combination of upstream path and downstream interface. With support for both IPv4 and IPv6, we can currently store about 160,000 hop fields, and with support for either IPv4 or IPv6, we can store about 200,000 hop fields. In the latter case, we could, for example, support three upstream paths and fifteen downstream interfaces. In practice, that will depend on the exact beaconing policy and settings in use. Note that there might still be room for improvement, as we have not yet optimised the implementation so as to maximise the MAC verification table capacity. Also, the number of supported hop fields might be higher if there’s more than one border router in the AS. Also note that we did not take into account the one-hop paths here, but there are unlikely to be many of them, and we could set a relatively short expiration time for their hop fields.
Changes to the SCION headers
When we started to implement the protocol, we ran into several issues with the header format of the SCION protocol, which made it hard or inefficient to implement on our hardware. For example, the forwarding path was structured in a fairly complex manner. The forwarding path was structured as a nested list: the first-level lists contained information fields, and each of those information fields was followed by another list, which contained the hop fields. This structure was then repeated for each path segment. While the structure is fairly straightforward to program in software, it represents a challenge when implemented directly on hardware where all resources need to be statically allocated.
Another example is that the headers contained a field that indicated the types of address that were used for the end hosts. Examples of such address types are IPv4 and IPv6 addresses. Those address types have different lengths. The fact that just the type, not the length, of these addresses was included in the packet meant that every node between the end hosts needed to be aware of the address types. This despite the fact that intermediate nodes only need to know the length of the addresses as they do not process the addresses themselves. We fed those and other observations back to the SCION team, together with suggested solutions. The solutions were then included in a new version of the SCION headers. For example, the issue with the structure of the forwarding path was resolved by first including all the general information fields for the different segments, followed by a list of all the hop fields. This allows us to implement SCION more efficiently and support a higher number of hop fields within a path. The old and new path layout designs are illustrated in Figure 2.
Figure 2: The old and new layouts of the forwarding path in the SCION header
Implemented functionality in our prototype
We have implemented the basic forwarding functionality for SCION. However, we do not yet support all SCION functionality. For example, we have not yet added support for peering or the return of SCMP error messages. In addition, with the EPIC and COLIBRI extensions for SCION that provide additional security and QoS, the cryptographic MACs in the hop fields will be bound to individual packets. As a result, for those packets we will no longer be able to make use of the approach where we store the complete hop fields in a table for verification. Depending on how many packets make use of those extensions, it might be a solution to forward them to the control plane at the switch for verification or to an external node that does provide cryptographic support on hardware.
Running our prototype
We started by testing the implementation on the Tofino software model to verify its functionality. Once that worked, we knew that the implementation would also run without real issues on the actual hardware. To run our implementation in practice, we made use of the 2STiC testbed, which consists of P4 programmable equipment located on the premises of various 2STiC consortium partners in the Netherlands. For our tests, we used three Edgecore Wedge 100BF-32X switches, which have 32 QSFP28 ports supporting up to 100 Gbit/sec per port. In total, each switch has a maximum throughput of 3.2 Tbit/sec. In addition, we are in the process of connecting one APS Networks BF2556X-1T switch, which provides a maximum throughput of 2.0 Tbit/s through its 48 SFP28 and 8 QSFP28 ports.
We ran our implementation on a small topology consisting of three SCION ASs at different sites of the 2STiC testbed and are in the process of connecting a fourth AS (depicted in Figure 3 below). Every SCION AS in our test consists of one of the programmable switches above to act as the border router and one server connected to the corresponding switch to provide the other SCION services. With this setup, we showed that the implementation works in a practical scenario. A more detailed evaluation, including a performance analysis, will be done in the future.
Figure 3: The topology with which we tested our implementation on the 2STiC testbed, including one additional AS which has yet to be connected. In this topology, AS 112 is the core of the isolation domain, which contains all ASs.
Conclusion
With our implementation, we have shown the strength and flexibility of P4 and programmable network devices, which allowed us to run an implementation of the SCION protocol directly on an ASIC. At the same time, we have shown that we can run SCION on hardware at high speeds.
Our implementation is now available as open source. Ideas for future work include optimisation of the implementation to achieve even higher performance, as well as adding additional functionality and experimenting with support for extensions to SCION, such as EPIC and COLIBRI. We plan to discuss our implementation and the lessons learned in more detail in an upcoming publication.
