Multipath allows the simultaneous (or conditional) aggregation of multiple physical links into a bond for increased total throughput, load balancing, redundancy, and fault tolerance. There is a set of standard bonding policies available that can be used right out of the box with no configuration. These policies are inspired by the policies offered by the Linux kernel. A bonding policy can be used easily without specifying any additional parameters. For example:
- active-backup: Use only one primary link at a time and failover to another designated link.
- broadcast: Duplicate traffic across all available links at all times.
- balance-rr: Stripe packets across multiple links (not for use with TCP.)
- balance-xor: Hash flows to specific links.
- balance-aware: Auto-balance flows across links.
To customize a bonding policy simply specify a standard policy as
basePolicy and override its parameters. Note that a custom policy cannot have the same name as a standard policy. For example, to create a more rapid
active-backup policy that will failover
1 second after it detects a link failure:
You can only override parameters that are valid for the chosen
basePolicy. The list below shows parameters that can apply to any policy. Parameters that are unique to a given policy type are described in that policy's section.
Bonds are composed of multiple links (known paths over system network interfaces). By default if no interfaces are specified ZeroTier will attempt to use all network interfaces, including your expensive wireless links. In order to avoid this you should tell ZeroTier which interfaces are okay to talk on and also under what conditions. If a set of links is defined, ZeroTier will use only those links and ignore everything else.
The behavior of links can be changed by setting various properties:
A warm spare is a link that is not actively used to process traffic until all other available links are down. Note however that some ambient probing traffic is still sent to guarantee its performance. Spare links are not available when using the
broadcast policy. Cold spares are left as an exercise for the reader.
It is possible to use a policy for one peer and another policy for a different peer. For instance, if one were to want
active-backup for all peers by default but for certain peers to be bonded with a custom load-balanced bond such as
my-custom-balance-xor one could do the following:
Traffic is sent on only one link at any time. A different link becomes active if the current link fails. This mode provides fault tolerance with a nearly immediate failover depending on the
failoverInterval you set. This mode does not increase total throughput. If no
spare links are defined ZeroTier will attempt to pick the best one.
primary|spareLink option which specifies which link is the primary device. The specified device is intended to always be the active link while it is available. There are exceptions to this behavior when using different
linkSelectMethodmodes. There can only be one
primarylink in this bonding policy.
linkSelectMethod: Specifies the selection policy for the active link during failure and/or recovery events. This is similar to the Linux Kernel's
primary_reselectoption but with a minor extension:
optimize: (default if user provides no failover guidance) The primary link can change periodically if a superior link is detected.
always: (default when links are explicitly specified): Primary link regains status as active link whenever it comes back up.
better: Primary link regains status as active link when it comes back up and (if) it is better than the currently-active link.
failure: Primary link regains status as active link only if the currently-active link fails.
Traffic is sent on all available links simultaneously. This mode provides fault tolerance and effectively immediate failover due to transmission redundancy. This mode is a poor utilization of throughput resources and will not increase throughput but can prevent packet loss during a link failure.
Traffic is striped across multiple links. This mode offers partial fault tolerance immediately, full fault tolerance eventually. This policy is unaware of protocols and is primarily intended for use with protocols that are not sensitive to reordering delays. The only option available for this policy is
packetsPerLink which specifies the number of packets to transmit via a link before moving to the next link in the round-robin sequence. When set to
0 a link is chosen at random for each outgoing packet (doing so would be inefficient). The default value is
64, lower values can begin to add overhead to packet processing. This mode is not suitable for traffic that is sensitive to re-ordering such as TCP.
Traffic is categorized into flows based on source port, destination port, and protocol type these flows are then hashed onto available links where they will persist for the duration of their life. Traffic that does not have an assigned port (such as ICMP pings) will be randomly distributed across links. This mode offers partial fault tolerance immediately, full fault tolerance eventually. This mode is suitable for traffic that is sensitive to re-ordering such as TCP.
This mode operates similarly to
balance-xor in that it hashes flows onto specific links. However, it may reassign flows mid-conversation and perform other types of optimizations. This mode may surprise you more often than
balance-xor by causing re-ordering delays for certain flows but it should lead to a better total experience when all flows are considered. This policy allows you to specify not only relative link capacities but also a notion of quality expressed as a weighted vector with a sum of
1.0. See below:
In the above example if any links begin to show signs of saturation (for instance if latency increases beyond
400ms) flows will be moved from it until it is no longer judged to be saturated.
As seen in the balance-aware example configuration, you can provide hints to ZeroTier as to when a link is no longer suitable for use. You can set limits on the following:
lat_max: Maximum (mean) latency observed over many samples
pdv_max: Maximum packet delay variance (similar to jitter)
plr_max: Maximum packet loss ratio
per_max: Maximum packet error ratio
Then, weights must also be provided to tell ZeroTier how important your limits are (as a reminder, the weights must sum to
If any one of these limits are violated ZeroTier will attempt to avoid assigning new flows to the link in question as well as begin moving flows from that link to other higher quality links. ZeroTier will first try to find a link that doesn't violate any of your limits but if it is unable to do so it will pick the next best according to a quality ranking derived from your weights. There is no guarantee that links will be entirely avoided or that all of their flows will be moved if there are no better links to move to so these limits are merely strong suggestions to ZeroTier.
Any set of physical links regardless of their relative performance may be combined using any of the policies. However if the relative performance among the links vary too wildly (for instance a fiber link and an LTE link) you may find that blindly striping or hashing traffic across them doesn't turn out so well. For these scenarios it is recommended that you use balance-aware and provide
linkQuality hints to ZeroTier.
The term is meant to be a one-size-fits-all numerical value which represents the diameter of the pipe so to speak. That is, how much traffic it can process before it becomes clogged and congestion controls kick in. This will suggest to ZeroTier that it should assign traffic flows proportionally.
Providing quality hints to ZeroTier can be done in conjunction with the aforementioned
capacity property without issue. See the balance-aware section for an example of how to do this.
The above configuration will roughly result in the following allocation:
A good rule of thumb would be to express your known link speeds in units of
Mbit/s. This isn't strictly necessary as long as the quantities make sense relative to each other.
#Using the CLI
Currently most configuration is handled via manual editing of each node's
local.conf. There are only a few available CLI commands.
To view bonds to all peers:
If you'd like to ingest this data into your own monitoring solution use
zerotier-cli -j bond list to emit a JSON blob instead.
To see the current state of a bond to a given peer:
The reported allocation percentages have an uncertainty value of
To forcibly rotate to a different link in an