Terraform Multicloud Quickstart
We are living in an ephemeral world
And I am an ephemeral girl
Welcome!#
This quickstart tutorial creates a lab environment for using ZeroTier in combination with multiple Terraform cloud providers. If you're a ZeroTier user that's new to Terraform, You might be looking for the Terraform Quickstart instead.
If you're a Terraform user that's new to ZeroTier, you're in the right place. Make yourself a coffee a and buckle up.
Prerequisites#
To follow along step by step, you will need:
- A Github account,
- A ZeroTier Central account,
- A Terraform Cloud account.
- Accounts on multiple cloud providers
The full-blown multicloud demo uses:
- Digital Ocean
- Amazon Web Services
- Google Compute Engine
- Microsoft Azure
- Oracle Cloud Infrastructure
- Alibaba Cloud
- IBM Cloud
- Vultr
- Equinix Metal
That's a lot of service providers. You'll need at least two for demonstration purposes, but I recommend using them all for dramatic effect. Digital Ocean was chosen at random to provide DNS service for the lab.
The first time through, you will encounter a few hurdles. Each cloud vendor brings their own special brand of pain. For example, on AWS, you will need to accept the Marketplace agreement for the Ubuntu AMI. On GCP, you will be prompted to enable Cloud APIs. Others have stringent account verification procedures.
To lower the bar of entry, you can toggle which clouds are
enabled variables.tf. The process for creating service accounts and
gathering credentials is outside the scope of this document, but
should be pretty straight forward.
You can do this. We believe in you.
Import repo#
Import the Terraform Multicloud Quickstart to your Github account. We are "importing" instead of "cloning" so that we can set the repository as private. If you're comfortable on the command line, feel free to clone to your laptop and commits from there, otherwise, we will use Github's in-browser editing feature to drive the tutorial.
Create a Terraform workspace#
Next, we create a Terraform workspace and attach it to our private Github repository. Be sure to select version control workflow, select the correct Github account, (we want the private copy, not the original), and give it a unique name.
Create ZeroTier Central variables#
Next, we will use Terraform to create some resources in the ZeroTier Central API. Before we can do this, we need to give Terraform credentials as Environment Variables.
Create ZeroTier Central resources#
Examine main.tf At the top, you will see Terraform resources for
creating Identities,
Networks,
and Members. There
is also a Token
that we will use later.
Normally, to kick off a Terraform plan, we would make commits to our repository. However, since we have a fresh workspace and nothing to change, we'll need to manually queue our first plan in the Terraform webUI.
Confirm the plan by clicking "Confirm & Apply"
Congratulations! You have just successfully created your first ZeroTier network using Terraform! Go over to ZeroTier Central and check out your new network. Alice and Bob are both authorized onto the network, but don't worry, they aren't real. We will replace them shortly.
Edit variables.tf#
Terraform has two kinds of variables. We have already seen some Environment Variables, which we used to make credentials available to the ZeroTier Terraform Provider. The other kinds of variables are known as Input Variables. We will use these to supply some usernames and SSH keys, as well as toggle which clouds we want to use.
Use Github's editor to set the users, devices, and instances
variables. Replace Alice and Bob's information with your own SSH keys
and ZeroTier Node ID's. In the instances variable, toggle the clouds
you plan on using to enabled.
Save your work by clicking "Commit changes" at the bottom of the page.
Go back to your workspace and see that it now says "Planned". Every time a commit is pushed to the repo, Terraform will queue a plan. This is the essence of the "Version control workflow" we selected earlier.
Navigate through "Runs" and then "confirm and apply". There is a setting to make this step automatic, but we will leave it manual for now.
We now have pre-generated ZeroTier Identities that we will inject into our cloud instances when we bring them up. They are stored in the workspace's Terraform State on Terraform Cloud. Be careful about who has access to your account, as well as source repository that drives it.
Create Digital Ocean resources#
Add your DIGITALOCEAN_TOKEN to the workspace's Environment Variables using the same procedure as before.
Next, edit main.tf and uncomment the Digital Ocean module.
Join laptop to Network#
The ZeroTier Network can be found in the Terraform output. Find it by navigating to the "Outputs" tab of the latest run.
You can also find it in the ZeroTier Central webUI.
Join your laptop to the network. Make sure to check "Allow DNS"
You will be able to SSH into the box. If this does not work, make sure
username, ssh_pubkey and member_id are correct in variables.tf.
Spin up Multiple Clouds#
Baton Bunny - Warner Bros. 1959
Next, spin up the rest of the cloud instances. Go through each cloud provider,
one by one, adding Environment Variables to the Terraform workspace,
then uncommenting out the corresponding module in main.tf.
Here's a complete list of Environment Variables to set if you plan on spinning up every cloud the tutorial supports.
Hit the web servers#
Each node is running a web server with an example nginx page, accessible with an internal DNS address.
For example, http://aws.demo.lab.
Understanding ZeroTier VL2#
ZeroTier networks are virtual Ethernet switches. This means that anything you can do on a physical LAN segment, ZeroTier can over the Internet, securely, across clouds, and through NAT devices.
Down the Rabbit Hole - Valerie Hinojosa 2006
Ping all the boxen (v4)#
Examine the ARP cache#
As you can see, the ARP table now contains an entry for each node on our network, just as it would on a local Ethernet network.
Examine the interfaces#
Run the ip link command to examine the interfaces on each box.
You'll see a virtual Ethernet interface for each ZeroTier network the node is joined to. (in this case, one)
The name of the interface is derived from the network ID it is joined to. Note that the name of the interface is the same on each machine.
Ethernet Tapping#
You may have noticed the flow_rules
section in the zerotier_network while examining main.tf
earlier.
We will use these to gain visibility into our network with tshark. You
can see them reflected in the Central WebUI under the "Flow Rules"
section for the demo.lab network. They are documented in in-depth in
chapter 3 of the Design Whitepaper.
Edit flow_rules.tpl, uncommenting the tee rule.
Flow Rules are applied to every member of the network. tee tells
ZeroTier to mirror a copy of every packet to Digital Ocean. Apply the
rule set by saving the file and running Terraform.
Watching traffic with tshark#
On the Digital Ocean machine, view traffic by running tshark on your network's ZeroTier interface.
Open another terminal window, log into AWS, and ping GCP.
You will be able to observe the traffic from Digital Ocean.
You'll see duplicates, as the tee is picking up both the incoming and outgoing packets from both nodes.
The watch rule, combined with the inbound characteristic is a
little friendlier.
Edit flow_rules.tpl, this time using the watch rule.
Apply the rule set again with Terraform.
You can also see the the traffic from your laptop when hitting the web servers. Load the page on IBM Cloud by visiting http://ibm.demo.lab, and observe the traffic in your Digital Ocean terminal.
Manually manipulate IP addresses#
Because ZeroTier behaves like Ethernet, we can assign multiple IP addresses to an interface, just like on a physical network.
Our network is configured as a /16, which means we can add any of
the 65,536 available IPv4 addresses to any zerotier interface, on any
machine, and it will work as expected via ARP resolution.
Experiment with this by adding ip addresses from the command line.
Clean up after yourself by deleting them.
Native Container Routing#
Amy Gizienski - whale
We would be remiss not to mention containers in the year 2021. A great attribute of Layer 2 networks is that containers can talk directly to each other using native routing.
No really.
Pick a box, any box, and start a shell in Docker.
Then, pick another random box and do the same.
Ping the IPv4 and IPv6 addresses of the container, from the other container.
What black magic is this? Let's examine the routing table.
At the bottom of the lab boot script we've installed a routing daemon and gave it a simple OSPF configuration. This propagates the routes of the Docker networks among all the instances so they can talk over the ZeroTier network.
But what about IPv6? For that, we've enabled the ZeroTier 6PLANE.
ZeroTier 6PLANE encodes the network's name (8bd5124fd6f45ffe) into IPv6 addresses, and emulates NDP. This allows for private IPv6 networking to work at massive scales, without actually having to send the discovery traffic.
Tear it all down#
When you're done experimenting with the lab, tear everything down by queueing a destroy plan.
