nm-cloud-setup: NetworkManager Reference Manual

nm-cloud-setup

nm-cloud-setup — Overview of Automatic Network Configuration in Cloud

Overview

When running a virtual machine in a public cloud environment, it is desirable to automatically configure the network of that VM. In simple setups, the VM only has one network interface and the public cloud supports automatic configuration via DHCP, DHCP6 or IPv6 autoconf. However, the virtual machine might have multiple network interfaces, or multiple IP addresses and IP subnets on one interface which cannot be configured via DHCP. Also, the administrator may reconfigure the network while the machine is running. NetworkManager's nm-cloud-setup is a tool that automatically picks up such configuration in cloud environments and updates the network configuration of the host.

Multiple cloud providers are supported. See the section called “Supported Cloud Providers”.

Use

The goal of nm-cloud-setup is to be configuration-less and work automatically. All you need is to opt-in to the desired cloud providers (see the section called “Environment Variables”) and run /usr/libexec/nm-cloud-setup.

Usually this is done by enabling the nm-cloud-setup.service systemd service and let it run periodically. For that there is both a nm-cloud-setup.timer systemd timer and a NetworkManager dispatcher script.

Details

nm-cloud-setup configures the network by fetching the configuration from the well-known meta data server of the cloud provider. That means, it already needs the network configured to the point where it can reach the meta data server. Commonly that means, that a simple connection profile is activated that possibly uses DHCP to get the primary IP address. NetworkManager will create such a profile for ethernet devices automatically if it is not configured otherwise via "no-auto-default" setting in NetworkManager.conf. One possible alternative may be to create such an initial profile with nmcli device connect "$DEVICE" or nmcli connection add type ethernet ....

By setting the user-data org.freedesktop.nm-cloud-setup.skip=yes on the profile, nm-cloud-setup will skip the device.

nm-cloud-setup modifies the run time configuration akin to nmcli device modify. With this approach, the configuration is not persisted and only preserved until the device disconnects.

/usr/libexec/nm-cloud-setup

The binary /usr/libexec/nm-cloud-setup does most of the work. It supports no command line arguments but can be configured via environment variables. See the section called “Environment Variables” for the supported environment variables.

By default, all cloud providers are disabled unless you opt-in by enabling one or several providers. If cloud providers are enabled, the program tries to fetch the host's configuration from a meta data server of the cloud via HTTP. If configuration could be not fetched, no cloud provider are detected and the program quits. If host configuration is obtained, the corresponding cloud provider is successfully detected. Then the network of the host will be configured.

It is intended to re-run nm-cloud-setup every time when the configuration (maybe) changes. The tool is idempotent, so it should be OK to also run it more often than necessary. You could run /usr/libexec/nm-cloud-setup directly. However it may be preferable to restart the nm-cloud-setup systemd service instead or use the timer or dispatcher script to run it periodically (see below).

nm-cloud-setup.service systemd unit

Usually /usr/libexec/nm-cloud-setup is not run directly, but only by systemctl restart nm-cloud-setup.service. This ensures that the tool only runs once at any time. It also allows to integrate with the nm-cloud-setup systemd timer, and to enable/disable the service via systemd.

As you need to set environment variable to configure nm-cloud-setup binary, you can do so via systemd override files. Try systemctl edit nm-cloud-setup.service.

nm-cloud-setup.timer systemd timer

/usr/libexec/nm-cloud-setup is intended to run whenever an update is necessary. For example, during boot when when changing the network configuration of the virtual machine via the cloud provider.

One way to do this, is by enabling the nm-cloud-setup.timer systemd timer with systemctl enable --now nm-cloud-setup.timer.

/usr/lib/NetworkManager/dispatcher.d/90-nm-cloud-setup.sh

There is also a NetworkManager dispatcher script that will run for example when an interface is activated by NetworkManager. Together with the nm-cloud-setup.timer systemd timer this script is to automatically pick up changes to the network.

The dispatcher script will do nothing, unless the systemd service is enabled. To use the dispatcher script you should therefor run systemctl enable nm-cloud-setup.service once.

Environment Variables

The following environment variables are used to configure /usr/libexec/nm-cloud-setup. You may want to configure them with a drop-in for the systemd service. For example by calling systemctl edit nm-cloud-setup.service and configuring [Service] Environment=, as described in systemd.exec(5) manual.

NM_CLOUD_SETUP_LOG: control the logging verbosity. Set it to one of TRACE, DEBUG, INFO, WARN, ERR or OFF. The program will print message on stdout and the default level is WARN.
NM_CLOUD_SETUP_AZURE: boolean, whether Microsoft Azure support is enabled. Defaults to no.
NM_CLOUD_SETUP_EC2: boolean, whether Amazon EC2 (AWS) support is enabled. Defaults to no.
NM_CLOUD_SETUP_GCP: boolean, whether Google GCP support is enabled. Defaults to no.
NM_CLOUD_SETUP_ALIYUN: boolean, whether Alibaba Cloud (Aliyun) support is enabled. Defaults to no.

Example Setup for Configuring and Predeploying nm-cloud-setup

As detailed before, nm-cloud-setup needs to be explicitly enabled. As it runs as a systemd service and timer, that basically means to enable and configure those. This can be done by dropping the correct files and symlinks to disk.

The following example enables nm-cloud-setup for Amazon EC2 cloud:

dnf install -y NetworkManager-cloud-setup

mkdir -p /etc/systemd/system/nm-cloud-setup.service.d
cat > /etc/systemd/system/nm-cloud-setup.service.d/10-enable-ec2.conf << EOF
[Service]
Environment=NM_CLOUD_SETUP_EC2=yes
EOF

# systemctl enable nm-cloud-setup.service
mkdir -p /etc/systemd/system/NetworkManager.service.wants/
ln -s /usr/lib/systemd/system/nm-cloud-setup.service /etc/systemd/system/NetworkManager.service.wants/nm-cloud-setup.service

# systemctl enable nm-cloud-setup.timer
mkdir -p /etc/systemd/system/timers.target.wants/
ln -s /etc/systemd/system/timers.target.wants/nm-cloud-setup.timer /usr/lib/systemd/system/nm-cloud-setup.timer

# systemctl daemon-reload

Supported Cloud Providers

Amazon EC2 (AWS)

For AWS, the tools tries to fetch configuration from http://169.254.169.254/. Currently, it only configures IPv4 and does nothing about IPv6. It will do the following.

First fetch http://169.254.169.254/latest/meta-data/ to determine whether the expected API is present. This determines whether EC2 environment is detected and whether to proceed to configure the host using EC2 meta data.
Fetch http://169.254.169.254/2018-09-24/meta-data/network/interfaces/macs/ to get the list of available interface. Interfaces are identified by their MAC address.
Then for each interface fetch http://169.254.169.254/2018-09-24/meta-data/network/interfaces/macs/$MAC/subnet-ipv4-cidr-block and http://169.254.169.254/2018-09-24/meta-data/network/interfaces/macs/$MAC/local-ipv4s. Thereby we get a list of local IPv4 addresses and one CIDR subnet block.
Then nm-cloud-setup iterates over all interfaces for which it could fetch IP configuration. If no ethernet device for the respective MAC address is found, it is skipped. Also, if the device is currently not activated in NetworkManager or if the currently activated profile has a user-data org.freedesktop.nm-cloud-setup.skip=yes, it is skipped.

Then, the tool will change the runtime configuration of the device.
- Add static IPv4 addresses for all the configured addresses from local-ipv4s with prefix length according to subnet-ipv4-cidr-block. For example, we might have here 2 IP addresses like "172.16.5.3/24,172.16.5.4/24".
- Choose a route table 30400 + the index of the interface and add a default route 0.0.0.0/0. The gateway is the first IP address in the CIDR subnet block. For example, we might get a route "0.0.0.0/0 172.16.5.1 10 table=30401".
- Finally, add a policy routing rule for each address. For example "priority 30401 from 172.16.5.3/32 table 30401, priority 30401 from 172.16.5.4/32 table 30401".
With above example, this roughly corresponds for interface eth0 to nmcli device modify "eth0" ipv4.addresses "172.16.5.3/24,172.16.5.4/24" ipv4.routes "0.0.0.0/0 172.16.5.1 10 table=30401" ipv4.routing-rules "priority 30401 from 172.16.5.3/32 table 30401, priority 30401 from 172.16.5.4/32 table 30401". Note that this replaces the previous addresses, routes and rules with the new information. But also note that this only changes the run time configuration of the device. The connection profile on disk is not affected.

Google Cloud Platform (GCP)

For GCP, the meta data is fetched from URIs starting with http://metadata.google.internal/computeMetadata/v1/ with a HTTP header "Metadata-Flavor: Google". Currently, the tool only configures IPv4 and does nothing about IPv6. It will do the following.

First fetch http://metadata.google.internal/computeMetadata/v1/instance/id to detect whether the tool runs on Google Cloud Platform. Only if the platform is detected, it will continue fetching the configuration.
Fetch http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/ to get the list of available interface indexes. These indexes can be used for further lookups.
Then, for each interface fetch http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/$IFACE_INDEX/mac to get the corresponding MAC address of the found interfaces. The MAC address is used to identify the device later on.
Then, for each interface with a MAC address fetch http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/$IFACE_INDEX/forwarded-ips/ and then all the found IP addresses at http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/$IFACE_INDEX/forwarded-ips/$FIPS_INDEX.
At this point, we have a list of all interfaces (by MAC address) and their configured IPv4 addresses.

For each device, we lookup the currently applied connection in NetworkManager. That implies, that the device is currently activated in NetworkManager. If no such device was in NetworkManager, or if the profile has user-data org.freedesktop.nm-cloud-setup.skip=yes, we skip the device. Now for each found IP address we add a static route "$FIPS_ADDR/32 0.0.0.0 100 type=local" and reapply the change.

The effect is not unlike calling nmcli device modify "$DEVICE" ipv4.routes "$FIPS_ADDR/32 0.0.0.0 100 type=local [,...]" for all relevant devices and all found addresses.

Microsoft Azure

For Azure, the meta data is fetched from URIs starting with http://169.254.169.254/metadata/instance with a URL parameter "?format=text&api-version=2017-04-02" and a HTTP header "Metadata:true". Currently, the tool only configures IPv4 and does nothing about IPv6. It will do the following.

First fetch http://169.254.169.254/metadata/instance?format=text&api-version=2017-04-02 to detect whether the tool runs on Azure Cloud. Only if the platform is detected, it will continue fetching the configuration.
Fetch http://169.254.169.254/metadata/instance/network/interface/?format=text&api-version=2017-04-02 to get the list of available interface indexes. These indexes can be used for further lookups.
Then, for each interface fetch http://169.254.169.254/metadata/instance/network/interface/$IFACE_INDEX/macAddress?format=text&api-version=2017-04-02 to get the corresponding MAC address of the found interfaces. The MAC address is used to identify the device later on.
Then, for each interface with a MAC address fetch http://169.254.169.254/metadata/instance/network/interface/$IFACE_INDEX/ipv4/ipAddress/?format=text&api-version=2017-04-02 to get the list of (indexes of) IP addresses on that interface.
Then, for each IP address index fetch the address at http://169.254.169.254/metadata/instance/network/interface/$IFACE_INDEX/ipv4/ipAddress/$ADDR_INDEX/privateIpAddress?format=text&api-version=2017-04-02. Also fetch the size of the subnet and prefix for the interface from http://169.254.169.254/metadata/instance/network/interface/$IFACE_INDEX/ipv4/subnet/0/address/?format=text&api-version=2017-04-02. and http://169.254.169.254/metadata/instance/network/interface/$IFACE_INDEX/ipv4/subnet/0/prefix/?format=text&api-version=2017-04-02.
At this point, we have a list of all interfaces (by MAC address) and their configured IPv4 addresses.

For each device, we lookup the currently applied connection in NetworkManager. That implies, that the device is currently activated in NetworkManager. If no such device was in NetworkManager, or if the profile has user-data org.freedesktop.nm-cloud-setup.skip=yes, we skip the device. Now for each found IP address we add a static address "$ADDR/$SUBNET_PREFIX". Also we configure policy routing by adding a static route "$ADDR/$SUBNET_PREFIX $GATEWAY 10, table=$TABLE" where $GATEWAY is the first IP address in the subnet and table is 30400 plus the interface index. Also we add a policy routing rule "priority $TABLE from $ADDR/32 table $TABLE".

The effect is not unlike calling nmcli device modify "$DEVICE" ipv4.addresses "$ADDR/$SUBNET [,...]" ipv4.routes "$ADDR/32 $GATEWAY 10 table=$TABLE" ipv4.routing-rules "priority $TABLE from $ADDR/32 table $TABLE" for all relevant devices and all found addresses.

Alibaba Cloud (Aliyun)

For Aliyun, the tools tries to fetch configuration from http://100.100.100.200/. Currently, it only configures IPv4 and does nothing about IPv6. It will do the following.

First fetch http://100.100.100.200/2016-01-01/meta-data/ to determine whether the expected API is present. This determines whether Aliyun environment is detected and whether to proceed to configure the host using Aliyun meta data.
Fetch http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/ to get the list of available interface. Interfaces are identified by their MAC address.
Then for each interface fetch http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/$MAC/vpc-cidr-block , http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/$MAC/private-ipv4s and http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/$MAC/netmask. Thereby we get a list of private IPv4 addresses, one CIDR subnet block and private IPv4 addresses prefix.
Then nm-cloud-setup iterates over all interfaces for which it could fetch IP configuration. If no ethernet device for the respective MAC address is found, it is skipped. Also, if the device is currently not activated in NetworkManager or if the currently activated profile has a user-data org.freedesktop.nm-cloud-setup.skip=yes, it is skipped.

Then, the tool will change the runtime configuration of the device.
- Add static IPv4 addresses for all the configured addresses from private-ipv4s with prefix length according to netmask. For example, we might have here 2 IP addresses like "10.0.0.150/24,10.0.0.152/24".
- Choose a route table 30400 + the index of the interface and add a default route 0.0.0.0/0. The gateway is the default gateway retrieved from metadata server. For example, we might get a route "0.0.0.0/0 10.0.0.253 10 table=30400".
- Finally, add a policy routing rule for each address. For example "priority 30400 from 10.0.0.150/32 table 30400, priority 30400 from 10.0.0.152/32 table 30400".
With above example, this roughly corresponds for interface eth0 to nmcli device modify "eth0" ipv4.addresses "10.0.0.150/24,10.0.0.152/24" ipv4.routes "0.0.0.0/0 10.0.0.253 10 table=30400" ipv4.routing-rules "priority 30400 from 10.0.0.150/32 table 30400, priority 30400 from 10.0.0.152/32 table 30400". Note that this replaces the previous addresses, routes and rules with the new information. But also note that this only changes the run time configuration of the device. The connection profile on disk is not affected.