NetworkManager Revisited - All your rebases are belong to us?

History of NetworkManager in RHEL/CentOS

NetworkManager was first introduced in EL6 with NM 0.8.1 which was highly limited in its capabilities and the accepted position, outside of a very small WiFi use case, was to disable it. NM was never rebased to something more recent in the lifetime of EL6.

When it comes to EL7 on the other hand, the 7.0 GA had 0.9.9.1 which, despite the deceptively small version difference, was a massive improvement in usability over the older version and had the rather nice nmcli tool included - as discussed in my previous article.

This was updated to 1.0.0 at the 7.1 milestone, 1.0.6 at the 7.2 milestone and most recently to 1.4.0 at the 7.3 milestone release.

To get an understanding of just how much this has changed, despite the apparently small version increment, it's interesting to take a brief look through the changelog upstream ...

Much of the previous article is still valid but there are some key differences in the update to pay attention to.

Now this is a long list of changes and this article is not going to exhaustively go through them. It is, however, going to pick on the most prominent changes that provide functional improvements or deviations in behaviour from the first EL7 release.

Super lightweight environments

In an environment where every daemon running counts against you and networking is very static with no need to react to udev or dbus events the 1.0 NetworkManager release included a useful new option configure-and-quit.

This is a NetworkManager configuration, rather than an interface configuration, so it needs to go into /etc/NetworkManager/ and is simple to just drop in:

cat > /etc/NetworkManager/conf.d/c-and-q.conf <<EOF
[main]
configure-and-quit=true
EOF

With this in place NetworkManager will carry out the configuration of the interface and then gracefully exit leaving the network up as desired and notifying systemd that networking is up but without the NM daemon left running in the background listening for, and responding to, udev, dbus or similar events.

Automatically respond to changes to config files

The default behaviour for NetworkManager is to only respond to changes to configuration files on a restart or when directed to through nmcli conn reload

However, if a more dynamic response to changes to the files is preferred it is possible to have NM monitor all the configuration files and have it immediately make changes to network state from this.

cat > /etc/NetworkManager/conf.d/monitor.conf <<EOF
[main]
monitor-connection-files=true
EOF

Be very careful with this option as any changes to the files will be immediately applied which could quite easily break network connectivity if the wrong thing is put in place.

RTFM? But isn't this blog meant to tell me what to do?

A substantial effort was made to improve the documentation for both using nmcli and editing files. When trying to do anything new it's worth looking at man nm-settings, man nm-settings-ifcfg-rh, man nmcli-examples and man NetworkManager.conf as there may be a clear example or documentation on how to handle that situation already.

IPv6 changes everywhere!

There's been many, many bug fixes on this side of things such as not losing a static IPv6 configuration if SLAAC fails, respecting router advertisement MTUs, waiting for Duplicate Address Detection (DAD) to complete before declaring the connection activated and more.

The more interesting areas here though are a few changes in behaviour which have happened over time, mostly to address privacy concerns over IPv6 address generation and behaviour.

Firstly NetworkManager handles all the IPv6 configuration itself rather than allowing the kernel to do any automated stuff through SLAAC or similar. This makes the resulting configuration more predictable. This also means only network config files that need to be changed for almost everything, with only a couple of behaviours being changed via sysctl configuration. For many of the configuration options though the default is to use the kernel defaults to minimise surprises.

Next there was a bug fix in place (which was backported to 1.0.X but still a change from the EL7.0 GA) where if an interface only has an IPv6 link-local (fe80::) address it is no longer considered as 'connected'

The 7.0GA had a version of NetworkManager that used EUI64 based address generation (so predictably based on MAC) but this has been considered a privacy issue for some time.

With the NM 1.2 release (so after the 7.3 milestone) this was changed for new builds to follow RFC 7217 for SLAAC address generation.

The key to modify this with nmcli is ipv6.addr-gen-mode or in the ifcfg file configure the key IPV6_ADDR_GEN_MODE with eui64 for the older behaviour, or stable-privacy for the newer behaviour.

This does not change behaviour for existing connections on an upgrade from an existing system based on EL7.2 (or older) to one that passes the 7.3 milestone, but for any new system built from 7.3 media (or if a new network connection is added) this could be unexpected and confusing with the very different SLAAC IPv6 address configured on the connection from what was there on older builds.

The stable address is based on a combination of a machine unique value which remains stable which is 256bits of data from urandom stored in /var/lib/NetworkManager/secret_key and the connection.stable-id value, which defaults to the UUID of the connection if not defined. If you need multiple connections on a system that result in the same stable-privacy address you just need to set connection.stable-id to the same for each connection, although I cannot imagine this being a common requirement.

Related to this, the IPv6 Privacy Extensions were added as a configurable option as well. This may trip you up on a new build with unexpected IPv6 addresses on systems, if you have an IPv6 capable network - this would be the implementation of RFC 4941.

The default at this point in time is to not enable these, but it is trivial to do so.

Enabling system wide it's recommended to do this at the kernel level:

cat > /etc/sysctl.d/99-default-ip6-privacy.conf <<EOF
net.ipv6.conf.default.use_tempaddr=2
net.ipv6.conf.default.temp_prefered_lft = 7200
EOF
systemctl restart systemd-sysctl

To do it at a global NetworkManager level for all connections:

cat > /etc/NetworkManager/conf.d/ipv6-pe.conf <<EOF
 [connection]
ipv6.ip6-privacy=2
EOF

There is no setting to configure the preferred lifetime though so it'll use the kernel default of 1 day for rotation of addresses.

Finally it's possible to configure on a per connection basis in NetworkManager:

nmcli c mod <connection-name> ipv6.ip6-privacy 2

Return of the MAC

Following on with privacy concerns and tracking techniques there have been changes at a Layer2 level as well. There's two separate ways this is carried out, one specific to wifi and one to ethernet as well.

The default now when carrying out scans for wifi networks to associate with is to use a random MAC during that scan. Since this is a device level activity before association where a connection profile gets activated it needs to be configured with a device level setting in /etc/NetworkManager.

[device]
wifi.scan-rand-mac-address=no

If set to no it will use the MAC configured for the wifi device when it's not connected to anything, either the native address or manipulated via macchanger or udev.

Note this is not the address used to actually associate with the wifi network though.

The previous article mentioned the use of the cloned-mac-address property to adjust the MAC address of an interface, physical or wifi, which is used to actually connect to a network. If manipulating the configuration directly through D-Bus this has been deprecated by the new property assigned-mac-address, but if using nmcli or the key-value format of the files cloned-mac-address is still the property to alter.

The reason for the change on D-Bus is that the existing type is a byte array, but for the new features a string was required and this could not be changed. Of course this strong type behaviour is not an issue with using nmcli or editing files.

The new feature here is the ability to automatically generate a private MAC rather than broadcast the actual MAC to all systems on the network. The various options for this are:

  • permanent - use the native built in MAC address of the device.
  • preserve - use any MAC address already assigned to the device. If this is not already changed via macchanger, udev or networkd this will be the normal MAC of the device.
  • random - use a random MAC each time this connection is activated.
  • stable - use a random generated MAC that is persistent for the connection, using very similar techniques to RFC7217 stable private IPv6 addressing and using the same secret_key and connection.stable-id token used by that method.
  • NULL/unset - follow any globally configured default (which is the default and the global default is permanent in 1.4.0 and changes to preserve in 1.6.0)
  • ##:##:##:##:##:## - use a specific hard coded MAC address

Dispatching activities

Dispatcher scripts are stored in /etc/NetworkManager/dispatcher.d and are called when a connection goes up or down.

These can be quite useful to carry out certain activities such as notifying status, updating records, running configuration management utilities such as puppet, or handling more complicating routing arrangements via policy routing (see NetworkManager-dispatcher-routing-rules on CentOS/RHEL but this is part of NetworkManager on Fedora).

The key change with the rebased NetworkManager in 7.3 are that these get executed for any connectivity status change and before a system suspends as well to handle services or notifications.

Link Layer Discovery Protocol communications

Which port is your system plugged into? At home this is an easy question, in an office with underfloor cables and centralised switch racks this is not so simple.

LLDP is a protocol that provides details about the switch connected. This can be name, port, vlans allowed and so on. Although in some environments this is disabled to prevent "information leakage" in the name of security, in many others LLDP (or the Cisco specific CDP) is enabled to ease troubleshooting.

The version of NetworkManager in EL7.3 includes an LLDP listener to receive and display these details, but not a transmitter. So from a system if the switch is sending out LLDP packets it is possible to gather details from the switch, but from the switch you wouldn't see the details of the system connected.

This is disabled by default on a connection but the property involved is connection.lldp and enabling the functionality on a connection is as simple as:

nmcli c mod <connection-name> connection.lldp enable

To view the neighbour information use nmcli device lldp

Since it is fairly limited in what it presents and has no transmitter part I'd generally suggest something like lldpd instead.

Bridging the gap, forming bonds with tags and teams

Previously there was a special connection type of "bridge-slave", "bond-slave" and "team-slave" and there were limitations with the nmcli tool about how these could be stacked (eg they required an interface type of ethernet to be one of these).

One of the significant improvements is that this limitation is no longer in place and these connection types can be arbitrarily stacked using nmcli. The *-slave types are no longer present and instead the existence of a 'master' declares it is a slave to another connection.

To recap briefly to create an environment that involves three interfaces in an active-backup bond with three bridges that virtual machines can be attached to, of which two are vlan tagged, it required this prior to EL7.3:

nmcli connection add type bond con-name mybond mode active-backup
nmcli connection add type bond-slave ifname eth2 master mybond
nmcli connection add type bond-slave ifname ens9 master mybond
nmcli connection add type bond-slave ifname ens10 master mybond
nmcli connection add type bridge con-name bridge0 ifname bridge0 connection.autoconnect yes ipv4.method manual "10.0.0.1/24" 
nmcli connection add type bridge con-name bridge60 ifname bridge60 connection.autoconnect yes ipv4.method manual "10.0.60.1/24" 
nmcli connection add type bridge con-name bridge100 ifname bridge100 connection.autoconnect yes ipv4.method manual "10.0.100.1/24" 
nmcli connection add type vlan con-name vlan-60 dev mybond id 60
nmcli connection add type vlan con-name vlan-100 dev mybond id 100
nmcli connection down mybond
nmcli connection down vlan-60
nmcli connection down vlan-100
nmcli connection modify mybond connection.master bridge0 connection.slave-type bridge
nmcli connection modify vlan-60 connection.master bridge60 connection.slave-type bridge
nmcli connection modify vlan-100 connection.master bridge100 connection.slave-type bridge
nmcli connection up bond-slave-eth2
nmcli connection up bond-slave-ens9
nmcli connection up bond-slave-ens10
nmcli connection up mybond
nmcli connection up bridge0
nmcli connection up vlan-60
nmcli connection up bridge60
nmcli connection up vlan-100
nmcli connection up bridge100

This is obviously quite confusing, quite involved and requires modifying properties after the connection is created to make things link up.

To do the same in EL7.3 just this is required, and in my testing the connections were automatically activated in the full stack:

nmcli c add type bridge ifname br0 con-name br0 connection.autoconnect yes ipv4.method manual ipv4.addr "10.0.0.2/24"
nmcli c add type bridge ifname br60 con-name br60 connection.autoconnect yes ipv4.method manual ipv4.addr "10.0.60.2/24"
nmcli c add type bridge ifname br100 con-name br100 connection.autoconnect yes ipv4.method manual ipv4.addr "10.0.100.2/24"
nmcli c add type vlan con-name vlan-100 dev bond0 id 100 master br100 connection.autoconnect yes 
nmcli c add type vlan con-name vlan-60 dev bond0 id 60 master br60 connection.autoconnect yes 
nmcli c add type bond ifname bond0 con-name bond0 connection.autoconnect yes master br0 bond.options mode=active-backup 
nmcli c add type ethernet ifname eth1 con-name eth1 master bond0 connection.autoconnect yes
nmcli c add type ethernet ifname eth2 con-name eth2 master bond0 connection.autoconnect yes
nmcli c add type ethernet ifname eth3 con-name eth3 master bond0 connection.autoconnect yes
[root@c7-template ~]# nmcli c
NAME         UUID                                  TYPE            DEVICE    
System eth0  1a672fc0-3b5e-4040-ac3c-356d2968413b  802-3-ethernet  eth0      
bond0        751244b0-7a3c-4d4e-bc15-d0af59a4c43c  bond            bond0     
br0          45f5895d-3f5c-4ffa-b0b0-759d2d662da0  bridge          br0       
br100        70f9d11f-8173-4435-acbd-9d0d57bdc2d4  bridge          br100     
br60         86b4cc04-061b-42f2-8b50-44a9264f9066  bridge          br60      
eth1         9f55056a-beaa-4c01-8b14-b315ae0b7eb5  802-3-ethernet  eth1      
eth2         ca97ead2-45b5-49d2-8e1f-3706ed35b5d6  802-3-ethernet  eth2      
eth3         8008ce99-e491-4498-8a5a-43c7b935028e  802-3-ethernet  eth3      
vlan-100     c0226632-a6d7-4fee-9d80-93b22a98c9dc  vlan            bond0.100 
vlan-60      bd2a6aff-a21b-4465-9984-6f15cc363a6d  vlan            bond0.60  

As you can see it is much simpler, and doesn't use any special *-slave types but rather just allows these to be stacked as required, and it can trivially be changed to use a team device instead with very little difference:

nmcli c add type bridge ifname br0 con-name br0 connection.autoconnect yes ipv4.method manual ipv4.addr "10.0.0.2/24"
nmcli c add type bridge ifname br60 con-name br60 connection.autoconnect yes ipv4.method manual ipv4.addr "10.0.60.2/24"
nmcli c add type bridge ifname br100 con-name br100 connection.autoconnect yes ipv4.method manual ipv4.addr "10.0.100.2/24"
nmcli c add type vlan con-name vlan-100 dev team0 id 100 master br100 connection.autoconnect yes 
nmcli c add type vlan con-name vlan-60 dev team0 id 60 master br60 connection.autoconnect yes 
nmcli c add type team ifname team0 con-name team0 connection.autoconnect yes master br0 team.config '{"runner": {"name": "activebackup"}, "link_watch": {"name": "ethtool"}}'
nmcli c add type ethernet ifname eth1 con-name eth1 master team0 connection.autoconnect yes
nmcli c add type ethernet ifname eth2 con-name eth2 master team0 connection.autoconnect yes
nmcli c add type ethernet ifname eth3 con-name eth3 master team0 connection.autoconnect yes

The bridge command is quite useful to determine the state of the bridges as well:

[root@c7-template ~]# bridge link
59: team0 state UP :  mtu 1500 master br0 state forwarding priority 32 cost 100 
60: team0.100 state UP @team0:  mtu 1500 master br100 state forwarding priority 32 cost 100 
61: team0.60 state UP @team0:  mtu 1500 master br60 state forwarding priority 32 cost 100 

Concluding summary

The NetworkManager included in 7.0, as per my previous article, was already very capable and was the recommended way to handle various things. The changes made, especially in 7.3, have been significant and leaves this as even more capable cementing it as the preferred method of configuring most environments.

Keep in mind this article isn't even exhaustive of the changes, and I strongly urge you to review the changelog for more details into things like enabling management of more device types (tun & tap, macvlan, vxlan, ip tunnels (ipip, gre, sit, ip6ip6 and ipip6)... so just a few).

For your environments take a long look before dismissing NetworkManager as it'll only improve management and diagnosis of networking issues going forwards and the edgecase for preferring the legacy network service is now basically just openvswitch usage.

Add new comment