Two of my five worker nodes – both HP servers – went NotReady at the same time. I hadn’t touched the cluster. I had touched the USB wifi adapters plugged into the back of those machines.

That was the whole bug. The interesting part is why putting them back broke both nodes instead of one.

The Setup

Three mini-PCs on onboard wifi, two HP servers with USB wifi dongles. All five nodes on a flat 10.0.0.0/24 home network. The HP servers don’t have built-in wifi, so each has a USB adapter providing its only link to the cluster.

Netplan on Ubuntu 24.04 manages the wifi config. The config looked something like:

network:
  version: 2
  wifis:
    wlx1234abcdef56:
      dhcp4: true
      access-points:
        "home-ssid":
          password: "..."

That interface name – wlx1234abcdef56 – is not arbitrary. It’s derived from the adapter’s MAC address by systemd’s predictable network interface naming. Onboard NICs get eno1, enp3s0, etc. USB wifi adapters get wlx<MAC>.

The Failure

$ kubectl get nodes
NAME          STATUS     ROLES           AGE   VERSION
hp-server-1   NotReady   <none>          9d    v1.33.10
hp-server-2   NotReady   <none>          9d    v1.33.10
mini-pc-1     Ready      <none>          13d   v1.31.14
mini-pc-2     Ready      <none>          13d   v1.32.13
mini-pc-3     Ready      control-plane   13d   v1.33.10

$ ping 10.0.0.75
From 10.0.0.11 icmp_seq=1 Destination Host Unreachable

ARP table confirmed it: both HP IPs had FAILED entries. Nothing was answering at L2. The mini-PCs were fine, so this wasn’t the router.

I walked to hp-server-2, plugged in a keyboard, and logged in locally. The OS was up. ip link showed lo, eno1 (onboard, unused), and a wlx... interface. The USB adapter was detected.

sudo netplan apply hung for about a minute, then exited with status 1. Journal logs were full of kubelet errors about not being able to reach the API server – symptoms, not causes. Nothing obviously wrong with the wifi itself.

The “Wait, Are These Different?” Moment

Two machines. Two USB wifi adapters. I’d pulled both out and put them back in. What are the odds I put them back in the machines they came from?

$ ip link show | grep wlx
wlxaabbccdd1122: ...

$ sudo grep wlx /etc/netplan/*.yaml
    wlx998877665544:

The interface on the machine was wlxaabbccdd.... The interface in netplan was wlx998877.... Netplan was waiting for an adapter that was now plugged into the other HP server.

wlx<MAC> naming meant each adapter had its own identity baked into its interface name, and the netplan config on each node referenced one specific adapter. Swap the adapters and both configs point to the wrong device. Both nodes fail. Symmetrically.

The Fix

Swap them back:

# after physically moving the USBs back to their original machines
$ sudo netplan apply
$ ip -4 addr show wlxaabbccdd1122
    inet 10.0.0.75/24 ...
$ ping 10.0.0.1
PING 10.0.0.1: 56 data bytes
64 bytes from 10.0.0.1: ...

From the admin node:

$ kubectl get nodes
hp-server-1   Ready   <none>   9d    v1.33.10
hp-server-2   Ready   <none>   9d    v1.33.10

Kubelet reconnected on its own once the network came back.

The re-insertion alone wasn’t enough – the kernel brought the interface up, but NetworkManager didn’t auto-associate. sudo netplan apply after the swap was required to actually trigger DHCP and wifi association.

The Lesson

If you use USB wifi adapters with MAC-based interface names, your netplan config is bound to a specific piece of hardware, not a specific physical port or machine. Two identical-looking dongles are not interchangeable. Label them.

If you actually want them to be interchangeable, match by something more stable and use set-name:

wifis:
  wifi0:
    match:
      driver: "rtl8192*"   # or match any single-adapter-per-host criterion
    set-name: wifi0
    dhcp4: true
    access-points:
      "home-ssid":
        password: "..."

This is the same pattern people use for servers with multiple NICs where they want predictable names regardless of PCI enumeration order. For a single-adapter host, matching on driver is usually enough to make the config portable across hardware swaps.

But honestly? For a homelab with two USB dongles, a strip of masking tape on each one solves the same problem in five seconds.