18.04.04: Raise Network Interfaces on boot, Problems with LAN driver

[vellofrell]

Running a recently built server rig on 18.04.04 with a Gigabyte Aorus Z490 AX motherboard. I've had nothing but trouble with networking compatibility between this motherboard and 18.04 from the start. I cannot reliably raise the wired ethernet connection on reboot without performing a hard restart and only plugging in the ethernet cable to the back after the startup sequence is finished.


Every time during bootup sequence, I see:


[FAILED] Failed to start Raise network interfaces.
see systemctl status networking.service for details.


and... "a start job is running for Wait for Network to be Configured" which takes a full two minutes.


Here are the results of 'systemctl status networking.service'



systemctl status networking.service
● networking.service - Raise network interfaces
Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2020-07-19 21:57:50 CDT; 2 days ago
Docs: man:interfaces(5)
Process: 1107 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=1/FAILURE)
Process: 1010 ExecStartPre=/bin/sh -c [ "$CONFIGURE_INTERFACES" != "no" ] && [ -n "$(ifquery --read-environment --list --exclude=lo)" ] && udevadm settle (code=e
Main PID: 1107 (code=exited, status=1/FAILURE)


Jul 19 21:57:50 server systemd[1]: Starting Raise network interfaces...
Jul 19 21:57:50 server ifup[1107]: Missing required variable: address
Jul 19 21:57:50 server ifup[1107]: Missing required configuration variables for interface enp4s0/inet.
Jul 19 21:57:50 server ifup[1107]: Failed to bring up enp4s0.
Jul 19 21:57:50 server systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
Jul 19 21:57:50 server systemd[1]: networking.service: Failed with result 'exit-code'.
Jul 19 21:57:50 server systemd[1]: Failed to start Raise network interfaces.


I have tried installing an updated driver file (or maybe a kernel module is a better way to say it?) - as instructed on this driver page of the motherboard manufacturer: https://www.gigabyte.com/us/Motherboard/Z490-AORUS-PRO-AX-rev-1x/support#support-dl-driver-lan (https://www.gigabyte.com/us/Motherboard/Z490-AORUS-PRO-AX-rev-1x/support#support-dl-driver-lan)


Using their NVMEUpdate utility has gotten me exactly nowhere.



sudo ./nvmupdate64e
[sudo] password for $USER:


Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.35.23.3
Copyright (C) 2013 - 2020 Intel Corporation.




WARNING: To avoid damage to your device, do not stop the update or reboot or power off the system during this update.
Inventory in progress. Please wait [|.........]




Num Description Ver.(hex) DevId S:B Status
=== ================================== ============ ===== ====== ==============
01) Intel(R) Ethernet Controller (2) 1.69(1.45) 15F3 00:004 Update not
I225-V available




Tool execution completed with the following status: Device not found.
Press any key to exit.

Ideas as to how I get this done and make this machine raise the network interface automatically on all reboots?

[TheFu]

lshw -C network
What is in the netplan.yaml file(s) in /etc/netplan/ ?

ifupdown was deprecated in 17.10.

[vellofrell]

sudo lshw -C network
[sudo] password for USER:
*-network
description: Wireless interface
product: Intel Corporation
vendor: Intel Corporation
physical id: 14.3
bus info: pci@0000:00:14.3
logical name: wlo1
version: 00
serial: 58:96:1d:b2:ee:a9
width: 64 bits
clock: 33MHz
capabilities: pm msi pciexpress msix bus_master cap_list ethernet physical wireless
configuration: broadcast=yes driver=iwlwifi driverversion=5.3.0-62-generic firmware=48.13675109.0 latency=0 link=no multicast=yes wireless=IEEE 802.11
resources: irq:16 memory:51514000-51517fff
*-network
description: Ethernet interface
product: Intel Corporation
vendor: Intel Corporation
physical id: 0
bus info: pci@0000:04:00.0
logical name: enp4s0
version: 02
serial: b4:2e:99:f9:85:e4
size: 1Gbit/s
capacity: 1Gbit/s
width: 32 bits
clock: 33MHz
capabilities: pm msi msix pciexpress bus_master cap_list rom ethernet physical 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=igc driverversion=0.0.1-k duplex=full ip=192.168.1.2 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
resources: irq:16 memory:51200000-512fffff memory:51300000-51303fff memory:51100000-511fffff




cat /etc/netplan/config.yaml
network:
version: 2
renderer: networkd
ethernets:
enp4s0:
addresses: [ 192.168.1.2/24 ]
gateway4: 192.168.1.1
nameservers:
addresses: [ 208.67.222.222, 208.67.220.220 ]

[LHammonds]

network:
version: 2
renderer: networkd
ethernets:
enp4s0:
addresses: [ 192.168.1.2/24 ]
gateway4: 192.168.1.1
nameservers:
addresses: [ 208.67.222.222, 208.67.220.220 ]


All the extra spaces where none should be "might" be messing with you. Example of modified config:


network:
version: 2
renderer: networkd
ethernets:
enp4s0:
dhcp4: false
dhcp6: false
addresses: [192.168.1.2/24]
gateway4: 192.168.1.1
nameservers:
addresses: [208.67.222.222,208.67.220.220]
What is the output of this command?

sudo netplan --debug generate
If there is a line in there that says "Configuration is valid" then it isn't a syntax issue.

LHammonds

[TheFu]

https://askubuntu.com/questions/1244745/ubuntu-20-04-intel-network-connectivity-issue-bug-in-igc says it is an Intel firmware issue and provides instructions for Gigabyte and Asus MBs. If these don't work, $25 for an Intel PRO/1000 PCIe NIC that uses older drivers is a solution to consider. Having a spare NIC is handy.

I don't see anything wrong that jumps out to me. Don't know if disabling dhcp is required or not. I always do.

May want to disable the wifi adapter just to be certain too - I'd use rfkill for that.

I'm just happy my MBs have Intel NICs that are a little older - using e1000e or igb drivers. Newer isn't always better.

[aljames2]

Also if you see Intel Boot Agent in your EFI boot sequence you can disable that, and you can also disable PXE Network boot if that is enabled. Anything that is not a boot device in your boot sequence can be disabled. Mine had a USB printer in the boot sequence once and it slowed down my bootup. One more thing about the Intel Boot Agent...some nics have it enabled in the firmware. This caused me lots of problems once and I did what TheFu said and replaced it with a older $25 Intel nic manuf by HP.

+1 on disable the wireless, this may cause further heartburn.

+1 on having a clean netplan file. I think I would check this out first.

[vellofrell]

All the extra spaces where none should be "might" be messing with you. Example of modified config:


network:
version: 2
renderer: networkd
ethernets:
enp4s0:
dhcp4: false
dhcp6: false
addresses: [192.168.1.2/24]
gateway4: 192.168.1.1
nameservers:
addresses: [208.67.222.222,208.67.220.220]
What is the output of this command?

sudo netplan --debug generate
If there is a line in there that says "Configuration is valid" then it isn't a syntax issue.

LHammonds

LHammonds - thanks for the tips regarding refining of Netplan. I have updated per your recommendations. Netplan Generate and Netplan Apply reveal no issues.

Because I have some kind of driver/kernel module compatibility issue, I'm certain this won't fix my ultimate problem with raising network interfaces automatically on every reboot, but I appreciate the help on refining this Netplan config.

[vellofrell]

TheFU & aljames2 - thanks for these tips. I will continue with this advice and find out what happens.

TheFU - regarding that post you link to - I have been through that thread over and over again, and even started THIS THREAD (https://askubuntu.com/questions/1257188/18-04-04-server-needs-new-eth0-driver-for-networking-help-installing) to further inquire on this problem. Going through those instructions in that initial thread you linked to, I still receive this:


$ sudo ./nvmupdate64e


Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.35.23.3
Copyright (C) 2013 - 2020 Intel Corporation.




WARNING: To avoid damage to your device, do not stop the update or reboot or power off the system during this update.
Inventory in progress. Please wait [|.........]




Num Description Ver.(hex) DevId S:B Status
=== ================================== ============ ===== ====== ==============
01) Intel(R) Ethernet Controller (2) 1.69(1.45) 15F3 00:004 Update not
I225-V available




Tool execution completed with the following status: Device not found.
Press any key to exit.


All I can reason now is that I'm not using the correct .bin file for this exact motherboard? (Intel Gigabyte Aorus Z490 AX). Maybe "FXVL_15F3_V_2MB_1.45.bin" doesn't apply to my precise version of this MoBo?

What am I missing here? I have re-flashed one MoBo BIOS update so far (one update later than factory setting). For now, all I am relying on is waiting for a combo of more BIOS updates from Gigabyte and the release of 18.04.5 in August.

RE problems of using bleeding-edge releases of hardware for server applications: YEP - this was precisely my worry from the beginning. The Coronavirus supply chain problems caused the supply of slightly-older hardware such as Motherboards to dry up, and I realized I had to buy newer, more expensive components like the board and the CPU in order to put the parts together for this new server rig when I was ready. I think I'd rather wait for updates to fix this problem - or actually get the thing hacked into working properly - rather than finding a less troublesome board and going back to dissect this server rig and half-way rebuild it all over.

Thanks all...

[vellofrell]

TheFU & aljames2 - thanks for these tips. I will continue with this advice and find out what happens.


TheFU - regarding that post you link to - I have been through that thread over and over again, and even started THIS THREAD to further inquire on this problem. Going through those instructions in that initial thread you linked to, I still receive this:



$ sudo ./nvmupdate64e


Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.35.23.3
Copyright (C) 2013 - 2020 Intel Corporation.



WARNING: To avoid damage to your device, do not stop the update or reboot or power off the system during this update.
Inventory in progress. Please wait [|.........]



Num Description Ver.(hex) DevId S:B Status
=== ================================== ============ ===== ====== ==============
01) Intel(R) Ethernet Controller (2) 1.69(1.45) 15F3 00:004 Update not
I225-V available


Tool execution completed with the following status: Device not found.
Press any key to exit.



All I can reason now is that I'm not using the correct .bin file for this exact motherboard? (Intel Gigabyte Aorus Z490 AX). Maybe "FXVL_15F3_V_2MB_1.45.bin" doesn't apply to my precise version of this MoBo?


What am I missing here? I have re-flashed one MoBo BIOS update so far (one update later than factory setting). For now, all I am relying on is waiting for a combo of more BIOS updates from Gigabyte and the release of 18.04.5 in August.


RE problems of using bleeding-edge releases of hardware for server applications: YEP - this was precisely my worry from the beginning. The Coronavirus supply chain problems caused the supply of slightly-older hardware such as Motherboards to dry up, and I realized I had to buy newer, more expensive components like the board and the CPU in order to put the parts together for this new server rig when I was ready. I think I'd rather wait for updates to fix this problem - or actually get the thing hacked into working properly - rather than finding a less troublesome board and going back to dissect this server rig and half-way rebuild it all over.

EDIT: This rig had no problems at all automatically raising the network (ethernet) when I was first testing it out with Ubuntu Desktop - either 18.04 or 20.04 - which one exactly is a key detail I can't recall. It's the Ubuntu Server 18.04.04 in which network interfaces combined with the MoBo don't seem to work. With Server 18 installed, every time at boot, I see a line saying 'Failed to Raise Network Interfaces' along with 'A start job is running for Wait for Network to be configured.'

Thanks all...

[TheFu]

If Desktop versions are working, then I'd check that the driver used is the same between the desktop and server installs.

Also, Desktops use network-manager (usually) to handle all network settings. You can try to use nmcli and see if that matters. I'd be surprised if it does, but anything is possible.

The suggestion for a $25 Intel NIC was to get a PCIe adapter, not change the motherboard. Installing a card is 5 minutes.

[vellofrell]

Okay, I went out and bought this NIC and installed -

https://www.microcenter.com/product/360498/intel-expi9301ct-gigabit-pcie-x1-adapter

I still could not get the rig to auto-raise network interfaces on startup. In fact I couldn't even get it online at all. 'ifconfig' did show a change in the Ethernet interfaces, and I attempted to alter netplan to reflect this. However I got nowhere, and had to remove the new NIC device for now, to at least get this server rig connected again, albeit with its problems raising network interfaces unless plugging in the Ethernet cable after the full boot process, and following a hard shutdown.

No matter what, the boot sequence would still show


"Failed to start raise network interfaces"




Jul 27 20:44:42 server systemd[1]: Starting resolvconf-pull-resolved.service...
Jul 27 20:44:42 server systemd[1]: Started resolvconf-pull-resolved.service.


While the new NIC device was still installed, it would give me this messages:


[FAILED] Failed to start resolvconf-pull-resolved.service.
See 'systemctl status resolvconf-pull-resolved.service' for details.



Here are those results... Inactive/dead

resolvconf-pull-resolved.service
Loaded: loaded (/lib/systemd/system/resolvconf-pull-resolved.service; static; vendor preset: enabled)
Active: inactive (dead) since Mon 2020-07-27 20:44:42 CDT; 21min ago
Process: 2036 ExecStart=/bin/sh -c cat /run/systemd/resolve/stub-resolv.conf | grep -v edns0 | /sbin/resolvconf -a systemd-resol
Main PID: 2036 (code=exited, status=0/SUCCESS)



Okay n00b question here, but does ifup/ifdown have any relevance here?

[TheFu]

ifup/ifdown were deprecated as user commands pre-18.04, but they are still used in the systemd Unit file for networking on 20.04. I don't have any 18.04 systems and we will be skipping that LTS, it appears.

On a server, the resolver doesn't need to be complex. I purge systemd-resolved and resolvconf from my system and manually edit the /etc/resolv.conf file to point at the DNS I wish to use. I have internal DNS that does caching, so having a local cache on each system really isn't needed (regardless of what Canonical thinks). Many consumer routers will provide DNS caching too.

Where DNS resolving becomes more complex is with laptops and other portable devices. Fortunately, only 1 of my systems is considered "portable" and that one uses DoH already.

But starting at the first steps is where I'd begin.

Driver loaded?
Driver used?
Card Up manually?
Card Up automatically?


Only then would I deal with DNS issues. People confuse DNS problems with network and routing problems all the time. They are not the same and need completely different troubleshooting. To tell whether an issue is DNS or driver+device or router, the simple ping command should be used.
Ping by IP, in this order:

local router (192.168.x.x)
1.1.1.1

What does the output mean?

If you can ping the router, then the NIC is fine. The driver is fine.
If you can ping 1.1.1.1 or any other well-know IP (8.8.8.8, 8.8.4.4, 1.0.0.1 are other examples), then networking, including routing, is fine.
If you cannot ping the local router, the problem is on the LAN - router, cable, NIC, computer.
If there isn't any NIC available, then it is either a driver, software or hardware problem on the computer.


Is it routing?
If you cannot ping public IPs, but all local IPs work, then it is likely routing. That can be on your side, in your router, or the ISP or the far side having issues. Trying different IPs - if none work, then the "far side" systems are NOT the issue. It is the ISP or something on your side.

Is it DNS?

ping using names. How does that go?
ping google.com
ping {your-ISP-name}.com, .org, .net.

There are different error messages based on whether the DNS lookup happened and worked correctly or not.
If DNS doesn't seem to be working, disable resolvconf and systemd-resolved using typical commands. sudoedit the /etc/resolv.conf file and put

nameserver 1.1.1.1
inside at the top. Save the file. Try pinging again using names. Sometimes the /etc/resolv.conf is a symbolic link to a file managed by one of those disabled tools and there is a claim at the top not to edit directly - it will be overwritten. With those other programs disabled, that won't happen ... until you reboot or enable them again.

Of course, a minimal knowledge of IPv4 networking is needed to better understand the different messages.

Is it a hardware/driver issue? Start by ensuing that a driver is actually being loaded. Which driver and version is it? sudo lshw -C network provides that information is a clear way. If multiple NICs have drivers loaded and there are issues, it is best to ensure only 1 at a time gets a configuration on the same subnet. For a beginner, having 2 NICs on the same subnet will lead to confusion.

[TheFu]

The first error way above says this:

Missing required variable: address
Did that get fixed by the netplan suggestions above?

When posting, please show the command and all output unless it gets over a page long together. Here's mine on a 20.04 box:


$ systemctl status networking.service
● networking.service - Raise network interfaces
Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: >
Active: active (exited) since Sun 2020-07-26 10:14:22 EDT; 1 day 23h ago
Docs: man:interfaces(5)
Main PID: 629 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 2279)
Memory: 0B
CGroup: /system.slice/networking.service

Jul 26 10:14:21 regulus systemd[1]: Starting Raise network interfaces...
Jul 26 10:14:22 regulus systemd[1]: Finished Raise network interfaces.

Odd to see Docs: man:interfaces(5), since the interfaces file was deprecated and mine is empty. 20.04 uses netplan. My working netplan file is exactly:

$ more 01-ens3-static.yaml
network:
version: 2
renderer: networkd
ethernets:
ens3:
addresses:
- 172.22.22.3/24
dhcp4: false
dhcp6: false
gateway4: 172.22.22.1
nameservers:
addresses: [ "172.22.22.80","172.22.22.81" ]

So people don't get confused. THIS IT NOT THE OP's information. Data in this post is shown only as an example.

[vellofrell]

Fu, thanks for the additional advice.

Turned out that although the machine was saying that it 'Failed to raise network interfaces' on all startups,' after installing PCI device network interface controller, the device was thankfully recognized by Ubuntu Server 18 due to its older and well-tested driver/module (so I assume).

It was in fact able to ping both the router and external IPs such as 1.1.1.1 and 8.8.8.8 (and anything else) despite saying 'Failed to raise network interfaces' when the ethernet cable was plugged into the NIC from the start.

Eventually, this whole problem worked itself out (as long as I keep using the newly-installed NIC, by way of adjusting the iptables and netplan settings to reflect the new eth0 address/name. The addition of the network interface controller changed the eth0 interface name from 'enp4s0' to two separate eth0 interfaces called 'enp1s0' and 'enp5s0.'

Running 'ifconfig' showed the newly installed NIC was the 'enp1s0' device. After adjusting both Netplan and /etc/iptables/rules.v4 to reflect the change from 'enp4s0' to 'enp1so,' my entire problem was fixed. AND, I had deleted '/etc/network/interfaces.d/setup' --just the 'setup' file. That step seemed to put the kibosh on all the 'Failed to raise network interfaces' mess. Although now I can't be certain if that was a major factor, given the subsequent changes to Netplan and Iptables.

TL,DR;




Installed new NIC device
Deleted /etc/network/interfaces.d/setup (some problem interfering with /etc/network/interfaces settings. This stopped "Failed to start Raise network interfaces" even before correcting Netplan and IPtables to reflect eth0 changes.
The extra network interface (NIC device) now shows 'enp1s0' and 'enp5s0' instead of just 'enp4s0.' Adjusted Netplan and ipconfig.rules.v4 settings to refer to 'enp1s0.' This allows successful raising of network interfaces and all connectivity available on startup.


Because I'm just insane and OCD enough to stress out about making the original motherboard's Intel ethernet controller work just as well, I am awaiting further updates from Ubuntu (18.04.5 as of July 2020) and further BIOS updates to the Gigabyte Aorus Z490 Pro AX motherboard. I've had no success working directly with these hardware modules/drivers to fix the problem, until the devs roll out updates. Is this thinking correct? As it stands now, something still is completely bugged up and wrong with one or the other, it seems.

[TheFu]

Was looking to build a new system and I want Intel NICs. Seems this MB has 2.5Gb Ethernet - that's what the i225-v from Intel provides. That seems to be bleeding edge. Intel has claimed they fixed a huge problem, but I can't find confirmation for Linux, just Windows.

Also saw earlier today that Intel is making updating NIC firmware easier on Linux. https://www.phoronix.com/scan.php?page=news_item&px=Intel-PLDM-Firmware-Update-ICE Don't think that is related, but maybe ... someone who knows more can say?

If this is solved by the new NIC, please use the "Thread Tools" button to mark it that way. At some point, the i225-v will have great support, so I'd check every few months. Put a reminder on your calendar. OTOH, if you don't have 2.5Gbps switch or router, then that NIC isn't all that useful, yet.

[vellofrell]

Thanks, I'll look at that link. I can't tell what the Gbps specs are for the router (Google Fiber Network Box GFRG200); I know max speeds in and out from the Internet are about 1000 Mbps. So stressing out over it will be a waste of time, I assume?

Yeah, that Intel i225 thing - I didn't even want a motherboard this recent and up to date - because of both pricing and fears of something like this. This bleeping virus going around caused the supply of prior gen motherboards I was looking at to dry up by the time I managed to get started making this server.

Thanks again.

[TheFu]

Thanks, I'll look at that link. I can't tell what the Gbps specs are for the router (Google Fiber Network Box GFRG200); I know max speeds in and out from the Internet are about 1000 Mbps. So stressing out over it will be a waste of time, I assume?

Yeah, that Intel i225 thing - I didn't even want a motherboard this recent and up to date - because of both pricing and fears of something like this. This bleeping virus going around caused the supply of prior gen motherboards I was looking at to dry up by the time I managed to get started making this server.

Thanks again.

If you only have 1 computer at home, then the WAN connection speeds are the most important limiting factor, but if you have multiple devices and want those networked and able to talk to each other, then usually faster is better.

Why use 10base-tx if 100base-tx is available cheap?
Why use 100base-tx if 1000base-tx is available cheap?
Why use 1000base-tx if 2500 or 10Kbase-tx is available and cheap?

Of course, having a NIC that goes 2.5 Gbps (in theory) doesn't do any good if the network gear is only 1 Gbps. Lots of people have 10 Gbps network gear at home these days. Not me, I'll wait until 10 Gbps switches are $30, but some people don't mind paying $300 for a faster switch. The 2.5Gbps standard is sorta a gap filler because wifi can't really get 10 Gbps for the next few years in the real world, but 2.5 Gbps is looking possible in very specific situations. But who will spurge for new wifi APs/routers if their home "servers" have slower connections - at least on the boxes/marketing. Wifi reality is usually about 25% of whatever the box says.

My home network has multiple subnets and about 20 systems. Most traffic is within my LANs, not over the internet. My internet connection is about 30 Mbps / 5 Mbps. That's plenty fast. I remember running a business on a 128 Kbps ISDN connection, so 5 Mbps up is huge!

Today, 1 Gbps is the minimal wired ethernet anyone should deploy. It is cheap - going slower will probably mean using more expensive equipment, since nobody makes 100base-tx stuff anymore.

2.5 Gbps probably will only be used in home environments. Businesses moved to 10 Gbps and 40 Gbps networks for servers about 5 yrs ago. Without businesses buying, 2.5Gbps will probably be a desktop-only thing. In a few years, 10 Gbps will get cheap, so everyone will get it when they get new routers. Then the desktop 10Gbps NIC flood will begin and those prices will drop to $25.