8.8 11 Troubleshoot System Startup 3

You're staring at a black screen. Now, or a spinning circle. Or — worst of all — an error message that tells you exactly nothing.

System startup failures have a way of showing up at the worst possible time. Right before a presentation. And on a deadline. When you just want to check one thing real quick.

And if you've ever seen 8.Worth adding: 8. 8.On top of that, 4. 8.8 or 8.4 pop up in your network logs during boot, you've probably wondered: *is DNS breaking my startup?

Short answer: sometimes. But it's rarely the whole story.

Let's walk through what actually happens when a system won't start cleanly — and how to fix it without guessing.

What "System Startup" Actually Means

Most people think "startup" is one thing. It's not. It's a chain.

Power on → firmware (BIOS/UEFI) → bootloader → kernel init → userspace services → login prompt → desktop/app load.

A failure at any link breaks the chain. And the symptom — black screen, hang, error code — often shows up downstream from the real cause.

That's why "troubleshoot system startup" isn't a single checklist. It's a process of elimination.

The DNS Red Herring

Here's where 8.8.8.8 enters the chat.

Google's public DNS servers (8.8.That's why 8. Also, 8 and 8. Even so, 8. 4.4) are hardcoded into countless routers, containers, VMs, and IoT devices as fallbacks. Now, when a system boots and can't reach its configured DNS — maybe DHCP hasn't handed out a lease yet, maybe the local resolver isn't up — it falls back to 8. 8.8.8.

If that fails, you'll see timeouts in logs. NetworkManager retries. systemd-resolved complaints. Apps hanging on "resolving host Practical, not theoretical..

But DNS didn't break your boot. DNS is just the first network thing that tries to talk — and fails — because the network isn't ready yet.

Big difference Simple, but easy to overlook..

Why Startup Failures Happen (The Real Culprits)

After years of debugging everything from headless servers to dual-boot laptops, the patterns are surprisingly consistent.

1. Storage Issues You Didn't See Coming

A nearly full root partition (/) will kill boot silently. On the flip side, the journal fills what little space remains. systemd can't write logs. tmpfiles can't create runtime dirs. Next reboot: emergency shell The details matter here..

Check it: df -h / from a live USB. If usage is >90%, that's your problem.

2. Kernel / Initramfs Mismatch

You updated the kernel but didn't regenerate the initramfs. Here's the thing — or you rolled back a kernel but the bootloader still points to the new initramfs. Result: kernel panics, missing modules, "can't find root device.

Fix: mkinitcpio -P (Arch), update-initramfs -u -k all (Debian/Ubuntu), dracut --regenerate-all (Fedora/RHEL). Then update grub: grub-mkconfig -o /boot/grub/grub.cfg And that's really what it comes down to..

3. Broken systemd Units

One bad .service file — a typo in ExecStart, a missing After=network-online.And waits. Now, target — can stall the entire boot. Because of that, systemd waits. Default timeout: 90 seconds per unit Practical, not theoretical..

Diagnose: systemd-analyze blame and systemd-analyze critical-chain from a working boot. Or boot with systemd.debug-shell=1 and poke from the emergency shell Worth knowing..

4. Network Dependencies That Don't Exist

Services declared Wants=network-online.target but the network never comes "online" — because the interface is renamed, or DHCP is slow, or the bridge isn't up.

Real talk: network-online-target is a lie on many systems. It fires when any interface gets a carrier. Not when your interface has an IP.

Better: use After=systemd-networkd-wait-online.service or NetworkManager-wait-online.But service explicitly. Or write a drop-in that requires your specific interface.

5. Hardware That's Quietly Dying

NVMe controller timeouts. So naturally, sATA cables going bad. RAM errors that only show under thermal stress. These don't always throw SMART alerts or kernel oopses. They just... hang. Sometimes at the same spot. Sometimes randomly That's the part that actually makes a difference..

Test: smartctl -a /dev/nvme0, memtest86+ (run 4+ passes), dmesg -T | grep -i -e nvme -e ata -e scsi -e mce Nothing fancy..

How to Actually Troubleshoot (Step by Step)

Don't guess. Follow the chain.

Step 1: Get Visibility

If you see a prompt — any prompt — you're already ahead.

• GRUB menu? Press e on the boot entry. Remove quiet splash. Add systemd.log_level=debug. Boot with Ctrl+x.
• No GRUB? Live USB. arch-chroot / chroot into the broken system. Check logs: journalctl -b -1 -p err..alert (previous boot, errors only).
• Headless server? Serial console. IPMI. KVM. If you don't have out-of-band access, you're flying blind.

Step 2: Identify the Last Successful Milestone

systemd logs milestones. Look for:

Reached target Local File Systems.
Reached target Network.
Reached target Graphical Interface.

The last one reached = the last thing that worked. The next one = where it stalled Simple as that..

Step 3: Isolate the Blocking Unit

From emergency shell or chroot:

systemctl list-units --state=failed
systemctl status 
journalctl -u  -b -1

Check TimeoutStartSec in the unit file. If it's 90s and the service hangs, that's your 90s boot delay.

Step 4: Test Fixes Incrementally

Don't change five things at once.

• Mask the suspect unit: systemctl mask broken.service
• Reboot. Does it get further?
• If yes, fix the unit. If no, unmask and look at the next failure.

Step 5: Verify the Full Chain

Once it boots cleanly:

systemd-analyze verify
systemd-analyze critical-chain
systemd-analyze plot > boot.svg  # visualize it

Look for services that take >5s without good reason. Drop After=network-online.Parallelize where possible. target unless you really need it.

Common Mistakes (And What to Do Instead)

Mistake: Blaming DNS for Everything

You see resolving 8.8.Think about it: 8. 8 in logs. You hardcode /etc/resolv.conf. You add nameserver 1.On top of that, 1. 1.Worth adding: 1. Boot still hangs.

Why: DNS timeout is a symptom of network not being ready. Fix the network bring-up. Or make the

Fixing the DNS‑Related Hang

If you really need to resolve hostnames early, do it the right way:

1. Create a minimal resolv.conf in the initramfs
   Add a file like /etc/initramfs-tools/conf.d/resolv.conf (Debian/Ubuntu) or drop a drop‑in in /etc/dracut.conf.d/ (Fedora) that contains only the nameservers you trust. Example for dracut:

   # /etc/dracut.So naturally, conf. Because of that, d/resolv. conf
   add_dracutmodules+=" dns"
   # The following line is parsed by dracut’s resolv.conf generator
   rd.
   
   Then rebuild the initramfs: `dracut -f`.
   
   

2. Avoid blocking services
   If a unit has After=network-online.target and Wants=network-online.target, make sure it also declares Before=multi-user.target. Otherwise it can become a hard dependency that stalls the boot. A quick fix is to add TimeoutStartSec=10s to the unit or replace the dependency with a lighter one like network.target.

3. Test the change
   Reboot and watch the journal: journalctl -b -p err..alert -u <unit> – you should see the DNS lookup finish well before the 90‑second timeout expires.

Kernel‑Parameter Tweaks That Actually Work

Sometimes the problem isn’t a service at all; it’s the kernel’s default timeout behavior. A few well‑placed kernel command‑line options can shave seconds off the boot without touching any unit files:

Add any of these to /etc/default/grub (or the appropriate GRUB configuration file) and run update-grub / grub2-mkconfig. Remember to keep a copy of the original line in case you need to revert.

When the Boot Is Stalled by a Custom Initramfs Script

Many distributions ship with a custom initramfs that runs scripts to mount encrypted volumes, set up RAID, or start remote filesystems. A stray sleep or an infinite while loop in one of those scripts can look like a “random 90‑second hang”.

How to debug it:

1. Regenerate the initramfs with verbose output

   • Debian/Ubuntu: update-initramfs -u -k all -v
   • Fedora: dracut -f --verbose

2. Inspect the generated cpio archive

   mkdir /tmp/initrd
   cd /tmp/initrd
   zcat /boot/initrd.img-$(uname -r) | cpio -idmv
   

   Look for scripts under scripts/, init, or setup-*.sh. Grep for sleep, while, or exec statements that could block That alone is useful..

3. Add a debug echo
   Insert echo "DEBUG: <script> line <n>" > /dev/kmsg at the top of the suspect script. Rebuild and boot; the message will appear in dmesg or journalctl -b Worth keeping that in mind..

4. Temporarily replace the script with a stub
   If you identify a particular script (e.g., /scripts/local-top/cryptroot), rename it to cryptroot.disabled and rebuild. If the system boots past the previous hang point, you’ve isolated the culprit.

The “It Works on Another Machine” Trap

If the same configuration boots fine on one hardware platform but hangs on another, the difference is almost always in the hardware initialization sequence. And the BIOS/UEFI may be setting a different SATA mode (AHCI vs. RAID), or the firmware may be exposing a different PCIe link speed Small thing, real impact..

Steps to align the environments:

• Force a known SATA mode in the firmware settings (e.g., always AHCI).
• Disable fast boot options that skip POST checks.
• **Update the

Continue the article without friction. Do not repeat previous text. Finish with a proper conclusion.

When the hardware‑initialisation sequence is the culprit

Even after confirming that the kernel parameters are correct, a mismatch between the firmware’s early‑boot configuration and the kernel’s expectations can still cause a stall. The most common culprits are:

1. SATA/NVMe mode – Some UEFI firmwares default to RAID or IDE mode, which the Linux kernel cannot recognise without additional drivers. Switching the SATA controller to AHCI (or to the native NVMe mode for PCIe SSDs) often eliminates the pause while the kernel probes for devices Still holds up..

2. PCIe link speed – On newer platforms the firmware may negotiate a link speed that the kernel’s PCIe driver does not support out‑of‑the‑box. Adding the kernel parameter pcie_aspm=off or pcie=noaspm forces a fallback to a compatible speed, allowing the boot process to continue.

3. Secure Boot / unsigned modules – If Secure Boot is enabled and the initramfs contains unsigned third‑party modules (e.g., proprietary storage drivers), the boot loader may halt while the signature verification runs. Temporarily disabling Secure Boot or signing the modules resolves the dead‑lock.

4. ACPI/UEFI quirks – Certain motherboards expose non‑standard ACPI tables that trigger a kernel panic in the early‑initramfs stage. Adding acpi=off or acpi=force can force the kernel to ignore the offending tables. Conversely, acpi=off may be needed on older hardware that mis‑interprets the tables.

5. Firmware updates – A known bug in the UEFI firmware can manifest as a 90‑second pause during the “Loading Linux…” stage. Checking the vendor’s release notes for a BIOS/UEFI update that addresses “boot hangs” or “PCIe link training” is often the quickest fix Most people skip this — try not to..

Practical checklist for aligning the boot environment

• Enter the firmware setup and verify that the SATA controller is set to AHCI (or the appropriate NVMe mode).
• Disable any Fast Boot or Quick Boot options that skip POST diagnostics.
• If the platform uses UEFI Secure Boot, either turn it off or enroll the required keys for the custom initramfs modules.
• Apply the latest BIOS/UEFI firmware from the manufacturer; many “boot hangs” are fixed in minor updates.
• Add the kernel parameter pcie_aspm=off (or the variant that matches your hardware) to the GRUB line and run update-grub / grub2-mkconfig.
• As a last resort, boot with nomodeset or intel_idle.max_cstate=1 to sidestep graphics or CPU‑idle quirks that may interfere with early kernel initialization.

Diagnosing the stall with systemd‑based tools

Once the firmware side is ruled out, the remaining delay is usually accounted for by systemd services or the initramfs itself. The following commands provide a clear picture of where time is being spent:

systemd-analyze blame          # shows the services that take the longest
systemd-analyze critical-chain initrd-udevd.service
journalctl -b -1 | grep -iE 'timeout|wait|hang'
dmesg | grep -iE 'timeout|wait|error'

If a particular service (e.In real terms, g. , systemd-modules-load.service or a custom cryptsetup.Now, service) is flagged as the bottleneck, you can temporarily mask it (systemctl mask <service>) to see whether the boot proceeds. Remember to re‑enable the service after testing.

Wrapping up

When a system appears to “randomly” hang during boot, the investigation typically follows a three‑tiered approach:

1. Firmware alignment – Ensure the hardware initialisation mode (SATA, PCIe, Secure Boot) matches the kernel’s expectations, and keep the firmware up to date.
2. Boot‑parameter hygiene – Verify that the kernel command line contains only the parameters you truly need, and consider adding debug, rd.debug, or systemd.unit=shutdown.target to isolate the offending stage.
3. Service‑level inspection – Use systemd-analyze and the journal to pinpoint services or initramfs scripts that consume excessive time, then mask, edit, or replace the problematic unit.

By systematically checking the firmware settings, refining the kernel command line, and drilling into systemd’s timing data, most “random 90‑second hang” symptoms can be identified and resolved without resorting to trial‑and‑error reboots. Once the root cause is addressed, the system will boot reliably, and the time spent troubleshooting can be redirected to more productive tasks.