Solved: Help: DNS Broken in 29.1

#devops #programming #tutorial #cloud

🚀 Executive Summary

TL;DR: After a system update, DNS resolution can fail due to systemd-resolved losing its upstream DNS configuration, even if direct IP pings work. The recommended solution involves configuring /etc/systemd/resolved.conf to specify DNS servers and domains, ensuring systemd-resolved correctly manages /etc/resolv.conf.

🎯 Key Takeaways

In modern Linux, /etc/resolv.conf is often a symlink managed by systemd-resolved, which acts as a local DNS stub resolver on 127.0.0.53.
System updates, network manager changes, or cloud-init scripts can overwrite systemd-resolved’s internal configuration, leading to name resolution failures.
The most robust solution is to configure systemd-resolved directly via /etc/systemd/resolved.conf by setting DNS, FallbackDNS, and Domains directives, then restarting the service.

Struggling with DNS resolution failures after a system update? Learn the root cause of the infamous systemd-resolved issue and explore three practical solutions, from a quick temporary hack to a permanent configuration fix.

So, an Update Broke Your DNS. Let’s Talk About `systemd-resolved`.

I remember a 2 AM outage call. The monitoring dashboards were screaming red, claiming half our services were down. But I could SSH into the boxes just fine. ping 8.8.8.8? Worked like a charm. ping prod-db-01.internal.mycorp? Timeout. Every single time. The new deployment hadn’t touched DNS, so what gives? The culprit, as it often is these days, was a minor OS patch that reset our network config, and a “helpful” little service called systemd-resolved decided my carefully crafted /etc/resolv.conf was merely a suggestion. If that feeling of dread sounds familiar, you’re in the right place. Let’s get this sorted.

What’s Actually Happening Here? The “Why” Behind the Pain

In modern Linux distributions, you’re not writing to /etc/resolv.conf directly anymore. That file is often just a symlink managed by systemd-resolved, a local DNS stub resolver. The goal is noble: to manage DNS settings more dynamically, especially for things like VPNs. In reality, it means your server now points all its DNS queries to a local address, usually 127.0.0.53, and lets systemd-resolved handle the forwarding.

The problem arises when an update, a network manager change, or a cloud-init script overwrites the configuration that systemd-resolved itself uses. When that happens, your server can still talk to IPs, but it has no idea how to look up a name. It’s like having a phone that can call numbers but has lost its entire contact list.

Taming the Beast: Three Ways to Fix Your DNS

We’ve got options, ranging from a quick-and-dirty fix to get you back online to the “proper” long-term solution. I’ve used all three in different situations.

Solution 1: The “Get It Working NOW” Fix (chattr)

This is the classic “brute force” method. We’re going to manually create the /etc/resolv.conf file with the correct settings and then make it immutable, preventing any process (including systemd-resolved) from changing it.

Step 1: First, stop and disable the service that’s causing the problem.

sudo systemctl disable systemd-resolved
sudo systemctl stop systemd-resolved

Step 2: The real /etc/resolv.conf is often managed through a symlink. We need to break that link and create a real file.

sudo rm /etc/resolv.conf
sudo nano /etc/resolv.conf

Step 3: Add your nameservers to the new file. This could be your internal DNS, a public one like Google’s, or your cloud provider’s.

# Our internal DNS servers
nameserver 10.0.1.10
nameserver 10.0.1.11
# Fallback to Google
nameserver 8.8.8.8
search internal.mycorp

Step 4: This is the magic. We lock the file using chattr.

sudo chattr +i /etc/resolv.conf

Darian’s Take: Be warned, this is a hack. It works, and it’ll save you during an outage. But you’re fighting the system, and a future OS update might have unpredictable results. Someone (maybe you, six months from now) will be very confused when they can’t edit that file. Remember to use chattr -i before you try to change it.

Solution 2: The “Do It Right” Permanent Fix (Configure resolved.conf)

This is the method I recommend for production systems. Instead of fighting systemd-resolved, we’re going to tell it how to behave correctly. This way, we’re working *with* the operating system, not against it.

Step 1: Edit the systemd-resolved configuration file.

sudo nano /etc/systemd/resolved.conf

Step 2: Find the [Resolve] section. Uncomment and set the DNS, FallbackDNS, and Domains directives. This tells the service which upstream servers to use.

[Resolve]
DNS=10.0.1.10 10.0.1.11
FallbackDNS=8.8.8.8 8.8.4.4
Domains=internal.mycorp mycorp.com

Step 3: Ensure /etc/resolv.conf is pointing to the systemd stub file. This is the “correct” symlink setup.

sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

Step 4: Restart the service to apply the new configuration.

sudo systemctl restart systemd-resolved
sudo systemctl status systemd-resolved

You can verify it’s working with resolvectl status.

Solution 3: The “Nuclear Option” (Disable and Go Old-School)

Sometimes, you just don’t want to deal with it. Maybe you have legacy scripts or configuration management (like an old version of Puppet) that expects to manage /etc/resolv.conf directly. In this case, you can completely disable systemd-resolved and revert to a more traditional networking setup.

Step 1: Completely stop and disable the service.

sudo systemctl disable systemd-resolved
sudo systemctl stop systemd-resolved

Step 2: Mask the service to prevent any other process from starting it again, ever.

sudo systemctl mask systemd-resolved

Step 3: At this point, you’re on your own. You’ll need to configure your primary networking service (like NetworkManager or systemd-networkd) to manage /etc/resolv.conf for you, or you’ll have to manage it manually as we did in Solution 1 (but without needing chattr). For NetworkManager, you’d typically edit /etc/NetworkManager/NetworkManager.conf and set dns=default or dns=none to give you back control.

Warning: This is a major change. Disabling a core systemd component can have side effects, especially with complex networking setups involving VPNs or containers that might rely on its features. Only do this if you know what you’re doing and are prepared to manage DNS resolution yourself.

Which Path Should You Choose?

Here’s how I break it down for my team:


Solution	Best For	Pros	Cons
1. The Quick Fix (chattr)	Emergency outage mitigation.	Fast, simple, effective immediately.	Brittle, confusing for future troubleshooting, fights the OS.
2. The Permanent Fix (resolved.conf)	Most production systems.	Works with the system, stable, “correct” method.	Requires understanding `systemd` config.
3. The Nuclear Option (disable)	Systems with specific legacy requirements or for experts who prefer manual control.	Gives you full, direct control.	Can have unintended side effects; you lose modern features.

Ultimately, DNS is one of those foundational services that must be rock solid. While it’s frustrating when an update knocks it over, understanding the components at play is the key to building resilient systems. My advice? Take the time to learn Solution 2. Your future self, at 2 AM, will thank you.