Ubuntu Fundamentals: passwd

The Unsung Hero: Deep Dive into passwd on Ubuntu

Introduction

A recent production incident involving compromised SSH keys on a fleet of Ubuntu 22.04 VMs highlighted a critical gap in our password policy enforcement. While key-based authentication is preferred, service accounts and emergency access often rely on passwords. The incident wasn’t a failure of key management, but a lack of granular control and auditing around password complexity, age, and reuse – all managed, at the core, by the passwd utility. Mastering passwd isn’t about simply changing passwords; it’s about understanding the underlying system architecture, security implications, and operational considerations that impact the entire infrastructure, especially in long-term support (LTS) production environments. This post aims to provide a deep dive for experienced system administrators and DevOps engineers.

What is "passwd" in Ubuntu/Linux context?

passwd is the command-line utility for managing user account passwords. On Ubuntu (and Debian-based systems), it interacts directly with /etc/shadow, which stores the encrypted passwords and password aging information. Crucially, passwd doesn’t store the password itself; it hashes it using a strong one-way algorithm (currently SHA-512, configurable via PAM). The /etc/passwd file contains user account information, but not the password hash.

Key components involved:

• PAM (Pluggable Authentication Modules): /etc/pam.d/common-password is the primary configuration file controlling password complexity, hashing algorithms, and other security policies.
• shadow package: Provides the passwd command and manages the /etc/shadow file.
• libpam-modules: Contains the PAM modules used for authentication and password management.
• useradd, usermod: Commands that interact with passwd during user creation and modification.
• chage: Specifically for managing password aging information.

Use Cases and Scenarios

1. Automated User Provisioning: Cloud-init scripts use passwd (via chpasswd) to set initial passwords for newly provisioned VMs, often based on randomly generated strings.
2. Emergency Access: A system administrator needs to reset a user's password remotely after they've forgotten it, requiring secure remote access and passwd execution.
3. Service Account Management: Regularly rotating passwords for service accounts (e.g., database users, application credentials) to minimize the impact of potential compromises.
4. Password Policy Enforcement: Implementing and enforcing strong password policies (length, complexity, age) across the entire infrastructure using PAM configuration.
5. Container Image Hardening: Disabling password authentication entirely in container images intended for key-based access, and ensuring no default passwords exist.

Command-Line Deep Dive

• Changing a user's password: sudo passwd <username> – prompts for the new password.
• Forcing password change on next login: sudo chage -d 0 <username>
• Checking password aging information: chage -l <username>
• Setting a password non-interactively: echo "<username>:<password>" | sudo chpasswd (use with extreme caution, avoid in production scripts without proper security measures).
• Inspecting PAM configuration: cat /etc/pam.d/common-password
• Example PAM configuration snippet (enforcing minimum length):

password requisite pam_cracklib.so retry=3 minlen=12 dcredit=-1 ucredit=-1 ocredit=-1 lcredit=-1

• Checking shadow file permissions: ls -l /etc/shadow (should be 600, owned by root).
• Systemd journal logs related to password changes: sudo journalctl -u passwd (often minimal, but useful for troubleshooting PAM errors).

System Architecture

graph LR
    A[User] --> B(passwd command);
    B --> C{PAM};
    C --> D[pam_unix.so];
    C --> E[pam_cracklib.so];
    D --> F(/etc/shadow);
    E --> F;
    F --> G[Kernel];
    G --> H(Authentication);
    B --> I[Systemd];
    I --> J(/var/log/auth.log);

passwd leverages PAM to delegate authentication and password management tasks. pam_unix.so handles the actual password hashing and storage in /etc/shadow. pam_cracklib.so enforces password complexity rules. Systemd manages the passwd service, and authentication events are logged to /var/log/auth.log. The kernel is ultimately responsible for verifying the password hash during authentication.

Performance Considerations

passwd operations are generally fast, but hashing complex passwords can introduce a slight delay. The primary performance bottleneck is I/O to /etc/shadow, especially on systems with slow storage.

• Benchmarking: Use time passwd <username> to measure the execution time.
• I/O Monitoring: iotop can identify if /etc/shadow is a source of I/O contention.
• Sysctl Tuning: While not directly related to passwd, optimizing filesystem caching (vm.vfs_cache_pressure) can indirectly improve performance.
• Hashing Algorithm: SHA-512 is computationally more expensive than older algorithms like MD5, but provides significantly better security. Changing the algorithm requires careful consideration and testing.

Security and Hardening

• Shadow File Permissions: Ensure /etc/shadow is owned by root and has permissions 600.
• PAM Configuration: Implement strong password policies using PAM modules like pam_cracklib.so, pam_pwquality.so, and pam_unix.so.
• Password History: Configure pam_unix.so to prevent password reuse.
• Account Lockout: Use fail2ban or pam_tally2.so to lock accounts after multiple failed login attempts.
• Auditd: Monitor /etc/shadow for unauthorized modifications using auditd. Example rule: auditctl -w /etc/shadow -p wa -k passwd_changes
• UFW/iptables: Restrict SSH access to authorized networks.

Automation & Scripting

Ansible Example (rotating service account password):

- name: Rotate service account password
  become: true
  user:
    name: <service_account>
    password: "{{ lookup('password', '/dev/null length=16 chars=ascii_letters,digits') }}"
    update_password: on_create

Important: Avoid storing passwords directly in scripts. Use secrets management tools like HashiCorp Vault or Ansible Vault. Ensure scripts are idempotent and include error handling.

Logs, Debugging, and Monitoring

• /var/log/auth.log: Contains authentication-related events, including password changes.
• journalctl -u passwd: Logs from the passwd service.
• dmesg: Kernel messages, potentially useful for diagnosing PAM errors.
• strace passwd <username>: Traces system calls made by passwd, useful for debugging complex issues.
• Monitoring: Monitor /var/log/auth.log for failed password attempts and unauthorized password changes. Use tools like logstash or splunk for centralized logging and analysis.

Common Mistakes & Anti-Patterns

1. Storing passwords in plain text: Never store passwords in scripts or configuration files.
2. Weak password policies: Failing to enforce minimum length, complexity, and age requirements.
3. Incorrect shadow file permissions: Allowing unauthorized access to /etc/shadow.
4. Disabling password aging: Leaving passwords valid indefinitely.
5. Using chpasswd without proper security: Exposing passwords in command history or logs.

Correct vs. Incorrect:

• Incorrect: echo "user:password" | sudo chpasswd
• Correct: Use a secrets management tool to generate and securely pass the password to passwd.

Best Practices Summary

1. Enforce strong password policies with PAM.
2. Regularly rotate passwords for service accounts.
3. Monitor /etc/shadow for unauthorized changes.
4. Use secrets management tools for automation.
5. Implement account lockout mechanisms.
6. Regularly audit password policies and configurations.
7. Keep the shadow package updated.
8. Utilize centralized logging for authentication events.
9. Disable password authentication where possible (favor key-based access).
10. Document password management procedures and standards.

Conclusion

passwd is a foundational component of Ubuntu system security. While often overlooked, a deep understanding of its architecture, configuration, and operational implications is crucial for maintaining a secure and reliable infrastructure. Proactive auditing of password policies, automated password rotation, and robust monitoring are essential steps to mitigate the risks associated with compromised credentials. Take the time to review your current passwd configuration, build automated scripts for password management, and establish clear standards for password security across your environment.