DEV Community

whiteSama001
whiteSama001

Posted on

Debugging Redis Remote Access on RHEL: A Four-Layer Problem That Looked Like One

#redis #linux

When a request comes in as "the app can't connect to Redis," it's tempting to assume it's a one-line fix — open a port, flip a config flag, done. In practice, on a hardened RHEL/CentOS host, a single "connection refused" error can be hiding four separate problems stacked on top of each other: network-level firewall rules, SELinux policy, file-level permissions, and Redis's own safety defaults. Each layer will silently swallow the connection and give you almost no useful signal about which one is actually blocking you.

This is a walkthrough of how I diagnosed and fixed exactly that scenario — a Redis instance that needed to accept connections from an application server on the same internal subnet, but wouldn't.

The symptom

A service running on a separate host in the same internal network needed to read/write to a Redis instance. The redis-cli -h <redis-host> ping command from the app server just hung, then timed out. No obvious error in the Redis logs. systemctl status redis showed it running happily on the Redis host itself, and redis-cli ping worked fine locally on that box.

That local-works, remote-fails split is the first useful clue: it rules out a broken Redis process and points squarely at something in the path between the two hosts, or in how Redis is bound.

Layer 1: Is Redis even listening on the right interface?

The default Redis config binds to 127.0.0.1 only — deliberately, since Redis has no authentication by default and was never meant to be exposed carelessly. First check:

sudo ss -tlnp | grep redis
Enter fullscreen mode Exit fullscreen mode

If you only see 127.0.0.1:6379, that's your answer for this layer. The fix is in redis.conf:

bind 0.0.0.0 -::1
# or, more safely, bind to the specific internal interface:
bind 127.0.0.1 10.0.x.x
Enter fullscreen mode Exit fullscreen mode

Restart the service and re-check with ss. If the socket now shows the right interface, move to the network layer — this alone rarely fixes remote access on a hardened host.

Layer 2: The firewall

RHEL-family systems default to firewalld. Even if the app team swears "the network team already opened the port," verify it yourself:

sudo firewall-cmd --list-all
Enter fullscreen mode Exit fullscreen mode

If 6379/tcp isn't in the services or ports list for the active zone, add it:

sudo firewall-cmd --zone=internal --add-port=6379/tcp --permanent
sudo firewall-cmd --reload
Enter fullscreen mode Exit fullscreen mode

Don't forget any cloud-level security group or NSG sitting in front of the host — firewalld being correct doesn't mean the packet ever arrives if a cloud ACL drops it first. Test from the app server with:

nc -zv <redis-host> 6379
Enter fullscreen mode Exit fullscreen mode

If that now succeeds but redis-cli ping still hangs or gets refused, the packet is arriving — the block is happening inside the host, which is where most people stop looking too early.

Layer 3: SELinux — the layer everyone forgets

This is the one that trips people up because it fails silently and looks identical to a network problem from the outside. SELinux enforces policy independently of standard file/network permissions, and a denial doesn't show up in the application's own logs — it shows up in the audit log.

Check for denials:

sudo ausearch -m avc -ts recent
Enter fullscreen mode Exit fullscreen mode

or, more readably:

sudo sealert -a /var/log/audit/audit.log
Enter fullscreen mode Exit fullscreen mode

Two common findings in this scenario:

  1. Port context: if you changed Redis to listen on a non-default port, SELinux won't know that port is allowed for the redis_port_t type until you tell it: 
   sudo semanage port -a -t redis_port_t -p tcp <new_port>
Enter fullscreen mode Exit fullscreen mode
  1. File context on data/config paths: if the Redis data directory or config file was moved, copied, or restored from a backup in a way that didn't preserve SELinux labels, the daemon can be denied access to its own files: 
   sudo restorecon -Rv /var/lib/redis /etc/redis
Enter fullscreen mode Exit fullscreen mode

Never disable SELinux to make this go away — setenforce 0 "fixes" the symptom by removing a security boundary you presumably want on a production host. Diagnose the specific denial and allow exactly that.

Layer 4: File ownership

Related but distinct from SELinux context: standard Unix ownership. If the Redis data directory, RDB/AOF files, or config file aren't owned by the redis user (often because of a manual cp or a restore run as root), the daemon can fail to start cleanly or fail to persist — which then surfaces upstream as flaky connectivity when the process restarts unexpectedly.

sudo chown -R redis:redis /var/lib/redis
sudo chmod 750 /var/lib/redis
Enter fullscreen mode Exit fullscreen mode

Layer 5: Redis's own protected mode

Even once the network, SELinux, and ownership are all correct, Redis has one more guard rail: protected mode. If Redis is bound to a non-loopback address without a password configured, it refuses external connections by default and logs a warning about it. This is Redis protecting you from accidentally exposing an unauthenticated instance to the world.

The correct fix is not to disable protected mode — it's to set a password:

requirepass <strong-password>
Enter fullscreen mode Exit fullscreen mode

Then connect as:

redis-cli -h <redis-host> -a <strong-password> ping
Enter fullscreen mode Exit fullscreen mode

If for some internal-only, tightly firewalled reason you genuinely don't want auth, you can explicitly set protected-mode no — but treat that as a deliberate, documented exception, not a quick fix.

Why this matters beyond Redis

The specific commands here are Redis-and-RHEL-flavored, but the debugging shape generalizes to almost any "service X can't talk to service Y" ticket on a hardened Linux estate:

  1. Confirm the process is actually listening where you think it is (ss/netstat).
  2. Confirm the packet can physically arrive (firewall, security groups, routing).
  3. Confirm the kernel-level mandatory access control isn't silently dropping it (SELinux/AppArmor — check the audit log, don't guess).
  4. Confirm file/process ownership is correct.
  5. Only then look at the application's own safety defaults.

Skipping straight to "just open the port" or "just disable SELinux" will often look like it worked in a rushed fix, while actually just removing a control someone put there on purpose. Working through the layers in order — and confirming each one with a command, not an assumption — gets you to a fix you can actually explain and defend later, which matters a lot more in a regulated environment (this happened on banking infrastructure) than in a hobby project.

A couple of open questions I'd flag to anyone in a similar spot

Two things I left as follow-ups rather than closing out immediately, because they're policy decisions, not bugs:

  • dir setting: where RDB/AOF snapshots actually land matters for backup and disk-capacity planning — worth confirming it points somewhere with monitored, appropriately-sized storage rather than leaving the default.
  • Eviction policy: maxmemory-policy needs to match how the application actually uses Redis (cache vs. durable store). Getting this wrong either silently drops data you needed or lets Redis OOM under load.

Neither is a "fix" — they're the kind of thing worth a short design conversation with whoever owns the data before you consider the job fully done.

If you've hit a similar multi-layer wall with Redis, Postgres, or any other service on a hardened RHEL box, I'd be curious to hear which layer got you — SELinux is my usual bet.
Enter fullscreen mode Exit fullscreen mode

Top comments (0)