Callandor

Just spent an hour troubleshooting why I couldn’t SSH into a RHEL 5.4 machine.

Symptoms:

- Can’t SSH in, even as root; login succeeds but before a shell is opened, it exits. Running SSHD in debug mode on the server, the following is printed on the client:

debug1: PAM: reinitializing credentials
debug1: permanently_set_uid: 0/0
debug3: PAM session not opened, exiting

And /var/log/secure has the following to say:

Apr 19 09:22:14 rh02 sshd[5679]: Accepted password for root from 10.1.1.1 port 47152 ssh2
Apr 19 09:22:15 rh02 sshd[5679]: pam_unix(sshd:session): session opened for user root by (uid=0)
Apr 19 09:22:15 rh02 sshd[5679]: error: PAM: pam_open_session(): Permission denied
Apr 19 09:36:05 rh02 login: pam_unix(login:session): session opened for user root by LOGIN(uid=0)
Apr 19 09:36:05 rh02 login: Permission denied

- Even an su fails, with message: could not open session

Apr 19 09:36:16 rh02 login: pam_unix(login:session): session opened for user alucard by LOGIN(uid=0)
Apr 19 09:36:16 rh02 login: Permission denied

- GUI / VTY login doesn’t work either.

I checked all the obvious things, like:

• Permissions on /etc/passwd were 0644
• Permissions on /etc/shadow were 0700
• /var/empty existed
• /, /var, and /var/empty were NOT world-writable
• Setting UsePAM to no in /etc/ssh/sshd_config made it work
• /etc/pam.d/* was identical with a working machine

Finally this bug report led me to open /etc/security/limits.conf. Sure enough, the “nofiles” hard and soft ulimits were set to “unlimited,” and sure enough when I changed them to “10240” I was able to SSH in normally.

That was stupid.