Crackpot Ideas

In my day job, I rarely get the opportunity to get as “hands on” as I used to – I still work with Oracle, and of course I still work with our software every day.  But the days when I get to roll up my sleeves and dig into a UNIX, storage, or networking problem are few and far between, and I miss it sometimes.

So, today a customer contacted me and asked me for a favor – they were setting up a server to be part of an Oracle RAC cluster, and the ssh equivalence wasn’t working, and could I help try to figure it out?  I was more than happy to give it a shot.

For those of you who aren’t as familiar with Oracle RAC, Oracle’s distributed clustering technology, Oracle basically requires that all the nodes in a RAC cluster be able to ssh or rsh to each other as the database user without passwords.  This even includes that the node be able to ssh to itself without a password.

Prior to ssh, people used rsh for this kind of remote command execution, but there were a number of notable drawbacks to using rsh.  The first was that communications were unencrypted – that may or may not be an issue, considering that if you weren’t using passwords, there wasn’t necessarily sensitive information to sniff on the wire.  The second was that hosts were authenticated based on IP address or hostname – both of which can be spoofed, and hence it wasn’t possible to authoritatively identify hosts.  The third issue was that the user equivalence basically trusted that the originating host was being truthful about who the user is – that is, I might connect to server A using rsh, and say, “I’m mzito @ mycomputer”, and serverA has no definitive way to authenticate that I am actually mzito.

So, SSH is the vastly more enlightened way to do this sort of passwordless authentication, as it addresses all of the complaints about rsh through strong cryptography.  Basically, SSH encrypts all client-server communications, and provides users and services with a key-based mechanism for identifying themselves to other parties, very similarly to how SSL works with HTTPS.  Like many things, it works really well until it doesn’t, and then you’re stuck in troubleshooting hell.

Let’s quickly look at an overview of the process for setting this up.  Normally when you ssh to a machine, you get a password prompt:

matthew-zitos-macbook:~ mzito$ ssh testacct@crackpot1-dev
testacct@crackpot1-dev's password:
Last login: Thu Jan 29 21:44:15 2009 from (my ip)
[testacct@crackpot1-dev ~]$

Simple enough.  So let’s say we want to ssh to crackpot1-dev as testacct without a password.  The first thing we have to do is generate a key:

matthew-zitos-macbook:~ mzito$ whoami
mzito
matthew-zitos-macbook:~ mzito$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/mzito/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/mzito/.ssh/id_rsa.
Your public key has been saved in /Users/mzito/.ssh/id_rsa.pub.
The key fingerprint is:
6c:ee:e7:a5:42:e9:48:9e:20:4a:9d:ca:fd:a2:b2:e1 mzito@matthew-zitos-macbook.local

What did we just do?  The short answer is that we generated a unique identifier for my user.  The longer answer is that we have generated a public and a private key that can be used to authenticate me to any ssh daemon that is configured to trust my key.  We’ll cover the technology behind public/private key encryption later, but suffice it to say that I can give my public key (id_rsa.pub) to someone,  keep my private key (id_rsa) to myself, and I can prove I am who I say I am to the person who has the public key.  Looking at it another way, as long as i keep my private key private, i can generate a signature with my private key that anyone with my public key can verify is me.

This trust is what allows us to configure our server to accept the key as an identifier instead of a password.  Here’s how:

1. Copy your public key to the server:

matthew-zitos-macbook:~ mzito$ cd .ssh/
matthew-zitos-macbook:.ssh mzito$ ls -l
total 32
-rw-------  1 mzito  staff  1675 Jan 29 21:49 id_rsa
-rw-r--r--  1 mzito  staff   415 Jan 29 21:49 id_rsa.pub
-rw-r--r--  1 mzito  staff  5351 Jan 29 21:45 known_hosts
matthew-zitos-macbook:.ssh mzito$ scp id_rsa.pub testacct@crackpot1-dev:~
testacct@crackpot1-dev's password:
id_rsa.pub                                                  100%  415     0.4KB/s   00:00
matthew-zitos-macbook:.ssh mzito$

2. Next, on the server, log in and copy or append the contents of the id_rsa.pub file to an authorized_keys file (note, if you have not ssh’ed OUT from that server before, you may not have a .ssh directory in your home directory – simply running ssh localhost will create it):

matthew-zitos-macbook:.ssh mzito$ ssh testacct@blog.crackpotideas.com
testacct@blog.crackpotideas.com's password:
Last login: Thu Jan 29 21:44:19 2009 from cpe-72-229-230-230.nyc.res.rr.com
[testacct@crackpot1-dev ~]$ ls
id_rsa.pub
[testacct@crackpot1-dev ~]$ ssh localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is c0:7d:6f:ea:ae:cf:ea:9b:a2:43:1d:01:ec:71:75:83.
Are you sure you want to continue connecting (yes/no)? 

[testacct@crackpot1-dev ~]$ ls
id_rsa.pub
[testacct@crackpot1-dev ~]$ ls -la
total 40
drwx------ 3 testacct testacct 4096 Jan 29 22:13 .
drwxr-xr-x 5 root     root     4096 Jan 29 21:43 ..
-rw-r--r-- 1 testacct testacct   33 Jan 29 21:43 .bash_logout
-rw-r--r-- 1 testacct testacct   24 Jan 29 21:43 .bash_logout.rpmnew
-rw-r--r-- 1 testacct testacct  176 Jan 29 21:43 .bash_profile
-rw-r--r-- 1 testacct testacct  176 Jan 29 21:43 .bash_profile.rpmnew
-rw-r--r-- 1 testacct testacct  124 Jan 29 21:43 .bashrc
-rw-r--r-- 1 testacct testacct  124 Jan 29 21:43 .bashrc.rpmnew
drwx------ 2 testacct testacct 4096 Jan 29 22:13 .ssh
-rw-r--r-- 1 testacct testacct  415 Jan 29 22:07 id_rsa.pub
[testacct@crackpot1-dev ~]$ cd .ssh/
[testacct@crackpot1-dev .ssh]$ ls -l
total 0
[testacct@crackpot1-dev .ssh]$ cat ../id_rsa.pub >> authorized_keys
[testacct@crackpot1-dev .ssh]$

3. Alright, so – now we’ve told sshd on my crackpot1-dev server that if someone shows up identifying themself with the private key that corresponds to the public key in authorized_keys, we should let them in without a password.  Let’s give it a shot:

matthew-zitos-macbook:.ssh mzito$ ssh testacct@blog.crackpotideas.com
testacct@blog.crackpotideas.com's password:

4. Whoops – what happened?  This, as it turns out will dovetail nicely with the next blog post on this subject – but for the moment, let’s just fix this.  Set the permissions on authorized_keys to 600:

testacct@crackpot1-dev .ssh]$ ls -l authorized_keys
-rw-rw-r-- 1 testacct testacct 415 Jan 29 22:16 authorized_keys
[testacct@crackpot1-dev .ssh]$ chmod 600 authorized_keys
[testacct@crackpot1-dev .ssh]$ ls -l
total 4
-rw------- 1 testacct testacct 415 Jan 29 22:16 authorized_keys
[testacct@crackpot1-dev .ssh]$

and try again:

matthew-zitos-macbook:.ssh mzito$ ssh testacct@crackpot1-dev
Last login: Thu Jan 29 22:12:35 2009 from (my ip)
[testacct@crackpot1-dev ~]$

And there we go.  Next section, we’ll explain exactly what went wrong, and how you can troubleshoot these problems on your own – “the bad and the ugly”.