Debian Ssl Keys Compromised!

[Lloyd]

This is an emergency security issue for nearly all Rimuhosting customers, whether or not they run Debian Etch or Debian derivative systems.

It was announced on May 13 that the Debian openssl package in use since 2006 generates ssl keys that fall in a very small, predictable group of keys. These keys can be broken with only about 20 minutes of brute-forcing. The brute-forcing scripts have already been released into the "wild." You can run software on your server to identify compromised keys from a "blacklist" of key signatures in a fraction of a second.

Rimuhosting customers are affected in 2 ways.

First, Rimuhosting staff ssh keys are (most of them, anyway) compromised. I believe these keys are present on every machine that has had a support ticket answered. With only the presence of a single compromised public key in your file /root/.ssh/authorized_keys, your machine can easily be hacked into. The detection software mentioned in resources links below will allow you to identify the compromised Rimuhosting staff keys. If you are unable to run tests for compromised keys on your system immediately, it is best to remove all the Rimuhosting staff private keys from your authorized_keys file, leaving only the "master" Rimuhosting key. Or, you can turn off SSL authentication for your sshd server.

Second, if you have generated ssl keys (server host key, ssh user login key, secure cert, Postfix TTL key, etc.) on a Debian-based system since 2006, the keys will be compromised. If your keys are vulnerable you need to upgrade your openssl packages and generate new keys NOW.

The Debian developers have included, with the new openssl packages, a program to identify compromised keys, and you will find instructions below for replacing keys used by various programs.

Resources:
http://www.debian.org/security/2008/dsa-1576
http://metasploit.com/users/hdm/tools/debian-openssl/
http://www.debian.org/security/key-rollover/

[retep]

FYI we had a _few_ staff (a minority) who had created weak keys (on debian-based boxes).

We are running a ssh job on all our customers servers right now that will replace or remove those affected public keys. This will work on servers where customers have left our ability to access the server.

i.e. you should not need to remove rimuhosting staff keys. (You can do that if you really wish to, of course).

The master rimuhosting key is known not to be affected by this debian bug.

You can check your .ssh/authorized_keys file using the ssh-vulnkey command. This command is available on (just some?) debian/ubuntu distros.

If you find one of our keys in there that is vulnerable, please report it at http://rimuhosting.com/ticket/enterticketdetails.jsp?t_type=TT_OTHER so that we can add that to the script we have fixing up keys.

[goopot]

> The master rimuhosting key is known not to be affected by this debian bug.

Please could you confirm the fingerprint of the rimuhosting master key so that I can check I've left the correct one in place?

Thanks,

Dave.

[Lloyd]

I'm very glad to hear that Rimuhosting has already put together to script to scan for and clean up compromised keys found in customers' authorized_keys files.

I checked my old authorized_keys file and see that I was wrong when I said "most" staff keys listed in authorized_keys are compromised. I had 4 compromised keys listed, and only 2 belonged to Rimuhosting staff.

However, the number of staff keys involved in not important. It only takes a single compromised ssh key in an authorized_keys file to allow root access to an intruder.

I would like to point out that by posting here I am in no way pointing a finger of blame at Rimuhosting. I am only drawing attention to what I feel is a very serious and urgent server security issue which, due to the possibility of compromised keys in authorized_keys, potentially affects all Rimuhosting customers. (Users of Debian Etch - such as myself - have additional work to do.)

Peter, I would like to suggest that you alter your script that is scanning authorized_keys files to detect any compromised key rather than to simply check against a list of compromised Rimuhosting staff keys. This will allow the script to detect compromised keys other than staff keys.

Most of the work for such a detection script has already been done (http://security.debian.org/project/extra/dowkd/dowkd.pl.gz). Note that non-Debian users can use this Python script to detect compromised keys on their server, without having to have the Debian openssl packages installed. In addition to the authorized_keys files, it can automatically check the server host key, any root ssh login key, and any user ssh login key.

By the way, if you are running Debian or Ubuntu and are looking for the ssh-vulnkey command, you can get it by installing the openssh-blacklist package.

Note that the detection software mentioned (ssh-vulnkey and dowkd.pl) will only detect 2048 bit compromised keys. Keys of other lengths (such as 1024 bit) should also be replaced if there is any doubt (possibly created by Debian after Sept. 2006).

Dave, the Rimuhosting "master" key fingerprint is: 1023 0f:9f:90:dd:c0:d5:7d:ae:16:9f:50:b5:f4: ee:47:64.

Lloyd

[jgbillings]

> The master rimuhosting key is known not to be affected by this debian bug.

Please could you confirm the fingerprint of the rimuhosting master key so that I can check I've left the correct one in place?

Thanks,

Dave.

Rimuhosting main key is available here: http://bliki.rimuhosting.com/space/knowledgebase/rimuhosting/rimuhosting+ssh+access

It wasn't created on a Debian system afaik, and was created before the flawed OpenSSL was introduced into Debian anyway.