Secure file distribution with apache, openssl, and curl

This is a long post. Be warned.

I have a problem to solve. I need to share a file on a regular basis with a small group of users. It has to be done securely and upload and download must be automated. It also has to be cheap and simple.

The plan is to layer HTTP Basic and SSL Client Auth for directory access to the BSA files. This provides encryption, strong authentication, and with HTTP Basic, the ability to disable an account easily (even though their client certificate may still be good. I didn't want to deal with certificate revocation lists). For retrieval, curl is a great option and meets all requirements.

Tools and assumptions

I used Ubuntu (8.04 or later), apache2 (2.2), and the latest curl and openssl packages. You know some *nix.

Apache2 configuration

The BSA file will be placed in a directory called 'bsa', so create this in /var/www (the default)

Add the following to the /etc/apache2/httpd.conf file to protect your new directory.

<Directory /var/www/bsa>

AuthType Basic

AuthName "Approved BSA users"

AuthUserFile /usr/local/apache/passwd/passwords

AuthGroupFile /usr/local/apache/passwd/groups

Require group BsaUsers

</Directory>

Note that this will be changed later for the final solution. I am just showing how to do this in general.

You will need to create the directories apache and passwd so that the path /usr/local/apache/passwd exists. Here is how to create the files 'passwords' and 'groups'.

group file creation

You simply make a text file (called 'groups') with an entry like this:

BsaUsers: user1 user2 user3

Everything after "BsaUsers:" are the user names for who has access. Each will need a password.

passwords file creation

For this you must use an apache tool for creating password files, htpasswd. The command below creates the passwords file (-c option) and adds one user, user1. You will be prompted to add their password.

>htpasswd -c /usr/local/apache/passwd/passwords user1

Adding more users to the password file

>htpasswd /usr/local/apache/passwd/passwords user2

>htpasswd /usr/local/apache/passwd/passwords user3

(each time you are prompted for the password for the user)

Note: htpasswd is in /usr/local/apache2/bin in case that location is not on your path.

Adding a new user

1. add the user name to the 'group's file.
2. add a password for the new user as above

Now test that http basic auth is working for the direcory 'bsa' in the root of the web server. After writing to the httpd.conf file you may need to restart the server.

TIP: apache2 service name is not httpd, it is apache2. So restart is this:

>service apache2 restart

Now on to how to access an http basic protected directory using curl. First, when you try curl against the protected directory it will appear to work. If you look at what you downloaded, though, it will be a 401 Authentication Required file.

Here is how to use curl on a resource (URL) requiring http basic authentication:

>curl -u name:password www.example.com

So for my example, the command is this:

>curl -u user1:password1 -O localhost/bsa/testfile.zip

This will send the credentials to the server and download and save the requested file. It's that simple.

Now for the main event: SSL CLient authentication set up

(see http://httpd.apache.org/docs/2.2/ssl/ssl_howto.html and http://httpd.apache.org/docs/2.2/ssl/ssl_faq.html for complete details)

Enabling SSL

>a2enmod ssl

>service apache2 restart

You can verify that /etc/apache2/mods-enabled now has the ssl.conf and ssl.load symbolic links in it.

Creating Certs and Keys

I tried CA.pl, a script for working with openssl for creating server certificates and it caused no end of problems arround the private key not being found. I believe it is all due to a format issue but not worth hassling with. By using openssl directly, as follows, I got it working easily.

Create a RSA private key for your server (will be Triple-DES encrypted and PEM formatted):

>openssl genrsa -des3 -out server.key 1024

Please backup this host.key file and the pass-phrase you entered in a secure location. You can create a stronger key by changing 1024 to something like 4096.

You can see the details of this RSA private key by using the command:

>openssl rsa -noout -text -in server.key

Create a self-signed Certificate (X509 structure) with the RSA key you just created (output will be PEM formatted):

>openssl req -new -x509 -nodes -sha1 -days 365 -key server.key -out server.crt

This signs the server CSR and results in a server.crt file. Set 'days' to whatever you like.

Copy the server.crt file and the server.key file to a directory under /etc/apache2. You can make your own- just don't use conf.d since that will cause other errors.

You must set up apache2 so your server uses SSL and your new certs. I did this using a virtual host for SSL. You will find these under /etc/apache2/sites-enabled/000-default (I tried using the main config file, but ran into problems. Using a virtual host works.)

Add the following to 000-default:

<VirtualHost *:443>

DocumentRoot /var/www

SSLEngine on

SSLCertificateFile /etc/apache2/conf/ssl/ca.crt

SSLCertificateKeyFile /etc/apache2/conf/ssl/server.key

<Directory /var/www/bsa>

#ssl client auth set up

SSLVerifyClient require

SSLVerifyDepth 1

SSLCACertificateFile conf/ssl/ca.crt

# http basic auth set up

AuthType Basic

AuthName "Approved BSA users"

AuthUserFile /usr/local/apache/passwd/passwords

AuthGroupFile /usr/local/apache/passwd/groups Require group BsaUsers

</Directory>

</VirtualHost>

Second, I ensure you can only access the private area using https with this:

<Directory /var/www/bsa>

Deny from all

</Directory>

This goes in the virtual host for port 80 (in 000-default) and blocks all access except through port 443. At this point, you have a web server that supports both http and https traffic with a protected directory (bsa) that can only be accessed as follows:

• https
• with username and password
• and a client certificate

Now when I try to run the server, here's what I get. Note that the cert private key is found, as evidenced by being challenged for its passphrase.

>/etc/apache2# service apache2 restart

* Restarting web server apache2 Apache/2.2.12 mod_ssl/2.2.12 (Pass Phrase Dialog)

Some of your private key files are encrypted for security reasons.

In order to read them you have to provide the pass phrases.

Server localhost:443 (RSA)

Enter pass phrase:

OK: Pass Phrase Dialog successful.

When accessing using a browser I am getting the message I want- the server certificate is untrusted. This confirms apache is using my self-signed cert.

I accept the 'untrusted' cert, and get the following error in Firefox:

An error occurred during a connection to localhost.

SSL peer was unable to negotiate an acceptable set of security parameters.

(Error code: ssl_error_handshake_failure_alert)

This is also expected, since I am requiring client authentication, but my client has no certificate. Just to confirm, I removed the requirement for client auth from httpd.conf, and restarted the server. Now when I use https, I can get to the web page after accepting the cert. This is all good.

Next step- use a client cert for authentication

First, create a sample client certificate using own CA created earlier (and stored in 'selfCA').

Create a private key

>openssl genrsa -des3 -out server.key 1024

Create a CSR using your new key

>openssl req -new -key server.key -out server.csr

Sign CSR using your own CA

>openssl x509 -req -days 1100 -in server.csr -CA ../selfCA/server.crt -CAkey ../selfCA/server.key -set_serial 01 -out test.crt

You will be asked for the passphrase for server.key.

Now to use test.crt and the key. To work with curl, the cert and key need to be in the same file.

>cat test.crt server.key > client.crt

This just combines the files into one text file. Take a look.

Here's the finished curl command to download the file using client authentication and basic authentication.

>curl -u user1:password1 -k https://localhost/bsa/testfile.zip -E client.crt:passphrase1 -O https://localhost/bsa/testfile.zip

The explanation of everything

• -u username and password set up on the server for http basic authentication
• -k ignore certificate warnings from my server (since using self-signed CA with mismatched CN and hostname)
• -E specifies file with the client cert and key to use plus the key passphrase
• -O download and save the file from give URL and using the filename in the URL

Pretty cool, huh?