Masking 2-Way “Mutual” SSL Authentication using F5 LTM or HAProxy

Hello folks,

So a recent post I published talked about 1-Way vs 2-way SSL Authentication in some decent detail. We learned that 2-Way “Mutual” SSL Authentication can be used to enforce both parties attempting to communicate securely to provide authenticity. In other words, prove to each other that they are who they say they are. This can be very powerful from a security standpoint, but is it practical? The answer is, yes and no. The constraint comes from the aspect of administration (actually create certificates for each client) and manageability (keep accounting and maintaining actively lists of trusts) with the trade-off of proper authenticity. For example at first administering and managing 10 client certificates may be okay, but then imaging 100, or even a 1,000! So in this post I wanted to approach the idea of utilizing some tools we can use to offload some of this administration and management while maintaining Mutual Authentication with another entity. The idea revolves around one major assumption, users of a particular service (In this case a web-server) reside on a privately controlled and trusted network

My idea is if we have a group of clients residing on an internal privately addressed network, we can use either an F5 LTM or HAProxy to proxy our users’s connections destined for a service that is enforcing 2-Way SSL “Mutual” Authentication. The F5 LTM or HAProxy would perform the 2-Way SSL Mutual Authentication on behalf of each connecting user, eliminating the technical need to generate certificates for each client, while maintaining an element of mutual trust to the end service.

The basic idea is: (notice only our F5 LTM/HAproxy and the web-server perform 2-Way “Mutual” Authentication)

Preliminary Steps:

For the following steps please read my 1-Way vs 2-Way SSL Authentication Post.

  1. Create a the web-server’s CSR and Key
    root@ca:/opt# openssl req -config openssl-rootca.conf -extensions server_req_ext -new -nodes -newkey rsa:2048 -keyout web-server.key -out web-server.csr -days 365
    Generating a 2048 bit RSA private key
    ................................+++
    ............................................................................+++
    writing new private key to 'web-server2.key'
    -----
    You are about to be asked to enter information that will be incorporated
    into your certificate request.
    What you are about to enter is what is called a Distinguished Name or a DN.
    There are quite a few fields but you can leave some blank
    For some fields there will be a default value,
    If you enter '.', the field will be left blank.
    -----
    Enter Country [US]:
    State or Province Name (full name) [Connecticut]:
    Locality Name (eg, city) [Wethersfield]:
    Organization Name [thejimmahknows]:
    Unit Name [Test Unit]:
    Common Name (e.g. server FQDN or YOUR name) []:web-server
    Contact email for this Certificate [admin@example.com]:
    

     

  2. Create the F5 & HAproxy Server-Side CSR and Key
    root@ca:/opt# openssl req -config openssl-rootca.conf -extensions client_req_ext -new -nodes -newkey rsa:2048 -keyout ha-client1.key -out ha-client1.csr -days 365
    Generating a 2048 bit RSA private key
    ...................+++
    .....+++
    writing new private key to 'ha-client1.key'
    -----
    You are about to be asked to enter information that will be incorporated
    into your certificate request.
    What you are about to enter is what is called a Distinguished Name or a DN.
    There are quite a few fields but you can leave some blank
    For some fields there will be a default value,
    If you enter '.', the field will be left blank.
    -----
    Enter Country [US]:
    State or Province Name (full name) [Connecticut]:
    Locality Name (eg, city) [Wethersfield]:
    Organization Name [thejimmahknows]:
    Unit Name [Test Unit]:
    Common Name (e.g. server FQDN or YOUR name) []:ha-client1
    Contact email for this Certificate [admin@example.com]:
    

     

  3. Create the F5 & HAproxy Client-Side (connection the client will actual connect to)
    root@ca:/opt# openssl req -config openssl-rootca.conf -extensions server_req_ext -new -nodes -newkey rsa:2048 -keyout virtual-service.key -out virtual-service.csr -days 365
    Generating a 2048 bit RSA private key
    ........................+++
    .............................................................+++
    writing new private key to 'virtual-service.key'
    -----
    You are about to be asked to enter information that will be incorporated
    into your certificate request.
    What you are about to enter is what is called a Distinguished Name or a DN.
    There are quite a few fields but you can leave some blank
    For some fields there will be a default value,
    If you enter '.', the field will be left blank.
    -----
    Enter Country [US]:
    State or Province Name (full name) [Connecticut]:
    Locality Name (eg, city) [Wethersfield]:
    Organization Name [thejimmahknows]:
    Unit Name [Test Unit]:
    Common Name (e.g. server FQDN or YOUR name) []:mytestvip
    Contact email for this Certificate [admin@example.com]:
    
  4. Using our CA from my previous ariticle, sign all three of these certificates

    Sign the web-server’s CSR

    root@ca:/opt# openssl ca -config openssl-rootca.conf -extensions server_req_ext -in web-server2.csr -out web-server2.crt
    Using configuration from openssl-rootca.conf
    Enter pass phrase for /opt/rootCA.key:
    Check that the request matches the signature
    Signature ok
    The Subject's Distinguished Name is as follows
    countryName           :PRINTABLE:'US'
    stateOrProvinceName   :ASN.1 12:'Connecticut'
    localityName          :ASN.1 12:'Wethersfield'
    organizationName      :ASN.1 12:'thejimmahknows'
    organizationalUnitName:ASN.1 12:'Test Unit'
    commonName            :ASN.1 12:'web-server'
    Certificate is to be certified until Feb 25 15:10:27 2017 GMT (365 days)
    Sign the certificate? [y/n]:y
    
    
    1 out of 1 certificate requests certified, commit? [y/n]y
    Write out database with 1 new entries
    Data Base Updated
    

    Sign the F5 & HAProxy Server-Side CSR (This is the connection the F5 makes to our backend pool members)

    root@ca:/opt# openssl ca -config openssl-rootca.conf -extensions client_req_ext -in ha-client1.csr -out ha-client1.crt
    Using configuration from openssl-rootca.conf
    Enter pass phrase for /opt/rootCA.key:
    Check that the request matches the signature
    Signature ok
    The Subject's Distinguished Name is as follows
    countryName           :PRINTABLE:'US'
    stateOrProvinceName   :ASN.1 12:'Connecticut'
    localityName          :ASN.1 12:'Wethersfield'
    organizationName      :ASN.1 12:'thejimmahknows'
    organizationalUnitName:ASN.1 12:'Test Unit'
    commonName            :ASN.1 12:'ha-client1'
    Certificate is to be certified until Feb 25 15:12:16 2017 GMT (365 days)
    Sign the certificate? [y/n]:y
    
    
    1 out of 1 certificate requests certified, commit? [y/n]y
    Write out database with 1 new entries
    Data Base Updated
    

    Sign the F5 & HAProxy Client-Side CSR (This is the connection our users will connect to, the VIP)

    root@ca:/opt# openssl ca -config openssl-rootca.conf -extensions server_req_ext -in virtual-service.csr -out virtual-service.crt
    Using configuration from openssl-rootca.conf
    Enter pass phrase for /opt/rootCA.key:
    Check that the request matches the signature
    Signature ok
    The Subject's Distinguished Name is as follows
    countryName           :PRINTABLE:'US'
    stateOrProvinceName   :ASN.1 12:'Connecticut'
    localityName          :ASN.1 12:'Wethersfield'
    organizationName      :ASN.1 12:'thejimmahknows'
    organizationalUnitName:ASN.1 12:'Test Unit'
    commonName            :ASN.1 12:'mytestvip'
    Certificate is to be certified until Feb 25 15:15:09 2017 GMT (365 days)
    Sign the certificate? [y/n]:y
    
    
    1 out of 1 certificate requests certified, commit? [y/n]y
    Write out database with 1 new entries
    Data Base Updated
    

     

  5. Configure Apache web-server to enforce the 2-Way “Mutual” Authentication

    Apache Config from previous post.

    root@web-server:/opt# vi /etc/apache2/sites-available/default.conf
    Listen 443
    <VirtualHost *:443>
            DocumentRoot           "/var/www/"
    
            SSLEngine               on
            SSLCACertificateFile   /opt/rootCA.crt
            SSLCertificateFile     /opt/web-server.crt
            SSLCertificateKeyFile  /opt/web-server.key
           SSLCARevocationFile    /opt/rootCRL.crl
            SSLStrictSNIVHostCheck on
            SSLVerifyClient        require
            SSLVerifyDepth         1
    
    # Allows PHP to read Certificate info
            SSLOptions +stdEnvVars
    
            LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
            CustomLog "/tmp/access.log" combined
            ErrorLog "/tmp/error.log"
    
    </VirtualHost>
    

    Restart Apache

    root@web-server:/opt# service apache2 restart

    For Troubleshooting create this index.php file

    <?php
    echo "Date       =   " . date('Y-m-d H:i:s') . "<br>";
    
    echo "Client Cert =  " . $_SERVER['SSL_CLIENT_S_DN_CN'] . "<br>";
    echo "Server Cert =  " . $_SERVER['SSL_SERVER_S_DN_CN'] . "<br>";
    echo "Server Serial =  " . $_SERVER['SSL_SERVER_M_SERIAL'] . "<br>";
    ?>
    

     

Masking with F5 LTM:

  1. Importing Certificates

    • Import virtual-service certificate
    • Import ha-client1 certificate
      • Use the same steps from above.
    • Import the rootCA certificate (used to authenticate the web-server)
      • Use the same steps from above.
    • All 3 Certificates imported
  2. Create the SSL Profiles
    • Create the client-side-connection SSL profile
    • Create the server-side-connection SSL profile
  3. Create the Server Pool
  4. Create the Virtual Server aka VIP
  5. Test by using Chrome to connect to our virtual-service

  6. Certificate revocation
    • Revoke the ha-client1.crt (The certificate the F5 authenticates with when connecting to the web-server)
      root@ca:/opt# openssl ca -config openssl-rootca.conf -revoke ha-client1.crt -crl_reason keyCompromise
      Using configuration from openssl-rootca.conf
      Enter pass phrase for /opt/rootCA.key:
      Revoking Certificate 1005.
      Data Base Updated
      
    • Re-generate the CRL
      root@ca:/opt# openssl ca -config openssl-rootca.conf -gencrl -out rootCRL.crl
      Using configuration from openssl-rootca.conf
      Enter pass phrase for /opt/rootCA.key:
      
      root@ca:/opt# openssl crl -noout -text -in rootCRL.crl
      Certificate Revocation List (CRL):
              Version 2 (0x1)
          Signature Algorithm: sha256WithRSAEncryption
              Issuer: /C=US/ST=Connecticut/L=Wethersfield/O=thejimmahknows/OU=Test Unit/CN=MyRootAuthority/emailAddress=admin@example.com
              Last Update: Feb 26 18:42:33 2016 GMT
              Next Update: Mar 27 18:42:33 2016 GMT
              CRL extensions:
                  X509v3 Authority Key Identifier: 
                      keyid:E3:A6:FD:69:23:0A:25:AF:7B:77:7A:B8:03:0B:B6:8A:CF:F2:B2:B8
      
                  Authority Information Access: 
                      CA Issuers - URI:http://ocsp.thejimmahknows.com/rootCA.crt
      
                  X509v3 CRL Number: 
                      4100
      Revoked Certificates:
          Serial Number: 1003
              Revocation Date: Jan 16 18:55:22 2016 GMT
          Serial Number: 1005
              Revocation Date: Feb 26 18:38:04 2016 GMT
              CRL entry extensions:
                  X509v3 CRL Reason Code: 
                      Key Compromise
      

      Notice the Serial Number 1005 is revoked in the CRL file now.

    • Replace the rootCRL.crl file on the web-server and restart Apache
      scp /opt/rootCRL.crl root@web-server:/opt/rootCRL.crl
      
      root@web-server:/opt# service apache2 restart
    • Test using the virtual-service VIP on the F5 again.

Masking with HAProxy:

If you are unfamiliar with HAProxy, I recommend checking out my article on setting up HAProxy. Or my articles on using HAProxy as a F5 LTM replacement.

  1. Before we being, we have to generate and sign another certificate and key because we revoked the ha-client.crt perviously.
    root@ca:/opt# openssl req -config openssl-rootca.conf -extensions client_req_ext -new -nodes -newkey rsa:2048 -keyout ha-client2.key -out ha-client2.csr -days 365
    Generating a 2048 bit RSA private key
    .+++
    ....................................+++
    writing new private key to 'ha-client2.key'
    -----
    You are about to be asked to enter information that will be incorporated
    into your certificate request.
    What you are about to enter is what is called a Distinguished Name or a DN.
    There are quite a few fields but you can leave some blank
    For some fields there will be a default value,
    If you enter '.', the field will be left blank.
    -----
    Enter Country [US]:
    State or Province Name (full name) [Connecticut]:
    Locality Name (eg, city) [Wethersfield]:
    Organization Name [thejimmahknows]:
    Unit Name [Test Unit]:
    Common Name (e.g. server FQDN or YOUR name) []:ha-client2
    Contact email for this Certificate [admin@example.com]:
    

    And Sign it

    root@ca:/opt# openssl ca -config openssl-rootca.conf -extensions client_req_ext -in ha-client2.csr -out ha-client2.crt
    Using configuration from openssl-rootca.conf
    Enter pass phrase for /opt/rootCA.key:
    Check that the request matches the signature
    Signature ok
    The Subject's Distinguished Name is as follows
    countryName           :PRINTABLE:'US'
    stateOrProvinceName   :ASN.1 12:'Connecticut'
    localityName          :ASN.1 12:'Wethersfield'
    organizationName      :ASN.1 12:'thejimmahknows'
    organizationalUnitName:ASN.1 12:'Test Unit'
    commonName            :ASN.1 12:'ha-client2'
    Certificate is to be certified until Feb 25 19:47:40 2017 GMT (365 days)
    Sign the certificate? [y/n]:y
    
    
    1 out of 1 certificate requests certified, commit? [y/n]y
    Write out database with 1 new entries
    Data Base Updated
    
  2. Copy All 3 certificates to our HAProxy server
    root@ca:/opt# scp virtual-service.* root@172.16.0.44:/opt/
    root@ca:/opt# scp ha-client2.* root@172.16.0.44:/opt/
    root@ca:/opt# scp rootCA.crt root@172.16.0.44:/opt/
  3. We need to chain the virtual-service certificate with the root CA certificate for HAProxy to accept it. (For help reference this)
    root@test-haproxy:/opt# cat virtual-service.key >> virtual-service-chain.pem
    root@test-haproxy:/opt# cat virtual-service.crt >> virtual-service-chain.pem
    root@test-haproxy:/opt# cat rootCA.crt >> virtual-service-chain.pem

    And ha-client2

    root@test-haproxy:/opt# cat ha-client2.key >> ha-client2.pem
    root@test-haproxy:/opt# cat ha-client2.crt >> ha-client2.pem 
    
  4.  Edit your haproxy.conf file to match
    global
            log /dev/log    local0
            log /dev/log    local1 notice
            chroot /var/lib/haproxy
            stats socket /run/haproxy/admin.sock mode 660 level admin
            stats timeout 30s
           user haproxy
           group haproxy
            daemon
    
            # Default SSL material locations
            ca-base /etc/ssl/certs
            crt-base /etc/ssl/private
    
            # Default ciphers to use on SSL-enabled listening sockets.
            # For more information, see ciphers(1SSL). This list is from:
            #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
            ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
            ssl-default-bind-options no-sslv3
            tune.ssl.default-dh-param 2048
    
    defaults
            log     global
            timeout connect 5000
            timeout client  50000
            timeout server  50000
            errorfile 400 /etc/haproxy/errors/400.http
            errorfile 403 /etc/haproxy/errors/403.http
            errorfile 408 /etc/haproxy/errors/408.http
            errorfile 500 /etc/haproxy/errors/500.http
            errorfile 502 /etc/haproxy/errors/502.http
            errorfile 503 /etc/haproxy/errors/503.http
            errorfile 504 /etc/haproxy/errors/504.http
    
    
    frontend vs_172.16.0.44_443
            bind 172.16.0.44:443 ssl crt /opt/virtual-service-chain.pem
            default_backend pool_test2waySSL
    
    
    backend pool_test2waySSL
            server testweb01 172.16.0.25:443 ssl verify required ca-file /opt/rootCA.crt crt /opt/ha-client2.pem
    
  5. Success!!
  6. Revocation test
    • Revoke and re-generate the rootCRL.crl file
      root@ca:/opt# openssl ca -config openssl-rootca.conf -revoke ha-client2.crt -crl_reason keyCompromise
      Using configuration from openssl-rootca.conf
      Enter pass phrase for /opt/rootCA.key:
      Revoking Certificate 1007.
      Data Base Updated
      

      And re-generate the CRL

      root@ca:/opt# openssl ca -config openssl-rootca.conf -gencrl -out rootCRL.crl
      Using configuration from openssl-rootca.conf
      Enter pass phrase for /opt/rootCA.key:
      
    • Reload Apache on our web-server to pick up the new CRL file
      root@web-server:/opt# service apache2 restart
    • Now we test….

 

 

 

 

 

 

 

Only 1 comment left Go To Comment

  1. webjockey /

    Yet another comprehensive article on 2 way SSL, not many people understand and know how to make it work, thanks

Leave a Reply

↓