Finally managed to solve all the issues, so I'll answer my own question. These are the settings/files I've used to manage to get my particular problem(s) solved;
The client's keystore is a PKCS#12 format file containing
- The client's public certificate (in this instance signed by a self-signed CA)
- The client's private key
To generate it I used OpenSSL's pkcs12
command, for example;
openssl pkcs12 -export -in client.crt -inkey client.key -out client.p12 -name "Whatever"
Tip: make sure you get the latest OpenSSL, not version 0.9.8h because that seems to suffer from a bug which doesn't allow you to properly generate PKCS#12 files.
This PKCS#12 file will be used by the Java client to present the client certificate to the server when the server has explicitly requested the client to authenticate. See the Wikipedia article on TLS for an overview of how the protocol for client certificate authentication actually works (also explains why we need the client's private key here).
The client's truststore is a straight forward JKS format file containing the root or intermediate CA certificates. These CA certificates will determine which endpoints you will be allowed to communicate with, in this case it will allow your client to connect to whichever server presents a certificate which was signed by one of the truststore's CA's.
To generate it you can use the standard Java keytool, for example;
keytool -genkey -dname "cn=CLIENT" -alias truststorekey -keyalg RSA -keystore ./client-truststore.jks -keypass whatever -storepass whatever
keytool -import -keystore ./client-truststore.jks -file myca.crt -alias myca
Using this truststore, your client will try to do a complete SSL handshake with all servers who present a certificate signed by the CA identified by myca.crt
.
The files above are strictly for the client only. When you want to set-up a server as well, the server needs its own key- and truststore files. A great walk-through for setting up a fully working example for both a Java client and server (using Tomcat) can be found on this website.
Issues/Remarks/Tips
- Client certificate authentication can only be enforced by the server.
- (Important!) When the server requests a client certificate (as part of the TLS handshake), it will also provide a list of trusted CA's as part of the certificate request. When the client certificate you wish to present for authentication is not signed by one of these CA's, it won't be presented at all (in my opinion, this is weird behaviour, but I'm sure there's a reason for it). This was the main cause of my issues, as the other party had not configured their server properly to accept my self-signed client certificate and we assumed that the problem was at my end for not properly providing the client certificate in the request.
- Get Wireshark. It has great SSL/HTTPS packet analysis and will be a tremendous help debugging and finding the problem. It's similar to
-Djavax.net.debug=ssl
but is more structured and (arguably) easier to interpret if you're uncomfortable with the Java SSL debug output.
It's perfectly possible to use the Apache httpclient library. If you want to use httpclient, just replace the destination URL with the HTTPS equivalent and add the following JVM arguments (which are the same for any other client, regardless of the library you want to use to send/receive data over HTTP/HTTPS):
-Djavax.net.debug=ssl
-Djavax.net.ssl.keyStoreType=pkcs12
-Djavax.net.ssl.keyStore=client.p12
-Djavax.net.ssl.keyStorePassword=whatever
-Djavax.net.ssl.trustStoreType=jks
-Djavax.net.ssl.trustStore=client-truststore.jks
-Djavax.net.ssl.trustStorePassword=whatever
You basically have four potential solutions to fix a "Not Trusted" exception on Android using httpclient:
- Trust all certificates. Don't do this, unless you really know what you're doing.
- Create a custom SSLSocketFactory that trusts only your certificate. This works as long as you know exactly which servers you're going to connect to, but as soon as you need to connect to a new server with a different SSL certificate, you'll need to update your app.
- Create a keystore file that contains Android's "master list" of certificates, then add your own. If any of those certs expire down the road, you are responsible for updating them in your app. I can't think of a reason to do this.
- Create a custom SSLSocketFactory that uses the built-in certificate KeyStore, but falls back on an alternate KeyStore for anything that fails to verify with the default.
This answer uses solution #4, which seems to me to be the most robust.
The solution is to use an SSLSocketFactory that can accept multiple KeyStores, allowing you to supply your own KeyStore with your own certificates. This allows you to load additional top-level certificates such as Thawte that might be missing on some Android devices. It also allows you to load your own self-signed certificates as well. It will use the built-in default device certificates first, and fall back on your additional certificates only as necessary.
First, you'll want to determine which cert you are missing in your KeyStore. Run the following command:
openssl s_client -connect www.yourserver.com:443
And you'll see output like the following:
Certificate chain
0 s:/O=www.yourserver.com/OU=Go to
https://www.thawte.com/repository/index.html/OU=Thawte SSL123
certificate/OU=Domain Validated/CN=www.yourserver.com
i:/C=US/O=Thawte, Inc./OU=Domain Validated SSL/CN=Thawte DV SSL CA
1 s:/C=US/O=Thawte, Inc./OU=Domain Validated SSL/CN=Thawte DV SSL CA
i:/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c)
2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA
As you can see, our root certificate is from Thawte. Go to your provider's website and find the corresponding certificate. For us, it was here, and you can see that the one we needed was the one Copyright 2006.
If you're using a self-signed certificate, you didn't need to do the previous step since you already have your signing certificate.
Then, create a keystore file containing the missing signing certificate. Crazybob has details how to do this on Android, but the idea is to do the following:
If you don't have it already, download the bouncy castle provider library from: http://www.bouncycastle.org/latest_releases.html. This will go on your classpath below.
Run a command to extract the certificate from the server and create a pem file. In this case, mycert.pem.
echo | openssl s_client -connect ${MY_SERVER}:443 2>&1 | \
sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > mycert.pem
Then run the following commands to create the keystore.
export CLASSPATH=/path/to/bouncycastle/bcprov-jdk15on-155.jar
CERTSTORE=res/raw/mystore.bks
if [ -a $CERTSTORE ]; then
rm $CERTSTORE || exit 1
fi
keytool \
-import \
-v \
-trustcacerts \
-alias 0 \
-file <(openssl x509 -in mycert.pem) \
-keystore $CERTSTORE \
-storetype BKS \
-provider org.bouncycastle.jce.provider.BouncyCastleProvider \
-providerpath /path/to/bouncycastle/bcprov-jdk15on-155.jar \
-storepass some-password
You'll notice that the above script places the result in res/raw/mystore.bks
. Now you have a file that you'll load into your Android app that provides the missing certificate(s).
To do this, register your SSLSocketFactory for the SSL scheme:
final SchemeRegistry schemeRegistry = new SchemeRegistry();
schemeRegistry.register(new Scheme("http", PlainSocketFactory.getSocketFactory(), 80));
schemeRegistry.register(new Scheme("https", createAdditionalCertsSSLSocketFactory(), 443));
// and then however you create your connection manager, I use ThreadSafeClientConnManager
final HttpParams params = new BasicHttpParams();
...
final ThreadSafeClientConnManager cm = new ThreadSafeClientConnManager(params,schemeRegistry);
To create your SSLSocketFactory:
protected org.apache.http.conn.ssl.SSLSocketFactory createAdditionalCertsSSLSocketFactory() {
try {
final KeyStore ks = KeyStore.getInstance("BKS");
// the bks file we generated above
final InputStream in = context.getResources().openRawResource( R.raw.mystore);
try {
// don't forget to put the password used above in strings.xml/mystore_password
ks.load(in, context.getString( R.string.mystore_password ).toCharArray());
} finally {
in.close();
}
return new AdditionalKeyStoresSSLSocketFactory(ks);
} catch( Exception e ) {
throw new RuntimeException(e);
}
}
And finally, the AdditionalKeyStoresSSLSocketFactory code, which accepts your new KeyStore and checks if the built-in KeyStore fails to validate an SSL certificate:
/**
* Allows you to trust certificates from additional KeyStores in addition to
* the default KeyStore
*/
public class AdditionalKeyStoresSSLSocketFactory extends SSLSocketFactory {
protected SSLContext sslContext = SSLContext.getInstance("TLS");
public AdditionalKeyStoresSSLSocketFactory(KeyStore keyStore) throws NoSuchAlgorithmException, KeyManagementException, KeyStoreException, UnrecoverableKeyException {
super(null, null, null, null, null, null);
sslContext.init(null, new TrustManager[]{new AdditionalKeyStoresTrustManager(keyStore)}, null);
}
@Override
public Socket createSocket(Socket socket, String host, int port, boolean autoClose) throws IOException {
return sslContext.getSocketFactory().createSocket(socket, host, port, autoClose);
}
@Override
public Socket createSocket() throws IOException {
return sslContext.getSocketFactory().createSocket();
}
/**
* Based on http://download.oracle.com/javase/1.5.0/docs/guide/security/jsse/JSSERefGuide.html#X509TrustManager
*/
public static class AdditionalKeyStoresTrustManager implements X509TrustManager {
protected ArrayList<X509TrustManager> x509TrustManagers = new ArrayList<X509TrustManager>();
protected AdditionalKeyStoresTrustManager(KeyStore... additionalkeyStores) {
final ArrayList<TrustManagerFactory> factories = new ArrayList<TrustManagerFactory>();
try {
// The default Trustmanager with default keystore
final TrustManagerFactory original = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
original.init((KeyStore) null);
factories.add(original);
for( KeyStore keyStore : additionalkeyStores ) {
final TrustManagerFactory additionalCerts = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
additionalCerts.init(keyStore);
factories.add(additionalCerts);
}
} catch (Exception e) {
throw new RuntimeException(e);
}
/*
* Iterate over the returned trustmanagers, and hold on
* to any that are X509TrustManagers
*/
for (TrustManagerFactory tmf : factories)
for( TrustManager tm : tmf.getTrustManagers() )
if (tm instanceof X509TrustManager)
x509TrustManagers.add( (X509TrustManager)tm );
if( x509TrustManagers.size()==0 )
throw new RuntimeException("Couldn't find any X509TrustManagers");
}
/*
* Delegate to the default trust manager.
*/
public void checkClientTrusted(X509Certificate[] chain, String authType) throws CertificateException {
final X509TrustManager defaultX509TrustManager = x509TrustManagers.get(0);
defaultX509TrustManager.checkClientTrusted(chain, authType);
}
/*
* Loop over the trustmanagers until we find one that accepts our server
*/
public void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException {
for( X509TrustManager tm : x509TrustManagers ) {
try {
tm.checkServerTrusted(chain,authType);
return;
} catch( CertificateException e ) {
// ignore
}
}
throw new CertificateException();
}
public X509Certificate[] getAcceptedIssuers() {
final ArrayList<X509Certificate> list = new ArrayList<X509Certificate>();
for( X509TrustManager tm : x509TrustManagers )
list.addAll(Arrays.asList(tm.getAcceptedIssuers()));
return list.toArray(new X509Certificate[list.size()]);
}
}
}
Best Solution
The issue was not having the actual root certificate in the trusted certs path.
This was easily solved by exporting certificates using OpenSSL:
and copying all the certificates from there.