Using WebScarab with existing SSL certificate

I needed to monitor/debug some HTTP and HTTPS traffic to a site from a closed source software. I had access to the real SSL certificates and via some quick DNS rewrites I was able to fool the application that the site's IP is my local address. At this point all I needed was some kind of reverse proxy that allows me to log requests as they are forwarded to the real site. I can't use Wireshark because it doesn't decode HTTPS traffic. I could have used nginx (or other similar) but I wanted something more adapted to this task. While googling I run into fiddler (only for Windows), charlesproxy (commercial), paros and its succesor ZAP (good, but listens/forwards only to one port) and others. The one that seemed to offer most features was WebScarab.

WebScarap is a Java app (selfcontained jar) that allows you to monitor HTTPS requests via it's build-in proxy. In order to handle HTTPS traffic, in the proxy listener settings you need to enable a proxy listener that forwards traffic to a HTTPS server. Something like: address=127.0.0.1, port=443, and baseurl=https://example.com:443/. 

By default, Webscarab will generate a self-signed SSL certificate for your setup. This means that browsers will complain about it and the application I was testing would refuse to use it. One can create a certs directory inside the webscarab application and put existing certficates inside that as described here. There are some things to be aware of:

1. If you're proxying requests on privileged ports (e.g. 80 or 443), you might need to start WebScarab as root (but you already knew that, right?)
2. Most webserver certificates come in PEM format (.crt extension) with a key file. WebScarab nees a PKCS12 format (.p12 extension) file. You can convert them with openssl:

   openssl pkcs12 -export -in example.com.crt -inkey example.com.key -certfile CertificationAuthority.crt -out example.com.p12
   

3. When openssl asks you for a password for the .p12 file, you must enter the word password. This is hardcoded into WebScarab.
4. The name of the .p12 file must match exactly the server name to which you are forwarding requests (as setup in the proxy listeners settings) and not the name of the server for which you are receiving requests (the two might be different). If you're forwarding to www1.example.com, then the name of the file must be www1.example.com.p12. I learned the hard way (i.e. reading webscarab code) that because I'm forwarding the requests to an IP address, the certificate file needs to be named as for that address. For example, if baseurl is https://192.168.1.1:443/ then the certificate file must be named 192.168.1.1.p12.
5. Having a custom certificate inside the certs directory (as described in item 4) will NOT work (keep reading for a few solutions). :(

Due to item 5, I prepared everything and WebScarab was still not working. Then I noticed in the terminal window that it was throwing an exception, that the X509 key manager can not be located. After a few hours of googleing, I found this post that seems was never treated as a bug report. I edited that line of code and tried to recompile webscarab as indicated here (and other places, too).

--- a/src/org/owasp/webscarab/plugin/proxy/Proxy.java
+++ b/src/org/owasp/webscarab/plugin/proxy/Proxy.java
@@ -476,7 +476,7 @@ public class Proxy implements Plugin {
                        SSLContext sslcontext = null;
                        KeyStore ks = KeyStore.getInstance("PKCS12");
                        ks.load(is, _keystorepass);
-                       kmf = KeyManagerFactory.getInstance("X509");
+                       kmf = KeyManagerFactory.getInstance("SunX509");
                        kmf.init(ks, _keypassword);
                        sslcontext = SSLContext.getInstance("SSLv3");
                        sslcontext.init(kmf.getKeyManagers(), null, null);

Running ant build (or simply ant) to recompile webscarab failed due to many missing classes. I found out the some of the jar files that it needed to compile are not included in the git repo (maybe due to licensing terms or something). In particular, I needed to add the following jars into the lib directory (you can find them in the official webscarab jar or by googleing): bcprov-jdk15-1.45.jar, joda-time-1.6.2.jar, openid4java-nodeps-0.9.6.jar, xalan-2.7.1.jar.

After I got the jars, running ant still not finished successfully. The command that worked was: ant -lib lib build. That created a webscarab.jar file. Trying to run that jar directly (using following command) failed:

java -cp .:lib/bcprov-jdk15-1.45.jar:lib/commons-logging-1.0.4.jar:lib/flex-messaging-remoting.jar:lib/jfreechart-1.0.13.jar:lib/openid4java-nodeps-0.9.6.jar:lib/bsf-2.3.0.jar:lib/concurrent.jar:lib/htmlparser.jar:lib/jhall-2.0_02.jar:lib/tagsoup-1.0rc2.jar:lib/bsh-2.0b1.jar:lib/flex-messaging-common.jar:lib/jcifs-1.3.14.jar:lib/joda-time-1.6.2.jar:lib/xalan-2.7.1.jar:lib/chardet.jar:lib/flex-messaging-core.jar:lib/jcommon-1.0.16.jar:lib/openamf.jar:lib/xmlsec-1.4.3.jar -jar webscarab.jar  

I don't know why it failed, maybe I'm missing something. However, I opened the build directory and tried to run this:

java -cp .:../lib/bcprov-jdk15-1.45.jar:../lib/commons-logging-1.0.4.jar:../lib/flex-messaging-remoting.jar:../lib/jfreechart-1.0.13.jar:../lib/openid4java-nodeps-0.9.6.jar:../lib/bsf-2.3.0.jar:../lib/concurrent.jar:../lib/htmlparser.jar:../lib/jhall-2.0_02.jar:../lib/tagsoup-1.0rc2.jar:../lib/bsh-2.0b1.jar:../lib/flex-messaging-common.jar:../lib/jcifs-1.3.14.jar:../lib/joda-time-1.6.2.jar:../lib/xalan-2.7.1.jar:../lib/chardet.jar:../lib/flex-messaging-core.jar:../lib/jcommon-1.0.16.jar:../lib/openamf.jar:../lib/xmlsec-1.4.3.jar org.owasp.webscarab.Main

This command finally worked (ran from inside the build directory).

Update: After posting this, I found this discussion which points out that -jar and -classpath arguments are mutually exclusive. So I rewrote the first command to:

java -cp .:lib/bcprov-jdk15-1.45.jar:lib/commons-logging-1.0.4.jar:lib/flex-messaging-remoting.jar:lib/jfreechart-1.0.13.jar:lib/openid4java-nodeps-0.9.6.jar:lib/bsf-2.3.0.jar:lib/concurrent.jar:lib/htmlparser.jar:lib/jhall-2.0_02.jar:lib/tagsoup-1.0rc2.jar:lib/bsh-2.0b1.jar:lib/flex-messaging-common.jar:lib/jcifs-1.3.14.jar:lib/joda-time-1.6.2.jar:lib/xalan-2.7.1.jar:lib/chardet.jar:lib/flex-messaging-core.jar:lib/jcommon-1.0.16.jar:lib/openamf.jar:lib/xmlsec-1.4.3.jar:webscarab.jar org.owasp.webscarab.Main

That means that I'm including the webscarab.jar in the classpath and explicitly naming the main class in command line. It worked like a charm!

Update: After more poking around, I found out that I can avoid typing the long classpath by editing the manifest file inside the webscarab.jar (in META-INF/manifest.mf) and replace the current Class-Path entry with:

Class-Path: lib/bcprov-jdk15-1.45.jar lib/commons-logging-1.0.4.jar lib/flex-messaging-remoting.jar lib/jfreechart-1.0.13.jar lib/openid4java-nodeps-0.9.6.jar lib/bsf-2.3.0.jar lib/concurrent.jar lib/htmlparser.jar lib/jhall-2.0_02.jar lib/tagsoup-1.0rc2.jar lib/bsh-2.0b1.jar lib/flex-messaging-common.jar lib/jcifs-1.3.14.jar lib/joda-time-1.6.2.jar lib/xalan-2.7.1.jar lib/chardet.jar lib/flex-messaging-core.jar lib/jcommon-1.0.16.jar lib/openamf.jar lib/xmlsec-1.4.3.jar

After doing so, running WebScarab is (again) as easy as running:

java -jar webscarab.jar