Using WebScarab with existing SSL certificate

I needed to monitor/debug some HTTP and HTTPS traffic to a site from a closed source software. I had access to the real SSL certificates and via some quick DNS rewrites I was able to fool the application that the site's IP is my local address. At this point all I needed was some kind of reverse proxy that allows me to log requests as they are forwarded to the real site. I can't use Wireshark because it doesn't decode HTTPS traffic. I could have used nginx (or other similar) but I wanted something more adapted to this task. While googling I run into fiddler (only for Windows), charlesproxy (commercial), paros and its succesor ZAP (good, but listens/forwards only to one port) and others. The one that seemed to offer most features was WebScarab.

WebScarap is a Java app (selfcontained jar) that allows you to monitor HTTPS requests via it's build-in proxy. In order to handle HTTPS traffic, in the proxy listener settings you need to enable a proxy listener that forwards traffic to a HTTPS server. Something like: address=127.0.0.1, port=443, and baseurl=https://example.com:443/. 

By default, Webscarab will generate a self-signed SSL certificate for your setup. This means that browsers will complain about it and the application I was testing would refuse to use it. One can create a certs directory inside the webscarab application and put existing certficates inside that as described here. There are some things to be aware of:

1. If you're proxying requests on privileged ports (e.g. 80 or 443), you might need to start WebScarab as root (but you already knew that, right?)
2. Most webserver certificates come in PEM format (.crt extension) with a key file. WebScarab nees a PKCS12 format (.p12 extension) file. You can convert them with openssl:

   openssl pkcs12 -export -in example.com.crt -inkey example.com.key -certfile CertificationAuthority.crt -out example.com.p12
   

3. When openssl asks you for a password for the .p12 file, you must enter the word password. This is hardcoded into WebScarab.
4. The name of the .p12 file must match exactly the server name to which you are forwarding requests (as setup in the proxy listeners settings) and not the name of the server for which you are receiving requests (the two might be different). If you're forwarding to www1.example.com, then the name of the file must be www1.example.com.p12. I learned the hard way (i.e. reading webscarab code) that because I'm forwarding the requests to an IP address, the certificate file needs to be named as for that address. For example, if baseurl is https://192.168.1.1:443/ then the certificate file must be named 192.168.1.1.p12.
5. Having a custom certificate inside the certs directory (as described in item 4) will NOT work (keep reading for a few solutions). :(

Due to item 5, I prepared everything and WebScarab was still not working. Then I noticed in the terminal window that it was throwing an exception, that the X509 key manager can not be located. After a few hours of googleing, I found this post that seems was never treated as a bug report. I edited that line of code and tried to recompile webscarab as indicated here (and other places, too).

--- a/src/org/owasp/webscarab/plugin/proxy/Proxy.java
+++ b/src/org/owasp/webscarab/plugin/proxy/Proxy.java
@@ -476,7 +476,7 @@ public class Proxy implements Plugin {
                        SSLContext sslcontext = null;
                        KeyStore ks = KeyStore.getInstance("PKCS12");
                        ks.load(is, _keystorepass);
-                       kmf = KeyManagerFactory.getInstance("X509");
+                       kmf = KeyManagerFactory.getInstance("SunX509");
                        kmf.init(ks, _keypassword);
                        sslcontext = SSLContext.getInstance("SSLv3");
                        sslcontext.init(kmf.getKeyManagers(), null, null);

Running ant build (or simply ant) to recompile webscarab failed due to many missing classes. I found out the some of the jar files that it needed to compile are not included in the git repo (maybe due to licensing terms or something). In particular, I needed to add the following jars into the lib directory (you can find them in the official webscarab jar or by googleing): bcprov-jdk15-1.45.jar, joda-time-1.6.2.jar, openid4java-nodeps-0.9.6.jar, xalan-2.7.1.jar.

After I got the jars, running ant still not finished successfully. The command that worked was: ant -lib lib build. That created a webscarab.jar file. Trying to run that jar directly (using following command) failed:

java -cp .:lib/bcprov-jdk15-1.45.jar:lib/commons-logging-1.0.4.jar:lib/flex-messaging-remoting.jar:lib/jfreechart-1.0.13.jar:lib/openid4java-nodeps-0.9.6.jar:lib/bsf-2.3.0.jar:lib/concurrent.jar:lib/htmlparser.jar:lib/jhall-2.0_02.jar:lib/tagsoup-1.0rc2.jar:lib/bsh-2.0b1.jar:lib/flex-messaging-common.jar:lib/jcifs-1.3.14.jar:lib/joda-time-1.6.2.jar:lib/xalan-2.7.1.jar:lib/chardet.jar:lib/flex-messaging-core.jar:lib/jcommon-1.0.16.jar:lib/openamf.jar:lib/xmlsec-1.4.3.jar -jar webscarab.jar  

I don't know why it failed, maybe I'm missing something. However, I opened the build directory and tried to run this:

java -cp .:../lib/bcprov-jdk15-1.45.jar:../lib/commons-logging-1.0.4.jar:../lib/flex-messaging-remoting.jar:../lib/jfreechart-1.0.13.jar:../lib/openid4java-nodeps-0.9.6.jar:../lib/bsf-2.3.0.jar:../lib/concurrent.jar:../lib/htmlparser.jar:../lib/jhall-2.0_02.jar:../lib/tagsoup-1.0rc2.jar:../lib/bsh-2.0b1.jar:../lib/flex-messaging-common.jar:../lib/jcifs-1.3.14.jar:../lib/joda-time-1.6.2.jar:../lib/xalan-2.7.1.jar:../lib/chardet.jar:../lib/flex-messaging-core.jar:../lib/jcommon-1.0.16.jar:../lib/openamf.jar:../lib/xmlsec-1.4.3.jar org.owasp.webscarab.Main

This command finally worked (ran from inside the build directory).

Update: After posting this, I found this discussion which points out that -jar and -classpath arguments are mutually exclusive. So I rewrote the first command to:

java -cp .:lib/bcprov-jdk15-1.45.jar:lib/commons-logging-1.0.4.jar:lib/flex-messaging-remoting.jar:lib/jfreechart-1.0.13.jar:lib/openid4java-nodeps-0.9.6.jar:lib/bsf-2.3.0.jar:lib/concurrent.jar:lib/htmlparser.jar:lib/jhall-2.0_02.jar:lib/tagsoup-1.0rc2.jar:lib/bsh-2.0b1.jar:lib/flex-messaging-common.jar:lib/jcifs-1.3.14.jar:lib/joda-time-1.6.2.jar:lib/xalan-2.7.1.jar:lib/chardet.jar:lib/flex-messaging-core.jar:lib/jcommon-1.0.16.jar:lib/openamf.jar:lib/xmlsec-1.4.3.jar:webscarab.jar org.owasp.webscarab.Main

That means that I'm including the webscarab.jar in the classpath and explicitly naming the main class in command line. It worked like a charm!

Update: After more poking around, I found out that I can avoid typing the long classpath by editing the manifest file inside the webscarab.jar (in META-INF/manifest.mf) and replace the current Class-Path entry with:

Class-Path: lib/bcprov-jdk15-1.45.jar lib/commons-logging-1.0.4.jar lib/flex-messaging-remoting.jar lib/jfreechart-1.0.13.jar lib/openid4java-nodeps-0.9.6.jar lib/bsf-2.3.0.jar lib/concurrent.jar lib/htmlparser.jar lib/jhall-2.0_02.jar lib/tagsoup-1.0rc2.jar lib/bsh-2.0b1.jar lib/flex-messaging-common.jar lib/jcifs-1.3.14.jar lib/joda-time-1.6.2.jar lib/xalan-2.7.1.jar lib/chardet.jar lib/flex-messaging-core.jar lib/jcommon-1.0.16.jar lib/openamf.jar lib/xmlsec-1.4.3.jar

After doing so, running WebScarab is (again) as easy as running:

java -jar webscarab.jar

Solr replication monitoring

I am using Solr replication on some projects for load balancing (not that Solr really needs that ;) ) and high-availability. Recently, due to a network problem, replication was stalled on one of the slave without me noticing it, so I wrote a small script to check the index version difference between master and slave and which admins can integrate with current monitoring system (nagios, zenoss, munin, whatever).

#!/bin/bash
if [ $# -lt 2 ]; then
 echo Usage: `basename $0` SolrMasterIP[:port] SolrSlaveIP[:port] [AlertThreshold]
 exit 1
fi
# read command params
MASTER=$1
SLAVE=$2
THRESHOLD=${3:-1}

# read index version from master and slave
MASTER_INDEX=`wget -qO - http://$1/solr/admin/replication/index.jsp | awk '/Index Version/ {print substr($3,0,length($3)-1)}'` 
[ -z "$MASTER_INDEX" ] && echo Error reading Master index && exit 2
SLAVE_INDEX=`wget -qO - http://$2/solr/admin/replication/index.jsp | awk '/Index Version/ {print substr($3,0,length($3)-1)}'`
[ -z "$SLAVE_INDEX" ] && echo Error reading Master index && exit 3

# check if master-slave index difference is within acceptable threshold
let "$MASTER_INDEX - $SLAVE_INDEX < $THRESHOLD" && echo Slave index version \($SLAVE_INDEX\) is behind master \($MASTER_INDEX\) && exit 4
echo Slave index version \($SLAVE_INDEX\) is in sync with master \($MASTER_INDEX\)

Speed up Ubuntu Quantal Quetzal

I recently upgraded my Ubuntu 12.04 Precise to Ubuntu 12.10. Overall, I didn't see much difference, probably because I didn't run/notice the bugs/limitations that were fixed in this new release. One thing that I (subjectively) noticed is that boot up and application startup is slower, with the UI experiencing occasional freezes. I guess these slowness/freezes were already in 12.04, but they become more obvious after the upgrade. To the point that they become frustrating. So I looked into some ways that might speed up the system (not minor or complete tweakings, but just a few to get over the current situation).

Mount options

First thing I tried was adding nodiratime and noatime options to the mount options of / and /home partitions. To do that, just edit /etc/fstab and add these options alongside the existing ones. My fstab looks something like this now:

/dev/sda1 / ext4 nodiratime,noatime,errors=remount-ro 0 1
UUID=..... /home ext4 defaults,nodiratime,noatime 0 2

I didn't notice any major speedup after these changes, but they do seem common sense and lack of speedup might be due to my particular setup (some VMs I'm using).

Install ureadahead

Next I installed the ureadahead package which is supposed to lower boot up times. Install was easy:

sudo apt-get install ureadahead

According to what I read, on first run, ureadahead logs bootup files accesses and then aggregates these files into cached file that will be used in next bootups to speed things up. It also optimizes this cached file according to the type of disk that you have (SSD or HDD). After the system boots, ureadahead process seems to terminate and just gets out of the way.
I didn't notice any major speedup on this either. It could be some, but bootup time is still several times more of the Ubuntu advertised "10 seconds bootup".

Install preload

Preload is a daemon that logs statistics about programs usage and requirements and use these statistics to speed up programs startup. I guess this is what Windows has on by default in that prefetched folder. As mentioned in the man pages, the speed up is not immediately, because preload first needs to collect the stats and create some caches before it can use them. Even so, I did see programs starting faster pretty soon after installing preload. I just installed using:

sudo apt-get install preload

Install prelink

Preling is a tool that improves memory usage and load times by optimizing programs and libraries. Install it using:

sudo apt-get install prelink

After install, you need to start the optimization process. Before I did that, first I edited /etc/prelink.conf to include /opt directory in the optimization process because I have some custome software there. After that, just run

sudo prelink -avmR

The option v is for verbose so that you can see what's processing. You could remove that if you don't care. I guess I'll need to run such a command when I update/install new software, but that's not a big issue.
Although I installed it mostly for the "memory optimizations" promise, Prelink also seemed to help with speed. I'm not sure how much since I installed it after preload which already helped in that area.