Like a lot of the web development community, Vox Product has been increasing its focus on the front end performance of our sites. In order to improve, we must first quantify what "front end performance" means to us, and have a repeatable way to measure it so that we can tell if we are really improving or not.

While attending Velocity NYC last October, WebPageTest.org was repeatedly brought up as a the best way to start measuring front end performance. Their Speed Index metric was exactly what we were looking for - a number to represent how fast a user feels like a page is loading.

Jonathan Klein gave a talk titled DIY Synthetic - Private WebPageTest Magic, which convinced me that a private Web Page Test instance was what we needed. Our needs are probably a little different than most, because we wanted to run tests against the home and article pages for all of our networks (at the time, Polygon, SB Nation and Verge) for both desktop and mobile sizes. Testing all these variations with multiple runs to ensure accuracy would easily put us over the public API limits, so we started looking into setting up our own instance.

Setting up our instance on EC2 involved a bit more trial and error (particularly around setting up the test agents), but it's been a great investment for us. Here is how we did it.

Walkthrough

Before getting started, you should be acquainted with the primary WPT Private Instance documentation, since we'll be using terms introduced there.

Begin by booting up a t1.micro instance running Ubuntu 14.04. This will be the the web server that coordinates running tests, and actually hosts our WPT pages, so make sure ports for SSH (22) and HTTP (80) are open in the security group. Upon booting, make note of the public DNS (hereafter, public_dns) for your server - we'll use it later.

SSH in using your EC2 private key file and install the prerequisite packages mentioned in the docs:

Time for a little config file editing. This server is just going to be dedicated to serving WPT, so let's edit the default DocumentRoot in Apache's config in /etc/apache2/sites-enabled/000-default.conf to point to /var/www. After that, open up /etc/php5/apache2/php.ini and set upload_max_filesize to "15M" and post_max_size to "15M" as well. Finally, run sudo service apache2 restart to make these config changes take effect.

Now we are ready to actually install WPT.  Check the releases page to get the tar.gz download url for the latest version.

After all this is done, you should be able to navigate your browser over to http://public_dns and you should see your own WPT instance! http://public_dns/install should hopefully confirm that all systems are a go.

Depending on your needs, it's probably a good idea to setup a cron task to clear out old log data so your disk doesn't fill up. We persist our data elsewhere, so our disk space is small and have this in sudo's crontab:

The last thing we need to do on this server is setup locations.ini, which can be found in /var/www/settings/locations.ini. I was never able to fully grok the labeling of browser configurations for test agents and the documentation is a bit confusing, but this is what we ended up with:

Make note of the 'location' setting you use - in this case, ours is "EC2-WPT". This variable will be used later to correctly register and identify your test agents with your server.

At this point, everything on the server should be setup, and we can boot up a test agent. We use a m1.medium instance running the IE9/Chrome/Firefox/Safari AMI from the documentation. When booting, you'll need to set two pieces of user data so the node knows where the WPT master is. This is what our user data looks like:

After this instance boots up (wait till EC2 shows the image as totally initialized), you should see the test agent register at http://public_dns/getLocations.php.

One last step: it's a good idea to update the agents using the directions in the documentation to make sure they are current. With that done, your own WPT server is ready to go.

We have another application running that constantly queues up tests, looks at the results and pushes the metrics we care about to StatsD. By watching these graphs and paying attention to our waterfall charts, we have a much better understanding of what is causing slowness on our pages, and gives us insight into how to improve.