Adventures With Redis PubSub, SSE and Docker At Vox Product

If you were to watch and read every story and every video published through Chorus, on any given day, you would be reading or watching something like 1,700 different pieces of content. That is a lot of content! And that doesn't even include the stories our audience publishes in our Forums and Fan Post sections.

To help myself and others keep up with all these great stories, I built an internal tool called "Soda Stream" that leverages Redis, Server Sent Events, and Docker to present a realtime view of content as it flows through Chorus.

Here's the explainer:

What is Soda Stream and why did you call it that?

Soda Stream is an app that seeks to present a digestible, scannable, enjoyable way to stay in touch with the approximately 10 zillion stories our editorial teams and users publish through Chorus.

It is called Soda Stream because I am addicted to seltzer. Also, this app is hooked up to Chorus in such a way that stories stream to it in realtime: as soon as a story gets published, it flows into Soda Stream!

This looks just like Tweetdeck! What's the scoop?

Tweetdeck is amazing. The column/lane interface is fantastic for organizing and keeping up with many streams of content and notifications. I love it. Soda Stream was obviously inspired by this interface. If any Twitter/Tweetdeck folks are reading this: your app is great! Thank you for creating it!

Stories flow into Soda Stream in realtime. How does that work?

Chorus uses Redis pub/sub and webhooks to broadcast all sorts of events. Soda Stream listens to these Redis pub/sub channels and when a "publish" event happens Soda Stream broadcasts that event to all connected browser clients using a Server Sent Events / EventSource channel.

So how does Docker fit into all of this?

Soda Stream was a lot of fun to build because it gave me an opportunity to explore using Redis pub/sub in conjunction with SSE/EventSource, but it was also a great opportunity to explore building and deploying an application using Docker.

We had some experience at Vox Product using Docker to facilitate local development, but we had not yet made the leap to running any general use applications and services in containers.

With Soda Stream, I decided to learn about building Docker images, managing persistent data and shared services, and deploying container-based applications without disruption.

Docker, you say? Did you use --insert container orchestration tool-- ?

Rather than use one of the approximately 10,000 container orchestration services that are available, I decided to try to build out some tooling myself. The goal was to dig into Docker and really screw things up so I could learn first-hand about the pain that Docker orchestration tools seek to ease.

The whole effort was a blast. The toolchain / workflow I ended up with involves a Capistrano plugin, a pipeline of Jenkins jobs that builds and deploys containers, and a custom-built application called Sauron that plugs into Docker hosts, discovers new containers and uses HaProxy to balance traffic.

The big takeaway for me from all of this was that, in addition to helping make development environments easier to manage, Docker and containers really help make research and development efforts and prototyping easier and fun. More time is spent imagining and creating, and less time is spent configuring and provisioning.

With this simple orchestration workflow in place, we have been able to rapidly develop and deploy all sorts of apps and services using Docker, Redis pub/sub, and SSE. Here are a couple of fun ones:

Where is this all headed?

Our next big Vox Product Hack Week (which we call Vax) is coming up and I think that Docker-based workflows are going to help our teams build some interesting, useful, and fun things. Stay tuned!