Blog Posts - July 2011

Leader Election with Zookeeper


Recently we had to implement an active-passive redundancy of a singleton service in our production environment where the general rule is always have “more than one of anything”. The main motivation is to alleviate the need to manually monitor and manage these services, whose presence is crucial to the overall health of the site.

This means that we sometime have a service installed on several machines for redundancy, but only one of the is active at any given moment. If the active services goes down for some reason, another service rises to do its work. This is actually called leader election. One of the most prominent open source implementation facilitating the process of leader election is Zookeeper. So what is Zookeeper?

Originally developed by Yahoo reasearch, Zookeepr acts as a service providing reliable distributed coordination. It is highly concurrent, very fast and suitable mainly for read-heavy access patterns. Reads can be done against any node of a Zookeeper cluster while writes a quorum-based. To reach a quorum, Zookeeper utilizes an atomic broadcast protocol. So how does it work?

Read more >

Feature Flags Made Easy

I recently participated in the ILTechTalk week. Most of the talks discussed issues like Scalability, Software Quality, Company Culture, and Continuous Deployment (CD). Since the talks were hosted at Outbrain, we got many direct questions about our concrete implementations. Some of the questions and statements claimed that Feature Flags complicate your code. What bothered most participants was that committing code directly to trunk requires addition of feature flags in some cases and that it may make their code base more complex.

While in some cases, feature flags may make the code slightly more complicated, it shouldn’t be so in most cases. The main idea I’m presenting here is that conditional logic can be easily replaced with polymorphic code. In fact, conditional logic can always be replaced by polymorphism.

Enough with the abstract talk…

Suppose we have an application that contains some imaginary feature, and we want to introduce a feature flag. Below is a code snippet that developers normally come up with:

public void runApplication() { // ... if (useNewImplementation) { executeNewImaginaryFeatureImplementation(); } else { executeOldImaginaryFeatureImplementation(); } // ... }

While this is a legitimate implementations in some cases, it does complicate your code base by increasing the cyclomatic complexity of your code. In some cases, the test for activation of the feature may recur in many place in the code, so this approach can quickly turn into a maintenance nightmare.

Luckily, implementing a feature flag using polymorphism is pretty easy. First, let’s define an interface for the imaginary feature and two implementations (old and new):

public interface ImaginaryFeature { public void executeFeature(); } class OldImaginaryFeature implements ImaginaryFeature { @Override public void executeFeature() { System.out.println("old feature implementation"); } } class NewImaginaryFeature implements ImaginaryFeature { @Override public void executeFeature() { System.out.println("new feature implementation"); } }

Now, let’s use the feature in our application, selecting the implementation at runtime:

public class PolymorphicApplication { private final ImaginaryFeature imaginaryFeature; public PolymorphicApplication() { this.imaginaryFeature = createImaginaryFeature(); } private ImaginaryFeature createImaginaryFeature() { final String featureClass = System.getProperty("PolymorphicApplication.imaginaryFeature.class"); try { return (ImaginaryFeature) Class.forName(featureClass).newInstance(); } catch (final Exception e) { throw new IllegalStateException("Failed to create ImaginaryFeature of class " + featureClass, e); } } public void runApplication() { // ... imaginaryFeature.executeFeature(); // ... } }

Here, we initialized the imaginary feature member by reflection, using a class name specified as a system property. The createImaginaryFeature() method above is usually abstracted into a factory but kept as is here for brevity. But we’re still not done. Most of the readers would probably say that the introduction of a factory and reflection makes the code less readable and less maintainable. I have to agree — and apart from that, adding dependencies to the concrete implementations will complicate the code even more. Luckily, I have a secret weapon at my disposal. It is called IoC, (or DI). When using an IoC container such as Spring or Guice, your code can be made extremely flexible, and implementing feature flags becomes a walk in the park.

Below is a rewrite of the PolymorphicApplication using Spring dependency injection:

public class SpringPolymorphicApplication { private final ImaginaryFeature imaginaryFeature; public SpringPolymorphicApplication(final ImaginaryFeature imaginaryFeature) { this.imaginaryFeature = imaginaryFeature; } public void runApplication() { // ... imaginaryFeature.executeFeature(); // ... } }

The spring code above defines an application and 2 imaginary feature implementations. By default, the application is initialized with the oldImaginaryFeature, but this behavior can be overridden by specifying a -DimaginaryFeature.implementation.bean=newImaginaryFeature command line argument. Only a single feature implementation will be initialized by Spring, and the implementations may have dependencies.

Bottom line is: with a bit of extra preparation and correct design decisions, feature flags shouldn’t be a burden on your code base. By extra preparation, I mean extracting interfaces for your domain objects, using an IoC container, etc, which is something we should be doing in most cases anyway.


Eran Harel is a Senior Software Developer at Outbrain.