The Annotated Timeline
This graph may seem intimidating at first, so don’t be scared and let’s dive right into it… BTW, you may click on the image to enlarge it.

In this graph the x axis shows the time (date and time of day) and the y axis shows the svn revision number. Each colored line represents a single module (so we have one line for www and one line for the BehavioralEngine etc).

What you would usually see is for each line (representing a module) a monotonically increasing value over time, a line from the bottom left corner towards the top right corner, however, in relatively rare cases where a developer wants to deploy an older version of his module, then you clearly see it by the line suddenly dropping down a bit instead of climbing up; this is really nice, helps find unusual events.

The Histogram
In the next graph you see an overview of deployments per day. (click to enlarge)

This is more of a holistic view of how things went the last couple of days, it just shows how many deployments took place each day (counts production clusters only) and colors the successful ones in green and the failed ones in red.

This graph is like an executive summary that can tell the story of – in case there are too many reds (or there are reds at all), then someone needs to take that seriously and figure out what needs to be fixed (usually that someone is me…) – or in case the bars aren’t high enough, then someone needs to kick developer’s buts and get them deploying somethin already…

Like many other graphs from Google’s library (this one’s a Stacked Column Chart, BTW), it shows nice tooltips when hovering over any of the columns with their x values (the date) and their y value (number of successful/failed deployments)

Versions DNA Mapping
The following graph shows the current variety of versions that we have in our production systems for each and every module. It was attributed as a DNA mapping by one of our developers b/c of the similarity in how they look but that’s how far this similarity goes…

The x axis lists the different modules that we have (names were intentionally left out, but you can imaging having www and other folks there). The y axis shows the svn versions of them in production. It uses glu’s live model as reported by glu’s agents to zookeeper.

Let’s zoom in a bit:

What this diagram tells us is that the module www has versions starting from 41268 up to 41463 in production. This is normal as we don’t necessarily deploy everything to all servers at once, but this graph helps us easily find hosts that are left behind for too long, so for example if one of the modules had not been deployed in a while then you’d see it falling behind low on the graph. Similarly, if a module has a large variability in versions in production, chances are that you want to close that gap pretty soon. The following graph illustrates both cases:

To implement this graph I used a crippled version of the Candle Stick Chart, which is normally used for showing stock values; it’s not ideal for this use case but it’s the closest I could find.

That’s all, three charts is enough for now and there are other news regarding our evolving deployment system, but they are not as visual; if you have any questions or suggestions for other types of graphs that could be useful don’t be shy to comment or tweet (@rantav).

Recently we had to implement an active-passive redundancy of a singleton service in our production environment where the general rule is always have “more than one of anything”. The main motivation is to alleviate the need to manually monitor and manage these services, whose presence is crucial to the overall health of the site.

This means that we sometime have a service installed on several machines for redundancy, but only one of the is active at any given moment. If the active services goes down for some reason, another service rises to do its work. This is actually called leader election. One of the most prominent open source implementation facilitating the process of leader election is Zookeeper. So what is Zookeeper?

Originally developed by Yahoo reasearch, Zookeepr acts as a service providing reliable distributed coordination. It is highly concurrent, very fast and suitable mainly for read-heavy access patterns. Reads can be done against any node of a Zookeeper cluster while writes a quorum-based. To reach a quorum, Zookeeper utilizes an atomic broadcast protocol. So how does it work?

Connectivity, State and Sessions

Zookeeper maintains an active connection with all its clients using a heartbeat mechanism. Furthermore, Zookeeper keeps a session for each active client that is connected to it. When a client is disconnected from Zookeeper for more than a specified timeout the session expires. This means that Zookeeper has a pretty good picture of all the animals in its zoo.

Data Model

The Zookeeper data model consists of a hierarchy of nodes, called ZNodes. ZNodes can hold a relatively small (efficiency is key here) amount of data, they are versioned and timestamped . There are several properties a ZNode can have that make them particularly useful for different use cases. Each node in Zookeeper can have the persistent, ephemeral and sequential flags. These determine the naming of the node and its behavior with respect to the client session.

• The persistent node is basically a managed data bin
• The ephemeral node exists for the lifetime of its client session
• The sequential node, when created, gets a unique number (sequence) suffixed to its name

The latter two provide the means to implementing a variety of distribution tasks such as locks, queues, barriers, transactions, elections and other synchronization related tasks.

Here’s what an election path looks like in the Solr Cloud Admin Console:

Events

Zookeeper allows its clients to watch for different events in its node hierarchy. This way clients can get notified of different changes in the distributed state of affairs and act accordingly. These watches are one timers and should be persisted again by the client after notification. The client is also responsible of handling session expiration which means that ephemeral nodes should be re-persisted after an expiration.

Client Implementation

Zookeeper requires a lot of boiler plate code, mostly around connectivity and for the majority of the time you will be doing the same things over and over. Luckily Stefan Groschupf and Patrick Hunt wrote a client abstraction called ZkClient. I published a maven artifact for this on OSS so it’s available to our build system. The library also provides a persistent event notification mechanism in the form of listeners.

The next thing to do was to cook up a Spring factory bean for ZkClient and a template style class to act as an abstraction layer to Zookeeper operations. This ties in nicely into the Spring container which we use extensively:

    <bean id="zkClient" class="org.projectx.zookeeper.ZkClientFactoryBean">
        <property name="ensemble" value="localhost:2181,localhost:2182,localhost:2183" />
        <property name="connectionTimeout" value="2000" />
        <property name="sessionTimeout" value="10000" />
        <property name="stateListeners">
            <list>
                <ref local="zkStatsCollector" />
            </list>
        </property>
    </bean>
    
    <bean id="zkStatsCollector" class="org.projectx.zookeeper.ZookeeperClientStatsCollector" />
 
    <bean id="zkTemplate" class="org.projectx.zookeeper.ZookeeperTemplate">
        <constructor-arg ref="zkClient" />
    </bean>

The ZooKeeperClientStatsCollector is a listener implementation which collects stats about session connects/disconnects, exported to JMX as an MBean.

Now that we have a working data access layer we can start with the good stuff.

Leader Election

The Zookeeper documentation describes in general terms how leader election is to be performed. The general idea is that all participants of the election process create an ephemeral-sequential node on the same election path. The node with the smallest sequence number is the leader. Each “follower” node listens to the node with the next lower sequence number to prevent a herding effect when the leader goes away. In effect this creates a linked list of nodes. When a node’s local leader dies it goes to election either find a smaller node or becoming the leader if it has the lowest sequence number.

The following image describes a scenario with 3 clients participating in the election process:

Each client participating in this process has to:

1. Create an ephemeral-sequential node to participate under the election path
2. Find its leader and follow (watch) it
3. Upon leader removal go to election and find a new leader, or become the leader if no leader is to be found
4. Upon session expiration check the election state and go to election if needed

One thing to consider here is the nature of the work being done by the leader. Make sure it’s state can be preserved if its leadership is revoked. Leader loss could be caused by any number of reasons including initiated restarts due to maintenance and releases. It could also be brought about by network partitioning.

Designing services for graceful recovery is a requirement for distributed systems not leader election.

Spring helps here because interception can be used to suppress method invocations of various services based on leadership status. Below is an example of an interception based leadership control:

<bean id="leaderElectionProxyTemplate" class="org.springframework.aop.framework.ProxyFactoryBean"
    abstract="true">
    <property name="interceptorNames">
        <list>
            <value>leaderElectionTarget</value>
        </list>
    </property>
</bean>
 
<bean id="leaderElectionTarget"  class="org.projectx.zookeeper.election.LeaderElectionTargetInterceptor" />
 
<bean id="myService" parent="leaderElectionProxyTemplate">
    <property name="target">
        <bean class="org.projetx.service.MyService">
    </property>
</bean>

The service myService is the one controlled by leader election, all it’s method are going to be suppressed or invoked based on leadership status.

Another implementation uses a quartz scheduler instance as its target target:

<bean id="leaderElectionTarget"
    class="org.projectx.zookeeper.election.quartz.SchedulerElectionTarget">
    <constructor-arg ref="myScheduler" />
</bean>

This implementation puts a quartz scheduler on standby mode when leadership is revoked and resumes it when it’s granted (notice it will not actually stop running tasks, this will be allowed their natural completion, so in effect you may have a scheduled task running on two services due to partitioning scenarios. This means that whatever is scheduled has to be aware of another service possibly doing the same work. This problem can be easily solved with a Zookeeper barrier implementation, more on that in another post.

But there’s more than leader election you could do with Zookeeper

If you wish to run your data center the democratic way, where important decisions are made in coordination with other stakeholders, Zookeeper certainly helps.

Leader Election with Spring is on GitHub, Source shown in this post can be found on Gist.

Zookeeper documentation and wiki.

Happy Zookeeping!

While in some cases, feature flags may make the code slightly more complicated, it shouldn’t be so in most cases. The main idea I’m presenting here is that conditional logic can be easily replaced with polymorphic code. In fact, conditional logic can always be replaced by polymorphism.

Enough with the abstract talk…

Suppose we have an application that contains some imaginary feature, and we want to introduce a feature flag. Below is a code snippet that developers normally come up with:

  public void runApplication() {
 
    // ...
    if (useNewImplementation) {
      executeNewImaginaryFeatureImplementation();
    } else {
      executeOldImaginaryFeatureImplementation();
    }
    // ...
  }

While this is a legitimate implementations in some cases, it does complicate your code base by increasing the cyclomatic complexity of your code. In some cases, the test for activation of the feature may recur in many place in the code, so this approach can quickly turn into a maintenance nightmare.

Luckily, implementing a feature flag using polymorphism is pretty easy. First, let’s define an interface for the imaginary feature and two implementations (old and new):

public interface ImaginaryFeature {
  public void executeFeature();
}
 
class OldImaginaryFeature implements ImaginaryFeature {
  @Override
  public void executeFeature() {
    System.out.println("old feature implementation");
  }
}
 
class NewImaginaryFeature implements ImaginaryFeature {
  @Override
  public void executeFeature() {
    System.out.println("new feature implementation");
  }
}

Now, let’s use the feature in our application, selecting the implementation at runtime:

public class PolymorphicApplication {
  
  private final ImaginaryFeature imaginaryFeature;
  
  public PolymorphicApplication() {
    this.imaginaryFeature = createImaginaryFeature();
  }
 
  private ImaginaryFeature createImaginaryFeature() {
    final String featureClass = System.getProperty("PolymorphicApplication.imaginaryFeature.class");
    try {
      return (ImaginaryFeature) Class.forName(featureClass).newInstance();
    } catch (final Exception e) {
      throw new IllegalStateException("Failed to create ImaginaryFeature of class " + featureClass, e);
    }
  }
 
  public void runApplication() {
 
    // ...
    imaginaryFeature.executeFeature();
    // ...
  }
}

Here, we initialized the imaginary feature member by reflection, using a class name specified as a system property. The createImaginaryFeature() method above is usually abstracted into a factory but kept as is here for brevity. But we’re still not done. Most of the readers would probably say that the introduction of a factory and reflection makes the code less readable and less maintainable. I have to agree — and apart from that, adding dependencies to the concrete implementations will complicate the code even more. Luckily, I have a secret weapon at my disposal. It is called IoC, (or DI). When using an IoC container such as Spring or Guice, your code can be made extremely flexible, and implementing feature flags becomes a walk in the park.

Below is a rewrite of the PolymorphicApplication using Spring dependency injection:

public class SpringPolymorphicApplication {
 
  private final ImaginaryFeature imaginaryFeature;
 
  public SpringPolymorphicApplication(final ImaginaryFeature imaginaryFeature) {
    this.imaginaryFeature = imaginaryFeature;
  }
 
  public void runApplication() {
 
    // ...
    imaginaryFeature.executeFeature();
    // ...
  }
}

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd">
 
  <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
    <property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE"/>
    <property name="properties">
      <map>
        <entry key="imaginaryFeature.implementation.bean" value="oldImaginaryFeature"/>
      </map>
    </property>  
  </bean>
 
  <bean id="application" class="com.eranharel.SpringPolymorphicApplication">
    <constructor-arg ref="imaginaryFeature"/>
  </bean>
  
  <alias name="${imaginaryFeature.implementation.bean}" alias="imaginaryFeature"/>
  
  <bean id="newImaginaryFeature" class="com.eranharel.NewImaginaryFeature" lazy-init="true"/>
  
  <bean id="oldImaginaryFeature" class="com.eranharel.OldImaginaryFeature" lazy-init="true"/>
</beans>

The spring code above defines an application and 2 imaginary feature implementations. By default, the application is initialized with the oldImaginaryFeature, but this behavior can be overridden by specifying a -DimaginaryFeature.implementation.bean=newImaginaryFeature command line argument. Only a single feature implementation will be initialized by Spring, and the implementations may have dependencies.

Bottom line is: with a bit of extra preparation and correct design decisions, feature flags shouldn’t be a burden on your code base. By extra preparation, I mean extracting interfaces for your domain objects, using an IoC container, etc, which is something we should be doing in most cases anyway.

Eran Harel is a Senior Software Developer at Outbrain.

Yeah — I know, monitoring is a “must have” tool for every web application/operation functionality. If you have clients or partners that are dependant on your system, you don’t want to hurt their business (or your business) and react in time to issues. At Outbrain, we acknowledge that it is a tech system we are running on and tech systems are bound to fail. All you need is to catch the failure soon enough, understand the reason, react and fix. On DevOps terminology it is called TTD (time to detect) and TTR (time to recover).  To accomplish that, you need a good system that will tell the story and wake you up if something is wrong long before it effects the business.

This is the main reason why we invested a lot in a highly capable monitoring system. With it, we are doing Continuous Deployment and a superb monitoring system is integral part of the Immune System that allows us to react fast to flaws in the continuous stream of system changes.

Note: Most of the stuff we have used to build it are open-source parts and projects that we gathered for our use.

A monitoring system is usually worthless if no one is looking at it. The common practice here is to have a NOC that is staffed 24/7. At Outbrain, we took another approach where instead of seating low level engineers in front of the screen all the time and alerting high level engineers when something happens, we built a system which is smart enough to alert the high level engineers in shift and tell them the story. We assume this alert can catch them wherever they are (at the playground with the kids, at the supermarket, etc…) and they can react to it if needed.

The only thing an Ops engineer in shift needs next to him is his smartphone and his notebook. Most of the time in order to understand the issue — he needs no more then a smartphone Gmail mailbox that looks like this:

Gmail Labels are used to color alerts according to the alert type — making best use of the short space available on the Android smartphone.

Nagios alerts are also sent via Jabber… to handle situations where Mail is down. The Nagios alert that comes with it includes the details about the alert and if it is a trend graph that is going over a threshold it also has the graph that shows the trend and tells the story.

And…. yes — we use Nagios, one of the oldest and most robust monitoring systems. Its biggest advantage is that it doesn’t know to do ANYTHING. That system just says “script for me anything you want me to monitor” and by that gives you full freedom to monitor everything you wish. We did great things with Nagios from monitoring the most basic vitals of the machines up to each and every JMX metric that our services expose and on into the bandwidth between our data centers.

One of the things that Nagios does not do well enough is supplying an aggregated view of the system state. For that we have put a “Nagios dashboard” application that puts the state in front of the engineer looking at it.

Our dashboard is based on NagLite3 with some local modifications to differently handle some situations and our naming scheme.

Let me tell you a little on how we actually give content recommendations here at Outbrain. This will be only a short introduction. We might elaborate on some of the below issues in future posts.

Our main goal is to serve good content recommendations to readers on the Internet. The typical situation is a user reading a content page. We want to recommend content for further reading, which is a “good” recommendation.

What is a good recommendation? We believe a good recommendation is one that is interesting to the user and is both timely and relevant. The user should not only want to click the recommendation title or image we put on the recommendations widget, but also like the page that he/she sees after the click and even want to continue investigating more content on the recommendation’s site. All recommendations should give a good experience to the user so he/she will become familar with our widget and know that it gives good recommendations. So with this goal in mind,  we do not use any “click traps” — we want a long-term relationship with the user, not a single click.

Note the target here is to serve content which is interesting to the user, and not to serve content which is relevant to content the user is already reading. Relevancy becomes just one of the methods to get interesting recommendations, not the target.

In order to serve good recommendations we run a large set of algorithms in parallel and get a set of candidate recommendations. Then, we decide which recommendations to serve to the user by machine learning techniques. I.e., We try to learn what the user or a group of users (the simplest group can be the readers of a specific site, but it can be more complex) like to read and serve it more often.

We can divide the algorithmic methods used to contextual algorithms, behavioral algorithms and personal algorithms:

Contextual algorithms analyze the context the user is reading now and finds relevant content. Relevant content can be interesting to the user. We use the Solr search engine with some enhacements we did here for the search, and we can also classify content into categories and use categories matching instead of a search.

Behavioral algorithms learn a set of statistical behaviors of groups of users. The simplest algorithms can bring the most visited documents in a site, the most rated documents, the ones with most social sharing events and so on. More complex algorithms can apply colleborative filtering methods to get other content which people who “liked” this content also liked.

From our experience we have seen behavioral algorithms perform differently than contextual algorithms. The best performance comes from giving a few recommendations from each type. On different sites, different algorithms give different results.
Personal algorithms are a specific subtype of behavioral algorithms, which learn the properties and history of a user, or a group of users, and give recommendations which will be interesting specifically to them. Note personalization brings with it both privacy issues and scalability issues. For example, we need to decide where to save the user data for fast access. We currently use cookies for this purpose (which are limited in size) but are considering a storage in the server level in the future. A user can opt-out from saving data on his actions.

Scalability is an issue in recommendations serving. We serve recommendations in an average of about 30 milliseconds. To achieve this fast serving time we do most processing in offline, saving results in a memory cache tool Memcached. We use key-value databases (like Cassandra) on top of traditional rational databases (MySql) to get a good response time for getting offline prefetched answers to queries (Data needed to calculate recommendations for documents, for example).

Time relevancy is a big issue — how do you decide if a document is still relevant? Some documents are always good, “evergreen” as we can them, but many age very fast. An article on a future sports event will age when the event happens. Stock market status reports become irrelevant very fast. We have some behavioral methods to try to understand users like these recommendations less over time, thus we stop serving them. Still some titles will make people click over and over even when the content is not relevant any more. Identifying relevancy is an interesting challenge.
We are totally measureable. We use Hive/Hadoop to create statistics about various aspects of the system. As an example, we know how good each algorithm performed in any environment (e.g. data center) in any source every hour, so we can always monitor logical performance of our algorithms and make intelligence decisions. We use the historic data for research. We even have learning algorithms that analyze current algorithmic performance and give best performing recommendations in a page or site more often for this page or site.

Development is done mostly in Java. We develop really Agile and fast. We use continuous deployment and have a staging environment to which we can deploy new algorithms and ideas very quickly. This means we can see how a new algorithm performs on real production data (a small fraction of it) a very short time after it was developed. We can do AB testing on algorithm properties and decide which value works best for each parameter.

Shlomy Boshy is Outbrain’s Algorithms Team Leader

Enjoy!!!

Some of you might ask, ”why is he telling us about datacenter architecture? Don’t Cloud Services solve this already?” and some of you that already know me and what my opinions are on the subject will not be surprised. Yes, I’m not a fan of the Cloud Services and that is another discussion, however, there are some advantages for using Cloud Services that giving them up by establishing a datacenter felt somehow wrong for us.

Here are 2 of them:

1. Grow As You Go – When you build a datacenter you take on commitments for space (racks or cages) and high profile network gear that are investments you have to pay for in advance or before you really need to use them. This is not an issue for a Cloud-based setup because as you grow you spin up more instances.

2. Disaster Recovery Headroom – With a datacenter-based setup, in order to properly handle disaster recovery you need to double your setup so you can always move all your traffic to the other datacenter in case of disaster, which means doubling the hardware you buy. In the Cloud, this is also a non-issue.

These 2 arguments are very much correct, however even taking those into consideration, our setup is much more efficient in cost then any Cloud offering. The logic behind it is what I want to share here.

Traditionally, when a company’s business grows, a single rack or maybe 2 are not sufficient and you have the need to allocate adjacent racks space in a co-located datacenter. This makes your recurring expenses grow since you actually pay for reserved space that you don’t really use. It’s a big waste of your $$$. Once we managed to set more than one location for our service we found out that it will be much cheaper to build multiple small datacenters with a small space footprint than committing to a large space that we will not use most of the time. Adjacent space of at least 4 racks is much easier to find in most co-location facilities.  More than that, our co-location provider agreed to give us 2 active racks with first right of refusal for the adjacent 2 racks so we actually pay for those we use.

This architecture also simplified much of our network gear requirements. Assuming each “LEGO Brick” is small, it needs to handle only a portion of the traffic and not all of it. This does not require high profile network gear and very cheap Linux machines are sufficient for handling most of the network roles including load balancing, etc.

We continued this approach for choosing the intra-LEGO Brick switching gear. Here we decided to use Brocade stackable switching technology. In general, it means that you can put a switch per cabinet and wire all the machines to it. When you add another cabinet you simply connect them in a chain that looks and acts like a single switch. You can grow such a stack up to 8 switches. At Outbrain, we try to eliminate single points of failure, so we have 2 stacks and machines are connected to both of them. Again, the stacking technology gave us the ability to not pay for network gear before we actually need it.

But what about Disaster Recovery (DR) headroom? (We decided to implement more than one location for disaster recovery as soon as we started generating revenue for our partners.)  As I said, this is a valid argument. When we had 2 datacenters, 50% of our computing power was dedicated to DR and not used in normal time. This was not ideal and we needed to improve that. Actually, the LEGO bricks helped here once again. This week we opened our 3rd datacenter in Chicago. The math is simple, by adding another location our headroom dropped to only 33% which is a lot of $$$ savings when your business grows. When we add the 4th it will drop to 25%, etc.

I guess now you understand the logic and we can mention some fun info about the DC implementation itself:

• Datacenters communicate via a dedicated link, powered by our co-location vendor.
• We use a Global DNS service to balance traffic between the datacenters.
• In our newer datacenters, the power billing is a pay-per-use — no flat fees which again enable us to not pay for power we don’t use. It also motivates us to power off unneeded hardware and save power costs while saving the planet
• Power is 208V which is more efficient than the regular 110v.
• All servers are connected to a KVM to enable remote access to BIOS config if needed — much easier to manage from Israel and in general.
• We have a lot of Dell C6100s in our datacenters so each node there is also connected to an IPMI network in order to remotely restart each node without rebooting all 4 nodes in that chassis.
• You can read more about assembling these C6100s in Nathan’s detailed post.

I guess your question is “what does it take to manage this in terms of labor?” That answer is… not too much.

The Outbrain Operations team is a group of 4 Ops engineers. Most of the time they are not doing much related to the physical infrastructure, but like other ops teams, most of the time they handle the regular tasks of configuring infrastructure softwares (we use all of them from open source like MySQL, Cassandra, Hadoop, Hive, ActiveMQ, etc…), monitoring, code and system deployment (we heavily use Chef) etc.

In general, Operations’ role in the company is to keep the serving fast, reliable and (very important) cost-efficient.  This is the main reason why we invest time, knowledge and innovation in architecting our datacenters wisely.

I guess one of the next posts will be about our new Chicago datacenter and the concept of the “Dataless Datacenter.”

Ori