Tech Blog

Hurricane Sandy – Outbrain Service updates

Oct 28, 2012 at 9:52 PM by admin | No Comments

Hi all As Hurricane Sandy is about to hit the east coast US, and as Outbrain’s main Datacenter is located in downtown Manhattan, we are taking measures to make as little service interruption as possible for our partners and customers. Outbrain is normally serving from 3 data centers and in case of NY data center loss, we will supply the service from one the other data centers. On this page, below – we will update on any service interruption and ETAs for problem solving. We assume all will go well and we will not have to update but… just in case

[UPDATE - Nov 3rd 3:45pm EST] - At this time Utility power is back to all our datacenters and HQ office. It is now time to restore the service from NY and get the office back to work. This will take some time but systems will gradually be put back up over the next week or so. There should be no effect on users, publishers or clients.

Our HQ will also start working gradually depending on the availability of public transportation.

We are here closing this reporting post – if you see any issues, please report to am@outbrain.com or your rep.

I hope the storm of the century will be the last one for the next century (at least).

[UPDATE - Nov 1st  9:30am EST] - Our HQ, located on 13^th between 5^th and 6^th in downtown New York City is still without power and therefore closed. Thankfully, our NY-based team is safe and in dry locations, and will continue to try and work as best they can. We highly appreciate the concern and best wishes we received from our partners and clients across the globe; thank you!

We are doing our best to continue to provide the best in class service, one we hope you’ve come to expect from us. As an update, our datacenter in NY is still without power and we expect it to be down for a few more days. We will continue to serve from our other datacenters located in Chicago and Los Angeles. To reiterate, our service did not go down, and we are currently still serving across our client’s sites. As of this morning, we recovered and updated all our reporting capabilities, so we should be back to 100%.

If you are experiencing any difficulties or seeing different, please reach out to your respective contacts. We’ll also continue to operate under emergency mode until Monday, you can reach us 24/7 at am-emergency-support@outbrain.com (am = Account Management).

[UPDATE - Oct 31st 6:46am EST] - Serving still holds strong from our LA and Chicago data centers and we are not aware of any disruption to our service. We are working hard to recover our dashboard reporting capabilities, but it will probably take a couple more days before we’re able to get back to normal mode. Sorry for any inconvenience caused by this. Send us a note to am-emergency-support@outbrain.com if you have any request, and one of us from around the world will respond as soon as possible.

[UPDATE - 6:51pm EST]  - Again, not much to update – All is stable with both LA and Chicago datacenters. It’s the end of the day here in Israel and we are trying to get some rest. Our team mates in the US are keeping an eye on the system and will alert us if there is anything wrong. Good night.

[UPDATE - 3:35am EST] - Actually not much to update about the service. All is pretty much stable. we are safely serving from LA and Chicago. most back-end services are running in LA Datacenter and our tech team in Israel and NY are monitoring and handling issues as they raise. Our Datacenter vendors in NY are working with FDNY to pump the water from the flooded generator room so it will take a while to recover this datacenter

[UPDATE - 10:50am EST] - The clients dashboard is back up.

[UPDATE - 10am EST] – The clients dashboard on our site is periodically down – we are handling the issues there and will update soon.

[UPDATE - 5am EST] Our NY Data center went down. Our service is fully operational and we are  serving through our Chicago and LA Data centers. If you’re accessing your Outbrain dashboard you may experience some delays in data freshness. We are working to resolve this issue and will continue to update.

[UPDATE - 2am EST] – Our NY Data center went completely off – We are fully serving from our Chicago and LA Data centers. External reports on our site are still down but we are working to fail over all services from the LA Datacenter. – we will follow with updates.

[Update - 12:50am EST] – power just went all off in our NY Datacenter  and provider has evacuated the facility – we are taking our measures to move all functionality to other datacenters.

[UPDATE]  - at 9pm EST]  commercial power went down on our NY Datacenter. Provider failed over to generator and we continue to serve smoothly from this Datacenter. We continue to monitor the service closely and ready to take actions if needed.

Posted in Uncategorized| No Comments

How to “Outbrain” Selenium Tests with Ext framework

Dec 22, 2011 at 6:15 PM by asaf | 19 Comments

Many of our internal applications were developed using the Extjs framework.

Extjs Is a very powerful JavaScript framework and one of the most popular javascript user interface open source framework , however when it comes to automated test with selenium the real challenge begin.

It is very difficult to write automated test to Ext application with selenium because Ext generates many <div> and <span> tags with an automatically-generated ID (something like “ext-comp-11xx”). Accessing these tags through Selenium is the big challenge we are trying to solve. We wanted to find a way to get these automatically-generated IDs automatically.
How do we approach this?

Ext has a component manager, where all of the developers’ components are being saved.  We can “ask” the component manager for the component ID by sending it a descriptor of the component. To simplify – we (the selenium server) tell the component manager “I need the ID of the current visible window which, btw, is labeled as ‘campaign editor’”.

This will look something like:


ComponentLocatorFactory  extjsCmpLoc = new ComponentLocatorFactory(selenuim);

Window testWin = new Window(extjsCmpLoc.createLocator(”campaign editor”Xtype.WINDOW));
Then we can to use Ext window method like close -> testWin.close();

Anther Example :

ComponentLocatorFactory  extjsCmpLoc = new ComponentLocatorFactory(selenuim);

Button newButton = new  Button(extjsCmpLoc.createLocator(“Add Campaign”, ExtjsUtils.Xtype.BUTTON));

newButton.click();

You can ask for all of the visible components by type, by label or both:

TextField flyfromdate = new TextField( extjsCmpLoc.createLocator(ExtjsUtils.Xtype.DATEFIELD, 0));

flyfromdate.setValue(“10/12/2011″);

TextField flytodate = new TextField(extjsCmpLoc.createLocator(ExtjsUtils.Xtype.DATEFIELD,  1));

flytodate.setValue(“10/31/2011″);

Here’s a simple diagram of our solution:

link to project in git-hub : https://github.com/simbal/SelenuimExtend

This solution is Open Source. In the meantime, if you have any questions, feel free to contact me directly. Asaf at outbrain dot com.

Asaf Levy

Asaf@outbrain.com

Posted in Dev Methods, Research, Uncategorized| 19 Comments

Leader Election with Zookeeper

Jul 10, 2011 at 8:19 AM by Erez Mazor | 2 Comments

Recently we had to implement an active-passive redundancy of a singleton service in our production environment where the general rule is always have “more than one of anything”. The main motivation is to alleviate the need to manually monitor and manage these services, whose presence is crucial to the overall health of the site.

This means that we sometime have a service installed on several machines for redundancy, but only one of the is active at any given moment. If the active services goes down for some reason, another service rises to do its work. This is actually called leader election. One of the most prominent open source implementation facilitating the process of leader election is Zookeeper. So what is Zookeeper?

Originally developed by Yahoo reasearch, Zookeepr acts as a service providing reliable distributed coordination. It is highly concurrent, very fast and suitable mainly for read-heavy access patterns. Reads can be done against any node of a Zookeeper cluster while writes a quorum-based. To reach a quorum, Zookeeper utilizes an atomic broadcast protocol. So how does it work?

(more…)

Posted in Dev Methods, Uncategorized, leader election, spring, zookeeper| 2 Comments

Under the Hood of Our Algorithmic Engine – How We Serve Content Recommendations

Apr 17, 2011 at 8:59 AM by Shlomy Boshy | No Comments

Let me tell you a little on how we actually give content recommendations here at Outbrain. This will be only a short introduction. We might elaborate on some of the below issues in future posts.

Our main goal is to serve good content recommendations to readers on the Internet. The typical situation is a user reading a content page. We want to recommend content for further reading, which is a “good” recommendation.

(more…)

Posted in Uncategorized| No Comments

Hello Techies!

Mar 12, 2011 at 8:50 PM by admin | No Comments

A lot has been going with technology on outrain in the last year or two and we haven’t had the time to sit and tell it to the world. Well, some of it we share in our Hebrew podcast, Ran have shared some of it in his blog but I felt it  is time to set-up a formal technology blog for Outbrain.

Some of the posts here will be written by myself but I truly hope that most of them will be written by the team members (those that actually do the work ).

what is it going to be about? Well, that’s easy, Tech stuff like:

• Software geeks goodies that we do in Outbrain.
• infrastructures and systems we use and enthusiastic about.
• we will try to share some of our architectural designs and considerations.
• Ops/IT stuff, how we choose to do things.
• our culture and how we work to “Get things done”
• cool algorithms and how to make them scale.

OK, enough with the build-up, let’s see how much time will it take to write the first tech post.

Enjoy the blog

Ori Lahav

Outbrin CTO

Posted in Uncategorized| No Comments