Hi all As Hurricane Sandy is about to hit the east coast US, and as Outbrain’s main Datacenter is located in downtown Manhattan, we are taking measures to make as little service interruption as possible for our partners and customers. Outbrain is normally serving from 3 data centers and in case of NY data center loss, we will supply the service from one the other data centers. On this page, below – we will update on any service interruption and ETAs for problem solving. We assume all will go well and we will not have to update but… just in case
[UPDATE - Nov 3rd 3:45pm EST] - At this time Utility power is back to all our datacenters and HQ office. It is now time to restore the service from NY and get the office back to work. This will take some time but systems will gradually be put back up over the next week or so. There should be no effect on users, publishers or clients.
Our HQ will also start working gradually depending on the availability of public transportation.
We are here closing this reporting post – if you see any issues, please report to email@example.com or your rep.
I hope the storm of the century will be the last one for the next century (at least).
[UPDATE - Nov 1st 9:30am EST] - Our HQ, located on 13th between 5th and 6th in downtown New York City is still without power and therefore closed. Thankfully, our NY-based team is safe and in dry locations, and will continue to try and work as best they can. We highly appreciate the concern and best wishes we received from our partners and clients across the globe; thank you!
We are doing our best to continue to provide the best in class service, one we hope you’ve come to expect from us. As an update, our datacenter in NY is still without power and we expect it to be down for a few more days. We will continue to serve from our other datacenters located in Chicago and Los Angeles. To reiterate, our service did not go down, and we are currently still serving across our client’s sites. As of this morning, we recovered and updated all our reporting capabilities, so we should be back to 100%.
If you are experiencing any difficulties or seeing different, please reach out to your respective contacts. We’ll also continue to operate under emergency mode until Monday, you can reach us 24/7 at am-emergency-support@outbrain.
[UPDATE - Oct 31st 6:46am EST] - Serving still holds strong from our LA and Chicago data centers and we are not aware of any disruption to our service. We are working hard to recover our dashboard reporting capabilities, but it will probably take a couple more days before we’re able to get back to normal mode. Sorry for any inconvenience caused by this. Send us a note to am-emergency-support@outbrain.
[UPDATE - 6:51pm EST] - Again, not much to update – All is stable with both LA and Chicago datacenters. It’s the end of the day here in Israel and we are trying to get some rest. Our team mates in the US are keeping an eye on the system and will alert us if there is anything wrong. Good night.
[UPDATE - 3:35am EST] - Actually not much to update about the service. All is pretty much stable. we are safely serving from LA and Chicago. most back-end services are running in LA Datacenter and our tech team in Israel and NY are monitoring and handling issues as they raise. Our Datacenter vendors in NY are working with FDNY to pump the water from the flooded generator room so it will take a while to recover this datacenter
[UPDATE - 10:50am EST] - The clients dashboard is back up.
[UPDATE - 10am EST] – The clients dashboard on our site is periodically down – we are handling the issues there and will update soon.
[UPDATE - 5am EST] Our NY Data center went down. Our service is fully operational and we are serving through our Chicago and LA Data centers. If you’re accessing your Outbrain dashboard you may experience some delays in data freshness. We are working to resolve this issue and will continue to update.
[UPDATE - 2am EST] – Our NY Data center went completely off – We are fully serving from our Chicago and LA Data centers. External reports on our site are still down but we are working to fail over all services from the LA Datacenter. – we will follow with updates.
[Update - 12:50am EST] – power just went all off in our NY Datacenter and provider has evacuated the facility – we are taking our measures to move all functionality to other datacenters.
[UPDATE] - at 9pm EST] commercial power went down on our NY Datacenter. Provider failed over to generator and we continue to serve smoothly from this Datacenter. We continue to monitor the service closely and ready to take actions if needed.
A lot has been going with technology on outrain in the last year or two and we haven’t had the time to sit and tell it to the world. Well, some of it we share in our Hebrew podcast, Ran have shared some of it in his blog but I felt it is time to set-up a formal technology blog for Outbrain.
Some of the posts here will be written by myself but I truly hope that most of them will be written by the team members (those that actually do the work ).
what is it going to be about? Well, that’s easy, Tech stuff like:
- Software geeks goodies that we do in Outbrain.
- infrastructures and systems we use and enthusiastic about.
- we will try to share some of our architectural designs and considerations.
- Ops/IT stuff, how we choose to do things.
- our culture and how we work to “Get things done”
- cool algorithms and how to make them scale.
OK, enough with the build-up, let’s see how much time will it take to write the first tech post.
Enjoy the blog