As we are now few weeks into this POC and we gave you first glimpse into it in the first post. We would like to continue and update on what are the latest developments.
However before we come to the update. There are 2 things that we needed tell you about the setup that did not get into the first post just for length reasons.
The first thing to tell is the data structure we use as Doron explains it:
Our data is held using a single key as partition key, and additional 2 columns as clustering keys, in a manner of key/value structure:
- A – partition key (number).
- B – Clustering key 1 (text). This is the value type (i.e. “name”, “age” etc…).
- C – Clustering key 2 (text).
- D – The data (text). This is the value itself (i.e. “David”, “19”, etc…).
When storing the data we store either all data for a partition key, or partial data.
When reading we always read all data for the partition, meaning:
select * from cf_name where A=XXX;
When we need multiple partitions we just fire the above query in parallel and wait for the slowest result. We have come to understand that this approach is the fastest one.
As this meant to be the fastest, we need to understand that such reads latency is measured always by the slowest key to read. If you fire tens of reads into the cluster and If, by the chance, you bump into a slow response of one of the nodes (GC can be a good example for such case) your all read attempt is delayed.
This where we thought Scylla can help us improve the latency of the system.
The second thing we wanted to tell you about is what made this POC so easily done in Outbrain and this is our Dual DAO mechanism that Doron explains below.
Dual DAO implementation for Data Store migration/test
We have come to see that in many cases we need a dual DAO implementation to replace the regular DAO implementation.
The main use-case is data migration from one cluster to the other:
- Start dual writes to both clusters.
- Migrate the data – by streaming it into the cluster (for example).
- Start dual read – read from the new cluster and fallback to the old.
- Remove the old cluster once there are no fallbacks (if the migration was smooth, there should not be any).
The dual DAO we have written holds both instances of the old and new DAO, and also a phase – a number between 1 and 5. In addition, the dual DAO implements the same interface like the regular DAO, so it is pretty easy to inject it in and out when done.
- Write to old cluster and read from old cluster only.
- Write to both cluster and read from old cluster only.
- Write to both cluster and read from new cluster, fallback to old cluster if missing.
- Write to new cluster and read from new cluster, fallback to old cluster if missing.
- Write to new cluster and read from new cluster only.
The idea here is to support gradual and smooth transition between the clusters. We have come to understand that in some cases transition done only by the dual DAO can be very slow (we can move to 5 only after no fallback is done), so we had 2 ways to accelerate it:
- Read-repair – when in phase 3 or 4, and we fallback to the old cluster – write the data we fetch to the new cluster. This approach usually fits to use cases of heavy reads.
- Stream the data in manually by Data guys – this has proven to be the more effective and quick way – and this is what we have done in this case.
Dual DAO for data store test:
The ScyllaDB POC introduces a slightly different use-case. We do not want to migrate to Scylla (at least not right now) – but add it to our system in order to test it.
In order to match those demands, we have added a new flavor to our dual DAO:
- The write to the new DAO is done in the background – logging its results in case of failures – but does not fail the write process. When working in dual DAO for migration we want to fail if either the new or old DAO fail. When working in test-mode we do not. At first we did not implement it like this – and we had a production issue when ScyllaDB upgrade (to a pre GA version, from 0.17 to 0.19) caused the save queries to fail (on 9/3/16). Due to those issues we have changed the dual write to not fail upon the new DAO failure.
- The read from the new DAO is never used. On phases 3-5 we do read from the new DAO, and place a handler in the end to time the fetch and compare the results to the old DAO. However, the actual response gets back from the old DAO, when it is done. This is not bullet-proof but reduces the chances of production issues due to issues in the test cluster.
Mid April Update:
Last time we have reported was when ScyllaBD was loaded with partial and ongoing data while the C* cluster was loaded with the full historic data. Scylla was performing much better as we showed.
The next step we had to do was to load the full data into ScyllaDB cluster. This was done by Scylla streaming process that read the data from the C* cluster and wrote it to the ScyllaDB cluster. This process uncovered a bug with it that failed to extract the data from the newest version of C* that we’ve used.
Shlomi’s explanation is as follows:
Sstables upgraded from cassandra 2.0 to cassandra 2.1 can contain range tombstone with a different format from tombstones written in cassandra 2.1 – we had to add support for the additional format
The Scylla team figured it out and fixed it pretty quickly.More info about this issue can be found in the project GitHub.
After all the data was filled correctly into the ScyllaDB cluster, We’ve seen degrade in the ScyllaDB cluster which made it perform slightly worse than the C* (Surprise!!!!). A faulty configuration we’ve found in the C* “Speculative retries” mechanism and fixed it, actually made the C* latency about 50% better than the ScyllaDB. We need to remember we are measuring latency at its 99th percentile. In our case – as Doron mentioned above it’s while using multiple reads which is even harder.
ScyllaBD guys were as surprised as we are, They took a look into the system and found a bug with their cross DC read-repair, they fixed the issue and installed a new version.
Here is the description of the issue as explained by Tzach from Scylla: (reference to issue – https://github.com/scylladb/scylla/issues/1165)
After investigation, we found out the Scylla latency issue was caused by a subtle bug in the read-repair, causing unnecessary, synchronize, cross DC query.
When creating a table with read-repair chance (10% in this case) in Scylla or Cassandra, 10% of the reads send background queries from coordinator to ALL replications, including cross DC. Normally, the coordinator waits for responses only from a subset of the replicas, based on Consistency Level, Local Quorum in our case, before responding to the client.
However, when the first replica responses, does not match, coordinator will wait for ALL responses before sending response to the client. This is where Scylla bug was hidden.
Turnout, response digest was wrongly calculated on some cases, which cause the coordinator to believe local data is not in sync, and wait for remote DC nodes to response.
This issue is now fix and backport to Scylla 1.0.1
So, we upgraded to Scylla 1.0.1 and things look better. We can now say that C* and Scylla are in same range of latency but still C* is better. Or as Doron phrased it in this project google group:
Performance definitely improved starting 13/4 19:00 (IL time). It is clear.
I would say C* and Scylla are pretty much the same now… Scylla spikes a bit lower…
That did not satisfy the Scylla guys they went back to the lab and came back with the following resolution as described by Shlomi:
In order to test the root cause for the latency spikes we decided to try poll-mode:
Scylla supports two modes of operations – the default, event triggered mode and another extreme poll-mode. In the later, scylla constantly polls the event loop without no idle time at all. The cpu waits for work, always consuming 100% per core. Scylla was initially designed with this mode only and later switched to a more sane mode of operation with calls epoll_wait. The later requires complex synchronization before blocking on the OS syscall to prevent races between the cores going in/out sleep.
We recently discovered that the default mode suffers from increased latency and thus fixed it upstream, this fix did not propagate to 1.0.x branch yet and thus we recommend to use poll-mode here.
The journey didn’t stop here since another test execution in our lab revealed that hardware Hyper Threads (HT) may increase latencies during poll mode.
Yesterday, following our tests with and without HT, I have updated scylla configuration to use poll-mode and running only on 12 physical cores (allowing the kernel to schedule the other processes on the HT).
Near future investigation will look into why poll-mode with HT is an issue (Avi suspects it’s L1 cache eviction). Once the default, event-based code will be back ported to the release, we’ll switch to it as well.
This is how the graphs look today:
As you can see in the graphs above: (Upper graph is C*/ Lower graph is Scylla)
- Data was loaded into Scylla in 4/1 (lower graph).
- On 4/4 Doron did the fix in C* configuration and its latency improved dramatically.
- On 4/13 Scylla 1.0.1 was installed and did an improvement.
- On 4/20 Shlomi did the HyperThreading poll-mode config change.
- The current status is that Scylla base is nicely on the 50ms spiking to above 90ms whereas C* is at 60-65 spiking to 120ms in some cases. It worth to remember that those measurements are 99th percentile taken from within Outbrain’s client service which is Java and by itself suffers from unstable performance.
There is a lot to learn from such POC. The guys from Scylla are super responsive and don’t hesitate to take learnings from every unexpected results.
Our next steps are:
- We still see some level of results inconsistencies between the 2 systems that we want to verify where they come from and fix.
- Move to throughput test by decreasing the number of machines in the Scylla cluster.
That’s all for now. More to come.