Full speed ahead

When Kindred Group launched its new Kindred Futures initiative last year it pledged to “transform the experience” it offered to customers.

With this in mind, Kindred Futures recently partnered with start-up Cockroach Labs to develop a global database solution that will help the operator fulfil its promise to the player.

As a development partner, Kindred Futures is working with Cockroach Labs to investigate specific challenges that the operator is facing in terms of data privacy and technical requirements in a regulatory environment.

One of the key business needs Kindred wants to address is the ability to deliver a fast and reliable service to its customers across the globe.

Customers who are located far from Kindred’s datacentre tend to experience more lag but by working with the start-up the operator expects to be able to operate multiple active datacentres all synchronised with each other.

EGR Technology talks to head of Kindred Futures Will Mace, Kindred Group CTO Marcus Smedman and Cockroach Labs’ CEO and co-founder Spencer Kimball to find out how the collaboration came about and why data sovereignty compliance is an integral part of the database solution.

EGR Technology: Why did Kindred decide to develop a database solution with Cockroach Labs?

Will Mace (WM): We’ve traditionally had a single datacentre and customers wherever they’ve been in the world have accessed our products or services that run from that datacentre. But as we grow rapidly internationally it’s become apparent that that’s not a great solution to serve a globally diverse range of customers. The further a customer is away from a datacentre, the slower performance they receive. Given we have global ambitions and indeed an increasingly global customer base we needed a way of serving customers wherever they were in the world.

EGR Technology: How did the partnership with Cockroach Labs come about?

Spencer Kimball (SK): Marcus [Smedman] came out to New York a few years ago. Some of the staff on Marcus’s team were aware of Cockroach DB through sources like Hacker News where we had a lot of blog posts discussing the design of the system and early prototypes.

The staff let Marcus know that the technology might help solve some of their problems. So when Marcus was in New York the two of us met. It was the first time I’d heard about Kindred [Unibet at the time] and I didn’t know much about gambling in the EU.

The use case he described seemed like a good fit for what we were building. When the beta version came out I made a trip to Stockholm, met with the team and gave them a much more in-depth overview of the technology.

We realised some of the stuff we’d been thinking of, but hadn’t committed to implementing, around making Cockroach a suitable database for data sovereignty compliance would be fundamentally important in solving some of the problems Kindred was having in terms of satisfying the global customer base.

Kindred’s use case really inspired us and has heavily influenced the direction of our product development.

EGR Technology: And how does the partnership fit in with the ethos of Kindred Futures?

WM: Kindred Futures is the capability within the business that different people use to help explore strategically interesting opportunities, threats or problems we need to solve. The model Kindred Futures tries to operate under is that it takes a business problem, challenge or opportunity and it tries to find partners to help us solve these problems or exploit the opportunities.

We look externally when the problems are beyond our expertise and our core business. In this case we have a known business challenge and Cockroach is proving to be an ideal partner in helping us solve this.

EGR Technology: Why was a multiple on premise solution deemed the most suitablefor Kindred?

Marcus Smedman (MS): We want to be able to deliver a step change in the performance of our products and services by reducing latency as much as possible.

There is nothing more annoying to a customer than pages and screens taking an age [in terms of seconds] to load, or for bets to take an age to be accepted.

Customers are demanding almost instant responses; even just two or three seconds is far too long.

As our customer base is increasingly global they get further and further from our traditional datacentre locations and so the problem of latency increases.

The geo-replication element of CockroachDB means we can put data in multiple locations across the world, so it is as close as possible to our global customers, but at the same time ensure it is always in sync, wherever it is.

The ability to have data in several locations enables a multi active datacentre set-up, with improved performance, scalability and resilience as a result.

Obviously, data sovereignty compliance and customer trust is always most important, so the ability to control exactly where data is replicated and not is central in the solution.

SK: The meaning of cloud is rapidly changing. I think of cloud as a situation where you’re not having to manage the machines yourself. You’re able to spin them up with very little friction and very little time required.

That could be a public cloud such as Google or Amazon or it could be that you’re running things like Kubernetes and your whole organisation is running a private cloud. It’s still a cloud because people in your business are requisitioning the hardware and are responsible for monitoring and maintaining it.

I think a lot of the reason Kindred has been running a lot of the hardware itself as opposed to using the public cloud is regulatory.

There’s pretty strict oversight in the EU so they weren’t able to run the public cloud. Going forward they have a strong focus on quality control in terms of the network latency and uptime for the facilities they’re planning to use. They are very keen on maintaining a level of control both for regulatory and performance purposes.

Will Mace, Kindred Futures

EGR Technology: What kind of features or capabilities was Kindred looking for?

SK: Kindred was looking for one of the core capabilities of CockroachDB, which we call multi-active availability. CockroachDB is based on Google’s Spanner technology, which enables it to replicate the data consistently across facilities. It’s pushing up the expectations for SLAs.

That’s substantially better when you can sustain the loss of an entire facility. But CockroachDB has been evolving. Version 1.2 is going to be out in April 2018 with a new geo-partitioning feature. That allows you to keeps users’ data close to the user even when you have a global user base.

For instance, Kindred has a global customer base so it’s essential that users’ data is replicated in their geographic location, which is important for latency and data sovereignty.

Data privacy regulations, and with GDPR coming online in May 2018, put a lot of constraints on what you can do when you are building a global service.

WM: The compliance environment is getting ever more complex particularly with data privacy. It’s front of mind for consumers and regulators more so than five to 10 years ago.

So not transferring data across borders means the storage and processing of a customer’s data can be governed by their own nation’s rules, which makes the management of data a lot simpler.

EGR Technology: How does CockroachDB work and how is it built?

SK: It’s one of the few databases that was built post-cloud. It’s built in an environment where you have this expectation of being run, even if it’s on-premise, in an environment where you can give it additional resources and you expect it to grow elastically.

That’s something Google has been developing for a decade at least. But it’s new for other companies that are used to the database being very monolithic.

With CockroachDB we’ve simplified it as much as possible so it’s easy to run, whether you’re a start-up or a Fortune 500 enterprise.

It treats every node in a distributed system as symmetric. The nodes don’t have the same data but they act in the same way. Any piece of data would be replicated among say three of 10 nodes.

Any given node doesn’t have all the data but a fraction of it. Cockroach provides these symmetric nodes and when you get close to capacity you can add more nodes.

It will rebalance nodes that are full compared to the ones that are empty and you can decommission nodes. You can also elastically grow and shrink these clusters.

The point of having these replicas throughout the distributed cluster is to sustain integrity in the event of loss. For example, if a disk drive fails or a machine loses power, you have a set of replicas you can draw from.

Unlike traditional databases, CockroachDB uses a consensus replication mechanism. It eliminates the possibility of inconsistencies if there’s a fail over. With Cockroach, all the nodes evenly share the workload at the same time.

If one goes away, another one can pick up more activity in order to cover the loss. In terms of scale, if you keep adding nodes in the system it will move things around and rebalance.

You can actually make the system quite large. We’ve tested it with up to 128 nodes, which is a pretty big system.

Because Cockroach can do replication over geo-diverse replicas, it sets it up to maintain data for the customers wherever they are located.

It’s about building global data architecture so we can have customers in different geographical locations and not have to sustain super high latency.

EGR Technology: Cockroach Labs was founded by ex-Google employees. Have you brought any knowledge from your time at Google into CockroachDB?

SK: Absolutely. Google is one of the leading companies out there in terms of building these distributed systems. They’ve pushed very hard on storage infrastructure that works in a cloud setting.

Three founders of Cockroach Labs started at Google in 2002 and worked together on many projects including Google’s storage system called Colossus. It’s where Google still stores all of its data eg. all of YouTube, all of Gmail etc.

That knowledge was instrumental in how we approached the problem of building CockroachDB. We got to see how and why to build something like this.

CockroachDB was initially inspired by Google’s Spanner technology. Google invested in R&D to build Spanner and we got to see over 10 years the pain points and solutions that culminated in the design of Spanner.

When we left in 2012 we got out into the world of open source again. We were dismayed by the current offerings. The public cloud was still getting underway at that time in terms of adoption.

Things were changing. Kubernetes didn’t exist then.

After Google we came away with a detailed understanding of why companies need these kinds of solutions.

Spencer Kimball, Cockroach Labs

WM: At Kindred, we’re able to benefit from the experience of Spencer and his team to solve similar problems that Google had been facing.

SK: Kindred is much more advanced than many companies out there that haven’t even begun to grasp how to deal with this problem effectively. Kudos to Kindred for getting ahead of the curve here.

EGR Technology: What are some of the benefits Kindred hopes to see from CockroachDB?

WM: Having data closer to the customer and reducing the issues of latency so we can deliver a better customer experience. We know from all our customer research that speed and performance is absolutely crucial.

If you’re trying to place an in-play sports bet or play poker/slots the last thing you want is for the website or app to take ages to respond. And that’s what you get if you have a latency problem.

Also other benefits like data privacy and data resilience. The system can hold the data in the location of the customer, if laws and regulations permit it, so it’s governed by data protection that a customer expects it to be governed by.

We’re not live with CockroachDB in our production environment yet. We’re at the first stage of a multi-stage journey over the next months and years. These benefits will come in increments over that period.

One of the advantages of a mutually beneficial partnership is we have reasonably good visibility of the Cockroach Labs roadmap so we can plan their future enhancements into our service.

EGR Technology: What are the regulatory considerations you need to take into account with this database solution?

MS: Data privacy and data security is rightly a very important area in which we as an operator, as well as regulators and indeed customers take a keen interest.

Because there are very different data privacy rules in different parts of the world it is of the utmost importance that we continue to have a compliant, secure and scalable solution for data going into the future.

The geo-partitioning element of CockroachDB facilitates data sovereignty and facilitates data being kept in the environment in which it was provided.

This is really valuable to help customers know that their data is being protected by laws they are familiar with and is not being transported to places with different laws that they are not familiar with and might not be happy with.

EGR Technology: What’s the next step in the partnership?

WM: The current release [version 1.1] has just been rolled out and the formal release of the next launch is April 2018 for version 1.2. As a development partner we’ll see the contents of that early and we’ll be able to start integrating it.

Marcus Smedman, Cockroach Labs