How to Perform Cloud Migration by Leveraging Apache ShardingSphere Database Mesh Capabilities

Trista Pan at UXDX Community: APAC Online Series. Video: https://youtu.be/djgC8_7qdTo

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction

[00:00:02] Host: Trista Pan, who will be talking to us about how to perform cloud migration by leveraging Apache ShardingSphere database mesh capabilities. And just for context before I finish introducing Trista: the idea of UXDX is to pull together product, UX, design and dev. So this is going to be a more technical talk, but the purpose is exactly what we were just talking about in the last session. If you can build your T-shaped skill, if you can understand a bit more about what's happening in some of the other parts of the product development life cycle, you'll be able to contribute better as a team member, and it kind of builds that generalist capability. So with that kind of introduction, I would like to introduce Trista, who will be sharing her story. Welcome, Trista.

Making a technical topic a little more product

[00:01:00] Trista: Hello everyone. Wait a moment, let me share my screen. All right, hello everyone. I agree with what we discussed in the previous talk, that no matter whether it's about the product or about the tech part, if we can share our ideas, make us on the same page, then that will be great to make the cooperation between the product team and the tech team work well, to work together.

[00:01:42] So today my talk is a little tech part, because you can see this topic, it's about your database, it's about the distributed system, it's about Kubernetes. So when I mention three hot words today, people will consider that, oh, it's a little far from my work. But today I just want to make this topic become a little more product.

[00:02:15] I just gave the introduction about this solution, about what is happening in the database and in Kubernetes, in this distributed system. So if you are a product manager, it will be great for you to get such useful messages for your company's product, or for you to give some suggestions for the other team members, right?

[00:02:47] So let's get started with my talk, because today I only have 25 minutes. In this slide it's basically a demo show. I have no time to introduce the demo. If you are interested in the demo, I guess you can later on refer to this slide and do the demo by yourself. So today we will just focus on the solution, and focus on the three keywords: distributed system, database system and Kubernetes.

About me and ShardingSphere

[00:03:24] This profile is about myself. I'm Trista Pan and now I'm the SphereEx co-founder and CTO. Also, I spend a lot of time in the Apache open source software foundation, so I'm an Apache member and an incubator project mentor. That means my role actually is a mixed one. Sometimes I need to help our tech team to do the technical development part. The second one, because our company's product is about the infrastructure, database product, so sometimes I need to switch my role to be the product manager, to help to design the database distributed system from the product manager perspective.

[00:04:23] So that's my role. And now I'm the ShardingSphere PMC. ShardingSphere is a top level project of the Apache Software Foundation, and today we will leverage this product to introduce how to use it for your distributed database system on Kubernetes. Sometimes I will post some articles about open source, about the community, about the product, about the tech part, around all the stuff. So if you are interested in such topics you can just give a look at my Twitter or LinkedIn.

Digital transformation and the new needs for databases

[00:05:04] Today, first, let us get into the background about digital transformation, because I guess everyone knows this word, digital transformation. People want to leverage the new technology to help them to provide more productive and great product to our end user, or totally change the development paradigm, or to change the delivery culture, all the stuff. So that's about digital transformation. First about the new technology, second one it's the paradigm, or it's the culture change.

[00:05:48] So that's the background, and because of this background people actually have new needs for our database system and for our cloud. Because you know that nowadays, when people get into the big data era, you find that your databases... oh, maybe you found your website visiting become so slow. So why does it become so slow? Because your databases need to help you manage the tremendous data. And in that situation, how to make your product, make your database product support your visiting from your application, from your website, and how to make it become secure, and how to make it become elastic, scale out, elastic scale? All of the new needs from big data, from the digital transformation background.

[00:06:49] Here I list the people's new requests for your database system: the large data to manage, the efficient querying, and you need to do the traffic governance, and how to reach the elastic scaling goal and an out-of-the-box deployment. So that means, at the beginning maybe you have your good PostgreSQL or MySQL, they work so great on premises or on the cloud, but now you need to do some change about this database infrastructure.

[00:07:32] In order to reach such goals, actually you need the following techniques. First, data sharding. Data sharding means that at the beginning you only have one single database, but now you want to make the single database split into a lot of small portions, which will be located in different machines or different locations. And that's one. Then your traffic governance, that means you need to improve the high availability of your system and to govern each of your queries, to leverage the replicas of your databases to make the throughput become higher.

[00:08:23] And then what about resharding? That means you want your database system to have the elastic scaling feature, so your new database system needs to be able to help you reshard your databases. And that's one. Then we can use Helm or an operator on Kubernetes, this mature and wonderful platform or cluster, to help you manage your database cluster and also at the same time retrieve the three points. So that's the new needs, and we need the new technique.

[00:09:05] And now we have a solution to help us include all the new techniques into one solution. That means this solution can provide an out-of-the-box product or project for you just to use. So in the next part I will introduce how to present this solution for you to reach the following new requirements.

From monolithic databases to a distributed system

[00:09:39] I need to first give a brief introduction about our monolithic database, because if we want to solve the issues we need to first consider what's the database, and then we use the database's functions to upgrade it, become the current modern database system. So what's a database? Actually there are two major functions of your database system. The first one, computing capability. The second one, storage capability.

[00:10:17] So for the monolithic databases, for example PostgreSQL, actually each of them have the computing capability and the storage capability, but they will put the two layers, or the two components, to work together in the same machine. That means they are monolithic databases. But in a distributed database system, or in a sharding database system, actually the computing components and the storage components are distributed in different locations in this world. So that's the major difference between the monolithic databases and the modern distributed or sharding database system.

[00:11:00] So now if we want to make our traditional databases upgrade to become a distributed one, that means we need to split the computing nodes and storage nodes into different machines or different locations. So that's the approach to solve this issue.

Adding a computing layer in front of your existing databases

[00:11:25] That means before, maybe your database infrastructure looked like this one: your application just directly visits your PostgreSQL or MySQL cluster, or visits these primary nodes or replica nodes. But now, if we want to, here, afterwards you will have this modern database infrastructure. How to reach that? That means we need to import another component between your application and your traditional databases.

[00:12:07] So your application will just visit this new component, and this new component will work as a computing node of this distributed database system. And your traditional databases, at this point, will become the storage nodes of this distributed database system. That means this new component plus your traditional databases will become a new distributed database system, and in this system the new component is a computing layer, and your traditional databases, no matter if it's MySQL, Oracle, SQL Server or PostgreSQL, will act as the storage nodes.

[00:13:04] So that's the new solution. Because we use this new one, we can upgrade your previous database infrastructure to become a distributed one, and your application can just visit this computing node, or this database proxy, and this database proxy will help your end user or application to manage all of your thousands or hundreds of database clusters.

[00:13:36] And you don't need to care about the autoscaling, or about the traffic governance, or about the data sharding. All of the features will be taken over by this database proxy, or this computing node, for your application. Your application just thinks that there is a distributed system, and it doesn't care about all the technical details about how to autoscale, how to do the traffic governance, how to improve the high availability of this database system. Now all of the features are transparent for our application.

What ShardingSphere is

[00:14:25] So here we leverage the new component and make the new component merge with your traditional databases, and make it become distributed. And what's this new component? That's ShardingSphere. ShardingSphere is an Apache top level project. Actually you can see here its slogan: ShardingSphere is the ecosystem, or it's a proxy, to transfer any of your databases to become a distributed one and enhance it with the sharding feature, elastic scaling feature or data encryption feature. So by using this ShardingSphere Proxy project, then we can upgrade your traditional MySQL or PostgreSQL database to become a distributed one.

[00:15:22] Actually ShardingSphere has been open sourced for more than five years. So you can see here, this project has a mature community. We also have more than 400 contributors and have already released nearly 50 versions. So I guess if you have any questions, this community will answer your question, will provide their help as soon as possible, and its documentation also can help you to quickly set up this project for your environment.

Two clients: ShardingSphere-JDBC and ShardingSphere-Proxy

[00:16:04] So by using this one we can upgrade your existing database system to become a distributed one. Actually this project provides two clients for the end user to choose. The first one is named ShardingSphere-JDBC. It's for your Java application. It's a lightweight Java driver, and if your application was developed in Java then you can just use the Maven artifact and use this client, and it can help you to do all the features I mentioned before, and it can upgrade all of the databases to become a distributed one and help any user or your application to manage all the database clusters.

[00:17:02] The next client is ShardingSphere-Proxy. It's a database proxy, it works as the database server, the distributed server, so you need to deploy ShardingSphere-Proxy on a single machine. That means, look here, compared to the JDBC one, it needs to be deployed independently, and JDBC just needs to be deployed with your application together, right? So with ShardingSphere-Proxy you can just use your MySQL protocol or your PostgreSQL, the third party connection approach, to connect to ShardingSphere-Proxy. And that means it will not totally change your previous way to connect to your monolithic databases. And then, actually ShardingSphere-Proxy and ShardingSphere-JDBC share the same features, like I mentioned before. They are just the two clients for you to choose.

[00:18:13] So no matter whether you use ShardingSphere-JDBC or ShardingSphere-Proxy, it can help you finish the following features: data sharding, elastic scale out, distributed transactions, read/write splitting and authentication. All of them, the data sharding issue or the scale out, scale in issue. But today in my demo I just use ShardingSphere-Proxy, and help the user to solve the distributed data issue, that means the data sharding issue.

Delivering it on Kubernetes with charts and the operator

[00:18:52] So the last part, I will give the solution. Because I mentioned a lot about this architecture of your new database infrastructure, the next one is how do we efficiently deliver this product to an end user? That means we need to use the Kubernetes system.

[00:19:16] Here, in this repo, ShardingSphere-on-Cloud, it provides many ShardingSphere charts. That means you use Helm, then it can help you deploy the ShardingSphere cluster and the ShardingSphere Operator into your Kubernetes system. That means you can just use a few commands on your CLI, then you can do the demo for testing the ShardingSphere charts. It can help you deploy and upgrade the ShardingSphere cluster. And the ShardingSphere Operator, it can guarantee the high availability and the scale out, the HPA of the ShardingSphere clients. So there are different charts for different functions, but we recommend you can use them together, so you don't worry about the deployment, the upgrade, HPA, high availability, all the features. So we recommend our end users use them together.

[00:20:29] All right, so that's all about today's solution, and the next part. If you use ShardingSphere charts to deploy ShardingSphere on Kubernetes, in the next part maybe you need to use one of the PostgreSQL charts to deploy your PostgreSQL on Kubernetes. That means by using the PostgreSQL charts you have your storage nodes, and by using ShardingSphere charts you will have your computing nodes. So like I mentioned before, your computing nodes plus storage nodes will become the final distributed database system for your application.

[00:21:13] So for your application, they can just use the previous approach to connect to this new distributed database system, and it helps you do the database sharding and improve the throughput, and also can make each query become faster than before. So that's all of the capabilities of this new database system.

[00:21:43] And now, for example, you deploy PostgreSQL here and also ShardingSphere, and your application just visits ShardingSphere Proxy, to tell ShardingSphere Proxy and your databases that, hey, I just want to split this order table into one, two, three, four, five subtables in your database system. Could you help me do the data sharding, or could you help me do the read/write splitting? You can just tell ShardingSphere all of what you want by SQL, and then ShardingSphere Proxy can automatically deal with all of your requests from our end user, and manage your databases and do the global computing for each of your queries.

How SQL is handled, and the demo

[00:22:45] So the next part, handling the process of SQL. That means your application just sends your queries to this ShardingSphere Proxy, and ShardingSphere Proxy, by using some sharding algorithm or some strategies, helps you find the target row or target record, and gathers the records to do the global computing and send a result to your end user.

[00:23:18] So even though we upgraded by using ShardingSphere Proxy, the process of the SQL actually has no changes compared to the previous one, because for the database system we don't change its computing power and the storage power, we just make it become stronger, become more powerful. But actually for the handling process of SQL, they are unchangeable[?].

[00:23:53] Okay, so that's all about the solution's introduction. The next part is about the demo show, but we have not enough time to introduce them step by step. Actually I want to say that I gave each step for you to refer to this slide to do the demo by yourself.

[00:24:24] And you can see here, for example, if you execute the SQL, this one, from the ShardingSphere Proxy, then you can get the result right here. But if you log in to your actual storage nodes or PostgreSQL instance, you will find that in your actual databases there are many sub rows, or sub readouts from each PostgreSQL instance. So that's the function of our ShardingSphere Proxy. It will hide all of the sub readouts or the actual rows. It will just give you the final readout, so people will not worry about all of the middle readouts from this database system. All right, so that's all about today's talk. Any questions?

Q&A

[00:25:45] Host: Thank you for that. I think you covered that in quite a bit of detail, so it was interesting to see, and it's a problem that people might not be aware of happens as frequently, where your database just kind of starts getting slow and you have to figure out how to solve it. One thing that did come to mind, and a question that came up, is what would be the benefit and when should companies start looking down this kind of sharding route, or maybe looking at a less hands-on approach, migrate to CockroachDB or something like that, where the database itself and the hosting is kind of managed elsewhere, a serverless host, maybe Aurora or something like that?

[00:26:30] Trista: All right, okay. Actually, when we see CockroachDB or other distributed databases, they also use the same technique here, like data sharding, because CockroachDB also has a lot of distributed instances. But here, the major difference is that in CockroachDB, their storage layer, they use another storage component like a KV or other storage types. But here, this solution also shares the sharding technique, but it uses your traditional databases to work as the storage layer.

[00:27:22] Trista: So the benefit from today's solution is that, if you want to choose a distributed database, for example CockroachDB or Aurora databases, that means you need to migrate your original data into a new one, right? So that's a general action for our DevOps team or DBA. But this solution, today's solution, you don't need to do any change about your previous database cluster, and just import a new computing node into your previous database system. So you can share the same data sharding technique with the new databases, I mean CockroachDB or Aurora.

[00:28:15] Trista: So I guess for any users, you need to base it on your scenarios to choose all of the databases. And if you want to just develop from scratch, so it's okay for you to choose CockroachDB or YugabyteDB or the other databases, that's fine. But if you are a little worried about that, oh, if I don't want to do anything to my existing databases and just make it upgrade, then you can choose today's solution.

[00:28:54] Host: Great, no, that's a great answer, and thanks for clarifying, because yeah, migrating your database is no easy feat.

[00:29:02] Trista: Oh yeah.

[00:29:06] Host: So yeah, I think that brings us to the end. So thank you very much for sharing, really enjoyed the talk, and I hope everybody out there did as well. So thank you very much.

[00:29:16] Trista: Yeah. See you next time, guys. I'm sorry you had to endure my tedious talk[?] to listen to this tech speak. Thanks everyone.

[00:29:27] Host: Thank you.