Open-source Projects in Modern Data Stacks

William Guo at UXDX Community: UX Research - Facts & Feelings. Video: https://youtu.be/4O4pm_V8aSE

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction and the modern data stack

[00:00:00] Thank you all for joining our session. Today I want to share some experiences on open source projects in modern data stacks. First of all, I want to introduce myself. I'm William, and I am an Apache Software Foundation member. Being an Apache Software Foundation member means there are a lot of Apache open source projects, like Hadoop, Spark and other things. I used to work at IBM, Teradata, CSCC [?], and I also used to work as CTO of Analysys [?], and I've been big data director of Lenovo.

[00:00:44] Today I want to share some experience on how to build your big data platform using some new open source projects. There are many, many requirements on big data, especially now that we have a new concept we call the data community. That means not only the engineers, but a lot of product managers, UX designers, your sales analysts or your boss will use the data directly, not only the engineers. And there will be many, many new open source projects in the world.

[00:01:32] There's also a new concept we call the modern data stack. What is the modern data stack? It means there will be a set of tools that can handle big data analysis. It's not only one. In the old days, for example, we had Microsoft SQL Server or Teradata or something like that, but nowadays we have many, many different open source projects or tools to handle the whole big data problem.

Open source tools for each layer

[00:02:13] I'd like to introduce some of the open source projects. For example, now we have many, many different data sources. There are some open source projects to collect the data from different data sources. For example, you can collect the data from Google Analytics, or you can collect the data from your database, or from an Excel file or some other files. Then you can integrate all the data into your data warehouse to do some data analysis. There are many open source data integration projects, such as Airbyte, which is from America; SeaTunnel, which is from the Apache Software Foundation; Jitsu, which is from Japan; DataX, which is from China; and RudderStack, which is also from America. These are open source data integration tools.

[00:03:19] We also have many, many data warehouses. I know that Snowflake is very popular now, but we are talking about open source data warehouses. I think Trino is very popular in America, ClickHouse is very popular in Russia and Europe, and Pinot is very popular in America. Iceberg is also a popular project, in America and in China. Materialize, I think, is very popular in Europe. And Doris, StarRocks, big hidden [?], I think they are very popular in China.

[00:04:04] We also have some open source projects to do transformation. Data transformation means you can transform the data from one column to many other columns, or you want to do some calculation in big data. There are some open source projects like Flink, which is an Apache open source project; I think it was born in Europe, and now it's very popular in China. dbt is very popular in the US. Apache Beam is now very popular in the US, because it's a Google-based open source project. Spark, I think, is very popular all over the world. And Samza, I think, is popular in Europe.

[00:05:06] We also have some open source projects in the orchestration area. Big data orchestration means it can orchestrate many workflows, because you will have many, many big data jobs. For example, perhaps you have 1,000 SQL jobs, so you have to arrange them: task A is the first to run, then you run task B, then you have to run task C. How to orchestrate them together is what these orchestration tools do. Airflow is very popular all over the world. But now we have another orchestration tool called DolphinScheduler. I think it's very easy for non-engineering users to use for big data orchestration, because you drag and drop to create workflows. Flyte is very popular in AI, very popular in America.

[00:06:29] There are also many, many BI and analysis tools that are open source, and you can use them to analyze your big data, such as Superset, new chart [?], Querybook, Redash, Metabase, or Dash [?]. These are very popular open source projects all over the world. And I know that ChatGPT is very hot these days. Actually, the model training for ChatGPT is based on our way [?], let's say, which is also an open source project. We also have PyTorch and Jupyter and TensorFlow, and MindSpore [?] and PaddlePaddle and OpenMLDB [?]. These are some open source AI projects. So you can see there are many, many open source modern data stack tools, and you can choose one if you want.

The ETLT architecture

[00:07:35] Today I want to introduce some basic ideas of how to use these kinds of modern data stack tools to build your own data analysis. Now we have what we call the ETLT architecture. Perhaps you have a lot of data, but your data is stored, for example, in SaaS, or in your spreadsheet, or in your cloud database. So you have to extract the data from different data sources, and then you do what we call a transform, so we can load this kind of data into your data warehouse. This process we call ETL: extract, transform and load.

[00:08:36] Then, when you ETL the data source into your data warehouse, you can do a lot of business transformation in your data warehouse, in your Snowflake or in your Trino or Hadoop. And then perhaps some business analysts or data application developers, or you, if you can write SQL, even a product manager can write SQL in Snowflake or Trino to do your data analysis. And some data scientists can use the AI tools to do more advanced analysis algorithms. For a small company, you can use this big data architecture to build a very easily used big data application.

Real-time and batch data for small and medium enterprises

[00:09:38] Nowadays we have two different types of data. One we call real-time data, and the other we call batch data. Real-time data means you have to collect the data in real time, you have to compute the data in real time, and you have to analyze the data in real time. Now we have SeaTunnel. It's an Apache open source project, and it's free, and you can collect the data whether it's in your IoT devices, your database, your spreadsheet, your logs, or your SAP or ERP or SaaS software. You can collect the data in real time and then load it into Kafka. Kafka is a middleware that stores messages in real time.

[00:10:40] Or you can just load the data into your real-time data warehouse. For example, you can load the data into ClickHouse, or into AWS Redshift, or into another real-time data warehouse. You can also send it into your alert system. That means you can send it to the API and you can alert whether some IoT devices are wrong or not. You can monitor the whole real-time data flow.

[00:11:18] The other way we call batch data. You can also use SeaTunnel to load the data from your spreadsheet, from your Google Doc, from your Salesforce or some other source. You can also extract the data from Oracle, from MySQL or SQL Server or some database. And it's very easy to use. That means you don't need to be an engineer or a programmer; you can just use a very, very easy script to do these kinds of things. And then you can load the data into your data warehouse, your data lake or your OLAP engine.

[00:12:07] If you want to do some data analysis, to write some SQL, and handle the whole orchestration, the whole workflow, you can use Apache DolphinScheduler. You can just drag a job to create ETL jobs, or your Hadoop jobs, or your Flink jobs. You can also manage what we call MLOps, machine learning ops, and you can manage your AI training process with DolphinScheduler. So it's something like the traffic light in a big data platform. And then you can also use SeaTunnel to load the data from your data warehouse back to your SaaS or back to your database. So this is simply a big data architecture for a small or medium enterprise.

Apache SeaTunnel

[00:13:10] Now I want to introduce two of the projects. One of them we call Apache SeaTunnel, and it is in the Apache incubator. Because we have many, many data sources, that's why we have to use Apache SeaTunnel. There are so many data sources, Google Docs and any other database, and sometimes you have to use real-time and batch data together, and you have to monitor the whole data extraction process. SeaTunnel is a very easy-to-use project that can extract the data from different data sources and then load it into different data targets, and the performance is very good.

[00:14:10] Now it has more than 100 connectors to help you synchronize the data to anywhere. You can see there are many, like cloud connectors, database connectors or SaaS connectors. And it is free, because it's an open source project. So I think it's a very, very useful open source project to integrate your data from different databases, different clouds or different SaaS into your database or your data warehouse.

[00:14:57] And I think it has some very good features. For example, it's very easy to deploy and use. You don't need to deploy Hadoop or Flink or Spark, because it doesn't depend on a Hadoop environment. You can deploy only one server that can synchronize many, many records in one second. So I think it's easy to use and easy to deploy, and the performance is very good. And it just uses SQL-like code; I think everyone can use SeaTunnel to synchronize data. There are many, many other features.

[00:15:42] You can see here the example for SeaTunnel. You just write a script like this. It's just like a SQL select and replace or something like that. You write the connector, like the TypeScript, and then you can write a sink in this code sample. So I think it's very easy for somebody who is not an engineer to synchronize data from different data sources. There are many, many other features as well. It's very popular in China; it has many users in China, and also in Singapore and, I think, in India. So I think this is a very good data integration project.

Apache DolphinScheduler

[00:16:43] I want to introduce another project, called Apache DolphinScheduler. I think it makes it easier for other people to use the big data platform. We often meet some challenges in our big data platform: bad performance, latency. And sometimes we have to manage real-time jobs and batch jobs together. Sometimes we lack resource control, and we have inefficient data processes. Moreover, I think it's very hard for people who have no engineering background to use orchestration tools, because with, for example, Azkaban or Airflow, you have to use Python to design the whole process.

[00:17:58] DolphinScheduler is designed so you just drag and drop to create your workflow, and you can also use Python to generate your workflow. Now its active users, I think, are more than 3,000 users all over the world, and the stars of this Apache project are more than 10,000. So I think it's a very strong open source project. The interface is very, very easy. You can just drag and drop to create a SQL or Hadoop job, and then you can drag and drop jobs to create the whole big data analysis process.

[00:18:47] And it's very powerful, because it is what we call multi-master and multi-worker. That means it scales, and it is very stable. So you can use DolphinScheduler to manage a very, very big amount of workflows in one cluster. You don't need to create many, many small clusters to manage the workflows.

[00:19:20] There will be some new features. In the old version you could just drag a job; we call that what you see is what you get, to create your big data workflow. It has high reliability, it has rich workflow functions, it's cloud native, and the extensibility is very good. In the new version we support machine learning orchestration, we support real-time tasks, we support Python and YAML workflows to generate the workflow, and we also support Kubernetes. Those are some technical features, but for people who are not very familiar with engineering, I think the what-you-see-is-what-you-get way of creating a big data workflow is very easy to use.

[00:20:20] There's a comparison between different open source projects. I don't want to dive into the details, but I think the three projects have their own pros and cons. If you are a beginner big data user, or you are not a user with an engineering background, I think DolphinScheduler is really easy for you to use to orchestrate the whole thing. Now it has many, many users, for example Cisco and IBM, and also many users in China and in India. So if you are interested in this project, you can use it for your enterprise.

[00:21:20] There are also a lot of use cases in China, in Singapore and in India. I can mention one, for Shane [?] and Cisco, because DolphinScheduler can deploy on the cloud and it can handle a very big amount of workflows. It is also very easy to use, so you yourself or your team can use DolphinScheduler, not only the engineers. There are some resources for DolphinScheduler, so if you are interested, you can go to the Apache website, and you can also see some things on Medium or Twitter, something like that.

[00:22:15] So I've gone through the whole presentation. If you are interested in big data, or you're interested in the modern data stack, you can contact me on LinkedIn [?], or you can email me at this email, and you can also add my Twitter. That's all for my presentation, and I think the time is just okay for everyone. So do you have some questions about the data stack open source projects?

Q&A

[00:22:51] Host: Perfect, thank you very much. I really enjoyed that. It's a deep dive into the technical side that some people don't often see. People think it's all just at the front end, but you really need to move that data around at the back end. One thing I had a question on is related to security, which is coming up a lot more these days, and I've seen some CTOs starting to get a little bit nervous of open source. I don't know if you've heard the research from Socket Security. This is more about the JavaScript UI front end, but every module that you import, you're implicitly trusting 300 people, and a moderately sized one can have thousands of people. All it takes is one bad actor to introduce a security hole. So what are you doing to make sure that on every upgrade, and I know it's a free piece of software, how do you manage that kind of concern that people might have?

[00:24:03] William: Yeah, that is what the Apache Software Foundation is doing, because we have a whole process. For example, for Apache Software Foundation open source projects, we have a community. We have what we call a PMC, the project management committee. So it's not only one person or one company controlling this open source project. For example, for Hadoop there will be more than three or four companies, and I think more than 30 or 40 PMC members controlling Hadoop's progress.

[00:25:03] And when a new version is released, the security team of the Apache Foundation will check the whole thing. Sometimes there will be some leaks or some issues they did not catch, because every software has bugs or has issues. But we have a whole process to ensure that open source projects in the Apache Software Foundation are more secure than if you just download an open source project from nowhere. That is what the Apache Software Foundation is doing.

[00:25:51] Host: Excellent. That sounds really good, because for a while the goodwill and good nature of people worked, but unfortunately now the bad actors are coming in a bit more in force, particularly with the front-end JavaScript packages. Some of the Socket examples are really entertaining.

[00:26:13] William: Yeah, because those are also open source projects, but not Apache Software Foundation ones, so we cannot help with them.

[00:26:24] Host: All right. Well, thank you very much, William.

[00:26:28] William: Yep. Thank you, everyone, and thank you for listening.