KubeCon + CloudNativeCon | Open Source Summit China 2019

JD.com serves 99% of China's consumers, we have always had to innovate on infrastructure in order to meet the demands of scale and speed.
In 2017, we adopted Vitess to help us scale MySQL. Two years on, JD.com now operates the world' largest Vitess on Kubernetes deployments. In this presentation, I will introduce how we use Vitess in JD.com. I will also share my personal understanding of Vitess in the Chinese context. Finally, I present a demo to show how we fully exploit the value of Vitess.
My presentation will demonstrate:
1. The value that Vitess brings to JD.com: how we use Vitess to reduce costs on machines and development
2. Frequently encountered problems: as one of the world's largest Vitess users, we want to share the problems we have encountered with Vitess and how to solve them
3. Practical examples in my demo: how to quickly built our projects with reusable Vitess modules

Kubernetes support schedule workloads based on CPU and memory resource with node affinity, pod affinity and anti-affinity. This works very well for stateless workloads. For stateful workloads, especially big data workloads, scheduling compute close to data source can greatly boost performance, reliability and availability. However, in many cloud based storage systems, the data locality info is either unavailable or not exposed to container orchestra.

In this talk, we will first compare the data locality support from mainstream container attached storage for Kubernetes. Then we will introduce network topology support from Apache Hadoop Ozone and how to use it as locality aware container attached storage via Ozone CSI plugin for better workloads scheduling. Last, we will use Spark on K8s to demo the benefits of data locality aware scheduling with Apache Hadoop Ozone.

As companies grow their infrastructure on the cloud in Kubernetes, questions of high availability arise. To be truly cloud native, you must be able to handle failure at any point in your stack.

Vitess has many features that can help in failure identification and handling, such as vtgate-as-proxy and data recovery through duplication (instead of backup).

Vitess also works with many third party tools like Prometheus for easy monitoring and observability.

In this talk, PlanetScale cofounder and CEO Jiten Vaidya will present the case for planning for failure. Using a live demo, Jiten will show how designing for high availability using Vitess allows you to prepare for risk, avert disasters, and recover from catastrophic failure.

Rook was developed as a storage provider for Kubernetes to automatically deploy and attach storage to pods. Significant effort within Rook has been devoted to integrating the open-source storage platform Ceph with Kubernetes. Ceph is a distributed storage system in broad use today that presents unified file, block, and object interfaces to applications.

This talk will present completed work in the Ceph Nautilus release to dynamically create highly-available and scalable NFS server clusters that export the Ceph file system (CephFS) for use within Kubernetes or as a standalone appliance. CephFS provides applications with a friendly programmatic interface for creating shareable volumes. For each volume, Ceph and Rook cooperatively manage the details of dynamically deploying a cluster of NFS-Ganesha pods with minimal operator or user involvement.

Kubernetes not only becomes predominant in public cloud area these days, but also becomes a new trend in on-premises big data cluster environment, as an alternative of Hadoop YARN, a resource schedule component. In on-premise big data cluster, majority data are saved in HDFS. How to consume big data in HDFS with Kubernetes is a new challenge to users.
In the talk we will introduce our CSI compatible HDFS plugin design and architecture first. Then, we will share our best practices and knowledge about how big data workload Spark use HDFS CSI plugin to access HDFS data when running on K8s. In the end, the TPC-DS benchmark suite will be used to analysis performance comparison between Spark on K8s with HDFS and Spark on YARN with HDFS.