The vision of Submarine

From the Submarine Github home page:

Apache Submarine (Submarine for short) is the ONE PLATFORM to allow Data Scientists to create end-to-end machine learning workflow. ONE PLATFORM means it supports Data Scientists to finish their jobs on the same platform without frequently switching their toolsets. From dataset exploring data pipeline creation, model training (experiments), and push model to production (model serving and monitoring). All these steps can be completed within the ONE PLATFORM.

Comparing to other open-source projects, Submarine is targeted to bring best-of-breed User-Interface (API/SDK/GUI, etc.) to run training workload at scale, repeatable, easy to use which doesn’t require Data-Scientist to understand implementations behind the scene (such as how K8s works).

Look back, what we have done

Submarine was part of Hadoop subprojects (in parallel with HDFS/YARN/OZone, etc.). It spun off from Hadoop and became an Apache TLP (Top Level Project) in Nov 2019.

The previous Submarine release is 0.3.0 (Release Note) which is released on Feb 2020. Till 0.3.0 release, Submarine is mainly focused on running deep learning and machine learning workload on Apache Hadoop YARN (YARN for short) platform.

It supports running distributed Tensorflow, PyTorch workloads on YARN together with other batch (like Spark/Tez/Map-Reduce), or streaming (like Flink, etc.) workloads on YARN. Internally, Submarine uses LinkedIn’s TonY project to run deep-learning applications on YARN.

By leveraging GPU/Docker support on YARN, Submarine also supports workloads request GPU, and run workloads inside Docker container.

What’s new for 0.4.0 — Overview

Starting from 0.4.0, we will focus to provide a full training/model-management/notebook experiences for end-users. We will also start to focus on Kubernetes since Kubernetes is becoming another choice to run a machine-learning/AI platform.

0.4.0 is the first step of our vision of ONE Platform. At a high level, we provide Python SDK and REST APIs in order to allow data scientists to run distributed/standalone Tensorflow/PyTorch training workloads (experiments in Submarine’s term) with a few commands.

Let’s look at the highlights of the 0.4.0 release.

What’s new for 0.4.0 — Highlights

Easy to deploy on Kubernetes

Submarine provides a one-click Helm chart for easy installation on K8s. This Helm chart consists of Submarine server, database, and other Submarine dependencies.

helm install submarine ./helm-charts/submarine

Submit distributed training experiments to K8s via Python SDK

More details please see: Jupyter example

submarine_client = submarine.ExperimentClient(host='http://submarine:8080')

experiment_meta = ExperimentMeta(name='mnist',
   namespace='submarine',
   framework='Tensorflow',
   cmd='python mnist.py'
   ' - log_dir=/train/log',
   ' - batch_size=150',
   env_vars={'ENV1': 'ENV1'})

worker_spec = ExperimentTaskSpec(resources='cpu=4,memory=2048M',
   replicas=2)

ps_spec = ExperimentTaskSpec(resources='cpu=4,memory=2048M',
   replicas=1)

env = Environment(
   image='gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0')
   experiment_spec = ExperimentSpec(meta=experiment_meta,
   environment=env,
   spec={'Worker': worker_spec,
         'Ps': ps_spec})

# Submit experiment to K8s 
submarine_client.create_experiment(experiment_spec=experiment_spec)

# Get experiment spec
id = experiment['experimentId']
submarine_client.get_experiment(id)

# List all running experiments
status = 'running'
submarine_client.list_experiments(status=status)

# Get experiment log
submarine_client.get_log(id)
Also include how to retrieve logs, status, etc.

Submit distributed training experiments to K8s via REST API

curl -X POST -H “Content-Type: application/json” -d ‘
{
 “meta”: {
 “name”: “tf-mnist-json”,
 “namespace”: “default”,
 “framework”: “TensorFlow”,
 “cmd”: “python /var/tf_mnist/mnist_with_summaries.py — log_dir=/train/log — learning_rate=0.01 — batch_size=150”,
 “envVars”: {
 “ENV_1”: “ENV1”
 }
 },
 “environment”: {
 “image”: “gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0”
 },
 “spec”: {
 “Ps”: {
 “replicas”: 1,
 “resources”: “cpu=1,memory=512M”
 },
 “Worker”: {
 “replicas”: 1,
 “resources”: “cpu=1,memory=512M”
 }
 }
}
‘ http://127.0.0.1:8080/api/v1/experiment

# Get experiment spec

curl -X GET http://127.0.0.1/api/v1/experiment/experiment_1592057447228_0001

# List all running experiments

curl -X GET http://127.0.0.1:8080/api/v1/experiment?status=running

# Get experiment log

curl -X GET http://127.0.0.1:8080/api/v1/experiment/logs/experiment_1592057447228_0001

What’s coming?

We’re planning to release the next version v0.5.0 by end of Auguest, 2020, the main features planned for 0.5.0 includes the following (tentative):

• Submarine SDK
• Environment API
• Pre-defined Experiment API to use out-of-box algorithms like deepFM
• Model management API
• Submarine UI
• Integration with Jupyter notebook

Here’re screenshots for 0.5.0 Submarine experiment management UI:

Community

Community Statistics

During the 0.4.0 release, we have 34 unique contributors (which is the largest number of contributors for all Submarine releases) that have PR merged or reported JIRAs. And we have a total of 172 JIRAs resolved in community.

0.4.0 also includes 51,392 lines of code additions and 7,797 lines of deletions.

Acknowledgments

Thanks all the contributors who have PRs or created JIRAs:

Eroschang, Huang Ting Yao, Jasoonn, Lleohao, Qin Yao, SegFault6666, andrewhsiehth, arbaobao, cchung100m, chiajou, cku328, danielthank, flirmnave, gumartinm, jason_jane, jiwq, jotjohnting, khchen, khu, kobe860219, liushaohui, liuxun, lowc1012, pingsutw, snemeth, stana, tangzhankun, wangda, weichiu, xzh_dz, yuan_zac, zhulinhao, ztang

Without all of you, we cannot get the release done!

Have an interest in joining the Apache Submarine community? Please follow https://github.com/apache/submarine/blob/master/docs/community/contributing.md to become a contributor!