PaddlePaddle On Kubernetes

In this article, we will introduce how to run PaddlePaddle training job on single CPU machine using Kubernetes. In next article, we will introduce how to run PaddlePaddle training job on distributed cluster.

Build Docker Image

In distributed Kubernetes cluster, we will use Ceph or other distributed storage system for storing training related data so that all processes in PaddlePaddle training can retrieve data from Ceph. In this example, we will only demo training job on single machine. In order to simplify the requirement of the environment, we will directly put training data into the PaddlePaddle Docker Image, so we need to create a PaddlePaddle Docker image that includes the training data.

The production Docker Image paddlepaddle/paddle:cpu-demo-latest has the PaddlePaddle source code and demo. (Caution: Default PaddlePaddle Docker Image paddlepaddle/paddle:latest doesn't include the source code, PaddlePaddle's different versions of Docker Image can be referred here: Docker Installation Guide), so we run this Docker Image and download the training data, and then commit the whole Container to be a new Docker Image.

Run Docker Container

$ docker run --name quick_start_data -it paddlepaddle/paddle:cpu-demo-latest

Download Training Data

Getting into /root/paddle/demo/quick_start/data Directory,using get_data.sh to download training data. Then getting into /root/paddle/demo/quick_start Directory, using preprocess.sh to pre-process training data.

$ root@fbd1f2bb71f4:~/paddle/demo/quick_start/data# ./get_data.sh

Downloading Amazon Electronics reviews data...
--2016-10-31 01:33:43--  http://snap.stanford.edu/data/amazon/productGraph/categoryFiles/reviews_Electronics_5.json.gz
Resolving snap.stanford.edu (snap.stanford.edu)... 171.64.75.80
Connecting to snap.stanford.edu (snap.stanford.edu)|171.64.75.80|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 495854086 (473M) [application/x-gzip]
Saving to: 'reviews_Electronics_5.json.gz'

 10% [=======>                                         ] 874,279     64.7KB/s  eta 2h 13m

Modify Startup Script

After downloading the data,modify /root/paddle/demo/quick_start/train.sh file contents are as follows (one more cd cmd):

set -e
cd /root/paddle/demo/quick_start
cfg=trainer_config.lr.py
#cfg=trainer_config.emb.py
#cfg=trainer_config.cnn.py
#cfg=trainer_config.lstm.py
#cfg=trainer_config.bidi-lstm.py
#cfg=trainer_config.db-lstm.py
paddle train \
  --config=$cfg \
  --save_dir=./output \
  --trainer_count=4 \
  --log_period=20 \
  --num_passes=15 \
  --use_gpu=false \
  --show_parameter_stats_period=100 \
  --test_all_data_in_one_period=1 \
  2>&1 | tee 'train.log'

Commit Docker Image

$ docker commit quick_start_data mypaddle/paddle:quickstart

Use Kubernetes For Training

We will use Kubernetes job for training process, following steps shows how to do the training with Kubernetes.

Create Yaml Files

The output result in container will be demolished when job finished (container stopped running), so we need to mount the volume out to the local disk when creating the container to store the training result. Using our previously created image, we can create a Kubernetes Job, the yaml contents are as follows:

apiVersion: batch/v1
kind: Job
metadata:
  name: quickstart
spec:
  parallelism: 1
  completions: 1
  template:
    metadata:
      name: quickstart
    spec:
      volumes:
      - name: output
        hostPath: 
          path: /home/work/paddle_output     
      containers:
      - name: pi
        image: mypaddle/paddle:quickstart
        command: ["bin/bash",  "-c", "/root/paddle/demo/quick_start/train.sh"]
        volumeMounts:
        - name: output
          mountPath: /root/paddle/demo/quick_start/output
      restartPolicy: Never

Start PaddlePaddle Job

Using the above yaml file to start the Kubernetes job.

$ kubectl  create -f paddle.yaml

Get the detailed status of the job:

$ kubectl  get job
NAME         DESIRED   SUCCESSFUL   AGE
quickstart   1         0            58s

$ kubectl  describe job quickstart
Name:       quickstart
Namespace:  default
Image(s):   registry.baidu.com/public/paddle:cpu-demo-latest
Selector:   controller-uid=f120da72-9f18-11e6-b363-448a5b355b84
Parallelism:    1
Completions:    1
Start Time: Mon, 31 Oct 2016 11:20:16 +0800
Labels:     controller-uid=f120da72-9f18-11e6-b363-448a5b355b84,job-name=quickstart
Pods Statuses:  0 Running / 1 Succeeded / 0 Failed
Volumes:
  output:
    Type:   HostPath (bare host directory volume)
    Path:   /home/work/paddle_output
Events:
  FirstSeen LastSeen    Count   From            SubobjectPath   Type        Reason          Message
  --------- --------    -----   ----            -------------   --------    ------          -------
  1m        1m      1   {job-controller }           Normal      SuccessfulCreate    Created pod: quickstart-fa0wx

Get Training Result

We can use kubectl command to take a look at the status of related pod.

$ kubectl  describe pod quickstart-fa0wx
Name:       quickstart-fa0wx
Namespace:  default
Node:       paddle-demo-let02/10.206.202.44
Start Time: Mon, 31 Oct 2016 11:20:17 +0800
Labels:     controller-uid=f120da72-9f18-11e6-b363-448a5b355b84,job-name=quickstart
Status:     Succeeded
IP:     10.0.0.9
Controllers:    Job/quickstart
Containers:
  quickstart:
    Container ID:   docker://b8561f5c79193550d64fa47418a9e67ebdd71546186e840f88de5026b8097465
    Image:      registry.baidu.com/public/paddle:cpu-demo-latest
    Image ID:       docker://18e457ce3d362ff5f3febf8e7f85ffec852f70f3b629add10aed84f930a68750
    Port:
    Command:
      bin/bash
      -c
      /root/paddle/demo/quick_start/train.sh
    QoS Tier:
      cpu:      BestEffort
      memory:       BestEffort
    State:      Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 31 Oct 2016 11:20:20 +0800
      Finished:     Mon, 31 Oct 2016 11:21:46 +0800
    Ready:      False
    Restart Count:  0
    Environment Variables:
Conditions:
  Type      Status
  Ready     False
Volumes:
  output:
    Type:   HostPath (bare host directory volume)
    Path:   /home/work/paddle_output

We can also ssh to Kubernetes node to take a look at the training result.

[root@paddle-demo-let02 paddle_output]# ll
total 60
drwxr-xr-x 2 root root 4096 Oct 31 11:20 pass-00000
drwxr-xr-x 2 root root 4096 Oct 31 11:20 pass-00001
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00002
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00003
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00004
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00005
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00006
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00007
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00008
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00009
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00010
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00011
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00012
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00013
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00014