Build PAI Containers from Source

This document describes hot to build PAI containers from source and deploy the newly-built image. You can change the source code, and apply the code change to your cluster by following these steps.

Prepare the Config in a Dev-box Container

To build the PAI cluster from source, please first follow our installation guide to deploy an OpenPAI cluster, and follow this document to launch a dev-box container. Please use sudo docker exec -it <name-of-your-dev-box-container> bash to go into the dev-box container.

In the dev-box container, use the following config to pull the cluster config to a config folder.

./paictl.py config pull -o <config-folder>

You can choose a suitable config folder by your own.

Modify the Cluster Config

Use an editor to open <config-folder>/services-configuration.yaml. In services-configuration.yaml, the following config decides the docker images that OpenPAI will use in cluster:

• cluster.docker-registry.domain: The registry which stores the image. Default value is the official docker registry docker.io.
• cluster.docker-registry.username: The username of the registry.
• cluster.docker-registry.password: The password of the username.
• cluster.docker-registry.tag: The tag of the images. All images use the same tag.
• cluster.docker-registry.namespace: The common prefix string used in all images. Default value is openpai.

By default, OpenPAI will use <namespace>/<docker-image-name-of-each-service>:<tag> as the image name. For example, if you have deployed OpenPAI v1.3.0, it will use images like openpai/webportal:v1.3.0, openpai/rest-server:v1.3.0, ..., etc.

To build and use your own images, you should first modify the config. Then build and push your images. Finally restart the cluster. Your images must be stored in one registry so that OpenPAI will be able to use them. If you also want to use the official docker.io to store your images, please modify services-configuration.yaml as follows:

• cluster.docker-registry.domain: Remains the same.
• cluster.docker-registry.username: Uncomment the corresponding config line, and change it to a valid docker.io username.
• cluster.docker-registry.password: Uncomment the corresponding config line, and change it to the password of the username.
• cluster.docker-registry.tag: Change it to a casual string. All images will share one tag.
• cluster.docker-registry.namespace: Your can change it to the docker.io username. Or you can create an organization in DockerHub, and change it to the organization name.

If your network to docker.io is slow, you can also set up a local registry. You can run sudo docker run -d -p 5000:5000 --name registry registry:2 on a local machine, then use <registry-machine-ip>:5000 as the registry. Please notice the registry you set up doesn't support https by default, thus Docker will refuse to pull images from it. Please edit docker's daemon.json to add <registry-machine-ip>:5000 as a known insecure registry, and restart docker of all machines in the cluster.

To use a local registry, change services-configuration.yaml as follows:

• cluster.docker-registry.domain: Please change it to <registry-machine-ip>:5000.
• cluster.docker-registry.username: No need to change.
• cluster.docker-registry.password: No need to change.
• cluster.docker-registry.tag: Change it to a casual string. All images will share one tag.
• cluster.docker-registry.namespace: No need to change.

Build and Push Images

After the modification of services-configuration.yaml, you can build and push the image. Make sure you are using the code you modified or have changed to your branch:

./build/pai_build.py build -c <config-folder>
./build/pai_build.py push -c <config-folder>

Restart PAI Service

First, stop all PAI services:

./paictl.py service stop

Then, push the modifed config to the cluster. Only pushed config will take effect in cluster deployment:

./paictl.py config push -p <config-folder> -m service

Finally, start all PAI services:

./paictl.py service start

Update One Service

Sometimes, you are only modifying one PAI service, thus updating containers of this service is enough. But we recommend you to build all images and restart all services at least once in the beginning. The reason is that: PAI services share one docker registry, one image namespace and one tag, and all images should be present. For example, in a newly-installed v1.3.0 cluster, the images used are openpai/webportal:v1.3.0, openpai/rest-server:v1.3.0, ..., etc. If Alice changes the cluster config to use her docker registry and tag, like alice/<image-name>:alice-test, and only build image alice/webportal:alice-test, other images like alice/rest-server:alice-test will be missing, which will cause the cluster to behave abnormally.

To conclude, if you are going to update one service, please first follow the instructions above to build all images and restart all services. In the following time, you can build the containers in that service only and use the same tag as before. Please refer to the following commands:

./build/pai_build.py build -c  <config-folder> -s <service-name>
./build/pai_build.py push -c <config-folder> -i <container-name-1> <container-name-2> ...
./paictl.py service stop -n <service-name>
./paictl.py service start -n <service-name>

One service may contain multiple containers, you can read the log produced by ./build/pai_build.py build to figure out the container names.

For example, if you are modifying webportal, you can only build / push webportal container, and restart webportal service:

./build/pai_build.py build -c  <config-folder> -s webportal
./build/pai_build.py push -c <config-folder> -i webportal
./paictl.py service stop -n webportal
./paictl.py service start -n webportal