Seldon Core Installation on Kubernetes (OpenShift)

AI & Data Engineering

6 min readJun 30, 2021

In this post, we will explore the installation of Seldon Core (the latest stable) with OpenShift Kubernetes.

Seldon Core is an open-source platform for deploying machine learning models on a Kubernetes cluster.

Pre-requisites:

Kubernetes cluster ≥ 1.12. For Openshift it requires version 4.2 or higher
Installer method: Helm ≥ 3.0
Ingress: Istio & Ambassador

Let's first understand in brief, what is HELM and Ingress like Istio, and Ambassador.

HELM

In short, HELM is a package manager for Kubernetes. It is just like “yum” or “Maven” or “apt”. You can find packages in ArtifactHub https://artifacthub.io/

Ingress

Ingress is the incoming traffic from the outside world into the cluster.

Ambassador is a Kubernetes ingress controller/API gateway that uses Envoy. And Istio is a service mesh that happens to have its own ingress mechanism. Istio also uses Envoy as sidecar proxies. We will discuss in a separate post about service mesh and sidecar patterns.

Installation Steps

Step1: Install HELM

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Step2: Install Istio

Download the Istio release

curl -L https://istio.io/downloadIstio | sh -

2. Change the directory and get into the installed root directory of Istio

cd istio-1.10.2
export PATH=$PWD/bin:$PATH
ls

3. By default, OpenShift doesn’t allow containers running with user ID 0. You must enable containers running with UID 0 for Istio’s service accounts by running the command below. Here, we are assuming that we will maintain the default Istio namespace: “istio-system”

$ oc adm policy add-scc-to-group anyuid system:serviceaccounts:istio-system

4. Install Istio. Here, use the profile as openshift

$ istioctl install --set profile=openshift

Step3: Install Seldon

Create namespaces for Seldon

$ kubectl create namespace seldon
$ kubectl create namespace seldon-system

2. Apply the Security context constraints (scc) on both Seldon namespaces: seldon and seldon-system

$ oc adm policy add-scc-to-group anyuid system:serviceaccounts:seldon-system$ oc adm policy add-scc-to-group anyuid system:serviceaccounts:seldon

3. Update the kubeconfig for the seldon namespace

kubectl config set-context $(kubectl config current-context) --namespace=seldon

4. Install seldon-core using HELM. Here, I am enabling both Istio and Ambassador

helm install seldon-core seldon-core-operator \
 --repo https://storage.googleapis.com/seldon-charts \
 --set ambassador.enabled=true \
 --set istio.enabled=true \
 --set usageMetrics.enabled=true \
 --namespace seldon-system

5. Check the seldon-core deployment status

kubectl get deployment -n seldon-system

6. We need an istio gateway installed in the istio-system namespace. By default, we assume one called seldon-gateway.

Let's create a YAML file with the name seldon-gateway.yaml and paste the below content as mentioned here: https://github.com/SeldonIO/seldon-core/blob/master/doc/source/ingress/istio.md

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: seldon-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"

Apply the file for deployment using kubectl

kubectl create -f seldon-gateway.yaml

We can get the status of the deployment using kubectl rollout.

kubectl rollout status deploy/seldon-controller-manager -n seldon-system

Step4: Install Ambassador

Add HELM repo for Ambassador

helm repo add datawire https://www.getambassador.io
helm repo update

helm repo add datawire https://www.getambassador.io
helm repo update

2. Installing ambassador

helm install ambassador datawire/ambassador \
 --set image.repository=quay.io/datawire/ambassador \
 --set enableAES=false \
 --set crds.keep=false \
 --namespace seldon

3. Check the deployment status

kubectl rollout status deployment.apps/ambassador -n seldon

4. Check the final status of deployment under seldon and seldon-system

kubectl get deployments -n seldon
kubectl get deployments -n seldon-system
kubectl get pods -n seldon-system
kubectl get pods -n seldon
kubectl get pods -n istio-system

5. The below command will forward traffic from the local 8003 port to port 8080 of the Ambassador ingress gateway pod within Kubernetes. Ambassador will then route this traffic to your Seldon deployments.

kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080

This listener will wait for ingress traffic to forward on from port 8003 to the pod port of 8080

Step5: ML Model Deployment

Define the configuration of the SeldonDeployment, which will orchestrate the serving of your model. Here, we are taking an example of “Scikitlearn IRIS” ML model deployment.

The yaml file looks as below:

The deployment file will download sklearn-iris image for 0.1 version.The imagePullPolicy is “IfNotPresent” and name of the container is“sklearn-iris-classifier”. The endpoint is “REST”, type is MODEL and replica is 1.

apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  name: seldon-deployment-example
spec:
  name: sklearn-iris-deployment
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/sklearn-iris:0.1
          imagePullPolicy: IfNotPresent
          name: sklearn-iris-classifier
    graph:
      children: []
      endpoint:
        type: REST
      name: sklearn-iris-classifier
      type: MODEL
    name: sklearn-iris-predictor
    replicas: 1

Apply the yaml file for the deployment using kubectl create

kubectl create -f sklearn_iris_deployment.yaml

Check the deployment using “rollout” command. Please note that there is a seldon-xxx deployment and a pod seldon-xxx running under seldon namespace.

It shows, deployment successfully rolled out.

Step6: Test the Model deployment

You can test the deployment by throwing some features to the REST endpoint

The classifier expects an array of the form [Sepal Length, Sepal Width, Petal Length, Petal Width]. You can implement this simply using curl by running the following command:

curl -s http://localhost:8003/seldon/seldon/seldon-deployment-example/api/v0.1/predictions -H "Content-Type: application/json" -d '{"data":{"ndarray":[[5.964,4.006,2.081,1.031]]}}'

This should return a response that confirms the classification of your iris alongside confidence scoring

e.g. {"data":{"names":["t:0","t:1","t:2"],"ndarray":[[0.9548873249364169,0.04505474761561406,5.7927447968952436e-05]]},"meta":{}}

This example is suggesting that the iris is a Setosa with 95% confidence

So, that it!

In summary, we have seen

End-to-end deployment of Seldon Core
Deployment of Ingress like Istio and Ambassador
Deployment of ScikitLearn ML model in Seldon Core
Finally, testing the model

Referred Links

Seldon Core Installation on Kubernetes (OpenShift)

Installation Steps

Written by AI & Data Engineering