Online learning
Kubeflow实战系列
Prepare
了解ksonnet初探Google之Kubeflow （采用的是0.8.0）install depinstall ksKubeflow 安利：在 Kubernetes 上进行机器学习ksonnet功能类似helm kubernetes应用的包管理工具。 可以对比存裸部署sparkKubeflow 入门
Jupyter notebook入门教程（上） - CSDN博客 Jupyter notebook教程 （more)
Docker 入门教程
最新的安装方式：
KSURL="https://github.com/kubeflow/kubeflow/releases/latest" LASTURL="" LASTURL=$(curl $KSURL -s -L -I -o /dev/null -w '%{url_effective}') if (test $? -ne 0); then LASTURL=$(wget -O /dev/null --content-disposition $KSURL 2>&1 |awk '/^Location: /{print $2}') if [ $? -ne 0 ]; then echo "Error: Can not find the latest ksonnet version by wget." fi fi LATES=${LASTURL##*/} echo $LATES mkdir kubeflow_$LATES cd kubeflow_$LATES export KUBEFLOW_TAG=$LATES curl https://raw.githubusercontent.com/kubeflow/kubeflow/${KUBEFLOW_TAG}/scripts/download.sh | bash
echo "Please run:"
echo "export KUBEFLOW_SRC=`pwd`"
echo "export KFAPP=~/kfapp"
cat > `pwd`/kubeflow.env <<EOL
export KUBEFLOW_TAG=$LATES
export KUBEFLOW_SRC=`pwd`
export KFAPP=~/kfapp
EOL
echo "Please run the commnad before doing any kubeflow setting up:"
echo "source `pwd`/kubeflow.env"
------------------------------------------------
以下是陈旧的安装方式
源代码安装ksonnet脚本（ksonnet的变化很大， 经常出问题。 不建议源码安装）
go get -u github.com/golang/dep/cmd/dep go get github.com/ksonnet/ksonnet # Build and install binary under shortname `ks` into $GOPATH/bin cd $GOPATH/src/github.com/ksonnet/ksonnet sudo -E -H make install export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig
安装最最新发布版的ksonnet
KSURL="https://github.com/ksonnet/ksonnet/releases/latest" LASTURL="" LASTURL=$(curl $KSURL -s -L -I -o /dev/null -w '%{url_effective}') if (test $? -ne 0); then LASTURL=$(wget -O /dev/null --content-disposition $KSURL 2>&1 |awk '/^Location: /{print $2}') if [ $? -ne 0 ]; then echo "Error: Can not find the latest ksonnet version by wget." fi fi LATES="" if [ "$LASTURL" = "$KSURL" ]; then echo "Error: Not find the latest ksonnet version by curl." else if test -n "$LASTURL"; then echo "OK: Find the latest ksonnet version." else echo "Error: Not find the latest ksonnet version by wget." fi fi LATES=${LASTURL##*/} if [[ $LATES =~ ^v[0-9.]{4,}$ ]]; then LATES=${LATES#v} fi wget https://github.com/ksonnet/ksonnet/releases/download/v$LATES/ks_${LATES}_linux_amd64.tar.gz
GOROOT=/usr/local
tar -C $GOROOT/bin -xzf ks_${LATES}_linux_amd64.tar.gz --wildcards --no-anchored 'ks' ln -s $GOROOT/bin/ks_${LATES}_linux_amd64/ks $GOROOT/bin/ks # tar --extract --file=ks_0.11.0_linux_amd64.tar.gz ks_0.11.0_linux_amd64/ks
Using Kubeflow （翻译） (中文) （latest EN)
(中文简要介绍)
ks init my-kubeflow sudo chown $USER:$USER ~/my-kubeflow -R
cd my-kubeflow ks registry add kubeflow github.com/google/kubeflow/tree/master/kubeflow ks pkg install kubeflow/core ks pkg install kubeflow/tf-serving ks pkg install kubeflow/tf-job ks generate core kubeflow-core --name=kubeflow-core
check kubectl contexts:
kubectl config view apiVersion: v1 clusters: - cluster: certificate-authority-data: REDACTED server: https://localhost:6443/ name: local-up-cluster contexts: - context: cluster: local-up-cluster user: local-up-cluster name: local-up-cluster current-context: local-up-cluster kind: Config preferences: {} users: - name: local-up-cluster user: client-certificate-data: REDACTED client-key-data: REDACTED
Jupyter启动之后运行tf的example
Troubleshooting
1.
$ sudo -E -H ks pkg install kubeflow/core ERROR GET https://api.github.com/repos/google/kubeflow/commits/master: 403 API rate limit exceeded for 134.134.139.74. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) [rate reset in 26m21s]
403 API rate limit of 60 still exceeded
GTOKEN=801c4f41553571107afa06d459c0910900aeb5fb
GTOKEN=5d67c71ea35d85a5459e6e05a3075363d438b8c7
export GITHUB_TOKEN=$GTOKEN
echo "GITHUB_TOKEN=$GTOKEN" >> ~/.profile
2. Can not ks apply default -c kubeflow-core with ks 0.9.5
rollback to v0.8.0
@seekyiyi I have the same problem using v0.9.1(or dev build) version of ksonnet, but I use v0.8.0 can be solved.
centos 下安装go
git clone https://github.com/jamesarems/gocentos.git cd gocentos sh InstallGO.sh source /etc/profile go get -u github.com/golang/dep/cmd/dep echo 'GOROOT=`go env |grep "GOROOT" |cut -d "=" -f2` GOROOT=${GOROOT#"} GOROOT=${GOROOT%"} echo $GOROOT GOPATH=`go env |grep GOPATH |cut -d "=" -f 2` GOPATH=${GOPATH%"} GOPATH=${GOPATH#"} echo $GOPATH declare -x GOROOT declare -x GOPATH export PATH=${PATH}:${GOROOT}/bin:${GOPATH}/bin' >> ~/.bashrc source ~/.bashrc echo $GOPATH go get github.com/ksonnet/ksonnet cd $GOPATH/src/github.com/ksonnet/ksonnet make install export KUBECONFIG=/root/config ks init my-kubeflow
wget -O /dev/null --content-disposition https://github.com/ksonnet/ksonnet/releases/latest --server-response 2>&1 | awk '(NR==1){SRC=$3;} /^ Location: /{DEST=$2} END{ print SRC, DEST}' https://github.com/ksonnet/ksonnet/releases/latest https://github.com/ksonnet/ksonnet/releases/tag/v0.11.0
setup local registry （git hub)
Sharing a local registry with minikube(推荐)
Kubernetes镜像仓库-Harbor的Helm部署
在kubernetes 集群上搭建docker 私有仓库Harbor
Get latest Github app version:
# cat github-latest.sh
#!/bin/bash URL=$1 APPURL=${URL%/releases/latest*} APP=${APPURL##*/} LASTURL="" LASTURL=$(curl $URL -s -L -I -o /dev/null -w '%{url_effective}') if (test $? -ne 0); then LASTURL=$(wget -O /dev/null --content-disposition $URL 2>&1 |awk '/^Location: /{print $2}') if [ $? -ne 0 ]; then echo "Error: Can not find the latest $APP version by wget." fi fi LATES="" if [ "$LASTURL" = "$URL" ]; then echo "Error: Not find the latest $APP version by curl." else if test -n "$LASTURL"; then echo "OK: Find the latest $APP version." else echo "Error: Not find the latest $APP version by wget." fi fi LATES=${LASTURL##*/} if [[ $LATES =~ ^v[0-9.]{4,}$ ]]; then LATES=${LATES#v} fi echo "Version:$LATES" exit 0
URL=https://github.com/goharbor/harbor/releases/latest V=`./github-latest.sh $URL|grep Version: |cut -d ":" -f2` BIN=https://storage.googleapis.com/harbor-releases/harbor-offline-installer-v$V.tgz wget $BIN