トップ > おうちkubernetes > おうちkubernetes（インストール編）

おうちkubernetes（インストール編）

おうちkubernetes 開発環境 開発 環境設定

takeiteasy-09.net takeiteasy-09.net

に続くものです。

kubernetesのインストール

f:id:akky97:20211024110744p:plain

github.com

上記の「証明書の生成」から実施していきます。

証明書の生成と転送

generate-cert.sh をダウンロードして実行します。

ubuntu@sapphire:~/kubernetes$ chmod +x generate-cert.sh 
ubuntu@sapphire:~/kubernetes$ ./generate-cert.sh 
Hostname of Node1: sapphire       
Hostname of Node2: opal
Hostname of Node3: peridot
Addresses of Node1 (x.x.x.x[,x.x.x.x]): 10.0.0.11,192.168.11.120
Addresses of Node2 (x.x.x.x[,x.x.x.x]): 10.0.0.12,192.168.11.121
Addresses of Node3 (x.x.x.x[,x.x.x.x]): 10.0.0.13,192.168.11.122
Address of Kubernetes ClusterIP (first address of ClusterIP subnet): 10.32.0.1
---> Generate CA certificate
:
:
---> Complete to generate certificate

処理が完了すると「cert」ディレクトリが作成されてその中に証明書一式が格納されています。 手順書にある通り、ca.pen と <hostname>.pem と <hostname>-key.pem は他のサーバへも転送します。

ubuntu@sapphire:~/kubernetes$ scp cert/{ca.pem,opal.pem,opal-key.pem} ubuntu@opal:/home/ubuntu/
ubuntu@opal's password: 
ca.pem                                                                            100% 1314     1.1MB/s   00:00    
opal.pem                                                                          100% 1480     1.4MB/s   00:00    
opal-key.pem                                                                      100% 1675     1.6MB/s   00:00    
ubuntu@sapphire:~/kubernetes$ scp cert/{ca.pem,peridot.pem,peridot-key.pem} ubuntu@peridot:/home/ubuntu/
ubuntu@peridot's password: 
ca.pem                                                                            100% 1314     1.2MB/s   00:00    
peridot.pem                                                                       100% 1489     1.4MB/s   00:00    
peridot-key.pem                                                                   100% 1679     1.6MB/s   00:00    
ubuntu@sapphire:~/kubernetes$

kubectlのインストール

手順の通りダウンロードとファイルの配置をします。

ubuntu@sapphire:~/kubernetes$ wget -q --show-progress --https-only --timestamping "https://storage.googleapis.com/kubernetes-release/release/v1.18.6/bin/linux/arm64/kubectl"
kubectl                      100%[==============================================>]  39.81M  11.8MB/s    in 3.4s    
ubuntu@sapphire:~/kubernetes$ chmod +x kubectl
ubuntu@sapphire:~/kubernetes$ sudo mv kubectl /usr/local/bin
ubuntu@sapphire:~/kubernetes$

バージョン確認します。

ubuntu@sapphire:~/kubernetes$ kubectl version --client
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/arm64"}
ubuntu@sapphire:~/kubernetes$

kubeconfigの生成と転送

generate-kubeconfig.sh をダウンロードして実行します。

ubuntu@sapphire:~/kubernetes$ chmod +x generate-kubeconfig.sh
ubuntu@sapphire:~/kubernetes$ ./generate-kubeconfig.sh 
Hostname of Node1: sapphire
Hostname of Node2: opal
Hostname of Node3: peridot
Address of Master Node: 10.0.0.11
---> Generate kubelet kubeconfig
Cluster "kubernetes-the-hard-way" set.
User "system:node:sapphire" set.
:
:
Context "default" created.
Switched to context "default".
---> Complete to generate kubeconfig
ubuntu@sapphire:~/kubernetes$

作成されたconfigファイルを転送します。

ubuntu@sapphire:~/kubernetes$ scp kubeconfig/{opal.kubeconfig,kube-proxy.kubeconfig} ubuntu@opal:/home/ubuntu/
ubuntu@opal's password: 
opal.kubeconfig                                                                   100% 6347     3.8MB/s   00:00    
kube-proxy.kubeconfig                                                             100% 6309     4.9MB/s   00:00    
ubuntu@sapphire:~/kubernetes$ scp kubeconfig/{peridot.kubeconfig,kube-proxy.kubeconfig} ubuntu@peridot:/home/ubuntu/
ubuntu@peridot's password: 
peridot.kubeconfig                                                                100% 6369     3.7MB/s   00:00    
kube-proxy.kubeconfig                                                             100% 6309     4.9MB/s   00:00    
ubuntu@sapphire:~/kubernetes$

etcdのブートストラップ

kubernetesで利用するデータの管理をするものとのことで、まずはこれをインストールします。

ubuntu@sapphire:~/kubernetes$ wget -q --show-progress --https-only --timestamping "https://github.com/etcd-io/etcd/releases/download/v3.4.13/etcd-v3.4.13-linux-arm64.tar.gz"
etcd-v3.4.13-linux-arm64.tar 100%[==============================================>]  15.36M  11.6MB/s    in 1.3s    
ubuntu@sapphire:~/kubernetes$ tar -xvf etcd-v3.4.13-linux-arm64.tar.gz
etcd-v3.4.13-linux-arm64/
etcd-v3.4.13-linux-arm64/etcdctl
etcd-v3.4.13-linux-arm64/README.md
etcd-v3.4.13-linux-arm64/etcd
:
:
etcd-v3.4.13-linux-arm64/README-etcdctl.md
etcd-v3.4.13-linux-arm64/READMEv2-etcdctl.md
ubuntu@sapphire:~/kubernetes$ sudo mv etcd-v3.4.13-linux-arm64/etcd* /usr/local/bin/
ubuntu@sapphire:~/kubernetes$

各種設定を行います。

ubuntu@sapphire:~/kubernetes$ sudo mkdir -p /etc/etcd /var/lib/etcd
ubuntu@sapphire:~/kubernetes$ sudo chmod 700 /var/lib/etcd
ubuntu@sapphire:~/kubernetes$ sudo cp cert/{ca.pem,kubernetes-key.pem,kubernetes.pem} /etc/etcd/
ubuntu@sapphire:~/kubernetes$ ll /etc/etcd/
total 20
drwxr-xr-x   2 root root 4096 Oct 22 18:04 ./
drwxr-xr-x 103 root root 4096 Oct 23 20:49 ../
-rw-r--r--   1 root root 1314 Oct 23 21:24 ca.pem
-rw-------   1 root root 1679 Oct 23 21:24 kubernetes-key.pem
-rw-r--r--   1 root root 1688 Oct 23 21:24 kubernetes.pem
ubuntu@sapphire:~/kubernetes$ 
ubuntu@sapphire:~/kubernetes$ ETCD_NAME=$(hostname -s)
ubuntu@sapphire:~/kubernetes$ INTERNAL_IP=10.0.0.11
ubuntu@sapphire:~/kubernetes$ cat <<EOF | sudo tee /etc/systemd/system/etcd.service
> [Unit]
> Description=etcd
> Documentation=https://github.com/coreos
> 
> [Service]
> Type=notify
> ExecStart=/usr/local/bin/etcd \\
>   --name ${ETCD_NAME} \\
>   --cert-file=/etc/etcd/kubernetes.pem \\
>   --key-file=/etc/etcd/kubernetes-key.pem \\
>   --peer-cert-file=/etc/etcd/kubernetes.pem \\
>   --peer-key-file=/etc/etcd/kubernetes-key.pem \\
>   --trusted-ca-file=/etc/etcd/ca.pem \\
>   --peer-trusted-ca-file=/etc/etcd/ca.pem \\
>   --peer-client-cert-auth \\
>   --client-cert-auth \\
>   --initial-advertise-peer-urls https://${INTERNAL_IP}:2380 \\
>   --listen-peer-urls https://${INTERNAL_IP}:2380 \\
>   --listen-client-urls https://${INTERNAL_IP}:2379,https://127.0.0.1:2379 \\
>   --advertise-client-urls https://${INTERNAL_IP}:2379 \\
>   --initial-cluster-token etcd-initial-token \\
>   --initial-cluster ${ETCD_NAME}=https://${INTERNAL_IP}:2380 \\
>   --initial-cluster-state new \\
>   --data-dir=/var/lib/etcd
> Restart=on-failure
> RestartSec=5
> Environment=ETCD_UNSUPPORTED_ARCH=arm64
> 
> [Install]
> WantedBy=multi-user.target
> EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \
  --name sapphire \
  --cert-file=/etc/etcd/kubernetes.pem \
  --key-file=/etc/etcd/kubernetes-key.pem \
  --peer-cert-file=/etc/etcd/kubernetes.pem \
  --peer-key-file=/etc/etcd/kubernetes-key.pem \
  --trusted-ca-file=/etc/etcd/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth \
  --initial-advertise-peer-urls https://10.0.0.11:2380 \
  --listen-peer-urls https://10.0.0.11:2380 \
  --listen-client-urls https://10.0.0.11:2379,https://127.0.0.1:2379 \
  --advertise-client-urls https://10.0.0.11:2379 \
  --initial-cluster-token etcd-initial-token \
  --initial-cluster sapphire=https://10.0.0.11:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
Environment=ETCD_UNSUPPORTED_ARCH=arm64

[Install]
WantedBy=multi-user.target
ubuntu@sapphire:~/kubernetes$

起動します。

ubuntu@sapphire:~/kubernetes$ sudo systemctl daemon-reload
ubuntu@sapphire:~/kubernetes$ sudo systemctl enable etcd
Created symlink /etc/systemd/system/multi-user.target.wants/etcd.service → /etc/systemd/system/etcd.service.
ubuntu@sapphire:~/kubernetes$ sudo systemctl start etcd
ubuntu@sapphire:~/kubernetes$ sudo systemctl status etcd
● etcd.service - etcd
     Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-10-23 21:29:55 JST; 9s ago
       Docs: https://github.com/coreos
   Main PID: 3079 (etcd)
      Tasks: 16 (limit: 9257)
     CGroup: /system.slice/etcd.service
             └─3079 /usr/local/bin/etcd --name sapphire --cert-file=/etc/etcd/kubernetes.pem --key-file=/etc/etcd/k>

手順の結果が出力されるか確認します。

ubuntu@sapphire:~/kubernetes$ sudo ETCDCTL_API=3 etcdctl member list \
>   --endpoints=https://127.0.0.1:2379 \
>   --cacert=/etc/etcd/ca.pem \
>   --cert=/etc/etcd/kubernetes.pem \
>   --key=/etc/etcd/kubernetes-key.pem
e67187a477e79e67, started, sapphire, https://10.0.0.11:2380, https://10.0.0.11:2379, false
ubuntu@sapphire:~/kubernetes$

kube-apiserver のインストール

kube-apiserver は Kubernetes の中核を担うコンポーネントとのこと。まだ何のことなのか理解できていませんがインストールを続けていきます。

バイナリのダウンロードと配置を行います。

ubuntu@sapphire:~/kubernetes$ wget -q --show-progress --https-only --timestamping "https://storage.googleapis.com/kubernetes-release/release/v1.18.6/bin/linux/arm64/kube-apiserver"
kube-apiserver               100%[==============================================>] 108.75M  12.2MB/s    in 9.1s    
ubuntu@sapphire:~/kubernetes$ chmod +x kube-apiserver
ubuntu@sapphire:~/kubernetes$ sudo mv kube-apiserver /usr/local/bin/
ubuntu@sapphire:~/kubernetes$

データ暗号化のための設定を実施します。

ubuntu@sapphire:~/kubernetes$ ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
ubuntu@sapphire:~/kubernetes$ cat > encryption-config.yaml <<EOF
> kind: EncryptionConfig
> apiVersion: v1
> resources:
>   - resources:
>       - secrets
>     providers:
>       - aescbc:
>           keys:
>             - name: key1
>               secret: ${ENCRYPTION_KEY}
>       - identity: {}
> EOF
ubuntu@sapphire:~/kubernetes$

設定に必要なフォルダ作成や証明書を配置します。

ubuntu@sapphire:~/kubernetes$ sudo mkdir -p /etc/kubernetes/config
ubuntu@sapphire:~/kubernetes$ sudo mkdir -p /var/lib/kubernetes/
ubuntu@sapphire:~/kubernetes$ sudo cp -ai cert/{ca.pem,ca-key.pem,kubernetes-key.pem,kubernetes.pem,service-account-key.pem,service-account.pem} /var/lib/kubernetes/
ubuntu@sapphire:~/kubernetes$ sudo cp -ai encryption-config.yaml /var/lib/kubernetes/
ubuntu@sapphire:~/kubernetes$ ll /var/lib/kubernetes/
total 52
drwxr-xr-x  2 root   root   4096 Oct 23 10:46 ./
drwxr-xr-x 39 root   root   4096 Oct 23 12:02 ../
-rw-------  1 ubuntu ubuntu 1675 Oct 23 21:00 ca-key.pem
-rw-rw-r--  1 ubuntu ubuntu 1314 Oct 23 21:00 ca.pem
-rw-rw-r--  1 ubuntu ubuntu  240 Oct 23 21:34 encryption-config.yaml
-rw-------  1 ubuntu ubuntu 1679 Oct 23 21:02 kubernetes-key.pem
-rw-rw-r--  1 ubuntu ubuntu 1688 Oct 23 21:02 kubernetes.pem
-rw-------  1 ubuntu ubuntu 1679 Oct 23 21:02 service-account-key.pem
-rw-rw-r--  1 ubuntu ubuntu 1436 Oct 23 21:02 service-account.pem

kube-apiserver を動かすためのユニットファイルを作成します。

ubuntu@sapphire:~/kubernetes$ INTERNAL_IP=10.0.0.11
ubuntu@sapphire:~/kubernetes$ CLUSTER_IP_NETWORK=10.32.0.0/24
ubuntu@sapphire:~/kubernetes$ cat <<EOF | sudo tee /etc/systemd/system/kube-apiserver.service
> [Unit]
> Description=Kubernetes API Server
> Documentation=https://github.com/kubernetes/kubernetes
> 
> [Service]
> ExecStart=/usr/local/bin/kube-apiserver \\
>   --advertise-address=${INTERNAL_IP} \\
>   --allow-privileged=true \\
>   --apiserver-count=3 \\
>   --audit-log-maxage=30 \\
>   --audit-log-maxbackup=3 \\
>   --audit-log-maxsize=100 \\
>   --audit-log-path=/var/log/audit.log \\
>   --authorization-mode=Node,RBAC \\
>   --bind-address=0.0.0.0 \\
>   --client-ca-file=/var/lib/kubernetes/ca.pem \\
>   --enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \\
>   --etcd-cafile=/var/lib/kubernetes/ca.pem \\
>   --etcd-certfile=/var/lib/kubernetes/kubernetes.pem \\
>   --etcd-keyfile=/var/lib/kubernetes/kubernetes-key.pem \\
>   --etcd-servers=https://${INTERNAL_IP}:2379 \\
>   --event-ttl=1h \\
>   --encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \\
>   --kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \\
>   --kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem \\
>   --kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem \\
>   --kubelet-https=true \\
>   --runtime-config='api/all=true' \\
>   --service-account-key-file=/var/lib/kubernetes/service-account.pem \\
>   --service-cluster-ip-range=${CLUSTER_IP_NETWORK} \\
>   --service-node-port-range=30000-32767 \\
>   --tls-cert-file=/var/lib/kubernetes/kubernetes.pem \\
>   --tls-private-key-file=/var/lib/kubernetes/kubernetes-key.pem \\
>   --v=2
> Restart=on-failure
> RestartSec=5
> 
> [Install]
> WantedBy=multi-user.target
> EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
  --advertise-address=10.0.0.11 \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/audit.log \
  --authorization-mode=Node,RBAC \
  --bind-address=0.0.0.0 \
  --client-ca-file=/var/lib/kubernetes/ca.pem \
  --enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --etcd-cafile=/var/lib/kubernetes/ca.pem \
  --etcd-certfile=/var/lib/kubernetes/kubernetes.pem \
  --etcd-keyfile=/var/lib/kubernetes/kubernetes-key.pem \
  --etcd-servers=https://10.0.0.11:2379 \
  --event-ttl=1h \
  --encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \
  --kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \
  --kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem \
  --kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem \
  --kubelet-https=true \
  --runtime-config='api/all=true' \
  --service-account-key-file=/var/lib/kubernetes/service-account.pem \
  --service-cluster-ip-range=10.32.0.0/24 \
  --service-node-port-range=30000-32767 \
  --tls-cert-file=/var/lib/kubernetes/kubernetes.pem \
  --tls-private-key-file=/var/lib/kubernetes/kubernetes-key.pem \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
ubuntu@sapphire:~/kubernetes$

続けて起動します。

ubuntu@sapphire:~/kubernetes$ sudo systemctl daemon-reload
ubuntu@sapphire:~/kubernetes$ sudo systemctl enable kube-apiserver
Created symlink /etc/systemd/system/multi-user.target.wants/kube-apiserver.service → /etc/systemd/system/kube-apiserver.service.
ubuntu@sapphire:~/kubernetes$ sudo systemctl start kube-apiserver
ubuntu@sapphire:~/kubernetes$ sudo systemctl status kube-apiserver
● kube-apiserver.service - Kubernetes API Server
     Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-10-23 21:39:43 JST; 7s ago
       Docs: https://github.com/kubernetes/kubernetes
   Main PID: 3209 (kube-apiserver)
      Tasks: 14 (limit: 9257)
     CGroup: /system.slice/kube-apiserver.service
             └─3209 /usr/local/bin/kube-apiserver --advertise-address=10.0.0.11 --allow-privileged=true --apiserver>...

kube-controller-manager のインストール

kube-controller-manager は Kubernetes におけるリソース管理などのコントローラー類を束ねたコンポーネントとのこと。例によってまだ理解はできていません…

詳しくは後に勉強することとして、まずはバイナリのダウンロードと配置をします。

ubuntu@sapphire:~/kubernetes$ wget -q --show-progress --https-only --timestamping "https://storage.googleapis.com/kubernetes-release/release/v1.18.6/bin/linux/arm64/kube-controller-manager"
kube-controller-manager      100%[==============================================>]  99.00M  12.1MB/s    in 8.4s    
ubuntu@sapphire:~/kubernetes$ chmod +x kube-controller-manager
ubuntu@sapphire:~/kubernetes$ sudo mv kube-controller-manager /usr/local/bin/
ubuntu@sapphire:~/kubernetes$

configファイルを配置します。

ubuntu@sapphire:~/kubernetes$ sudo cp -ai kubeconfig/kube-controller-manager.kubeconfig /var/lib/kubernetes/
ubuntu@sapphire:~/kubernetes$

kube-controller-manager を動かすためのユニットファイルを作成します。

ubuntu@sapphire:~/kubernetes$ POD_NETWORK=10.10.0.0/16
ubuntu@sapphire:~/kubernetes$ CLUSTER_IP_NETWORK=10.32.0.0/24
ubuntu@sapphire:~/kubernetes$ cat <<EOF | sudo tee /etc/systemd/system/kube-controller-manager.service
> [Unit]
> Description=Kubernetes Controller Manager
> Documentation=https://github.com/kubernetes/kubernetes
> 
> [Service]
> ExecStart=/usr/local/bin/kube-controller-manager \\
>   --bind-address=0.0.0.0 \\
>   --cluster-cidr=${POD_NETWORK} \\
>   --cluster-name=kubernetes \\
>   --cluster-signing-cert-file=/var/lib/kubernetes/ca.pem \\
>   --cluster-signing-key-file=/var/lib/kubernetes/ca-key.pem \\
>   --kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \\
>   --leader-elect=true \\
>   --root-ca-file=/var/lib/kubernetes/ca.pem \\
>   --service-account-private-key-file=/var/lib/kubernetes/service-account-key.pem \\
>   --service-cluster-ip-range=${CLUSTER_IP_NETWORK} \\
>   --use-service-account-credentials=true \\
>   --v=2
> Restart=on-failure
> RestartSec=5
> 
> [Install]
> WantedBy=multi-user.target
> EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
  --bind-address=0.0.0.0 \
  --cluster-cidr=10.10.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/var/lib/kubernetes/ca.pem \
  --cluster-signing-key-file=/var/lib/kubernetes/ca-key.pem \
  --kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \
  --leader-elect=true \
  --root-ca-file=/var/lib/kubernetes/ca.pem \
  --service-account-private-key-file=/var/lib/kubernetes/service-account-key.pem \
  --service-cluster-ip-range=10.32.0.0/24 \
  --use-service-account-credentials=true \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
ubuntu@sapphire:~/kubernetes$

起動します。

ubuntu@sapphire:~/kubernetes$ sudo systemctl daemon-reload
ubuntu@sapphire:~/kubernetes$ sudo systemctl enable kube-controller-manager
Created symlink /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service → /etc/systemd/system/kube-controller-manager.service.
ubuntu@sapphire:~/kubernetes$ sudo systemctl start kube-controller-manager
ubuntu@sapphire:~/kubernetes$ sudo systemctl status kube-controller-manager
● kube-controller-manager.service - Kubernetes Controller Manager
     Loaded: loaded (/etc/systemd/system/kube-controller-manager.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-10-23 21:58:18 JST; 8s ago
       Docs: https://github.com/kubernetes/kubernetes
   Main PID: 3336 (kube-controller)
      Tasks: 12 (limit: 9257)
     CGroup: /system.slice/kube-controller-manager.service
             └─3336 /usr/local/bin/kube-controller-manager --bind-address=0.0.0.0 --cluster-cidr=10.10.0.0/16 --cl>...

kube-scheduler のインストール

kube-scheduler は Pod のスケジューリングを担うコンポーネントとのこと。例によって（以下略

ダウンロードと配置を実施します。

ubuntu@sapphire:~/kubernetes$ wget -q --show-progress --https-only --timestamping "https://storage.googleapis.com/kubernetes-release/release/v1.18.6/bin/linux/arm64/kube-scheduler"
kube-scheduler               100%[==============================================>]  38.81M  12.0MB/s    in 3.3s    
ubuntu@sapphire:~/kubernetes$ chmod +x kube-scheduler
ubuntu@sapphire:~/kubernetes$ sudo mv kube-scheduler /usr/local/bin/
ubuntu@sapphire:~/kubernetes$

configファイルの配置をします。

ubuntu@sapphire:~/kubernetes$ sudo cp -ai kubeconfig/kube-scheduler.kubeconfig /var/lib/kubernetes/
ubuntu@sapphire:~/kubernetes$

ユニットファイルを作成します。

ubuntu@sapphire:~/kubernetes$ cat <<EOF | sudo tee /etc/kubernetes/config/kube-scheduler.yaml
> apiVersion: kubescheduler.config.k8s.io/v1alpha1
> kind: KubeSchedulerConfiguration
> clientConnection:
>   kubeconfig: "/var/lib/kubernetes/kube-scheduler.kubeconfig"
> leaderElection:
>   leaderElect: true
> EOF
apiVersion: kubescheduler.config.k8s.io/v1alpha1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: "/var/lib/kubernetes/kube-scheduler.kubeconfig"
leaderElection:
  leaderElect: true
ubuntu@sapphire:~/kubernetes$ cat <<EOF | sudo tee /etc/systemd/system/kube-scheduler.service
> [Unit]
> Description=Kubernetes Scheduler
> Documentation=https://github.com/kubernetes/kubernetes
> 
> [Service]
> ExecStart=/usr/local/bin/kube-scheduler \\
>   --config=/etc/kubernetes/config/kube-scheduler.yaml \\
>   --v=2
> Restart=on-failure
> RestartSec=5
> 
> [Install]
> WantedBy=multi-user.target
> EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-scheduler \
  --config=/etc/kubernetes/config/kube-scheduler.yaml \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
ubuntu@sapphire:~/kubernetes$

続けて起動します。

ubuntu@sapphire:~/kubernetes$ sudo systemctl daemon-reload
ubuntu@sapphire:~/kubernetes$ sudo systemctl enable kube-scheduler
Created symlink /etc/systemd/system/multi-user.target.wants/kube-scheduler.service → /etc/systemd/system/kube-scheduler.service.
ubuntu@sapphire:~/kubernetes$ sudo systemctl start kube-scheduler
ubuntu@sapphire:~/kubernetes$ sudo systemctl status kube-scheduler
● kube-scheduler.service - Kubernetes Scheduler
     Loaded: loaded (/etc/systemd/system/kube-scheduler.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-10-23 22:02:40 JST; 8s ago
       Docs: https://github.com/kubernetes/kubernetes
   Main PID: 3440 (kube-scheduler)
      Tasks: 13 (limit: 9257)
     CGroup: /system.slice/kube-scheduler.service
             └─3440 /usr/local/bin/kube-scheduler --config=/etc/kubernetes/config/kube-scheduler.yaml --v=2

動作チェック

ubuntu@sapphire:~/kubernetes$ kubectl get componentstatuses --kubeconfig kubeconfig/admin.kubeconfig
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-0               Healthy   {"health":"true"}

Nodeの準備

ここからはすべてのNodeに対して作業を実施しますが、記事中はsapphireのみで記載します。

Memory Subsystemの有効化を必要とのことで、すべての環境（今回だとsapphire,opal,peridot）で/boot/firmware/cmdline.txtファイルに以下の行を追記します。

cgroup_memory=1 cgroup_enable=memory

追記したら再起動します。

ubuntu@sapphire:~/kubernetes$ sudo reboot

必要なパッケージをインストールします。

ubuntu@sapphire:~/kubernetes$ sudo apt update
Hit:1 http://ports.ubuntu.com/ubuntu-ports focal InRelease
Hit:2 http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease
Hit:3 http://ports.ubuntu.com/ubuntu-ports focal-backports InRelease
Hit:4 http://ports.ubuntu.com/ubuntu-ports focal-security InRelease
Reading package lists... Done
Building dependency tree       
Reading state information... Done
All packages are up to date.
ubuntu@sapphire:~/kubernetes$ sudo apt -y install socat conntrack ipset
Reading package lists... Done
Building dependency tree       
Reading state information... Done
:
:
Processing triggers for man-db (2.9.1-1) ...
ubuntu@sapphire:~/kubernetes$

kubeletのインストール

kubeletはPodを動かすためのコンポーネント。ぼんやりイメージはできてるようなできていないような感じではありますが、こちらもまずは手順に従ってインストールを実施します。

ubuntu@sapphire:~/kubernetes$ sudo mkdir -p \
>   /etc/cni/net.d \
>   /opt/cni/bin \
>   /var/lib/kubelet \
>   /var/lib/kubernetes \
>   /etc/containerd
ubuntu@sapphire:~/kubernetes$ cd cert/
ubuntu@sapphire:~/kubernetes/cert$ sudo cp -ai ${HOSTNAME}-key.pem ${HOSTNAME}.pem /var/lib/kubelet/
ubuntu@sapphire:~/kubernetes/cert$ cd  ../kubeconfig/
ubuntu@sapphire:~/kubernetes/kubeconfig$ sudo cp -ai ${HOSTNAME}.kubeconfig /var/lib/kubelet/kubeconfig
ubuntu@sapphire:~/kubernetes/kubeconfig$ cd ../cert/
ubuntu@sapphire:~/kubernetes/cert$ sudo cp -ai ca.pem /var/lib/kubernetes/
ubuntu@sapphire:~/kubernetes/cert$ cd ..
ubuntu@sapphire:~/kubernetes$ wget -q --show-progress --https-only --timestamping \
>   https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.18.0/crictl-v1.18.0-linux-arm64.tar.gz \
>   https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-arm64-v0.8.6.tgz \
>   https://storage.googleapis.com/kubernetes-release/release/v1.18.6/bin/linux/arm64/kubelet
crictl-v1.18.0-linux-arm64.t 100%[==============================================>]  11.18M  10.5MB/s    in 1.1s    
cni-plugins-linux-arm64-v0.8 100%[==============================================>]  33.08M  11.7MB/s    in 2.8s    
kubelet                      100%[==============================================>] 101.93M  12.0MB/s    in 8.8s    
ubuntu@sapphire:~/kubernetes$ tar -xvf crictl-v1.18.0-linux-arm64.tar.gz
crictl
ubuntu@sapphire:~/kubernetes$ sudo tar -xvf cni-plugins-linux-arm64-v0.8.6.tgz -C /opt/cni/bin/
./
./flannel
./ptp
./host-local
./firewall
./portmap
./tuning
./vlan
./host-device
./bandwidth
./sbr
./static
./dhcp
./ipvlan
./macvlan
./loopback
./bridge
ubuntu@sapphire:~/kubernetes$ chmod +x crictl kubelet
ubuntu@sapphire:~/kubernetes$ sudo mv crictl kubelet /usr/local/bin/
ubuntu@sapphire:~/kubernetes$ sudo apt -y install containerd runc
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  containerd runc
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 28.1 MB of archives.
After this operation, 140 MB of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 runc arm64 1.0.0~rc95-0ubuntu1~20.04.2 [3545 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 containerd arm64 1.5.2-0ubuntu1~20.04.3 [24.5 MB]
Fetched 28.1 MB in 6s (5032 kB/s)      
Selecting previously unselected package runc.
(Reading database ... 117808 files and directories currently installed.)
Preparing to unpack .../runc_1.0.0~rc95-0ubuntu1~20.04.2_arm64.deb ...
Unpacking runc (1.0.0~rc95-0ubuntu1~20.04.2) ...
Selecting previously unselected package containerd.
Preparing to unpack .../containerd_1.5.2-0ubuntu1~20.04.3_arm64.deb ...
Unpacking containerd (1.5.2-0ubuntu1~20.04.3) ...
Setting up runc (1.0.0~rc95-0ubuntu1~20.04.2) ...
Setting up containerd (1.5.2-0ubuntu1~20.04.3) ...
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service.
Processing triggers for man-db (2.9.1-1) ...
ubuntu@sapphire:~/kubernetes$

PODネットワーク設定を実施します。

ubuntu@sapphire:~/kubernetes$ POD_CIDR=10.10.1.0/24
ubuntu@sapphire:~/kubernetes$ cat <<EOF | sudo tee /etc/cni/net.d/10-bridge.conf
> {
>     "cniVersion": "0.3.1",
>     "name": "bridge",
>     "type": "bridge",
>     "bridge": "cnio0",
>     "isGateway": true,
>     "ipMasq": true,
>     "ipam": {
>         "type": "host-local",
>         "ranges": [
>           [{"subnet": "${POD_CIDR}"}]
>         ],
>         "routes": [{"dst": "0.0.0.0/0"}]
>     }
> }
> EOF
{
    "cniVersion": "0.3.1",
    "name": "bridge",
    "type": "bridge",
    "bridge": "cnio0",
    "isGateway": true,
    "ipMasq": true,
    "ipam": {
        "type": "host-local",
        "ranges": [
          [{"subnet": "10.10.1.0/24"}]
        ],
        "routes": [{"dst": "0.0.0.0/0"}]
    }
}
ubuntu@sapphire:~/kubernetes$ cat <<EOF | sudo tee /etc/cni/net.d/99-loopback.conf
> {
>     "cniVersion": "0.3.1",
>     "name": "lo",
>     "type": "loopback"
> }
> EOF
{
    "cniVersion": "0.3.1",
    "name": "lo",
    "type": "loopback"
}
ubuntu@sapphire:~/kubernetes$

containerd の設定をします。

ubuntu@sapphire:~/kubernetes$ cat << EOF | sudo tee /etc/containerd/config.toml
> [plugins]
>   [plugins.cri.containerd]
>     snapshotter = "overlayfs"
>     [plugins.cri.containerd.default_runtime]
>       runtime_type = "io.containerd.runtime.v1.linux"
>       runtime_engine = "/usr/sbin/runc"
>       runtime_root = ""
> EOF
[plugins]
  [plugins.cri.containerd]
    snapshotter = "overlayfs"
    [plugins.cri.containerd.default_runtime]
      runtime_type = "io.containerd.runtime.v1.linux"
      runtime_engine = "/usr/sbin/runc"
      runtime_root = ""
ubuntu@sapphire:~/kubernetes$

設定ファイルとユニットファイルを作成します。

ubuntu@sapphire:~/kubernetes$ POD_CIDR=10.10.1.0/24
ubuntu@sapphire:~/kubernetes$ cat <<EOF | sudo tee /var/lib/kubelet/kubelet-config.yaml
> kind: KubeletConfiguration
> apiVersion: kubelet.config.k8s.io/v1beta1
> authentication:
>   anonymous:
>     enabled: false
>   webhook:
>     enabled: true
>   x509:
>     clientCAFile: "/var/lib/kubernetes/ca.pem"
> authorization:
>   mode: Webhook
> clusterDomain: "cluster.local"
> clusterDNS:
>   - "10.32.0.10"
> podCIDR: "${POD_CIDR}"
> resolvConf: "/run/systemd/resolve/resolv.conf"
> runtimeRequestTimeout: "15m"
> tlsCertFile: "/var/lib/kubelet/${HOSTNAME}.pem"
> tlsPrivateKeyFile: "/var/lib/kubelet/${HOSTNAME}-key.pem"
> EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: "/var/lib/kubernetes/ca.pem"
authorization:
  mode: Webhook
clusterDomain: "cluster.local"
clusterDNS:
  - "10.32.0.10"
podCIDR: "10.10.1.0/24"
resolvConf: "/run/systemd/resolve/resolv.conf"
runtimeRequestTimeout: "15m"
tlsCertFile: "/var/lib/kubelet/sapphire.pem"
tlsPrivateKeyFile: "/var/lib/kubelet/sapphire-key.pem"
ubuntu@sapphire:~/kubernetes$ cat <<EOF | sudo tee /etc/systemd/system/kubelet.service
> [Unit]
> Description=Kubernetes Kubelet
> Documentation=https://github.com/kubernetes/kubernetes
> After=containerd.service
> Requires=containerd.service
> 
> [Service]
> ExecStart=/usr/local/bin/kubelet \\
>   --config=/var/lib/kubelet/kubelet-config.yaml \\
>   --container-runtime=remote \\
>   --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \\
>   --image-pull-progress-deadline=2m \\
>   --kubeconfig=/var/lib/kubelet/kubeconfig \\
>   --network-plugin=cni \\
>   --register-node=true \\
>   --v=2
> Restart=on-failure
> RestartSec=5
> 
> [Install]
> WantedBy=multi-user.target
> EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=containerd.service
Requires=containerd.service

[Service]
ExecStart=/usr/local/bin/kubelet \
  --config=/var/lib/kubelet/kubelet-config.yaml \
  --container-runtime=remote \
  --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \
  --image-pull-progress-deadline=2m \
  --kubeconfig=/var/lib/kubelet/kubeconfig \
  --network-plugin=cni \
  --register-node=true \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
ubuntu@sapphire:~/kubernetes$

起動します。

ubuntu@sapphire:~/kubernetes$ sudo systemctl daemon-reload
ubuntu@sapphire:~/kubernetes$ sudo systemctl enable kubelet
Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /etc/systemd/system/kubelet.service.
ubuntu@sapphire:~/kubernetes$ sudo systemctl start kubelet
ubuntu@sapphire:~/kubernetes$ sudo systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
     Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2021-10-24 10:13:48 JST; 5s ago
       Docs: https://github.com/kubernetes/kubernetes
   Main PID: 7297 (kubelet)
      Tasks: 13 (limit: 9257)
     CGroup: /system.slice/kubelet.service
             └─7297 /usr/local/bin/kubelet --config=/var/lib/kubelet/kubelet-config.yaml --container-runtime=remote>...

kube-proxyのインストール

kubernetes内のネットワークを制御するためのコンポーネントをインストールします。

ubuntu@sapphire:~/kubernetes$ wget -q --show-progress --https-only --timestamping https://storage.googleapis.com/kubernetes-release/release/v1.18.6/bin/linux/arm64/kube-proxy
kube-proxy                   100%[==============================================>]  34.62M  11.0MB/s    in 3.2s    
ubuntu@sapphire:~/kubernetes$ chmod +x kube-proxy
ubuntu@sapphire:~/kubernetes$ sudo mv kube-proxy /usr/local/bin/
ubuntu@sapphire:~/kubernetes$

設定ファイルを配置します。

ubuntu@sapphire:~/kubernetes$ sudo mkdir -p /var/lib/kube-proxy
ubuntu@sapphire:~/kubernetes$ sudo mv kubeconfig/kube-proxy.kubeconfig /var/lib/kube-proxy/kubeconfig
ubuntu@sapphire:~/kubernetes$

ユニットファイルを作成します。

ubuntu@sapphire:~/kubernetes$ POD_NETWORK=10.10.0.0/16
ubuntu@sapphire:~/kubernetes$ cat <<EOF | sudo tee /var/lib/kube-proxy/kube-proxy-config.yaml
> kind: KubeProxyConfiguration
> apiVersion: kubeproxy.config.k8s.io/v1alpha1
> clientConnection:
>   kubeconfig: "/var/lib/kube-proxy/kubeconfig"
> mode: "iptables"
> clusterCIDR: "${POD_NETWORK}"
> EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:
  kubeconfig: "/var/lib/kube-proxy/kubeconfig"
mode: "iptables"
clusterCIDR: "10.10.0.0/16"
ubuntu@sapphire:~/kubernetes$ cat <<EOF | sudo tee /etc/systemd/system/kube-proxy.service
> [Unit]
> Description=Kubernetes Kube Proxy
> Documentation=https://github.com/kubernetes/kubernetes
> 
> [Service]
> ExecStart=/usr/local/bin/kube-proxy \\
>   --config=/var/lib/kube-proxy/kube-proxy-config.yaml
> Restart=on-failure
> RestartSec=5
> 
> [Install]
> WantedBy=multi-user.target
> EOF
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-proxy \
  --config=/var/lib/kube-proxy/kube-proxy-config.yaml
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
ubuntu@sapphire:~/kubernetes$

起動します。

ubuntu@sapphire:~/kubernetes$ sudo systemctl daemon-reload
ubuntu@sapphire:~/kubernetes$ sudo systemctl enable kube-proxy
Created symlink /etc/systemd/system/multi-user.target.wants/kube-proxy.service → /etc/systemd/system/kube-proxy.service.
ubuntu@sapphire:~/kubernetes$ sudo systemctl start kube-proxy
ubuntu@sapphire:~/kubernetes$ sudo systemctl status kube-proxy
● kube-proxy.service - Kubernetes Kube Proxy
     Loaded: loaded (/etc/systemd/system/kube-proxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2021-10-24 10:38:42 JST; 6s ago
       Docs: https://github.com/kubernetes/kubernetes
   Main PID: 12020 (kube-proxy)
      Tasks: 11 (limit: 9257)
     CGroup: /system.slice/kube-proxy.service
             └─12020 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/kube-proxy-config.yaml

kube-apiserver 認証 RBAC 設定

kube-apiserver からの接続を許可する設定をします。

ubuntu@sapphire:~/kubernetes$ cat <<EOF | kubectl apply --kubeconfig kubeconfig/admin.kubeconfig  -f -
> apiVersion: rbac.authorization.k8s.io/v1beta1
> kind: ClusterRole
> metadata:
>   annotations:
>     rbac.authorization.kubernetes.io/autoupdate: "true"
>   labels:
>     kubernetes.io/bootstrapping: rbac-defaults
>   name: system:kube-apiserver-to-kubelet
> rules:
>   - apiGroups:
>       - ""
>     resources:
>       - nodes/proxy
>       - nodes/stats
>       - nodes/log
>       - nodes/spec
>       - nodes/metrics
>     verbs:
>       - "*"
> EOF
clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
ubuntu@sapphire:~/kubernetes$ cat <<EOF | kubectl apply --kubeconfig kubeconfig/admin.kubeconfig -f -
> apiVersion: rbac.authorization.k8s.io/v1beta1
> kind: ClusterRoleBinding
> metadata:
>   name: system:kube-apiserver
>   namespace: ""
> roleRef:
>   apiGroup: rbac.authorization.k8s.io
>   kind: ClusterRole
>   name: system:kube-apiserver-to-kubelet
> subjects:
>   - apiGroup: rbac.authorization.k8s.io
>     kind: User
>     name: kubernetes
> EOF
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created
ubuntu@sapphire:~/kubernetes$

Nodeの動作チェック

Nodeの状態をチェックします。

ubuntu@sapphire:~/kubernetes$ kubectl get node --kubeconfig kubeconfig/admin.kubeconfig 
NAME       STATUS     ROLES    AGE   VERSION
opal       NotReady   <none>   36m   v1.18.6
peridot    NotReady   <none>   36m   v1.18.6
sapphire   NotReady   <none>   38m   v1.18.6
ubuntu@sapphire:~/kubernetes$

一応無事インストールできた…かな。 これでいろいろと遊んでみたいと思います。