记录下k8s环境的搭建,中间重装系统、重新部署集群几十次。
0 环境准备:
节点数量 : 3 台 centos 7.9.2009 (Core)
配置: 4c 16g
网络互通
1. 集群规划
k8s-n1 10.8.8.31 主
k8s-n2 10.8.8.32
k8s-n3 10.8.8.33
2. 设置主机名
这一步,在三台节点上,各自执行一条。3-9步在三台节点上全都要执行。
$ hostnamectl set-hostname k8s-n1
$ hostnamectl set-hostname k8s-n2
$ hostnamectl set-hostname k8s-n3
3. 设置hosts文件
cat >> /etc/hosts <<EOF
10.8.8.31 k8s-n1
10.8.8.32 k8s-n2
10.8.8.33 k8s-n3
EOF
# 预留以下ip和hostname
cat >> /etc/hosts <<EOF
10.8.8.31 k8s-n1
10.8.8.32 k8s-n2
10.8.8.33 k8s-n3
10.8.8.34 k8s-n4
10.8.8.35 k8s-n5
10.8.8.36 k8s-n6
10.8.8.37 k8s-n7
10.8.8.38 k8s-n8
10.8.8.39 k8s-n9
EOF
4. 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
5. 关闭SELINUX
# 将 SELinux 设置为 permissive 模式(相当于将其禁用)
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
6. 关闭swap分区

swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
7. 开启ipvs
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack
8. 安装containerd并配置
参考 :
# 配置网络 转发 IPv4 并让 iptables 看到桥接流量
cat > /etc/modules-load.d/k8s.conf << EOF
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
# 应用 sysctl 参数而不重新启动
sysctl --system
下载安装containerd
wget https://github.com/containerd/containerd/releases/download/v1.6.14/cri-containerd-cni-1.6.14-linux-amd64.tar.gz
cri-containerd-cni-1.6.14-linux-amd64.tar.gz压缩包中已经按照官方二进制部署推荐的目录结构布局好。 里面包含了systemd配置文件,containerd以及cni的部署文件。 将解压缩到系统的根目录/中:
tar -zxvf cri-containerd-cni-1.6.14-linux-amd64.tar.gz -C /
经测试cri-containerd-cni-1.6.4-linux-amd64.tar.gz包中包含的runc在CentOS 7下的动态链接有问题,这里从runc的github上单独下载runc,并替换上面安装的containerd中的runc:
wget https://github.com/opencontainers/runc/releases/download/v1.1.2/runc.amd64
chmod +x runc.amd64
mv /usr/local/sbin/runc /usr/local/sbin/runc.bak
mv runc.amd64 /usr/local/sbin/runc
接下来生成containerd的配置文件:
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
修改前面生成的配置文件/etc/containerd/config.toml:
搜索
SystemdCgroup
改为true
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
...
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
...
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://suoqz334.mirror.aliyuncs.com", "https://registry-1.docker.io"]
搜索
sandbox_image
改为registry.aliyuncs.com/google_containers/pause:3.9
# sandbox_image = "k8s.gcr.io/pause:3.6"
sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"
配置containerd开机启动,并启动containerd
systemctl daemon-reload
systemctl enable containerd --now
使用crictl测试一下,确保可以打印出版本信息并且没有错误信息输出:
crictl version
Version: 0.1.0
RuntimeName: containerd
RuntimeVersion: v1.6.14
RuntimeApiVersion: v1
第一次测试报错了:
[root@k8s-templete ~]# crictl version
E0215 22:11:47.458647 1587 remote_runtime.go:168] "Version from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
FATA[0000] getting the runtime version: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
[root@k8s-templete ~]# systemctl status containerd.service -l
● containerd.service - containerd container runtime
Loaded: loaded (/etc/systemd/system/containerd.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2023-02-15 22:12:58 CST; 27s ago
Docs: https://containerd.io
Process: 1732 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Main PID: 1735 (containerd)
Tasks: 9
Memory: 20.9M
CGroup: /system.slice/containerd.service
└─1735 /usr/local/bin/containerd
Feb 15 22:12:58 k8s-templete containerd[1735]: time="2023-02-15T22:12:58.116386085+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
Feb 15 22:12:58 k8s-templete containerd[1735]: time="2023-02-15T22:12:58.116403276+08:00" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
Feb 15 22:12:58 k8s-templete containerd[1735]: time="2023-02-15T22:12:58.116428745+08:00" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
Feb 15 22:12:58 k8s-templete containerd[1735]: time="2023-02-15T22:12:58.116446848+08:00" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
Feb 15 22:12:58 k8s-templete containerd[1735]: time="2023-02-15T22:12:58.116474383+08:00" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
Feb 15 22:12:58 k8s-templete containerd[1735]: time="2023-02-15T22:12:58.116515607+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
Feb 15 22:12:58 k8s-templete containerd[1735]: time="2023-02-15T22:12:58.116769338+08:00" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri" error="invalid plugin config: `systemd_cgroup` only works for runtime io.containerd.runtime.v1.linux"
有一个报错:invalid plugin config: systemd_cgroup only works for runtime io.containerd.runtime.v1.linux
修改
[plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
...
runtime_type = "io.containerd.runtime.v1.linux"
重启再试下
[root@k8s-templete ~]# systemctl restart containerd
[root@k8s-templete ~]# crictl version
Version: 0.1.0
RuntimeName: containerd
RuntimeVersion: v1.6.17
RuntimeApiVersion: v1
9. 安装k8s
# 添加源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
# 安装
yum install -y kubelet kubeadm kubectl
# 配置开机自启
systemctl enable kubelet
10. 初始化主节点
使用kubeadm config print init-defaults --component-configs KubeletConfiguration >kubeadm.default.yaml
可以创建集群初始化默认的使用的配置。
自定义出本次使用kubeadm初始化集群所需的配置文件kubeadm.yaml:
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 10.8.8.34
bindPort: 6443
nodeRegistration:
criSocket: unix:///run/containerd/containerd.sock
taints:
- effect: PreferNoSchedule
key: node-role.kubernetes.io/master
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: 1.26.0
imageRepository: registry.aliyuncs.com/google_containers
networking:
podSubnet: 10.244.0.0/16
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
failSwapOn: false
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
这里定制了imageRepository
为阿里云的registry,避免被墙,无法直接拉取镜像。criSocket
设置了容器运行时为containerd。 同时设置kubelet的cgroupDriver
为systemd,设置kube-proxy代理模式为ipvs
。
选择node1节点作为主节点,在主节点上初始化集群:
kubeadm init --config kubeadm.yaml
kubeadm init --apiserver-advertise-address=10.8.8.31 \
--image-repository registry.aliyuncs.com/google_containers
初始化结束后配置kubectl:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
如果初始化出现问题,可以使用kubeadm reset
清理集群。这个命令在搭建过程中执行过十几次QAQ。
在初始化完成的最后给出了其他节点加入集群的命令:
kubeadm join 10.8.8.31:6443 --token 15s5fj.sklsq0mpgakjs1h6kqd9p \
--discovery-token-ca-cert-hash sha256:34985938744910cf33d849e0f2d48fb6529ef028dkjwqxb7sa03475848304a18bbfc0
11. 部署Pod Network组件calico
# 安装 Pod 网络插件(CNI)
wget --no-check-certificate https://projectcalico.docs.tigera.io/archive/v3.25/manifests/calico.yaml
kubectl apply -f calico.yaml