Logo

菜单

miaodi
发布于 2023-03-08 / 347 阅读
0

1.26版本k8s集群搭建记录

#k8s

记录下k8s环境的搭建，中间重装系统、重新部署集群几十次。

0 环境准备：

  • 节点数量 : 3 台 centos 7.9.2009 (Core)

  • 配置： 4c 16g

  • 网络互通

1. 集群规划

  • k8s-n1 10.8.8.31 主

  • k8s-n2 10.8.8.32

  • k8s-n3 10.8.8.33

2. 设置主机名

这一步，在三台节点上，各自执行一条。3-9步在三台节点上全都要执行。

$ hostnamectl set-hostname k8s-n1
$ hostnamectl set-hostname k8s-n2
$ hostnamectl set-hostname k8s-n3

3. 设置hosts文件

cat >> /etc/hosts <<EOF
10.8.8.31 k8s-n1
10.8.8.32 k8s-n2
10.8.8.33 k8s-n3
EOF
​
# 预留以下ip和hostname
cat >> /etc/hosts <<EOF
10.8.8.31 k8s-n1
10.8.8.32 k8s-n2
10.8.8.33 k8s-n3
10.8.8.34 k8s-n4
10.8.8.35 k8s-n5
10.8.8.36 k8s-n6
10.8.8.37 k8s-n7
10.8.8.38 k8s-n8
10.8.8.39 k8s-n9
EOF

4. 关闭防火墙

systemctl stop firewalld
systemctl disable firewalld

5. 关闭SELINUX

# 将 SELinux 设置为 permissive 模式（相当于将其禁用）
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

6. 关闭swap分区

![image-20230211232308111](https://cntz6zilryza.compat.objectstorage.ap-seoul-1.oraclecloud.com/bucket-20230202-1610/picgo/20230211_6d969809991fef34bf37e6e02a895c70.png)

swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

7. 开启ipvs

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack

8. 安装containerd并配置

参考 ：

容器运行时 | Kubernetes

使用kubeadm部署Kubernetes 1.26 - 架构小白

# 配置网络 转发 IPv4 并让 iptables 看到桥接流量 
cat > /etc/modules-load.d/k8s.conf << EOF
overlay
br_netfilter
EOF
​
sudo modprobe overlay
sudo modprobe br_netfilter
​
cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
​
# 应用 sysctl 参数而不重新启动
sysctl --system

下载安装containerd

wget https://github.com/containerd/containerd/releases/download/v1.6.14/cri-containerd-cni-1.6.14-linux-amd64.tar.gz

cri-containerd-cni-1.6.14-linux-amd64.tar.gz压缩包中已经按照官方二进制部署推荐的目录结构布局好。 里面包含了systemd配置文件，containerd以及cni的部署文件。 将解压缩到系统的根目录/中:

tar -zxvf cri-containerd-cni-1.6.14-linux-amd64.tar.gz -C /

经测试cri-containerd-cni-1.6.4-linux-amd64.tar.gz包中包含的runc在CentOS 7下的动态链接有问题，这里从runc的github上单独下载runc，并替换上面安装的containerd中的runc:

wget https://github.com/opencontainers/runc/releases/download/v1.1.2/runc.amd64
chmod +x runc.amd64
mv /usr/local/sbin/runc /usr/local/sbin/runc.bak
mv runc.amd64 /usr/local/sbin/runc

接下来生成containerd的配置文件:

mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml

修改前面生成的配置文件/etc/containerd/config.toml：

  • 搜索 SystemdCgroup 改为true

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true
    
    
  ...
  [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors] 
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://suoqz334.mirror.aliyuncs.com", "https://registry-1.docker.io"]

  • 搜索 sandbox_image改为registry.aliyuncs.com/google_containers/pause:3.9

# sandbox_image = "k8s.gcr.io/pause:3.6"
sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"

配置containerd开机启动，并启动containerd

systemctl daemon-reload
​
systemctl enable containerd --now

使用crictl测试一下，确保可以打印出版本信息并且没有错误信息输出:

crictl version
​
Version:  0.1.0
RuntimeName:  containerd
RuntimeVersion:  v1.6.14
RuntimeApiVersion:  v1

第一次测试报错了：

[root@k8s-templete ~]# crictl version
E0215 22:11:47.458647    1587 remote_runtime.go:168] "Version from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
FATA[0000] getting the runtime version: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
[root@k8s-templete ~]# systemctl status containerd.service  -l
● containerd.service - containerd container runtime
   Loaded: loaded (/etc/systemd/system/containerd.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2023-02-15 22:12:58 CST; 27s ago
     Docs: https://containerd.io
  Process: 1732 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
 Main PID: 1735 (containerd)
    Tasks: 9
   Memory: 20.9M
   CGroup: /system.slice/containerd.service
           └─1735 /usr/local/bin/containerd
​
Feb 15 22:12:58 k8s-templete containerd[1735]: time="2023-02-15T22:12:58.116386085+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
Feb 15 22:12:58 k8s-templete containerd[1735]: time="2023-02-15T22:12:58.116403276+08:00" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
Feb 15 22:12:58 k8s-templete containerd[1735]: time="2023-02-15T22:12:58.116428745+08:00" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
Feb 15 22:12:58 k8s-templete containerd[1735]: time="2023-02-15T22:12:58.116446848+08:00" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
Feb 15 22:12:58 k8s-templete containerd[1735]: time="2023-02-15T22:12:58.116474383+08:00" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
Feb 15 22:12:58 k8s-templete containerd[1735]: time="2023-02-15T22:12:58.116515607+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
Feb 15 22:12:58 k8s-templete containerd[1735]: time="2023-02-15T22:12:58.116769338+08:00" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri" error="invalid plugin config: `systemd_cgroup` only works for runtime io.containerd.runtime.v1.linux"

有一个报错：invalid plugin config: systemd_cgroup only works for runtime io.containerd.runtime.v1.linux

修改

      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
       ...
        runtime_type = "io.containerd.runtime.v1.linux"

重启再试下

[root@k8s-templete ~]# systemctl restart containerd
[root@k8s-templete ~]# crictl version
Version:  0.1.0
RuntimeName:  containerd
RuntimeVersion:  v1.6.17
RuntimeApiVersion:  v1
​

9. 安装k8s

# 添加源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
​
# 安装
yum install -y kubelet kubeadm kubectl
​
# 配置开机自启
systemctl enable kubelet

10. 初始化主节点

使用kubeadm config print init-defaults --component-configs KubeletConfiguration >kubeadm.default.yaml可以创建集群初始化默认的使用的配置。

自定义出本次使用kubeadm初始化集群所需的配置文件kubeadm.yaml：

apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.8.8.34
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
  taints:
  - effect: PreferNoSchedule
    key: node-role.kubernetes.io/master
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: 1.26.0
imageRepository: registry.aliyuncs.com/google_containers
networking:
  podSubnet: 10.244.0.0/16
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
failSwapOn: false
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

这里定制了imageRepository为阿里云的registry，避免被墙，无法直接拉取镜像。criSocket设置了容器运行时为containerd。 同时设置kubelet的cgroupDriver为systemd，设置kube-proxy代理模式为ipvs。

选择node1节点作为主节点，在主节点上初始化集群：

kubeadm init --config kubeadm.yaml
kubeadm init --apiserver-advertise-address=10.8.8.31 \
--image-repository registry.aliyuncs.com/google_containers

初始化结束后配置kubectl:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

如果初始化出现问题，可以使用kubeadm reset清理集群。这个命令在搭建过程中执行过十几次QAQ。

在初始化完成的最后给出了其他节点加入集群的命令：

kubeadm join 10.8.8.31:6443 --token 15s5fj.sklsq0mpgakjs1h6kqd9p \
        --discovery-token-ca-cert-hash sha256:34985938744910cf33d849e0f2d48fb6529ef028dkjwqxb7sa03475848304a18bbfc0

11. 部署Pod Network组件calico

# 安装 Pod 网络插件(CNI) 
wget --no-check-certificate https://projectcalico.docs.tigera.io/archive/v3.25/manifests/calico.yaml
kubectl apply -f calico.yaml


k8s部署nfs存储类
windows系统下linux环境