$ kubectl apply -fhttps://raw.githubusercontent.com/kubernetes-sigs/ip-masq-agent/master/ip-masq-agent.yaml2 安装配置 Node
A Join
在两个 Node 节点上运行 join 命令：
sudo kubeadm join10.122.16.10:6443 --token youzhi.0123456789abcdef--discovery-token-ca-cert-hash sha256:e5c96b2d0499287b6884c27b8ec7293e6aab1ab09540a0386b91ecbadfb38d4fB 查看 Node 状态
此时 Node 是 NotReady 状态 ， 因为 kubelet 的网络插件配置有问题 。
$ kubectl get nodeNAMESTATUSROLESAGEVERSIONkub-mReadycontrol-plane,master137mv1.22.0kub-n-1NotReady<none>15mv1.22.0C 修改 Node 上 kubelet 配置
 
这时 node 的状态变成 Ready：
$ kubectl get nodeNAMESTATUSROLESAGEVERSIONkub-mReadycontrol-plane,master147mv1.22.0kub-n-1Ready<none>25mv1.22.0
D 修改 Node providerID
为了 Service 能自动创建和配置谷歌云负载均衡 ， 需要配置每个 Node 的 providerID 参数 。对于谷歌云上的 Node ， providerID 的格式是 gce://<Project ID>/<Zone>/<Instance Name> 。
在 Master 节点 ， 运行 kubectl 修改每个 Node 的 prividerID：
$ kubectl patch node kub-n-1-p '{"spec":{"providerID":"gce://youzhi-lab/asia-southeast1-b/kub-n-1"}}'node/kub-n-1 patched$ kubectl patch node kub-n-2-p '{"spec":{"providerID":"gce://youzhi-lab/asia-southeast1-b/kub-n-2"}}'node/kub-n-2 patched
E 给 GCE 添加 Network Tag
在 Kubernetes 通过 cloud-provider 创建实现 Service 的谷歌云负载均衡的时候 ， 需要通过 network tag 创建 firewall rules 。所以需要给 GCE 实例添加 network tag 。创建的规则是所有 node 相同的前缀 ， 比如这里两台 VM 相同的前缀是 kub-n
gcloud compute instancesadd-tags kub-n-1--zone asia-southeast1-b --tags kub-ngcloud compute instances add-tags kub-n-2--zone asia-southeast1-b --tags kub-n
F 给 Node 虚机实例添加 Alias IP
kubectl get nodes 列出各个 Node 的名称 。
查看每个 Node 的 Pod CIDR ， 如下 。
$ kubectl get node kub-n-1 -ojsonpath={.spec.podCIDR}192.168.18.0/24使用谷歌云命令行工具 ， 根据上面的打印结果 ， 分别更新各个 Node 的 Alias IP ， 使其与该 Node 的 Pod CIDR 一致 。
● Node-1
gcloud compute instancesnetwork-interfaces update kub-n-1--zone asia-southeast1-b--aliases"asia-southeast1-sub-1-pod:192.168.17.0/24"
● Node-2
gcloud compute instancesnetwork-interfaces update kub-n-2--zone asia-southeast1-b--aliases"asia-southeast1-sub-1-pod:192.168.18.0/24"3 检测配置
A 创建 deployment 和 service
kubectl create deploy Nginx--image nginxkubectl scale deploy/nginx --replicas=6kubectl expose deployment nginx--port 80 --target-port 80 --name nginx-lb --type LoadBalancer
B 查看部署情况查看 pod 情况：
$kubectl get pod -o wideNAMEREADYSTATUSRESTARTSAGEIPNODEnginx-6799fc88d8-68fld1/1Running017h192.168.18.7kub-n-2nginx-6799fc88d8-gssw51/1Running017h192.168.17.15kub-n-1nginx-6799fc88d8-jcdhc1/1Running017h192.168.17.14kub-n-1nginx-6799fc88d8-k7sxg1/1Running017h192.168.18.6kub-n-2nginx-6799fc88d8-l6nnv1/1Running017h192.168.18.5kub-n-2nginx-6799fc88d8-tcbmf1/1Running017h192.168.17.13kub-n-1查看 Service：
$kubectl get svcNAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGEkubernetesClusterIP172.16.16.1<none>443/TCP29hnginx-lbLoadBalancer172.16.16.15334.126.129.7580:32761/TCP7m31s获取 Service 的 LB IP
ip=$(kubectl get svc -lapp=nginx-o jsonpath={.items[0].status.loadBalancer.ingress[0].ip})测试 Pod 的同 Node 和跨 Node 访问
$ kubectl exec -it nginx-6799fc88d8-68fld bashkubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a futureversion. Use kubectl exec [POD] -- [COMMAND] instead.root@nginx-6799fc88d8-68fld:/# curl -I 192.168.18.6HTTP/1.1 200 OKServer: nginx/1.21.1Date: Thu, 12 Aug 2021 07:25:29 GMTContent-Type: text/htmlContent-Length: 612Last-Modified: Tue, 06 Jul 2021 14:59:17 GMTConnection: keep-aliveETag: "60e46fc5-264"Accept-Ranges: bytesroot@nginx-6799fc88d8-68fld:/# curl -I 192.168.17.15HTTP/1.1 200 OKServer: nginx/1.21.1Date: Thu, 12 Aug 2021 07:25:35 GMTContent-Type: text/htmlContent-Length: 612Last-Modified: Tue, 06 Jul 2021 14:59:17 GMTConnection: keep-aliveETag: "60e46fc5-264"Accept-Ranges: bytes测试通过 Cluser IP 访问 Service
root@nginx-6799fc88d8-68fld:/# curl-I 172.16.16.153HTTP/1.1 200 OKServer: nginx/1.21.1Date: Thu, 12 Aug 2021 07:25:43 GMTContent-Type: text/htmlContent-Length: 612Last-Modified: Tue, 06 Jul 2021 14:59:17 GMTConnection: keep-aliveETag: "60e46fc5-264"Accept-Ranges: bytes