由 SuKai June 9, 2022
之前文章介绍了如何通过Kube-OVN实现Kubernetes网络租户隔离，今天我们一起看一下KubeVirt通过Kube-OVN使用Underlay网络，使虚拟机接入物理网络。
实践操作
检查Kubernetes使用的Pod网络地址段和Service网络地址段。
sukai@ubuntuserver:/etc/kubernetes/manifests$ sudo grep cidr kube-controller-manager.yaml
- --allocate-node-cidrs=true
- --cluster-cidr=10.244.0.0/16
sukai@ubuntuserver:/etc/kubernetes/manifests$ sudo grep range kube-controller-manager.yaml
- --service-cluster-ip-range=10.211.0.0/16
检查kube-ovn-controller配置项是否正确
kube-ovn在版本v1.11.0开始支持keep-vm-ip参数，虚拟机使用固定IP地址。
Containers:
kube-ovn-controller:
Container ID: containerd://0f8d53403c9821c7b535c64d2c98e1ca130a15e1b2b878270cd9388c36d0e48a
Image: kubeovn/kube-ovn:v1.11.0
Image ID: docker.io/kubeovn/kube-ovn@sha256:fea623e68a2a81ef78102c6cfe95b96c16c054c57f4c8b9d168bd3ff620b6779
Port: <none>
Host Port: <none>
Args:
/kube-ovn/start-controller.sh
--default-cidr=10.244.0.0/16
--default-gateway=10.244.0.1
--default-gateway-check=true
--default-logical-gateway=false
--default-exclude-ips=
--node-switch-cidr=100.64.0.0/16
--service-cluster-ip-range=10.211.0.0/16
--network-type=geneve
--default-interface-name=
--default-vlan-id=100
--pod-nic-type=veth-pair
--enable-lb=true
--enable-np=true
--enable-eip-snat=true
--enable-external-vpc=true
--keep-vm-ip=true
创建服务网络
指定物理服务器上访问物理网络的网卡。
apiVersion: kubeovn.io/v1
kind: ProviderNetwork
metadata:
name: office
spec:
defaultInterface: eno1
创建VLAN
创建在指定服务网络上创建一个VLAN，这里VLAN ID为0表示不属于任何VLAN。
apiVersion: kubeovn.io/v1
kind: Vlan
metadata:
name: defaultvlan
spec:
id: 0
provider: office
创建子网
在defaultvlan上，创建一个子网，这个子网作用的namespace为mail263，指定子网分配的物理网络地址段，物理网络网关地址。
---
apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
name: mail263
spec:
protocol: IPv4
cidrBlock: 172.16.3.0/24
gateway: 172.16.3.2
excludeIps:
- 172.16.3.1..172.16.3.200
vlan: defaultvlan
# natOutgoing: false
#private: false
namespaces:
- mail263
查看kube-ovn信息
sukai@ubuntuserver:~$ kubectl get vlan -o wide
NAME ID PROVIDER
defaultvlan 0 office
sukai@ubuntuserver:~$ kubectl get subnet
NAME PROVIDER VPC PROTOCOL CIDR PRIVATE NAT DEFAULT GATEWAYTYPE V4USED V4AVAILABLE V6USED V6AVAILABLE EXCLUDEIPS
join ovn ovn-cluster IPv4 100.64.0.0/16 false false false distributed 1 65532 0 0 ["100.64.0.1"]
mail263 ovn ovn-cluster IPv4 172.16.3.0/24 false false false distributed 1 53 0 0 ["172.16.3.1..172.16.3.200"]
ovn-default ovn ovn-cluster IPv4 10.244.0.0/16 false true true distributed 19 65514 0 0 ["10.244.0.1"]
sukai@ubuntuserver:~$
sukai@ubuntuserver:~$ kubectl ko nbctl show
switch e0b180e4-1e4c-49e0-9bea-d5aa384196b0 (ovn-default)
port cert-manager-8568b6f9cb-wf74c.cert-manager
addresses: ["00:00:00:E9:A4:E5 10.244.0.15"]
port cert-manager-cainjector-c9c77b797-xwlgh.cert-manager
addresses: ["00:00:00:39:46:D0 10.244.0.14"]
port coredns-6d4b75cb6d-tt7cb.kube-system
addresses: ["00:00:00:F1:29:07 10.244.0.3"]
port virt-handler-9st4d.kubevirt
addresses: ["00:00:00:A0:72:A1 10.244.0.22"]
port hostpath-provisioner-csi-5mcbc.hostpath-provisioner
addresses: ["00:00:00:D0:28:68 10.244.0.18"]
port virt-api-748cfcdb48-vz6fk.kubevirt
addresses: ["00:00:00:D8:BF:D4 10.244.0.20"]
port nginx.default
addresses: ["00:00:00:F3:E4:1C 10.244.0.5"]
port virt-controller-5756d69f5f-ffs5r.kubevirt
addresses: ["00:00:00:12:42:FC 10.244.0.23"]
port cdi-deployment-7d587548cd-w55fh.cdi
addresses: ["00:00:00:D3:7E:DF 10.244.0.34"]
port cdi-uploadproxy-6c64d5cd59-5lvkc.cdi
addresses: ["00:00:00:B2:C7:B7 10.244.0.35"]
port virt-operator-5fcd4ff76f-j6x24.kubevirt
addresses: ["00:00:00:5A:C9:3E 10.244.0.7"]
port cdi-apiserver-c6cdc9489-g57k4.cdi
addresses: ["00:00:00:11:AF:5B 10.244.0.33"]
port virt-controller-5756d69f5f-lmkpk.kubevirt
addresses: ["00:00:00:CE:9F:04 10.244.0.21"]
port virt-operator-5fcd4ff76f-h8sx6.kubevirt
addresses: ["00:00:00:C2:56:F1 10.244.0.6"]
port cert-manager-webhook-885b8ffcb-hrzjt.cert-manager
addresses: ["00:00:00:11:A1:8B 10.244.0.16"]
port hostpath-provisioner-operator-7c8c499c77-46vk8.hostpath-provisioner
addresses: ["00:00:00:1D:58:B7 10.244.0.17"]
port coredns-6d4b75cb6d-l6cjw.kube-system
addresses: ["00:00:00:A6:76:70 10.244.0.2"]
port kube-ovn-pinger-tb5t7.kube-system
addresses: ["00:00:00:A7:BB:47 10.244.0.4"]
port cdi-operator-997bdf879-ht6kx.cdi
addresses: ["00:00:00:DD:1E:29 10.244.0.32"]
port ovn-default-ovn-cluster
type: router
router-port: ovn-cluster-ovn-default
switch 0629dbe7-beab-45f1-8d42-0ce2f2a89a79 (mail263)
port localnet.mail263
type: localnet
addresses: ["unknown"]
port mailserver.mail263
addresses: ["00:00:00:68:56:C6 172.16.3.250"]
switch 73e8db33-6e71-4f1e-af8a-516afaf13019 (join)
port node-ubuntuserver
addresses: ["00:00:00:B0:35:E1 100.64.0.2"]
port join-ovn-cluster
type: router
router-port: ovn-cluster-join
router e2c95d29-f93b-4f70-95b7-0f1c5e3c982a (ovn-cluster)
port ovn-cluster-join
mac: "00:00:00:E2:21:46"
networks: ["100.64.0.1/16"]
port ovn-cluster-ovn-default
mac: "00:00:00:C9:39:03"
networks: ["10.244.0.1/16"]
sukai@ubuntuserver:~$
创建虚拟机
因为我们需要一个CentOS 6.8操作系统，这个古老的系统因为驱动原因无法在现在的服务器硬件上安装，所以我们通过虚拟机来解决这个问题。当然使用虚拟化也需要解决一些新的虚拟化技术兼容CentOS 6.8的问题。
上传虚拟机镜像
sukai@ubuntuserver:/etc/kubernetes/manifests$ wget https://cloud.centos.org/centos/6/images/CentOS-6-x86_64-GenericCloud-1605.qcow2
sukai@ubuntuserver:/etc/kubernetes/manifests$ kubectl -n cdi get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cdi-api ClusterIP 10.211.69.105 <none> 443/TCP 20h
cdi-prometheus-metrics ClusterIP 10.211.27.150 <none> 8080/TCP 20h
cdi-uploadproxy ClusterIP 10.211.41.23 <none> 443/TCP 20h
sukai@ubuntuserver:/etc/kubernetes/manifests$ virtctl image-upload dv centos68 -vm-disk --size=10Gi -n mail263 --storage-class hostpath-csi --image-path=/home/sukai/CentOS-6-x86_64-GenericCloud-1907.qcow2 --uploadproxy-url=https://10.211.41.23 --insecure --wait-secs=240
创建虚拟机磁盘的PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: centos-mail263-disk
namespace: mail263
spec:
accessModes:
- ReadWriteOnce
storageClassName: hostpath-csi
resources:
requests:
storage: 100Gi
创建虚拟机
1，CPU model使用host-passthrough，不然会性能告警提示：Performance Events: unsupported p6 CPU model 58 no PMU driver, software events only.
2，disk的bus磁盘设备类型使用sata
3，网卡设备类型使用e1000
4，初始化配置用户密码
---
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
labels:
kubevirt.io/vm: mailserver
name: mailserver
namespace: mail263
spec:
running: false
template:
metadata:
labels:
kubevirt.io/vm: mailserver
spec:
domain:
cpu:
model: host-passthrough
devices:
disks:
- name: osdisk
disk:
bus: sata
- disk:
bus: sata
name: cloudinitdisk
- name: datadisk
disk:
bus: sata
interfaces:
- name: default
model: e1000
bridge: {}
resources:
requests:
cpu: "8"
memory: 16Gi
networks:
- name: default
pod: {}
terminationGracePeriodSeconds: 0
volumes:
- name: osdisk
dataVolume:
name: centos68
- name: cloudinitdisk
cloudInitNoCloud:
userData: |-
#cloud-config
user: centos
password: password
ssh_pwauth: True
chpasswd: { expire: False }
- name: datadisk
persistentVolumeClaim:
claimName: centos-mail263-disk
启动虚拟机
sukai@ubuntuserver:/etc/kubernetes/manifests$ virtctl -n mail263 start mailserver
sukai@ubuntuserver:/etc/kubernetes/manifests$ kubectl -n mail263 get vms
NAME AGE STATUS READY
mailserver 4h18m Running True
sukai@ubuntuserver:/etc/kubernetes/manifests$ kubectl -n mail263 get vmi
NAME AGE PHASE IP NODENAME READY
mailserver 133m Running 172.16.3.250 ubuntuserver True
sukai@ubuntuserver:/etc/kubernetes/manifests$
Console登录虚拟机
sukai@ubuntuserver:/etc/kubernetes/manifests$ virtctl console -n mail263 mailserver
Successfully connected to mailserver console. The escape sequence is ^]
login: root
Password:
Last login: Thu Jun 9 06:31:57 from 172.16.3.25
[root@mailserver ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:00:00:68:56:c6 brd ff:ff:ff:ff:ff:ff
inet 172.16.3.250/24 brd 172.16.3.255 scope global eth0
inet 192.168.10.95/24 brd 192.168.10.255 scope global eth0:0
inet6 fe80::200:ff:fe68:56c6/64 scope link
valid_lft forever preferred_lft forever
[root@mailserver ~]# ip route
172.16.3.0/24 dev eth0 proto kernel scope link src 172.16.3.250
192.168.10.0/24 dev eth0 proto kernel scope link src 192.168.10.95
10.65.0.0/16 via 192.168.10.99 dev eth0
default via 172.16.3.2 dev eth0 proto static
[root@mailserver ~]# ping www.sina.com
PING ww1.sinaimg.cn.w.alikunlun.com (163.181.42.223) 56(84) bytes of data.
64 bytes from 163.181.42.223: icmp_seq=1 ttl=55 time=75.5 ms
--- ww1.sinaimg.cn.w.alikunlun.com ping statistics ---
2 packets transmitted, 1 received, 50% packet loss, time 1018ms
rtt min/avg/max/mdev = 75.505/75.505/75.505/0.000 ms
[root@mailserver ~]# ping www.sina.com -c 1
PING ww1.sinaimg.cn.w.alikunlun.com (163.181.42.223) 56(84) bytes of data.
64 bytes from 163.181.42.223: icmp_seq=1 ttl=55 time=75.1 ms
--- ww1.sinaimg.cn.w.alikunlun.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 78ms
rtt min/avg/max/mdev = 75.104/75.104/75.104/0.000 ms
[root@mailserver ~]# sukai@ubuntuserver:/etc/kubernetes/manifests$
sukai@ubuntuserver:/etc/kubernetes/manifests$
容器与虚拟机的网络通信
sukai@ubuntuserver:~$ kubectl exec -it nginx -- /bin/sh
/ # ping 172.16.3.250 -c1
PING 172.16.3.250 (172.16.3.250): 56 data bytes
64 bytes from 172.16.3.250: seq=0 ttl=62 time=0.586 ms
--- 172.16.3.250 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.586/0.586/0.586 ms
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
18: eth0@if19: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1400 qdisc noqueue state UP
link/ether 00:00:00:f3:e4:1c brd ff:ff:ff:ff:ff:ff
inet 10.244.0.5/16 brd 10.244.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::200:ff:fef3:e41c/64 scope link
valid_lft forever preferred_lft forever
/ # ip r
default via 10.244.0.1 dev eth0
10.244.0.0/16 dev eth0 scope link src 10.244.0.5
/ #
sukai@ubuntuserver:~$
注意
Kube-OVN在使用Underlay网络模式或者在自定义VPC中，都需要Kubernetes节点主机配置相应的网络安全策略，比如PortSecurity, 允许混杂模式，允许MAC更改，允许伪传输等。在使用VMware和Openstack虚拟机部署Kubernetes集群时，尤其需要注意。
总结
Kube-OVN提供了Underlay网络模式，在此模式下，提供了容器和虚拟机接入物理网络的能力，并且能够让容器与虚拟机在默认的VPC下进行通信。