由 SuKai August 23, 2022
今天我们一起来看一下Kubernetes如何使用不同厂商存储的，了解一下CSI基本的概念。
基本概念
CSI
容器存储接口Container Storage Interface规范，定义了一个行业标准，使用存储厂商开发的插件可以在多个容器编排系统运行。
PV
持久卷PersistentVolume，Kubernetes集群中的一块存储卷，提供给Pod容器存储数据，通过卷插件(provisioner)来管理，拥有独立于Pod的生命周期。
PVC
持久卷声明PersistentVolumeClaim，用户对存储的资源请求声明。
StorageClass
存储类，集群中可以提供的存储类型。管理员可以创建不同的存储类来对应不同的存储资源，卷插件和策略。
Porvisioner
卷插件根据资料请求信息完成存储设备的卷资源操作。
dynamic provisioning
动态卷配置，基于StorageClass的自动化存储资源的生命周期管理，按用户需求自动动态创建和调整存储资源，用户无需过多关注存储的管理。
下面我们一起来看一下，如何使用动态卷配置。
动态卷配置流程
1，创建一个StorageClass，指定卷插件，回收策略，存储卷绑定模式。
2，用户创建一个PVC持久卷声明，指定卷使用的存储类，读写模式，资源大小，卷类型。
3，卷插件创建持久卷。
4，Kubernetes绑定持久卷和持久卷声明。
5，容器启动时，挂载到容器中使用。
使用示例
创建CSIDriver
创建一个CSI存储插件，当我们的插件注册时会使用相同的名称，Kubernetes的组件根据名称调用插件服务。
---
# Create the CSI Driver object
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
name: local.csi.sukai.io
spec:
# do not require volumeattachment
attachRequired: false
podInfoOnMount: true
storageCapacity: true
创建StorageClass
指定provisioner为local.csi.sukai.io
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: openebs-lvmsc
allowVolumeExpansion: true
parameters:
volgroup: "lvmvg"
provisioner: local.csi.sukai.io
默认的回收策略为Delete删除策略，当与PV绑定的PVC被删除的时候，删除PV存储资源。另一种策略为Retain保留策略。持久卷绑定模式为Immediate即刻绑定，立即创建持久卷并将其绑定到持久卷声明。WaitForFirstConsumer为首次使用时绑定，第一次被容器组使用时，才创建持久卷，并将其绑定到持久卷声明。
sukai@sukai:~$ kubectl get sc -o wide
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
hostpath-csi kubevirt.io.hostpath-provisioner Delete WaitForFirstConsumer false 76d
openebs-lvmsc local.csi.sukai.io Delete Immediate true 22h
sukai@sukai:~$
创建PVC
持久卷声明指定存储类为openebs-lvmsc,
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: block-claim
spec:
volumeMode: Block
storageClassName: openebs-lvmsc
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 4Gi
查看PVC
sukai@sukai:~$ kubectl get pvc -o wide
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE
block-claim Bound pvc-67484f57-29b4-4b56-8322-09487f04bba5 4Gi RWO openebs-lvmsc 22h Block
sukai@sukai:~$
查看自动创建的PV
sukai@sukai:~$ kubectl get pv -o wide
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE VOLUMEMODE
data-sukai-sukai-02 50Gi RWO Retain Bound sukai/sukai-02 14d Filesystem
pvc-67484f57-29b4-4b56-8322-09487f04bba5 4Gi RWO Delete Bound default/block-claim openebs-lvmsc 22h Block
查看一下存储插件的调用过程
这里有两个容器组，一个是openebs-lvm-node，一个是sukai-lvm-controller-0。
sukai@sukai:~$ kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
openebs-lvm-node-dxtz9 2/2 Running 0 22h
sukai-lvm-controller-0 5/5 Running 0 22h
查看一下openebs-lvm-node日志，Kubernetes通过GRPC调用了插件的信息和节点的信息
sukai@sukai:~$ kubectl -n kube-system logs openebs-lvm-node-dxtz9 openebs-lvm-plugin
I0822 08:07:29.344418 1 main.go:68] LVM Driver Version :- 1.0
I0822 08:07:29.344444 1 main.go:69] DriverName: local.csi.sukai.io Plugin: agent EndPoint: unix:///plugin/csi.sock NodeID: ubuntuserver
I0822 08:07:29.344461 1 driver.go:26] enabling volume access mode: SINGLE_NODE_WRITER
I0822 08:07:29.344674 1 grpc.go:150] Listening for connections on address: &net.UnixAddr{Name:"//plugin/csi.sock", Net:"unix"}
I0822 08:07:30.250073 1 grpc.go:51] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I0822 08:07:30.252047 1 grpc.go:60] GRPC response: {"name":"local.csi.sukai.io","vendor_version":"1.0"}
I0822 08:07:30.868233 1 grpc.go:51] GRPC call: /csi.v1.Node/NodeGetInfo requests {}
I0822 08:07:30.868350 1 grpc.go:60] GRPC response: {"accessible_topology":{"segments":{"sukai.io/nodename":"ubuntuserver"}},"node_id":"ubuntuserver"}
查看sukai-lvm-controller-0日志，Kubernetes通过GRPC调用了插件的基本信息，支持的能力，调用了CreateVolume创建持久卷。
sukai@sukai:~$ kubectl -n kube-system logs sukai-lvm-controller-0 openebs-lvm-plugin
W0822 08:07:31.083122 1 volume.go:27] NodeID environment variable not set
I0822 08:07:31.084830 1 main.go:68] LVM Driver Version :- 1.0
I0822 08:07:31.084842 1 main.go:69] DriverName: local.csi.sukai.io Plugin: controller EndPoint: unix:///var/lib/csi/sockets/pluginproxy/csi.sock NodeID:
I0822 08:07:31.084856 1 driver.go:26] enabling volume access mode: SINGLE_NODE_WRITER
I0822 08:07:31.085050 1 grpc.go:150] Listening for connections on address: &net.UnixAddr{Name:"//var/lib/csi/sockets/pluginproxy/csi.sock", Net:"unix"}
I0822 08:07:31.728688 1 grpc.go:51] GRPC call: /csi.v1.Identity/Probe requests {}
I0822 08:07:31.731391 1 grpc.go:60] GRPC response: {}
I0822 08:07:31.732823 1 grpc.go:51] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I0822 08:07:31.732950 1 grpc.go:60] GRPC response: {"name":"local.csi.sukai.io","vendor_version":"1.0"}
I0822 08:07:31.733658 1 grpc.go:51] GRPC call: /csi.v1.Identity/GetPluginCapabilities requests {}
I0822 08:07:31.733742 1 grpc.go:60] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
I0822 08:07:31.735101 1 grpc.go:51] GRPC call: /csi.v1.Controller/ControllerGetCapabilities requests {}
I0822 08:07:31.735131 1 grpc.go:60] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":4}}}]}
I0822 08:07:31.799109 1 grpc.go:51] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I0822 08:07:31.799257 1 grpc.go:60] GRPC response: {"name":"local.csi.sukai.io","vendor_version":"1.0"}
I0822 08:07:31.801741 1 grpc.go:51] GRPC call: /csi.v1.Identity/Probe requests {}
I0822 08:07:31.802523 1 grpc.go:60] GRPC response: {}
I0822 08:07:31.802981 1 grpc.go:51] GRPC call: /csi.v1.Controller/ControllerGetCapabilities requests {}
I0822 08:07:31.803051 1 grpc.go:60] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":4}}}]}
I0822 08:07:32.010074 1 grpc.go:51] GRPC call: /csi.v1.Identity/Probe requests {}
I0822 08:07:32.010157 1 grpc.go:60] GRPC response: {}
I0822 08:07:32.011294 1 grpc.go:51] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I0822 08:07:32.011438 1 grpc.go:60] GRPC response: {"name":"local.csi.sukai.io","vendor_version":"1.0"}
I0822 08:07:32.012080 1 grpc.go:51] GRPC call: /csi.v1.Identity/GetPluginCapabilities requests {}
I0822 08:07:32.012128 1 grpc.go:60] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
I0822 08:07:32.013280 1 grpc.go:51] GRPC call: /csi.v1.Controller/ControllerGetCapabilities requests {}
I0822 08:07:32.013367 1 grpc.go:60] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":4}}}]}
I0822 08:10:08.481307 1 grpc.go:51] GRPC call: /csi.v1.Controller/CreateVolume requests {"accessibility_requirements":{"preferred":[{"segments":{"sukai.io/nodename":"ubuntuserver"}}],"requisite":[{"segments":{"sukai.io/nodename":"ubuntuserver"}}]},"capacity_range":{"required_bytes":4294967296},"name":"pvc-67484f57-29b4-4b56-8322-09487f04bba5","parameters":{"csi.storage.k8s.io/pv/name":"pvc-67484f57-29b4-4b56-8322-09487f04bba5","csi.storage.k8s.io/pvc/name":"block-claim","csi.storage.k8s.io/pvc/namespace":"default","volgroup":"lvmvg"},"volume_capabilities":[{"AccessType":{"Block":{}},"access_mode":{"mode":1}}]}
I0822 08:10:08.481904 1 controller.go:62] request create volume default/block-claim
I0822 08:10:08.481920 1 grpc.go:60] GRPC response: {"volume":{"capacity_bytes":4294967296,"volume_id":"pvc-67484f57-29b4-4b56-8322-09487f04bba5"}}
查看csinode节点信息，可以看到存储驱动列表
sukai@sukai:~$ kubectl get CSINode -o wide
NAME DRIVERS AGE
ubuntuserver 2 76d
sukai@sukai:~$ kubectl get CSINode ubuntuserver -o yaml
apiVersion: storage.k8s.io/v1
kind: CSINode
metadata:
annotations:
storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd
creationTimestamp: "2022-06-07T13:10:43Z"
managedFields:
- apiVersion: storage.k8s.io/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:storage.alpha.kubernetes.io/migrated-plugins: {}
f:ownerReferences:
.: {}
k:{"uid":"def564d2-2f59-482e-9ab7-b3562fd92343"}: {}
f:spec:
f:drivers:
k:{"name":"kubevirt.io.hostpath-provisioner"}:
.: {}
f:name: {}
f:nodeID: {}
f:topologyKeys: {}
k:{"name":"local.csi.sukai.io"}:
.: {}
f:name: {}
f:nodeID: {}
f:topologyKeys: {}
manager: kubelet
operation: Update
time: "2022-08-22T08:07:31Z"
name: ubuntuserver
ownerReferences:
- apiVersion: v1
kind: Node
name: ubuntuserver
uid: def564d2-2f59-482e-9ab7-b3562fd92343
resourceVersion: "61158650"
uid: d80d96e4-34f7-44a3-a6fe-3e1fc4789765
spec:
drivers:
- name: kubevirt.io.hostpath-provisioner
nodeID: ubuntuserver
topologyKeys:
- topology.hostpath.csi/node
- name: local.csi.sukai.io
nodeID: ubuntuserver
topologyKeys:
- sukai.io/nodename
sukai@sukai:~$ kubectl get node ubuntuserver -o yaml
apiVersion: v1
kind: Node
metadata:
annotations:
csi.volume.kubernetes.io/nodeid: '{"kubevirt.io.hostpath-provisioner":"ubuntuserver","local.csi.sukai.io":"ubuntuserver"}'
Kubernetes如何调用CSI驱动
上面的过程可以让我们对Kubernetes如何使用CSI驱动有了大体了解，那么Kubernetes又是如何来与CSI插件一起工作的呢？下面我们来看一下Kubernetes如何通过各个组件支持不同厂商存储的CSI插件。
为了能够让各个存储厂商只关注存储相关代码开发，CSI规范将存储工作分为两部分编排调度系统Container Orchestration system(CO)和插件，这两部分服务通过RPC进行通信，在部署时CO组件作为Sidecar的方式和驱动插件一起工作。所以存储厂商根据CSI规范开发对应的服务接口，那么Kubernetes就可以使用该厂商存储设备了。
+-------------------------------------------+
| |
| +------------+ +------------+ |
| | CO | gRPC | Controller | |
| | +--+--------> Plugin | |
| +------------+ | +------------+ |
| | |
| | |
| | +------------+ |
| | | Node | |
| +--------> Plugin | |
| +------------+ |
| |
+-------------------------------------------+
编排调度系统Container Orchestration system(CO)主要组件
external-provisioner
external-provisioner监控PVC资源，根据指定的StorageClass存储，调用ControllerService的存储卷操作接口CreateVolume, DeleteVolume，实现动态配置。
external-attacher
external-attacher监控VolumeAttachment资源，调用ControllerService的ControllerPublishVolume，ControllerUnpublishVolume接口，实现存储卷挂接到主机节点。
external-snapshotter
external-snapshotter监控SnapshotClass的VolumeSnapshot资源，调用ControllerService的CreateSnapshot，DeleteSnapshot接口，实现存储卷的镜像操作。
external-resizer
external-resizer监控PVC资源的容量变更，调用ControllerService的ControllerExpandVolume接口，实现存储卷的容量大小变更。
node-driver-registrar
node-driver-registrar获取CSI插件信息，在kubelet上注册驱动。kubelet负责发布NodeGetInfo, NodeStageVolume, NodePublishVolume调用。
插件服务接口
插件服务只需要根据CSI接口规范，完成下面服务接口的开发，就可以将存储系统接入到Kubernetes中使用了。
控制服务
// ControllerServer is the server API for Controller service.
type ControllerServer interface {
CreateVolume(context.Context, *CreateVolumeRequest) (*CreateVolumeResponse, error)
DeleteVolume(context.Context, *DeleteVolumeRequest) (*DeleteVolumeResponse, error)
ControllerPublishVolume(context.Context, *ControllerPublishVolumeRequest) (*ControllerPublishVolumeResponse, error)
ControllerUnpublishVolume(context.Context, *ControllerUnpublishVolumeRequest) (*ControllerUnpublishVolumeResponse, error)
ValidateVolumeCapabilities(context.Context, *ValidateVolumeCapabilitiesRequest) (*ValidateVolumeCapabilitiesResponse, error)
ListVolumes(context.Context, *ListVolumesRequest) (*ListVolumesResponse, error)
GetCapacity(context.Context, *GetCapacityRequest) (*GetCapacityResponse, error)
ControllerGetCapabilities(context.Context, *ControllerGetCapabilitiesRequest) (*ControllerGetCapabilitiesResponse, error)
CreateSnapshot(context.Context, *CreateSnapshotRequest) (*CreateSnapshotResponse, error)
DeleteSnapshot(context.Context, *DeleteSnapshotRequest) (*DeleteSnapshotResponse, error)
ListSnapshots(context.Context, *ListSnapshotsRequest) (*ListSnapshotsResponse, error)
ControllerExpandVolume(context.Context, *ControllerExpandVolumeRequest) (*ControllerExpandVolumeResponse, error)
ControllerGetVolume(context.Context, *ControllerGetVolumeRequest) (*ControllerGetVolumeResponse, error)
}
节点服务
// NodeServer is the server API for Node service.
type NodeServer interface {
NodeStageVolume(context.Context, *NodeStageVolumeRequest) (*NodeStageVolumeResponse, error)
NodeUnstageVolume(context.Context, *NodeUnstageVolumeRequest) (*NodeUnstageVolumeResponse, error)
NodePublishVolume(context.Context, *NodePublishVolumeRequest) (*NodePublishVolumeResponse, error)
NodeUnpublishVolume(context.Context, *NodeUnpublishVolumeRequest) (*NodeUnpublishVolumeResponse, error)
NodeGetVolumeStats(context.Context, *NodeGetVolumeStatsRequest) (*NodeGetVolumeStatsResponse, error)
NodeExpandVolume(context.Context, *NodeExpandVolumeRequest) (*NodeExpandVolumeResponse, error)
NodeGetCapabilities(context.Context, *NodeGetCapabilitiesRequest) (*NodeGetCapabilitiesResponse, error)
NodeGetInfo(context.Context, *NodeGetInfoRequest) (*NodeGetInfoResponse, error)
}
标识服务
// IdentityServer is the server API for Identity service.
type IdentityServer interface {
GetPluginInfo(context.Context, *GetPluginInfoRequest) (*GetPluginInfoResponse, error)
GetPluginCapabilities(context.Context, *GetPluginCapabilitiesRequest) (*GetPluginCapabilitiesResponse, error)
Probe(context.Context, *ProbeRequest) (*ProbeResponse, error)
}
CSI容器存储接口规范，将存储厂商驱动开发更加简单化标准化，方便存储厂商接入Kubernetes系统。后面我们将一起看看OpenEBS如何实现LVM存储的。