Overview

K3s is small because a lot different not-mandatory components of Kubernetes are not included by default. As example K3s has only one available storageclass - local-path.

# kubectl get storageclass 
NAME                   PROVISIONER                                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path (default)   rancher.io/local-path                          Delete          WaitForFirstConsumer   false                  33d

Default local-path is more than enough to play around with storagesclass, persistent volumes and persistent volume claims.

When there is cluster of several nodes, volumes (which are used by pods) have to be external to nodes, otherwise each pod will have different data when it runs on different node (as it will use local volume). In home environment the most simple external storage can be NAS (network attached storage) accessible via NFS/CIFS/AFS protocols. So the same volume which is accessible via NFS can be mounted on all nodes and in that case local-path storageclass can be used still.

I started from there using NAS via NFS and it worked fine for most of the cases. However it means you need to make node configuration external to K3s - /etc/fstab has to be updated and managed, required mount modules have to be install on all nodes and the same volume has to be mounted on all nodes and if something goes wrong - you need to investigate the problem outside of K3s.

So next step was - discovery of nfs-client, it brings nfs-client as new storageclass into kubernetes cluster and allows to use it for persistent volume claims (pvc). nfs-client class can create (and manage afterwards) NFS volumes based on defined configuration and specified pvc-requests.

Installation

Before to install nfs-client into kubernetes cluster your NFS server has to be configured the following way:

  • NFS should be accessible to all nodes via ip or hostname (192.168.8.20 on example below)
  • each node has to have write permissions to nfs-path which will be used as base for all kubernetes nfs-volumes (/mnt/kubcluster on example below)

The easiest way to install nfs-client is to use helm and create storage namespace for it before:

helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
helm search repo | grep nfs-subdir
nfs-subdir-external-provisioner/nfs-subdir-exte...	4.0.4        	4.0.0                  	nfs-subdir-external-provisioner is an automatic...

kubectl create namespace storage

helm install -n storage --set nfs.server=192.168.8.20 --set nfs.path=/mnt/kubcluster storage nfs-subdir-external-provisioner/nfs-subdir-external-provisioner

As result you should see new storageclass available and pod running:

# kubectl get pod -n storage
NAME                                                     READY   STATUS    RESTARTS   AGE
storage-nfs-subdir-external-provisioner-dd4dbdf5-5rmrw   1/1     Running   0          9m57s
# kubectl get storageclass
NAME                   PROVISIONER                                             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path (default)   rancher.io/local-path                                   Delete          WaitForFirstConsumer   false                  6d17h
nfs-client             cluster.local/storage-nfs-subdir-external-provisioner   Delete          Immediate              true                   10m

Usage

Usage is very simple - just specify volume.beta.kubernetes.io/storage-class: nfs-client as annotations in metadata for pvc you create. As example the following yaml-file will create grafana-nfs-pvc which will automatically create pv with automatically generated name:

# cat monitoring/grafana-nfs-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: "monitoring"
  name: "grafana-nfs-pvc"
  annotations:
    volume.beta.kubernetes.io/storage-class: nfs-client
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: "50Mi"

# kubectl get pvc grafana-nfs-pvc -n monitoring
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
grafana-nfs-pvc   Bound    pvc-1f903c25-823b-466d-b226-800c1e0a05d2   50Mi       RWO            nfs-client     27d

# kubectl get pv pvc-1f903c25-823b-466d-b226-800c1e0a05d2
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                        STORAGECLASS   REASON   AGE
pvc-1f903c25-823b-466d-b226-800c1e0a05d2   50Mi       RWO            Delete           Bound    monitoring/grafana-nfs-pvc   nfs-client              27d

This is what exists on NFS-server:

# ls -l /mnt/kubcluster | grep pvc-1f903c25-823b-466d-b226-800c1e0a05d2
drwxrwxrwx  4 root root 4096 Feb 20 12:38 monitoring-grafana-nfs-pvc-pvc-1f903c25-823b-466d-b226-800c1e0a05d

If pvc is deleted volume will be renamed into archived, but data will be left on NFS share. It is managed by Reclaim policy setting in storageclass (for more details).