Deploying K8s on Raspberry Pi4 with Hypriot and Cloud-Init

I was reading the most excellent “Kubernetes Up & Running” book by Brendon Burns, Joe Beda and Kelsey Hightower earlier this month and decided to build a small K8s cluster on Raspberry Pi 4 (2GB) to learn. In the appendix they have a short chapter on how to do this, but there is a fair amount of detail left for the reader to divine. What follows are my notes about my experience of deploying a four node system. I hope you find it informative and useful!

The authors recommend using Hypriot since it comes with Docker built in. You can download the bits off their GitHub site here: https://github.com/hypriot/image-builder-rpi/releases Since I was going to be building four boxes I wanted to use cloud-init to build the images. The process goes something like this.

  1. Download latest Hypriot image builder file (hypriotos-rpi-v1.11.5.img.zip as of this writing)
  2. Unzip and burn the image file to a Micro-SD card
  3. Create a specific cloud-init configuration for each system
  4. Replace the ‘user-data’ file in the root of the SD card with your specially crafted YML file
  5. Boot the Raspberry with modified SD card and if everything works out, you have a working, pre-configured system in about five minutes!

Step #3 is really the only that requires a significant amount of work 😉

Building a cloud-init Script

So lets break down what I ended up with, section by section. The full example file is at the end of this article.

#cloud-config Each cloud-init file begins with the #cloud-config line. Some YAML linters will replace this top line with “—” in which case cloud-init will warn you “File user-data needs to begin with “#cloud-config”” when you try to run it.
hostname: What you want the entry in /etc/hostname to reflect
manage_etc_hosts: {true, false} By default this is set to in the Hypriot examples. I wanted to manually manage my hosts file so I set this to false. You can read up on how to work with the “true” setting in the cloud-init documentation.
package_update: {true, false} I have this set to false since I specifically invoke an apt update while installing the K8s tools.
package_upgrade: {true, false} I have this set to false so I can manually control what gets loaded on the machines.
runcmd: This specifies commands that cloud-init should invoke during system customization. The “systemctl restart avahi-daemon” and “ifup wlan 0” entries are there to get the wireless card up and running and the system name advertised out on the network. The “systemctl daemon-reload” was necessary to get the DNS information updated that I added to /etc/systemd/resolved.conf. The last four commands are there to get the administrative tools for K8s downloaded and installed so they are ready to rock.

#cloud-config
hostname: kubernetes
manage_etc_hosts: false
package_update: false
package_upgrade: false
runcmd: 
  - "systemctl restart avahi-daemon"
  - "ifup wlan0"
  - "ifup eth0"
  - "systemctl daemon-reload"
  - "curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -"
  - "echo 'deb https://apt.kubernetes.io/ kubernetes-xenial main' >> /etc/apt/sources.list.d/kubernetes.list"
  - "apt-get update"
  - "apt-get install -y kubelet kubeadm kubectl"
 

The users section allows you to create users. You can read more about that here. I specified public ssh keys in this section so I can SSH into and between the K8s nodes without using a password. No, those aren’t my actual SSH keys 😉

users: 
  - 
    chpasswd: 
      expire: false
    gecos: "Hypriot Pirate"
    groups: "users,docker,video"
    lock_passwd: false
    name: aaron
    plain_text_passwd: password
    shell: /bin/bash
    ssh-authorized-keys: 
      - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpVazw0Hsh4p9Uuq/pM3HP0A3tGJuTTO4sRxFluu7byVVDMevhMLFZ80yEIS809jfiDM5YLc/o96GJhMSbrJ4eGa3sn1k9jGvXEXGAvPKsZk92DQAhubueWwOns0Pd/NccFa8vlgcHzfrxKNuI6ZtXsESM+2aIBV8LfWYx0s/StNSH09LwUnGkVkWVPivxJSjGWGtA/YAt4URfUgbpYnm40iJWuJZbxh1g8qAEGt2uNPEi5OBQOBWfpX5Ud/VI3YvYKjn2/1LpaxNSsNts9UJ5163Y8kXTkUt/iZZT1atA+IV6FsoaMLYqBfsAH7ChTr0h9MgpcJLRmWP/uAGLrfvL [email protected]"
      - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7fvwjRVd3y1AKFdBq9i1ja3MvxXxalLC7D1Ml6CH1cPBpgxMFnJPZTUza5fdY1i+NhBs4EqM73K4j6iSSNry7qVd+sL0rgmY7lvuIqcAG87R73bPxq84lU/RsqIDbAnFsXRcZYEX6xa/GsP6bFVyU3w9wWtMV7eiLjzFwIIjjFNheVt1Badn+ZnYf7X/s1uriXcTkArA28aD8uv5HB3VRVgUiLGMg1bRcDNkL+/lTVTR28a3sz9qFGeiBkOKnw8ymKYp6jGzIlobqGciZlEImlcwDPGXT3CuD4yZ9IYlk1/Jd9UwxxgJgk1vD+0QRJrHP02jBQHdVFAfy8rix5erl azuread\[email protected]"
    ssh_pwauth: true
    sudo: "ALL=(ALL) NOPASSWD:ALL"

The write_files: section allows you to write out files to the system during customization. Each stanza is started with the “-” character followed by “content:” and then whatever you want written to the file. This is concluded with the “path:” statement indicating where in the file system the data should be written.

Since I’m using wireless for the “public” interfaces of my K8s nodes I needed to configure some details in three files to get wireless working. Those files are:

  • /etc/network/interfaces.d/wlan0
  • /etc/wpa_supplicant/wpa_supplicant.conf
  • /etc/network/interfaces

The first file I’m writing out is /etc/network/interfaces.d/wlan0. This just hot-plug on the interface and tells wpa-conf where to find its configuration file. You can read all about that here.

write_files: 
  - 
    content: |
        allow-hotplug wlan0
        wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf
    path: /etc/network/interfaces.d/wlan0

The /etc/wpa_supplicant/wpa_supplicant.conf file contains all the details needed to connect to the wireless network. Details can be found here. You can use this Raspberry Pi WiFi Config Generator to get the correct details for the network={} details. Just for the sake of completeness:

proto could be either RSN (WPA2) or WPA (WPA1).
key_mgmt could be either WPA-PSK (most probably) or WPA-EAP (enterprise networks)
pairwise could be either CCMP (WPA2) or TKIP (WPA1)
auth_alg is most probably OPEN, other options are LEAP and SHARED

  - 
    content: |
        country=US
        ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
        update_config=1
        network={
        ssid="CasaDePatten"
        psk="wireless"
        proto=RSN
        key_mgmt=WPA-PSK
        pairwise=CCMP
        auth_alg=OPEN
        }
    path: /etc/wpa_supplicant/wpa_supplicant.conf

The /etc/network/interfaces file contains the static IP addresses I’m assigning for my wireless and wired networks.

  - 
    content: |
        auto wlan0
        allow-hotplug wlan0
        iface wlan0 inet static
        address 192.168.3.50
        netmask 255.255.255.0
        gateway 192.168.3.1
        wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf
        
        auto eth0
        iface eth0 inet static
        address 10.0.0.50
        netmask 255.255.255.0
    path: /etc/network/interfaces

The /etc/systemd/resolved.conf file has my DNS settings specified. Since systemd is managing DNS, you can’t just edit the /etc/resolv.conf file directly or it will just get overwritten. You can read more here about systemd and DNS configuration.

I didn’t try this particular example, but this may be another way to do the same thing. https://cloudinit.readthedocs.io/en/latest/topics/examples.html#configure-an-instances-resolv-conf

  - 
    content: |
        [Resolve]
        DNS=192.168.3.1
        FallbackDNS=1.1.1.1
    path: /etc/systemd/resolved.conf

The last file I’m writing out is /etc/hosts. Remember that if you have manage_etc_hosts: true set in your user-data file, this will get overwritten.

  - 
    content: |
        127.0.0.1 localhost        
        10.0.0.50 kubernetes.cluster.home kubernetes
        10.0.0.51 node-1.cluster.home node-1
        10.0.0.52 node-2.cluster.home node-2
        10.0.0.53 node-3.cluster.home node-3
    path: /etc/hosts

Full cloud-init Example

So here is an example of the full cloud-init script I ended up building for my master node. Don’t worry, passwords and ssh keys have been changed to protect the innocent.

#cloud-config
hostname: kubernetes
manage_etc_hosts: false
package_update: false
package_upgrade: false
runcmd: 
  - "systemctl restart avahi-daemon"
  - "ifup wlan0"
  - "ifup eth0"
  - "systemctl daemon-reload"
  - "curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -"
  - "echo 'deb https://apt.kubernetes.io/ kubernetes-xenial main' >> /etc/apt/sources.list.d/kubernetes.list"
  - "apt-get update"
  - "apt-get install -y kubelet kubeadm kubectl kubernetes-cni"

users: 
  - 
    chpasswd: 
      expire: false
    gecos: "Hypriot Pirate"
    groups: "users,docker,video"
    lock_passwd: false
    name: aaron
    plain_text_passwd: password
    shell: /bin/bash
    ssh-authorized-keys: 
      - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpVazw0Hsh4p9Uuq/pM3HP0A3tGJuTTO4sRxFluu7byVVDMevhMLFZ80yEIS809jfiDM5YLc/o96GJhMSbrJ4eGa3sn1k9jGvXEXGAvPKsZk92DQAhubueWwOns0Pd/NccFa8vlgcHzfrxKNuI6ZtXsESM+2aIBV8LfWYx0s/StNSH09LwUnGkVkWVPivxJSjGWGtA/YAt4URfUgbpYnm40iJWuJZbxh1g8qAEGt2uNPEi5OBQOBWfpX5Ud/VI3YvYKjn2/1LpaxNSsNts9UJ5163Y8kXTkUt/iZZT1atA+IV6FsoaMLYqBfsAH7ChTr0h9MgpcJLRmWP/uAGLZRbL [email protected]"
      - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7fvwjRVd3y1AKFdBq9i1ja3MvxXxalLC7D1Ml6CH1cPBpgxMFnJPZTUza5fdY1i+NhBs4EqM73K4j6iSSNry7qVd+sL0rgmY7lvuIqcAG87R73bPxq84lU/RsqIDbAnFsXRcZYEX6xa/GsP6bFVyU3w9wWtMV7eiLjzFwIIjjFNheVt1Badn+ZnYf7X/s1uriXcTkArA28aD8uv5HB3VRVgUiLGMg1bRcDNkL+/lTVTR28a3sz9qFGeiBkOKnw8ymKYp6jGzIlobqGciZlEImlcwDPGXT3CuD4yZ9IYlk1/Jd9UwxxgJgk1vD+0QRJrHP02jBQHdVFAfy8rix5erl azuread\[email protected]"
      - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDR1sDqGyQFb35XO8NQQ+7VAzsLpV9v62uo1dSBFs4SHZ5Djwfl5mri/mxyqvbpg1PO8TiYd+ieNTdDFnxpCOz3uMTfHegbu9AFC5o78Qo16PHywiJSvhnGqdoitFkMek+qxCmOn3puCEAseDHJ+0q9eFNkM+7w8EOqEJ+2y94AOERj+dAhXRig4CDi1IO/gpPKl1w5SkQcu/+8Y6fAV8If1brkRAN0OW+jv41kD0cNPRbSxbZA+wADi8p9JlEYSY/vZyYCBQpE3pWwpZGC60O6RtTjJ8gKM+4BCQ3cjtTGaEB0zvNaRA3glS3w/Gv4M7kuedbOgCFq+bIw0UUFaKlD [email protected]"
    ssh_pwauth: true
    sudo: "ALL=(ALL) NOPASSWD:ALL"
write_files: 
  - 
    content: |
        allow-hotplug wlan0
        wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf
    path: /etc/network/interfaces.d/wlan0
  - 
    content: |
        country=US
        ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
        update_config=1
        network={
        ssid="CasaDePatten"
        psk="wireless"
        proto=RSN
        key_mgmt=WPA-PSK
        pairwise=CCMP
        auth_alg=OPEN
        }
    path: /etc/wpa_supplicant/wpa_supplicant.conf
  - 
    content: |
        auto wlan0
        allow-hotplug wlan0
        iface wlan0 inet static
        address 192.168.3.50
        netmask 255.255.255.0
        gateway 192.168.3.1
        wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf
        
        auto eth0
        iface eth0 inet static
        address 10.0.0.50
        netmask 255.255.255.0
    path: /etc/network/interfaces
  - 
    content: |
        [Resolve]
        DNS=192.168.3.1
        FallbackDNS=1.1.1.1
    path: /etc/systemd/resolved.conf
  - 
    content: |
        127.0.0.1 localhost        
        10.0.0.50 kubernetes.cluster.home kubernetes
        10.0.0.51 node-1.cluster.home node-1
        10.0.0.52 node-2.cluster.home node-2
        10.0.0.53 node-3.cluster.home node-3
    path: /etc/hosts

The images can be written to your SD cards using your favorite imaging tool. On Windows I use Win32DiskImager. On the Mac, I use the ‘flash’ utility provided by Hypriot. https://github.com/hypriot/flash

If you are using flash, you just grab the latest release and then run it like so:

[email protected] flash % flash --userdata ./configs/node-3.yml --bootconf ./sample/no-uart-config.txt ~/Downloads/hypriotos-rpi-v1.11.5.img.zip
Using cached image /tmp/hypriotos-rpi-v1.11.5.img

Is /dev/disk2 correct? y
Unmounting /dev/disk2 ...
Unmount of all volumes on disk2 was successful
Unmount of all volumes on disk2 was successful
Flashing /tmp/hypriotos-rpi-v1.11.5.img to /dev/rdisk2 ...
1.27GiB 0:00:27 [46.9MiB/s] [====================================================================================================>] 100%            
0+20800 records in
0+20800 records out
1363148800 bytes transferred in 27.655770 secs (49289852 bytes/sec)
Mounting Disk
Mounting /dev/disk2 to customize...
Copying ./sample/no-uart-config.txt to /Volumes/HypriotOS/config.txt ...
Copying cloud-init ./configs/node-3.yml to /Volumes/HypriotOS/user-data ...
Unmounting /dev/disk2 ...
"disk2" ejected.
Finished.

Troubleshooting

You will want to run your yaml file through a linter. Super handy for making sure your indentations are correct and that all the right markup is present. http://www.yamllint.com/ is a fine one but be advised, it replaces #cloud-config with “—” at the top of the file. You will need to manually change that back if you copy the YAML from this site.

You can also check the YAML out on the cli using the built in capabilities of cloud-init itself. You can ignore the “FutureWarning”. It’s a known bug in cloud-init 18.3 that has already been fixed in the 19.0 release.

$ cloud-init devel schema --config-file /boot/user-data
 /usr/lib/python3/dist-packages/cloudinit/config/cc_rsyslog.py:205: FutureWarning: Possible nested set at position 23
   r'^(?P[@]{0,2})'
 Valid cloud-config file /boot/user-data

Once the config file checks out, just burn a SD card for each node and replace the user-data file on each of them with a node-specific version you crafted. Power up the nodes and roughly five to ten minutes later you should have a functional set of Debian nodes ready to have Kubernetes installed.

For writing the SD cards, I use Win32 Disk Imager if running from a Windows box and if I’m on the Mac I use Hypriots ‘flash’ utility available here. https://github.com/hypriot/flash

Flash is easy to use, just grab the latest copy and run it like so:

% flash --userdata ./configs/node-3.yml --bootconf ./sample/no-uart-config.txt ~/Downloads/hypriotos-rpi-v1.11.5.img.zip
Using cached image /tmp/hypriotos-rpi-v1.11.5.img

Is /dev/disk2 correct? y
Unmounting /dev/disk2 ...
Unmount of all volumes on disk2 was successful
Unmount of all volumes on disk2 was successful
Flashing /tmp/hypriotos-rpi-v1.11.5.img to /dev/rdisk2 ...
1.27GiB 0:00:27 [46.9MiB/s] [====================================================================================================>] 100%            
0+20800 records in
0+20800 records out
1363148800 bytes transferred in 27.655770 secs (49289852 bytes/sec)
Mounting Disk
Mounting /dev/disk2 to customize...
Copying ./sample/no-uart-config.txt to /Volumes/HypriotOS/config.txt ...
Copying cloud-init ./configs/node-3.yml to /Volumes/HypriotOS/user-data ...
Unmounting /dev/disk2 ...
"disk2" ejected.
Finished.

Now, if something goes off the rails and you have to fix something in your user-data file, you can actually make the change and then re-run cloud-init instead of reflashing the SD card 🙂

$ sudo cloud-init clean
$ sudo cloud-init init

Remember that cloud-init only runs at first boot, so after re-initializing you will need to reboot in order for the changes to be applied.

Once you have everything ironed out get some tmux/iTerm2 loving going on and work on all your boxes at the same time ;-). I have a simple little tmux script I run from a WSL prompt on my Windows 10 box for this.

tmux new-session \; \
 select-pane -t 0 \; \
 split-window -v \; \
 split-window -h \; \
 select-pane -t 0 \; \
 split-window -h \; \
 select-pane -t 0 \; \
 send-keys 'ssh [email protected]' \; \
 select-pane -t 1 \; \
 send-keys 'ssh [email protected]' \; \
 select-pane -t 2 \; \
 send-keys 'ssh [email protected]' \; \
 select-pane -t 3 \; \
 send-keys 'ssh [email protected]' \; \
 select-pane -t 0 \;
 bind C-p setw synchronize-panes
tmux panes for my four nodes

Installing Kubernetes

From here out, I’m just parroting the last page of instructions from the book.

#use kubeadm to bootstrap the K8s cluster
sudo kubeadm init --pod-network-cidr 10.0.0.0/24 \
--apiserver-advertise-address 10.0.0.50 \
--apiserver-cert-extra-sans kubernetes.cluster.home

<snip>
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.0.0.50:6443 --token ti443b.z9opmdg3dokqvcjw \
    --discovery-token-ca-cert-hash sha256:08273e133da472e7611a5ec537fd83c94619c015ca9a63b327e8393162ff6d15

$ sudo kubeadm join 10.0.0.50:6443 --token ti443b.z9opmdg3dokqvcjw --discovery-token-ca-cert-hash sha256:08273e133da472e7611a5ec537fd83c94619c015ca9a63b327e8393162ff6d15
<snip>
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

#I then installed Flannel per the books instructions:  
curl https://rawgit.com/coreos/flannel/master/Documentation/kube-flannel.yml > kube-flannel.yaml

#Replace amd64 and vxlan with RPI friendly values
sed -i 's/amd64/arm/g' kube-flannel.yaml
sed -i 's/vxlan/host-gw/g' kube-flannel.yaml

#Apply the config map
kubectl apply -f kube-flannel.yaml

$ kubectl get nodes
NAME         STATUS     ROLES    AGE     VERSION
kubernetes   Ready   master   13m     v1.17.0
node-1       Ready   <none>   5m38s   v1.17.0
node-2       Ready   <none>   5m17s   v1.17.0
node-3       Ready   <none>   5m18s   v1.17.0

And there we have it.

SFCollector version .5 is now available

I’ve had some time to make extensive improvements to the SFCollector over the past couple of weeks. The major changes for the v.5 release include
  • Extensively reworked dashboards that now cover ESXi as well as SolidFire components. These dashboards are also now published on grafana.com
  • Added more metrics to the collections
  • Updated to Trident 18.01 for persistent storage
  • Tons of other small under the covers changes
Everything you need to get rolling is available at github.com/jedimt/sfcollector so head on over and take it for a spin!

Updates to the sfcollector

I’ve made some updates to the sfcollector to help improve the accuracy of graphs and make the delivery of stats more timely. If you have the prior version of this deployed consider this a mandatory upgrade 🙂 It’s also been tested against the new NetApp HCI product.

Future work will include improving the collector for scale as well as using a new metric set that moves away from point in time measurements that should be more representative of the actual IO demands on the system.

Head on over to https://github.com/jedimt/sfcollector to pick up the new version!

SolidFire stats collection w/Grafana and Graphite

I’ve been working on putting together a completely container based Grafana+Graphite dashboard for SolidFire for the past two weeks and I’ve got something I’m ready to call an early beta ready to roll.  

At a high level, there are three containers built using docker-compose that make up the application stack. 
  1. SFCollector -> runs the actual Python collection script that makes API calls to one or more SolidFire clusters to collect metrics, parse them and then push them into the Graphite container.
  2. Graphite -> stores the time series data pushed from the collector
  3. Grafana -> graphs the data from Graphite



If you are a SolidFire customer and you want to give it a shot, head over to https://github.com/jedimt/sfcollector, clone the repo and let me know what you think. There is a detailed install PDF included in the repo, but I’ve also cut a quick video of me doing an install. 

Please note, this IS a beta and there are some rough edges that need to be smoothed out. 

vSphere Replication w/SolidFire VVols

I just realized I had written up a quick ‘how to’ on configuring vSphere replication 6 with SolidFire VVols and then forgot to ever publish it. So yeah… without further ado…

Initial Install and Configuration for vSphere Replication

This link has a fairly detailed walkthrough for deploying the vSphere replication appliances.

http://www.settlersoman.com/how-to-install-and-configure-vmware-vsphere-replication-hypervisor-based-replication-6-0/

If the VMware self-signed certificates are being replaced by new certificates from a CA then the following KB article should be followed to ensure the correct SSL certificates are being used by the VMware Solutions

https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2109074

Configure a SPBM Policy for Replicated VMs

1. Connect to the vCenter that will host the replicated VM

2. Navigate to Home -> Policies and Profiles -> VM Storage Policies

3. Create a new storage policy as follows

a. Step 1 Name and Description -> Select the target vCenter server and enter a name for the policy
clip_image002

b. Step 2a Rule-Set 1 -> Select ‘com.solidfire.vasa.capabilities’ from the Rule based on data services drop down box. Set the data VVol minimum, maximum and burst IOPS to 1000/100000/100000
clip_image004

c. Step 3 Storage Compatibility -> Ensure the VVol datastore to be used for replication shows as compatible and then click Finish.

Configuring a VM for Replication with PIT Instances

1. Right click the VM to be replicated and select All vSphere Replication Actions -> Configure Replication
clip_image006

2. In the Configure Replication for <VM Name> wizard that appears configure as follows

a. Step 1 Replication Type -> Select Replicate to a vCenter Server

b. Step 2 Target Site -> Select the target vCenter instance

c. Step 3 Replication Server -> Select “Auto-assign vSphere Replication server”

d. Step 4 Target Location -> Click the ‘Edit’ link and choose the target VVol datastore
clip_image008
clip_image010

e. Step 5 Replication Options -> Enable Guest OS quiescing and Network Compression if desired.

f. Step 6 Recovery Settings -> Set the desired RPO for the virtual machine and enable Point in time instances with the desired number of instances to keep for the VM. These will be converted to snapshots during recovery of the virtual machine. The maximum number of PIT copies for a VM is 24.
clip_image012

g. Click Next and then Finish to complete configuration of replication

3. The replication will now start an initial sync after a couple minutes. After the initial sync the replication status should show a status of OK.
clip_image013

4. Navigate to Home -> vSphere Replication -> and select the target vCenter server. Click ‘Monitor’
clip_image014

5. Select ‘Incoming Replications’ and then select the virtual machine. Select Replication Details tab to verify details of the replication relationship for the VM.
clip_image016

6. Select the Point in Time tab to show PIT instances of the VM on the replicated site
clip_image018

Recovering a VM at the Replication Target Site

1. On the recovery vCenter navigate to Home -> vSphere Replication. Highlight the target vCenter instance and then click the monitor tab.
clip_image019

2. Highlight the VM to recover under “Incoming Replications” and then click the red ‘play’ button to start the recovery process
clip_image020

3. In the recovery wizard perform the following actions

a. Step 1 Recovery Options -> Select ‘Synchronize recent changes’ if the source VM is unavailable or shut down. If the source VM is still powered on and reachable this will not be an option. In this case, select “Use latest available data” and click Next. clip_image022

b. Step 2 Folder -> Select a folder on the target vCenter to receive the recovered VM

c. Step 3 Resource -> Select a host or cluster to host the restored VM. Click Next then Finish to complete the recovery of the VM.

4. The Recover Virtual Machine task will now run and register the restored VM on the target vCenter server. After a successful recovery the status of the VM will be ‘Recovered’. clip_image024

5. The recovered VM will be powered on with no network connectivity to avoid potential IP conflicts. clip_image026

6. If the VM needs to be restored to a previous PIT instance, right click the VM and navigate to Snapshots -> Manage Snapshots to select the PIT image to restore
clip_image028

7. To restore networking for the recovered VM (assuming the primary VM is offline), edit the settings of the VM and check the ‘Connected’ box for the network adapter for the VM.
clip_image029

Re-protecting a Recovered VM

1. Right click the recovered VM and select Configure vSphere Replication.

a. Step 1 Replication Type -> Select Replicate to a vCenter Server

b. Step 2 Target Site -> Select the vCenter to restore the VM to

c. Step 3 Replication Server -> Accept the default unless there are more than one replication appliances deployed

d. Step 4 Target Location -> Ensure the source VM is powered off. Then click the Edit link and select the original storage location for the VMclip_image031

i. Click OK. This will bring up a pop-up stating the target folder already exists. Click Use Existing button to replace the original VM with the contents of the recovered VMclip_image032

ii. Select the “Use all seeds” link to tell vSphere Replication to use the old VMDK files as a seed for the reverse replication to save on bandwidth during the recovery and click Nextclip_image034

e. Step 5 Replication Options -> Select replication options that fit the requirements for re-seeding the original production VM

f. Step 6 Recovery Settings -> Set the RPO and PIT options as required. Then click Finish to start the re-seeding of the recovered VM to the production vCenter.

Element OS 9.2

FluorineBanner

Earlier this month SolidFire has released Element OS 9 Patch 2 (Fluorine build 9.2.0.43). It includes support for all platforms and is a valid upgrade for any SolidFire cluster running Element OS 8.2 (Oxygen) or higher.

Some of the new features in patch 2 include

  • MDSS feature (more metadata) is not supported on all platforms. It is a support enabled feature at this time.
  • The vCenter Plugin has been updated to version 3. This was a complete re-write of the plugin and it now includes support for vSphere 6.5 and VVols 1/VASA 2. The new plugin focuses on pulling much more functionality from the SF GUI into vCenter and includes about 75% coverage of all storage tasks. I particularly like the VVols piece 🙂

There are also some very notable bug fixes that are included with this patch that will go a long way to improving the behavior of the cluster in a number of areas.

  • Fixed some issues when doing VM level replication to a VVol datastore
  • Resolved issues with tagging the primary storage VLAN and other general network improvements related to VLANs
  • Better performance under conditions with a very high number of iSCSI sessions
  • Better behavior when adding Element OS 8.x nodes to a 9.x cluster
  • Fix to allow faster loading of the SF GUI in ‘dark sites’
  • Lots of other under the covers fixes for general improvements in performance and stability

Get your update on!

Capacity and Efficiency per VM

This (probably poorly written) Powershell code will go grab all the VVols on a SolidFire system, group them by VM and then give you a read out of allocated and consumed capacity. It also reports on the overall efficiency of each VM.

image

#Connect to SolidFire system if no connection already exists
if(!$sfconnection){
Connect-SFCluster 172.27.40.200 -Username admin -Password solidfire | Out-Null
}

#Misc stuff
$separator = ".","_"

#Get a list of all the VVols and their stats
$VVols = Get-SFVirtualVolume -Details $true

#Group VVols by VM
#The VMW_VmID uniquely identifies the VVols associated with a VM
#Get list of unique VMW_VmID

foreach ($VmID in $VVols.metadata.VMW_VmID | Sort | Get-Unique) {

    #Get a unique VM object
    $VMObject = $VVols | select VirtualVolumeID,VolumeInfo,Metadata | where-object {$_.Metadata.VMW_VmID -match $VmID}
    $VMName = $VMObject.metadata.VMW_VVolName 
    $NumVVols = $VMObject.count
    
    $VMVVolStats = $VMObject.VolumeInfo.VirtualVolumeID | Get-SFVolumeStatsByVirtualVolume
    $VMVolEfficiencies = $VMObject.VolumeInfo.VolumeID | Get-SFVolumeEfficiency
      
    #Do some math for Agreegate capacity
    foreach ($VMVVolStat in $VMVVolStats) {
        $VMAllocatedCapactiy += $VMVVolStat.volumesize 
        $VMConsumedBlocks += $VMVVolStat.NonZeroBlocks
    }    
        
    #Create estimated efficiency per volume
    For ($i=0; $i -lt $VMVVolStats.Count; $i++) {

    #Catch if there is a 0 compression or deduplication number, if so set to 1
    If ($VMVolEfficiencies[$i].Deduplication -eq "0") { 
        $VMVolEfficiencies[$i].Deduplication = 1
    }
    If ($VMVolEfficiencies[$i].Compression -eq "0") {
        $VMVolEfficiencies[$i].Compression = 1
    }

    $VMVVolEfficiency = $VMVolEfficiencies[$i].Deduplication * $VMVolEfficiencies[$i].Compression
    $VMDiskEffectiveSize = (($VMVVolstats[$i].NonZeroBlocks * 4096) /1gb) / $VMVVolEfficiency
    $TotalVMEffective += [math]::Round($VMDiskEffectiveSize,2)                
    }

    #Turn blocks into bytes (not quite as exciting as water to wine)
    $VMConsumedCapacity = [math]::Round($VMConsumedBlocks * 4096 /1gb,2)
    $VMEffectiveEfficiency = [math]::Round($VMConsumedCapacity / $TotalVMEffective,2)

    #Total Counters
    $AllVMConsumed += $VMConsumedCapacity
    $AllVMEffective += $TotalVMEffective
    $AllEfficiency = [math]::Round(($AllVMConsumed / $AllVMEffective),2)

    #Note, all the capacity values are in GiB, not GB
    Write-Host "`nVM Name: $($VMName.split($separator) | select -first 1)" -ForegroundColor DarkGray
    Write-Host "Number of VVols: $NumVVols" -ForegroundColor Green
    Write-Host "VM Allocated GiB: $($VMAllocatedCapactiy /1gb)" -ForegroundColor Green
    Write-Host "VM Consumed GiB: $VMConsumedCapacity" -ForegroundColor Green
    Write-Host "VM Effective GiB: $TotalVMEffective" -ForegroundColor Green
    Write-Host "VM Effective Efficiency: $VMEffectiveEfficiency" -ForegroundColor Green

    #Clear out counters
    $VMAllocatedCapactiy = 0
    $VMConsumedBlocks = 0
    $TotalVMEffective = 0
    $VMDiskEffectiveSize = 0
    $TotalVMEffective = 0
    $VMConsumedCapacity = 0
    $VMEffectiveEfficiency = 0
}
Write-Host "`nTotal Consumed GiB Used by VMs:"  $AllVMConsumed
Write-Host "Total Effective GiB Used by VMs:" $AllVMEffective
Write-Host "Total Effective Efficiency:" $AllEfficiency

I’ll just come right out and say that the efficiency number really isn’t all that useful by itself. The reason being that the GetVolumeEfficiency call looks at a volume in isolation. This means it only reports the efficiency within the individual volumes, not across volumes. Practically that means the reported number is a worst case scenario.

However, it is useful in the sense that it provides a pretty decent expected ratio among a set of VMs. In the example above, I can see that my vra-2012r2 template reduced 1.91x and my vra-sql2014 template reduced 1.62x – a difference of ~ 15%. Generally speaking that means I can expect to get roughly 15% better efficiency rates with the vra-2012r2 VMs vs the vra-SQL2014 VMs. That might be useful information to have.

To test if this holds in a deployment, I created two storage containers (vra-2012r2 and vra-SQL2014) and deployed 50 VMs to each from the respective VM templates.

imageimage

This allows SolidFire to report the efficiency for all the VMs in those two storage containers independently so we can see how well we dedupe/compress across the VMs in the same storage container. Given the numbers reported per-VM earlier I would expect the vra-2012r2 storage container to get roughly 15% better numbers.

image

So we did, in fact, get better efficiency for the vra-2012r2 VMs and the difference in efficiency is ~20% which is in the ball park Smile After 24 hours the the difference was ~17%, almost exactly matching the original difference in the templates. So, not an exact science, but useful for general sizing estimates.

Measuring Efficiencies on VVols

Something that always comes up with discussions around VVols is “how do I report on how much I’ve allocated and consumed?”. This usually is expressed in one of the following ways:

  • Can I know total capacity allocated and consumed for the system and what efficiencies am I getting?
  • Can I know capacity allocated/consumed and efficiencies for each storage container?
  • Can I know capacity allocated/consumed and efficiencies for each VM?

Fortunately, the answer is yet to all three Smile There are a number of ways to get at this, but for the sake of simplicity I’ll be showing examples using the SolidFire PowerShell module. If you don’t have the PSM installed you can grab it and the installation instructions here: https://github.com/solidfire/powershell

Total System Capacity and Efficiencies

Total system metrics can be viewed by running Get-SFClusterCapacity

C:\> $ClusterCapacity = Get-SFClusterCapacity
C:\> $ClusterCapacity

ActiveBlockSpace : 48349817348
ActiveSessions : 16
AverageIOPS   : 4
ClusterRecentIOSize   : 5602
CurrentIOPS   : 4
MaxIOPS   : 200000
MaxOverProvisionableSpace : 276546135777280
MaxProvisionedSpace   : 55309227155456
MaxUsedMetadataSpace : 432103337164
MaxUsedSpace : 8642066743296
NonZeroBlocks : 52358440
PeakActiveSessions : 16
PeakIOPS : 502
ProvisionedSpace : 662498705408
SnapshotNonZeroBlocks : 0
Timestamp : 2017-02-17T18:05:17Z
TotalOps : 3586717
UniqueBlocks : 16370852
UniqueBlocksUsedSpace : 49468393216
UsedMetadataSpace : 876314624
UsedMetadataSpaceInSnapshots : 876314624
UsedSpace : 49468393216
ZeroBlocks : 271127256

Let me decode those bolded entries for you

  • NonZeroBlocks – The number of 4K blocks in the system that have been ‘touched’. Basically, there has been data written to them.
  • ProvisionedSpace – The number of bytes that have been provisioned. In the VVol case, this will line up with the aggregate size of all the VMDKs. For example adding an 80GB VMDK to a VM will increase this counter 80GB. 
  • UsedSpace – The actual amount of consumed block space on SolidFire for all NonZeroBlocks post deduplication/ compression savings. 

So the answers to our questions are:

  • Total Capacity Allocated = ProvsionedSpace (662498705408 bytes; 662GB)
  • Total Capacity Consumed [Pre-Efficiencies] = nonZeroBlocks * 4096 = 52358440 * 4096 = 214460170240 bytes; 214GB)
  • Total Capacity Consumed [Post Efficiencies] = UsedSpace (49468393216 bytes; 49GB) –> This implies a 4.6 cluster efficiency metric (deduplication * compression)

This lines up nicely with the numbers reported on the cluster reporting overview screen.

imageimage

 

So if you wanted to pull those numbers into some PowerShell variables you could do this. Note values would be stored in GiB, not GB notation when using the “/1gb” built in PowerShell constant.

$ClusterStats = Get-SFClusterCapacity
$ClusterAllocated = $ClusterStats.ProvisionedSpace /1gb
$ClusterConsumedPre = $ClusterStats.NonZeroBlocks * 4096 /1gb
$ClusterConsumedPost = $ClusterStats.UsedSpace /1gb

Capacity and Efficiency per Storage Container

[Note: If this section seems a little convoluted it is. There is a bug in the SDK that leaves out some details needed to match up VVols and storage containers, so I had to follow a bit of a circuitous route to get at what I wanted. I’ll amend this post when the hotfix is available.]

Ok, so lets take a look at how this breaks down with storage containers in the mix. There is not an API call to get all the capacity related counters for a storage container directly. We have to get at it by summing up the values for all VVols in the storage container.

There are a couple ways to go about this, but (until the SDK is fixed) I settled on the following logic.


#First, get storage containers and account objects
$StorageContainers = Get-SFStorageContainer
$StorageContainerAccounts = Get-SFAccount -AccountID $StorageContainers.AccountID

foreach ($StorageContainerAccount in $StorageContainerAccounts){

    #Get VVols for the Storage Container
    $VVols = Get-SFVolume -accountID $StorageContainerAccount.AccountID
    $VVolStats = $VVols | Get-SFVolumeStatsByVirtualVolume

        #Add up capacity stats for all VVols under a single storage container
        foreach ($vvol in $VVolStats) {
        $StorageContainerAllocatedBytes += $vvol.VolumeSize 
        $StorageContainerNonZeroblocks += $vvol.NonZeroBlocks 
        }

    #Do some math
    $StorageContainerConsumedGB = [math]::Round(($StorageContainerNonZeroblocks * 4096) /1gb,2)
    $StorageContainerAllocatedGB = $StorageContainerAllocatedBytes /1gb
    $TempSC = $storagecontainers | select AccountID,StoragecontainerID | where-object {$_.AccountID -match $StorageContainerAccount.AccountID}
    $StorageContainerEfficiency = Get-SFStorageContainerEfficiency -StorageContainerID $TempSC.StorageContainerID 
    $scdedupe = [math]::Round($StorageContainerEfficiency.Deduplication,2)
    $sccompres = [math]::Round($StorageContainerEfficiency.Compression,2)
    $sctotalefficiency = [math]::Round($StorageContainerEfficiency.Deduplication * $StorageContainerEfficiency.Compression,2)

    #Write out results to console on each pass
    Write-Host "`nStorage Container $($StorageContainerAccount.Username) VVol Stats (Account ID $($StorageContainerAccount.AccountID)):" -ForegroundColor DarkGray
    Write-Host "Allocated GB: $($StorageContainerAllocatedGB)" -ForegroundColor Green
    Write-Host "Consumed GB: $($StorageContainerConsumedGB)" -ForegroundColor Green
    Write-Host "Storage Container Efficiency: " -ForegroundColor Green
    Write-Host "Deduplication: $($scdedupe)" -ForegroundColor Green
    Write-Host "Compression: $($sccompres)" -ForegroundColor Green
    Write-Host "Total: $($sctotalefficiency)" -ForegroundColor Green

#Clear out variables
$StorageContainerAllocatedBytes = 0
$StorageContainerNonZeroblocks= 0
$StorageContainerConsumedGB = 0
$StorageContainerAllocatedGB = 0
$TempSC = $null
$StorageContainerEfficiency = 0
$scdedupe = $null
$sccompres = $null
$sctotalefficiency = $null
}


That produces the following results:

image

Note that the cluster wide efficiency numbers will look better than the individual storage container metrics because the deduplication domain is larger when you look at the full cluster. 

Now to get to work on that per-VM reporting…

Capturing API SSL Traffic with Wireshark

I ran into an issue with Postman today that required me to examine the API responses from my SolidFire cluster. However, since the API payload is encrypted it takes a little extra work. When I first captured the event in question I couldn’t find the API traffic I was expecting in the Wireshark trace.

To decrypt the traffic I needed the private key used to secure the exchange. Once I had that, in Wireshark I navigated to Edit -> Preferences -> Protocols -> SSL -> RSA Keys List to get to the SSL Decrypt dialog. Here I added the management IP address  (MVIP) of the SolidFire cluster, specified http as the protocol and loaded the private key file.

Once that was done I could see the HTTP JSON POST and response calls I was interested in. Not something I think I’ll do often, but it is a neat tool to have in your back pocket. 

Incidentally, this helped me verify that some garbage characters I was seeing in a API response for GetStorageContainerEfficiency was an issue with Postman itself, not the API. Reinstalling Postman cleared up the issue.