VIC appliance disk space is filled by Registry blobs / Cleanup VIC Registry

We have been using vSphere Integrated Containers for almost a year now and whenever the developers have trouble pushing or pulling images from the private registry, its usually because the disk is full.

So we just add more space to the appliance and reboot it to expand the filesystem.

Here is how it is done - 

Login to the VIC appliance and run df -h 

You will see the filesystem /storage/data is almost full 

Go to the vSphere Console and expand the drive to the desired value. 

Note : You can hot add drive space. 

The VIC appliance has 4 virtual disks attached to it. Refer to this table to find out which disk to expand.

This is a quick and dirty solution but in our case we kept adding space and went well over 1 TB. It was time to cleanup and put a permanent fix to this problem. 

Harbor is the open source registry that runs in the VIC appliance. Behind the scenes it is nothing but an open source docker distribution with added functionalities such as security, identity and management.

As per the VIC documentation, you can simply enable garbage collection and after you reboot the appliance it should take effect. Unfortunately this did not work for us.

Here is how I got around this issue and manually ran the Garbage collection  -

If you have not noticed yet, within the VIC appliance all the components run as individual containers (pretty cool)

SSH to the VIC appliance and run the simple command 

                       docker ps -a 

You will notice that all services are running as individual containers. 

We are going to work on the highlighted Harbor registry container.

A garbage collector is the one that actually deletes the leftover blobs from different image tags. 

If you are familiar with the docker exec command, we will use this simple command on the registry container to see what blobs can be marked for deletion.

Update:- If you have upgraded to VIC 1.5 then you need to su to user harbor. 

Older VIC versions - docker exec registry bin/registry garbage-collect --dry-run  /etc/registry/config.yml

VIC 1.5 and above - docker exec registry su -c "registry garbage-collect --dry-run  /etc/registry/config.yml" harbor

This will not actually delete the blobs but will output all that can be marked for deletion.

To mark these blobs for deletion, run the above command without the --dry-run flag - 

Older VIC versions - docker exec registry bin/registry garbage-collect /etc/registry/config.yml

VIC 1.5 and above - docker exec registry su -c "registry garbage-collect /etc/registry/config.yml" harbor

Eligible blobs will be marked for deletion. 

Next just restart the harbor service and the blogs will be actually deleted from the disk. 

systemctl restart harbor

If that still does not cleanup space, reboot the VIC appliance, SSH back and check with a df -h

Happy days !

docker: Error response from daemon: Server error from portlayer

docker: Error response from daemon: Server error from portlayer:

I got this error while running a new container in VIC.

I tried to move the VCH to a different ESXi Host, but no luck.

It is not advisable to reboot the VCH from vCenter, so had to skip that.

Tried to search the internet and came up with a similar error but a different cause - https://github.com/vmware/vic/issues/5782

This did not get me anywhere so I started looking from a higher level and this is what I noticed.

Turned out I had upgraded the VIC from 1.2.1 to 1.3 last weekend and complete forgot to upgrade this last VCH.

As you can see in the image above the VCH I was trying to run a container against was running on 1.2.1. Duh !

So I upgraded the VCH by running the following command -

./vic-machine-linux upgrade --id xxxxxx

And here is the result -

And ta-da I was able to run the new nginx container I was trying to deploy earlier.

Sharing NFS-backed Volumes Between Containers

vSphere Integrated Containers supports two types of volumes, each of which has different characteristics.

• VMFS virtual disks (VMDKs), mounted as formatted disks directly on container VMs. These volumes are supported on multiple vSphere datastore types, including NFS, iSCSI and VMware vSAN. They are thin, lazy zeroed disks. 
• NFS shared volumes. These volumes are distinct from a block-level VMDK on an NFS datastore. They are Linux guest-level mounts of an NFS file-system share.

VMDKs are locked while a container VM is running and other containers cannot share them.

NFS volumes on the other hand are useful for scenarios where two containers need read-write access to the same volume.

To use container volumes, one must first declare or create a volume store at the time of VCH creation

You must use the vic-machine create --volume-store option to create a volume store at the time of VCH creation.

You can add a volume store to an existing VCH by using the vic-machine configure --volume-store option. If you are adding volume stores to a VCH that already has one or more volume stores, you must specify each existing volume store in a separate instance of --volume-store.

Note: If you do not specify a volume store, no volume store is created by default and container developers cannot create or run containers that use volumes.

In my example, I have assigned a whole vSphere datastore as a volume store and would like to add a new NFS volume store to the VCH. The syntax is as follows - 

$ vic-machine-operating_system configure 

--target vcenter_server_username:password@vcenter_server_address 

--thumbprint certificate_thumbprint --id vch_id 

--volume-store datastore_name/datastore_path:default 

--volume-store nfs://datastore_name/path_to_share_point:nfs_volume_store_label

nfs://datastore_name/path_to_share_point:nfs_volume_store_label is adding a NFS datastore in vSphere as the volume store. Which means when a volume is created on it, it will be a VMDK file which cannot be shared between containers.

Add a NFS mountpoint to be able to share between containers. You need to specify the URL, UID, GID and access protocol. 

Note:You cannot specify the root folder of an NFS server as a volume store.

The syntax is as follows - 

--volume-store nfs://datastore_address/path_to_share_point?uid=1234&gid=5678&proto=tcp:nfs_volume_store_label

If you do not specify the UID and GID the default is 1000. Read more and UID and GID here.

Before adding NFS Volume Store

After adding NFS Volume Store

Two things to note in my example- 

1) I am running a VM based RHEL backed NFS and the UID and GID did not work for me. 

2) The workaround was to manually change permissions on the test2  folder on the NFS Share point by using  chmod 777 test2

Now that the NFS volume store is added, lets create a volume - 

Next, deploy two containers with My_NFS_Volume  mounted on both.

To check if the volume was mounted docker inspect containername and you will see the details under Mounts. You will see the name as well as the read/write mode.

Now lets create a .txt file from one container and check from the other if it can be seen.

This concludes we can share NFS-backed volumes between containers.

Configure Static IP on PhotonOS

To obtain the name of your Ethernet link run the following command:  networkctl

If this is the first time you are using Photon OS, you will only see the first 2 links. The others got created because I ran some docker swarms and created customer network bridges.

The network configuration file is located at -

                   /etc/systemd/network/

You might see the file 10-dhcp-eth0.network. I renamed this file to static.

You can do this by running the following command -

root@photon [ ~ ]# mv /etc/systemd/network/10-dhcp-eth0.network  /etc/systemd/network/10-static-eth0.network

Use vi editor to edit the file and add your static IP, Gateway, DNS, Domain and NTP.

This is how the file would look like. 

root@photon [ ~ ]# cat /etc/systemd/network/10-static-eth0.network

[Match]

Name=eth0  <<<<<<< “Make sure to change this to your adapter. ipconfig to check adapter name”

[Network]

Address=10.xx.xx.xx/24

Gateway=10.xx.xx.1

DNS=10.xx.xx.xx 10.xx.xx.xx

Domains=na.xx.com

NTP=time.nist.gov

Apply the changes by running -

systemctl restart systemd-networkd

Try to ping out form the OS. 

Note: You will not able able to ping this VM as by default the iptables firewall blocks everything except SSH. In my next blog I will explain how to allow ping on iptables.

Testing a MySQL restore in a container using VIC

We had a requirement yesterday where we had to lift and shift a static website from a major Cloud provider to our in-house Datacenter (why would someone want to do that is another topic ;-) ). It was a typical 3-tier application. The Database was MySQL. The customer shipped us a database dump and our System Admins got busy building a new RHEL VM for it. I got curious to see if the dump can be tested before restoring in the RHEL VM. Docker Containers to the rescue!

With limited MySQL knowledge I started digging deeper. Following was the thought process -

1) Can the backup be copied inside a container running MySQL image?
2) That would require a volume store in VCH which can be mounted inside the MySQL container.
3) Test if the database is restored and running - locally as well as using a remote client.

Lets dive into how this was done using vSphere Integrated Containers.

I have already deployed the VCH.

Lets create a volume store where the DB Backup can be copied.

Next create and mount the volume in a MySQL container.

--name: Name of container

-v: Bind mount a volume. soure_volume:container_destination_options

-e: The environment variable.

MYSQL_ROOT_PASSWORD: Password option is mandatory when running a MySQL container. You need to specify one of MYSQL_ROOT_PASSWORD, MYSQL_ALLOW_EMPTY_PASSWORD or MYSQL_RANDOM_ROOT_PASSWORD
Refer to "How to use this image" in the official repository

--network: connecting the container to a container network defined while creating the VCH. This was defined with a IP block so it will assign an IP from that block.

mysql: image name.

Now that the container is created, lets copy the mysql_db_dump.sql  dump file to the volume that is mounted to the container.

Now that the database dump is copied in the container, lets start it. Once that container is running lets confirm if the dump file is copied at /var/lib/mysql

Next, lets check what databases are currently running inside the container. There are multiple ways you can achieve this -

Opening a bash shell to the container:

As you can see we land directly at the root prompt inside the container.

mysql -uroot -pversecurepassword

Login to mysql with root as the username and verysecurepassword as the password. The password was set at the time of container creation. 

This gives you a mysql prompt. From here you can run all your queries against the Database. 

Using Docker exec: 

You can also use docker exec -i to just pass commands to the container in an interactive mode instead of the Bash shell. Here is how its done - 

Lets create a new Database named TEST_RESTORE and restore the mysql_db_dump.sql

As you can see the database has been restored and "show tables;" lists all tables.

Lets create a client container to see if we can connect to the Database from it.

Note: We did not run legacy --link command. We ran the client container on a Container Network which is able to communicate with our mysqlrestore container.

This concludes that we can connect to the restored database locally as well as from a remote client.

vSphere Integrated Containers (Docker Swarm) - Part 2

In my last post we created a Docker Swarm using DCH now lets look at a few cool things you can do with it.

The setup remains the same as the last post.

Scaling up and down

In our original example we had 6 replicas running.

Run the following to check the running processes -

             docker -H 10.156.134.141 service ps web

Lets bump it up to 10. To do this we use the docker service scale command

            docker -H 10.156.134.141 service scale web=10

And then check to see if the instances have been scaled to 10

To scale down just run the same command with the desired number of instances

            docker -H 10.156.134.141 service scale web=6

And check to see if the instances have been scaled down back to 6.

Draining the Node


Before draining a node let us make sure the number of running processes/instances and the corresponding worker nodes they are running on.

• 1 on manager1
• 1 on worker1
• 2 on worker2
• 2 on worker3

Run the following command to drain worker3

            docker -H 10.156.134.141 node update --availability drain worker3

As you can see web.8 and web.9 are shutdown on worker3 and are scheduled on manager1 and worker1 respectively.

Applying Rolling Updates 

The simplest of all commands. If the nginx image you deployed needs to be updated, just run - 

docker service update --image <imagename>:<version> web

Removing a service

Be careful using this command. It does not ask for confirmation.

docker service rm web

vSphere Integrated Containers (Docker Swarm)

vSphere Integrated Containers v1.2 includes the ability to provision native Docker container hosts (DCH). The DCH is distributed by VMware through Docker Hub.

VIC v1.2 does not support docker build and docker push and developers can use this dch-photon image to perform these operations as well as deploy a swarm (not natively supported in VIC)

dch-photon is also pre-loaded in the default-project in vSphere Integrated Containers Registry. If you are not familiar with dch-photon, please read the VIC Admin Guide for reference.

Lets jump right in to see how a Developer can use DCH to create a Docker Swarm and deploy an application.

You can write a simple shell script that deploys Docker swarm manager node and then create and join worker nodes to the swarm. In this example I will deploy the manager and worker nodes manually.

Create a Virtual Container Host (VCH) - This will act as an endpoint for deploying the master and worker DCH. I have deployed a VCH named swarm_test with a container network IP range so the containers will pick an IP from the range provided. The VCH IP in 10.156.134.35.

Creating a docker volume for the master image cache

docker -H 10.156.134.35 volume create --opt Capacity=10GB --name registrycache

Creating a volume for each worker image cache

                docker -H 10.156.134.35 volume create --opt Capacity=10GB --name worker1

                docker -H 10.156.134.35 volume create --opt Capacity=10GB --name worker2

                docker -H 10.156.134.35 volume create --opt Capacity=10GB --name worker3

Lets create a Master instance

               docker -H 10.156.134.35 create -v registrycache:/var/lib/docker \

--net VIC-Container \

--name manager1 \

--hostname=manager1 \

vmware/dch-photon:17.06        

Create the worker instances worker 1,2,3

docker -H 10.156.134.35 create -v worker1:/var/lib/docker \

--net VIC-Container \

--name worker1 \

--hostname=worker1 \

vmware/dch-photon:17.06

Here is how the deployed setup looks like -

Connect the master and worker nodes to the appropriate Bridge network.

            docker -H 10.156.134.35 network connect bridge manager1

docker -H 10.156.134.35 network connect bridge worker1

docker -H 10.156.134.35 network connect bridge worker2

docker -H 10.156.134.35 network connect bridge worker3 

This is where I spent most of the time trying to figure out why my worker nodes were not talking to the manager. I highly recommend reading the network use cases in the documentation. As per my network setup I had to combine bridge networks with a container network.

Now, start the master and all worker nodes.

                docker –H 10.156.134.35 start manager1

                docker –H 10.156.134.35 start worker1 worker2 worker3

Create a Swarm on the Master (note my manager node IP in the screenshot above)

                docker -H 10.156.134.141 swarm init --advertise-addr 10.156.134.141

I am advertising the manager IP so the nodes can communicate on it. This is your eth0 in Docker networking world.The output of the above command will give you a Token.

This Token is required for the worker nodes to join the swarm. Don't panic if you missed copying the token string. You can get it back by running the following command 

               docker -H 10.156.134.141 swarm join-token worker  (make sure you get the worker token. If you replace worker my manager you will get the manager token. This is useful if you want to have more than one manager in your swarm)

Add each worker to the swarm

      docker -H 10.156.134.142 swarm join 4stkz3wrziufhq8qjwkszxpzet6o3tlut1lf9o9ijqkhsvb5va-dtopt9a6bp9r03q52q2ea6mo4 10.156.134.141:2377

Once all worker nodes are added, run the following to list the nodes - 

                      docker -H 10.156.134.141 node ls

Now that the nodes are up lets create a simple nginx web service.

Create a service 

                     docker -H 10.156.134.141 service create --replicas 6 -p 80:80 --name web nginx

Check the status of the service - 

                 docker –H 10.156.134.141 service ls

                    docker –H 10.156.134.141 service ps web

You can see the replicas are preparing and not ready yet. 

Whats happening in the background is the manager node is distributing the nginx image on the worker nodes and orchestration layer is scheduling containers on the manager and worker nodes. If you run docker service ps web again you will see the service is now running which means the nginx daemon has been launched and ready to serve requests.

Go to your favorite browser and hit the IP of any worker or manager nodes. Even if a container is not scheduled on one of the nodes, you should still be able to get to the Welcome to nginx! webpage. Thats the whole idea of a swarm. 

In my next blog I will talk about Scaling up and down, inspecting nodes, draining nodes, removing a service and applying rolling updates to a service.