VIC appliance disk space is filled by Registry blobs / Cleanup VIC Registry

We have been using vSphere Integrated Containers for almost a year now and whenever the developers have trouble pushing or pulling images from the private registry, its usually because the disk is full.

So we just add more space to the appliance and reboot it to expand the filesystem.

Here is how it is done - 

Login to the VIC appliance and run df -h 

You will see the filesystem /storage/data is almost full 

Go to the vSphere Console and expand the drive to the desired value. 

Note : You can hot add drive space. 

The VIC appliance has 4 virtual disks attached to it. Refer to this table to find out which disk to expand.

This is a quick and dirty solution but in our case we kept adding space and went well over 1 TB. It was time to cleanup and put a permanent fix to this problem. 

Harbor is the open source registry that runs in the VIC appliance. Behind the scenes it is nothing but an open source docker distribution with added functionalities such as security, identity and management.

As per the VIC documentation, you can simply enable garbage collection and after you reboot the appliance it should take effect. Unfortunately this did not work for us.

Here is how I got around this issue and manually ran the Garbage collection  -

If you have not noticed yet, within the VIC appliance all the components run as individual containers (pretty cool)

SSH to the VIC appliance and run the simple command 

                       docker ps -a 

You will notice that all services are running as individual containers. 

We are going to work on the highlighted Harbor registry container.

A garbage collector is the one that actually deletes the leftover blobs from different image tags. 

If you are familiar with the docker exec command, we will use this simple command on the registry container to see what blobs can be marked for deletion.

Update:- If you have upgraded to VIC 1.5 then you need to su to user harbor. 

Older VIC versions - docker exec registry bin/registry garbage-collect --dry-run  /etc/registry/config.yml

VIC 1.5 and above - docker exec registry su -c "registry garbage-collect --dry-run  /etc/registry/config.yml" harbor

This will not actually delete the blobs but will output all that can be marked for deletion.

To mark these blobs for deletion, run the above command without the --dry-run flag - 

Older VIC versions - docker exec registry bin/registry garbage-collect /etc/registry/config.yml

VIC 1.5 and above - docker exec registry su -c "registry garbage-collect /etc/registry/config.yml" harbor

Eligible blobs will be marked for deletion. 

Next just restart the harbor service and the blogs will be actually deleted from the disk. 

systemctl restart harbor

If that still does not cleanup space, reboot the VIC appliance, SSH back and check with a df -h

Happy days !