Part 4/4 - Deploying Openshift/OKD 4.5 on Proxmox VE Homelab
This is the last part of a 4-part series on Running Openshift at Home. Some information here will have some references to the previous part of the series. Please find the links below for the previous posts.
Part 1/4 - Homelab Hardware
Part 2/4 - Building a Silent Server Cabinet
Part 3/4 - Installing a Two-Node Proxmox VE Cluster
The installation process uses OKD4.5, the upstream community project of Openshift. It's like the Fedora Linux of Red Hat Enterprise Linux.
Openshift has 2 types of installation. The IPI, which is the fully automated deployment over known cloud providers such as AWS, Azure and GCP. On the other hand, UPI or user-provisioned infrastructure is a partially automated process which we will talk about here.
The installation process is automatically performed by bootstrapping. A bootstrap machine is a temporary machine used by
Openshift/Kubernetes to host the services required in the bootstrap procedure. The bootstrap machine will create an etcd cluster and starts a few Kubernetes services. The master machines will then join the etcd cluster through ignition. The Kubernetes services will then be transferred from the bootstrap machine to the master nodes as soon as they become ready. The last step of the bootstrap process is that the bootstrap machine will be removed from the etcd cluster. At this point, the bootstrap machine can be shut down and deleted forever.
Though the bootstrap process is automatic, the preparation of the installation, however, has to be done manually. Note that in 4.6 which was released a couple of days ago, there is now support for automated installation on bare-metal infrastructure using IPMI/BMC. We will not cover this here.
As described in the previous post, my homelab infrastructure looks like this.
The servers are running Proxmox Virtualization Environment, an opensource hypervisor. I also have a physical router and a physical DNS server. We will configure this device as well for the OKD bootstrap process to work. You will need a good amount of RAM on the host to run the following configuration. In Proxmox, we can over-provision RAM. So even if the total RAM of the VMS is 100GB, the setup should run if you have at least 64GB of RAM available on the host. In my case, the total RAM usage after installing a 5-node Openshift was around 56GB.
For a 5-node Openshift/OKD cluster you will need to spin up 6x Fedora Core OS VMs, and 1x Centos 8, assuming you have a physical router and an external DNS server. Otherwise, you may also run your router and DNS server in a VM. But this will eat up even more RAM on your host.
Start by creating the following VMs in Proxmox VE as detailed in the following sections and take note of their MAC addresses after creation. We will use the table below as a reference for VM creation and DHCP address reservation configuration.
Download the OS Images
Download the latest Fedora Core OS installer from https://getfedora.org/en/coreos/download. Select the Bare Metal & Virtualized Tab. Download the Bare Metal ISO package.
Upload the installer to the local storage of the Proxmox node where you will create the VMs (just in case you have multiple nodes).
Create the VMs
From the Proxmox VE Web interface, right-click in the node and select create VM. Name the VM according to the table above starting with the okd4-bootstrap.
Select the Fedora Core OS image we uploaded earlier.
Leave the system tab with default values. Proxmox VE has already pre-selected the optimum setting for the selected Guest OS type. Set the size of the disk according to the table above.
Select 4 cores in the CPU tab as per the table above. Leave the rest of the setting unchanged by default, unless you know what you are doing.
Set the Memory to 16GB (16384 Mib).
If you followed the above instructions correctly, you should see the following values in the confirmation screen. Then just click finish.
After the VM is created. Proxmox will generate a MAC Address for the virtual network interface card. Take note of the MAC Address. Create a table similar to the above, but with the MAC Addresses column. You will need this later.
Repeat the above procedure for the rest of the VMs. Take note that the last VM in the table, okd4-services, is a CentOS 8 VM. You need to download the CentOS 8 installer ISO and upload it to Proxmox local storage. The latest CentOS 8 release can be downloaded here: http://isoredirect.centos.org/centos/8/isos/x86_64.
Download the one that ends with dvd1.iso.
Upload this file to the local storage of Proxmox VE and create the okd4-services VM as per the above procedure. Take note that this VM only has 4GB of RAM. Not 16GB.
You should have the following list of VMs at the end.
DHCP Address Reservation
Using the list of MAC Addresses of the VMs created earlier, we need to assign IP addresses to these MAC Addresses through DHCP Address reservation. Depending on the router, the process may be slightly different.
For my case, I have an ASUS Router and this is how the Address reservation looks like. Just a table of MAC addresses and their pre-assigned IP address.
When the VMs are started, they will get these IP Addresses via DHCP.
If you do not have a physical router or don't want to use your home router, you can run a PfSense router on a VM and configure the above VMs to be behind this router. Then you need to configure the same DHCP address reservation configuration. We will also need to revisit this router configuration after setting up a DNS server.
The okd4-services VM will run several things required to install and run Openshift/OKD.
- DNS Server - If you do not have an external/Raspberry Pi DNS server
- HA Proxy - Load Balancer
- Apache Web Server (httpd) - to host the OS images and ignition files during PXE booting. This service can be stopped after the installation.
- NFS server - to be used as Persistent Volume by Openshift Image Registry
Start the okd4-services VM. Navigate to console.
In the Installation Destination option, select custom, then done.
Delete the /home partition and leave the desired capacity for / empty.