Part 4/4 - Deploying Openshift/OKD 4.5 on Proxmox VE Homelab
This is the last part of a 4-part series on Running Openshift at Home. Some information here will have some references to the previous part of the series. Please find the links below for the previous posts.
Part 1/4 - Homelab Hardware
Part 2/4 - Building a Silent Server Cabinet
Part 3/4 - Installing a Two-Node Proxmox VE Cluster
The installation process uses OKD4.5, the upstream community project of Openshift. It's like the Fedora Linux of Red Hat Enterprise Linux.
Openshift has 2 types of installation. The IPI, which is the fully automated deployment over known cloud providers such as AWS, Azure and GCP. On the other hand, UPI or user-provisioned infrastructure is a partially automated process which we will talk about here.
The installation process is automatically performed by bootstrapping. A bootstrap machine is a temporary machine used by
Openshift/Kubernetes to host the services required in the bootstrap procedure. The bootstrap machine will create an etcd cluster and starts a few Kubernetes services. The master machines will then join the etcd cluster through ignition. The Kubernetes services will then be transferred from the bootstrap machine to the master nodes as soon as they become ready. The last step of the bootstrap process is that the bootstrap machine will be removed from the etcd cluster. At this point, the bootstrap machine can be shut down and deleted forever.
Though the bootstrap process is automatic, the preparation of the installation, however, has to be done manually. Note that in 4.6 which was released a couple of days ago, there is now support for automated installation on bare-metal infrastructure using IPMI/BMC. We will not cover this here.
Infrastructure
As described in the previous post, my homelab infrastructure looks like this.
The servers are running Proxmox Virtualization Environment, an opensource hypervisor. I also have a physical router and a physical DNS server. We will configure this device as well for the OKD bootstrap process to work. You will need a good amount of RAM on the host to run the following configuration. In Proxmox, we can over-provision RAM. So even if the total RAM of the VMS is 100GB, the setup should run if you have at least 64GB of RAM available on the host. In my case, the total RAM usage after installing a 5-node Openshift was around 56GB.
Virtual Machines
For a 5-node Openshift/OKD cluster you will need to spin up 6x Fedora Core OS VMs, and 1x Centos 8, assuming you have a physical router and an external DNS server. Otherwise, you may also run your router and DNS server in a VM. But this will eat up even more RAM on your host.
Start by creating the following VMs in Proxmox VE as detailed in the following sections and take note of their MAC addresses after creation. We will use the table below as a reference for VM creation and DHCP address reservation configuration.
| VM Name | Role | IP Address | OS | vCPU | RAM | Storage |
| okd4-bootstrap | bootstrap | 192.168.1.200 | Fedora Core OS | 4 | 16 | 120 |
| okd4-control-plane-1 | master | 192.168.1.201 | Fedora Core OS | 4 | 16 | 120 |
| okd4-control-plane-2 | master | 192.168.1.202 | Fedora Core OS | 4 | 16 | 120 |
| okd4-control-plane-3 | master | 192.168.1.203 | Fedora Core OS | 4 | 16 | 120 |
| okd4-compute-1 | worker | 192.168.1.204 | Fedora Core OS | 4 | 16 | 120 |
| okd4-compute-2 | worker | 192.168.1.205 | Fedora Core OS | 4 | 16 | 120 |
| okd4-services | Load Balancer,
DNS Server,
Web, NFS | 192.168.1.210 | CentOS 8 | 4 | 4 | 100 |
Download the OS Images
Download the latest Fedora Core OS installer from https://getfedora.org/en/coreos/download. Select the Bare Metal & Virtualized Tab. Download the Bare Metal ISO package.
Upload the installer to the local storage of the Proxmox node where you will create the VMs (just in case you have multiple nodes).
Create the VMs
From the Proxmox VE Web interface, right-click in the node and select create VM. Name the VM according to the table above starting with the okd4-bootstrap.
Select the Fedora Core OS image we uploaded earlier.
Leave the system tab with default values. Proxmox VE has already pre-selected the optimum setting for the selected Guest OS type. Set the size of the disk according to the table above.
Select 4 cores in the CPU tab as per the table above. Leave the rest of the setting unchanged by default, unless you know what you are doing.
Set the Memory to 16GB (16384 Mib).
If you followed the above instructions correctly, you should see the following values in the confirmation screen. Then just click finish.
After the VM is created. Proxmox will generate a MAC Address for the virtual network interface card. Take note of the MAC Address. Create a table similar to the above, but with the MAC Addresses column. You will need this later.
Repeat the above procedure for the rest of the VMs. Take note that the last VM in the table, okd4-services, is a CentOS 8 VM. You need to download the CentOS 8 installer ISO and upload it to Proxmox local storage. The latest CentOS 8 release can be downloaded here: http://isoredirect.centos.org/centos/8/isos/x86_64.
Download the one that ends with dvd1.iso.
Upload this file to the local storage of Proxmox VE and create the okd4-services VM as per the above procedure. Take note that this VM only has 4GB of RAM. Not 16GB.
You should have the following list of VMs at the end.
DHCP Address Reservation
Using the list of MAC Addresses of the VMs created earlier, we need to assign IP addresses to these MAC Addresses through DHCP Address reservation. Depending on the router, the process may be slightly different.
For my case, I have an ASUS Router and this is how the Address reservation looks like. Just a table of MAC addresses and their pre-assigned IP address.
When the VMs are started, they will get these IP Addresses via DHCP.
If you do not have a physical router or don't want to use your home router, you can run a PfSense router on a VM and configure the above VMs to be behind this router. Then you need to configure the same DHCP address reservation configuration. We will also need to revisit this router configuration after setting up a DNS server.
OKD Services
The okd4-services VM will run several things required to install and run Openshift/OKD.
- DNS Server - If you do not have an external/Raspberry Pi DNS server
- HA Proxy - Load Balancer
- Apache Web Server (httpd) - to host the OS images and ignition files during PXE booting. This service can be stopped after the installation.
- NFS server - to be used as Persistent Volume by Openshift Image Registry
Start the okd4-services VM. Navigate to console.
In the Installation Destination option, select custom, then done.
Delete the /home partition and leave the desired capacity for / empty.
Select the Network and Hostname. Enable the ethernet adapter, set the hostname and tick the automatically connect.
Then click begin installation and set the root password.
After installation is complete, run the following to add the EPEL repository to DNF/yum and update the OS.
sudo dnf install -y epel-release
sudo dnf update -y
sudo systemctl reboot
From this point on, we will do the rest of the installation and configuration from this VM. SSH to this new VM.
Create a DNS Server
The Openshift bootstrap procedure uses FQDNs to address the nodes. Because of this, we need to set up a DNS server. I used an old Raspberry Pi as DNS Server. But for simplicity, we will run the DNS server process on the okd4-services VM.
SSH to okd4-services VM, it should have picked up the IP address 192.168.1.210 if the DHCP address reservation configuration was done correctly.
This repo contains the configuration files we need.
The default domain name and cluster name for this installation is okd.home.lab. Where okd is the cluster name and home.lab is the base domain name. I created a script to help you update the configuration files if you want to change the domain name and cluster name. The following command will update the command to mycloud.mydomain.com.
./setdomain.sh mycloud mydomain.com
Then we need to install named as our DNS server. Copy down the configuration files and open up firewall port 53/UDP for DNS.
#DNS Server
cd
sudo dnf -y install bind bind-utils
sudo cp named.conf /etc/named.conf
sudo cp named.conf.local /etc/named/
sudo mkdir /etc/named/zones
sudo cp db* /etc/named/zones
sudo systemctl enable named
sudo systemctl start named
sudo systemctl status named
sudo firewall-cmd --permanent --add-port=53/udp
sudo firewall-cmd --reload
Now we need to go back to our router/DHCP server to set this DNS server as the DNS server of the router. All devices connected to the router should use this DNS server.
Create a Load Balancer
Copy the HA Proxy configuration file to the /etc/haproxy and then start HA Proxy service.
cd
sudo cp okd4_files/haproxy.cfg /etc/haproxy/haproxy.cfg
sudo setsebool -P haproxy_connect_any 1
sudo systemctl enable haproxy
sudo systemctl start haproxy
sudo systemctl status haproxy
Open TCP ports for Openshift/etcd clustering.
sudo firewall-cmd --permanent --add-port=6443/tcp
sudo firewall-cmd --permanent --add-port=22623/tcp
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload
Serve the Installation Files Over HTTP
Now, we need to install Apache webserver (HTTPD). We will host here, the files that we need to PXE boot the nodes on http port 8080.
sudo dnf install -y httpd
sudo sed -i 's/Listen 80/Listen 8080/' /etc/httpd/conf/httpd.conf
sudo setsebool -P httpd_read_user_content 1
sudo systemctl enable httpd
sudo systemctl start httpd
sudo firewall-cmd --permanent --add-port=8080/tcp
sudo firewall-cmd --reload
curl localhost:8080
A quick recap of what we just did.
- We created the VMs we need for the cluster
- We configured a router/DHCP server
- We installed Centos 8 on okd4-service VM
- We configured a DNS Server (named) on Okd4-services VM
- We created a load balancer using HA Proxy running in okd4-services VM
- We created an Apache webserver to host installation files in okd4-services VM
Installing OKD
Now that the infrastructure is ready, it's time to install OKD. The open-source upstream project of Openshift. The latest OKD release can be found at https://github.com/openshift/okd/releases. Update the links below accordingly to get the latest version.
Download the Openshift installer and the OC client. Extract the downloaded file and move the extracted binaries to /usr/local/bin. SSH to okd4-services VM and execute the following.
cd
wget https://github.com/openshift/okd/releases/download/4.5.0-0.okd-2020-10-15-235428/open\
shift-client-linux-4.5.0-0.okd-2020-10-15-235428.tar.gz
wget https://github.com/openshift/okd/releases/download/4.5.0-0.okd-2020-10-15-235428/open\
shift-install-linux-4.5.0-0.okd-2020-10-15-235428.tar.gz
#Extract the okd version of the oc client and openshift-install:
tar -zxvf openshift-client-linux-4.5.0-0.okd-2020-10-15-235428.tar.gz
tar -zxvf openshift-install-linux-4.5.0-0.okd-2020-10-15-235428.tar.gz
#Move the kubectl, oc, and openshift-install to /usr/local/bin and show the version:
sudo mv kubectl oc openshift-install /usr/local/bin/
#Test oc client and openshift-install command
oc version
openshift-install version
You may also check the status of the release builds at https://origin-release.apps.ci.l2s4.p1.openshiftapps.com/.
Setup the Openshift Installer
If you haven't done so, create an SSH key without a password. We will provide the SSH key to the install_config.yaml so that we can login to the VMs without password prompts.
#Generate an SSH key if you do not already have one.
ssh-keygen
Your ssh key public key is usually located at ~/.ssh/id_rsa.pub.
Create an installation directory and copy the installer config file to it. We will use this directory to hold the files generated by the opnshift-install command.
cd
mkdir install_dir
cp okd4_files/install-config.yaml ./install_dir
Get a Red Hat pull secret by logging in to https://cloud.redhat.com. Navigate to Cluster Manager > Create Cluster > Red Hat Openshift Container Platform > Run on Baremetal > User-Provisioned Infrastructure > Copy pull secret.
Edit the install_config.yaml. Replace the value of pull secret field with your pull secret or leave it as is if you don't have a Red Hat pull secret. Then, replace the value of sshKey field with your SSH key.
The last two lines of your install_config.yaml file should look like this.
Generate the Ignition Files
Run the installer to generate files and host the files in httpd.
cd
openshift-install create manifests --dir=install_dir/
# This lines disables schedule application pods on the master nodes
sed -i 's/mastersSchedulable: true/mastersSchedulable: False/' install_dir/manifests/clust\
er-scheduler-02-config.yml
openshift-install create ignition-configs --dir=install_dir/
rm -drf /var/www/html/okd4
sudo mkdir /var/www/html/okd4
sudo cp -R install_dir/* /var/www/html/okd4/
sudo chown -R apache: /var/www/html/
sudo chmod -R 755 /var/www/html/
curl localhost:8080/okd4/metadata.json
Download the Fedora Core OS image and signature. Rename with a shorter name and host the image and signature files in httpd under okd4 directory.
cd /var/www/html/okd4/
sudo wget https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20201004\
.3.0/x86_64/fedora-coreos-32.20201004.3.0-metal.x86_64.raw.xz
sudo wget https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20201004\
.3.0/x86_64/fedora-coreos-32.20201004.3.0-metal.x86_64.raw.xz.sig
sudo mv fedora-coreos-32.20201004.3.0-metal.x86_64.raw.xz fcos.raw.xz
sudo mv fedora-coreos-32.20201004.3.0-metal.x86_64.raw.xz.sig fcos.raw.xz.sig
sudo chown -R apache: /var/www/html/
sudo chmod -R 755 /var/www/html/
The latest Fedora Core OS release is available at https://getfedora.org/coreos/download?tab=cloud_launchable&stream=stable. Update the wget links accordingly to get the latest versions.
Starting the VMs
Now that the ignition files are generated. It's time to start the VMs. Select the okd4-bootstrap VM and navigate to Console. Start the VM. When you see the Fedora CoreOS startup screen, press TAB on the keyboard. This will initiate the PXE boot (booting over the network). In the command line below the screen, append the following arguments. This will install the Fedora CoreOS to /dev/sda disk using the image file and the ignition file we hosted in http in the earlier steps.
Bootstrap Node
coreos.inst.install_dev=/dev/sda
coreos.inst.image_url=http://192.168.1.210:8080/okd4/fcos.raw.xz
coreos.inst.ignition_url=http://192.168.1.210:8080/okd4/bootstrap.ign
In the console screen, it should look like this.
This step is painful when you make typo errors. Review what you typed before pressing "Enter". You also cannot copy-paste the arguments for these steps because Proxmox VE has no way of forwarding the clipboard to the VNC console session.
Master Nodes
Repeat the above step for all the other VMs, starting with the master nodes, okd4-control-plane-X. For the master nodes you need to replace the last argument ignition file name to master.ign.
coreos.inst.install_dev=/dev/sda
coreos.inst.image_url=http://192.168.1.210:8080/okd4/fcos.raw.xz
coreos.inst.ignition_url=http://192.168.1.210:8080/okd4/master.ign
Worker Nodes
Repeat the above steps for worker nodes.
coreos.inst.install_dev=/dev/sda
coreos.inst.image_url=http://192.168.1.210:8080/okd4/fcos.raw.xz
coreos.inst.ignition_url=http://192.168.1.210:8080/okd4/worker.ign
Bootstrap Progress
You can monitor the installation progress by running the following command.
cd
openshift-install --dir=install_dir/ wait-for bootstrap-complete --log-level=info
Once the bootstrap process completes, you should see the following messages.
Removing the Bootstrap Node
You can now shutdown/stop the okd4-bootstrap VM. Then we need to remove/comment out the bootstrap node from the load balancer so that API requests do not get routed to the bootstrap IP.
Edit the /etc/haproxy/haproxy.cfg file and reload the HAProxy configuration.
sudo sed '/ okd4-bootstrap /s/^/#/' /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy
Approving Certificate Signing Requests
We need to interact with the API in order to approve CSRs and to check the status/readiness of the cluster operators. Under the install_dir/auth there is a generated kubeconfig file. We will use this to access the API.
#login to the cluster
export KUBECONFIG=~/install_dir/auth/kubeconfig
oc whoami
oc get nodes
oc get csr
You should see only the master nodes when you ran the oc get nodes. This is because the worker nodes are still waiting for the CSR to be approved.
To quickly approve all CSR at once instead of doing it one-by-one, we will install jq and then use it to run oc adm certificate approve.
wget -O jq https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64
chmod +x jq
sudo mv jq /usr/local/bin/
jq --version
oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | \
xargs oc adm certificate approve
Approve the pending CSRs. You may need to run thing multiple times. Regularly run oc get csr. If you see a Pending CSR, run the below command again.
oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | \
xargs oc adm certificate approve
Check the Status of the Cluster Operators
You can check the status of cluster operators with the following command.
You should see an output like this.
Wait for the console to be available. Once it is available, we can point a browser to https://console-openshift-console.clustername.domain.name
You will get an SSL error because the certificate is not valid for this domain. That's normal. Just bypass the SSL error.
You will have to bypass SSL error twice because both the web console and the oAuth domains have an invalid certificate. After ignoring SSL errors for both web console and OAuth, you should see a login screen.
Login with user "kubeadmin".You can find the kubeadmin password in a file generated during the installation.
cat install_dir/auth/kubeadmin-password
Et voila!
Create a Cluster Admin User
The current kubeadmin user that we are using in the previous step is temporary. We need to create a permanent cluster administrator user. The fastest way to do this is by using
htpasswd as an authentication provider. We will create a secret under the openshift-config namespace and add a htpasswd provider in the cluster oAuth.
cd
cd okd4_files
htpasswd -c -B -b users.htpasswd testuser testpassword
oc create secret generic htpass-secret --from-file=htpasswd=users.htpasswd -n \
openshift-config
oc apply -f htpasswd_provider.yaml
The next time you login to the web console, you should have an option to choose an authentication provider and there should be an htpasswd option like below.
Login with credentials: testuser/testpassword. After authentication succeeds, we need to give this new user a cluster admin role. Note that the command below does not work before the first login.
oc adm policy add-cluster-role-to-user cluster-admin test
Now that we have a proper cluster admin user. We can then delete the kubeadmin temporary user.
oc delete secrets kubeadmin -n kube-system
You should not see the kube:admin login option the next time you log in.
Setup Image Registry
In order to complete the setup, we need to create persistent storage to be exposed as persistent volume to be used by the Openshift image registry.
In okd4_services VM, run the following, run the following to install NFS Server, start it and open up some ports in the firewall.
#Setting up NFS server
sudo dnf install -y nfs-utils
sudo systemctl enable nfs-server rpcbind
sudo systemctl start nfs-server rpcbind
sudo mkdir -p /var/nfsshare/registry
sudo chmod -R 777 /var/nfsshare
sudo chown -R nobody:nobody /var/nfsshare
echo '/var/nfsshare 192.168.1.0/24(rw,sync,no_root_squash,no_all_squash,no_wdelay)' | \
sudo tee /etc/exports
sudo setsebool -P nfs_export_all_rw 1
sudo systemctl restart nfs-server
sudo firewall-cmd --permanent --zone=public --add-service mountd
sudo firewall-cmd --permanent --zone=public --add-service rpc-bind
sudo firewall-cmd --permanent --zone=public --add-service nfs
sudo firewall-cmd --reload
Create a persistent volume in Openshift.
cd
oc create -f okd4_files/registry_pv.yaml
oc get pv
You should see an unclaimed PV.
Now change the management State to Managed and set the claim to blank.
oc edit configs.imageregistry.operator.openshift.io
Change according to below and save it.
managementState: Managed
storage:
pvc:
claim:
If you run another oc get pv, you should see that the PV has been claimed by the image-registry-storage.
Now you have an image registry backed by persistent storage via NFS.
Deploying our First Application
Now let's deploy a demo Java application called Spring Pet Clinic. The source code is available in Github repo:
https://github.com/spring-projects/spring-petclinic. We will use Openshift's S2I (Source-to-Image) feature to package and deploy a git repository into a running application.
First, create a project/namespace, then create an openshift app from the git source repo.
# Deploy Spring Pet Clinic
oc new-project spring-petclinic
oc new-app registry.redhat.io/ubi8/openjdk-11~https://github.com/spring-projects/spring-\
petclinic.git
oc expose svc/spring-petclinic
Open the web console. Select the "spring-petclinic" project. Click the Administrator and select the Developer view. In the Topology tab, you should see an application called spring-petclinic as shown below.
In the screenshot above, you can see that the application is being built. This may take a while depending on how fast your internet connection is. This is a Java Spring application that is known for having hundreds, maybe thousands, of dependencies. These dependencies are being downloaded from maven central during the build process.
Once the build and deployment are successful, the application icon will turn blue, and you should be able to navigate to the route URL.
We have just deployed our first application to our newly provisioned Openshift cluster running on Proxmox VE.
This concludes the "Running Openshift at Home" series. Check out the links below for the earlier parts.
