Red Hat have been pushing its automated installation of OpenStack with Director based on TripleO concept over the last years. Many Telcos are adopting this concept for the flexibility and agility that automation brings. This post called “IPMI trick for undercloud via VirtualBMC” is support material for your lab and use KVM instances simulating bare metals that can be set up via ironic and its IPMI (IP management interface) .
TripleO is an OpenStack project that aims to utilize OpenStack itself as the foundations for deploying OpenStack. To clarify, TripleO advocates the use of native OpenStack components, and their respective API’s to configure, deploy, and manage OpenStack environments itself. Undercloud is the OpenStack component that install the actual OpenStack called Overcloud. Containers are also a key component of this architecture.
IPMI trick for undercloud via VirtualBMC: High level topology
VirtualBMC installation
I am working with RHEL 8.2. You have, at least on my knowledge, two ways to install vbmc. IMPORTANT: I did it after installing the undercloud (tripleO).
VirtualBMC installation via pip3
vbmc is a python app , then you can use directly pip3 and adding the required packages. .
# those commands must run in the undercloud server (RHEL 8.2 in my case)
# add the repo
sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
# enable it
sudo ARCH=$( /bin/arch ) subscription-manager repos --enable "codeready-builder-for-rhel-8-${ARCH}-rpms"
# add the required packages
sudo yum -y install libvirt libvirt-devel gcc
# install virtualbmc (check dependencies in advanced)
sudo pip3 install virtualbmc
In this case, you will vbmc and vbmcd in a known path in ‘/usr/local/bin’
You have to enable the service and for this you will have to create the system file.
# I am assuming you're logged as root cat > /etc/systemd/system/vbmcd.service <<EOF [Install] WantedBy = multi-user.target [Service] BlockIOAccounting = True CPUAccounting = True ExecReload = /bin/kill -HUP $MAINPID ExecStart = /usr/local/bin/vbmcd --foreground Group = root MemoryAccounting = True PrivateDevices = False PrivateNetwork = False PrivateTmp = False PrivateUsers = False Restart = on-failure RestartSec = 2 Slice = vbmc.slice TasksAccounting = True TimeoutSec = 120 Type = simple User = root [Unit] After = libvirtd.service After = syslog.target After = network.target Description = vbmc service EOF systemctl daemon-reload systemctl enable vbmcd.service systemctl start vbmcd.service systemctl status vbmcd.service
it’s the service is in ‘active’ status, then you’re done with the installation
VirtualBMC installation via virtualenv
Or you can go with the one I used . It’s the same, however, the files we’ll be in a unknown path, then you will have to add it to your .bashrc file
# Assuming you have already install undercloud tripleO virtualenv --system-site-packages /opt/vbmc /opt/vbmc/bin/pip install virtualbmc cat > /etc/systemd/system/vbmcd.service <<EOF [Install] WantedBy = multi-user.target [Service] BlockIOAccounting = True CPUAccounting = True ExecReload = /bin/kill -HUP $MAINPID ExecStart = /opt/vbmc/bin/vbmcd --foreground Group = root MemoryAccounting = True PrivateDevices = False PrivateNetwork = False PrivateTmp = False PrivateUsers = False Restart = on-failure RestartSec = 2 Slice = vbmc.slice TasksAccounting = True TimeoutSec = 120 Type = simple User = root [Unit] After = libvirtd.service After = syslog.target After = network.target Description = vbmc service EOF systemctl daemon-reload systemctl enable vbmcd.service systemctl start vbmcd.service systemctl status vbmcd.service
You should see the service as ‘active’ like this:
[stack@undercloud ~]$ systemctl status vbmcd.service
● vbmcd.service - vbmc service
Loaded: loaded (/etc/systemd/system/vbmcd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2020-11-09 12:19:24 EST; 3 days ago
Main PID: 32504 (vbmcd)
Tasks: 5 (limit: 152850)
Memory: 43.8M
CPU: 4min 52.281s
CGroup: /vbmc.slice/vbmcd.service
├─32504 /opt/vbmc/bin/python3.6 /opt/vbmc/bin/vbmcd --foreground
└─34558 /opt/vbmc/bin/python3.6 /opt/vbmc/bin/vbmcd --foreground
Configuring vbmc and testing ipmi ports
Having the vbmcd service up, then we can start creating the first KVM instance and test it. My KVM host is at 10.5.0.101 (Centos 7.6, and yes, it’s a different distro and version), then I’ve generated the certificates in the undercloud server to access this KVM hosts directly via ssh and without password
After that I’ve tried to access virsh remotely via “virsh –connect” like this
[stack@undercloud ~]$ virsh --connect=qemu+ssh://root@10.5.0.101/system
Welcome to virsh, the virtualization interactive terminal.
Type: 'help' for help with commands
'quit' to quit
virsh # list --all
Id Name State
---------------------------------------
- overcloud-controller shut off
You can see the instance ‘overcloud-controller’ is off. keep it like this. You can check its configuration in our repo in github.
Let’s create the access to ipmi via port 6001
vbmc add overcloud-controller --port 6001 --username admin --password nuagenet --libvirt-uri qemu+ssh://root@10.5.0.101/system
Don’t forget to start i up.
/opt/vbmc/bin/vbmc start overcloud-controller
After that, you can test your ipmi port using ipmitool like this:
[stack@undercloud ~]$ ipmitool -I lanplus -U admin -P nuagenet -H 127.0.0.1 -p 6001 power status Chassis Power is off
Importing baremetal servers
Let’s create the JSON file to import this pseudo bare metal server:
[stack@undercloud ~]$ cat overcloud-compute1.json
{
"nodes": [
{
"arch": "x86_64",
"disk": "200",
"memory": "24576",
"name": "overcloud-compute1",
"pm_user": "admin",
"pm_addr": "127.0.0.1",
"pm_password": "nuagenet",
"pm_port": "6002",
"pm_type": "pxe_ipmitool",
"mac": [
"52:54:00:5e:68:b1"
],
"cpu": "12"
}
]
}
And then, just need to execute the import process. I am doing also the introspection.
openstack overcloud node import --introspect --provide overcloud-compute1.json
And we’re done.
Thanks for reading “IPMI trick for undercloud via VirtualBMC”. See ya!
Troubleshooting
No kernel image provided
If you get this message “RuntimeError: No kernel image provided and none of [‘x86_64/agent.kernel’, ‘agent.kernel’] found in /var/lib/ironic/httpboot” after executing command: “openstack overcloud node import –introspect –provide overcloud-compute1.json”
(undercloud) [stack@undercloud ~]$ openstack overcloud node import --introspect --provide overcloud-compute1.json
Exception occured while running the command
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
super(Command, self).run(parsed_args)
File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
return super(Command, self).run(parsed_args)
File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
return_code = self.take_action(parsed_args) or 0
File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_node.py", line 420, in take_action
http_boot=parsed_args.http_boot)
File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 1755, in update_nodes_deploy_data
http_boot))
RuntimeError: No kernel image provided and none of ['x86_64/agent.kernel', 'agent.kernel'] found in /var/lib/ironic/httpboot
No kernel image provided and none of ['x86_64/agent.kernel', 'agent.kernel'] found in /var/lib/ironic/httpboot
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.5.20.12', 34288), raddr=('10.5.20.12', 13000)>
Means you have missed to upload the image for your undercloud. Check this section: 4.8.1. Single CPU architecture overclouds.
In summary, you need to upload your images, for the CPU architecture you will have introspected.
[stack@director ~]$ source ~/stackrc (undercloud) [stack@director ~]$ sudo dnf install rhosp-director-images rhosp-director-images-ipa (undercloud) [stack@director ~]$ cd ~/images (undercloud) [stack@director images]$ for i in /usr/share/rhosp-director-images/overcloud-full-latest-16.1.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-16.1.tar; do tar -xvf $i; done (undercloud) [stack@director images]$ openstack overcloud image upload --image-path /home/stack/images/
Failed to bind DHCP server socket
My introspection was processing but was getting forever stuck trying to finish
(undercloud) [stack@undercloud ~]$ openstack overcloud node import --introspect --provide overcloud-compute1.json Waiting for messages on queue 'tripleo' with no timeout. 1 node(s) successfully moved to the "manageable" state. Successfully registered node UUID d954af44-a12e-4b8c-b377-146d82cbc0ad Waiting for introspection to finish... Waiting for messages on queue 'tripleo' with no timeout. Introspection of node attempt failed:d954af44-a12e-4b8c-b377-146d82cbc0ad. Retrying 1 nodes that failed introspection. Attempt 1 of 3 Introspection of node attempt failed:d954af44-a12e-4b8c-b377-146d82cbc0ad. Retrying 1 nodes that failed introspection. Attempt 2 of 3 Introspection of node attempt failed:d954af44-a12e-4b8c-b377-146d82cbc0ad. Retrying 1 nodes that failed introspection. Attempt 3 of 3
I did a tcpdump (“sudo tcpdump -i br-ctlplane”) at br-ctlplane (the bridge used by undercloud) and I could see the following
10:41:11.363487 IP undercloud.vim.nokialab.net > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 103, authtype none, intvl 1s, length 20 10:41:11.363584 IP undercloud.vim.nokialab.net > 224.0.0.18: VRRPv2, Advertisement, vrid 52, prio 103, authtype none, intvl 1s, length 20 10:41:12.364052 IP undercloud.vim.nokialab.net > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 103, authtype none, intvl 1s, length 20 10:41:12.364186 IP undercloud.vim.nokialab.net > 224.0.0.18: VRRPv2, Advertisement, vrid 52, prio 103, authtype none, intvl 1s, length 20 10:41:12.676965 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:5e:68:b1 (oui Unknown), length 396
I means the VM was sending the DHCP request and didn’t get answer from the undercloud DHCP service.
Then, I’ve realized the service ‘tripleo_ironic_inspector_dnsmasq.service’ was down. And I couldn’t make it work because some issue with t he container ‘ironic_inspector_dnsmasq’ with the log message /var/log/ironic-inspector/dnsmasq.log (I checked that inside the container) “failed to bind DHCP server socket: Address already in use”
Finally, I’ve realized I got a libvirt network using dnsmasq and blocking the port. After I remove it everything worked as a charm.
Thanks for reading: “IPMI trick for undercloud via VirtualBMC”
