Upgrading OpenStack – Let’s give it a try!

An OpenStack Upgrade – Let’s give it a try!

Ever since rolling upgrade has been introduced in the icehouse release of OpenStack.
Icehouse enables the OpenStack engineers to upgrade controller infrastructure first, and subsequently, upgrade individual compute nodes without requiring downtime of the entire cloud to complete the upgrade.

What’s so special about this post, Since there are many blogs online to talk about the live upgrades in OpenStack?

Yes, of course! But this post mainly focuses on an OpenStack enthusiastic, who is so curious about the rolling upgrade feature and who would like to give a try in the small POC environment before implementing it in a big scale environment.

Although, Upgrading the big scale infrastructure also follows the same procedure. OpenStack engineers need to handle the high availability / Load balancing environment by removing the nodes from its pool before upgrading the each node to avoid the downtime which is not the scope of this post.

As I mentioned, this post helps the openstack engineers to try the rolling upgrade in a small scale and even smallest before trying it in the production ready setup. Yes! I mean All-in-one node setup 🙂

Let’s Upgrade…

You could try the rolling upgrade from any OpenStack release starting from Icehouse. In this post, I guide you on upgrading from Kilo to Liberty release.

Step 1: BackUp is a Mantra
As you all aware that the procedure for upgrading the OpenStack is not like “Slide to upgrade option available for IOS” :-). You need to backup all the possible data from the current setup. This will help us for rolling back the failed upgrade in worst case scenario.

$ mysqldump -u root -p –all-databases > all-kilo-db_backup.sql

Use the above command to backup the entire database from your MysqlDB. This will take some time depends on your DB size.

Step 2: Migrate to new package repository
This is where the real upgrading process begins. You must change the package repository from your current release to the new one.

Follow the below steps to remove the current (Kilo) repo and add the latest liberty repo.

#cd /etc/apt/sources.list.d
#rm -rf cloudarchive-kilo.list

# apt-get install software-properties-common
# add-apt-repository cloud-archive:liberty
# apt-get update

Repeat the above step2 in each node if you are trying this in multi-node OpenStack setup.

Step 3: Upgrade the packages
Now it’s time to upgrade all the OpenStack packages in one go.
Run the below command to download and install the latest version of package respected to the newly added cloud-archive.

#apt-get dist-upgrade

During this process, the package installation terminal prompt for asking whether to replace the new configuration file from the package maintainers repo or to keep the existing file as it is.

you need to select the option to install/replace the package maintainer version of the configuration file.
Selecting this option will replace the existing configuration file by renaming the existing old file with .old suffix as a backup.

If the upgrade process includes a new kernel, reboot your host to activate it.

Step 4: Install the missing Package.
There may be few packages that are newly added in the latest release of the OpenStack.

For example, “python-openstackclient” package is introduced in the liberty release. But the dist-upgrade would only process for upgrading the existing packages with its dependencies.
So you need to manually install those missing packages by referring the liberty installation guide available at http://docs.openstack.org/

Step 5: Update the services configuration files
At this time, you could notice that all the services are in stop status. This is because all the OpenStack service configuration files are not yet configured.

Again you have to refer the OpenStack installation guide to configure the OpenStack service starting from keystone, glance, nova, neutron and so on.

Step 6: DB Sync
Before starting the service, you need to run DB Sync command for all the service to populate the new tables in the database with respect to the upgraded release.

This is because the latest release may contain new features which will get reflected in its DB schema. So those new tables need to be populated before starting the service of newly upgraded OpenStack.

Here, I have listed the DB-Sync commands for most common services in OpenStack. You could also refer it from the installation guide.

To populate the Identity service database:
# su -s /bin/sh -c “keystone-manage db_sync” keystone

 

To Populate the Image service database:
# su -s /bin/sh -c “glance-manage db_sync” glance

 

To Populate the Compute database:
# su -s /bin/sh -c “nova-manage db sync” nova

 

To Populate the database:
# su -s /bin/sh -c “neutron-db-manage –config-file /etc/neutron/neutron.conf \
–config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head” neutron

 

Step 7: Restart the services
After successfully populating the DB, restart all the service. If the services stop working, then check the /var/log/upstart/ directory for the appropriate service logs.

There could be some dependencies issue you need to fix it. After all, you have successfully upgraded the OpenStack.

Cheers,
Vinoth Kumar Selvaraj