da Agency

RDO CI promotion pipelines in a nutshell

One of the key goals in #RDO is to provide a set of well tested and up-to-date
repositories that can be smoothly used by our users:

Operators deploying OpenStack with any of the available tools.
Upstream projects using RDO repos to develop and test their patches, as OpenStack
puppet modules, TripleO or kolla.

To include new patches in RDO packages as soon as possible, in RDO Trunk
repos
we build and publish new packages when commits are merged in upstream repositories.
To ensure the content of theses packages is trustworthy, we run different tests which
helps us to identify any problem introduced by the changes committed.

This post provides an overview of how we test RDO repositories. If you
are interested in collaborate with us in running an improving it, feel free to
let us know in rdo channel in freenode or rdo-list mailing list.

Promotion Pipelines

Promotion pipelines are composed by a set of related CI jobs that are executed
for each supported OpenStack release to test the content of a specific RDO repository.
Currently promotion pipelines are executed in diferent phases:

Define the repository to be tested. RDO Trunk repositories are identified
by a hash based on the upstream commit of the last built package. The content of
these repos doesn’t change over time. When a promotion pipeline is launched, it
grabs the latest consistent hash repo and sets it to be tested in following phases.

Build TripleO images. TripleO is
the recommended deployment tool for production usage in RDO and as such, is tested
in RDO CI jobs. Before actually deploying OpenStack using TripleO the required
images are built.

Deploy and test RDO. We run a set of jobs which deploy and test OpenStack
using different installers and scenarios to ensure they behave as expected. Currently,
following deployment tools and configurations are tested:

TripleO deployments. Using tripleo-quickstart
we deploy two different configurations, minimal
and minimal_pacemaker
which apply different settings that cover most common options.
OpenStack Puppet scenarios. Project puppet-openstack-integration (a.k.a. p-o-i)
maintains a set of puppet manifest to deploy different OpenStack services
combinations and configurations (scenarios) in a single server using OpenStack
puppet modules and run tempest smoke tests for the deployed services. The
tested services on each scenario can be found in the
README
for p-o-i. Scenarios 1, 2 and 3 are currently tested in RDO CI as
Packstack deployment. As part of the upstream testing, packstack defines
three deployment scenarios
to verify the correct behavior of the existing options. Note that tempest smoke tests
are executed also in these jobs. In RDO-CI we leverage those scenarios to test
new packages built in RDO repos.

Repository and images promotion. When all jobs in the previous phase succeed,
the tested repository is considered good and it is promoted so that users can use these
packages:

The repo is published using CentOS CDN in https://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-<release>-tested/)
The images are copied to https://buildlogs.centos.org/centos/7/cloud/x86_64/tripleo_images/<release>/delorean/
to be used by TripleO users.

Tools used in RDO CI

Jobs definitions are managed using Jenkings Job Builder, JJB,
via gerrit review workflow in review.rdoproject.org
weirdo is the tool we
use to run p-o-i and Packstack testing scenarios defined upstream inside RDO CI.
It’s composed of a set of ansible roles and playbooks that prepares the environment
and deploy and test the installers using the testing scripts provided by
the projects.
TripleO Quickstart
provides a set of scripts, ansible roles and pre-defined configurations to deploy
an OpenStack cloud using TripleO
in a simple and fully automated way.
ARA is used to store and visualize
the results of ansible playbook runs, making easier to analize and troubleshoot
them.

Infrastructure

RDO is part of the CentOS Cloud Special Interest Group so we run promotion pipelines
in CentOS CI infrastructure where Jenkins
is used as continuous integration server..

Handling issues in RDO CI

An important aspect of running RDO CI is managing properly the errors found in the
jobs included in the promotion pipelines. The root cause of these issues sometimes
is in the OpenStack upstream projects:

Some problems are not catched in devstack-based jobs running in upstream gates.
In some cases, new versions of OpenStack services require changes in the deployment
tools (puppet modules, TripleO, etc…).

One of the contributions of RDO to upstream projects is to increase test coverage of
the projects and help to identify the problems as soon as possible. When we find them
we report it upstream as Launchpad bugs and propose fixes when possible.

Every time we find an issue, a new card is added to the TripleO and RDO CI Status
Trello board where
we track the status and activities carried out to get it fixed.

Status of promotion pipelines

If you are interested in the status of the promotion pipelines in RDO CI you can
check:

CentOS CI RDO view
can be used to see the result and status of the jobs for each OpenStack
release.

RDO Dashboard shows the overal
status of RDO packaging and CI.

More info

TripleO quickstart demostration by trown
Weirdo: A talk about CI in OpenStack and RDO by dmsimard.
ARA blog posts – from dmsimard blog
Ci in RDO: What do we test? – presentation in RDO and Ceph Meetup BCN.

Quelle: RDO

da Agency

A tale of Tempest rpm with Installers

Tempest is a set of integration tests to run against OpenStack Cloud.
Delivering robust and working OpenStack cloud is always challenging. To make sure what we deliver in #RDO is rock-solid, we use Tempest
to perform a set of API and scenario tests against a running cloud using different installers like puppet-openstack-integration,
packstack, and tripleo-quickstart.
And, it is the story of how we integrated RDO Tempest RPM package with installers so it can be consumed by various CI rather than using raw upstream sources.

And the story begins from here:

In RDO, we deliver Tempest as an rpm to be consumed by anyone to test their cloud. Till Newton release,
we maintained a fork of Tempest which contains the config_tempest.py script to auto generate tempest.conf
for your cloud and a set of helper scripts to run Tempest tests as well as with some backports for each release. From Ocata, we have changed the source of Tempest rpm from forked Tempest to upstream Tempest by keeping
the old source till Newton in RDO through rdoinfo. We are using rdo-patches branch
to maintain patches backports starting from Ocata release.

With this change, we have moved the config_tempest.py script from the forked Tempest repository to a separate project python-tempestconf
so that it can be used with vanilla Tempest to generate Tempest config automatically.

What have we done to make a happy integration between Tempest rpm and the installers?

Currently, puppet-openstack-integration, packstack, and tripleo-quickstart heavily use RDO packages. So using Tempest rpm with these installers will be the best match.
Before starting the integration, we need to make the initial ground ready. Till Newton release, all these installers are using Tempest from source in their respective CI.
We have started the match making of Tempest rpm with installers.
puppet-openstack-integration and packstack consume puppet-modules. So in order to consume Tempest rpm, first I need to fix the puppet-tempest.

puppet-tempest

It is a puppet-module to install and configure Tempest and openstack-services Tempest plugins based on the services available from source as well as packages.
So we have fixed puppet-tempest to install Tempest rpm from the package and created a Tempest workspace. In order to use that feature through puppet-tempest module [https://review.openstack.org/#/c/425085/].
you need to add install_from_source => ‘false’ and tempest_workspace => ‘path to tempest workspace’ to tempest.pp and it will do the job for you.
Now we are using the same feature in puppet-openstack-integration and packstack.

puppet-openstack-integration

It is a collection of scripts and manifests for puppet module testing (which powers the openstack-puppet CI).
From Ocata release, we have added a flag TEMPEST_FROM_SOURCE flag in run_tests.sh script.
Just change TEMPEST_FROM_SOURCE to false in the run_test.sh, Tempest is then installed and configured from packages using puppet-tempest.

packstack

It is a utility to install OpenStack on CentOS, Red Hat Enterprise Linux or other derivatives in proof of concept (PoC) environments. Till Newton, Tempest is installed and ran by packstack from the upstream source and behind the scenes, puppet-tempest does the job for us. From Ocata, we have replaced this feature by using Tempest RDO package. You can use this feature by running the following command:

$ sudo packstack –allinone –config-provision-tempest=y –run-tempest=y

It will perform packstack all in one installation and after that, it will install and configure Tempest and run smoke tests on deployed cloud.
We are using the same in RDO CI.

tripleo-quickstart

It is an ansible based project for setting up TripleO virtual environments.
It uses triple-quickstart-extras where validate-tempest roles exist which is used to install, configure and run Tempest on a tripleo deployment after installation. We have improved the validate-tempest role to use Tempest rpm package for all releases (supported by OpenStack upstream) by keeping the old workflow and as well as using Ocata Tempest rpm and using ostestr for running Tempest tests for all releases and using python-tempestconf to generate tempest.conf through this patch.

To see in action, Run the following command:

$ wget https://raw.githubusercontent.com/openstack/tripleo-quickstart/master/quickstart.sh
$ bash quickstart.sh –install-deps
$ bash quickstart.sh -R master –tags all $VIRTHOST

So finally the integration of Tempest rpm with installers is finally done and they are happily consumed in different CI and this will help to test and produce more robust OpenStack cloud in RDO as well as catch issues of Tempest with Tempest plugins early.

Thanks to apevec, jpena, amoralej, Haikel, dmsimard, dmellado, tosky, mkopec, arxcruz, sshnaidm, mwhahaha, EmilienM
and many more on rdo channel for getting this work done in last 2 and half months. It was a great learning experience.
Quelle: RDO

da Agency

RDO Ocata released

The #RDO community is pleased to announce the general availability of the RDO build for OpenStack Ocata for RPM-based distributions, #CentOS Linux 7 and Red Hat Enterprise Linux.
RDO is suitable for building private, public, and hybrid clouds. Ocata is the 15th release from the OpenStack project, which is the work of more than 2500 contributors from around the world (source).

The RDO community project curates, packages, builds, tests and maintains a complete OpenStack component set for RHEL and CentOS Linux and is a member of the CentOS Cloud Infrastructure SIG.
The Cloud Infrastructure SIG focuses on delivering a great user experience for CentOS Linux users looking to build and maintain their own on-premise, public or hybrid clouds.

All work on RDO, and on the downstream release, Red Hat OpenStack Platform, is 100% open source, with all code changes going upstream first.

Interesting things in the Ocata release include:

Significant Improvements to Tempest and Tempest plugin packaging in RDO

The OpenStack-Ansible project now supports deployment on top of CentOS with the help of RDO-packaged dependencies

For cloud operators, RDO now provides packages for some new OpenStack Services:

Tacker: an ETSI MANO NFV Orchestrator and VNF Manager
Congress: an open policy framework for the cloud
Vitrage: the OpenStack RCA (Root Cause Analysis) Service
Kolla: The Kolla project provides tooling to build production-ready container images for deploying OpenStack clouds

Some other notable additions:

novajoin: a dynamic vendordata plugin for the OpenStack nova metadata service to manage automatic host instantiation in an IPA server
ironic-ui: a new Horizon plugin to view and manage baremetal servers
python-virtualbmc VirtualBMC is a proxy that translates IPMI commands to libvirt calls. This allows projects such as OpenStack Ironic to test IPMI drivers using VMs.
python-muranoclient: a client for the Application Catalog service.
python-monascaclient: a client for the Monasca monitoring-as-a-service solution.
Shaker: the distributed data-plane testing tool built for OpenStack
Multi-architecture support: aarch64 builds are now provided through an experimental repository – enable the RDO ‘testing’ repositories to get started

From a networking perspective, we have added some new Neutron plugins that can help Cloud users and operators to address new use cases and scenarios:

networking-bagpipe: a mechanism driver for Neutron ML2 plugin using BGP E-VPNs/IP VPNs as a backend
networking-bgpvpn: an API and framework to interconnect BGP/MPLS VPNs to Openstack Neutron networks
networking-fujitsu: FUJITSU ML2 plugins/drivers for OpenStack Neutron
networking-l2gw: APIs and implementations to support L2 Gateways in Neutron
networking-sfc: APIs and implementations to support Service Function Chaining in Neutron

From the Packstack side, we have several improvements:

We have added support to install Panko and Magnum
Puppet 4 is now supported, and we have updated our manifests to cover the latest changes in the supported projects

Getting Started

There are three ways to get started with RDO.

To spin up a proof of concept cloud, quickly, and on limited hardware, try the All-In-One Quickstart. You can run RDO on a single node to get a feel for how it works.
For a production deployment of RDO, use the #TripleO Quickstart and you’ll be running a production cloud in short order.
Finally, if you want to try out OpenStack, but don’t have the time or hardware to run it yourself, visit TryStack, where you can use a free public OpenStack instance, running RDO packages, to experiment with the OpenStack management interface and API, launch instances, configure networks, and generally familiarize yourself with OpenStack. (TryStack is not, at this time, running Ocata, although it is running RDO.)

Getting Help

The RDO Project participates in a Q&A service at ask.openstack.org, for more developer-oriented content we recommend joining the rdo-list mailing list. Remember to post a brief introduction about yourself and your RDO story. You can also find extensive documentation on the RDO docs site.

The rdo channel on Freenode IRC is also an excellent place to find help and give help.

We also welcome comments and requests on the CentOS mailing lists and the CentOS and TripleO IRC channels (centos, centos-devel, and tripleo on irc.freenode.net), however we have a more focused audience in the RDO venues.

Getting Involved

To get involved in the OpenStack RPM packaging effort, see the RDO community pages and the CentOS Cloud SIG page. See also the RDO packaging documentation.

Join us in rdo on the Freenode IRC network, and follow us at @RDOCommunity on Twitter. If you prefer Facebook, we’re there too, and also Google+.
Quelle: RDO

da Agency

Writing RPM macro for OpenStack

RPM macro is a short string, always prefixed by % and generally surrounded by curly brackets ({}) which RPM will convert to a different and usually longer string.
Some macros can take arguments and some can be quite complex.
In RHEL, CentOS and Fedora, macros are provided by rpm package and from redhat-rpm-config.
In #RDO, OpenStack macros are provided by openstack-macros which comes from upstream rpm-packaging project.
You can find list of all macros under /usr/lib/rpm/macros.d/ directory.

To see the list of all available macros on your system:

$ rpm –showrc

For example: %{_bindir} is a rpm-macro which points to the binary directory where executables are usually stored.

To evaluate an rpm macro:

$ rpm –eval %{_bindir}

%py_build is the commonly used rpm-macro in RDO OpenStack packages which points to python setup.py build process.

$ rpm –eval %py_build

Motivation behind writing a new RPM macro for OpenStack packages
Currently, Tempest provides an external test plugin interface which enables anyone to integrate an external test suite as a part of Tempest run and each service Tempest plugin has an entrypoint
defined in setup.cfg through which tempest discovers it and list the Tempest plugins.
For example:

tempest.test_plugins =
heat_tests = heat_integrationtests.plugin:HeatTempestPlugin

In RDO OpenStack services RPM packages, In-tree Tempest plugins packages are provided by openstack-{service}-tests subpackage but the tempest plugin entrypoint is provided by the main package openstack-%{service}.
So once you have a working OpenStack environment with Tempest installed having no test subpackage installed. Then we tried to run tempest commands you would have encountered “No module heat_integrationtests.plugin found”
and you end up installing a hell lot of packages to fix this. The basic reason for the above error is tempest plugin entry point is installed by main OpenStack package but files pointing to entrypoint are not found.

To fix the above issue we have decided to separate out the tempest plugin entrypoint from the main package and move it to openstack-{service}-tests subpackage during rpmbuild process by creating a fake tempest plugin entry point
for all RDO services packages. Since it is a massive and similar change affecting all OpenStack services packages.
So, I have created %py2_entrypoint macro which is available in OpenStack Ocata release.
Here is the macro definition of %py2_entrypoint:
“`
# Create a fake tempest plugin entry point which will
# resides under %{python2_sitelib}/%{service}_tests.egg-info.
# The prefix is %py2_entrypoint %{modulename} %{service}
# where service is the name of the openstack-service or the modulename
# It should used under %install section
# the generated %{python2_sitelib}/%{service}_tests.egg-info
# will go under %files section of tempest plugin subpackage
# Example: %py2_entrypoint %{modulename} %{service}
# In most of the cases %{service} is same as %{modulename}
# but in case of neutron plugins it is different
# like servicename is neutron-lbaas and modulename is neutron_lbass
%py2_entrypoint()
egg_path=%{buildroot}%{python2_sitelib}/%{1}-*.egg-info
tempest_egg_path=%{buildroot}%{python2_sitelib}/%{1}_tests.egg-info
mkdir $tempest_egg_path
grep “tempest|Tempest” %{1}.egg-info/entry_points.txt >$tempest_egg_path/entry_points.txt
sed -i “/tempest|Tempest/d” $egg_path/entry_points.txt
cp -r $egg_path/PKG-INFO $tempest_egg_path
sed -i “s/%{2}/%{1}_tests/g” $tempest_egg_path/PKG-INFO
%nil

“`
Here is the list of tempest-plugin-entrypoint reviews.

Some learning from above macro:

[1.] You can use the shell script or Lua language to write macros.

[2.] %define is used to define a macro in spec file or you can directly place the macro in /usr/lib/rpm/macros.d/macros.openstack-rdo to consume it using rpmbuild process.

[3.] use %nil to showcase the end of the macro.

[4.] use %{1} to %{6} to pass variables in macros.

Above is a temporary solution. We are working upstream to separate out tempest plugins from OpenStack project to a new repo for easier management and packaging
in Pike release:https://review.openstack.org/#/c/405416/.

Thanks to Daniel, Alan Haikel and many others on rdo channel for getting the work done.
It was a great learning experience.
Quelle: RDO

da Agency

RDO Newton Released

The #RDO community is pleased to announce the general availability of the RDO build for OpenStack Newton for RPM-based distributions – #CentOS Linux 7 and Red Hat Enterprise Linux. RDO is suitable for building private, public, and hybrid clouds. Newton is the 14th release from the OpenStack project, which is the work of more than 2700 contributors from around the world.
(Source)

The RDO community project curates, packages, builds, tests, and maintains a complete OpenStack component set for RHEL and CentOS Linux and is a member of the CentOS Cloud Infrastructure SIG. The Cloud Infrastructure SIG focuses on delivering a great user experience for CentOS Linux users looking to build and maintain their own on-premise, public or hybrid clouds. At latest count, RDO contains 1157 packages.

All work on RDO, and on the downstream release, Red Hat OpenStack Platform, is 100% open source, with all code changes going upstream first.

Getting Started

There are three ways to get started with RDO.

To spin up a proof of concept cloud, quickly, and on limited hardware, try the RDO QuickStart You can run RDO on a single node to get a feel for how it works.

For a production deployment of RDO, use the TripleO Quickstart and you’ll be running a production cloud in short order.

Finally, if you want to try out OpenStack, but don’t have the time or hardware to run it yourself, visit TryStack, where you can use a free public OpenStack instance, running RDO packages, to experiment with the OpenStack management interface and API, launch instances, configure networks, and generally familiarize yourself with OpenStack

Getting Help

The RDO Project participates in a Q&A service at ask.openstack.org, for more developer oriented content we recommend joining the rdo-list mailing list. Remember to post a brief introduction about yourself and your RDO story. You can also find extensive documentation on the RDO docs site.

The rdo channel on Freenode IRC is also an excellent place to find help and give help.

We also welcome comments and requests on the CentOS Mailing lists and the CentOS IRC Channels (centos, and centos-devel, on irc.freenode.net), however we have a more focused audience in the RDO venues.

Getting Involved

To get involved in the OpenStack RPM packaging effort, see the RDO community pages and the CentOS Cloud SIG page. See also the RDO packaging documentation.

Join us in rdo on the Freenode IRC network, and follow us at @RDOCommunity on Twitter. If you prefer Facebook, we’re there too, and also Google+.

And, if you’re going to be in Barcelona for the OpenStack Summit two weeks from now, join us on Tuesday evening at the Barcelona Princess, 5pm – 8pm, for an evening with the RDO and Ceph communities. If you can’t make it in person, we’ll be streaming it on YouTube.
Quelle: RDO