One of the key goals in #RDO is to provide a set of well tested and up-to-date
repositories that can be smoothly used by our users:
Operators deploying OpenStack with any of the available tools.
Upstream projects using RDO repos to develop and test their patches, as OpenStack
puppet modules, TripleO or kolla.
To include new patches in RDO packages as soon as possible, in RDO Trunk
repos
we build and publish new packages when commits are merged in upstream repositories.
To ensure the content of theses packages is trustworthy, we run different tests which
helps us to identify any problem introduced by the changes committed.
This post provides an overview of how we test RDO repositories. If you
are interested in collaborate with us in running an improving it, feel free to
let us know in rdo channel in freenode or rdo-list mailing list.
Promotion Pipelines
Promotion pipelines are composed by a set of related CI jobs that are executed
for each supported OpenStack release to test the content of a specific RDO repository.
Currently promotion pipelines are executed in diferent phases:
Define the repository to be tested. RDO Trunk repositories are identified
by a hash based on the upstream commit of the last built package. The content of
these repos doesn’t change over time. When a promotion pipeline is launched, it
grabs the latest consistent hash repo and sets it to be tested in following phases.
Build TripleO images. TripleO is
the recommended deployment tool for production usage in RDO and as such, is tested
in RDO CI jobs. Before actually deploying OpenStack using TripleO the required
images are built.
Deploy and test RDO. We run a set of jobs which deploy and test OpenStack
using different installers and scenarios to ensure they behave as expected. Currently,
following deployment tools and configurations are tested:
TripleO deployments. Using tripleo-quickstart
we deploy two different configurations, minimal
and minimal_pacemaker
which apply different settings that cover most common options.
OpenStack Puppet scenarios. Project puppet-openstack-integration (a.k.a. p-o-i)
maintains a set of puppet manifest to deploy different OpenStack services
combinations and configurations (scenarios) in a single server using OpenStack
puppet modules and run tempest smoke tests for the deployed services. The
tested services on each scenario can be found in the
README
for p-o-i. Scenarios 1, 2 and 3 are currently tested in RDO CI as
Packstack deployment. As part of the upstream testing, packstack defines
three deployment scenarios
to verify the correct behavior of the existing options. Note that tempest smoke tests
are executed also in these jobs. In RDO-CI we leverage those scenarios to test
new packages built in RDO repos.
Repository and images promotion. When all jobs in the previous phase succeed,
the tested repository is considered good and it is promoted so that users can use these
packages:
The repo is published using CentOS CDN in https://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-<release>-tested/)
The images are copied to https://buildlogs.centos.org/centos/7/cloud/x86_64/tripleo_images/<release>/delorean/
to be used by TripleO users.
Tools used in RDO CI
Jobs definitions are managed using Jenkings Job Builder, JJB,
via gerrit review workflow in review.rdoproject.org
weirdo is the tool we
use to run p-o-i and Packstack testing scenarios defined upstream inside RDO CI.
It’s composed of a set of ansible roles and playbooks that prepares the environment
and deploy and test the installers using the testing scripts provided by
the projects.
TripleO Quickstart
provides a set of scripts, ansible roles and pre-defined configurations to deploy
an OpenStack cloud using TripleO
in a simple and fully automated way.
ARA is used to store and visualize
the results of ansible playbook runs, making easier to analize and troubleshoot
them.
RDO is part of the CentOS Cloud Special Interest Group so we run promotion pipelines
in CentOS CI infrastructure where Jenkins
is used as continuous integration server..
Handling issues in RDO CI
An important aspect of running RDO CI is managing properly the errors found in the
jobs included in the promotion pipelines. The root cause of these issues sometimes
is in the OpenStack upstream projects:
Some problems are not catched in devstack-based jobs running in upstream gates.
In some cases, new versions of OpenStack services require changes in the deployment
tools (puppet modules, TripleO, etc…).
One of the contributions of RDO to upstream projects is to increase test coverage of
the projects and help to identify the problems as soon as possible. When we find them
we report it upstream as Launchpad bugs and propose fixes when possible.
Every time we find an issue, a new card is added to the TripleO and RDO CI Status
Trello board where
we track the status and activities carried out to get it fixed.
Status of promotion pipelines
If you are interested in the status of the promotion pipelines in RDO CI you can
check:
CentOS CI RDO view
can be used to see the result and status of the jobs for each OpenStack
release.
RDO Dashboard shows the overal
status of RDO packaging and CI.
More info
TripleO quickstart demostration by trown
Weirdo: A talk about CI in OpenStack and RDO by dmsimard.
ARA blog posts – from dmsimard blog
Ci in RDO: What do we test? – presentation in RDO and Ceph Meetup BCN.
Quelle: RDO
Published by