OpenStack Third Party Testing: How to use Zuul for your Jenkins

When setting up OpenStack third party testing, it is useful to trigger rechcecking from Gerrit comment such as recheck no bug or recheck-<vendor> and there is an onging discussion whether recheck support should be a requirement.

This blog post describes the manual procedure of Zuul and how to migrate the trigger of Jenkins jobs from gerrit trigger jenkins plugin to Zuul.

After migrating to Zuul, I feel it is nice and makes it easy to control Jenkins jobs trigger flexibly.

If you are interested in setting up Zuul by puppet, please check the puppet manifests from Jay Pipes. https://github.com/jaypipes/os-ext-testing

Prerequisite

• Ubuntu 12.04.3 LTS

Zuul setup

Installation

Install required dependency packages:

aptitude install python-pip
aptitude install python-webob python-lockfile python-paste
aptitude install python-yaml python-paramiko python-daemon

Create a user and a group for zuul service:

groupadd zuul
useradd -m -d /home/zuul -s /bin/bash -g zuul zuul

Retrieve and install zuul:

git clone https://git.openstack.org/openstack-infra/zuul /opt/zuul
pip install /opt/zuul

Configuration files

Preparation (optional)

puppet module in openstack-infra/config has useful config examples. Checkout them first to somewhere. I refer to the directory where you checkedout openstack-infra/config git repository by <INFRA_CONFIG>.

git clone https://git.openstack.org/openstack-infra/config

/etc/zuul/zuul.conf

Customize the following parameters. The detail description is available at http://ci.openstack.org/zuul/zuul.html#zuul-conf.

The following parameters are important:

• [gerrit]
  • server : review.openstack.org
  • user : gerrit service account name
  • sshkey : gerrit service account ssh private key
  • baseurl : https://review.openstack.org/ if server field is different from review.openstack.org. It is useful if your zuul server is behind proxy ()or you are using SSH port forwarding to access OpenStack gerrit).
• [zuul]
  • url_pattern : URL of each Jenkins job. Usually used to point a build job.
  • zuul_url : It is used as git repo URL. Jenkins job fetches a commit from this URL.
  • status_url : There is no need to customize it unless you post a start message.

[gearman]
server=127.0.0.1

[gearman_server]
start=true
log_config=/etc/zuul/gearman-logging.conf

[gerrit]
server=review.openstack.org
user=jenkins
sshkey=/var/lib/zuul/ssh/id_rsa

[zuul]
layout_config=/etc/zuul/layout.yaml
log_config=/etc/zuul/logging.conf
state_dir=/var/lib/zuul
git_dir=/var/lib/zuul/git
push_change_refs=false
url_pattern=http://<your log server>/logs/{job.name}/{build.number}
status_url=http://status.openstack.org/zuul/
job_name_in_report=true
zuul_url=http://ostack10.svp.cl.nec.co.jp/p

/etc/zuul

Prepare logging configurations.

cp config/modules/openstack_project/files/zuul/gearman-logging.conf /etc/zuul
cp config/modules/openstack_project/files/zuul/logging.conf /etc/zuul

openstack_functions.py is optional but it is useful to define LOG_PATH or single_use_node (disable a used slave node after the job completes).

cp config/modules/openstack_project/files/zuul/openstack_functions.py /etc/zuul

/etc/init.d/zuul

Create a startup script of zuul.

cp config/modules/zuul/files/zuul.init /etc/init.d/zuul
chmod +x /etc/init.d/zuul

/etc/zuul/layout.yaml

layout.yaml is a heart of zuul. The following is an example I used. The detail description is available at http://ci.openstack.org/zuul/zuul.html#layout-yaml.

• check pipeline watches three gerrit events from master branch. branch parameter is optional. If you want to watch all branches, remove it.
  • Note that a target branch(es) can also be limited by branch parameter of jobs configuration. branch in jobs configuration selects jobs to be executed, but zuul post a result even when no job is selected. For such case, it looks better to limit a target branch in pipelines configuration.

• If you remove success and/or failure entries from gerrit trigger definition, a result of a build job will not be posted to gerrit. It is useful while you are preparing zuul and jenkins.

• If parameter-function: use_single_node is specified, a slave node where the build run will be offline. Enable it depending on your test infra.

includes:
  - python-file: openstack_functions.py

pipelines:
  - name: check
    description: Newly uploaded patchsets enter this pipeline to receive an initial +/-1 Verified vote from Jenkins.
    manager: IndependentPipelineManager
    precedence: low
    trigger:
      gerrit:
        - event: patchset-created
          branch: ^master$
        - event: change-restored
          branch: ^master$
        - event: comment-added
          branch: ^master$
          comment_filter: (?i)^\s*recheck(( (?:bug|lp)[\s#:]*(\d+))|( no bug))\s*$
    success:
      gerrit:
        verified: 1
    failure:
      gerrit:
        verified: -1

jobs:
  - name: ^.*$
    parameter-function: set_log_url
  - name: ^Neutron_Gate$
    parameter-function: single_use_node
  #- name: ^Neutron_Gate$
  #  branch: ^master$

projects:
  - name: openstack/neutron
    check:
      - Neutron_Gate
  - name: openstack-dev/sandbox
    testcheck:
      - TestJob

Create working directories

Create working directories for zuul.

# mkdir /var/log/zuul
# chown zuul /var/log/zuul

# mkdir /var/run/zuul
# chown zuul:zuul /var/run/zuul

# mkdir /var/lib/zuul
# chown zuul:zuul /var/lib/zuul
# mkdir /var/lib/zuul/git
# chown zuul /var/lib/zuul/git
# mkdir /var/lib/zuul/ssh
# chown zuul:zuul /var/lib/zuul/ssh
# chmod 700 /var/lib/zuul/ssh
# mkdir /var/lib/zuul/www

Apache

git repository

Zuul provides a git repository (if push_change_refs=false). The easiest way is to export git repositories via apache.

SetEnv GIT_PROJECT_ROOT /var/lib/zuul/git/
SetEnv GIT_HTTP_EXPORT_ALL

AliasMatch ^/p/(.*/objects/[0-9a-f]{2}/[0-9a-f]{38})$ /var/lib/zuul/git/$1
AliasMatch ^/p/(.*/objects/pack/pack-[0-9a-f]{40}.(pack|idx))$ /var/lib/zuul/git/$1
ScriptAlias /p/ /usr/lib/git-core/git-http-backend/

Enable required apache modues.

a2enmod proxy
a2enmod proxy_http
a2enmod rewrite

status page

(It is not tested yet.)

aptitude install libjs-jquery
cd /opt
git clone https://github.com/mathiasbynens/jquery-visibility.git
cd /var/lib/zuul/www
ln -s /opt/zuul/etc/status/public_html/index.html
ln -s /opt/zuul/etc/status/public_html/app.js
ln -s /usr/share/javascript/jquery/jquery.min.js
ln -s /opt/jquery-visibility/jquery-visibility.js
wget http://getbootstrap.com/2.3.2/assets/bootstrap.zip
unzip bootstrap.zip
rm bootstrap.zip

/etc/apache2/site-enabled/000-default

RewriteEngine on
RewriteRule ^/status.json$ http://127.0.0.1:8001/status.json [P]

Reload apache

Finally make sure apache loads a new configuration.

service apache2 reload

Start Zuul

service zuul start

The detail of command line under the hood is described at http://ci.openstack.org/zuul/zuul.html#starting-zuul.

Jenkins configuration for zuul

Zuul uses Gearman to dispatch build requests to Jenkins and receive build results. Jenkins Gearman plugin needs to be installed.

1. Install Gearman plugin
2. In Gearman Plugin config menu, set Gearman server host and port and test connection.
3. In Gearman Plugin config menu, ensure to check "Enable Gearman".

Migrate Jenkins job trigger to zuul

Jenkins Gearman plugin dispatch Jenkins builds when it received build requests from Zuul (via Gearman), so no build trigger is needed in Jenkins job configruation.

Make sure that remove "Gerrit Event" from "Build Triggers" in Jenkis job configuration.

Otherwise, one gerrit event triggers two Jenkins builds per job through Gerrit Trigger plugin and Zuul.

That's all.

Tips

Defining vendor specific recheck

It is useful to kick only your CI system without affecting other CI systems. The following layout.yaml configuration allows you to trigger your build by a comment recheck-XXXX in addition to the existing recheck no bug or recheck bug NNNNN comments.

    trigger:
      gerrit:
        - event: patchset-created
        - event: change-restored
        - event: comment-added
          comment_filter:
            - (?i)^\s*recheck(( (?:bug|lp)[\s#:]*(\d+))|( no bug))\s*$
            - (?i)^\s*recheck-XXXX\s*$