Personal tools
The Open Lighting Project has moved!

We've launched our new site at www.openlighting.org. This wiki will remain and be updated with more technical information.

Difference between revisions of "OLA Buildbot"

From wiki.openlighting.org

Jump to: navigation, search
m (Prerequisites & Warning: Add --enable-python-libs requirement, mention lint requirements)
m (OpenBSD Instructions: Add eg++ etc)
 
(24 intermediate revisions by 3 users not shown)
Line 2: Line 2:
  
 
*Web UI: http://buildbot.openlighting.org
 
*Web UI: http://buildbot.openlighting.org
*IRC: OLA-buildbot in #openlighting-buildbot on irc.freenode.org
+
*IRC: OLA-buildbot-ch in #openlighting-build on irc.freenode.org
  
 
We're currently only building ola, but this could be extended to do other projects such as the QT GUI etc.
 
We're currently only building ola, but this could be extended to do other projects such as the QT GUI etc.
  
 
== Master Configuration ==
 
== Master Configuration ==
The buildbot master configuration is stored in git at https://code.google.com/p/open-lighting/source/browse?repo=buildbot-config . It needs checking out, and symlinking to replace master.cfg in the buildbot master's config directory.
+
The buildbot master configuration is stored in git at https://github.com/OpenLightingProject/buildbot . It needs checking out, and symlinking to replace master.cfg in the buildbot master's config directory.
  
N.B. If the git repository is ever moved or changed, ensure the master's gitpoller-workdir folder is emptied out, or the poller won't work properly.
+
N.B. If the main open-lighting git repository (as opposed to buildbot config) is ever moved or changed, ensure the master's gitpoller-workdir folder is emptied out, or the poller won't work properly.
  
 
= Adding a Slave =
 
= Adding a Slave =
Line 19: Line 19:
 
Slaves execute code directly from the git repo. Even though submits to the git repo are locked down, this is still a possible attack vector for your machine. For this reason it's best to run build slaves within a virtual machine.  TODO: link to some VM solutions.
 
Slaves execute code directly from the git repo. Even though submits to the git repo are locked down, this is still a possible attack vector for your machine. For this reason it's best to run build slaves within a virtual machine.  TODO: link to some VM solutions.
  
At the very least you should run the buildslave as a separate user (not root!). Slave passwords aren't stored in the git repo for security, you'll need to get Simon to add new ones.
+
At the very least you should run the buildslave as a separate user (not root!). Slave passwords aren't stored in the git repo for security, you'll need to get Simon or Peter to add new ones.
  
 
The buildbot performs full build & test runs with all the options enabled. Please make sure you have all the necessary libraries installed on your system. You need to be able to complete a  
 
The buildbot performs full build & test runs with all the options enabled. Please make sure you have all the necessary libraries installed on your system. You need to be able to complete a  
Line 25: Line 25:
 
<pre>
 
<pre>
 
autoconf -i
 
autoconf -i
./configure --enable-e133 --enable-rdm-tests --enable-python-libs
+
./configure --enable-e133 --enable-rdm-tests --enable-python-libs --enable-ja-rule
 
make
 
make
 
make check
 
make check
Line 32: Line 32:
 
cycle before proceeding. If you have trouble ask on the mailing list.
 
cycle before proceeding. If you have trouble ask on the mailing list.
  
If you're running the lint check, you need ccplint.py in your path somewhere, see README.developer for info on how to obtain it.
+
If you're running the lint check, you need cpplint.py in your path somewhere, see README.developer for info on how to obtain it.
  
Buildbot slaves need to connect to  buildbot.openlighting.org:9989 . Make sure your firewall allows this. No port forwarding for inbound connections is required.
+
Buildbot slaves need to connect to  buildmaster.openlighting.org:9989 . Make sure your firewall allows this. No port forwarding for inbound connections is required.
  
 
== Debian / Ubuntu Instructions ==
 
== Debian / Ubuntu Instructions ==
Line 48: Line 48:
 
=== Slave Configuration ===
 
=== Slave Configuration ===
  
Create the slave:
+
Create the slave (the package creates the buildbot user automatically for you):
  
 
<pre>
 
<pre>
  sudo -u buildbot buildslave create-slave --usepty=0 /var/lib/buildbot/slaves/ola buildbot.openlighting.org:9989 <slave user> <slave password>
+
  sudo -u buildbot buildslave create-slave --usepty=0 /var/lib/buildbot/slaves/ola buildmaster.openlighting.org:9989 <slave user> <slave password>
 
</pre>
 
</pre>
  
Line 71: Line 71:
 
Check the log if there are any issues, or confirm the slave is registered at http://buildbot.openlighting.org/buildslaves:
 
Check the log if there are any issues, or confirm the slave is registered at http://buildbot.openlighting.org/buildslaves:
 
  tail -f /var/lib/buildbot/slaves/ola/twistd.log
 
  tail -f /var/lib/buildbot/slaves/ola/twistd.log
 +
 +
== FreeBSD Instructions ==
 +
 +
This was tested on FreeBSD 10.
 +
 +
=== Build Slave Installation ===
 +
As root:
 +
<pre>
 +
pkg install buildbot-slave
 +
</pre>
 +
 +
=== Slave Configuration ===
 +
 +
Setup a new user:
 +
 +
<pre>
 +
su -
 +
adduser
 +
</pre>
 +
Username: ola-build-slave
 +
Use password-based authentication? No
 +
 +
Setup the slave:
 +
 +
Switch to the slave user, I did:
 +
<pre>
 +
su - #To root
 +
su - ola-build-slave #To slave user
 +
cd ~
 +
</pre>
 +
 +
Export the variables needed for configure on FreeBSD:
 +
<pre>
 +
export CPPFLAGS="-I/usr/local/include/"
 +
export LDFLAGS="-L/usr/local/lib/"
 +
</pre>
 +
 +
Setup the slave itself:
 +
<pre>
 +
/usr/local/bin/buildslave create-slave ola-slave buildmaster.openlighting.org:9989 <slave user> <slave password>
 +
</pre>
 +
 +
Edit <tt>ola-slave/info/admin</tt> and <tt>ola-slave/info/host</tt> so your slave shows up correctly.
 +
 +
Then start the slave:
 +
 +
<pre>
 +
/usr/local/bin/buildslave start ola-slave
 +
</pre>
 +
 +
You can look at the logs by running
 +
 +
<pre>
 +
tail -f ola-slave/twistd.log
 +
</pre>
 +
 +
At this point you can go to http://buildbot.openlighting.org/buildslaves and you should see your slave connected. It's probably worth asking someone to kick off a build at this point so we can check your slave is working.
 +
 +
Finally, if everything looks good, configure your slave to launch on startup by editing the crontab for the ola-build-slave user (crontab -e). Add the following line:
 +
 +
<pre>
 +
@reboot /usr/local/bin/buildslave start /home/ola-build-slave/ola-slave
 +
</pre>
 +
 +
 +
== NetBSD Instructions ==
 +
 +
This was tested on NetBSD 6.
 +
 +
=== Build Slave Installation ===
 +
As root:
 +
<pre>
 +
pkgin install py27-buildbot-slave
 +
</pre>
 +
 +
=== Slave Configuration ===
 +
 +
Setup a new user:
 +
 +
<pre>
 +
su -
 +
useradd -m ola-build-slave
 +
</pre>
 +
Use password-based authentication? No
 +
 +
Setup the slave:
 +
 +
Switch to the slave user:
 +
<pre>
 +
su - ola-build-slave
 +
cd ~
 +
</pre>
 +
 +
Export the variables needed for git and configure on NetBSD (add these to the .shrc or equivalent for next time):
 +
<pre>
 +
export CPPFLAGS="-I/usr/pkg/include/"
 +
export LDFLAGS="-L/usr/pkg/lib/"
 +
export GIT_SSL_NO_VERIFY=true
 +
</pre>
 +
 +
Setup the slave itself:
 +
<pre>
 +
/usr/pkg/bin/buildslave create-slave ola-slave buildmaster.openlighting.org:9989 <slave user> <slave password>
 +
</pre>
 +
 +
Edit <tt>ola-slave/info/admin</tt> and <tt>ola-slave/info/host</tt> so your slave shows up correctly.
 +
 +
Then start the slave:
 +
 +
<pre>
 +
/usr/pkg/bin/buildslave start ola-slave
 +
</pre>
 +
 +
You can look at the logs by running
 +
 +
<pre>
 +
tail -f ola-slave/twistd.log
 +
</pre>
 +
 +
At this point you can go to http://buildbot.openlighting.org/buildslaves and you should see your slave connected. It's probably worth asking someone to kick off a build at this point so we can check your slave is working.
 +
 +
Finally, if everything looks good, configure your slave to launch on startup by editing the crontab for the ola-build-slave user (crontab -e). Add the following line:
 +
 +
<pre>
 +
@reboot /usr/pkg/bin/buildslave start /home/ola-build-slave/ola-slave
 +
</pre>
 +
 +
== OpenBSD Instructions ==
 +
 +
This was tested on OpenBSD 5.4.
 +
 +
=== Build Slave Installation ===
 +
As root:
 +
<pre>
 +
pkg_add py-buildslave
 +
</pre>
 +
 +
=== Slave Configuration ===
 +
 +
Setup a new user:
 +
 +
<pre>
 +
su -
 +
useradd -m ola-build-slave
 +
</pre>
 +
 +
Setup the slave:
 +
 +
Switch to the slave user:
 +
<pre>
 +
su - ola-build-slave
 +
cd ~
 +
</pre>
 +
 +
Export the variables needed for autoconf and configure on OpenBSD (add these to the .profile or equivalent for next time), the eg++/egcc lines are needed if the installed gcc is too old to pass configure, install the eg++ one and then use that line:
 +
<pre>
 +
export AUTOCONF_VERSION=2.69
 +
export AUTOMAKE_VERSION=1.13
 +
export CPPFLAGS="-I/usr/local/include/"
 +
export LDFLAGS="-L/usr/local/lib/"
 +
export CXX="eg++" CC="egcc"
 +
</pre>
 +
 +
Setup the slave itself:
 +
<pre>
 +
/usr/local/bin/buildslave create-slave ola-slave buildmaster.openlighting.org:9989 <slave user> <slave password>
 +
</pre>
 +
 +
Edit <tt>ola-slave/info/admin</tt> and <tt>ola-slave/info/host</tt> so your slave shows up correctly.
 +
 +
Then start the slave:
 +
 +
<pre>
 +
/usr/local/bin/buildslave start ola-slave
 +
</pre>
 +
 +
You can look at the logs by running
 +
 +
<pre>
 +
tail -f ola-slave/twistd.log
 +
</pre>
 +
 +
At this point you can go to http://buildbot.openlighting.org/buildslaves and you should see your slave connected. It's probably worth asking someone to kick off a build at this point so we can check your slave is working.
 +
 +
Finally, if everything looks good, configure your slave to launch on startup by editing the crontab for the ola-build-slave user (crontab -e). Add the following line:
 +
 +
<pre>
 +
@reboot /usr/local/bin/buildslave start /home/ola-build-slave/ola-slave
 +
</pre>
  
 
== Other Platforms ==
 
== Other Platforms ==
Line 103: Line 292:
 
su ola-build-slave
 
su ola-build-slave
 
cd ~
 
cd ~
buildslave create-slave ola-slave buildbot.openlighting.org:9989 <slave user> <slave password>
+
buildslave create-slave ola-slave buildmaster.openlighting.org:9989 <slave user> <slave password>
 +
</pre>
 +
 
 +
Edit <tt>ola-slave/info/admin</tt> and <tt>ola-slave/info/host</tt> so your slave shows up correctly.
 +
 
 +
Then start the slave:
 +
 
 +
<pre>
 +
buildslave start ola-slave
 +
</pre>
 +
 
 +
You can look at the logs by running
 +
 
 +
<pre>
 +
tail -f ola-slave/twistd.log
 +
</pre>
 +
 
 +
At this point you can go to http://buildbot.openlighting.org/buildslaves and you should see your slave connected. It's probably worth asking someone to kick off a build at this point so we can check your slave is working.
 +
 
 +
Finally, if everything looks good, configure your slave to launch on startup by editing the crontab for the ola-build-slave user (crontab -e). Add the following line:
 +
 
 +
<pre>
 +
@reboot buildbot start /home/ola-build-slave/ola-slave
 +
</pre>
 +
 
 +
== Enabling the C++ Lint Checker ==
 +
 
 +
The lint checker enforces C++ style. We only run this once per change but it's good to have multiple lint-enabled hosts in case one is down.
 +
 
 +
Download the cpplint checker and put it in /usr/local/bin
 +
 
 +
<pre>
 +
wget http://google-styleguide.googlecode.com/svn/trunk/cpplint/cpplint.py
 +
sudo cp cpplint.py /usr/local/bin
 +
sudo chmod a+x /usr/local/bin/cpplint.py
 +
</pre>
 +
 
 +
Finally let Simon or Peter know, or raise an issue, asking them to enable C++ lint for your slave.
 +
 
 +
== Enabling the Javascript Lint Checker ==
 +
 
 +
The lint checker enforces Javascript style. We only run this once per change but it's good to have multiple lint-enabled hosts in case one is down.
 +
 
 +
Either:
 +
*Install the closure-linter deb package if it's present on your OS
 +
*Install the gjslint checker by following the relevant instructions here (you man need to install python-setuptools to get easy_install on Debian/Ubuntu) https://developers.google.com/closure/utilities/docs/linter_howto
 +
 
 +
Finally let Simon or Peter know, or raise an issue, asking them to enable Javascript lint for your slave.
 +
 
 +
== Enabling the heap checker ==
 +
 
 +
First we need libunwind:
 +
 
 +
<pre>
 +
sudo apt-get install libunwind8-dev
 +
</pre>
 +
 
 +
Then download the latest gperftools from https://code.google.com/p/gperftools/downloads/list .
 +
 
 +
<pre>
 +
wget ....
 +
tar -zxf gperftools-2.1.tar.gz
 +
cd gperftools-2.1
 +
./configure
 +
make
 +
sudo make install
 +
sudo ldconfig
 
</pre>
 
</pre>
 +
Finally let Simon or Peter know, or raise an issue, asking them to enable the heap checker for your slave.

Latest revision as of 18:38, 29 December 2020

We run a Buildbot instance for OLA.

We're currently only building ola, but this could be extended to do other projects such as the QT GUI etc.

Master Configuration

The buildbot master configuration is stored in git at https://github.com/OpenLightingProject/buildbot . It needs checking out, and symlinking to replace master.cfg in the buildbot master's config directory.

N.B. If the main open-lighting git repository (as opposed to buildbot config) is ever moved or changed, ensure the master's gitpoller-workdir folder is emptied out, or the poller won't work properly.

Adding a Slave

Buildbot documentation is at http://docs.buildbot.net/current/manual/installation.html#creating-a-buildslave . The steps below should cover everything you need though.

Prerequisites & Warning

Slaves execute code directly from the git repo. Even though submits to the git repo are locked down, this is still a possible attack vector for your machine. For this reason it's best to run build slaves within a virtual machine. TODO: link to some VM solutions.

At the very least you should run the buildslave as a separate user (not root!). Slave passwords aren't stored in the git repo for security, you'll need to get Simon or Peter to add new ones.

The buildbot performs full build & test runs with all the options enabled. Please make sure you have all the necessary libraries installed on your system. You need to be able to complete a

autoconf -i
./configure --enable-e133 --enable-rdm-tests --enable-python-libs --enable-ja-rule
make
make check

cycle before proceeding. If you have trouble ask on the mailing list.

If you're running the lint check, you need cpplint.py in your path somewhere, see README.developer for info on how to obtain it.

Buildbot slaves need to connect to buildmaster.openlighting.org:9989 . Make sure your firewall allows this. No port forwarding for inbound connections is required.

Debian / Ubuntu Instructions

This requires wheezy or later. For squeeze you can use the easy_install method below.

Build Slave Installation

 sudo apt-get install buildbot-slave

Slave Configuration

Create the slave (the package creates the buildbot user automatically for you):

 sudo -u buildbot buildslave create-slave --usepty=0 /var/lib/buildbot/slaves/ola buildmaster.openlighting.org:9989 <slave user> <slave password>

Update the slave info, edit the files in /var/lib/buildbot/slaves/ola/info to be relevant to you.

Add config for the slave into /etc/default/buildslave (you'll need to increase the array id if you've got more than one slave on the same host), e.g.:

 SLAVE_ENABLED[1]=1                    # 1-enabled, 0-disabled
 SLAVE_NAME[1]="ola"         # short name printed on start/stop
 SLAVE_USER[1]="buildbot"              # user to run slave as
 SLAVE_BASEDIR[1]="/var/lib/buildbot/slaves/ola"                   # basedir to slave (absolute path)
 SLAVE_OPTIONS[1]=""                   # buildbot options
 SLAVE_PREFIXCMD[1]=""                 # prefix command, i.e. nice, linux32, dchroot

Start the slave

sudo /etc/init.d/buildslave start

Check the log if there are any issues, or confirm the slave is registered at http://buildbot.openlighting.org/buildslaves:

tail -f /var/lib/buildbot/slaves/ola/twistd.log

FreeBSD Instructions

This was tested on FreeBSD 10.

Build Slave Installation

As root:

 pkg install buildbot-slave

Slave Configuration

Setup a new user:

su -
adduser

Username: ola-build-slave Use password-based authentication? No

Setup the slave:

Switch to the slave user, I did:

su - #To root
su - ola-build-slave #To slave user
cd ~

Export the variables needed for configure on FreeBSD:

export CPPFLAGS="-I/usr/local/include/"
export LDFLAGS="-L/usr/local/lib/"

Setup the slave itself:

/usr/local/bin/buildslave create-slave ola-slave buildmaster.openlighting.org:9989 <slave user> <slave password>

Edit ola-slave/info/admin and ola-slave/info/host so your slave shows up correctly.

Then start the slave:

/usr/local/bin/buildslave start ola-slave

You can look at the logs by running

 tail -f ola-slave/twistd.log

At this point you can go to http://buildbot.openlighting.org/buildslaves and you should see your slave connected. It's probably worth asking someone to kick off a build at this point so we can check your slave is working.

Finally, if everything looks good, configure your slave to launch on startup by editing the crontab for the ola-build-slave user (crontab -e). Add the following line:

@reboot /usr/local/bin/buildslave start /home/ola-build-slave/ola-slave


NetBSD Instructions

This was tested on NetBSD 6.

Build Slave Installation

As root:

 pkgin install py27-buildbot-slave

Slave Configuration

Setup a new user:

su -
useradd -m ola-build-slave

Use password-based authentication? No

Setup the slave:

Switch to the slave user:

su - ola-build-slave
cd ~

Export the variables needed for git and configure on NetBSD (add these to the .shrc or equivalent for next time):

export CPPFLAGS="-I/usr/pkg/include/"
export LDFLAGS="-L/usr/pkg/lib/"
export GIT_SSL_NO_VERIFY=true

Setup the slave itself:

/usr/pkg/bin/buildslave create-slave ola-slave buildmaster.openlighting.org:9989 <slave user> <slave password>

Edit ola-slave/info/admin and ola-slave/info/host so your slave shows up correctly.

Then start the slave:

/usr/pkg/bin/buildslave start ola-slave

You can look at the logs by running

 tail -f ola-slave/twistd.log

At this point you can go to http://buildbot.openlighting.org/buildslaves and you should see your slave connected. It's probably worth asking someone to kick off a build at this point so we can check your slave is working.

Finally, if everything looks good, configure your slave to launch on startup by editing the crontab for the ola-build-slave user (crontab -e). Add the following line:

@reboot /usr/pkg/bin/buildslave start /home/ola-build-slave/ola-slave

OpenBSD Instructions

This was tested on OpenBSD 5.4.

Build Slave Installation

As root:

pkg_add py-buildslave

Slave Configuration

Setup a new user:

su -
useradd -m ola-build-slave

Setup the slave:

Switch to the slave user:

su - ola-build-slave
cd ~

Export the variables needed for autoconf and configure on OpenBSD (add these to the .profile or equivalent for next time), the eg++/egcc lines are needed if the installed gcc is too old to pass configure, install the eg++ one and then use that line:

export AUTOCONF_VERSION=2.69
export AUTOMAKE_VERSION=1.13
export CPPFLAGS="-I/usr/local/include/"
export LDFLAGS="-L/usr/local/lib/"
export CXX="eg++" CC="egcc"

Setup the slave itself:

/usr/local/bin/buildslave create-slave ola-slave buildmaster.openlighting.org:9989 <slave user> <slave password>

Edit ola-slave/info/admin and ola-slave/info/host so your slave shows up correctly.

Then start the slave:

/usr/local/bin/buildslave start ola-slave

You can look at the logs by running

 tail -f ola-slave/twistd.log

At this point you can go to http://buildbot.openlighting.org/buildslaves and you should see your slave connected. It's probably worth asking someone to kick off a build at this point so we can check your slave is working.

Finally, if everything looks good, configure your slave to launch on startup by editing the crontab for the ola-build-slave user (crontab -e). Add the following line:

@reboot /usr/local/bin/buildslave start /home/ola-build-slave/ola-slave

Other Platforms

Build Slave Installation

The easiest way to get started is by using easy_install. You need to have the Python headers available, so on Debian / Ubuntu run:

sudo apt-get install python-dev

Then install buildbot-slave:

easy_install buildbot-slave

Slave Configuration

Setup a new user:

sudo -s 
adduser ola-build-slave  # use a temp password for now
vi /etc/shadow  # delete the password entry

Setup the slave:

su ola-build-slave
cd ~
buildslave create-slave ola-slave buildmaster.openlighting.org:9989 <slave user> <slave password>

Edit ola-slave/info/admin and ola-slave/info/host so your slave shows up correctly.

Then start the slave:

buildslave start ola-slave

You can look at the logs by running

 tail -f ola-slave/twistd.log

At this point you can go to http://buildbot.openlighting.org/buildslaves and you should see your slave connected. It's probably worth asking someone to kick off a build at this point so we can check your slave is working.

Finally, if everything looks good, configure your slave to launch on startup by editing the crontab for the ola-build-slave user (crontab -e). Add the following line:

@reboot buildbot start /home/ola-build-slave/ola-slave

Enabling the C++ Lint Checker

The lint checker enforces C++ style. We only run this once per change but it's good to have multiple lint-enabled hosts in case one is down.

Download the cpplint checker and put it in /usr/local/bin

wget http://google-styleguide.googlecode.com/svn/trunk/cpplint/cpplint.py
sudo cp cpplint.py /usr/local/bin
sudo chmod a+x /usr/local/bin/cpplint.py

Finally let Simon or Peter know, or raise an issue, asking them to enable C++ lint for your slave.

Enabling the Javascript Lint Checker

The lint checker enforces Javascript style. We only run this once per change but it's good to have multiple lint-enabled hosts in case one is down.

Either:

Finally let Simon or Peter know, or raise an issue, asking them to enable Javascript lint for your slave.

Enabling the heap checker

First we need libunwind:

sudo apt-get install libunwind8-dev 

Then download the latest gperftools from https://code.google.com/p/gperftools/downloads/list .

wget ....
tar -zxf gperftools-2.1.tar.gz
cd gperftools-2.1
./configure
make
sudo make install
sudo ldconfig

Finally let Simon or Peter know, or raise an issue, asking them to enable the heap checker for your slave.