Frequent updates on TwitterSetting up a Buildbot Master With CMake
Code highlight key
Grey: Code unchanged
Red: Code removed
Green: Code added
Blue: Code not shown→ Denotes the current line and the next are really one line
It’s been a while since a new post graced this blog. Unfortunately (for the blog, at least), other work has got in the way and it is now it is necessary to start a new, year long project producing some software and an accompanying website. To write this new software CMake is going to be used as the build system (actually, CMake is only a Makefile/Project file generator, so the build system will be whatever is native to the operating system – make and Visual Studio – and nmake - in this case).
CMake wasn’t chosen because it’s any better (or worse) than SCons, but because of familiarity with it and the need to create an actual project rather than experimenting with example code. This, sadly, means setting the SCons tests on one side for the moment.
It was hoped the new SCons 2.0 series could be tested using the example code and different configurations on different platforms, but this work is now on hold. It is recommended, however, that anyone considering SCons to try out the new 2.0 series.
When it is thought that some element of the new project is worth blogging about, then this blog will be updated, hopefully frequently. Updates are also available via the Twitter account.
One thing that was on the original path was the automated compile/test of the example code using Buildbot. Originally, this blog was to detail the integration of SCons with Buildbot, but now will describe the integration of CMake instead. This first post in the mini-series describes the setting up and configuration of the Buildbot Master.
Buildbot is made up of two main components. The Master which decides when things need to be built, and collates and displays the results via web pages; and the Slave which is an individual machine that performs the building of the code and sends the results of the build to the Master. There is one Master and one or more Slaves. The more Slaves on differently configured machines the better!
Installation
The Master is going to be set up on a machine running openSUSE x86_64 version 11.2. Buildbot is written in Python, so you need to install Python (usually included by the default SUSE installation) first. The Python version installed is 2.6.2. Twisted also needs to be installed for Buildbot‘s networking functionality. This was installed through YaST (SUSE‘s system admin client) which installed Twisted version 8.2.0. Python and Twisted are required in systems running either the Master or the Slave. There are some further pre-requisites for the machine running the Master.
sqlite3 and simplejson are included with the Python version installed, so no further action was needed for this. Jinja2, however, needed to be downloaded and installed. Actually, it can be downloaded and installed using the command:
> easy_install Jinja2
but instead the source was downloaded, decompressed and installed from the source tar. Once downloaded and decompressed, the installation can be done as root using the command:
> python setup.py install
in the extracted directory. This, however, caused a problem as the Python setuptools module was not installed. A derivative of setuptools (distribute) was installed using the commands:
> curl -O http://python-distribute.org/distribute_setup.py > python distribute_setup.py
Then Jinja2 could be installed correctly using the command as above.
Now everything is ready to install Buildbot. The version of Buildbot used is 0.8.1. Both the Master and Slave tar.gz files were downloaded and decompressed. These were then installed by switching to each of the extracted directories and typing:
> python setup.py build > python setup.py install
The installations can then be tested. Firstly to check the version numbers:
> buildbot --version Buildbot version: 0.8.1 Twisted version: 8.2.0 > buildslave --version Buildslave version: 0.8.1 Twisted version: 8.2.0
and then by running the tests:
> PYTHONPATH=. trial buildbot.test
Unfortunately, running the above test suite showed two errors. These errors were both of type ImportError relating to the mock module missing from Python. The mock module was installed using:
> easy_install mock
The tests could then be run without error for both the Master and Slave components:
> PYTHONPATH=. trial buildbot.test > PYTHONPATH=. trial buildslave.test
Creating the Master
Finally the Master is ready to be set up.
Firstly a new user was created to act as the Master administrator. This was done through YaST, but could easily be done with useradd. This new user was then logged in. A new directory was created to house the Buildbot Master in the new user’s home directory:
> mkdir -p ~/Buildbot/master
This directory was then initialised as a new Master directory:
> buildbot create-master -r ~/Buildbot/master
Now it’s ready for the configuration.
Configuring the Master
The create-master option to the buildbot command creates a set of files in the specified directory. One of these files (master.cfg.sample) is the configuration file which needs to be edited. First, a copy of the sample file is made:
> cp master.cfg.sample master.cfg
This was then edited as follows (comments omitted):
c = BuildmasterConfig = {} c['db_url'] = "sqlite:///state.sqlite" from buildbot.buildslave import BuildSlave c['slaves'] = [BuildSlave("bot1name", "bot1passwd")] c['slaves'] = [ BuildSlave("oSUSE_11.2_x86_64_gcc","password1", → max_builds=2), BuildSlave("WinXP_VS2010","password2",max_builds=2) ] c['slavePortnum'] = 9989
This sets up the slaves on which the code will be built. More will be added to this list, but for now it will be tested with just these two. Obviously, the passwords need to be changed to something more secure! 
These passwords will be used later to set up the slaves. The max_builds=2 also tells the Master not to send more than two concurrent builds to the Slaves. The slavePortNum is the port on which the Master will listen for Slaves, and so must match the port used when setting up a Slave later.
Now the method for detecting changes to the source code can be specified. As Subversion will be used for this project, SVNPoller is used to check for updates to the code:
from buildbot.changes.pb import PBChangeSource from buildbot.changes.svnpoller import SVNPoller c['change_source'] = PBChangeSource() source_svn_url='http://svn.mereidea.com/main/trunk' svn_poller = SVNPoller( svnurl=source_svn_url, pollinterval=3600, # seconds histmax=10, svnbin='/usr/local/bin/svn' ) c['change_source'] = [ svn_poller ]
So the change_source is changed from PBChangeSource to the SVNPoller. The main repository is one that can be read by anyone, but a username and password can be specified to the SVNPoller if necessary with the svnuser and svnpasswd options.
Next the schedulers can be defined. These tell the Master when to call the Slaves to build the code. Two types of schedulers are required – a change scheduler (of type Scheduler) called whenever the repository changes, and a nightly scheduler (of type timed.Nightly) to rebuild the code every morning at 03:30. The build for change scheduler will simply update the code and run a build, the nightly scheduler will check out a clean copy of the code and build from scratch. The configuration file is changed as follows:
from buildbot.scheduler import Scheduler from buildbot.schedulers import timed c['schedulers'] = [] c['schedulers'].append(Scheduler(name="all", branch=None, treeStableTimer=2*60, builderNames=["buildbot-full"])) c['schedulers'].append(Scheduler(name="change-build",branch=None, treeStableTimer=600, builderNames=[ "Linux-change-build", "Win-change-build" ] ) ) c['schedulers'].append(timed.Nightly(name="nightly-build", hour=3, minute=30, builderNames=[ "Linux-nightly-build", "Win-nightly-build" ] ) )
This sets up the two required schedulers. The first (change-build) is triggered when the code is changed and detected with the SVNPoller set up earlier. Once triggered, it waits 10 minutes (treeStableTimer=600) to try to ensure all commits are complete before running the two builders – Linux-change-build and Win-change-build. The second (nightly-build) runs its builders – Linux-nightly-build and Win-nightly-build – at 3:30am (hour=3, minute=30).
This scheduler set up shows that four builders are going to be required – Linux-change-build, Linux-nightly-build, Win-change-build, and Win-nightly-build. The configuration is altered to include these builders as follows:
cvsroot = ":pserver:anonymous@cvs.sourceforge.net:/cvsroot/buildbot" cvsmodule = "buildbot" from buildbot.process import factory from buildbot.steps.source import CVS from buildbot.steps.shell import Compile from buildbot.steps import source, shell from buildbot.steps.python_twisted import Trial svnbaseurl='http://svn.mereidea.com/main/' svnup = source.SVN(mode='update', baseURL=svnbaseurl, defaultBranch='trunk') svnco = source.SVN(mode='clobber', baseURL=svnbaseurl, defaultBranch='trunk') setvcvars = shell.ShellCommand(command="setvcvars.bat") confg = shell.Configure(command="cmake ../src") wconfg = shell.Configure(command='cmake -G "NMake Makefiles" ../src') comp = shell.Compile(command='make') wcomp = shell.Compile(command='nmake') rtest = shell.Test(command='run_tests') f1 = factory.BuildFactory() f1.addStep(CVS(cvsroot=cvsroot, cvsmodule=cvsmodule, login="" → mode="copy")) f1.addStep(Compile(command=["python", "./setup.py", "build"])) f1.addStep(Trial(testChanges=True, testpath=".")) flchg = factory.BuildFactory() flchg.addStep(svnup) flchg.addStep(confg) flchg.addStep(comp) flchg.addStep(rtest) fwchg = factory.BuildFactory() fwchg.addStep(setvcvars) fwchg.addStep(svnup) fwchg.addStep(wconfg) fwchg.addStep(wcomp) fwchg.addStep(rtest) flngt = factory.BuildFactory() flngt.addStep(svnco) flngt.addStep(confg) flngt.addStep(comp) flngt.addStep(rtest) fwngt = factory.BuildFactory() fwngt.addStep(setvcvars) fwngt.addStep(svnco) fwngt.addStep(wconfg) fwngt.addStep(wcomp) fwngt.addStep(rtest) b1 = {'name': "buildbot-full", 'slavename': "bot1name", 'builddir': "full", 'factory': f1 } blchg = {'name'="Linux-change-build", 'slavename': "oSUSE_11.2_x86_64_gcc", 'builddir': "linus_change_build", 'factory': flchg } bwchg = {'name'="Win-change-build", 'slavename': "WinXP_VS2010", 'builddir': "win_change_build", 'factory': fwchg } blngt = {'name'="Linux-nightly-build", 'slavename': "oSUSE_11.2_x86_64_gcc", 'builddir': "linux_nightly_build", 'factory': flngt } bwngt = {'name'="Win-nightly-build", 'slavename': "WinXP_VS2010", 'builddir': "win_nightly_build", 'factory': fwngt } c['builders'] = [b1] c['builders'] = [blchg, bwchg, blngt, bwngt]
The builders are set with the factories to execute a set of commands appropriate to the build and architecture. The commands in this post are really just place holders, as the correct commands to run this correctly on any machine can only be decided once a few tests runs have been tried. The commands may be correct, but it is unlikely! For example, the setvcvars step needs to be set up and checked for persistence for later commands (are all commands executed in the same shell environment?). Further slaves that match a builder’s commands can be added to the builder by changing slavename to slavenames and providing a list:
'slavenames': ['WinXP_VS2010', 'Win7_VS2008'],
The correct build commands will be worked out in the next blog post when Slaves will be set up, too.
The final bits of the master configuration are the status targets and the project identity. The status targets tell Buildbot where to display the status results of the builds. These can be as web pages, by email, by IRC, etc. For now the MereIdea Buildbot will just display as web pages, though later the emailer will probably be set. The defaults for this section of the configuration are, therefore, fine. The http_port of the WebStatus may be changed, but this must correspond to the port set in the project identity section.
The project identity section provides a description of the project and how the web display can be accessed from the outside world. Simply change to whatever you want:
c['projectName'] = "Buildbot" c['projectName'] = "MereIdea Projects" c['projectURL'] = "http://buildbot.sourceforge.net" c['projectURL'] = "http://www.mereidea.com" c['buildbotURL'] = "http://localhost:8010" c['buildbotURL'] = "http://builds.mereidea.com"
These are pretty straightforward. If using the port number (8010) in the buildbotURL, this must match the http_port setting in the WebStatus of the status targets section of the configuration. In the above example, a redirect is set in the MereIdea Apache web server to convert http://builds.mereidea.com to http://www.mereidea.com:8010.
That’s the configuration of the Master complete. Now it can be checked for validity with:
> buildbot checkconfig master.cfg
This did produce a deprecation warning:
/usr/lib64/python2.6/site-packages/twisted/mail/smtp.py:10: → DeprecationWarning: the MimeWriter module is deprecated; → use the email package instead import MimeWriter, tempfile, rfc822 Config file is good!
This warning comes from within the Twisted code, so it’s just going to be ignored (possibly updating Twisted would make it go away). Apart from that warning, Buildbot says the configuration is good! Finally, the Master can be started with the command (executed in the directory containing the master’s configuration):
buildbot start
Then it can be checked by using a web browser to look at the buildbotURL. For the configuration above this means browsing to http://builds.mereidea.com. The Buildbot welcome page should be displayed. Looking on the Builders page it can be seen that the four builders specified in the configuration file are set up, and that they are currently offline:
So it’s all working so far! However, when the SVNPoller tried to poll the svn server an error occurred:
SVNPoller failed [Failure instance: Traceback (failure with →
no frames): <class 'twisted.internet.utils. →
_UnexpectedErrorOutput'>: got stderr: "svn: →
Repository moved temporarily to →
'http://www.mereidea.com/svn/main/trunk'; →
please relocate\n"
]
It seems the redirection in the Apache server from http://svn.mereidea.com to http://www.mereidea.com/svn has caused problems with the SVNPoller. The problem was easily solved by changing the configuration file:
from buildbot.changes.svnpoller import SVNPoller source_svn_url='http://svn.mereidea.com/main/trunk' source_svn_url='http://www.mereidea.com/svn/main/trunk' ... svnbaseurl='http://svn.mereidea.com/main/' svnbaseurl='http://www.mereidea.com/svn/main/'
The svnbaseurl was also changed to reduce the chance of any further problems.
Summary
This post described setting up the Master side of the Buildbot automatic build/test cycle software. It involved installing the necessary Python components, including Twisted and Jinja2. Then the installation of Buildbot itself. A Buildbot user was created to administer and run the Master daemon. A directory (~/Buildbot/master) was created to house the Master server files and it was initialised using the builbot create-master command. This also created a master.cfg.sample file which was copied to master.cfg and edited as detailed in the post to create four builds – two each for two Slaves. These were Linux and Windows builds when the code base (SVN) changed, and Linux and Windows nightly builds. The Master was then started and checked by accessing the WebStatus through http://builds.mereidea.com.
The configuration of the Master is quite straightforward, though it seemed daunting when first looking at the documentation. This is because the documentation is written to cover all eventualities and give a thorough outline of all types of builders, schedulers, build factories, etc.. The documentation is good, but the easiest way to configure Buildbot (and the way used to devise the configuration for the MereIdea set up) is to work through the master.cfg file using the comments as a guide and the Buildbot documentation as a reference.
With the Master now running, the next step is to put some code in the SVN repository and set up the Slaves. A continuous (short Periodic build) will the be set up so that the commands to run on the Slave can be fine tuned until it builds correctly. First the Linux Slave will be set up, then the Windows Slave. This is the subject of the next post, coming soon!
Leave a Reply
You must be logged in to post a comment.