Frequent updates on TwitterBuildbot slaves as a Windows service and adding a testing framework (UnitTest++)
Current repository revision: 12 (Main)
Code highlight key
Grey: Code unchanged
Red: Code removed
Green: Code added
Blue: Code not shown→ Denotes the current line and the next are really one line
The mini-series of posts about setting up Buildbot is coming to an end with this post about starting a Windows Slave as a service, adding the UnitTest++ testing framework to the code and running the tests with Buildbot. Earlier posts had described how to set up a Buildbot master with CMake and how to set up Buildbot slaves for CMake.
Starting the Buildbot Slave as a Windows Service
Note: Before setting up the Slave to run as a service, make sure it works running from the command line. This ensures everything is set up right!
.
Setting up the Buildbot Slave as a Service started with following the appropriate section (Windows Buildbot service setup) in this piece of documentation. A couple of problems occurred while following this procedure, but these have been solved and the details are below.
Firstly, the script buildbot_service.py was checked for in the Python Scripts directory. It didn’t exist! This was because the script comes with the Buildbot Master installation, but not the Slave. On Windows the Master was unnecessary, and so wasn’t installed. To save installing it now, the buildbot_service.py script was copied across from the Linux Master installer (from within the expanded buildbot-0.8.1/contrib/windows directory). Remember that PyWin32 has to be installed before adding as a service.
Windows XP
The first Slave is to run on a Windows XP machine, and this was set up with a single user (Chris). As this is the default user it should have administrator privileges. However, step 3 in the documentation was followed, and the user Chris was added to the Log on as a service user.
The service was then installed, using the command:
> python C:\Python26\Scripts\buildbot_service.py → --user BARRACUDA\Chris --password SomePassword --startup auto install
This command reported success, so the next step was to ensure the user (Chris) had permissions on the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BuildBot. Regedit was fired up, the key was browsed to, and the BuildBot folder right clicked on. Permissions… was selected from the context menu, and user Chris added.
As the user Chris created the buildbot Slave configuration directory it was unnecessary to change permissions on the configuration folder. Now the buildslave was ready to start with the command:
> buildbot_service.py start "C:\code\Buildbot\slave"
Unfortunately, that resulted in the error:
Error starting service: The service did not start due to a logon → failure
It was time to look at what the script had actually installed in the Services dialog (available through Control Panel | Administrative Tools | Services). In the list that pops up, the Buildbot service can be seen. Double-clicking on it brings up the properties, and everything looked sensible. Clicking the Start button brought up the same message as running at the command line did.
After much checking of permissions and retyping passwords, it was decided that the Administrator account should be tried instead of the user Chris, so the details were entered in the Log On tab:
The service was then started again, but this time the service started and stopped immediately. To find out what had happened it was necessary to look in the event viewer (Control Panel | Administrative Tools | Event Viewer). Double-clicking on the latest Warning related to the Buildbot service showed the following details:
There is an ImportError as the buildbot_service.py script tries to import runner from buildbot.scripts. This relates back to the initial problem of the missing buildbot_service.py script – only the Buildbot Slave code is installed on the Windows box.
Looking through the scripts called when the buildslave command is executed, it seems the code that does the actual running is in the file C:\Python26\Scripts\buildslave. This script just has just three lines of actual code, so these were copied and pasted into the buildbot_services.py file in the _RunChild() function in place of the Master running code (the problematic import and a runner.run() command):
def _RunChild(): ... # Start the buildbot app from buildbot.scripts import runner runner.run() __requires__ = 'buildbot-slave==0.8.1' import pkg_resources pkg_resources.run_script('buildbot-slave==0.8.1','buildslave') print "Service child process terminating normally."
Note: Remember – this code change is only necessary if the Buildbot Master code hasn’t been installed. In fact, if you make this change and hope to start a Buildbot Master it’s unlikely to work!
The service property dialog was then reopened, and the parameters normally passed to start the Slave were entered in the start parameters box:
The service was then started again, and this time all was successful! Checking on the MereIdea Buildbot web pages confirmed that the Slave was now connected to the Master. This could also be started with the command:
> buildbot_service.py start C:\code\Buildbot\slave
All that remained now was to check whether the Builders would still work with the service, and whether the service restarted correctly when the computer was rebooted.
The build did not work due to something mentioned in an earlier post in reference to the setvcvars step of the Windows build; a different command shell is used for each step, so the work done by the setvcvars step is lost by the next step. The call to NMake fails because the Visual Studio paths are no longer set.
This was fixed by removing setvcvars as a standalone first step, and editing setvcvars.bat to set up the environment and call NMake:
call "c:\Program Files\Microsoft Visual Studio → 10.0\VC\vcvarsall.bat" x86 nmake
This file was renamed run_nmake.bat, and then the Windows Compile step in the master.cfg file (wcomp) was changed to use it and the original setvcvars step removed:
... svnbaseurl='http://svn.mereidea.com/main/' svnup = source.SVN(mode='update', baseURL=svnbaseurl, defaultBranch='trunk') svnco = source.SVN(mode='clobber', baseURL=svnbaseurl, defaultBranch='trunk') setvcvars = shell.ShellCommand(command="setvcvars.bat") confg = shell.Configure(command="cmake ../src") wconfg = shell.Configure(command='cmake -G "NMake Makefiles" ../src') comp = shell.Compile(command='make') wcomp = shell.Compile(command='run_nmake') rtest = shell.Test(command='run_tests') ... fwchg = factory.BuildFactory() fwchg.addStep(setvcvars) fwchg.addStep(svnup) fwchg.addStep(wconfg) fwchg.addStep(wcomp) fwchg.addStep(rtest) ... fwngt = factory.BuildFactory() fwngt.addStep(setvcvars) fwngt.addStep(svnco) fwngt.addStep(wconfg) fwngt.addStep(wcomp) fwngt.addStep(rtest) ...
A new build was forced, and was now successful! 
The computer was rebooted, and the service restarted successfully. The build still worked.
Windows 7/Vista
Most of the Windows 7 setup for a Buildbot Slave as a service was the same as for Windows XP, except that the main steps need to be done in a command prompt that has been ‘Run As Administrator’.
It was also necessary to open the services dialog and reenter the user details in the Log On tab, then press Apply to set the permissions for the user to Lon on as a service. This gets automatically set when the user details are set. There is probably a way to set these permissions separately, but this works!
For Windows 7 the user was in the Administrators group, but when setting up the Vista service a non-administrative user was used. This still worked the same as Windows 7 (‘Run As Administrator’ shell needed to be used) except that the registry permissions needed granting on HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BuildBot, and the service had to be started (whether from the dialog or the command prompt) by the administrator user.
A final note on Window 7. There were problems with using Visual Studio 2010 under Windows 7. This problem was due to CMake and not Buildbot. Therefore, the test for this service was done with Visual Studio 2008. There may be a blog post on using CMake with Windows 7 and Visual Studio 2010 later.
Adding a testing framework
UnitTest++ was chosen as the testing framework. This section shows the addition of this framework to the MereIdea code, building the tests with CMake
and running them through Buildbot.
The first step is to add the UnitTest++ code to the MereIdea code. It was decided to simply import the code and build it as part of the mi_test module, which will be built with the rest of the code, rather than expect everyone to get the code and build it. The code is small and easily compiled to a library using a couple of CMake files. Including it also eliminates the main problem with using thirdparty code – if the UnitTest++code changes or becomes unavailable, it won’t require (possibly extensive) changes to the MereIdea code.
The first step, therefore, was to add a new module inside the mi_main source directory. The directory mi_test was created and another directory created inside it – unittest++. This will keep the UnitTest++ code separate from any other code added to the mi_test module. The UnitTest++ code was downloaded to a different directory using svn:
svn co https://unittest-cpp.svn.sourceforge.net/svnroot/unittest-cpp → unittest-cpp
and the code files (including the Posix, Win32 and test directories) were copied to the new mi_test/unittest++ module. The COPYING and README files were also copied to the top-level mi_test directory.
The CMake files were then added to the mi_test module. Firstly, the top-level CMakeLists.txt file was created. This simply declares the project name and adds the unittest++ subdirectory:
# Give this module its own project name so we can # refer to its build and source structures project(MI_TEST) # And build the unittest++ library add_subdirectory(unittest++)
Next the CMakeLists.txt file in the unittest++ subdirectory was written. This adds the source files to build the library, and then descends into the tests directory:
# Add the source files here
set(MI_TEST_UNITTEST_PP_SOURCES
AssertException.h
Test.h
Checks.h
....
CurrentTest.h
CheckMacros.h
Config.h
ExecuteTest.h
TestMacros.h
TestSuite.h
TimeHelpers.h
UnitTest++.h
AssertException.cpp
Test.cpp
Checks.cpp
...
CurrentTest.cpp
)
# Add the platform specific code
if (WIN32)
set (MI_TEST_UNITTEST_PP_SOURCES ${MI_TEST_UNITTEST_PP_SOURCES}
Win32/TimeHelpers.h
Win32/TimeHelpers.cpp
)
else ()
set(MI_TEST_UNITTEST_PP_SOURCES ${MI_TEST_UNITTEST_PP_SOURCES}
Posix/SignalTranslator.h
Posix/TimeHelpers.h
Posix/SignalTranslator.cpp
Posix/TimeHelpers.cpp
)
endif (WIN32)
# And add the library
add_library(miunittest++ ${MI_TEST_UNITTEST_PP_SOURCES})
# Put the library in a lib subdirectory
# Do for static, shared and module libraries
set_target_properties(miunittest++ PROPERTIES
ARCHIVE_OUTPUT_DIRECTORY ${MI_TEST_BINARY_DIR} →
/lib
LIBRARY_OUTPUT_DIRECTORY ${MI_TEST_BINARY_DIR} →
/lib
)
# Build the tests
add_subdirectory(tests)
All the source files and header files are included in the CMakeLists.txt file. This is because, even though some of the header files aren’t needed for building the library, the header files won’t appear in any generated Visual Studio projects if they are not included in the CMakeLists.txt file. The last new CMakeLists.txt file to be created is in the mi_test/unittest++/tests directory:
# Add the source files here
set(MI_TEST_UNITTEST_PP_TEST_SOURCES
RecordingReporter.h
ScopedCurrentTest.h
Main.cpp
TestAssertHandler.cpp
TestChecks.cpp
...
TestCurrentTest.cpp
TestTestSuite.cpp
)
# Create the test executable
add_executable(unittest++_tests ${MI_TEST_UNITTEST_PP_TEST_SOURCES})
# Set the libraries required
target_link_libraries(unittest++_tests miunittest++)
# Add as a test (so we can use 'make test')
add_test(unittest++_tester unittest++_tests)
Then the top-level source CMakeLists.txt file is edited to turn testing on (so add_test actually adds a test) and to include the mi_test module.
...
include_directories(${MI_MAIN_SOURCE_DIR})
# Make sure tests are built
enable_testing()
# Now set the subdirectories to work through. Ensure these are done
# in the correct order
add_subdirectory(mi_test)
add_subdirectory(mi_hello)
Now changing to the build directory in a Linux shell and typing:
> cmake ../src > make > make test
will generate the appropriate make files, build the code, and run the tests which should pass 100%. Under Windows it would be:
> cmake -G "NMake Makefiles" ../src > nmake > nmake test
On Windows the CMake GUI could also be run to generate the Visual Studio projects which can be built and run within the IDE.
Changing the mi_hello test to use UnitTest++
Now to change the mi_hello tests to use the UnitTest++ code. Before this is done, however, a small change to the mi_hello::Hello class is needed. Instead of sending the output to the console (std::cout) it needs capturing so that it can be tested. The header file becomes:
... #include <string> #include <ostream> ... //! Display a message /*! Display a string passed in (or "Hello, World!") Send a string parameter to a stream \param disp_str The string to display (defaults to → "Hello, World!") \param os The stream to send the string to */ void display(const std::string &disp_str = "Hello, World!", → std::ostream &os) const; ...
Note that the default for the display string (“Hello, World”) was removed as it will have to be entered anyway when changing the stream. As this class will only be used for testing, defaults don’t really make much sense. The cxx file is changed to match:
... #include <iostream> ... void mi_hello::Hello::display(const std::string &disp_str → /*= "Hello, World!"*/, std::ostream &os) const { std::cout << disp_str << std::endl; os << disp_str; }
Finally the test can be changed to use the new mi_hello::Hello class and the UnitTest++ code (comments omitted):
... #include <mi_hello/hello.h> #include <mi_test/unittest++/UnitTest++> #include <sstream> struct MI_HelloFixture { MI_HelloFixture() { } ~MI_HelloFixture() { } mi_hello::Hello hello_; std::string test_str_; std::stringstream str_; }; TEST_FIXTURE(MI_HelloTestFixture, TestHello) { test_str_ = "Hello, World!"; hello_.display(test_str_, str_); CHECK_EQUAL(test_str_, str_.str()); } TEST_FIXTURE(MI_HelloTestFixture, TestGoodbye) { test_str_ = "Goodbye, World!"; hello_.display(test_str_, str_); CHECK_EQUAL(test_str_, str_.str()); } int main() { mi_hello::Hello h; h.display(); h.display("Goodbye, World!"); return 0; }
Now that the main() function has been removed from this file (as there could be several of these test files), a new file needs to be created to run the tests. This file is named test_hello_main.cxx and is created alongside test_hello.cxx. This new file simply contains:
#include <mi_test/unittest++/UnitTest++.h>
#include <mi_test/unittest++/TestReporterStdout.h>
int main()
{
return UnitTest::RunAllTests();
}
Each tests directory in the libraries included in the MereIdea Main Projects will contain one or more test files (like test_hello.cxx) and one main file (like test_hello_main.cxx).
Finally, the CMakeLists.txt file in mi_hello/tests was edited for the new code:
... set(MI_HELLO_TEST_SOURCES test_hello_main.cxx test_hello.cxx ) add_executable(test_hello test_hello.cxx) add_executable(test_hello ${MI_HELLO_TEST_SOURCES}) target_link_libraries(test_hello miunittest++ mihello) add_test(test_hello_tester test_hello)
Now the code can be built and tested from within the build directory with the familiar:
> cmake ../src > make > make test
All tests should pass with some output similar to:
Running tests...
Test project /home/user/code/mereidea/mi_main/build
Start 1: unittest++_tester
1/2 Test #1: unittest++_tester ................ Passed 0.10 sec
Start 2: test_hello_tester
2/2 Test #2: test_hello_tester ................ Passed 0.01 sec
100% tests passed, 0 tests failed out of 2
Total Test time (real) = 0.12 sec
This is the output from CTest (part of CMake), which is called when using make with the test target. It shows this summary by default, which will be enough for displaying in
Buildbot as it will show which library test has the failure. It would then be up to the developer to go to that library’s test directory and type ctest -V or, with
CMake version 2.8 and later, ctest –output-on-failure. This will give more information about the problem test.
Updating Buildbot to use new tests
The last step in the new testing framework is to update the factories in the
Buildbot Master to call make test:
... rtest = shell.Test(command='mi_hello/tests/test_hello') rtest = shell.Test(command='ctest') wrtest = shell.Test(command='mi_hello\\tests\\test_hello') ... fwchg.addStep(wrtest) fwchg.addStep(rtest) ... fwngt.addStep(wrtest) fwngt.addStep(rtest) ...
Note that the factory uses ctest instead of make test. The two are equivalent in the
CMake world. Calling ctest means that the Windows and Linux commands can be identical. Otherwise make test would need to be called on Linux, and nmake test on Windows. Calling nmake test would also mean that another script would have to be created to set up the Visual Studio environment before calling NMake.
Fixing Visual Studio problems
Once the new testing framework was committed, the Master configuration updated and the Master restarted, the code was built and tested using the Linux and Windows builders. Two problems arose in the Visual Studio build: There were some warnings about strcpy being insecure and an unhandled exception occurred in the UnitTest++ test suite.
The unhandled exception was tackled first. This problem stemmed from the fact that CMake, by default, uses C++ exceptions without structured exceptions (switch /EHsc). This means that C++ exceptions will be caught (using throw, try and catch), but system exceptions will not.
To ensure all exceptions are handled as a test failure by UnitTest++ the exception switch to the compiler has to be changed to /EHa. This is done for all compiled programs (any part could use the UnitTest++ code) by setting the CMake variable CMAKE_CXX_FLAGS to include the corrected switches:
# The project is MI_MAIN, so variables like ${MI_MAIN_SOURCE_DIR} and # ${MI_MAIN_BINARY_DIR} can be used project(MI_MAIN) # Ensure the make flags are as we need (/EHa on Windows) if (MSVC) set(CMAKE_CXX_FLAGS "/DWIN32 /D_WINDOWS /W3 /Zm1000 /EHa /GR" CACHE STRING "Flags used by the compiler during all build types." FORCE) else() set(CMAKE_CXX_FLAGS "-Wall -pedantic -ansi" CACHE STRING "Flags used by the compiler during all build types." FORCE) endif(MSVC)
This is done in the top-level (src) CMakeLists.txt file. Note that, while compiler flags are being set, the flags for use with gcc are also set. If somebody uses another compiler, any necessary flags should be set here.
The final change is to fix the warnings produced by the Windows build. These warnings occur because strcpy() can be unsafe, and strcpy_s() should be used instead. However, strcpy_s() is not cross-platform, so it won’t be used here! Instead, it will just be assumed that the authors of UnitTest++ use strcpy() in a safe way and this warning can be ignored. It is possible to define _CRT_SECURE_NO_WARNINGS in the preprocessor switches, but this assumes that everyone who uses strcpy() anywhere in the code will use it safely. Instead, it’s better to make programmers leave an “I have thought about this and think it’s safe” message when using an unsafe function. This is done by expecting developers to disable the warning locally.
There are only three files in which the warning occurs – AssertException.cpp in the top-level of the UnitTest++ code, and RecordingReporter.h and TestDeferredTestReporter.cpp in the tests sub-directory. In those files any call to strcpy() is surrounded by #pragma warning() macros; one to turn the warning off, and another to turn it back on again. For example, in AssertException.cpp:
AssertException::AssertException(char const* description, → char const* filename, int lineNumber) : m_lineNumber(lineNumber) { using namespace std; #ifdef _MSC_VER # pragma warning( disable : 4996 ) #endif strcpy(m_description, description); strcpy(m_filename, filename); #ifdef _MSC_VER # pragma warning( default : 4996 ) #endif }
With the exception solved and the warnings removed from the build, the code can now be committed and the results viewed in the Buildbot Waterfall.
Final note
To start adding new Buildbot Slaves to the installation, it was necessary to rename the Builders more sensibly as a builder is needed for each machine to add. In doing this, the builds were moved to new directories and were rebuilt from scratch. The nightly build on Windows failed in the configuration step because the Visual Studio environment wasn’t set up when CMake was called and the compiler couldn’t be found. This hadn’t shown up before because CMake was run in a directory in which it had been previously run successfully (when the buildslave was run in a shell with the correct environment – before changing to a service). To fix this problem and to ensure anything like it is caught earlier in future, the master.cfg was changed to run a batch file for running CMake on Windows (similar to the run_nmake script) and an extra step was added to remove the CMakeCache.txt file on all nightly builds (when the code should be clobbered). The run_cmake.bat script is simply:
call "C:\Program File\Microsoft Visual Studio 10.0\VC\vcvarsall.bat" → x86 cmake -G "NMake Makefiles" ../src
and the changes to the master.cfg file were:
... confg = shell.Configure(command="cmake ../src") wconfg = shell.Configure(command=['cmake','-GNMake Makefiles',→ '../src']) wconfg = shell.Configure(command='run_cmake') rtest = shell.Test(command='ctest') rcmake = shell.ShellCommand(command='rm -f CMakeCache.txt') wrcmake = shell.ShellCommand(command='del /F /Q CMakeCache.txt') ... flngt = factory.BuildFactory() flngt.addStep(svnco) flngt.addStep(rcmake) flngt.addStep(confg) ... fwngt = factory.BuildFactory() fwngt.addStep(svnco) fwngt.addStep(wrcmake) fwngt.addStep(wconfg) ...
Summary
This post covered two major topics; setting up a Buildbot Slave as a service on Windows, and putting the Mere Idea Main Projects’ tests into the UnitTest++ framework.
The Slaves were installed as services following a page on the Buildbot website. The instructions on that page were mostly correct, but a couple of problems arose because the script to install the service (buildbot_service.py) isn’t included in the Slave installer, so had to be copied and modified from the one in the Master installer. Reading the Buildbot mailing list, a change to rectify this may soon be made.
The use of UnitTest++ in the code was straightforward – copying the code into the mi_main source code repository and changing the local tests to use it. The tests could then be run by using make test or ctest once the code was built.
The Buildbot Builders were then changed to carry out the testing correctly.
Repository Changes since last post
Revision 3 and 4 added Doxygen support for the Main Projects code. The result can be seen here.
Revision 5 added a README file on how to download and build the code (also available in the Doxygen.
Revision 7 changed the Doxygen revision filter to correctly display the current repository revision and the last revision at which a particular file was changed.
Revision 8 added the UnitTest++ code and changed the ‘hello’ test.
Revision 9 fixed a problem with using targets containing ‘++’ in their name by changing them to ‘pp’. The problem occurred with NMake.
Revision 10 and 11 fixed the problems mentioned in this post regarding building the UnitTest++ code under Windows (using /EHa and #pragma).
Revision 12 added support for MinGW as the code won’t compile with the -pedantic flag set, so turn it off if compiling with MinGW.
Leave a Reply
You must be logged in to post a comment.