Mere Idea Development Blog

Current repository revision: 60 (Testing)

Code highlight key

Grey: Code unchanged
Red: Code removed
Green: Code added
Blue: Code not shown

→ Denotes the current line and the next are really one line

The code can be checked out with:

svn checkout http://www.mereidea.com/svn/testing/trunk/mi_java_scons

---

Now that the initial Java / SCons code is in place, some tests can be added. The framework to be used for testing our Java code is JUnit.

This is not really a tutorial on JUnit, but will cover how we installed JUnit on both Linux and Windows, how we tested the installations with a simple test, how we added JUnit based tests and suites into our existing code, how the tests were built with SCons, and how they were run once built.
 

Installing JUnit

We want to be able to use JUnit on both Linux and Windows environments, and so it was installed on both.

Getting JUnit

At the time of writing, JUnit can be downloaded from github. We just downloaded the Basic jar. The version at writing was junit-4.10.jar.

Linux setup

1. Download the JUnit jar file.
2. Create a directory to put it in – this example assumes /usr/share/junit/.
3. Copy the jar file into the newly created directory.
4. Optionally rename the jar from, for example, junit-4.10.jar to junit.jar
5. Edit .bashrc or whatever shell initialization file works for you, and add

   export CLASSPATH=${CLASSPATH}:/usr/share/junit/junit.jar
   

   or whatever line your shell requires to add the directory and file name of the JUnit jar to your class path.

6. Save the .bashrc file, re-source it (source .bashrc), or open a new shell, and you’re ready to go!

Windows setup

1. Download the JUnit jar file.
2. Create a directory to put it in – this example assumes C:\junit\.
3. Copy the jar file into the newly created directory.
4. Optionally rename the jar from, for example, junit-4.10.jar to junit.jar
5. On Vista/Windows7 click the Windows (Start) button, right click on Computer and select Properties. On the window that pops up, select Advanced system settings and, finally, in the dialog that appears click Environment Variables….
6. On Windows XP click the Start button, right click on My Computer and select Properties. In the dialog that pops up select the Advanced tab then click the Environment Variables button.
7. If the CLASSPATH variable already exists in your System Variables, then select it and click Edit. If it doesn’t exist, click New.
8. Either add or enter the path to your junit.jar file (C:\junit\junit.jar)
9. Because, on Windows, if the CLASSPATH variable is set, the java command seems to ignore the current directory when searching for classes, ensure . is also in the CLASSPATH variable. For example, our CLASSPATH is:

   .;C:\junit\junit.jar
   

10. Click OK, close the dialogs, open a new command prompt, and you’re ready to go!

 

Using JUnit 4

The version of JUnit we have installed is version 4.10. Before JUnit 4, tests had to extend the TestCase class, and the methods had to have names beginning with test. We use the JUnit 4 annotation methods instead of the inherited TestCase as it seems much simpler and much neater to us. Below the same simple test is written to show both methods of creating tests. The class under test for both is:

public class StringUtils
{
  public String concatenateWithSpace(String s1, String s2)
  {
     return s1+" "+s2;
  }
}

Before JUnit 4

As mentioned above, before JUnit 4, the test class for the StringUtils needed to be inherited from TestCase. This can be written:

import junit.framework.*;

public class TestStringUtils extends TestCase
{
  private StringUtils str_utils;

  protected void setUp()
  {
    str_utils = new StringUtils();
  }

  public void testConcatenateWithSpace()
  {
    String s1 = "Test";
    String s2 = "me!";
    String expected = "Test me!";
    String actual = str_utils.concatenateWithSpace(s1,s2);
    assertEquals(actual, expected);
  }
}

Compile this with:

javac StringUtils.java TestStringUtils.java

then you can run the test with:

java junit.textui.TestRunner TestStringUtils

With JUnit 4

JUnit 4 enables you to define your tests using annotations. Below is the same test case as before, but rewritten to use JUnit 4. The annotation @Before identifies the setUp method to be called before each test, and the annotation @Test identifies actual tests:

import org.junit.Test;
import org.junit.Before;
import static org.junit.Assert.assertEquals;

public class TestStringUtils
{
  private StringUtils str_utils;

  @Before
  public void setUpStringUtils()
  {
    str_utils = new StringUtils();
  }

  @Test
  public void concatenateWithSpace()
  {
    String s1 = "Test";
    String s2 = "me!";
    String expected = "Test me!";
    String actual = str_utils.concatenateWithSpace(s1,s2);
    assertEquals(actual, expected);
  }
}

Once compiled (as above), this code can be run with:

java org.junit.runner.JUnitCore TestStringUtils

Note: With the JUnit 4 jar you can use either method for writing your tests.

 

Adding tests to the example code

Now we have written a couple of small, stand-alone tests to check our JUnit installation we can start adding tests to the mi_java_scons code.

Testing the Bar class

The first test we are going to write is for the Bar class in the mi_foo.bar package. In the directory src/mi_foo/bar there is a directory tests and in that directory is a file BarTests.java. This file will contain the tests for the Bar class. Below is part of the code for that class:

... Initial comments omitted ...
package mi_foo.bar.tests;

// The class to test
import mi_foo.bar.Bar;

// JUnit stuff
import org.junit.Test;
import org.junit.Before;
import static org.junit.Assert.assertEquals;

// Omit comment
public class BarTests
{
    // The class under test
    private Bar bar;

    // The setup function to create new instances of bar for each test
    @Before
    public void setUpBar()
    {
        bar = new Bar();
    }

    // Test initialisation
    @Test
    public void initialization()
    {
        // Check we start at zero
        assertEquals(0, bar.memoryRecall());
    }

    // Test addition
    @Test
    public void addition()
    {
        assertEquals(5, bar.add(5));
        assertEquals(5+3, bar.add(3));
        assertEquals(5+3+7, bar.add(7));
    }
... Rest of class omitted ...

This test class is relatively straightforward. We create a new package, mi_foo.bar.tests, to which all the tests in this directory will belong. We then include the class under test and the JUnit classes we want to use. We then define our BarTests class which contains five tests (two shown in the snippet above).

The BarTests class includes a private instance of the Bar class under test. The setUpBar function will ensure that a new instance of the Bar class is created before each test is run.

Note: If we wanted the Bar class to be persistent between tests, then we could create a static set up function that uses the @BeforeClass annotation instead of @Before, and make the Bar member static.

Following the set up function, the actual tests are defined. Each test begins with an @Test annotation. Inside the functions we can use the JUnit asserts to test our Bar class functions are working as expected. These tests tend to use the assertEquals assertion to check the return value of each function call against an expected value. For the expected values we do the sum (e.g. 5+3+7) rather than put the total because it can make it clearer why the result is what we expect, and can also avoid any errors in our mental arithmetic!

Testing the Raw class and its console output

Now that the Bar class is tested, we can move onto the Raw class. This is slightly complicated by the fact that the output of the function call is text written to the console. This is solved by using a ByteArrayOutputStream and using it to create a PrintStream to which the console output is redirected. We can then check the contents of this stream and compare it to the expected output.

In the RawTests.java file, the package is declared as in the BarTests example, and the necessary imports done, including import java.io.* for the streams. Then the test class itself can be created:

... Package declaration and imports omitted ...
public class RawTests
{
  //! The class under test
  private Raw raw;

  //! Handle things sent to System.out
  private ByteArrayOutputStream out_str = new ByteArrayOutputStream();

  //! The setup function to create a new instance of raw for each test
  @Before
  public void setUpRaw()
  {
    raw = new Raw();
    System.setOut(new PrintStream(out_str));
  }

  //! Test solving all world problems
  @Test
  public void solveAllWorldProblems()
  {
    raw.solveAllWorldProblems();
    String expected = "solveAllWorldProblems: Not Yet Implemented";
    expected += System.getProperty("line.separator");
    assertEquals(expected, out_str.toString());
  }

  //! Clean up the data as necessary
  @After
  public void tearDown()
  {
    System.setOut(null);
  }
}

After including a private instance of the class under test, a private instance of a ByteArrayOutputStream is also contained in the class. The setUpRaw function initializes the Raw class before each test and redirects the console output to the ByteArrayOutputStream.

In the test solveAllworldProblems, the Raw solveAllWorldProblems function is called, and the result (which will be in our stream) tested against an expected string. Note that we have to append System.getProperty(“line.separator”) to the end of our expected string as println is used in the Raw function. As the line separator differs on Windows and Linux, we need to make sure we append the correct system dependent line ending, or the test will fail on one system or the other. If, for example, we use ‘\n’, the test will pass on Linux, but fail on Windows.

Once the test is complete, the @After function is called. This simply redirects stdout back to the console.

This test enables us to test console output, but perhaps a better way is not to use console output directly in the first place. Using one of the logging systems would be a better option for output, and so the next blog post is going to be about adding logging to our code.

Note: Using ByteArrayOutputStream and PrintStream to capture console output was taken from this stackoverflow answer.

Adding a test suite

As we now have more than one test in the bar directory, it would be convenient to be able to run all the tests with one command. This is done by creating a test suite which runs everything in the bar/tests directory. The suite is created in a file in that directory and named MIFooBarTestSuite.java. The name includes the path to the suite. The code for the suite is pretty straightforward:

... Initial comments omitted ...
package mi_foo.bar.tests;

// The JUnit stuff for the suite
import org.junit.runner.RunWith;
import org.junit.runners.Suite;

//! A Suite class to handle the tests
@RunWith(Suite.class)
@Suite.SuiteClasses(
    {
      BarTests.class,
      RawTests.class
    }
  )
public class MIFooBarTestSuite
{
}

After the package declaration and the imports, the @RunWith annotation is used to denote that this function should be run with the Suite class. We then define which classes we want to run with the Suite class. In this case, BarTests and RawTests. These annotations need to belong to a public class, but no code is needed within that class. How we can run this suite, or individual tests is covered later.

Testing the Foo class

Once the bar sub-directory tests have been set up, the tests in the higher level mi_foo directory can be created.

The test for the Foo class is very similar to the test of Raw, except the building of the expected string is a little more complicated:

... Initial comments and setUp omitted ...
//! Test displaying a string
@Test
public void displayFoo()
{
  String disp_string = "Test Display String";
  foo.displayFoo(disp_string);
  String lsep = System.getProperty("line.separator");
  String expected = "solveAllWorldProblems: Not Yet Implemented";
  expected += lsep;
  expected += "foobar.Foo showing string: \n\t";
  expected += disp_string + lsep;
  assertEquals(expected, out_str.toString());
}
... tearDown function omitted ...

In the Foo class displayFoo function, “\n\t” is used for display formatting as well as using println. println and ‘\n’ don’t produce the same result (on Windows, at least), so where ‘\n’ is used in the display, ‘\n’ should also be put into the expected string, and where println is used we use the line separator.

Creating a hierarchical suite

It is useful to create a suite that can run the tests in the mi_foo directory as well as the tests for code it uses (in this case the tests in mi_foo/bar). To do this we create a hierarchical test suite that includes the Foo tests and the suites of any directories below the mi_foo directory. This is done in a file in mi_foo/tests that is named MIFooTestSuite.java:

... Initial comments omitted ...
package mi_foo.tests;

// Import the sub tests
import mi_foo.bar.tests.MIFooBarTestSuite;

// The JUnit stuff for the suite
import org.junit.runner.RunWith;
import org.junit.runners.Suite;

//! A Suite class to handle the tests
@RunWith(Suite.class)
@Suite.SuiteClasses(
    {
      MIFooBarTestSuite.class,
      FooTests.class
    }
  )
public class MIFooTestSuite
{
}

This is similar to the earlier suite, except that we import the MIFooBarTestSuite and add that to the SuiteClasses. This suite will now run everything in the mi_foo/bar/tests suite as well as the tests in this directory.

Adding a ‘run all’ suite

It is useful to be able to make a single call that will run all the tests. As it happens, we only have one module at the moment, so running MIFooTestSuite will do that, but as we add other modules we need a top-level class to run all suites for all modules. This suite is created in the top src directory in a file named TestRunner.java:

... Initial comments omitted ...
// Import the main test suites from the sub packages
import mi_foo.tests.MIFooTestSuite;

// The JUnit stuff for the suites
import org.junit.runner.RunWith;
import org.junit.runners.Suite;

//! A Suite class to handle all the tests
@RunWith(Suite.class)
@Suite.SuiteClasses(
    {
      MIFooTestSuite.class
    }
  )
public class TestRunner
{
}

Given the earlier suites, this code is simple enough. The MIFooTestSuite is imported and run by this suite. When other modules are created, their top-level suites will be included in this file and run. This is all we have to do as running MIFooTestSuite will run the tests in mi_foo and all the test suites below it.
 

Building the tests

Now we have all our test code in place, we want to be able to build it with SCons. Only one small change is required to the SConstruct file to do this, and that change is to ensure the CLASSPATH variable is set from the system environment:

...

# On Windows, the path is incorrect, so change the path to
# the environment one. Even though windows is the problem,
# do it for all platforms
newpath=os.environ.get('PATH')
env.Append(ENV = { 'PATH' : newpath })

# Set the classpath to the CLASSPATH environment variable
# for finding JUnit and maybe others
cpath=os.environ.get('CLASSPATH')
env.Append(JAVACLASSPATH = cpath)

# On Windows the Jar builder isn't loaded properly, so do
# it explicitly
SCons.Tool.jar.generate(env)

# Build the lot with a single command!
env.Java(env['bdir'], env['sdir'])
...

Now in the the mi_java_scons directory containing the SConstruct file we can type scons and the result should be something like:

> scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
javac -classpath :/usr/share/junit/junit.jar -d build →
  -sourcepath src src/TestRunner.java src/mi_foo/Foo.java →
  src/mi_foo/tests/FooTests.java →
  src/mi_foo/tests/MIFooTestSuite.java →
  src/mi_foo/bar/Bar.java src/mi_foo/bar/Raw.java →
  src/mi_foo/bar/tests/BarTests.java →
  src/mi_foo/bar/tests/MIFooBarTestSuite.java →
  src/mi_foo/bar/tests/RawTests.java
scons: done building targets.

 

Running the tests

Once built the tests can be run by changing into the build directory and calling the appropriate class with the org.junit.runner.JUnitCore test runner. For example, to run all the tests, we simply use the TestRunner class:

> java org.junit.runner.JUnitCore TestRunner
JUnit version 4.10
.......
Time: 0.023

OK (7 tests)

If we want to just run the tests in the mi_foo package (actually the same as all tests at the moment), still in the build directory, type:

> java org.junit.runner.JUnitCore mi_foo.tests.MIFooTestSuite
JUnit version 4.10
.......
Time: 0.021

OK (7 tests)

To only run the tests in the mi_foo.bar package, again in the build directory, type:

> java org.junit.runner.JUnitCore mi_foo.bar.tests.MIFooBarTestSuite
JUnit version 4.10
.......
Time: 0.019

OK (6 tests)

Finally, to run the tests for a single class (the tests for the Bar class for example), inside the build directory, simply type:

> java org.junit.runner.JUnitCore mi_foo.bar.tests.BarTests
JUnit version 4.10
.....
Time: 0.014

OK (5 tests)

Any test or suite of tests can be easily run in a similar way.
 

Summary

This post covered how we added tests to our existing mi_java_scons code using JUnit. We covered the installation of JUnit on both Windows and Linux, and how those installations were tested using a simple test example. The test example used both the pre-JUnit 4 method of extension of classes, and the JUnit 4 method of using annotations.

The annotation method was then used to add tests to the mi_java_scons code, creating tests for each class in the current code. Test suites were also created to collect together the tests at each level and enable them to be run with one call. A hierarchy of suites enabled one test suite to call another until, at the top-level, a single suite could be called to run all tests.

How this code was built using SCons was then described, followed by how the tests could be run once built.

---

Revision 51

Made some methods public due to a mistake in the initial draft of the classes

Revision 52

Added the class path to the SConstruct and renamed all the test classes from *Test.java to *Tests.java. Added tests for Bar and Raw classes

Revision 53

Changed line ending for expected string in RawTests to use the System.getProperty(“line.separator”) to work properly on Windows

Revision 54

Updated some Doxygen comments

Revision 55

Added test suites and the TestRunner so that multiple tests can be run with one call

Revision 56

Changed expected string in FooTests as ‘/n’ and System.getProperty(“line.separator”) do not produce the same output

Revision 57

Changed the Doxygen version filter to use awk instead of sed

Revision 59

Changed the README to correctly describe how to run the tests

Revisions 58 and 60

Changes to other, unrelated code in the Testing repository

Current repository revision: 50 (Testing)

Code highlight key

Grey: Code unchanged
Red: Code removed
Green: Code added
Blue: Code not shown

→ Denotes the current line and the next are really one line

The next blog post in this series building towards an Android development environment was supposed to be Using JUnit for testing in Java / SCons builds. However, before getting onto the testing, a lot of code and a fully functional SConstruct file have been created. In doing so, a couple of problems have been solved, including fixing a problem with SCons building even the simplest Java builds on Windows. It seems like enough has been done to include a post before the testing begins.

If you’re not interested in our code and SConstruct file, but just want to know how to fix the SCons Java and / or Jar builders on Windows, the answer is further down the post, here.

 

Adding some code for testing

The code being added to make up the mi_java_scons example is a Java version of the example code in mi_test_scons. That code experimented with using SCons to create a C++ build environment. So far, the foobar library from that example code has been implemented (without the tests). In the Java version, this package is called mi_foo instead of foobar. The structure of the code is as follows:

mi_java_scons/
+-- src/
|    +-- mi_foo/
|    |    +-- Foo.java
|    |    +-- bar/
|    |    |    +-- Bar.java
|    |    |    +-- Raw.java
|    |    |    +-- tests/
|    |    |    |    +-- BarTest.java
|    |    |    |    +-- RawTest.java
|    |    +-- tests/
|    |    |    +-- FooTest.java
|    +-- TestRunners.java
+-- docs/
|    +-- html_files/
|    |    +-- (various image and style files)
+-- Doxyfile
+-- intro.txt
+-- LICENSE.txt
+-- README.txt
+-- SConstruct
+-- versionfilter.sh

In the code, the mi_foo directory is the package mi_foo, with the bar directory having the package declared as mi_foo.bar. The .java files in the tests directories are, at present, empty, as is TestRunners.java.

The docs directory and the Doxyfile, intro.txt and versionfilter.sh files are all for Doxygen. README.txt contains instructions on getting and building the code, and LICENSE.txt includes the GNU General Public License under which the code is licensed.

The SConstruct file is what does the building work, and will be described in detail below. This post will not contain details of the contents of the .java files as they are straightforward. To get the code and have a look, simply use Subversion to check out the code from the repository with:

svn checkout http://www.mereidea.com/svn/testing/trunk/mi_java_scons

Alternatively, you can browse the code at the WebSVN repository. More details can be seen in the Doxygen documentation, which can be found here.

Note: The instructions in the documentation for running the tests don’t yet work as no testing code has been written.

 

The basic SConstruct file

In the previous post a simple SConstruct file was created to build Java files from within a src directory. The whole SConstruct file was:

Java('build','src')

This would actually have worked as the SConstruct file for the mi_java_scons project, but we wanted to offer choices, and enable the creation of .jar files for each package, so the SConstruct file, initially, looked like this:

# ...License comment ommitted...

# Set up allowable options (using options enables us to set up help →
and
# input arguments)

# Allow options to come from a file
optfile = ARGUMENTS.get('optfile','')
Help("""You can use the 'optfile' option to specify a file →
containing other
options (e.g.): optfile='myopts.py'
""")

vars = Variables(optfile)
vars.AddVariables(
    ('bdir', "Set to the build directory if you don't like 'build'", →
'build'),
    ('sdir', "Set to the source directory if you don't like 'src'", →
'src'),
    ('javaver', "Set to the Java version you are using. Defaults →
to '1.6'",
                '1.6'),
    BoolVariable('jar',  "Wrap the packages into .jar files",'false')
   )

# Create an environment containing all the options, and add help
env = Environment(variables = vars)
Help(vars.GenerateHelpText(env))

# Check for any unknown options
unknown = vars.UnknownVariables()
if unknown:
  print "You passed in the following unknown options:", unknown.keys()
  Exit(1)

# Set the java version to ensure correct dependencies
env.Replace(JAVAVERSION = env['javaver'])

# Build the lot with a single command!
env.Java(env['bdir'], env['sdir'])

Note: As we are now using SCons version 2.0.1, we decided to finally move from the long deprecated Options, AddOptions and BoolOption to the newer Variables, AddVariables and BoolVariable. Why the name of these methods were changed is not clear, but in a few versions of SCons (according to the SCons deprecation policy) our old scripts will no longer work. I’m sure the folks at SCons had a good reason for breaking backwards compatibility and losing some consistency by renaming something which handles options from Options to Variables, especially when they tend to be constants!

Even with the options added, the SConscript file is still short and easy to follow. Note that this doesn’t yet include the code to build the .jar files.

This script was run on a Linux based computer, and the result was:

> scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
javac -d build -sourcepath src src/TestRunners.java →
src/mi_foo/Foo.java src/mi_foo/tests/FooTest.java →
src/mi_foo/bar/Bar.java src/mi_foo/bar/Raw.java →
src/mi_foo/bar/tests/BarTest.java →
src/mi_foo/bar/tests/RawTest.java
scons: done building targets.

After the build, the build directory contained the directory mi_foo, which itself contained Foo.class and the directory bar. The bar directory contained the files Bar.class and Raw.class. This package structure is correct for running classes in the build directory, or simply using something similar to -classpath build when executing a class.

Note: Typing scons again without making code changes should tell you that ‘.’ is up to date. However, typing scons with the code in its current state rebuilds the code. This is because the TestRunners.java file is empty (as are others), and so gets rebuilt as no .class file exists. This problem should go away when this code is written.

Now that the SConstruct file seems to work, the part to create the .jar files (if requested) can be added:

# ...License comment ommitted...

import glob, re

# Set up allowable options (using options enables us to set up help →
and
# input arguments)

# Allow options to come from a file
optfile = ARGUMENTS.get('optfile','')
Help("""You can use the 'optfile' option to specify a file →
containing other
options (e.g.): optfile='myopts.py'
""")

vars = Variables(optfile)
vars.AddVariables(
    ('bdir', "Set to the build directory if you don't like 'build'", →
'build'),
    ('sdir', "Set to the source directory if you don't like 'src'", →
'src'),
    ('javaver', "Set to the Java version you are using. Defaults →
to '1.6'",
                '1.6'),
    BoolVariable('jar',  "Wrap the packages into .jar files",'false')
   )

# Create an environment containing all the options, and add help
env = Environment(variables = vars)
Help(vars.GenerateHelpText(env))

# Check for any unknown options
unknown = vars.UnknownVariables()
if unknown:
  print "You passed in the following unknown options:", unknown.keys()
  Exit(1)

# Set the java version to ensure correct dependencies
env.Replace(JAVAVERSION = env['javaver'])

# Build the lot with a single command!
env.Java(env['bdir'], env['sdir'])

# If jars are requested
if env['jar']:
  # Get a name for each package
  packs = glob.glob('%s/mi_*' % env['sdir'])

  # We need to make sure Jar operates in the build directory
  env.Replace(JARCHDIR=env['bdir'])

  # Go through the packages and jar each
  for nxtp in packs:
    nxtp = re.sub('%s' % env['sdir'], '', nxtp)
    env.Jar(target = '%s/jars/%s.jar' % (env['bdir'], nxtp),
            source = '%s/%s' % (env['bdir'],nxtp))

This code simply uses glob to get a list of all directories in the src directory beginning with mi_. These directories will be individual packages, each of which will be turned into a jar file.

The list of directories is iterated through. For each directory name, re (the regular expression library) is used to remove the source directory name from the package directory using the sub function. os.path.basename could be used, but it’s possible that would return an empty string in some circumstances, so it was thought safer to use re.sub. The resulting string can be used to create the appropriate .jar file using the SCons Jar command.

Note that the code also sets the JARCHDIR environment variable to the build directory. If this is not done then the build directory name is included as part of the path in the .jar file, and the packages will not work properly.

Running SCons with the jar option set results in:

> scons jar=true
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
javac -d build -sourcepath src src/TestRunners.java →
src/mi_foo/Foo.java src/mi_foo/tests/FooTest.java →
src/mi_foo/bar/Bar.java src/mi_foo/bar/Raw.java →
src/mi_foo/bar/tests/BarTest.java →
src/mi_foo/bar/tests/RawTest.java
jar cf build/jars/mi_foo.jar -C build mi_foo
scons: done building targets.

and the file mi_foo.jar is created in the build/jars/ directory.

 

Fixing the SCons Java build for Windows

The code discussed above was committed to the Testing repository under the name mi_java_scons. The code was then checked out under Windows and SCons was run. The result was:

> scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
javac -d build -sourcepath src src\TestRunners.java →
src\mi_foo\Foo.java src\mi_foo\bar\Bar.java →
src\mi_foo\bar\Raw.java →
src\mi_foo\bar\tests\BarTest.java src\mi_foo\bar\tests\RawTest.java →
src\mi_foo\tests\FooTest.java
'javac' is not recognized as an internal or external command,
operable program or batch file.
scons: *** [build\TestRunners.class] Error 1
scons: building terminated because of errors.

It seems that even though the javac command can be used at the command prompt, it cannot be found when called from SCons. A snippet from the SCons environment during a run (produced using print env.Dump() in the SConstruct file) shows why:

...
  'Dir': <SCons.Defaults.Variable_Method_Caller object at 0x028E8430>,
  'Dirs': <SCons.Defaults.Variable_Method_Caller object at 0x028E8450>,
  'ENV': { 'COMSPEC': 'C:\\Windows\\system32\\cmd.exe', 
           'PATH': u'C:\\Windows\\System32',
           'PATHEXT': '.COM;.EXE;.BAT;.CMD',
           'SystemDrive': 'C:',
           'SystemRoot': 'C:\\Windows',
           'TEMP': 'C:\\Users\\chris\\AppData\\Local\\Temp',
           'TMP': 'C:\\Users\\chris\\AppData\\Local\\Temp'},
  'ESCAPE': <function escape at 0x02A441F0>,
  'F77': 'gfortran',
...

The path is a minimal path only including the System32 directory.

Note: Sometimes this path includes more than this – on one of our Windows boxes it includes lots of Visual Studio stuff.

What we do not want to do is have everybody specify their own path somehow (maybe through an option) when they run SCons, or have it hard coded in the SConstruct file. Specifying the path for every run is a hassle, and hard coding the path is wrong, especially when the SConstruct file is being committed to a common repository. The solution is simply to import the system’s path, since typing javac at the command prompt works!

Note: If typing javac at the command prompt does not work, then you either need to reinstall the JDK correctly, or add its directory to your path manually.

We can edit our SConscript file to include the lines:

...
import glob, re, os
...
# Set the java version to ensure correct dependencies
env.Replace(JAVAVERSION = env['javaver'])

# On Windows, the path is incorrect, so change the path to
# the environment one. Even though windows is the problem,
# do it for all platforms
newpath=os.environ.get('PATH')
env.Append(ENV = { 'PATH' : newpath })

# Build the lot with a single command!
env.Java(env['bdir'], env['sdir'])
...

Now when run, all’s ok:

> scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
javac -d build -sourcepath src src\TestRunners.java →
src\mi_foo\Foo.java src\mi_foo\bar\Bar.java src\mi_foo\bar\Raw.java →
src\mi_foo\bar\tests\BarTest.java src\mi_foo\bar\tests\RawTest.java →
src\mi_foo\tests\FooTest.java
scons: done building targets.

But when run with the .jar file building option turned on:

> scons jar=true
scons: Reading SConscript files ...
AttributeError: 'SConsEnvironment' object has no attribute 'Jar':
  File "C:\code\mi_java_scons\SConstruct", line 62:
    env.Jar(target = '%s/jars/%s.jar' % (env['bdir'], nxtp),

Looking on the SCons forum, this post was found that describes a problem with the Jar builder not being created in a SCons environment properly. The post mentions that the builder is not created if the JDK bin directory is not included in the system path, but on our system, it is in the path, and still doesn’t seem to create the builder. However, a solution in the post was implemented:

...
import glob, re, os, SCons.Tool.jar
...
# On Windows, the path is incorrect, so change the path to
# the environment one. Even though windows is the problem,
# do it for all platforms
newpath=os.environ.get('PATH')
env.Append(ENV = { 'PATH' : newpath })

# On Windows the Jar builder isn't loaded properly, so do
# it explicitly
SCons.Tool.jar.generate(env)

# Build the lot with a single command!
env.Java(env['bdir'], env['sdir'])
...

This solution fixed the problem:

> scons jar=true
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
javac -d build -sourcepath src src\TestRunners.java →
src\mi_foo\Foo.java src\mi_foo\bar\Bar.java src\mi_foo\bar\Raw.java →
src\mi_foo\bar\tests\BarTest.java src\mi_foo\bar\tests\RawTest.java →
src\mi_foo\tests\FooTest.java
jar cf build\jars\mi_foo.jar -C build mi_foo
scons: done building targets.

Now that everything works, the final code can be committed and we can move on to include JUnit tests, one post late…

 

Summary

This post covered the initial setting up of the mi_java_scons test library. Some code was set up, and a SConstruct file was created that called the Java builder. Once this worked, extra SConstruct code was added to enable each package to be wrapped into a .jar file. This also worked nicely on a Linux based system, but when committed and checked out on a Windows machine the build was broken as the javac command could not be found by SCons.

This was fixed by including the system path in the SCons environment, but the Jar builder still did not work. A fix for this was found in the SCons forum and implemented, explicitly including the Jar builder in the SConstruct file.

The code was then tested on both Windows and Linux, and worked the same on both.

If SCons wants to be able to claim that the build tool is cross-platform, then it should really be tested on multiple platforms. These problems with Windows showed up on even the simplest Java builds, so should have shown up in any basic testing. This difficulty in even getting simple builds to work well under Windows is why, after testing SCons and blogging about it, we now prefer CMake for C++ builds. Tools should be there to help, not be something that you have to spend time getting working before you can start using them properly! It’s a shame, because SCons seems as if it could be a nice build system.

There are still a couple of problems with the SConstruct that need to be addressed to do with the way SCons builds the Java code, but we will leave that for the next post when we start adding the tests.

---

Revision 43: the initial import of mi_java_scons code.

Revision 44, 45, 46: maintenance to the repository – deleting accidentally committed binary files and adding the correct eol-style property to all .java files.

Revision 47: added the SConstruct code to build .jar files.

Revision 48: a final deletion of a binary file accidentally committed.

Revision 49: added code to set up correct path on Windows so that javac can be found.

Revision 50: changed various things as this blog post was written, including adding SConstruct code to explicitly add the Jar builder.

Current repository revision: 40 (Testing)

Code highlight key
Grey: Code unchanged
Red: Code removed
Green: Code added
Blue: Code not shown

→ Denotes the current line and the next are really one line

The next series of blog posts is going to highlight the path followed to reach the goal of using SCons to build a simple Android application. The steps taken to achieve this are:

1. Building Java with SCons
2. Using JUnit for testing in Java/SCons builds
3. Using Buildbot to run Java/SCons compile/test cycles
4. Integrating Java/SCons builds with Eclipse
5. Testing a GUI application with Java/SCons and JUnit
6. Including the Android API in a Java/SCons build
7. Building an Android app with Java/SCons
8. Dual GUI (Android and Swing) development

Java and Android are subjects that are new to us, so these blog posts will follow the learning process rather than being the advice of experts. As such we hope they will highlight and solve many of the problems people who are also coming to Java and Android for the first time may encounter. Hopefully the series will help anyone wanting to start out.

SCons is to be used for this development as it is a build system we have worked with before (and blogged about) which handles Java simply (or so it seems at the moment) and saves time on learning a new build system like Ant or Maven.

Now onto our first experience of building Java with SCons!

 

Hello, Java World!

To create Java programs using SCons it is necessary to have Java, the Java Development Kit (JDK), SCons and Python installed. Other requirements will be covered when they are first used.

Most computers will already have Java installed. To check whether it is installed, and whether it’s the latest version, simply go to the Java website and click on the Do I have Java? link (ensuring Java is enabled in the browser!). From that page Java can be downloaded and installed. At the time of writing our installed version is Version 6 Update 26.

To program in Java the JDK also needs to be installed. This includes the core packages of interfaces and classes used in the Java language, as well as the javac Java compiler and other development tools.

Now, for the first program (and a test that the Java and JDK are correctly installed), the Java version of Hello, World! can be written. In a directory called scons_test a directory named src was created. The src directory is created as it will be used by SCons later. Inside this src directory, a file named HelloWorld.java was edited with a text editor to contain:

/*
 * Java version of Hello, World!
 */
class HelloWorld
{
  public static void main(String args[])
  {
    System.out.println("Hello, Java World!");
  }
}

At a command prompt, in the src directory containing the HelloWorld.java file, the code was compiled using the command:

javac HelloWorld.java

This created a class file HelloWorld.class. The program was run with the command:

java HelloWorld

And the output was:

Hello, Java World!

 

Hello, SCons World!

Now we want to build our Hello, Java World! program using SCons. To use SCons requires a Python installation. Whether or not Python is installed on a system can be checked by opening a command prompt or shell and typing:

python --version

This will respond with either the Python version or a command not found style error. If Python is not installed already, then it can be downloaded and installed from the Python website.

A SCons installation can be checked with:

scons --version

Again, this will either return a SCons version number (and some other information) or a command not found style error message. If SCons is not installed, then it can be downloaded and installed from the SCons website. The version of SCons installed for this blog is 2.0.1.

Now we’re ready to build the HelloWorld.java code using SCons. A small change was made to the code to change the message output to refer to SCons rather than Java, but this was not really necessary :

/*
 * Java version of Hello, World!
 */
class HelloWorld
{
  public static void main(String args[])
  {
    System.out.println("Hello, Java World!");
    System.out.println("Hello, SCons World!");
  }
}

Also, the HelloWorld.class file created earlier in the src directory was deleted. Then, in the directory above src (the scons_test directory) a new text file named SConstruct was created and edited in a text editor. It only contains a single line:

Java('build','src')

A command prompt was opened in the directory containing the SConstruct file, and SCons was run with the command:

scons

The output should have been something like:

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
javac -d build -sourcepath src src/HelloWorld.java
scons: done building targets.

However, running it on a SuSE Linux based system, on the first run our output was more like:

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
javac -d build -sourcepath src src/HelloWorld.java
javac: unrecognized option '-sourcepath'
src: file not recognized: Is a directory
collect2: ld returned 1 exit status
scons: *** [build/HelloWorld.class] Error 1
scons: building terminated because of errors.

After a little investigation, the program javac turned out to be calling the Gnu program gcj instead of the JDK version of javac. To fix this gcj was uninstalled, and the JDK installed properly. After this was done the SCons run worked as above, and the HelloWorld.class file was generated in the automatically created build directory.

The newly compiled program was run by changing into the build directory and typing:

java HelloWorld

It could also be run by staying in the directory containing the SConstruct file and typing:

java -classpath build HelloWorld

Both will output:

Hello, SCons World!

Note that the directory containing the class has to be set using the -classpath option. Trying to just specify the directory as follows:

java build/HelloWorld

will not work, and will result in a long error something like:

Exception in thread "main" java.lang.NoClassDefFoundError:
build/HelloWorld (wrong name: HelloWorld)
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:634)
        at java.securoty.SecureClassLoader.defineClass(
SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(
URLClassLoader.java:277)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
        at sun.misc.Launcher$AppClassLoader.loadClass(
Launcher.java:294)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: build/HelloWorld. Program will exit

 

Summary

This post describes building Java with SCons. It shows a simple Hello, World! style example compiled both manually, and with a SConstruct file, and shows how to run the resulting class.

None of this code has been committed to the MereIdea Testing Repository, as it was only a simple example. The next post, however, will create Java code that is the equivalent to the C++ example code and will include a couple of packages, archived into jar files, and some tests created using JUnit, which will be stored in the repository for readers to check out.

Current repository revision: 12 (Main)

Code highlight key
Grey: Code unchanged
Red: Code removed
Green: Code added
Blue: Code not shown

→ Denotes the current line and the next are really one line

The mini-series of posts about setting up Buildbot is coming to an end with this post about starting a Windows Slave as a service, adding the UnitTest++ testing framework to the code and running the tests with Buildbot. Earlier posts had described how to set up a Buildbot master with CMake and how to set up Buildbot slaves for CMake.

---

Starting the Buildbot Slave as a Windows Service

Note: Before setting up the Slave to run as a service, make sure it works running from the command line. This ensures everything is set up right!

.
Setting up the Buildbot Slave as a Service started with following the appropriate section (Windows Buildbot service setup) in this piece of documentation. A couple of problems occurred while following this procedure, but these have been solved and the details are below.

Firstly, the script buildbot_service.py was checked for in the Python Scripts directory. It didn’t exist! This was because the script comes with the Buildbot Master installation, but not the Slave. On Windows the Master was unnecessary, and so wasn’t installed. To save installing it now, the buildbot_service.py script was copied across from the Linux Master installer (from within the expanded buildbot-0.8.1/contrib/windows directory). Remember that PyWin32 has to be installed before adding as a service.

Windows XP

The first Slave is to run on a Windows XP machine, and this was set up with a single user (Chris). As this is the default user it should have administrator privileges. However, step 3 in the documentation was followed, and the user Chris was added to the Log on as a service user.

The service was then installed, using the command:

> python C:\Python26\Scripts\buildbot_service.py →
--user BARRACUDA\Chris --password SomePassword --startup auto install

This command reported success, so the next step was to ensure the user (Chris) had permissions on the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BuildBot. Regedit was fired up, the key was browsed to, and the BuildBot folder right clicked on. Permissions… was selected from the context menu, and user Chris added.

As the user Chris created the buildbot Slave configuration directory it was unnecessary to change permissions on the configuration folder. Now the buildslave was ready to start with the command:

> buildbot_service.py start "C:\code\Buildbot\slave"

Unfortunately, that resulted in the error:

Error starting service: The service did not start due to a logon →
failure

It was time to look at what the script had actually installed in the Services dialog (available through Control Panel | Administrative Tools | Services). In the list that pops up, the Buildbot service can be seen. Double-clicking on it brings up the properties, and everything looked sensible. Clicking the Start button brought up the same message as running at the command line did.

After much checking of permissions and retyping passwords, it was decided that the Administrator account should be tried instead of the user Chris, so the details were entered in the Log On tab:

The service was then started again, but this time the service started and stopped immediately. To find out what had happened it was necessary to look in the event viewer (Control Panel | Administrative Tools | Event Viewer). Double-clicking on the latest Warning related to the Buildbot service showed the following details:

There is an ImportError as the buildbot_service.py script tries to import runner from buildbot.scripts. This relates back to the initial problem of the missing buildbot_service.py script – only the Buildbot Slave code is installed on the Windows box.

Looking through the scripts called when the buildslave command is executed, it seems the code that does the actual running is in the file C:\Python26\Scripts\buildslave. This script just has just three lines of actual code, so these were copied and pasted into the buildbot_services.py file in the _RunChild() function in place of the Master running code (the problematic import and a runner.run() command):

def _RunChild():
...
    # Start the buildbot app
    from buildbot.scripts import runner
    runner.run()
    __requires__ =  'buildbot-slave==0.8.1'
    import pkg_resources
    pkg_resources.run_script('buildbot-slave==0.8.1','buildslave')
    print "Service child process terminating normally."

Note: Remember – this code change is only necessary if the Buildbot Master code hasn’t been installed. In fact, if you make this change and hope to start a Buildbot Master it’s unlikely to work!

The service property dialog was then reopened, and the parameters normally passed to start the Slave were entered in the start parameters box:

The service was then started again, and this time all was successful! Checking on the MereIdea Buildbot web pages confirmed that the Slave was now connected to the Master. This could also be started with the command:

> buildbot_service.py start C:\code\Buildbot\slave

All that remained now was to check whether the Builders would still work with the service, and whether the service restarted correctly when the computer was rebooted.

The build did not work due to something mentioned in an earlier post in reference to the setvcvars step of the Windows build; a different command shell is used for each step, so the work done by the setvcvars step is lost by the next step. The call to NMake fails because the Visual Studio paths are no longer set.

This was fixed by removing setvcvars as a standalone first step, and editing setvcvars.bat to set up the environment and call NMake:

call "c:\Program Files\Microsoft Visual Studio →
10.0\VC\vcvarsall.bat" x86
nmake

This file was renamed run_nmake.bat, and then the Windows Compile step in the master.cfg file (wcomp) was changed to use it and the original setvcvars step removed:

...
svnbaseurl='http://svn.mereidea.com/main/'
svnup = source.SVN(mode='update',
                   baseURL=svnbaseurl,
                   defaultBranch='trunk')
svnco = source.SVN(mode='clobber',
                   baseURL=svnbaseurl,
                   defaultBranch='trunk')
setvcvars = shell.ShellCommand(command="setvcvars.bat")
confg = shell.Configure(command="cmake ../src")
wconfg = shell.Configure(command='cmake -G "NMake Makefiles" ../src')
comp = shell.Compile(command='make')
wcomp = shell.Compile(command='run_nmake')
rtest = shell.Test(command='run_tests')
...
fwchg = factory.BuildFactory()
fwchg.addStep(setvcvars)
fwchg.addStep(svnup)
fwchg.addStep(wconfg)
fwchg.addStep(wcomp)
fwchg.addStep(rtest)
...
fwngt = factory.BuildFactory()
fwngt.addStep(setvcvars)
fwngt.addStep(svnco)
fwngt.addStep(wconfg)
fwngt.addStep(wcomp)
fwngt.addStep(rtest)
...

A new build was forced, and was now successful! The computer was rebooted, and the service restarted successfully. The build still worked.

Windows 7/Vista

Most of the Windows 7 setup for a Buildbot Slave as a service was the same as for Windows XP, except that the main steps need to be done in a command prompt that has been ‘Run As Administrator’.

It was also necessary to open the services dialog and reenter the user details in the Log On tab, then press Apply to set the permissions for the user to Lon on as a service. This gets automatically set when the user details are set. There is probably a way to set these permissions separately, but this works!

For Windows 7 the user was in the Administrators group, but when setting up the Vista service a non-administrative user was used. This still worked the same as Windows 7 (‘Run As Administrator’ shell needed to be used) except that the registry permissions needed granting on HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BuildBot, and the service had to be started (whether from the dialog or the command prompt) by the administrator user.

A final note on Window 7. There were problems with using Visual Studio 2010 under Windows 7. This problem was due to CMake and not Buildbot. Therefore, the test for this service was done with Visual Studio 2008. There may be a blog post on using CMake with Windows 7 and Visual Studio 2010 later.

---

Adding a testing framework

UnitTest++ was chosen as the testing framework. This section shows the addition of this framework to the MereIdea code, building the tests with CMake
and running them through Buildbot.

The first step is to add the UnitTest++ code to the MereIdea code. It was decided to simply import the code and build it as part of the mi_test module, which will be built with the rest of the code, rather than expect everyone to get the code and build it. The code is small and easily compiled to a library using a couple of CMake files. Including it also eliminates the main problem with using thirdparty code – if the UnitTest++code changes or becomes unavailable, it won’t require (possibly extensive) changes to the MereIdea code.

The first step, therefore, was to add a new module inside the mi_main source directory. The directory mi_test was created and another directory created inside it – unittest++. This will keep the UnitTest++ code separate from any other code added to the mi_test module. The UnitTest++ code was downloaded to a different directory using svn:

svn co https://unittest-cpp.svn.sourceforge.net/svnroot/unittest-cpp →
unittest-cpp

and the code files (including the Posix, Win32 and test directories) were copied to the new mi_test/unittest++ module. The COPYING and README files were also copied to the top-level mi_test directory.

The CMake files were then added to the mi_test module. Firstly, the top-level CMakeLists.txt file was created. This simply declares the project name and adds the unittest++ subdirectory:

# Give this module its own project name so we can
# refer to its build and source structures
project(MI_TEST)

# And build the unittest++ library
add_subdirectory(unittest++)

Next the CMakeLists.txt file in the unittest++ subdirectory was written. This adds the source files to build the library, and then descends into the tests directory:

# Add the source files here
set(MI_TEST_UNITTEST_PP_SOURCES
    AssertException.h
    Test.h
    Checks.h
    ....
    CurrentTest.h
    CheckMacros.h
    Config.h
    ExecuteTest.h
    TestMacros.h
    TestSuite.h
    TimeHelpers.h
    UnitTest++.h

    AssertException.cpp
    Test.cpp
    Checks.cpp
    ...
    CurrentTest.cpp
)

# Add the platform specific code
if (WIN32)
  set (MI_TEST_UNITTEST_PP_SOURCES ${MI_TEST_UNITTEST_PP_SOURCES}
       Win32/TimeHelpers.h
       Win32/TimeHelpers.cpp
  )
else ()
  set(MI_TEST_UNITTEST_PP_SOURCES ${MI_TEST_UNITTEST_PP_SOURCES}
      Posix/SignalTranslator.h
      Posix/TimeHelpers.h
      Posix/SignalTranslator.cpp
      Posix/TimeHelpers.cpp
  )
endif (WIN32)

# And add the library
add_library(miunittest++ ${MI_TEST_UNITTEST_PP_SOURCES})

# Put the library in a lib subdirectory
# Do for static, shared and module libraries
set_target_properties(miunittest++ PROPERTIES
                      ARCHIVE_OUTPUT_DIRECTORY ${MI_TEST_BINARY_DIR} →
/lib
                      LIBRARY_OUTPUT_DIRECTORY ${MI_TEST_BINARY_DIR} →
/lib
)

# Build the tests
add_subdirectory(tests)

All the source files and header files are included in the CMakeLists.txt file. This is because, even though some of the header files aren’t needed for building the library, the header files won’t appear in any generated Visual Studio projects if they are not included in the CMakeLists.txt file. The last new CMakeLists.txt file to be created is in the mi_test/unittest++/tests directory:

# Add the source files here
set(MI_TEST_UNITTEST_PP_TEST_SOURCES
    RecordingReporter.h
    ScopedCurrentTest.h

    Main.cpp
    TestAssertHandler.cpp
    TestChecks.cpp
    ...
    TestCurrentTest.cpp
    TestTestSuite.cpp
)

# Create the test executable
add_executable(unittest++_tests ${MI_TEST_UNITTEST_PP_TEST_SOURCES})

# Set the libraries required
target_link_libraries(unittest++_tests miunittest++)

# Add as a test (so we can use 'make test')
add_test(unittest++_tester unittest++_tests)

Then the top-level source CMakeLists.txt file is edited to turn testing on (so add_test actually adds a test) and to include the mi_test module.

...
include_directories(${MI_MAIN_SOURCE_DIR})

# Make sure tests are built
enable_testing()

# Now set the subdirectories to work through. Ensure these are done
# in the correct order
add_subdirectory(mi_test)
add_subdirectory(mi_hello)

Now changing to the build directory in a Linux shell and typing:

> cmake ../src
> make
> make test

will generate the appropriate make files, build the code, and run the tests which should pass 100%. Under Windows it would be:

> cmake -G "NMake Makefiles" ../src
> nmake
> nmake test

On Windows the CMake GUI could also be run to generate the Visual Studio projects which can be built and run within the IDE.

---

Changing the mi_hello test to use UnitTest++

Now to change the mi_hello tests to use the UnitTest++ code. Before this is done, however, a small change to the mi_hello::Hello class is needed. Instead of sending the output to the console (std::cout) it needs capturing so that it can be tested. The header file becomes:

...
#include <string>
#include <ostream>

...
    //! Display a message
    /*!
         Display a string passed in (or "Hello, World!")
         Send a string parameter to a stream

         \param disp_str The string to display (defaults to →
"Hello, World!")
         \param os The stream to send the string to
    */
    void display(const std::string &disp_str = "Hello, World!", →
std::ostream &os) const;
...

Note that the default for the display string (“Hello, World”) was removed as it will have to be entered anyway when changing the stream. As this class will only be used for testing, defaults don’t really make much sense. The cxx file is changed to match:

...
#include <iostream>
...
void mi_hello::Hello::display(const std::string &disp_str →
/*= "Hello, World!"*/,
                              std::ostream &os) const
{
  std::cout << disp_str << std::endl;
  os << disp_str;
}

Finally the test can be changed to use the new mi_hello::Hello class and the UnitTest++ code (comments omitted):

...
#include <mi_hello/hello.h>
#include <mi_test/unittest++/UnitTest++>
#include <sstream>

struct MI_HelloFixture
{
  MI_HelloFixture() { }
  ~MI_HelloFixture() { }

  mi_hello::Hello hello_;
  std::string test_str_;
  std::stringstream str_;
};

TEST_FIXTURE(MI_HelloTestFixture, TestHello)
{
  test_str_ = "Hello, World!";
  hello_.display(test_str_, str_);
  CHECK_EQUAL(test_str_, str_.str());
}

TEST_FIXTURE(MI_HelloTestFixture, TestGoodbye)
{
  test_str_ = "Goodbye, World!";
  hello_.display(test_str_, str_);
  CHECK_EQUAL(test_str_, str_.str());
}

int main()
{
  mi_hello::Hello h;
  h.display();
  h.display("Goodbye, World!");

  return 0;
}

Now that the main() function has been removed from this file (as there could be several of these test files), a new file needs to be created to run the tests. This file is named test_hello_main.cxx and is created alongside test_hello.cxx. This new file simply contains:

#include <mi_test/unittest++/UnitTest++.h>
#include <mi_test/unittest++/TestReporterStdout.h>

int main()
{
  return UnitTest::RunAllTests();
}

Each tests directory in the libraries included in the MereIdea Main Projects will contain one or more test files (like test_hello.cxx) and one main file (like test_hello_main.cxx).

Finally, the CMakeLists.txt file in mi_hello/tests was edited for the new code:

...
set(MI_HELLO_TEST_SOURCES
    test_hello_main.cxx
    test_hello.cxx
)

add_executable(test_hello test_hello.cxx)
add_executable(test_hello ${MI_HELLO_TEST_SOURCES})

target_link_libraries(test_hello miunittest++ mihello)

add_test(test_hello_tester test_hello)

Now the code can be built and tested from within the build directory with the familiar:

> cmake ../src
> make
> make test

All tests should pass with some output similar to:

Running tests...
Test project /home/user/code/mereidea/mi_main/build
     Start 1: unittest++_tester
1/2 Test #1: unittest++_tester ................   Passed    0.10 sec
     Start 2: test_hello_tester
2/2 Test #2: test_hello_tester ................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) =   0.12 sec

This is the output from CTest (part of CMake), which is called when using make with the test target. It shows this summary by default, which will be enough for displaying in
Buildbot as it will show which library test has the failure. It would then be up to the developer to go to that library’s test directory and type ctest -V or, with
CMake version 2.8 and later, ctest –output-on-failure. This will give more information about the problem test.

---

Updating Buildbot to use new tests

The last step in the new testing framework is to update the factories in the
Buildbot Master to call make test:

...
rtest = shell.Test(command='mi_hello/tests/test_hello')
rtest = shell.Test(command='ctest')
wrtest = shell.Test(command='mi_hello\\tests\\test_hello')
...
fwchg.addStep(wrtest)
fwchg.addStep(rtest)
...
fwngt.addStep(wrtest)
fwngt.addStep(rtest)
...

Note that the factory uses ctest instead of make test. The two are equivalent in the
CMake world. Calling ctest means that the Windows and Linux commands can be identical. Otherwise make test would need to be called on Linux, and nmake test on Windows. Calling nmake test would also mean that another script would have to be created to set up the Visual Studio environment before calling NMake.

---

Fixing Visual Studio problems

Once the new testing framework was committed, the Master configuration updated and the Master restarted, the code was built and tested using the Linux and Windows builders. Two problems arose in the Visual Studio build: There were some warnings about strcpy being insecure and an unhandled exception occurred in the UnitTest++ test suite.

The unhandled exception was tackled first. This problem stemmed from the fact that CMake, by default, uses C++ exceptions without structured exceptions (switch /EHsc). This means that C++ exceptions will be caught (using throw, try and catch), but system exceptions will not.

To ensure all exceptions are handled as a test failure by UnitTest++ the exception switch to the compiler has to be changed to /EHa. This is done for all compiled programs (any part could use the UnitTest++ code) by setting the CMake variable CMAKE_CXX_FLAGS to include the corrected switches:

# The project is MI_MAIN, so variables like ${MI_MAIN_SOURCE_DIR} and
# ${MI_MAIN_BINARY_DIR} can be used
project(MI_MAIN)

# Ensure the make flags are as we need (/EHa on Windows)
if (MSVC)
  set(CMAKE_CXX_FLAGS "/DWIN32 /D_WINDOWS /W3 /Zm1000 /EHa /GR"
      CACHE
      STRING
      "Flags used by the compiler during all build types."
      FORCE)
else()
  set(CMAKE_CXX_FLAGS "-Wall -pedantic -ansi"
      CACHE
      STRING
      "Flags used by the compiler during all build types."
      FORCE)
endif(MSVC)

This is done in the top-level (src) CMakeLists.txt file. Note that, while compiler flags are being set, the flags for use with gcc are also set. If somebody uses another compiler, any necessary flags should be set here.

The final change is to fix the warnings produced by the Windows build. These warnings occur because strcpy() can be unsafe, and strcpy_s() should be used instead. However, strcpy_s() is not cross-platform, so it won’t be used here! Instead, it will just be assumed that the authors of UnitTest++ use strcpy() in a safe way and this warning can be ignored. It is possible to define _CRT_SECURE_NO_WARNINGS in the preprocessor switches, but this assumes that everyone who uses strcpy() anywhere in the code will use it safely. Instead, it’s better to make programmers leave an “I have thought about this and think it’s safe” message when using an unsafe function. This is done by expecting developers to disable the warning locally.

There are only three files in which the warning occurs – AssertException.cpp in the top-level of the UnitTest++ code, and RecordingReporter.h and TestDeferredTestReporter.cpp in the tests sub-directory. In those files any call to strcpy() is surrounded by #pragma warning() macros; one to turn the warning off, and another to turn it back on again. For example, in AssertException.cpp:

AssertException::AssertException(char const* description, →
char const* filename, int lineNumber)
    : m_lineNumber(lineNumber)
{
	using namespace std;
#ifdef _MSC_VER
#  pragma warning( disable : 4996 )
#endif
    strcpy(m_description, description);
    strcpy(m_filename, filename);
#ifdef _MSC_VER
#  pragma warning( default : 4996 )
#endif
}

With the exception solved and the warnings removed from the build, the code can now be committed and the results viewed in the Buildbot Waterfall.

---

Final note

To start adding new Buildbot Slaves to the installation, it was necessary to rename the Builders more sensibly as a builder is needed for each machine to add. In doing this, the builds were moved to new directories and were rebuilt from scratch. The nightly build on Windows failed in the configuration step because the Visual Studio environment wasn’t set up when CMake was called and the compiler couldn’t be found. This hadn’t shown up before because CMake was run in a directory in which it had been previously run successfully (when the buildslave was run in a shell with the correct environment – before changing to a service). To fix this problem and to ensure anything like it is caught earlier in future, the master.cfg was changed to run a batch file for running CMake on Windows (similar to the run_nmake script) and an extra step was added to remove the CMakeCache.txt file on all nightly builds (when the code should be clobbered). The run_cmake.bat script is simply:

call "C:\Program File\Microsoft Visual Studio 10.0\VC\vcvarsall.bat" →
x86
cmake -G "NMake Makefiles" ../src

and the changes to the master.cfg file were:

...
confg = shell.Configure(command="cmake ../src")
wconfg = shell.Configure(command=['cmake','-GNMake Makefiles',→
'../src'])
wconfg = shell.Configure(command='run_cmake')
rtest = shell.Test(command='ctest')
rcmake = shell.ShellCommand(command='rm -f CMakeCache.txt')
wrcmake = shell.ShellCommand(command='del /F /Q CMakeCache.txt')
...
flngt = factory.BuildFactory()
flngt.addStep(svnco)
flngt.addStep(rcmake)
flngt.addStep(confg)
...
fwngt = factory.BuildFactory()
fwngt.addStep(svnco)
fwngt.addStep(wrcmake)
fwngt.addStep(wconfg)
...

---

Summary

This post covered two major topics; setting up a Buildbot Slave as a service on Windows, and putting the Mere Idea Main Projects’ tests into the UnitTest++ framework.

The Slaves were installed as services following a page on the Buildbot website. The instructions on that page were mostly correct, but a couple of problems arose because the script to install the service (buildbot_service.py) isn’t included in the Slave installer, so had to be copied and modified from the one in the Master installer. Reading the Buildbot mailing list, a change to rectify this may soon be made.

The use of UnitTest++ in the code was straightforward – copying the code into the mi_main source code repository and changing the local tests to use it. The tests could then be run by using make test or ctest once the code was built.

The Buildbot Builders were then changed to carry out the testing correctly.

---

Repository Changes since last post

Revision 3 and 4 added Doxygen support for the Main Projects code. The result can be seen here.

Revision 5 added a README file on how to download and build the code (also available in the Doxygen.

Revision 7 changed the Doxygen revision filter to correctly display the current repository revision and the last revision at which a particular file was changed.

Revision 8 added the UnitTest++ code and changed the ‘hello’ test.

Revision 9 fixed a problem with using targets containing ‘++’ in their name by changing them to ‘pp’. The problem occurred with NMake.

Revision 10 and 11 fixed the problems mentioned in this post regarding building the UnitTest++ code under Windows (using /EHa and #pragma).

Revision 12 added support for MinGW as the code won’t compile with the -pedantic flag set, so turn it off if compiling with MinGW.

Current repository revision: 2 (Main)

Code highlight key
Grey: Code unchanged
Red: Code removed
Green: Code added
Blue: Code not shown

→ Denotes the current line and the next are really one line

The last post described how to create a Buildbot Master for compiling the MereIdea libraries. CMake is to be used to create the makefiles to build the code. This post will describe how some simple code was created and committed to the Main Subversion repository (http://www.mereidea.com/svn/main/trunk/), including the initial CMake files. Then a Linux Slave will be set up to check out and build the code. This will lead to an iterative process to correct the Builder commands in the Master to work with the Linux platform, before moving onto Windows and setting up a Windows based Slave.

An out-of-source build is preferred, as it avoids cluttering up the source tree and accidental commits of binary files to the code repository. The general directory structure for such code will be:

code/
+-- build/
+-- src/
|    +-- CMakeLists.txt
|    +-- mi_lib1/
|    |    +-- CMakeLists.txt
|    |    +-- foo.h
|    |    +-- foo.cxx
|    |    +-- bar/
|    |    |    +-- CMakeLists.txt
|    |    |    +-- bar.h
|    |    |    +-- bar.cxx
|    |    |    +-- tests/
|    |    |    |    +-- CMakeLists.txt
|    |    |    |    +-- test_bar.cxx
|    |    +-- tests/
|    |    |    +-- CMakeLists.txt
|    |    |    +-- test_foo.cxx
|    +--  mi_lib2/
|    +--  mi_lib3/

Everything within the src/ part of the tree (not including the src/ directory itself) will be committed to the repository.

---

Initial Code Tree

With this in mind, a src/ directory was created and a library directory (mi_hello) was added inside src/. The mi_hello library will always contain simple code (pretty much a C++ ‘Hello, World’) that can be used to test the compiler. It will be the first library built (with the exception of mi_test that will set up the testing framework) with at least one test to be linked. The files in here should be amended if C++ features are used in the other libraries which may or may not work on some compilers. Having this small library means you can get quick feedback as to whether the rest of the libraries should build (though not guaranteed) with your system.

The initial code that will be used to test the Buildbot set up was created in the actual directory structure:

code/
+-- src/
|    +-- CMakeLists.txt
|    +-- LICENSE.txt
|    +-- mi_hello/
|    |    +-- CMakeLists.txt
|    |    +-- hello.cxx
|    |    +-- hello.h
|    |    +-- tests/
|    |    |    +-- CMakeLists.txt
|    |    |    +-- test_hello.cxx

You can check out the code for yourself from http://svn.mereidea.com/main/trunk/ as revision 2, and you should get this structure. This can be done with:

svn co -r 2 http://www.mereidea.com/svn/main/trunk src

The LICENSE.txt file simply contains a copy of the GNU General Public License under which the code is available.

The files hello.h and hello.cxx contain the definition of a class (mi_hello::Hello) with a single method:

void display(const std::string &disp_str = "Hello, World!") const;

This method simply displays the string passed to it, or Hello,  World! if no other string is passed. The test file test_hello.cxx contains a main() function to call the display() method of the mi_hello::Hello class. The code for this is:

#include <mi_hello/hello.h>
int main()
{
  mi_hello::Hello h;
  h.display();
  h.display("Goodbye, World!");

  return 0;
}

This should compile to an executable and be run when the Buildbot Slave runs. Later this code will be placed in a proper testing framework, but until the Buildbot code is working correctly it’s not necessary to complicate things! The more things added at any testing stage, the more places things could go wrong. Putting things together one at a time makes it easier to locate where any problem is.

---

Initial CMake files

Now that there is some code to compile, the CMake files to help compile it can be added. The first file to add is the top-level CMakeLists.txt file, which defines the project and lists the sub-directories in which further CMakeLists.txt files exist to recursively create the make system. The CMakeLists.txt top-level file in the MereIdea Main code base is initially written as follows:

# Set the minimum version to >= 2.6
cmake_minimum_required(VERSION 2.6)

# The project is MI_MAIN, so variables like ${MI_MAIN_SOURCE_DIR} and
# ${MI_MAIN_BINARY_DIR} can be used
project(MI_MAIN)

# Set up the top-level as an include directory so we can use
# #include <library_name/optional_sub_dirs/file.h>
# in the source to include a file
include_directories( ${MI_MAIN_SOURCE_DIR} )

# Now set the subdirectories to work through. Ensure these are done
# in the correct order
add_subdirectory (mi_hello)

This will set up the project, the include directory and call each library sub-directory to build in turn. The only library so far is the mi_hello compiler test library. To build the mi_hello library requires a CMakeLists.txt file in the mi_hello sub-directory:

# Give this library its own project name so we can
# refer to its build and source structures
project(MI_HELLO)

# Add the source files here
set (MI_HELLO_SOURCES
     hello.h
     hello.cxx
    )

# And add the library
add_library(mihello ${MI_HELLO_SOURCES})

# Put the library in a lib sub-directory
# Do for any shared, static and module libraries
set_target_properties(mihello PROPERTIES
                      ARCHIVE_OUTPUT_DIRECTORY →
${MI_HELLO_BINARY_DIR}/lib
                      LIBRARY_OUTPUT_DIRECTORY →
${MI_HELLO_BINARY_DIR}/lib
                     )

# And build the tests
add_subdirectory(tests)

This builds the mi_hello library, places it in a lib directory and descends into the tests directory. The tests directory then has the final CMakeLists.txt file for this little test:

# Add each test as an executable
# This may change to add_test later when the testing
# framework is added
add_executable(test_hello test_hello.cxx)

# Set the libraries each executable needs to link against
target_link_libraries(test_hello mihello)

Now the makefiles can be generated by changing directory to the build directory and typing:

> cmake ../src
-- The C compiler identification is GNU
-- The CXX compiler identification is GNU
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /use/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Configuring done
-- Generating done
-- Build files have been written to: →
/home/mi_user/code/mereidea/mi_main/build

You should see something similar to the above on a Linux box – Windows is covered later. For more control over the settings of the build, run ccmake instead of cmake and the CMake interface will appear enabling you to alter the settings, such as the build type. The libraries and tests can now be built by simply typing make at the command line. The mi_hello library will end up in build/mi_hello/lib, and the test executable will end up in build/mi_hello/tests. Change to this tests directory and type:

> ./test_hello
Hello, World!
Goodbye, World!

It all appears to have worked! Now we can commit the code to the repository, and test with Buildbot installation.

---

Committing code to Subversion Repository

The code is to be committed to the MereIdea Main repository. Before checking it in it is worth making a small note about Subversion and cross platform code.

One of the annoyances of transferring text files (such as code files) from one platform to another has been the different line ending codes. For example, where Windows uses the carriage return and line feed ending (\r\n), Linux uses just the line feed (\n). Subversion, however, can solve this problem for you to some extend by enabling you to set the eol-style property to native on the files being committed. The good thing is that, rather than having to set this property on every file you commit, you can set the eol-style property (and any other property) automatically for different types (or specific names) of files. This is set in the Subversion conf file. On Linux this can be found in your home directory inside a sub-directory named .subversion, on Windows Vista it is in somewhere like C:\Users\mi_user\AppData\Roaming\Subversion, and on Windows XP it is somewhere like C:\Documents And Settings\Chris\Application Data\Subversion. Near the end of the configuration file is the line:

# enable-auto-props = yes

This is commented out by default, so simply remove the # to turn on the automatic properties. Then just set up the properties required on the specific file types:

### Section for configuring automatic properties.
[auto-props]
### The format of the entries is:
###   file-name-pattern = propname[=value][;propname[=value]...]
### The file-name-pattern can contain wildcards (such as '*' and
### '?').  All entries which match will be applied to the file.
### Note that auto-props functionality must be enabled, which
### is typically done by setting the 'enable-auto-props' option.
*.c = svn:eol-style=native
*.cpp = svn:eol-style=native
*.cxx = svn:eol-style=native
*.h = svn:eol-style=native
*.txt = svn:eol-style=native
SConstruct = svn:eol-style=native
SConscript = svn:eol-style=native
SConfig = svn:eol-style=native
*.cmake = svn:eol-style=native
*.py = svn:eol-style=native
*.sh = svn:eol-style=native;svn:executable
Makefile = svn:eol-style=native
*.png = svn:mime-type=image/png
*.jpg = svn:mime-type=image/jpeg

This is an example that includes SCons files used in other MereIdea projects. These changes only need to be made if you are planning to submit code to the repository.

Now the code can be committed easily with the command (assuming that the files are in a directory named src):

> svn import src http://www.mereidea.com/svn/main/trunk →
-m "Initial import"

Then this can be checked out again (importing doesn’t alter the src tree to become a working directory) by removing the existing source directory and typing:

> svn co http://www.mereidea.com/svn/main/trunk src

After checking it still builds and runs the code is now ready to set up with Buildbot.

Setting up a Buildbot Slave

After committing the code and waiting a few minutes (the Buildbot Master SVNPoller has a pollinterval of an hour) it can be seen that the Master knows it has to do some builds:

It can also be seen that there are five nightly builds queued up as the Master was started a few days ago! Now the Linux Slave can be started and an iterative process of getting the Builder commands correct can begin.

Using the same Linux user as used to set up the Master (and after checking this user can check out and build the code), a directory name slave is created alongside the one named master. This was configured as a Slave with the command:

> buildslave create-slave ~/Buildbot/slave/ www.mereidea.com:9989 →
oSUSE_11.2_x86_64_gcc password1

This creates a directory (info) containing two configuration files that needed editing. These were admin which was edited to include an administrator name and email address for this Slave, and host in which some system information was entered (this can be seen at http://builds.mereidea.com/buildslaves/oSUSE_11.2_x86_64_gcc – the details under the heading Administrator were entered in the admin file, and the details under Slave Information were entered in the host file). Then, inside the slave directory, the Slave can be started with:

> buildslave start

This ran the pending builds on the new Slave and put the results on the web pages.

---

Fixing the Linux Build

The Linux build succeeded in checking out the code, but the Configure step of running CMake failed. This is because when the code is checked out from the repository it is put in a directory named build.  This can be fixed by amending the source.SVN by adding the workdir option:

...
svnup = source.SVN(mode='update',
                   baseURL=svnbaseurl,
                   defaultBranch='trunk'),
                   workdir='src')
svnco = source.SVN(mode='clobber',
                   baseURL=svnbaseurl,
                   defaultBranch='trunk'),
                   workdir='src')
...

This will change the default directory to be src for the SVN checkout, but the configure step will still use build by default. The Master is then restarted to take account of the change, and a build forced (forceBuild was set temporarily to True on the WebStatus – set this in the authz_cfg variable in the master.cfg).

The whole thing now compiles, but the test fails. That is because a place holder (run_tests) was given as the command to execute for the test phase. This was changed to:

...
rtest = shell.Test(command='run_tests')
rtest = shell.Test(command='mi_hello/tests/test_hello')
...

The Linux build is now clean, and the Web Status shows green for both the Linux-change-build and the Linux-nightly-build.

---

Setting up the Windows Slave

Now the Linux build works with the simple code it is time to set up a Slave on the Windows machine and fix any problems with it.

The first step was to install all of the code necessary for Buildbot to work. This was done by simply following the instructions on this page with a couple of changes:

• Only the buildslave code is needed, so only buildbot-slave-0.8.1.zip was downloaded and extracted.
• The installed Python was 2.6.5, so in step 2 it was C:\Python26 and C:\Python26\scripts that were added to the path variable
• After installing Twisted and typing trial –version an ImportError occurred stating that the zope interface was missing. The Zope website didn’t seem to have an installer for Python 2.6, but luckily  a Python egg was available for 2.6 here.
• As the Zope interface was in egg form, setuptool (or distribute) needed to be installed so that easy_install could be used with the egg. Distribute was found here along with installation instructions.
• Once the Zope egg was installed using easy_install then the command trial –version returned the Twisted version number.
• It wasn’t necessary to alter the buildslave.bat file as detailed in step 9.

Once the buildslave code was installed, it was time to try a checkout and build of the code on the command line prior to setting up the Slave. First a batch file was created to call the appropriate vcvars file. This file just had the single line:

"c:\Program Files\Microsoft Visual Studio 10.0\VC\vcvarsall.bat" x86

in it and was named setvcvars.bat (the name in the master.cfg file). It was placed in the directory C:\bin which is on the executable path so it can be executed by the Slave. The binary CMake directory was also placed in the path variable.

TortoiseSVN is usually used for handling Subversion on MereIdea Windows boxes, but this doesn’t come with a command line client, so SlikSVN was installed. Now the test could begin with the sequence of commands the Slave would be asked to execute on the Windows machine:

> setvcvars.bat
Setting environment for using Microsoft Visual Studio 2010 x86 tools.
> svn co http://www.mereidea.com/svn/main/trunk src
A    src\LICENSE.txt
A    src\mi_hello
...
> cd build
> cmake -G "NMake Makefiles" ../src
-- The C compiler identification is MSVC
-- The CXX compiler identification is MSVC
...
-- Configuring done
-- Generating done
-- Build files have been written to: C:/code/mi_main/build
> nmake
...
> mi_hello\tests\test_hello
Hello, World!
Goodbye, World!

So the command set works fine! Two concerns are whether the commands are all run in the same environment (i.e. will the setvcvars call be persistent), and will the forward slashes in the shell.Test function be converted by Python?
The only way to find out is to set up the Slave and see what happens. A directory was created to house the Slave, and then the same buildslave command was run as on Linux:

> buildslave create-slave C:\code\Buildbot\slave\ →
www.mereidea.com:9989 WinXP_VS2010 password2

The created files in the slave\info sub-directory (admin and host) were edited as with the Linux Slave, and then the Slave was started by running:

> buildslave start

in the Slave directory.

Note: The command ‘buildslave start’ on Windows never returns to the command prompt as it runs in the foreground.

Once the slave was started, the queued builds were run. The result was that the configure step (CMake) failed. When looking at the output from the configure it was seen that the CMake error produced was:

CMake Error: Could not create named generator "NMake

This shows that the command wasn’t being passed correctly to the command line. The -G “NMake Makefiles” was being split at the space after NMake. The first thing tried was to change the configure step in the master.cfg to escape the quotes:

wconfg = shell.Configure(command='cmake -G \"NMake Makefiles\" →
../src')

This made no difference, and using a double set of quotes (“”NMake Makefiles”") only changed the error to:

CMake Error: Could not create named generator ""NMake

Next the command was split up into command and arguments:

...
wconfg = shell.Configure(command='cmake -G "NMake Makefiles" ../src')
wconfg = shell.Configure(command=['cmake', '-G "NMake Makefiles"', →
'../src'])
...

This changed the error to:

CMake Error: Could not create named generator  "NMake Makefiles"

So now it became apparent that the quotes were being passed to CMake as part of the argument. The quotes were removed altogether, and the error changed to:

CMake Error: Could not create named generator  NMake Makefiles

This looks correct now, so what was the problem? The answer is there are two spaces between generator and NMake in the error message! A space seems to be passed to CMake as part of the generator name! A final change was made to the master.cfg to fix this:

...
wconfg = shell.Configure(command=['cmake', '-G NMake Makefiles', →
'../src'])
wconfg = shell.Configure(command=['cmake', '-GNMake Makefiles', →
'../src'])
...

This seems strange, but fixes the problem. There must be a reason for this, and at least part of the blame must go to CMake for thinking that using generator names containing spaces was a good idea!

When the Master was restarted, and the build was forced again, the configure and compile passed, but the test failed. The reason for this was simple, and was easily fixed. Windows paths used backslashes, so the test execution requires backslashes. The change was made to the master.cfg:

...
rtest = shell.Test(command='mi_hello/tests/test_hello')
wrtest = shell.Test(command='mi_hello\\tests\\test_hello')
...
fwchg.addStep(rtest)
fwchg.addStep(wrtest)
...
fwngt.addStep(rtest)
fwngt.addStep(wrtest)
...

This fixes the test, and all builders now work!

---

Summary

This post describes the basics of setting up Buildbot Slaves on both Linux and Windows machines, using CMake as part of the build system. Some example code was created, along with the CMake files to generate the Makefiles necessary to build it. This was then committed to the MereIdea Main repository as Revision 2.

A Buildbot Slave was then created on the Linux box and errors in the build which occurred when it was started were fixed. The Windows Slave was then set up, which included installing all the necessary components, and errors resulting from this Slave were fixed.

There are still some outstanding issues to address in the next post:

• A proper testing framework needs to be set up and integrated into the configure/compile/test cycle.
• As the Windows Slave never returns from running, the environment under which the Slave is run should be used for running subsequent commands. This means if the Slave was run under a Visual Studio 2010 command prompt setvcvars may not be needed.
• The Windows Slave could be set up as a service so that it will be run in the background. This is likely to require the setvcvars step to remain.

All of these issues will be covered in the next post, which should complete the initial set up of the Buildbot system.

Code highlight key
Grey: Code unchanged
Red: Code removed
Green: Code added
Blue: Code not shown

→ Denotes the current line and the next are really one line

It’s been a while since a new post graced this blog. Unfortunately (for the blog, at least), other work has got in the way and it is now it is necessary to start a new, year long project producing some software and an accompanying website. To write this new software CMake is going to be used as the build system (actually, CMake is only a Makefile/Project file generator, so the build system will be whatever is native to the operating system – make and Visual Studio – and nmake - in this case).

CMake wasn’t chosen because it’s any better (or worse) than SCons, but because of familiarity with it and the need to create an actual project rather than experimenting with example code. This, sadly, means setting the SCons tests on one side for the moment.

It was hoped the new SCons 2.0 series could be tested using the example code and different configurations on different platforms, but this work is now on hold. It is recommended, however, that anyone considering SCons to try out the new 2.0 series.

When it is thought that some element of the new project is worth blogging about, then this blog will be updated, hopefully frequently. Updates are also available via the Twitter account.

One thing that was on the original path was the automated compile/test of the example code using Buildbot. Originally, this blog was to detail the integration of SCons with Buildbot, but now will describe the integration of CMake instead. This first post in the mini-series describes the setting up and configuration of the Buildbot Master.

Buildbot is made up of two main components. The Master which decides when things need to be built, and collates and displays the results via web pages; and the Slave which is an individual machine that performs the building of the code and sends the results of the build to the Master. There is one Master and one or more Slaves. The more Slaves on differently configured machines the better!

---

Installation

The Master is going to be set up on a machine running openSUSE x86_64 version 11.2. Buildbot is written in Python, so you need to install Python (usually included by the default SUSE installation) first. The Python version installed is 2.6.2. Twisted also needs to be installed for Buildbot‘s networking functionality. This was installed through YaST (SUSE‘s system admin client) which installed Twisted version 8.2.0. Python and Twisted are required in systems running either the Master or the Slave. There are some further pre-requisites for the machine running the Master.

sqlite3 and simplejson are included with the Python version installed, so no further action was needed for this. Jinja2, however, needed to be downloaded and installed. Actually, it can be downloaded and installed using the command:

> easy_install Jinja2

but instead the source was downloaded, decompressed and installed from the source tar. Once downloaded and decompressed, the installation can be done as root using the command:

> python setup.py install

in the extracted directory. This, however, caused a problem as the Python setuptools module was not installed. A derivative of setuptools (distribute) was installed using the commands:

> curl -O http://python-distribute.org/distribute_setup.py
> python distribute_setup.py

Then Jinja2 could be installed correctly using the command as above.

Now everything is ready to install Buildbot. The version of Buildbot used is 0.8.1. Both the Master and Slave tar.gz files were downloaded and decompressed. These were then installed by switching to each of the extracted directories and typing:

> python setup.py build
> python setup.py install

The installations can then be tested. Firstly to check the version numbers:

> buildbot --version
Buildbot version: 0.8.1
Twisted version: 8.2.0

> buildslave --version
Buildslave version: 0.8.1
Twisted version: 8.2.0

and then by running the tests:

> PYTHONPATH=. trial buildbot.test

Unfortunately, running the above test suite showed two errors. These errors were both of type ImportError relating to the mock module missing from Python. The mock module was installed using:

> easy_install mock

The tests could then be run without error for both the Master and Slave components:

> PYTHONPATH=. trial buildbot.test
> PYTHONPATH=. trial buildslave.test

---

Creating the Master

Finally the Master is ready to be set up.

Firstly a new user was created to act as the Master administrator. This was done through YaST, but could easily be done with useradd. This new user was then logged in. A new directory was created to house the Buildbot Master in the new user’s home directory:

> mkdir -p ~/Buildbot/master

This directory was then initialised as a new Master directory:

> buildbot create-master -r ~/Buildbot/master

Now it’s ready for the configuration.

---

Configuring the Master

The create-master option to the buildbot command creates a set of files in the specified directory. One of these files (master.cfg.sample) is the configuration file which needs to be edited. First, a copy of the sample file is made:

> cp master.cfg.sample master.cfg

This was then edited as follows (comments omitted):

c = BuildmasterConfig = {}

c['db_url'] = "sqlite:///state.sqlite"

from buildbot.buildslave import BuildSlave
c['slaves'] = [BuildSlave("bot1name", "bot1passwd")]
c['slaves'] = [
                BuildSlave("oSUSE_11.2_x86_64_gcc","password1", →
 max_builds=2),
                BuildSlave("WinXP_VS2010","password2",max_builds=2)
              ]
c['slavePortnum'] = 9989

This sets up the slaves on which the code will be built. More will be added to this list, but for now it will be tested with just these two. Obviously, the passwords need to be changed to something more secure! These passwords will be used later to set up the slaves. The max_builds=2 also tells the Master not to send more than two concurrent builds to the Slaves. The slavePortNum is the port on which the Master will listen for Slaves, and so must match the port used when setting up a Slave later.

Now the method for detecting changes to the source code can be specified. As Subversion will be used for this project, SVNPoller is used to check for updates to the code:

from buildbot.changes.pb import PBChangeSource
from buildbot.changes.svnpoller import SVNPoller
c['change_source'] = PBChangeSource()
source_svn_url='http://svn.mereidea.com/main/trunk'
svn_poller = SVNPoller( svnurl=source_svn_url,
                        pollinterval=3600, # seconds
                        histmax=10,
                        svnbin='/usr/local/bin/svn'
                      )
c['change_source'] = [ svn_poller ]

So the change_source is changed from PBChangeSource to the SVNPoller. The main repository is one that can be read by anyone, but a username and password can be specified to the SVNPoller if necessary with the svnuser and svnpasswd options.

Next the schedulers can be defined. These tell the Master when to call the Slaves to build the code. Two types of schedulers are required – a change scheduler (of type Scheduler) called whenever the repository changes, and a nightly scheduler (of type timed.Nightly) to rebuild the code every morning at 03:30. The build for change scheduler will simply update the code and run a build, the nightly scheduler will check out a clean copy of the code and build from scratch. The configuration file is changed as follows:

from buildbot.scheduler import Scheduler
from buildbot.schedulers import timed
c['schedulers'] = []
c['schedulers'].append(Scheduler(name="all", branch=None,
                                 treeStableTimer=2*60,
                                 builderNames=["buildbot-full"]))
c['schedulers'].append(Scheduler(name="change-build",branch=None,
                                 treeStableTimer=600,
                                 builderNames=[
                                        "Linux-change-build",
                                        "Win-change-build"
                                              ]
                                )
                       )
c['schedulers'].append(timed.Nightly(name="nightly-build",
                                     hour=3, minute=30,
                                     builderNames=[
                                            "Linux-nightly-build", 
                                            "Win-nightly-build"
                                                  ]
                                    )
                      )

This sets up the two required schedulers. The first (change-build) is triggered when the code is changed and detected with the SVNPoller set up earlier. Once triggered, it waits 10 minutes (treeStableTimer=600) to try to ensure all commits are complete before running the two builders – Linux-change-build and Win-change-build. The second (nightly-build) runs its builders – Linux-nightly-build and Win-nightly-build – at 3:30am (hour=3, minute=30).

This scheduler set up shows that four builders are going to be required – Linux-change-build, Linux-nightly-build, Win-change-build, and Win-nightly-build. The configuration is altered to include these builders as follows:

cvsroot = ":pserver:anonymous@cvs.sourceforge.net:/cvsroot/buildbot"
cvsmodule = "buildbot"
from buildbot.process import factory
from buildbot.steps.source import CVS
from buildbot.steps.shell import Compile
from buildbot.steps import source, shell
from buildbot.steps.python_twisted import Trial

svnbaseurl='http://svn.mereidea.com/main/'
svnup = source.SVN(mode='update',
                   baseURL=svnbaseurl,
                   defaultBranch='trunk')
svnco = source.SVN(mode='clobber',
                   baseURL=svnbaseurl,
                   defaultBranch='trunk')
setvcvars = shell.ShellCommand(command="setvcvars.bat")
confg = shell.Configure(command="cmake ../src")
wconfg = shell.Configure(command='cmake -G "NMake Makefiles" ../src')
comp = shell.Compile(command='make')
wcomp = shell.Compile(command='nmake')
rtest = shell.Test(command='run_tests')

f1 = factory.BuildFactory()
f1.addStep(CVS(cvsroot=cvsroot, cvsmodule=cvsmodule, login="" →
mode="copy")) 
f1.addStep(Compile(command=["python", "./setup.py", "build"])) 
f1.addStep(Trial(testChanges=True, testpath="."))

flchg = factory.BuildFactory()
flchg.addStep(svnup)
flchg.addStep(confg)
flchg.addStep(comp)
flchg.addStep(rtest)

fwchg = factory.BuildFactory()
fwchg.addStep(setvcvars)
fwchg.addStep(svnup)
fwchg.addStep(wconfg)
fwchg.addStep(wcomp)
fwchg.addStep(rtest)

flngt = factory.BuildFactory()
flngt.addStep(svnco)
flngt.addStep(confg)
flngt.addStep(comp)
flngt.addStep(rtest)

fwngt = factory.BuildFactory()
fwngt.addStep(setvcvars)
fwngt.addStep(svnco)
fwngt.addStep(wconfg)
fwngt.addStep(wcomp)
fwngt.addStep(rtest)

b1 = {'name': "buildbot-full",
        'slavename': "bot1name",
        'builddir': "full",
        'factory': f1      }

blchg = {'name'="Linux-change-build",
         'slavename': "oSUSE_11.2_x86_64_gcc",
         'builddir': "linus_change_build",
         'factory': flchg
        }

bwchg = {'name'="Win-change-build",
         'slavename': "WinXP_VS2010",
         'builddir': "win_change_build",
         'factory': fwchg
        }

blngt = {'name'="Linux-nightly-build",
         'slavename': "oSUSE_11.2_x86_64_gcc",
         'builddir': "linux_nightly_build",
         'factory': flngt
        }

bwngt = {'name'="Win-nightly-build",
         'slavename': "WinXP_VS2010",
         'builddir': "win_nightly_build",
         'factory': fwngt
        }

c['builders'] = [b1]
c['builders'] = [blchg, bwchg, blngt, bwngt]

The builders are set with the factories to execute a set of commands appropriate to the build and architecture. The commands in this post are really just place holders, as the correct commands to run this correctly on any machine can only be decided once a few tests runs have been tried. The commands may be correct, but it is unlikely! For example, the setvcvars step needs to be set up and checked for persistence for later commands (are all commands executed in the same shell environment?). Further slaves that match a builder’s commands can be added to the builder by changing slavename to slavenames and providing a list:

'slavenames': ['WinXP_VS2010', 'Win7_VS2008'],

The correct build commands will be worked out in the next blog post when Slaves will be set up, too.

The final bits of the master configuration are the status targets and the project identity. The status targets tell Buildbot where to display the status results of the builds. These can be as web pages, by email, by IRC, etc. For now the MereIdea Buildbot will just display as web pages, though later the emailer will probably be set. The defaults for this section of the configuration are, therefore, fine. The http_port of the WebStatus may be changed, but this must correspond to the port set in the project identity section.

The project identity section provides a description of the project and how the web display can be accessed from the outside world. Simply change to whatever you want:

c['projectName'] = "Buildbot"
c['projectName'] = "MereIdea Projects"
c['projectURL'] = "http://buildbot.sourceforge.net"
c['projectURL'] = "http://www.mereidea.com"
c['buildbotURL'] = "http://localhost:8010"
c['buildbotURL'] = "http://builds.mereidea.com"

These are pretty straightforward. If using the port number (8010) in the buildbotURL, this must match the http_port setting in the WebStatus of the status targets section of the configuration. In the above example, a redirect is set in the MereIdea Apache web server to convert http://builds.mereidea.com to http://www.mereidea.com:8010.

That’s the configuration of the Master complete. Now it can be checked for validity with:

> buildbot checkconfig master.cfg

This did produce a deprecation warning:

/usr/lib64/python2.6/site-packages/twisted/mail/smtp.py:10: →
DeprecationWarning: the MimeWriter module is deprecated; →
use  the email package instead
  import MimeWriter, tempfile, rfc822
Config file is good!

This warning comes from within the Twisted code, so it’s just going to be ignored (possibly updating Twisted would make it go away). Apart from that warning, Buildbot says the configuration is good! Finally, the Master can be started with the command (executed in the directory containing the master’s configuration):

buildbot start

Then it can be checked by using a web browser to look at the buildbotURL. For the configuration above this means browsing to http://builds.mereidea.com. The Buildbot welcome page should be displayed. Looking on the Builders page it can be seen that the four builders specified in the configuration file are set up, and that they are currently offline:

So it’s all working so far! However, when the SVNPoller tried to poll the svn server an error occurred:

SVNPoller failed [Failure instance: Traceback (failure with →
no frames): <class 'twisted.internet.utils. →
_UnexpectedErrorOutput'>: got stderr: "svn: →
Repository moved temporarily to →
'http://www.mereidea.com/svn/main/trunk'; →
please relocate\n"
    ]

It seems the redirection in the Apache server from http://svn.mereidea.com to http://www.mereidea.com/svn has caused problems with the SVNPoller. The problem was easily solved by changing the configuration file:

from buildbot.changes.svnpoller import SVNPoller
source_svn_url='http://svn.mereidea.com/main/trunk'
source_svn_url='http://www.mereidea.com/svn/main/trunk'
...
svnbaseurl='http://svn.mereidea.com/main/'
svnbaseurl='http://www.mereidea.com/svn/main/'

The svnbaseurl was also changed to reduce the chance of any further problems.

---

Summary

This post described setting up the Master side of the Buildbot automatic build/test cycle software. It involved installing the necessary Python components, including Twisted and Jinja2. Then the installation of Buildbot itself. A Buildbot user was created to administer and run the Master daemon. A directory (~/Buildbot/master) was created to house the Master server files and it was initialised using the builbot create-master command. This also created a master.cfg.sample file which was copied to master.cfg and edited as detailed in the post to create four builds – two each for two Slaves. These were Linux and Windows builds when the code base (SVN) changed, and Linux and Windows nightly builds. The Master was then started and checked by accessing the WebStatus through http://builds.mereidea.com.

The configuration of the Master is quite straightforward, though it seemed daunting when first looking at the documentation. This is because the documentation is written to cover all eventualities and give a thorough outline of all types of builders, schedulers, build factories, etc.. The documentation is good, but the easiest way to configure Buildbot (and the way used to devise the configuration for the MereIdea set up) is to work through the master.cfg file using the comments as a guide and the Buildbot documentation as a reference.

With the Master now running, the next step is to put some code in the SVN repository and set up the Slaves. A continuous (short Periodic build) will the be set up so that the commands to run on the Slave can be fine tuned until it builds correctly. First the Linux Slave will be set up, then the Windows Slave. This is the subject of the next post, coming soon!

Current Repository Revision: 38

Code highlight key
Grey: Code unchanged
Red: Code removed
Green: Code added
Blue: Code not shown

→ Denotes the current line and the next are really one line

On the 12th April 2010 Microsoft launched the first full (post Beta) version of their Visual Studio suite. This blog post covers testing the Mere Idea example code with SCons using Visual Studio 2010 Express. It tests the code with a system where 2010 is installed alongside 2008 and 2005, and a system where 2010 is installed alongside 2008 only. A system where 2010 is the only Visual Studio version installed will be tested in the next post, but the fix in this post should work fine.

Note: A summary of the fix in this post is given at the end. If you’re not interested in how this all applies to the organically grown example code and simply want to see how to get SCons working with MSVS 2010, then you could just read the summary to see if that helps.

A few weeks before the launch of the 2010 version of Visual Studio, SCons released version 1.3.0 which improved support MSVS 2008. This new version of SCons will also be installed on a new machine to see if it improves anything with regard to 2010. This, however, will be saved for the next post, as there were problems when version 1.3.0 was tried. Details of the problems are given at the end of this post.

Note: The correct version numbers for the Visual Studios 2005, 2008 and 2010 are 8.0, 9.0 and 10.0 respectively. These years and numbers are used interchangeably throughout this post. Also, when referring to Visual Studio 2010 Express in this post, it is referring to the C++ edition unless otherwise stated.

Before starting on Visual Studio it is worth mentioning that two commits were made to the example code since the last post. These were simple changes to set the Subversion eol-style property for the SConfig files. This property is normally set automatically due to settings in the Subversion config file. However, when adding the new SConfig files, the type was not added to the Subversion config file. The config file has now been updated, so future SConfig files should have the property correctly set to native. The two SConfig files were committed separately, so revisions 36 and 37 have this simple change. It’s not always easy to remember to keep all support files up-to-date!

Now back to Visual Studio 2010.

The first thing to try is to see whether the work done on the example code to get it to work properly with Visual Studio 2008 simply worked with 2010. The latest revision was checked out on a machine with three Visual Studios installed – 2005, 2008 and 2010. The Microsoft Visual Studio 2010 Command Prompt (from Start | Programs | Microsoft Visual Studio 2010 Express | Visual Studio Command Prompt (2010) ) was opened (using this command prompt ensures that the environment is set up properly for compiling with 2010 – it’s essentially the same as starting a normal command prompt and running the appropriate vcvars32.bat batch file), and  SCons was then run with the following result:

> scons
scons: Reading SConscript files ...
Checking for C++ header file string... no
ERROR: String header not present.
ERROR: Please ensure the Standard Library is installed and it
ERROR: contains <string>.

So there are problems simply checking for the header file string. To test for the string header, SCons compiles a small program that consists of:

#include "string"

To test that this works as a compilable program the same code was compiled without SCons:

>cl test_string.cxx /c
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version →
16.00.30319.01 for 80x86
Copyright (C) Microsoft Corporation.  All rights reserved.

test_10.cxx
C:\Program Files\Microsoft Visual Studio 10.0\VC\INCLUDE\ →
xlocale(323)  : warning
C4530: C++ exception handler used, but unwind semantics are →
not enabled. Specify
 /EHsc

The /c on the command is to ensure that no linking is attempted (there is no main(), so the linker would complain), and the warning about exception handlers comes from the C++ libraries. The SConscript script adds /EHsc to the compiler flags when used properly to avoid this warning. The successful compilation, however, shows that the compiler can compile the code simply by starting the 2010 command prompt and using cl. The problem must lie in the way it is handled in SCons.

A note on compiler versions in case of interest. The 9.0 (2008) and 10.0 (2010) version numbers refer only to the Visual Studio IDE. The same IDE version (with varied components) is used for all languages, so C++ and C# (for example) both use MSVS version 9.0 for 2008 and 10.0 for 2010. The language compilers have their own version numbers. The C++ compiler major version numbers are 15.00 in 2008 and 16.00 in 2010, whereas in C# the compiler major revisions are 3.5 in 2008 and 4.0 in 2010 (actually matching the .NET version number as they are developed together). The incremental linker major version numbers for C++ do match the Visual Studio numbers – 9.00 and 10.00 respectively.

The version of SCons installed on the test computer is old (version 1.0.1). This is so the example code can be tested with as many versions of SCons as possible to enable users with legacy systems to use the solutions laid out in this blog. As other computers are available for testing with newer versions of SCons, this post will look for a solution that will work with all installed SCons versions (including 1.0.1).

To check whether it is the version of SCons that is causing the problem Visual Studio 2010 Express was installed on a computer using SCons version 1.2.0. This computer had only MSVS 2008 already installed. SCons was run at the 2010 Command Prompt, and the same problem with the string configure test appeared.

A further version test was performed by temporarily installing the latest version of SCons (1.3.0) on the above computer (only temporarily, as 1.3.0 will be used later on a different computer, and 1.2.0 is needed on this one) and SCons was run again. Again the string configure test problem remains, showing that even the latest version of SCons has problems with multiple versions of Visual Studio installed (whether it has problems with just 2010 installed will be tested later).

Looking at the environment should give clues as to why the current code doesn’t work with the 2010 installation. The fix to make Visual Studio 2008 work with SCons was to use the 2008 Command Prompt (or run 2008′s vcvars32.bat) and copy the LIB and INCLUDE environment variables into the SCons environment. This code was put in the SConstruct file:

# If the compiler is msvs and the LIB and INCLUDE environment
# variables in the os exist, use them instead of SCons version
# to fix MSVS 2008 Express problems
if (cmplr=='cl') and (not glob_env['vs2005']):
  envlib = os.environ.get('LIB')
  envincl = os.environ.get('INCLUDE')
  if envlib is not None:
    glob_env.Append(ENV = {'LIB' : envlib})
  if envincl is not None:
    glob_env.Append(ENV = {'INCLUDE' : envincl})

As a quick recap from the earlier post, this code is necessary as SCons assumes the Microsoft SDK is below the Visual Studio directory, whereas from 2008 the SDK was in a separate directory. Hopefully this is fixed in SCons version 1.3.0.

So what happens when we start up the 2010 Command Prompt and run through this bit of code? The environment contains:

'ENV': { 'INCLUDE': 'C:\\Program Files (x86)\\Microsoft Visual →
Studio 10.0\\VC\\INCLUDE;C:\\Program Files (x86)\\Microsoft SDKs\\ →
Windows\\v7.0A\\include;',
'LIB': 'C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\ →
VC\\LIB;C:\\Program Files (x86)\\Microsoft SDKs\\Windows\\v7.0A\\ →
lib;',
'PATH': u'C:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\ →
Common7\\IDE;C:\\Program Files (x86)\\Microsoft Visual Studio →
9.0\\VC\\bin;C:\\Program Files (x86)\\Microsoft Visual Studio →
9.0\\Common7\\Tools;C:\\Program Files (x86)\\Microsoft Visual →
Studio 9.0\\Common7\\Tools\\bin;C:\\Windows\\Microsoft.NET\\ →
Framework\\v4.0.30319',
'PATHEXT': '.COM;.EXE;.BAT;.CMD',
'SYSTEMDRIVE': 'C:',
'SYSTEMROOT': 'C:\\Windows',
'SystemRoot': u'C:\\Windows',
'TEMP': 'C:\\Users\\chris\\AppData\\Local\\Temp',
'TMP': 'C:\\Users\\chris\\AppData\\Local\\Temp'},

You can see from this that the LIB and INCLUDE variables are correct, but the PATH variable still thinks it is using the Visual Studio 9.0 version. PATH could easily be added to the environment variables imported from the command prompt, but this includes lots of irrelevant paths, so it would be better to set this up correctly.

What compilers does the environment think are available? Again the environment dump can show this:

'MSVS': { 'FRAMEWORKDIR': u'C:\\Windows\\Microsoft.NET\\Framework\\',
'FRAMEWORKVERSION': u'v4.0.30319',
'FRAMEWORKVERSIONS': [ u'v4.0.30319',
u'v3.5',
u'v3.0',
u'v2.0.50727',
u'v1.1.4322',
u'v1.0.3705'],
'PROJECTSUFFIX': '.vcproj',
'SOLUTIONSUFFIX': '.sln',
'SUITE': 'EXPRESS',
'SUITES': ['EXPRESS'],
'VERSION': '9.0Exp',
'VERSIONS': ['9.0Exp', '10.0Exp']},

SCons seems to think the best version installed is 9.0Exp, but it recognises that 10.0Exp is available. Note that on the computer with three MSVS versions installed, 9.0Exp is still selected as the compiler:

'VERSION': '9.0Exp',
'VERSIONS': [ '9.0Exp',
'9.0',
'8.0Exp',
'8.0',
'3.5',
'10.0Exp',
'10.0'],

Notice, also, that this computer has versions with and without the ‘Exp’ on, even though only the Express editions are installed. This could be due to the old (1.0.1) version of SCons.

What would be good is if the Visual Studio version selected could be the one for which the Command Prompt was open. This would mean that opening a 2005 Command prompt would use 2005 (8.0), opening a 2008 Command Prompt would use 2008 (9.0), and opening a 2010 Command Prompt would use 2010 (10.0).

It would be nice if there was an MSVS_VER environment variable at the command prompt to help get the right version, but there isn’t. Instead something with the installation path in it can be used to get the version, as the installation path contains the version number in all MSVS versions (up to now). One environment variable that contains just the install path is DevEnvDir. This is set to the directory containing the development environment, and includes the string Microsoft Visual Studio x (where x is the version number) in its value. Given this we can write some code to work out what the correct version is for the open command prompt.

As the MSVS version is going to be selected based on the command prompt, the ‘vs2005‘ option is no longer needed (just open a 2005 Command Prompt), but an ‘msver‘ option is provided in case there is a problem with auto detecting the version number. The new code (all added into the SConstruct file) is:

...

opts = Options(optfile)
opts.AddOptions(
    ('bdir', "Set to the build directory if you don't like 'build'", →
'build'),
    ('sdir', "Set to the source directory if you don't like 'src'", →
'src'),
    ('smalllib',
 'Set to the number of object files below which a library is →
considered small',
     0),
    ('testname',
 "The name (or part of the path) of test(s) to run - sets →
'runtests=true'",
     ''),
    ('msver',
 "Set the MSVS version number. E.g. for 2008 use 'msver=9.0', useful →
if not auto detected",
     ''),
    BoolOption('debug','Set to apply debugging flags','false'),
    BoolOption('forgive',
        'If set the compiler will be more forgiving of non-standard →
code',
        'false'),
    BoolOption('msvs',
        "Set to create Visual Studio projects only (no build →
performed)",
        'false'),
    BoolOption('runtests',
        "Set to run the tests once built",
        'false'),
    BoolOption('mingw',
        "Set to use MinGW instead of the compiler SCons chooses",
        'false'),
    BoolOption('vs2005',
        "Set to use Visual Studio [Express] 2005 instead of a later →
compiler",
        'false')
)

glob_env = Environment(options = opts)

Help(opts.GenerateHelpText(glob_env))

# If vs2005 or mingw is set, get an environment with
# the appropriate tool instead
if glob_env['vs2005']:
 glob_env = Environment(options = opts,
 MSVS_VERSION = '8.0')
elif glob_env['mingw']:
 glob_env = Environment(options = opts,
 tools = ['mingw'])

# Set up the correct path separator
# If your platform uses a different separator, or
# uses back slash instead of forward slash, ammend this
# bit
plat = string.lower(glob_env['PLATFORM'])
if plat == "win32":
  glob_env.Replace(pathsep = '\\')
else:
  glob_env.Replace(pathsep = '/')

# Add the src directory to the CPPPATH so we can use
# #include <libname/subdir/class.h>
glob_env.Append(CPPPATH='#%s' % glob_env['sdir'])

# Set up specific compiler options
# If using msvs, set the correct version
#  dbgf will hold any debug flags
#  unff will hold any 'unforgiving' flags
#  genf will hold any general flags
cmplr = string.lower(glob_env['CC'])

# If the compiler is msvs and the LIB and INCLUDE environment
# variables in the os exist, use them instead of SCons version
# to fix MSVS 2008/2010 Express problems
if (cmplr=='cl') and (not glob_env['vs2005']):
  if (glob_env['msver']!=''):
    mver = glob_env['msver']
  else:
    # Select the right MSVS version number
    envdir = os.environ.get('DevEnvDir')
    chkstr = 'Microsoft Visual Studio'
    pos = envdir.find(chkstr)
    if pos == -1:
      # Warn we might not have the right compiler setup as we can't →
dicover the
      # correct version from the directory name
      print "WARNING: Unable to determine Visual Studio version - →
ensure you use"
      print "WARNING: the appropriate command prompt for your VS →
Version or use"
      print "WARNING: the msver option to set the version number."
      print "WARNING: Assuming default environment."
      mver=""
    else:
      # Try to get the version number
      pos=pos+len(chkstr)+1
      epos=envdir.find('\\',pos)
      mver = envdir[pos:epos]

  if (mver != ""):
    glob_env = Environment(options = opts,
                             MSVS_VERSION = mver)

  # Now set the LIB and INCLUDE variables
  envlib = os.environ.get('LIB')
  envincl = os.environ.get('INCLUDE')
  if envlib is not None:
    glob_env.Append(ENV = {'LIB' : envlib})
  if envincl is not None:
    glob_env.Append(ENV = {'INCLUDE' : envincl})

# Set up the correct path separator
# If your platform uses a different separator, or
# uses back slash instead of forward slash, ammend this
# bit
plat = string.lower(glob_env['PLATFORM'])
if plat == "win32":
  glob_env.Replace(pathsep = '\\')
else:
  glob_env.Replace(pathsep = '/')

# Add the src directory to the CPPPATH so we can use
# #include <libname/subdir/class.h>
glob_env.Append(CPPPATH='#%s' % glob_env['sdir'])
...

The first thing in the code is to add the msver option to enable the user to set the compiler version if it isn’t auto detected. This should be entered as, for example, msver=9.0 as an option to the scons command. The default is set to an empty string. The vs2005 option is removed, as is the code to set up an environment specifically for MSVS 2005 (version 8.0).

A block of code that creates some environment settings is then removed. As the new code  to select the correct compiler creates a new environment it would destroy the work this block of code does. The code is actually added back at the end after the compiler selection.

The check for the vs2005 option not being set is removed, and then a block of code is added that discovers the Visual Studio version. If the msver option is set, then that is used for the new environment. This just uses whatever the user enters – it could be checked to see if it’s a valid version number, but if something invalid is entered SCons just selects a compiler from the VERSIONS list. If msver is not set, the SConstruct script looks for the string Microsoft Visual Studio in the DevEnvDir command prompt environment variable. If the string isn’t found then a warning message is displayed and SCons is left to decide on the compiler, otherwise the version number is extracted from the DevEnvDir string.

If the version number is set to something, then a new SCons environment is created using that version number. This sets the compiler details correctly except for the LIB and INCLUDE paths, which still incorrectly assume the location of the Microsoft SDK to be under the Visual Studio directory. Therefore, the code to set them based on the command prompt environment is still included.

SCons can now be rerun with this new SConstruct file. Checking the SCons environment created it can be seen that the LIB, INCLUDE and PATH variables are now correct:

'ENV': { 'INCLUDE': 'C:\\Program Files\\Microsoft Visual Studio →
10.0\\VC\\INCLUDE;C:\\Program Files\\Microsoft SDKs\\Windows\\v7.0A →
\\include;',
'LIB': 'C:\\Program Files\\Microsoft Visual Studio 10.0\\VC\\LIB; →
C:\\Program Files\\Microsoft SDKs\\Windows\\v7.0A\\lib;',
'PATH': u'C:\\Program Files\\Microsoft Visual Studio 10.0\\Common7 →
\\IDE;C:\\Program Files\\Microsoft Visual Studio 10.0\\VC\\bin; →
C:\\Program Files\\Microsoft Visual Studio 10.0\\Common7\\Tools; →
C:\\Program Files\\Microsoft Visual Studio 10.0\\Common7\\Tools →
\\bin;C:\\WINDOWS\\Microsoft.NET\\Framework\\v4.0.30319',
'PATHEXT': '.COM;.EXE;.BAT;.CMD',
'SYSTEMDRIVE': 'C:',
'SYSTEMROOT': 'C:\\WINDOWS',
'SystemRoot': u'C:\\WINDOWS',
'TEMP': 'C:\\DOCUME~1\\Chris\\LOCALS~1\\Temp',
'TMP': 'C:\\DOCUME~1\\Chris\\LOCALS~1\\Temp'},

As is the selected compiler version:

'MSVS': { 'FRAMEWORKDIR': u'C:\\WINDOWS\\Microsoft.NET\\Framework\\',
'FRAMEWORKVERSION': u'v4.0.30319',
'FRAMEWORKVERSIONS': [ u'v4.0.30319',
 u'v3.5',
 u'v3.0',
 u'v2.0.50727',
 u'v1.1.4322',
 u'v1.0.3705'],
'PROJECTSUFFIX': '.vcproj',
'SOLUTIONSUFFIX': '.sln',
'SUITE': 'EXPRESS',
'SUITES': ['EXPRESS'],
'VCINSTALLDIR': u'C:\\Program Files\\Microsoft Visual Studio 10.0 →
\\VC\\',
'VERSION': '10.0',
'VERSIONS': [ '9.0Exp',
 '9.0',
 '8.0Exp',
 '8.0',
 '3.5',
 '10.0Exp',
 '10.0'],
'VSINSTALLDIR': u'C:\\Program Files\\Microsoft Visual Studio 10.0\\'},

And the code now builds using SCons 1.0.1 and 1.2.0. OK, so it gives hundreds of warnings from the MSVS 2010 includes (about unreferenced inline functions being removed), but this will be fixed in the next post!

---

Simple summary of fix

For people who aren’t interested in using the example code, but are just looking for a quick way to fix problems with Visual Studio 2010, this is the section for you! This solution uses SCons versions 1.2.0 or earlier, as some problems were found with version 1.3.0, the fixes for which will be detailed in a later post.

This section will use simple SConscript file containing:

Program('hello.c')

The file hello.c contains:

#include <stdio.h>
int main()
{
  printf("Hello World");
  return 0;
}

This should be compilable simply by typing scons on the command prompt (note that you should open an MSVS 2010 Command Prompt for this), but doing so produces an error:

> scons -Q
cl /Fohello.obj /c hello.c /nologo
hello.c
link /nologo /OUT:hello.exe hello.obj
LINK : fatal error LNK1104: cannot open file 'kernel32.lib'
scons: *** [hello.exe] Error 1104

Looking at the environment, it can seen that (on the test system that also has MSVS 2008 installed) the compiler is assuming version 9.0Exp (MSVS 2008), but that version 10.0Exp is available:

'MSVS': { 'FRAMEWORKDIR': u'C:\\Windows\\Microsoft.NET\\Framework\\',
 'FRAMEWORKVERSION': u'v4.0.30319',
...
 'VERSION': '9.0Exp',
 'VERSIONS': ['9.0Exp', '10.0Exp']},

This can easily be changed by changing the SConscript file to explicitly use version 10.0:

env=Environment(MSVS_VERSION = '10.0')
env.Program('hello.c')

Compiling with this SConscript file still produces the missing kernel32.lib problem. This problem is due to the fact that SCons (at least the versions 1.2.0 and earlier) assumes that the Microsoft SDK is in a directory below the Visual Studio installation directory, and the LIB and INCLUDE SCons environment variables are set accordingly. A solution to this is to get the correct LIB and INCLUDE directories from the command prompt. If you used a Visual Studio 2010 Command Prompt as suggested, or started a standard command prompt and ran vcvars32.bat, which is contained in the Visual Studio 2010 installation directory, then LIB and INCLUDE will be set up correctly. These can the be set in our SCons environment with the following change to the SConscript file:

import os

env = Environment(MSVS_VERSION = '10.0')

# Set the LIB and INCLUDE variables
envlib = os.environ.get('LIB')
envincl = os.environ.get('INCLUDE')

# If they are set in the command environment,
# set them in the SCons environment
if envlib is not None:
 env.Append(ENV = {'LIB' : envlib})
if envincl is not None:
 env.Append(ENV = {'INCLUDE' : envincl})

env.Program('hello.c')

Now we can run SCons again at the command line and…:

> scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
cl /Fohello.obj /c hello.c /nologo
hello.c
link /nologo /OUT:hello.exe hello.obj
scons: done building targets.

> hello
Hello World

…hey presto! It works.

If you don’t have multiple versions of Visual Studio installed, you can just start with an empty environment by replacing env.Environment( MSVS_VERSION = ’10.0′) with env.Environment(), but there is no harm in explicity stating the version to use.

Note: If you skipped it, the full version of the solution in this post actually gets the MSVS version to use automatically from the command prompt you opened, so opening a 2008 Command Prompt will use version 9.0, and opening a 2010 Command Prompt will use version 10.0.

---

SCons 1.3.0 (and 1.3.0.d20100404)

This section details problems encountered when trying to use SCons version 1.3.0 with even the simplest of SConscript files (as above). The SConscript file has just Program(‘hello.c’) in it, and hello.c was just Hello World. Running SCons gave the following error:

> scons -version
SCons by Steven Knight et al.:
 engine: v1.3.0.r4720, 2010/03/24 03:14:11, by jars on jars-desktop
Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, →
2010 The SCons Foundation
> scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
cl /Fohello.obj /c hello.c /nologo
'cl' is not recognized as an internal or external command,
operable program or batch file.
scons: *** [hello.obj] Error 1
scons: building terminated because of errors.

This is a step backwards, as now even the compiler can’t be located. At least before it only failed to link. Looking at the default environment:

'ENV': { 'COMSPEC': 'C:\\Windows\\system32\\cmd.exe',
 'PATH': u'C:\\Windows\\System32',
 'PATHEXT': '.COM;.EXE;.BAT;.CMD',
 'SystemDrive': 'C:',
 'SystemRoot': 'C:\\Windows',
 'TEMP': 'C:\\Users\\chris\\AppData\\Local\\Temp',
 'TMP': 'C:\\Users\\chris\\AppData\\Local\\Temp'},
...
'MSVC_VERSION': '10.0',
'MSVS': { 'PROJECTSUFFIX': '.vcproj', 'SOLUTIONSUFFIX': '.sln'},
...
'MSVS_VERSION': '10.0',

The environment (ENV) no longer has the LIB and INCLUDE variables, and the PATH has no reference to the MSVS paths. A new variable MSVC_VERSION is added which actually holds the Visual Studio version number (10.0) where you may expect the actual C++ version (16.00). At least the MSVS_VERSION is detected as 10.0!

As this version of SCons only claimed to fix Visual Studio 2008 problems, it was tested by setting the version number to 9.0 using env=Environment(MSVS_VERSION=’9.0) and env.Program(‘hello.c’) in the SConscript file and using a Visual Studio 2008 Command Prompt. The (slightly cut down) result:

scons: Reading SConscript files ...
scons: warning: MSVS_VERSION is deprecated: please use MSVC_VERSION →
instead
scons: warning: VC version 9.0 not installed.  C/C++ compilers are →
most likely not set correctly.
 Installed versions are: ['10.0', '9.0Exp']
scons: done reading SConscript files.
scons: Building targets ...
cl /Fohello.obj /c hello.c /nologo
'cl' is not recognized as an internal or external command,
operable program or batch file.
scons: building terminated because of errors.

This is strange for many reasons.In SCons version 1.2.0 specifying 10.0 matched with 10.0Exp, but in 1.3.0 specifying 9.0 doesn’t match with 9.0Exp. Only MSVS 10 Express is installed, but is shown as 10.0 rather than 10.0Exp. It appears MSVS_VERSION is deprecated in favour of MSVC_VERSION, but still expects the Visual Studio version number rather than the C++ version number, and it still can’t find the compiler!

Changing the environment to use 9.0Exp gives the following:

ValueError: Unrecognized version 9.0Exp:
 File "SConstruct", line 3:
 env = Environment(MSVC_VERSION = '9.0Exp')

So 9.0 isn’t a valid version because it’s not in the list, but 9.0Exp is an unrecognised version and so can’t be used either!

The SCons 1.3.0 investigation will have to wait for the next post. When trying 1.3.0.d20100404 things got even worse, reporting NameError: global name ‘msvc_version_numeric’ is not defined with even the simple one line SConscript!

The conclusion is to stick to SCons 1.2.0 and to use the fix detailed in this post to work with Visual Studio. It works with all versions so far!

---

The changes detailed in this post have been added to the example code repository. The current repository revision is 38.

Revision 36 and 37 updated the Subversion end-of-line property to set it to native for SConfig files.

Revision 38 updated the SConscript file to automatically detect the Visual Studio version number from the open command prompt. It also enables the example code to work with MSVS 2010.

The next post will have a look at SCons version 1.3.0 to see if the problems can be fixed for the simple code and then the example code. It will also look at the warnings thrown by the MSVS 2010 header files to fix (or possibly ignore) them.

The promised SCons configure check post will come after that. Getting SCons to work with MSVS 2010 is taking a little more work than was hoped!

Current Repository Revision: 35

Code highlight key
Grey: Code unchanged
Red: Code removed
Green: Code added
Blue: Code not shown

The first SCons configure style checks were added in the previous post. These simply checked for the existence of header files. This blog entry will add to the header checks by adding checks for functions.

As well as the addition of the configure checks, code will be added to handle the players, keeping track of who’s turn it is to go next and who should start the next game.

To enable any game which uses squares with three possible states to use the Board class (e.g. Reversi, Draughts (Checkers), etc. or even ones using two states (filled/empty) such as solitaire) the Board class can be changed to use any number of squares. As the size of the Board isn’t known before hand, an std::vector will be used.

The choice to use a std::vector is a simple one. To do it without STL containers would require pointers (managed by the MereIdea code) to allocate the appropriate amount of memory. This management is simple, but it’s always better not to do it if there’s an alternative as it avoids memory errors that can be hard to debug.

When using an STL container, std::vector should always be considered first, as it is the most efficient in terms of space and speed of access (it is essentially a management wrapper around a C-style array). It is perfect for this purpose, so there is no need to look further.

The code of the Board class is changed to use the std::vector in place of the nine element array:

/*!
   \file board.h

   \brief Maintain a board for a game of noughts and crosses
   \brief Maintain a board for a game using squares and up to →
three states

   \clog
...

#ifndef MI_TICTAC_CORE_BOARD_H_
#define MI_TICTAC_CORE_BOARD_H_

#include <vector>

#include <tictac/core/piece.h>
...
    //! Construct an empty board
    /*!
        \param n_squares The number of squares on the board. These →
will be
                         numbered 1 to n_squares.
        \post The board will be initialised with nine n_squares →
undefined squares
    */
    Board(void);
    Board(const unsigned int n_squares);
...
    void setSquare(const unsigned int num,
                   const Piece pce);

    //! Clear a single square on the board
    /*!
        \param num The number of the square to set
        \pre num contains a number in the range 1-n inclusive →
(where n is
                 the number of squares specified on contruction).
        \exception SquareOutOfRange thrown if num is not in the →
range 1-n
        \sa content(), setSquare()
    */
    void clearSquare(const unsigned int num);

    //! Clear the board so that all squares are undefined
...
  private:
    //! The squares for the noughts and crosses board
    Square board_m[9];
    std::vector<Square> board_m;
  };
}

The changes to the Board class are straightforward. The constructor has an argument added for the number of squares needed for the board. There is no need for a default constructor as the board always has to have some size, and there is no sensible default size. There is no need to restrict the board to square numbers, either, as it isn’t known how the Board will be used.

The second change is the addition of the clearSquare() method. This is now necessary as a use of the Board could include the removal of pieces.

Finally, the actual internal data type for the Board is changed from and array of Square to a std::vector of Square.

These changes don’t require much of a change in the board.cxx file (as both the array and std::vector are accessed with the square brackets operator), so the code isn’t shown here. The board_test.cxx was also altered to correct for the changes and add new tests for the clearSquare() method.

As the Board has been changed to have board sizes other than 3×3, it can be used for more than Noughts and Crosses (or tic-tac-toe). Therefore, it no longer makes sense to call the library tictac. The name was, therefore, changed to bdgme, for board game. This required changing all the namespaces from mi_tictac to mi_bdgme, changing the TictacException to BdGmeException, changing any comments containing tictac to bdgme, and changing the SConscript file in mi_test_scons/src to include the directory bdgme instead of tictac.

Finally, svn is used to move the tictac directory to bdgme:

svn mv tictac bdgme

Directory name changes should be done using svn otherwise the repository and working copies become out of sync and it is possible to end up with both tictac and bdgme directories active in the latest revision if you’re not careful!

The next step was to add a Player class. This class simply holds the name of a player and the Piece they are using. The class is declared as follows:

class Player
{
public:
  Player(const std::string &name = "anonymous",
         const Piece pce = "UNDEFINED_PIECE");
  ~Player(void);

  void setName(const std::string &name);
  void setPiece(const Piece pce);

  const std::string &name(void) const;
  const Piece &piece(void) const; 

private:
  std::string name_m;
  Piece piece_m;
};

The definition of the Player class is straightforward, and so is not shown. Some tests were also added in the file src/bdgme/core/tests/player_test.cxx.

To handle the number of players in the game, who’s turn it is next, and who’s turn it is to start the next game, a Players class is created. This is declared as follows:

class Players
{
public:
  Players(const Player &p);
  Players(const std::vector<Player> &vp);
  ~Players(void);

  void addPlayer(const Player &p);

  const Player &startPlay(bool rnd=false);
  const Player &currentPlayer(void) const;
  const Player &nextPlayer(void);

  unsigned int nPlayers(void);

private:
  std::vector<Player> players_m;
  unsigned int current_player_m;
  unsigned int next_start_m;
};

There is no default constructor for the Players class as we always need at least one player in any game! The player added through the single Player constructor becomes the first player, and other players remain in the order they are added through the addPlayer() method. By default the startPlay() method starts the next game from the next player in the std::vector. This behaviour can be changed by specifying true as the rnd argument of the startPlay() method. The player to start is then picked randomly from the number of players in the class. This could mean the same player being picked a number of times in a row. The standard C built-in rand() function is used. As their names suggest, currentPlayer() returns the current player taking their turn, nextPlayer() moves the current_player_m index by one (to point at the next Player in the std::vector) and returns that player, and nPlayers() returns the number of players held by the class. A set of tests were added in the file src/bdgme/core/players_test.cxx.

The definition of the constructors and the startPlay() functions are as follows:

...
mi_bdgme::Players::Players(const Player &p) :
current_player_m(0),
next_start_m(0)
{
  players_m.push_back(p);

  // Seed the random number generator based on time
  srand((unsigned int)time(0));
}
...
mi_bdgme::Players::Players(const std::vector<Player> &vp) :
players_m(vp),
current_player_m(0),
next_start_m(0)
{
  // Seed the random number generator based on time
  srand((unsigned int)time(0));
}
...
const mi_bdgme::Player &mi_bdgme::Players::startPlay(bool rnd
                                                     /*=false*/)
{
  unsigned int st;
  if (rnd)
  {
    st = rand() % players_m.size();
    next_start_m=st+1;
  }
  else
  {
    st = next_start_m++;
  }

  next_start_m = (next_start_m==players_m.size()) ? 0 : next_start_m;
  current_player_m = st;

  return players_m[st];
}
...

The adjusted Board and new Players classes include three new standard headers – vector, cstdlib and ctime. SCons configure style checks were added for these headers in src/bdgme/core/SConfig:

...
if not conf.CheckCXXHeader('vector'):
  print "ERROR: Vector header not present."
  print "ERROR: Please ensure the Standard Library is installed and it"
  print "ERROR: contains <vector>."
  Exit(1)

# Check for C library headers
if not conf.CheckCXXHeader('cstdlib'):
  print "ERROR: The C standard library header not present."
  print "ERROR: Please ensure your C library is properly installed and"
  print "ERROR: contains <cstdlib>."
  Exit(1)

if not conf.CheckCXXHeader('ctime'):
  print "ERROR: The C time header not present."
  print "ERROR: Please ensure your C library is properly installed and"
  print "ERROR: contains <ctime>."
  Exit(1)
...

The Players class also includes three standard functions – srand, time and rand. Configure checks can be added to the SConfig file for the existence of these functions using the CheckFunc() SCons function:

...
# Check for functions
if not conf.CheckFunc('srand'):
  print "ERROR: Function srand is not present."
  print "ERROR: Please ensure your C library is properly installed and"
  print "ERROR: contains srand()."
  Exit(1)

if not conf.CheckFunc('time'):
  print "ERROR: Function time is not present."
  print "ERROR: Please ensure your C library is properly installed and"
  print "ERROR: contains time()."
  Exit(1)

if not conf.CheckFunc('rand'):
  print "ERROR: Function rand is not present."
  print "ERROR: Please ensure your C library is properly installed and"
  print "ERROR: contains rand()."
  Exit(1)

# Finish up the configure environment
loc_env = conf.Finish()

Now running SCons without the -Q switch will show the new set of checks:

[mi_test_scons]$ scons
scons: Reading SConscript files ...
Checking for C++ header file string... yes
Checking for C++ header file exception... yes
Checking for C++ header file sstream... yes
Checking for C++ header file vector... yes
Checking for C++ header file cstdlib... yes
Checking for C++ header file ctime... yes
Checking for C function srand()... yes
Checking for C function time()... yes
Checking for C function rand()... yes
Checking for C++ header file iostream... yes
Checking for C++ header file string... yes
Checking for C++ header file vector... yes
Checking for C++ header file cstdlib... yes
Checking for C function srand()... yes
scons: done reading SConscript files.
scons: Building targets ...

There are repeated checks in this list. As SCons caches checks, it is hoped that these are not repeated (i.e. recompiled).

They exist because I like to have the checks local to the code (in this case in core and in core/tests). This is to make it easier to keep track of which checks are necessary. This is especially true in the case where, for example, a particular header file is removed from all files within a particular directory. The header check can be removed in the local SConfig file, but if other directories contain files including that header the necessary checks will remain. Having a global file would require searching of all directories to check if the header check is needed.

---

The changes detailed in this post have been added to the example code repository. The current revision is 35.

Revision 33 included the changes to the Board class to enable any number of squares to be specified. It also changed the library name from tictac to bdgme.

Revision 34 added the Player and Players classes. It also changed the TictacException class to BdGmeException. This revision also added the extra header checks and the function checks detailed in this post.

Revision 35 Simply added a newline to the end of the Players class and added a check for the time function missed in the earlier revision.

So far only simple function checks have been added. In the next post a further function check will be created that will fail to find the function, and will use a replacement function which is locally written. Also some custom checks will be added.

Current Repository Revision: 32

Code highlight key
Grey: Code unchanged
Red: Code removed
Green: Code added
Blue: Code not shown

In the last post a new library (tictac) was added to the SCons test example code. Now it’s time to put some code into that library, and check that the SCons setup can handle the two libraries. The first configure style checks in the code  – checks for C++ headers – will also be added.

Firstly, a walk through the new code. The tictac library is to implement a set of classes and functions to support a game of Noughts and Crosses (or TicTacToe) for two players (one of which could be the computer, but, for the purposes of this test code, won’t be). All of the code added so far is in the mi_test_scons/src/tictac/core and its relevant tests directories.

We start by creating something to represent the contents of a square (or the Piece). This can be a nought, a cross or empty (undefined). The Piece is simply declared as an enumerated type as follows:

enum Piece
{
  NOUGHT=0,
  UNDEFINED_PIECE,
  CROSS
};

The enumerations are defined in what may seem an unusual order. This is so that bitwise operations can be performed to set/clear a defined/undefined bit, or to set the piece type. This enables a Square class to be created that acts as a kind of tri-state variable (actually it’s more of a quad-state variable as it could hold defined-cross, defined-nought, undefined-cross and undefined-nought) to represent each square on the Board.

The Square class is simply a collection of setters and getters for the contents of a single square. The content is stored as an unsigned char, which is manipulated inside the setters and getters to set it to any of the four states as mentioned above. An unsigned char is used as it is the smallest type, and only two bits are actually needed to store the Piece. The Piece type itself isn’t used because an enumerated type is (usually) the same size as an int (though I had a little bit of a mad moment when writing this and forgot why I hadn’t used the enumerated type – see revision 31, and it’s reverse merge (revision 32) in the example code ).

A bitfield could have been used with just two bits in it, but this would have to be aligned to the size of a char anyway, and its usually better to go for the simpler option. A further option would have been to simply use two bools, but this would be at least double the size of the unsigned char type. This may all seem like a small saving, but if a library client wanted to create a board with millions of Squares, or if someone wants to include it in a small embedded system, it’ll turn out to be a huge saving! These things should always be considered. One of the test systems only has 64MB RAM. This helps keep things memory efficient!

The Square has two setter methods (set with a Piece which changes the content to defined with that Piece, and set without a piece that simply changes the content to defined), one clear method (to set the content as undefined) and a content method that returns the current Piece. These use bit logic to change the content of the square.

The content of the Square should be one of the following values:

Binary Value	Meaning
00	UNDEFINED_PIECE/NOUGHT
01	Defined/NOUGHT
10	UNDEFINED_PIECE/CROSS
11	Defined/CROSS

These values and the order we originally put the Piece enumerated type in makes the set with a Piece method easy to write:

void mi_ticac::Square::set(const Piece pce)
{
  content_m = static_cast<unsigned char>(pce)+1;
}

By adding one to the enumerated type, the content is set to the correct value:

Piece	Binary value	One added (binary)	Final Meaning
NOUGHT	00	01	Defined/NOUGHT
CROSS	10	11	Defined/CROSS
UNDEFINED_PIECE	01	10	UNDEFINED_PIECE/CROSS

Similarly, the set method without a Piece can be easily written. All that is needed in this method is ensure the defined/undefined (or least significant) bit is set to 1:

void mi_tictac::Square::set(void)
{
  content_m |= 1;
}

Using the OR operation, the least significant bit (LSB) is always set to 1 (? | 1 = 1) and the rest of the bits in the unsigned char are left as they are ( ? | 0 = ?).

Finally for the setters, we want to clear (set to 0) the defined/undefined bit. This is done in the clear method:

void mi_tictac::Square::clear(void)
{
  content_m &=2;
}

Using the AND operation with 2 (binary 10) will always clear the LSB (? & 0 = 0), and leave bit 1 (the nought/cross bit) as whatever it was (? & 1 = ?).

The last method in the Square is the getter – returning the Piece that the Square represents. This code isn’t just a case of returning the enumerated type, because it has to convert from the internal unsigned char type to the enumerated Piece type as an enumerated type cannot be set from another type. This makes sense because an enumerated type, by definition, is only permitted to hold certain values, and enabling it to be set from another type may break that rule – something which wouldn’t be known until runtime.

The code to convert is, again, pretty simple, and again it uses logical operators:

mi_tictac::Piece mi_tictac::Square::content(void) const
{
  Piece cont = UNDEFINED_PIECE;

  if (content_m & static_cast<unsigned char>(UNDEFINED_PIECE))
  {
    if (content_m & static_cast<unsigned char>(CROSS))
      cont = CROSS;
    else
      cont = NOUGHT;
  }
  return cont;
}

The logic is straightforward given what you already know from this post even if you didn’t know much about bitwise logic before!

The next class in the core library is the collection of Squares – the Board. The Board class simply holds an array of nine Squares. An array is used rather than, for example, an STL container as the number of Squares required is known (though this may change in a later version!), and the array is the simplest, most-efficient way to represent the board. Board has three methods – one to set a single Square, one to get the contents of a single Square, and one to clear all squares (set them back to UNDEFINED_PIECE).

The methods setSquare and contents take an unsigned short to identify the square whose contents are to be set/returned. This number should be in the range 1-9 (this is better for humans than 0-8, even if it means the code having to subtract one for the array index). If the number is outside the range, a SquareOutOfRange exception is thrown.

An exception is used here because the way the code is to be called by clients of the library is unknown. With an exception, they can deal with it at the level of the call, turn it into an error code, allow it to pass to another level, or simply ignore it and let the program crash. In other places a return code may be used, but when something can be called from outside the library it seems a good case for an exception. This is a choice that should be made based on the situation, and always using exceptions, or always using return codes means the programmer is not thinking about what is best in a particular situation. Never think that one-way is the only way. There is no programmer’s panacea!

A test was added for each class (in src/titac/core/tests) and the code compiled and tested on Linux and Windows. As the initial coding was done under Linux, there were some problems when building with MSVS.

Before moving on to the MSVS problems, a change was made to the way tests could be specified at the SCons command line. A new command line option, testname was added to enable the user to specify the name (or part of a name or path) of the test to run. This required a change to the SConstruct file:

...
    ('smalllib',
'Set to the number of object files below which a library →
small'.
     0),
    ('testname',
"The name (or part of the path) of test(s) to run - sets →
'runtests=true'",
     ''),
    BoolOption('debug','Set to apply debug flags',false),
...

A change was also necessary to the mi_scons.py file in the MI_BuildTestsAndExecute function to check for the test name in the test being run. It makes use of the fact the the Python string function find matches everything if it is trying to match an empty string:

...
trgt = env.Program(nxttst)

# Now execute
if [env['runtests'] or env['testname'] != '':
  cmnd = env['currbdir']+env['pathsep']+str(trgt[0])
  dmname = 'dummy_'+str(trgt[0])
  if cmnd.find(env['testname']) != -1:
  ··env.Command(dmname,trgt,cmnd)
...

So, for example, you can make the following calls to SCons to run specific tests:

# Run all tests
scons -Q runtests=true

# Run any test with 'board' in its name
scons -Q testname="board"

# Run all tests in the 'core' directory or anything else with
# 'core' in it's name
scons -Q testname="core"

Now to the MSVS problems. Firstly, there was a warning about structured exceptions (C4571). This is a warning that, when compiling with /EHsc structured exceptions will not be caught by catch(…). This warning has now been disabled in the SConstruct file:

#  /wd4571 - Disable this warning about not catching structured
             exceptions in catch(...) block due to many warnings
             for every catch(...) block.
#  /FC     - Display the full path of source files containing
...
genf = ' /Wall /EHsc /wd4820 /wd4710 /wd4571 /FC'
unff = ' /Za'

Another set of warnings, which have been appearing since this blog began and had begun to get too many to ignore, regarded argc and argv being unreferenced variables when compiling with MSVS. All argc and argv parameters were removed from main functions to correct this warning.

NOTE: The above paragraph may suggest that compiler warnings are occasionally ignored. You should never ignore compiler warnings as they often point to real problems. The next MSVS warning is a prime example!

MSVS also warned that the virtual function TictacException::what did not override a base class virtual member function (warning C4263). The reason for this was a missing const on the end of the function. This is an example of where a warning can point to a potential error, and also where using multiple compilers can help catch more problems than a single compiler (gcc gave no warning about this).

The final few warnings from MSVS were about type conversions. In the tests, unsigned int had been used for loop variables which call into the content function which takes an unsigned short. All loops were changed to unsigned short. A further conversion warning was given for the line that sets the square content by adding one to the Piece (Square::set). As the +1 was outside the cast, there was a warning that an unsigned char was being set from an int. This was easily solved:

void mi_tictac::Square::set(const Piece pce)
{
  // One is added to the value so that the value is set to →
defined (or in the case
  // of undefined, it will clear the defined bit).
  content_m = static_cast<unsigned char>(pce)+1;
  content_m = static_cast<unsigned char>(pce+1);
}

Once the MSVS warnings were fixed the code was committed, and rebuilt under gcc. This added a new warning about the exception specification being different between the derived TictacException::what() and the base std::exception::what(). This is simply resolved by adding throw() to the end of the what() method. Exception specifications won’t be used in the code normally, but are added here simply to clear the warning. Normally a list of exceptions thrown are put in the Doxygen documentation.

Now the code compiled cleanly with both gcc and MSVS, and we can move on to adding the first configure style checks to the code.

As the SConscript files at each level are functionally the same for each library, it would be better not to have the configure style checks just in the SConscript files that needed them, and missing from the others. To avoid this inconsistency, a new type of file was created – SConfig. A SConfig file is created only in parts of the source tree where the configure style checks need to be carried out, while other parts of the tree don’t contain one (or could contain an empty one, but that’s not necessary). The SConfig file can then be checked for in each directory’s SConscript file, and included if necessary. This only requires a small change to all SConscript files in the source tree:

# Set up the environment for this directory
loc_env.MI_SetDirectories()
srcf = loc_env.MI_GetSources()

# Make any configure style checks
if (os.path.isfile("%s/SConfig" % loc_env['currsdir'])):
  SConscript("SConfig", exports='loc_env')

# Store results
objs=[]

This change (or something similar) is included in all Library Level, Subdirectory Level and Test Builder SConscript files. A small change was also made to the SConstruct file to export the os for use in the SConscript files:

if not glob_env['forgive']:
  glob_env.Append(CCFLAGS = unff)

# Allow others to use the environment and mi_scons module
Export('glob_env', 'mi_scons', 'os')

# Pass on to the next stage
BuildDir(glob_env['bdir'],glob_env['sdir'],duplicate=0)
SConscript('%s/Sconscript' % glob_env['bdir'])

Finally the SConfig files can be added containing the configure style checks. To start with, some simple header file checks are made. The first SConfig file is placed in the src/tictac/core directory, and contains checks for the string, exception, and sstream headers used by files in that library:

# Copyright message cut out
...

# Import the environment
Import('loc_env')

# Set up the configure environment
conf = Configure(loc_env)

# Check for standard library headers
if not conf.CheckCXXHeader('string'):
  print "ERROR: String header not present."
  print "ERROR: Please ensure the Standard Library is installed →
and it"
  print "ERROR: contains <string>."
  Exit(1)

if not conf.CheckCXXHeader('exception'):
  print "ERROR: Exception header not present."
  print "ERROR: Please ensure the Standard Library is installed →
and it"
  print "ERROR: contains <exception>."
  Exit(1)

if not conf.CheckCXXHeader('sstream'):
  print "ERROR: String stream header not present."
  print "ERROR: Please ensure the Standard Library is installed →
and it"
  print "ERROR: contains <sstream>."
  Exit(1)

# Finish up the configure environment
loc_env = conf.Finish()

The tests in src/tictac/core/tests also use the iostream header, so a SConfig file is also included in that directory:

# Copyright message cut out
...

# Import the environment
Import('loc_env')

# Set up the configure environment
conf = Configure(loc_env)

# Check for standard library headers
if not conf.CheckCXXHeader('iostream'):
  print "ERROR: Input/Output stream header not present."
  print "ERROR: Please ensure the Standard Library is installed →
and it"
  print "ERROR: contains <iostream>."
  Exit(1)
# Finish up the configure environment
loc_env = conf.Finish()

Now running SCons without the -Q will show the configure checks (running with the -Q will only show any errors, not the list of checks):

[mi_test_scons]$ scons
scons: Reading SConscript files ...
Checking for C++ header file string... yes
Checking for C++ header file exception... yes
Checking for C++ header file sstream... yes
Checking for C++ header file iostream... yes
scons: done reading SConscript files.
scons: Building targets ...
...

---

The changes detailed in this post have been added to the example code repository. The current revision is 32.

Revision 23 added the initial tictac code, and also the testname variable to the SConstruct file and mi_scons.py.

Revision 24 corrected the Doxygen tag exceptions to exception (though the Doxygen manual claims that exceptions is a synonym for exception, Doxygen reported it as an unrecognised tag).

Revision 25 Simply added a link to the PDF version of the Doxygen documentation from the introductory page.

Revision 26 Fixed some errors in comments in exceptions.h.

Revision 27 Fixed the Windows problems outlined in this post.

Revision 28 Added the throw() exception specifier to the what() function in the TictacException to stop gcc warning.

Revision 29 Added the SConfig file in src/tictac/core and updated all the necessary SConscript files to check for and include and SConfig file.

Revision 30 Added the SConfig file to src/tictac/core/tests.

Revision 31 Was a mistake (mentioned in this post) where I forgot why I’d used unsigned char for the board rather than Piece.

Revision 32 Reverted to revision 30 to remove revision 31.

The next post will add to the tictac library, adding new checks for libraries and functions, and including replacement functions if necessary ones don’t exist. It was also noticed that the glob_env/loc_env change made in a previous post didn’t seem to work, as the foobar library wasn’t included when linking the tictac library. This will need further investigation in a later post.

When writing the code for tictac it seemed that a style guide may be useful in explaining why certain decision are made in the code. This guide will be created shortly, and will be organic – growing as thing are thought of which need to go in there. The guide will contain both coding and Doxygen stylings for the Mere Idea code base.

Current Repository Revision: 22

Code highlight key
Grey: Code unchanged
Red: Code removed
Green: Code added
Blue: Code not shown

Now it’s time to add a new library to the example code. This new library will be used for testing the SCons configure style options – checking for the existence of header files, libraries and functions.

The new library is called tictac and will contain a set of classes to implement a two player Noughts and Crosses (or Tic-Tac-Toe) game. It may be written in a very convoluted way to test some features of SCons, so if you really want to write a Noughts and Crosses game, don’t copy this!

The first thing needed is a place for the new library in the src directory. A new directory (tictac) was created in src. Then, inside tictac, the necessary sub-directories are created:

mkdir core tui lib tests

The tui directory will include the interface code (tui – Textual User Interface, but will simply be some couts and cins) and the core directory will include any other code for the game play.

There will be no code at the top (tictac) level of this library (unlike foobar which has foo.h and foo.cxx), so the tests directory at this level may be unnecessary, or it may contain tests combining all other code. Either way, it’s worth creating now for consistency. Consistency is a significant battle to win in software development! Maintenance is so much easier if things are as you expect them to be.

Just to test the scripts for multiple levels, the directory io is also created inside the tui directory. tests directories also need to be created in all the sub-directories.

The next step is to set up the SConscript files so that these directories are included in the build. When creating the foobar library 5 types of SConscript files were set up that can be reused in new projects or libraries:

• Top-level (e.g. mi_test_scons/src/SConscript) that specify which libraries to build. There is one per project.
• Library-level (e.g. mi_test_scons/src/foobar/SConscript) that collect all the object files together (through calls to sub-directories as well as compiling any code at the top-level of the library) and calls the library building SConscript, as well as calling the test building SConscript for any library-level tests. There is one per library.
• Sub-directory (e.g. mi_test_scons/src/foobar/bar/SConscript) that compile the code for that sub-directory, and pass the details back to the library-level SConscript. These can also call any further sub-directory scripts, and build any sub-directory tests. There is one for every sub-directory to any number of levels.
• Library builder (e.g. mi_test_scons/src/foobar/lib/SConscript) that build all of the object files gathered by the library-level SConscript and sub-directory SConscript files into the appropriate library. There is one per library.
• Test builder (e.g. mi_test_scons/src/foobar/tests/SConscript) that simply build any tests. There is one per tests directory, whether at the library level or in sub-directories.

These SConscript files need little change from project to project and library to library, so they can be just copied from one project or library to another. Once the development environment testing is over, all of the files useful for creating new projects and libraries will be put into a single Subversion module, but for now they’ll just be copied from library to library within mi_test_scons.

Firstly, the library-level SConscript is copied from foobar into the tictac directory and edited appropriately.

NOTE: One problem that always cropped up when using Makefiles was that people would just copy a working Makefile from one build to another, and it ended up with no one knowing how the Makefile worked. This made it difficult when something went wrong.

The SConscript files are short and simple, so you should read them carefully when copying them. That way you can check that you have edited everything that needs editing, and can ensure you understand them!

Before copying the library-level SConscript file it was noticed that the code that sets up the LIBPATH variable could be moved out of the SConscript file and into the mi_scons.py file containing the Mere Idea utility functions. A new function MI_SetLibraryPath was added:

...
def MI_SetLibraryPath(env):
  bloc = env['currbdir'].find(env['bdir'])
  if bloc==-1:
    # Comment omitted
    blibpath = '#lib'
  else:
    blibpath = '#'+env['currbdir'][bloc:]+env['pathsep']+'lib'

  env.Append(LIBPATH=[blibpath])
...

and included in the environment:

...
SConsEnvironment.MI_GetHeaders = MI_GetHeaders
SConsEnvironment.MI_SetLibraryPath = MI_SetLibraryPath
SConsEnvironment.MI_BuildObjects = MI_BuildObjects
...

NOTE: When appending to the LIBPATH variable in the mi_scons.py version of this code, the square brackets are used to add the new path as a new list item. These were not used in the SConscript version, and would result in the new path just being concatenated to the existing LIBPATH string (e.g. it would end up with something like ‘/build/release/foobar/lib#build/release/tictac/lib’ instead of two separate paths). This wasn’t apparent until the second library was added.

The LIBPATH appending code was stripped from the SConscript file, and replaced with a call to the new function:

...
# Only put the library together if we have object files
if len(objs)>0:
  # Subdirectories may have changed the directories, so reset them
  env.MI_SetDirectories()
  env.MI_SetLibraryPath()

  # Then work out where we are for the library path
  bloc = env['currbdir'].find(env['bdir'])
  if bloc==-1:
    # Comment omitted
    blibpath = '#lib'
  else
    blibpath = '#'+env['currbdir'][bloc:]+'/lib'

  env.Append(LIBPATH='%s' % blibpath)
  SConscript('lib/SConscript', exports='objs')

elif env['msvs']:
...

This simplifies the SConscript file, which can now be copied to the directory tictac, and edited accordingly:

#
# SConscript file for the foobar library
# SConscript file for the tictac library
#
...

# Enter any subdirectories here
subdirs = Split("""
                bar
                core
                tui
                """)
...

Those are the only changes needed – the rest of the SConscript just works!

If you added the tictac directory name to the top-level SConscript file now, the code could be compiled, and the missing sub-directory SConscript files would be ignored (and SCons would warn you about them). You could then add the other SConscript files only when needed. However, it’s better to get all the setup out of the way, so we’ll continue to copy the appropriate SConscript files from the foobar directory.

The sub-directory SConscript file for the core directory needs only one change from the one in the bar directory of the foobar library – the first comment in the file:

# SConscript file for foobar/bar
# SConscript file for tictac/core
...

The one copied into tui (from bar), however, needs a further change, as tui has a sub-directory (io):

# SConscript file for foobar/bar
# SConscript file for tictac/tui
...

# Set up some local variables
subdirs = Split("""
                io
                """)
...

The same SConscript file can be copied from bar into io, and the first comment changed:

# SConscript file for foobar/bar
# SConscript file for tictac/tui/io
...

Now the test builder SConscript files can be copied over. First the tictac/tests/SConscript file can be copied from foobar/tests/SConscript. All that needs changing in this file is that ever present first comment:

# SConscript file for foobar/tests
# SConscript file for tictac/tests
...

In fact, the SConscript files for all test directories are identical, except for that first comment. Just change it to the path (starting with the library-level directory) to that test directory.

Finally, the library builder SConscript file can be set up. It is just copied from foobar/lib to tictac/lib and changed appropriately. When looking at the script it was noticed that the constant ‘foobar’ was used in several places. To make it easier to change the library builder SConscript files a variable was added at the top of the library-level SConscript file (src/tictac/SConscript) to hold the library name, and this can be used throughout. It is exported to the library builder SConscript (and to any other SConscript file when needed). This keeps the required changes to a minimum. The src/tictac/lib/SConscript file, therefore, becomes:

# Library building SConscript for foobar
# Library building SConscript for tictac
...
Import('*')

loc_env.MI_SetDirectories()
...

if env['msvs']:
  libname = env['LIBPREFIX']+"foobar"+env['LIBSUFFIX']
  libname = env['LIBPREFIX']+thislib+env['LIBSUFFIX']
  env.MI_BuildMSVSProject(srcf,incf,libname)
else:
  env.Library('foobar',objs)
  env.Library(thislib,objs)

# Add it to the link list
env.Prepend(LIBS='foobar')
env.Prepend(LIBS=[thislib])

and the variable and exports are added to src/tictac/SConscript:

...
# <http://www.gnu.org/licenses/>.

# Set up this library name
thislib = 'tictac'

# Enter any subdirectories here
subdirs = Split("""
...

# Only put the library together if we have object files
if len(objs)>0:
  # Subdirectories may have changed the directories, so reset them,
  # and set the appropriate library path
  env.MI_SetDirectories()
  env.MI_SetLibraryPath()

  SConscript('lib/SConscript', exports=['objs', 'thislib'])

elif env['msvs']:
  SConscript('lib/SConscript', exports=['objs', 'thislib'])
...

The changes to use the variable thislib were also made in the src/foobar/SConscript and src/foobar/lib/SConscript files.

NOTE: The square brackets have been added to the Prepend line to ensure the library is prepended correctly. This change has also been made to the SConscript file in the foobar/lib directory.

Now the tictac directory can be added to the top-level src directory SConscript file:

...
# Simply call the SConscript file for each module
mods = Split("""
             foobar
             tictac
             """)
...

When adding the libraries to the top-level SConscript, you should put the lowest level first to ensure correct ordering.

The libraries should now build without any warnings or errors. A couple of potential problems are noticeable, though. Firstly, an empty tictac library is built, and, secondly, it is included on the link line for any foobar based executable. For example, the link line for test_foo (using gcc) is:

g++ -o build/release/foobar/tests/test_foo →
       build/release/foobar/tests/test_foo.o →
       -Lbuild/release/foobar/lib -Lsrc/foobar/lib →
       -Lbuild/release/tictac/lib -Lsrc/tictac/lib →
       -ltictac -lfoobar

The foobar library represents a lower level library than tictac, so shouldn’t require tictac to build. Also, tictac shouldn’t be built if there is no code to put in it, as this is unnecessary work for the build.

The first of these problems stems from the fact that SCons’ first job is to scan all the SConscript files. This sets all variables (including the libraries to link). To fix this, a different SCons environment will be set up for each library, with higher level libraries starting with the environment from the previous level and adding it’s own bits and pieces.

The first part of this is to change the name of env in the SConstruct file to glob_env (for global environment) so that variables that need to be passed from library to library can be stored. For example:

...
glob_env = Environment(options = opts)

Help(opts.GenerateHelpText(glob_env))

# If vs2005 or mingw is set, get and environment with
# the appropriate tool instead
if glob_env['vs2005']:
  glob_env = Environment(options = opts,
                         MSVS_VERSION = '8.0')
elif glob_env['mingw']:
  glob_env = Environment(options = opts,
                         tools = ['mingw']
...
# Allow others to use the environment
Export('glob_env')
...

This name change is done throughout the SConstruct file, and it’s now glob_env that is exported.

Next the glob_env variable needs to be cloned in each library to create loc_env (for local environment) in the library-level SConscript file. We then need to export the loc_env variable to each of the subsequent SConscript files. This is done as a ‘local’ export through the SConscript functions rather than globally through the Export function. Finally, the glob_env environment is reset with a clone of the loc_env environment so that the next library up the chain can use the lower level libraries and their settings. For example, in src/foobar/SConscript:

...
# Import the global variables
Import('*')

# Create a local environment for building this library
loc_env = glob_env.Clone()
...
# Build any subdirectories
for subdir in subdirs:
  if loc_env['msvs']:
    i, s = SConscript('%s/SConscript' % subdir, exports='loc_env')
    incf = incf + i
    srcf = srcf + s
  else:
    o = SConscript('%s/SConscript' % subdir, exports='loc_env')
    objs.append(o)

# Only put the library together if we have object files
if len(objs)>0:
  # Subdirectories may have changed the directories, so reset them
  # and set the appropriate library path
  loc_env.MI_SetDirectories()
  loc_env.MI_SetLibraryPath()

  SConscript('lib/SConscript', exports=['objs', 'thislib' →
, 'loc_env'])

elif loc_env['msvs']:
  SConscript('lib/SConscript', exports=['incf', 'srcf', 'thislib' →
, 'loc_env'])

# Now try to build and run any tests (this will normally be done in
# a testing framework, but here we are simply testing scons...)
SConscript('tests/SConscript', exports='loc_env')

# Put everything we've set up here back into the global environment
glob_env = loc_env.Clone()

Similar changes have to be made in all SConscript files, ensuring all env references are changed to loc_env, and that the loc_env variable is exported through any call to the SConscript function. The calls to Clone are only needed in the library-level SConscript file, however, not the sub-directory ones.

As the environment is local, the same name can be used in all the libraries, so a new library can be added as described at the start of this post – by copying the files from an existing library, and making minimal changes. Remember, though – you should still ensure you understand the script you are copying!

Now for the second problem – stopping the empty library from being built. The problem lies in the lines:

o = SConscript('%s/SConscript' % subdir, exports='loc_env')
objs.append(o)

# Only put the library together if we have object files
if len(objs)>0:

These exist in various SConscript files. In this code, even if o is empty it is still appended to the objs list. The objs variable then has a length of greater than zero, so the library is built.

To solve this is simply a matter of checking that the o variable isn’t empty before appending it to the objs variable in each SConscript file, and printing a warning if the objs list is empty:

o = SConscript('%s/SConscript' % subdir, exports='loc_env')
if len(o)>0:
··objs.append(o)

# Only put the library together if we have object files
if len(objs)>0:
  # Subdirectories may have changed the directories, so reset them
  # and set the appropriate library path
  loc_env.MI_SetDirectories()
  loc_env.MI_SetLibraryPath()

  SConscript('lib/SConscript', export=['objs', 'thislib', 'loc_env'])

elif loc_env['msvs']:
  SConscript('lib/SConscript', exports=['incf', 'srcf', 'thislib' →
,'loc_env'])
else:
  print "WARNING:", thislib, "library is empty."

After this change to all SConscript files the empty tictac library no longer builds. This means that, in the future, if developers begin to think that certain bits of code fit better into a different library, and move them about, and leave a library empty (believe me it does happen), then the library won’t be built, and any link line including it will fail! In fact, this can be taken a little further, and maybe a library with only a single object file (or maybe two) should be considered for refactoring, distributing it’s source files elsewhere. This can be done by adding a new command line option to the SConstruct file:

...
opts.AddOptions(
    ('bdir', "Set to the build directory if you don't like 'build'", →
'build'),
    ('sdir', "Set to the source directory if you don't like 'src'", →
'src'),
    ('smalllib',
 'Set to the number of object files below which a library is →
considered small',
     0),
    BoolOption('debug','Set to apply debugging flags','false'),
...

and using this command line option to issue a warning in the library builder SConscript files:

...
Import('*')

loc_env.MI_SetDirectories()

objs2 = list(mi_scons.MI_FlattenList(objs))
if len(objs2)<=int(loc_env['smalllib']):
  print "WARNING: tictac only contains", len(objs2) ,"object files."

if loc_env['msvs']:
...

You will notice that a new function was used in the above code – MI_FlattenList. This function is necessary to flatten the list to give the correct number of object files when using len(), as the list is nested when gathering object files from sub-directories. The new function is added to the mi_scons.py file:

...
def MI_FlattenList(lst):
  for i in lst:
    if type(i) is list:
      for j in MI_FlattenList(i):
        yield j
    else:
      yield i
...

This function isn’t added to the SCons environment like the others (as it doesn’t need or use the environment, it isn’t appropriate to add it), so to make it available to all scripts we have to export the mi_scons module in the SConstruct file:

...
# Allow others to use the environment and mi_scons module
Export('glob_env','mi_scons')
...

Now we can use the MI_FlattenList function in the library scripts.

The final structure of the tictac tree is:

src/
  +-- foobar/
  +-- SConscript
  +-- tictac/
  |     +-- core/
  |     |     +-- SConscript
  |     |     +-- tests/
  |     |     |     +-- SConscript
  |     +-- lib/
  |     |     +-- SConscript
  |     +-- SConscript
  |     +-- tests/
  |     |     +-- SConscript
  |     +-- tui/
  |     |     +-- io/
  |     |     |     +-- SConscript
  |     |     |     +-- tests/
  |     |     |     |     +-- SConscript
  |     |     +-- SConscript
  |     |     +-- tests/
  |     |     |     +-- SConscript

---

The above changes have been added to the example code’s repository as revision 22. Some of the added bits may need further tweaks once code is added to the tictac library. That’s the time we find out if this all works!