Mere Idea Development Blog

Current Repository Revision: 22

Code highlight key
Grey: Code unchanged
Red: Code removed
Green: Code added
Blue: Code not shown

Now it’s time to add a new library to the example code. This new library will be used for testing the SCons configure style options – checking for the existence of header files, libraries and functions.

The new library is called tictac and will contain a set of classes to implement a two player Noughts and Crosses (or Tic-Tac-Toe) game. It may be written in a very convoluted way to test some features of SCons, so if you really want to write a Noughts and Crosses game, don’t copy this!

The first thing needed is a place for the new library in the src directory. A new directory (tictac) was created in src. Then, inside tictac, the necessary sub-directories are created:

mkdir core tui lib tests

The tui directory will include the interface code (tui – Textual User Interface, but will simply be some couts and cins) and the core directory will include any other code for the game play.

There will be no code at the top (tictac) level of this library (unlike foobar which has foo.h and foo.cxx), so the tests directory at this level may be unnecessary, or it may contain tests combining all other code. Either way, it’s worth creating now for consistency. Consistency is a significant battle to win in software development! Maintenance is so much easier if things are as you expect them to be.

Just to test the scripts for multiple levels, the directory io is also created inside the tui directory. tests directories also need to be created in all the sub-directories.

The next step is to set up the SConscript files so that these directories are included in the build. When creating the foobar library 5 types of SConscript files were set up that can be reused in new projects or libraries:

• Top-level (e.g. mi_test_scons/src/SConscript) that specify which libraries to build. There is one per project.
• Library-level (e.g. mi_test_scons/src/foobar/SConscript) that collect all the object files together (through calls to sub-directories as well as compiling any code at the top-level of the library) and calls the library building SConscript, as well as calling the test building SConscript for any library-level tests. There is one per library.
• Sub-directory (e.g. mi_test_scons/src/foobar/bar/SConscript) that compile the code for that sub-directory, and pass the details back to the library-level SConscript. These can also call any further sub-directory scripts, and build any sub-directory tests. There is one for every sub-directory to any number of levels.
• Library builder (e.g. mi_test_scons/src/foobar/lib/SConscript) that build all of the object files gathered by the library-level SConscript and sub-directory SConscript files into the appropriate library. There is one per library.
• Test builder (e.g. mi_test_scons/src/foobar/tests/SConscript) that simply build any tests. There is one per tests directory, whether at the library level or in sub-directories.

These SConscript files need little change from project to project and library to library, so they can be just copied from one project or library to another. Once the development environment testing is over, all of the files useful for creating new projects and libraries will be put into a single Subversion module, but for now they’ll just be copied from library to library within mi_test_scons.

Firstly, the library-level SConscript is copied from foobar into the tictac directory and edited appropriately.

NOTE: One problem that always cropped up when using Makefiles was that people would just copy a working Makefile from one build to another, and it ended up with no one knowing how the Makefile worked. This made it difficult when something went wrong.

The SConscript files are short and simple, so you should read them carefully when copying them. That way you can check that you have edited everything that needs editing, and can ensure you understand them!

Before copying the library-level SConscript file it was noticed that the code that sets up the LIBPATH variable could be moved out of the SConscript file and into the mi_scons.py file containing the Mere Idea utility functions. A new function MI_SetLibraryPath was added:

...
def MI_SetLibraryPath(env):
  bloc = env['currbdir'].find(env['bdir'])
  if bloc==-1:
    # Comment omitted
    blibpath = '#lib'
  else:
    blibpath = '#'+env['currbdir'][bloc:]+env['pathsep']+'lib'

  env.Append(LIBPATH=[blibpath])
...

and included in the environment:

...
SConsEnvironment.MI_GetHeaders = MI_GetHeaders
SConsEnvironment.MI_SetLibraryPath = MI_SetLibraryPath
SConsEnvironment.MI_BuildObjects = MI_BuildObjects
...

NOTE: When appending to the LIBPATH variable in the mi_scons.py version of this code, the square brackets are used to add the new path as a new list item. These were not used in the SConscript version, and would result in the new path just being concatenated to the existing LIBPATH string (e.g. it would end up with something like ‘/build/release/foobar/lib#build/release/tictac/lib’ instead of two separate paths). This wasn’t apparent until the second library was added.

The LIBPATH appending code was stripped from the SConscript file, and replaced with a call to the new function:

...
# Only put the library together if we have object files
if len(objs)>0:
  # Subdirectories may have changed the directories, so reset them
  env.MI_SetDirectories()
  env.MI_SetLibraryPath()

  # Then work out where we are for the library path
  bloc = env['currbdir'].find(env['bdir'])
  if bloc==-1:
    # Comment omitted
    blibpath = '#lib'
  else
    blibpath = '#'+env['currbdir'][bloc:]+'/lib'

  env.Append(LIBPATH='%s' % blibpath)
  SConscript('lib/SConscript', exports='objs')

elif env['msvs']:
...

This simplifies the SConscript file, which can now be copied to the directory tictac, and edited accordingly:

#
# SConscript file for the foobar library
# SConscript file for the tictac library
#
...

# Enter any subdirectories here
subdirs = Split("""
                bar
                core
                tui
                """)
...

Those are the only changes needed – the rest of the SConscript just works!

If you added the tictac directory name to the top-level SConscript file now, the code could be compiled, and the missing sub-directory SConscript files would be ignored (and SCons would warn you about them). You could then add the other SConscript files only when needed. However, it’s better to get all the setup out of the way, so we’ll continue to copy the appropriate SConscript files from the foobar directory.

The sub-directory SConscript file for the core directory needs only one change from the one in the bar directory of the foobar library – the first comment in the file:

# SConscript file for foobar/bar
# SConscript file for tictac/core
...

The one copied into tui (from bar), however, needs a further change, as tui has a sub-directory (io):

# SConscript file for foobar/bar
# SConscript file for tictac/tui
...

# Set up some local variables
subdirs = Split("""
                io
                """)
...

The same SConscript file can be copied from bar into io, and the first comment changed:

# SConscript file for foobar/bar
# SConscript file for tictac/tui/io
...

Now the test builder SConscript files can be copied over. First the tictac/tests/SConscript file can be copied from foobar/tests/SConscript. All that needs changing in this file is that ever present first comment:

# SConscript file for foobar/tests
# SConscript file for tictac/tests
...

In fact, the SConscript files for all test directories are identical, except for that first comment. Just change it to the path (starting with the library-level directory) to that test directory.

Finally, the library builder SConscript file can be set up. It is just copied from foobar/lib to tictac/lib and changed appropriately. When looking at the script it was noticed that the constant ‘foobar’ was used in several places. To make it easier to change the library builder SConscript files a variable was added at the top of the library-level SConscript file (src/tictac/SConscript) to hold the library name, and this can be used throughout. It is exported to the library builder SConscript (and to any other SConscript file when needed). This keeps the required changes to a minimum. The src/tictac/lib/SConscript file, therefore, becomes:

# Library building SConscript for foobar
# Library building SConscript for tictac
...
Import('*')

loc_env.MI_SetDirectories()
...

if env['msvs']:
  libname = env['LIBPREFIX']+"foobar"+env['LIBSUFFIX']
  libname = env['LIBPREFIX']+thislib+env['LIBSUFFIX']
  env.MI_BuildMSVSProject(srcf,incf,libname)
else:
  env.Library('foobar',objs)
  env.Library(thislib,objs)

# Add it to the link list
env.Prepend(LIBS='foobar')
env.Prepend(LIBS=[thislib])

and the variable and exports are added to src/tictac/SConscript:

...
# <http://www.gnu.org/licenses/>.

# Set up this library name
thislib = 'tictac'

# Enter any subdirectories here
subdirs = Split("""
...

# Only put the library together if we have object files
if len(objs)>0:
  # Subdirectories may have changed the directories, so reset them,
  # and set the appropriate library path
  env.MI_SetDirectories()
  env.MI_SetLibraryPath()

  SConscript('lib/SConscript', exports=['objs', 'thislib'])

elif env['msvs']:
  SConscript('lib/SConscript', exports=['objs', 'thislib'])
...

The changes to use the variable thislib were also made in the src/foobar/SConscript and src/foobar/lib/SConscript files.

NOTE: The square brackets have been added to the Prepend line to ensure the library is prepended correctly. This change has also been made to the SConscript file in the foobar/lib directory.

Now the tictac directory can be added to the top-level src directory SConscript file:

...
# Simply call the SConscript file for each module
mods = Split("""
             foobar
             tictac
             """)
...

When adding the libraries to the top-level SConscript, you should put the lowest level first to ensure correct ordering.

The libraries should now build without any warnings or errors. A couple of potential problems are noticeable, though. Firstly, an empty tictac library is built, and, secondly, it is included on the link line for any foobar based executable. For example, the link line for test_foo (using gcc) is:

g++ -o build/release/foobar/tests/test_foo →
       build/release/foobar/tests/test_foo.o →
       -Lbuild/release/foobar/lib -Lsrc/foobar/lib →
       -Lbuild/release/tictac/lib -Lsrc/tictac/lib →
       -ltictac -lfoobar

The foobar library represents a lower level library than tictac, so shouldn’t require tictac to build. Also, tictac shouldn’t be built if there is no code to put in it, as this is unnecessary work for the build.

The first of these problems stems from the fact that SCons’ first job is to scan all the SConscript files. This sets all variables (including the libraries to link). To fix this, a different SCons environment will be set up for each library, with higher level libraries starting with the environment from the previous level and adding it’s own bits and pieces.

The first part of this is to change the name of env in the SConstruct file to glob_env (for global environment) so that variables that need to be passed from library to library can be stored. For example:

...
glob_env = Environment(options = opts)

Help(opts.GenerateHelpText(glob_env))

# If vs2005 or mingw is set, get and environment with
# the appropriate tool instead
if glob_env['vs2005']:
  glob_env = Environment(options = opts,
                         MSVS_VERSION = '8.0')
elif glob_env['mingw']:
  glob_env = Environment(options = opts,
                         tools = ['mingw']
...
# Allow others to use the environment
Export('glob_env')
...

This name change is done throughout the SConstruct file, and it’s now glob_env that is exported.

Next the glob_env variable needs to be cloned in each library to create loc_env (for local environment) in the library-level SConscript file. We then need to export the loc_env variable to each of the subsequent SConscript files. This is done as a ‘local’ export through the SConscript functions rather than globally through the Export function. Finally, the glob_env environment is reset with a clone of the loc_env environment so that the next library up the chain can use the lower level libraries and their settings. For example, in src/foobar/SConscript:

...
# Import the global variables
Import('*')

# Create a local environment for building this library
loc_env = glob_env.Clone()
...
# Build any subdirectories
for subdir in subdirs:
  if loc_env['msvs']:
    i, s = SConscript('%s/SConscript' % subdir, exports='loc_env')
    incf = incf + i
    srcf = srcf + s
  else:
    o = SConscript('%s/SConscript' % subdir, exports='loc_env')
    objs.append(o)

# Only put the library together if we have object files
if len(objs)>0:
  # Subdirectories may have changed the directories, so reset them
  # and set the appropriate library path
  loc_env.MI_SetDirectories()
  loc_env.MI_SetLibraryPath()

  SConscript('lib/SConscript', exports=['objs', 'thislib' →
, 'loc_env'])

elif loc_env['msvs']:
  SConscript('lib/SConscript', exports=['incf', 'srcf', 'thislib' →
, 'loc_env'])

# Now try to build and run any tests (this will normally be done in
# a testing framework, but here we are simply testing scons...)
SConscript('tests/SConscript', exports='loc_env')

# Put everything we've set up here back into the global environment
glob_env = loc_env.Clone()

Similar changes have to be made in all SConscript files, ensuring all env references are changed to loc_env, and that the loc_env variable is exported through any call to the SConscript function. The calls to Clone are only needed in the library-level SConscript file, however, not the sub-directory ones.

As the environment is local, the same name can be used in all the libraries, so a new library can be added as described at the start of this post – by copying the files from an existing library, and making minimal changes. Remember, though – you should still ensure you understand the script you are copying!

Now for the second problem – stopping the empty library from being built. The problem lies in the lines:

o = SConscript('%s/SConscript' % subdir, exports='loc_env')
objs.append(o)

# Only put the library together if we have object files
if len(objs)>0:

These exist in various SConscript files. In this code, even if o is empty it is still appended to the objs list. The objs variable then has a length of greater than zero, so the library is built.

To solve this is simply a matter of checking that the o variable isn’t empty before appending it to the objs variable in each SConscript file, and printing a warning if the objs list is empty:

o = SConscript('%s/SConscript' % subdir, exports='loc_env')
if len(o)>0:
··objs.append(o)

# Only put the library together if we have object files
if len(objs)>0:
  # Subdirectories may have changed the directories, so reset them
  # and set the appropriate library path
  loc_env.MI_SetDirectories()
  loc_env.MI_SetLibraryPath()

  SConscript('lib/SConscript', export=['objs', 'thislib', 'loc_env'])

elif loc_env['msvs']:
  SConscript('lib/SConscript', exports=['incf', 'srcf', 'thislib' →
,'loc_env'])
else:
  print "WARNING:", thislib, "library is empty."

After this change to all SConscript files the empty tictac library no longer builds. This means that, in the future, if developers begin to think that certain bits of code fit better into a different library, and move them about, and leave a library empty (believe me it does happen), then the library won’t be built, and any link line including it will fail! In fact, this can be taken a little further, and maybe a library with only a single object file (or maybe two) should be considered for refactoring, distributing it’s source files elsewhere. This can be done by adding a new command line option to the SConstruct file:

...
opts.AddOptions(
    ('bdir', "Set to the build directory if you don't like 'build'", →
'build'),
    ('sdir', "Set to the source directory if you don't like 'src'", →
'src'),
    ('smalllib',
 'Set to the number of object files below which a library is →
considered small',
     0),
    BoolOption('debug','Set to apply debugging flags','false'),
...

and using this command line option to issue a warning in the library builder SConscript files:

...
Import('*')

loc_env.MI_SetDirectories()

objs2 = list(mi_scons.MI_FlattenList(objs))
if len(objs2)<=int(loc_env['smalllib']):
  print "WARNING: tictac only contains", len(objs2) ,"object files."

if loc_env['msvs']:
...

You will notice that a new function was used in the above code – MI_FlattenList. This function is necessary to flatten the list to give the correct number of object files when using len(), as the list is nested when gathering object files from sub-directories. The new function is added to the mi_scons.py file:

...
def MI_FlattenList(lst):
  for i in lst:
    if type(i) is list:
      for j in MI_FlattenList(i):
        yield j
    else:
      yield i
...

This function isn’t added to the SCons environment like the others (as it doesn’t need or use the environment, it isn’t appropriate to add it), so to make it available to all scripts we have to export the mi_scons module in the SConstruct file:

...
# Allow others to use the environment and mi_scons module
Export('glob_env','mi_scons')
...

Now we can use the MI_FlattenList function in the library scripts.

The final structure of the tictac tree is:

src/
  +-- foobar/
  +-- SConscript
  +-- tictac/
  |     +-- core/
  |     |     +-- SConscript
  |     |     +-- tests/
  |     |     |     +-- SConscript
  |     +-- lib/
  |     |     +-- SConscript
  |     +-- SConscript
  |     +-- tests/
  |     |     +-- SConscript
  |     +-- tui/
  |     |     +-- io/
  |     |     |     +-- SConscript
  |     |     |     +-- tests/
  |     |     |     |     +-- SConscript
  |     |     +-- SConscript
  |     |     +-- tests/
  |     |     |     +-- SConscript

---

The above changes have been added to the example code’s repository as revision 22. Some of the added bits may need further tweaks once code is added to the tictac library. That’s the time we find out if this all works!