Mere Idea Development Blog

Current Repository Revision: 20

The previous post discussed how Doxygen can be used to produce on-line documentation of C++ code from structured comments. The methods outlined in that post have now been applied to the main mi_test_scons example code. The resulting documentation can be found here.

So that the documentation is always up-to-date, a cron job has been added to build the documentation overnight. The following script is used to do this:

#!/bin/sh
#
# Run the Doxygen command over mi_test_scons, and publish
# the results
#

echo "Starting Doxygen script (mi_test_scons)"
cd /dir/containing/mi_test_scons

# Get the latest code
svn up

# Get rid of the old files, run Doxygen,
# and then go to the produced html directory
rm -Rf docs/html
rm -Rf docs/latex
time doxygen
cd docs/html

# Copy the tab image files, remove old published
# documentation and move the new stuff in its place.
cp ../html_files/*.gif .
rm -Rf /wherever/webpages/are/docs/mi_test_scons/*
mv * /wherever/webpages/are/docs/mi_test_scons/

# Make the pdf and copy to web space
cd ../latex/
make
cp refman.pdf →
   /wherever/webpages/are/docs/mi_test_scons/mi_test_scons.pdf

echo "Done"

It’s pretty straightforward, and doesn’t require much explanation, but there are some things to note.

Firstly, when I initially tried to use this script I discovered that, in my Linux environment, cron used a different Subversion executable than the one called by logged in users. The version called by cron (/usr/bin/svn) was older than the one called by users (/usr/local/bin/svn) and so caused errors, warning about a need to upgrade. To solve this problem soft links were created from all the Subversion executables (svn, svnadmin, svnlook, etc.) in /usr/bin to their equivalent in /usr/local/bin. The old executables were kept in /usr/bin by renaming them with the suffix _bin in case of later problems where we may need to recover them. For example:

cd /usr/bin
mv svn svn_bin
ln -s /usr/local/bin/svn svn
mv svnadmin svnadmin_bin
ln -s /usr/local/bin/svnadmin svnadmin
...

The second thing to note is that we also build the PDF version of the documentation from the Doxygen latex build. This is all automatic once the Doxyfile latex variables are set:

GENERATE_LATEX        = YES
LATEX_OUTPUT          = latex
PAPER_TYPE            = a4
PDF_HYPERLINKS        = YES
USE_PDFLATEX          = YES

When Doxygen is run with these settings, a Makefile will be created in the ./docs/latex directory. Typing make in that directory will then generate the file refman.pdf containing all the documentation, as can be seen from the cron job script. The resulting PDF file can be seen here.

The version of Doxyfile committed to the Mere Idea Subversion repository in the example code has the latex variables set as above.

Finally, the reason the doxygen command is run using the time command (time doxygen) is so that the length of time it takes can be monitored (via the cron-sent email) and the Doxygen build can be split up if it’s taking too long. This can happen in large projects (though it is unlikely with our example).

Licensing

A couple of emails received recently have been giving positive feedback regarding this blog. These are always nice to receive. I welcome any feedback on the blog, including anything you’d like to see discussed (provided it’s related to setting up a development environment). You can give feedback using the Mere Idea contact form.

One thing mentioned in those emails was that it was unclear whether or not the example code could be used by others in their own development environments. As it stands, any of the example code from this blog is free to be used by anybody. To make its availability for use more explicit, I have now licensed the code under the GNU General Public License Version 3 (GPLv3).

If my understanding of the license is correct, this means that any other parts of your development environment that are derived from the Mere Idea code will need to be licensed under the GPLv3 (it’s only fair! I’m sharing my ideas with you, so you should really share too! ), but any code produced using that environment would not need to be GPLv3 licensed (as it wouldn’t actually be derived from any of the GPL’d code). This license is also compatible with the MIT license used by SCons.

If my understanding is wrong, please feel free to contact Mere Idea with a correction. I’m no lawyer!

---

Revision 19 of the example code added the Doxygen and GPLv3 comments to all source files, as well as the committing the Doxygen support files (configuration file, introductory page, etc.). The docs directory was also added (along with the html_files directory containing tab images and style sheet), as well as the main license document (LICENSE.txt).

Revision 20 added some Doxygen markup to the LICENSE.txt file so it appears in the code documentation.

The next post will see a return to SCons, adding a new library to test the ‘configure’ style checking, and then on to how to include replacement code for missing library functions.

Current Repository Revision: 18

Something else I was working on this week made me think about code documentation. None has been done in the example code so far, but as part of the development process it would be good to set up an automatic, nightly build of documentation from the code. It’s better to do this now, as retro-documenting is a big task if there is a lot of code already in place!

The tool I tend to use for producing code documentation is Doxygen. It does an excellent job and, looking around, there isn’t anything better in the price range (free ). I did look at NaturalDocs recently, but while I like the more natural documenting style and the output, it doesn’t have full language support for C/C++ or Python. This means you don’t get anything for free – undocumented functions produce no output and there are no useful inheritance diagrams. According to the last news page I could find, version 2.0 is in development, so I may look at this again when some of it’s features are announced.

For now, on with Doxygen…

Marking up the code

The Doxygen website contains a good manual which can be read online or downloaded, so I’m not going to give a tutorial here on all possible combinations of special commands and magical incantations. I will, however, show some examples of how I mark up code.

My preferred method (from the many choices) for specifying a Doxygen comment block is:

//! This is a single line, brief description
/*!
    This is a longer description

    \param p Some parameter called p
    \return Something is returned from this function
 */

This is referred to as the Qt style in the Doxygen manual.

I like to place the majority of the Doxygen documentation for C/C++ in the .h files, though a \file block is usually placed in the .cxx files and in test/example source files.

Note: I have found in the past that tabs in the comment block (for example, it may be tempting to tab the \param or \return lines) can put tabs into the final documentation, and make alignment messy. I always use spaces.

For documenting Python I use the style that enables the use of Doxygen’s special commands:

## This is a single line brief description
#
#  This is a longer description
#
#  \param p Some parameter called p
#  \return Something is returned from this function
#

For Python I only document the mi_scons.py file in the example code, not the SConstruct or SConscript files.

Below are some more detailed examples of how the code is documented with Doxygen. The full documentation output for this can be seen here.

File Headers

At the beginning of each .h and .py file I place a header boilerplate. This block begins with a \file command, and contains descriptions (brief and/or detailed), and a change log. An example of a .h file boilerplate would be:

/*!
   \file clever_class.h

   \brief  Defines MI_CleverClass to do something clever

   \clog
    Date        Comment           SVN Revision   Developer
    ==========  ================  ============   =========
    2009/11/20  Initial draft         3          chris
    2009/11/21  Added clever bit      4          chris
                of code
   \endclog
 */

The header template could also include license information. You could add a more detailed description if this file was defining more than just a single class, but you should remember that you are only stating here things relevant to the whole file, so don’t get into specifics about the contents. You could also add a \todo if it was relevant to the whole file. For example:

\todo Tidy up Doxygen comments

The equivalent in Python of the above boilerplate would be:

##
# \file clever_class.py
#
# \brief Defines MI_CleverClass to do something clever
#
# \clog
# Date        Comment           SVN Revision  Developer
# ==========  ================  ============  =========
# 2009/11/20  Initial draft         3         chris
# 2009/11/21  Added clever bit      4         chris
#             of code
# \endclog
#

The result of running these headers through Doxygen can be seen here for the .h file, and here for the Python file

Note: \clog is a command added as an alias in the Doxyfile configuration file (short for ‘change log’). If you use your own configuration file, \clog won’t work. Every \clog requires a matching \endclog.

For the rest of the examples you’ll see only the C++ style mark up, but you can easily infer the Python style from that.

Classes/Structures

For classes (and structures other than C-style, data-only structures) a brief description is given, followed by a more detailed description. Any \bug or \todo items relevant to the whole class can also be added. Again, be careful not to get into specifics here, just a general overview of the class, and any items relevant to the whole class. Try to avoid starting the comment with “A class to…” .

//! Do something very clever
/*!
    When something clever is needed, call on MI_CleverClass to do
    it.

    \todo Add a method to implement world peace.
*/
class MI_CleverClass
{
 ...
};

Methods/Functions

For methods and functions the style used is to give a brief, one line description, followed by a longer description (if necessary), similar to the class/structure comment. Then describe any parameters (for parameters that don’t change use \param and for any parameters expected to change we use \retval) and any return value (using \return). In addition, any known bugs for this method/function can be specified with \bug and anything that needs doing with \todo. If a method/function expects certain conditions to be true when entering, and they are not described in a \param or \retval command, then \pre commands should be used. If the function makes changes to the state of something, and that change is not described in a \retval or \return command, then a \post command should be used. Other, relevant functions should also be referenced with the \sa command.

//! Return the appropriate value for the caller's IQ
/*!
    The caller passes an IQ, and the appropriate value is
    returned.

    \param iq The IQ to get the appropriate value for
    \return A string containing a word appropriate to the
            iq parameter.
    \post The class will store the iq for use with calls
          to \ref stringFromLastIQ
    \sa stringFromLastIQ, displayStringFromIQ
    \bug The method returns "76" when it should return
         "mollycoddle"
*/
const std::string stringFromIQ(const unsigned int iq);

Variables

When marking up variables, whether they are member variables, global variables or any other, I usually use a single line brief-style comment.

//! Holds the last IQ used in \ref stringFromIQ
unsigned int last_iq_;

They can be expanded to multiple lines if necessary, but (as with all documentation) are best kept short.

Enumerated types, data-only structs, unions and the like

Anything whose form is similar to that of an enumerated type is usually documented with a brief, one line description, a longer description (if necessary), and then a short description for each element.

//! The types of smells
/*!
    Used when associating IQs with smells. The value of the
    enumeration isn't related to the IQ level!
*/
enum IQSmells
{
  NO_SMELL=0, //!< IQ levels which give off no smell
  FISH,       //!< Fishy IQs
  CHEESE,     //!< Cheesy IQs
  MILK,       //!< The milky IQs
  SYRUP       //!< Sweet smelling IQs
};

Examples and tests

Examples and tests can be added at the end of a header file using the \example command.  It is simply a comment block with the name of the file containing the test, and a description about each test.

/*!
 * \example clever_test.cxx
 *          Test the MI_CleverClass works
 */

I found that, for Doxygen to find the test, I had to change the Doxyfile configuration file to add an examples directory

EXAMPLE_PATH = src/

The example was in ./src/tests relative to the directory Doxygen was executed in.

---

That’s probably enough for now. The final documentation can be seen here.

More examples may be added as I think of them. I am also planning to add a style guide to the documentation pages on the main site which will include Doxygen styling. A link from this blog will be added once the guide has been created (it will be an organic guide – growing as I think of things to add). The code from this post has been added to the svn repository, and can be retrieved using the command

svn checkout

http://www.mereidea.com/svn/testing/trunk/doxy_example

The svn revision 15 in the testing repository was the initial import, revision 16 corrected a typo in the versionfilter.sh file, and revision 17 added the stylesheet and changes to Doxyfile to include it, as well as a revised version of the tab images (note that the tab images are manually copied into the html directory, otherwise the original images are used). Revision 18 updated a link in the intro.txt which points to this post. I couldn’t update that until this post was published!

The code in the doxy_example repository has never been compiled. Only the documentation has been built, so the code may not build or run!

The next thing to do is add Doxygen comments to the example code. Then set up an automatic overnight build of the documentation from the svn repository.

Note: Why the MI_CleverClass is clever, and how the smells are applied to IQ levels are left as exercises for the reader.

Current Repository Revision: 14

Code highlight key
Grey: Code unchanged
Red: Code removed
Green: Code added
Blue: Code not shown

We can now create Visual Studio projects for our example code and use them to help when debugging using the Visual Studio debugger.

Before we can start debugging, however, we need to fix the debug variant build so that it can be built with the Visual Studio compiler, both at the command line and in the IDE.

Fixing the debug build for Visual Studio command line

If you try to build a debug variant of the code, using Visual Studio as the compiler, with the command:

scons -Q debug=true

the following error will appear:

scons: *** Multiple ways to build the same target were
specified for: build\debug\mi_debug.pdb (from ['bar_test.obj']
and from ['raw_test.obj'])

The error occurs because the PDB variable has to be set for SCons to perform a debug build when using Microsoft Visual Studio. This is set in the SConstruct file (to the value mi_debug.pdb) and never changed, so every executable tries to use this as its Program Database file. This causes a clash in targets.

To solve this we simply need to set the PDB variable for each executable if debug=true is specified on the command line. This can be done inside the MI_BuildTestsAndExecute function inside mi_scons.py (at the moment this is the only function we have which builds executables):

nxttst = os.path.basename(nxttst)
if env['debug']:
      pdb_name = os.path.splitext(nxttst)[0]+'.pdb'
      env.Replace(PDB = env['currbdir']+env['pathsep']+pdb_name)
trgt = env.Program(nxttst)

The debug variant of the code can now be built successfully, and the correct .pdb files are created – essential for debugging with Visual Studio.

Fixing the debug build for Visual Studio IDE

The above code fixes the debug variant so it can be built on the command line when Visual Studio is your compiler, but if you build the Visual Studio projects for the debug build with:

scons -Q debug=true msvs=true

and load a resulting project into the Visual Studio IDE and select Build you will get an error similar to:

scons: *** Do not know how to make target
          `build\debug\foobar\lib\foobar.lib'.  Stop.

When we build on the command line we specify the option debug=true which does a lot of setting up (including setting the build directory to build/debug). So if we want to be able to build the debug variant in Visual Studio (which we may during a debugging session) we need to be able to pass the debug=true switch to SCons.

We add all our switches by appending them to the end of the MSVSSCONSFLAGS variable in our environment. This is done in the SConstruct file.

# We need to set up MSVS command lines before changing bdir
# We can add our flags to the MSVSSCONSFLAGS variable
# We just add them all, whether default or not
# msvs, runtests, mingw and vs2005 don't apply for obvious reasons.
if env['msvs']:
  env.Append(MSVSSCONSFLAGS = ' bdir='+env['bdir'])
  env.Append(MSVSSCONSFLAGS = ' sdir='+env['sdir'])
  env.Append(MSVSSCONSFLAGS = ' debug='+str(env['debug']))
  env.Append(MSVSSCONSFLAGS = ' forgive='+str(env['forgive']))

Now you can load the debug project files into your Visual Studio and build them.

Multiple build variants in one project file

Rather than have to specify debug=true at the command line, it would be nice if just specifying msvs=true would create both release and debug variants in a single project file, and put them in a directory named msvs instead of in the debug and release directories. According to the SCons manual:

variant: The name of this particular variant. For Visual Studio 7 projects, this can also be a list of variant names. These are typically things like “Debug” or “Release”, but really can be anything you want.

…

Multiple calls to MSVSProject with different variants are allowed; all variants will be added to the project file with their appropriate build targets and sources.

It seems the  first part of this (using a list of variants) isn’t much help to us, because we need to set the MSVSSCONSFLAGS variable to be different for the release and debug builds (debug=true for debug and debug=false for release). So instead we need to call the same target twice with different flags set. When trying this we get the error:

scons: warning: Two different environments were specified for
       target bar_test.vcproj, but they appear to have the same
       action: GenerateProject(target, source, env)
...

scons: *** Multiple ways to build the same target were specified
       for: bar_test.vcproj ...

I think this is because we try to change the environment (the MSVSSCONSFLAGS variable) between calls. Whatever the reason, this doesn’t work, and I have to go back to the part I said earlier ‘…isn’t much help to us…’.

Now, what follows is an ugly hack. I’m sure (hope!) that there is a proper way to do this, and someone will point it out to me. In the meantime, I made the following changes. Firstly I remove the debug reference from the MSVSSCONSFLAGS variable we had just added (as msvs=true will now build both variants) in the SConstruct file.

# We need to set up MSVS command lines before changing bdir
# We can add our flags to the MSVSSCONSFLAGS variable
# We just add them all on, whether default or not
# msvs, runtests, mingw and vs2005 don't apply for obvious reasons.
if env['msvs']:
  env.Append(MSVSSCONSFLAGS = ' bdir='+env['bdir'])
  env.Append(MSVSSCONSFLAGS = ' sdir='+env['sdir'])
  env.Append(MSVSSCONSFLAGS = ' debug='+str(env['debug']))
  env.Append(MSVSSCONSFLAGS = ' forgive='+str(env['forgive']))

We also want to build the projects in the directory msvs, so we need to make sure we set the build directory to that if msvs=true. This is also in the SConstruct file.

if env['msvs']:
  env.Append(bdir = env['pathsep']+'msvs')
elif env['debug']:
  env.Append(CCFLAGS = dbgf)
  env.Append(bdir = env['pathsep']+'debug')

  # MSVC is a special case where we have to set the PDB
  # variable to add debug flags
  if cmplr=='cl': env.Replace(PDB='%s/mi_debug.pdb' % env['bdir'])
else:
  env.Append(bdir = env['pathsep']+'release')

Then we can add some code to mi_scons.py which will set up and build both variants. The horrible hack is that we add the build=true switch to the build target, so it is used in the command line. To do this we must remember that the build target is usually surrounded in quotes, so we have to add some quotes to make the command correct. We also make sure the directory name is correct for the build targets.

# Create the variant build targets with a nasty hack for the debug!
# We add the debug=true switch so it is used in the command! We need
# to use some carefully placed quotes for this to work
relbldtar = string.replace(bldtar, 'msvs', 'release', 1)
dbgbldtar = 'debug=true" "'+string.replace(bldtar, 'msvs', 'debug', 1)

if env['debug']: vrnt = 'Debug'
else: vrnt = 'Release'

outmsvs = env.MSVSProject(target = projname,
                          srcs = sources,
                          incs = heads,
                          buildtarget = bldtar,
                          buildtarget = [relbldtar,dbgbldtar],
                          variant = vrnt,
                          variant = ['Release', 'Debug'],
                          runfile = rnfl,
                          auto_build_solution=0)

This code works for creating multiple variants in a single Microsoft Visual Studio Project, but any executables you create won’t run. This is because the runfile set in the project is just the name of the executable. This was fine when the projects were in the release or debug directories, but now Visual Studio looks for the executables in the msvs directory. To solve this we need to specify the runfile with the full path to the executable, and pass a list (one for each variant) to the MSVSProject builder.

# Whatever the current build target is, we need to calculate the
# full path to the final build location. This is what we need
# to get Visual Studio to run built programs, so use that
# as the run file
rnfl = bldtar
rnfl = env['currbdir']+env['pathsep']+bldtar
dbgrnfl = string.replace(rnfl, 'msvs', 'debug', 1)
relrnfl = string.replace(rnfl, 'msvs', 'release', 1)

# Other setup code
...

outmsvs = env.MSVSProject(target = projname,
                          srcs = sources,
                          incs = heads,
                          buildtarget = [relbldtar,dbgbldtar],
                          variant = ['Release', 'Debug'],
                          runfile = rnfl,
                          runfile = [relrnfl, dbgrnfl],
                          auto_build_solution=0)

Now you can run the code, but the dialog will still appear telling you the executable is out of date when it isn’t, as mentioned in the previous post. This dialog can be useful in other circumstances, so I’m hesitant about ticking the ‘Do not show this dialog again’ box in case it disappears altogether!

Debugging SCons builds with Visual Studio

Now all the projects are set up as we want them. All the Visual Studio rubbish ends up in the msvs directory tree (except for the pdb and incremental linker – ilk – files), and we have both our debug and release variants in single project files, one for each library and one for each executable.

The projects are useful because they enable us to debug the code effectively in Visual Studio. By creating a solution and loading all (or a subset) of the projects into it you allow Visual Studio’s Intellisense to browse through the code. Then you can use the features of Visual Studio to track down problems.

You can use the code definition window to see where variables, functions and classes are defined, and the code that makes up a class or function. The call graph window enables you to see what calls a particular function, or what functions it calls.  Then you can use the debugger, place breakpoints, edit and rebuild code, even add new files (our SConscript files look for all .c, .cxx and .h files rather than specific names, so there’s no need to edit the SConscript when adding new code files). You can edit the SConscript files inside Visual Studio if, for example, you add a new directory.

I will use Visual Studio projects for debugging SCons based code, but not generally for creating and editing code, or building and testing. This is because, for each project you need to build, Visual Studio needs to call SCons, and all the set up code is repeated, making it less efficient. I prefer just to compile from the command line – it’s what SCons is about. SCons is a full build system – not just a configuration system for other builders such as make or the Visual Studio environment.

Debugging SCons build in Visual Studio without the projects

Of course, you don’t need to use the project files to debug SCons built executables with Visual Studio.

You can debug any executable by building a debug version of the code at the command line (for example, use scons -Q debug=true to build our example code). Open Visual Studio, and create a new, general, empty project. Choose to insert an existing project, and in the dialog select Executable files as the file type. Then browse to the executable you want to debug, and load it in. Set it as your start up project, and then you can either open a file containing the code you want to debug, place a breakpoint and start debugging (F5), or start at the beginning of the executable with Step Into from the Debug menu (or F11).

You can also debug any executable in Visual Studio by attaching the debugger to a running process. This isn’t much use given the current example code as the processes will have ended before we can attach the debugger, but it can be useful in longer processes where you don’t see the problem when you start it in the debugger (Hint: if this is the case it often points to memory issues).

Again, build the debug version of the code on the command line, run the program, open Visual Studio and select Attach To Process… from the Tools menu. Then you can either open a file containing code you know the program is going to get to soon and put a breakpoint in it, or you can break the code (using the pause symbol), look at the stack trace and select a function you recognise as yours (rather than the deep, dark Microsoft ‘write a character to the screen’ function that I always end up breaking into!). Selecting the familiar function in the stack trace should open up the relevant file. Then place a breakpoint and away you go!

---

The code updates discussed in this post are committed as revision 14 in the repository.

Hopefully, that’s the Visual Studio projects finished with for the time being. Now we can get on with adding more code to test other features of SCons and see how well it can build our code.

Current Repository Revision: 13

Code highlight key
Grey: Code unchanged
Red: Code removed
Green: Code added
Blue: Code not shown

Now we are at the stage where we can create our Visual Studio projects. We will create a Visual Studio project for each library, and one for each executable.

In the current example code there is only one library (foobar) and three executable tests (test_foo, bar_test and raw_test – and that’s the first time I’ve noticed the inconsistent naming! ).

To be able to create a project for the foobar library we need to gather all the source and header files that are compiled into that library together to include in the project. We also need to stop creating the individual project files for each source file. To do this requires editing the SConscript files in the foobar directory and its subdirectory bar.

src\foobar\SConscript

The SConscript file in src\foobar was amended to remove the line that generates the Visual Studio project:

# Check if we just want to produce the msvs project
if env['msvs']:
  incf = env.MI_GetHeaders()
  env.MI_BuildMSVSProject(scrf,incf,'obj')
else:
  # Build the object files
  objs = env.MI_BuildObjects(srcf)

The include files are still collected if msvs=true is used when calling SCons.

We also need to alter this file to collect source and header files from the sub-directories, so code is added inside the loop iterating over the sub-directories to gather these files instead of the usual object files:

for subdir in subdirs:
  if env['msvs']:
    i, s = SConscript('%s/SConscript' % subdir)
    incf = incf + i
    srcf = srcf + s
  else:
    o = SConscript('%s/SConscript' % subdir)
    objs.append(o)

Finally in this file we need to pass the lists of source and header files to the library SConscript file to use when building the project:

if len(objs)>0:
  # Code for building the objects is here
  ...
elif env['msvs']:
  SConscript('lib/SConscript', exports=['incf', 'srcf']

src\foobar\bar\SConscript

The changes to the SConscript file in src\foobar\bar are similar to those in src\foobar\SConscript. The call to MI_BuildMSVSProject is removed, and the code to gather header and source files is added to the sub-directory loop. However, this file needs to return its source and header files to the calling SConscript file:

# Pass back the objects or headers and sources
if env['msvs']:
  Return('incf srcf')
else:
  Return('objs')

src\foobar\lib\SConscript

The library SConscript file needs changing to take the exported source and header files and create a Visual Studio project from them capable of building the library. We also need to make sure the internal directory variables are set correctly, and ensure the build target name is correct. Then we can call the MI_BuildMSVSProject function to create the project file for the library:

env.MI_SetDirectories()

if env['msvs']:
 libname = env['LIBPREFIX']+"foobar"+env['LIBSUFFIX']
 env.MI_BuildMSVSProject(srcf,incf,libname)
else:
 env.Library('foobar',objs)

mi_scons.py

The function to build the Visual Studio projects (MI_BuildMSVSProject) needs some changes to make sure the project created can be built in Visual Studio. As we are now passing the full name of the target through the bldtar variable, we can use this as a basis for the project name, removing the code that calculated the project name from the directory names, and replacing it with code to correctly calculate the build target directory (in the build tree relative to the base directory):

# Use the build target as a basis for the project name.
# Change the extension to the project one.
projname =  os.path.splitext(bldtar)[0]+env['MSVSPROJECTSUFFIX']

# Directory discovery code
...

if bloc==-1:
  projname = str(sources[0])
  projname = string.replace(projname,'.cxx','')
  projname = string.replace(projname,'.c','')

  # We can't find the path, we'll just have to use the whole
  # path and let the human sort out any errors!
  bldtar=bd+env['pathsep']+bldtar
else:
  projname = bd[bloc:]
  projname = string.replace(projname,fnd,'')
  projname = string.replace(projname,env['pathsep'],'_')

  # Found the correct path, so use it
  bldtar = bd[bloc:]+env['pathsep']+bldtar

projname += env['MSVSPROJECTSUFFIX']

These changes are all that is needed to create library project files correctly. Further changes are required for creating projects for each of the test executables.

src\foobar\bar\tests\SConscript

The extra problem with building our executable Visual Studio projects is highlighted by the src\foobar\bar\tests directory. This directory contains two source files to be built as executables, and we’d like them in separate Visual Studio projects. The standard MI_BuildMSVSProject function, however, bundles all source and header files together in the one project. We will be adding a new function (MI_BuildExeMSVSProject) to the mi_scons.py file, and so our SConscript file can call that function instead:

if env['msvs']:
  incf = env.MI_GetHeaders()
  env.MI_BuildMSVSProject(srcf,incf,'exe')
  env.MI_BuildExeMSVSProject(srcf)
else:
  env.MI_BuildTestsAndExecute(srcf)

Note: This code assumes that each executable has no header files, and there is only a single source file per executable. If you have header files or multiple source files, then load the project that has the file named after the target you want to build, and add any extra files manually (or not at all if you’re not interested in them). The executable will still build normally.

src\foobar\tests\SConscript

The change made to src\foobar\bar\tests\SConscript was also made to the scr\foobar\tests\SConscript file.

mi_scons.py Revisited

The MI_BuildExeMSVSProject function mentioned above for building individual test projects for each source file was added to the mi_scons.py file. This is simply a function that iterates through the source files and calls MI_BuildMSVSProject with the appropriate arguments:

# Build an executable MSVS Project. It has to build a project
# per source. The build targets are worked out from the source
# file names.
# There are no header files passed in
def MI_BuildExeMSVSProject(env, sources):
  # Run through the sources creating an executable file
  for nxtsrc in sources:
    nxttar = os.path.basename(nxtsrc)
    nxttar = string.replace(nxttar,'.cxx','.exe')
    nxttar = string.replace(nxttar,'.c', '.exe')

    # Now we can just call the project file builder
    env.MI_BuildMSVSProject(nxtsrc, '', nxttar)

This function then needs adding to the SCons environment:

SConsEnvironment.MI_SetDirectories = MI_SetDirectories
SConsEnvironment.MI_GetSources = MI_GetSources
SConsEnvironment.MI_GetHeaders = MI_GetHeaders
SConsEnvironment.MI_BuildObjects = MI_BuildObjects
SConsEnvironment.MI_BuildMSVSProject = MI_BuildMSVSProject
SConsEnvironment.MI_BuildExeMSVSProject = MI_BuildExeMSVSProject
SConsEnvironment.MI_BuildTestsAndExecute = MI_BuildTestsAndExecute

One final point about executables is that if you want Visual Studio to be able to run them when built, you need to set the runfile to be just the executable name. If you don’t do this SCons assumes that the executable name is the same as the build target. This is a problem because the build target contains the path necessary for a build to work, as discussed in the previous post. Visual Studio constructs the executable path from the target path plus the target name. Therefore, the path is included twice. Before bldtar is manipulated to include the current path, it contains what we need as the executable name, so to solve the problem we simply set the variable rnfl in the MI_BuildMSVSProject function, and pass it to the SCons MSVSProject builder as the runfile:

# Whatever the current build target is, it should be just
# the name of the file (with no path). This is what we need
# to get Visual Studio to run build programs, so use that
# as the run file
rnfl = bldtar

# Other jiggery-pokery here including adding path to
# build target
...

outmsvs = env.MSVSProject(target = projname,
                          srcs = sources,
                          incs = heads,
                          buildtarget = bldtar,
                          variant = vrnt,
                          runfile = rnfl,
                          auto_build_solution=0)

There is still one outstanding problem with using the Visual Studio projects built with SCons when running the executable. That is that Visual Studio is unable to confirm that the target is up to date, and so pops up the ‘The project is out of date’ dialog every time it’s run. At the moment I’m not looking into this any further, but any comments on how to fix this are welcome.

SConstruct

One final change we can make is to ensure Visual Studio shows the full path for source files when reporting errors and warnings in the Output window. This enables us to click on the file containing the error and it be opened in the editor at the error line. This is done by adding the \FC switch to the compile line:

genf = ' /Wall /EHsc /wd4820 /wd4710 /FC'

The code outlined in this post is now committed as revision 12 and 13 in the example code. Revision 12 has all the important changes in it (and will work as described), and revision 13 made some minor changes – removing the unnecessary call to MI_GetHeaders from the test SConscript files and correcting a comment.

Next I want to check that we can debug in Visual Studio using these projects without too much trouble. I already know that the debug build of our code doesn’t work for Windows, even on the command line, due to a problem with PDBs. This will be fixed in the next post, along with some discussion on debugging SCons builds with Visual Studio.

Current Repository Revision: 11

The next stage in our development process was to see how we could create Microsoft Visual Studio projects using SCons. We already had some code in our example code to build a project for each directory by calling:

scons -Q msvs=true

There is an error in this code, however. If you try and build Visual Studio projects with the code at revision 10, you will end up with some files in the build directories named, for example, foobar.vcproj that only contain the text:

This is just a placeholder file.
The real project file is here:
C:\mycpp\mi_test_scons\src\foobar\foobar.vcproj

or something similar. This is because when using SCons with BuildDir(), the src directory still gets the real Visual Studio project, while the build directory just gets the place holder. However, our example code assumes that the project file in the build directory is a duplicate of the one in the src directory, and deletes the one in src (because we like to keep the source tree as clean as possible, and the project files are generated files, so we feel they should be in the build directory). The first task, therefore, was to fix this.

Creating the real project file in the build directory

We can change the code to delete the dummy project file and move the project file from src to build as a post-build action. The original code for the MI_BuildMSVSProject function in the mi_scons.py file (as revision at 10) ended with the code to delete the project file from the source:

# This causes a project to be built in both the build and
# source directories. so we need to delete the one in source
srcfile = env['currsdir']+env['pathsep']+str(outmsvs[0])
env.AddPostAction(outmsvs,Defaults.Delete(srcfile))

So we can easily change this to delete the build dummy file, and move the source one to the build directory:

# This causes a project to be built in the source directory,
# and a place holder project to be built in the build directory.
# We want the project in the build directory, so delete the one
# there already, and move the source directory project into the build
srcfile = env['currsdir']+env['pathsep']+str(outmsvs[0])
bldfile = env['currbdir']+env['pathsep']+str(outmsvs[0])
env.AddPostAction(outmsvs,Defaults.Delete(bldfile))
env.AddPostAction(outmsvs,Defaults.Move(bldfile,srcfile))

Specifying build targets

After changing the code as specified above, and running SCons with msvs=true to create the Visual Studio projects, one was loaded into Visual Studio (foobar.vcproj) and a build was attempted.

The first thing to note was that, when loading into Microsoft Visual Studio Express 2008, the project needed converting (even though SCons had been run with Visual Studio 2008). The conversion simply changed the VisualStudioProject element by changing the Version attribute from “8.00″ to “9.00″ and removing the attributes SccProjectName, SccLocalPath and TargetFrameworkVersion. Apart from removing some newlines and the SCons Data comment from the end, there were no other changes.

Once the project was loaded a build was attempted. This resulted in the error:

scons: *** Do not know how to make target `obj'.  Stop.

The obj bit of that error is the value that was specified as the buildtarget when calling the SCons Builder MSVSProject. According to the SCons manual:

buildtarget: An optional string, node, or list of strings or nodes (one per build variant), to tell the Visual Studio debugger what output target to use in what build variant. The number of buildtarget entries must match the number of variant entries.

As it stated  that the buildtarget string is optional, it was removed from the my_scons.py MI_BuildMSVSProject function, and SCons rerun to create the new projects. The foobar.vcproj file was reloaded into Visual Studio, and a build was again attempted. This made matters worse, with the errors:

IndexError: string index out of range:
File
  "C:\Python26\Lib\site-packages\scons-1.2.0\SCons\Script\Main.py",
  line 1264:
    _exec_main(parser, values)
File
  "C:\Python26\Lib\site-packages\scons-1.2.0\SCons\Script\Main.py",
  line 1229:
    _main(parser)
File
  "C:\Python26\Lib\site-packages\scons-1.2.0\SCons\Script\Main.py",
  line 868:
    if a[0] == '-':

These errors stem from the fact that the build target was now an empty string (it gives the same result if you specify buildtarget = “” to the MSVSProject Builder). The problem occurs because SCons creates the BuildCommandLine attribute in the project and appends the contents of buildtarget to the end of that command line as the target for SCons to build (it does the same for the ReBuildCommandLine and CleanCommandLine attributes). To avoid any problems with the command lines, SCons encloses the targets in quotes (actually, " as the vjproj file is XML). This is like calling SCons at the command line with:

scons -Q ""

If you do this you’ll find it gives the same error as Visual Studio.

This means that for each Visual Studio project the correct build target has to be specified. There are a few things to note about build targets:

• In Windows, the ‘.exe’ extension is required on the end of an executable target, even if you specify the target name, e.g. Program(‘howdy’,'hello.c’) still requires: scons -Q howdy.exe
• If the target is in a subdirectory, you must specify that subdirectory in the target name. This makes sense as two subdirectories could contain files (and, therefore, targets) with the same name. For example, a program named goodbye in the subdirectory test2 would require scons -Q test2\goodbye.exe to build on Windows (and scons -Q test2/goodbye for Linux).
• If you are using BuildDir() to specify a build directory, then it is that directory which should be specified in the target name. In the mi_test_scons code, for example, to build bar.obj (under Windows – Linux would use bar.o) using scons -Q src\foobar\bar\bar.obj will fail, while scons -Q build\release\foobar\bar\bar.obj should succeed.
• When specifying a build directory for the target any error messages about not knowing how to build targets will give the source directory.

With these things in mind we can go on to write the code necessary to correctly create Visual Studio Projects from our SCons build. The projects will be built for each library and executable, but not for every object file (as happens at present). This will be the subject of the next post.

The code changes given in this post (to create actual vcproj files in the build directory instead of just placeholders) is now committed as revision 11 in the example code.

Current Repository Revision: 10

At the end of the last blog entry, it was mentioned that the next problem was to see how SCons handled having both Visual Studio Express 2005 and Visual Studio Express 2008 installed, and how to select which one to use. Luckily, SCons caters for this easily, and this post will detail the relevant changes made to the SConstruct file.

SCons always uses the latest installed version of Visual Studio by default. In the set up being tested this meant that running SCons with:

scons -Q

always selects Visual Studio Express 2008 as the compile tool.

It is possible with SCons to change the tool being used. We did this before so that MinGW could be used on Windows when Visual Studio is installed (Visual Studio taking precedence over MinGW by default in SCons) by using the switch:

scons -Q mingw=true

We can do something similar with Visual Studio Express 2005 by adding the switch vs2005 to the Options(). This requires an entry in the SConstruct file:

BoolOption('vs2005',
"Set to use Visual Studio [Express] 2005 instead of a later compiler",
'false')

Note: See this post as to the decision to continue to use the now deprecated BoolOption at present.

Now when the user enters the vs2005=true switch, we can reset SCons to use Visual Studio 2005 instead of 2008 by setting the variable MSVS_VERSION to the appropriate version number (note: Visual Studio 2005 is actually version 8.0) before constructing the environment:

# If vs2005 or mingw is set, get an environment with
# the appropriate tool instead
if env['vs2005']:
  env = Environment(options = opts,
                    MSVS_VERSION = '8.0')
elif env['mingw']:
  env = Environment(options = opts,
                    tools = ['mingw'])

As we have had to construct the Environment earlier (for the options), we could just reset the tool, rather than creating a whole new environment, by setting the MSVS_VERSION variable to 8.0 (env["MSVS_VERSION"]=”8.0″) and calling Tool(“msvs”)(env), but this doesn’t update the tool cleanly – it still leaves paths to Visual Studio 2008 in some of the variables – and I like it neat!

The final part of this update was to ensure that when a user specified they wanted to use Visual Studio 2005, it didn’t try to use our workaround for Visual Studio 2008, as SCons gets all the variables right for Visual Studio 2005 without the need to call that code:

if (cmplr=='cl') and (not env['vs2005']):

We tested what happens if you use all three tools in a standard Command Prompt, and in a Microsoft Visual Studio Express 2008 Command Prompt. The code was built and cleaned with each tool using the sequence:

scons -Q runtests=true
scons -Qc
scons -Q vs2005=true runtests=true
scons -Qc vs2005=true
scons -Q mingw=true runtests=true
scons -Qc mingw=true

In the standard command prompt Visual Studio Express 2005 and MinGW built the current code and ran the tests correctly, but Visual Studio Express 2008 failed to build the code  (due to the problems described in this post). In the Microsoft Visual Studio Express 2008 Command Prompt all three tools compiled the code and ran the tests correctly.

Other versions of Visual Studio (or any other compiler for that matter) could also be catered for, but this is left as an exercise for the reader…

The code is now committed as revision 10 (the code was originally committed revision 9, but would allow both vs2005 and mingw switches to be processed, so the mingw if was replaced with an elif in revision 10, and a comment was updated).

Note: The version of SCons used for these tests was 1.0.1, confirming that the workaround for Visual Studio 2008 works with older SCons installations.

Current Repository Revision: 8

I was visiting my parents this Easter, when I thought I’d try out the current development code on their Windows XP based computer. I installed Microsoft Visual Studio Express 2008 (C/C++), Python (version 2.6.1) and the latest stable SCons Windows installer (version 1.2.0).

After installing TortoiseSVN, I checked out the latest version (revision 6) of the example code, opened a command prompt, navigated to my mi_test_scons directory, and typed:

 scons -Q

It built the object files fine, but then I got to the linking stage, and the build was halted with the message:

LINK : fatal error LNK1104: cannot open file 'kernel32.lib'

Obviously my environment isn’t set up properly, I thought, so I searched for the vcvars32.bat file  which I have used in the past to set up the MSVS environment. I found this file (or rather its equivalent – vsvars32.bat) in the C:\Program Files\Microsoft Visual Studio 9.0\Tools directory. I ran it, and ran SCons again. The same error appeared.

Next I thought that, rather than use a standard command prompt, I’d use the Visual Studio 2008 Command Prompt available from the Microsoft Visual C++ 2008 Express Edition > Visual Studio Tools folder available on the Start Menu. Again I navigated to my mi_test_scons directory, and ran SCons. The same error appeared yet again.

I typed the link command that SCons said it was executing directly at the command prompt:

link /nologo /OUT:build\release\foobar\bar\tests\bar_test.exe
/LIBPATH:build\release\foobar\lib /LIBPATH:src\foobar\lib
foobar.lib build\release\foobar\bar\tests\bar_test.obj

This time, the link worked fine, so there had to be a difference between the environment SCons was using, and the MSVS environment.

After dumping out the SCons environment and looking at the command prompt environment, it was obvious there was a difference in the contents of the LIB variable. Specifically, the location of the Microsoft SDK directories was not being detected correctly by SCons. This is due to some assumptions made by SCons (that the SDK directories lie below the Visual Studio directories) that don’t hold true for Visual Studio 2008 Express. In 2008 Express Microsoft have separated the SDK from Visual Studio, so it is in its own directory tree.

From the SCons site it seems that this problem may be in the process of being fixed (and possibly has been fixed in the latest (non-stable) release). However, I only use stable releases, and I have older versions of SCons I want this to be able to work with, so I decided to add a quick workaround into the MereIdea code to  grab the LIB environment variable (and INCLUDE as problems may crop up there later) and use that in the SCons environment:

# If the compiler is msvs and the LIB and INCLUDE environment
# variables in the os exist, use them instead of SCons version
# to fix MSVS 2008 Express problems
if cmplr=='cl':
  envlib = os.environ.get('LIB')
  envincl = os.environ.get('INCLUDE')
  if envlib is not None:
    env.Append(ENV = {'LIB' : envlib})
  if envincl is not None:
    env.Append(ENV = {'INCLUDE' : envincl})

The above code fixes the problem with Visual Studio 2008 Express, and is now committed as revision 8 (revision 7 included an incorrect comment, corrected in revision 8). You have to remember to either run vsvars32.bat before running SCons or (my preferred option) use the Visual Studio2008 Command Prompt.

The next problem is to see how SCons handles having Visual Studio 2005 Express and 2008 Express installed on the same machine, and being able to select either to use for the build.

Updating SCons to 1.2.0 on one of the machines (puffer) caused deprecation warnings to appear with the current version (revision 6) of the repository code. The warnings:

scons: warning: The Options class is deprecated;
use the Variables class instead.

and

scons: warning: The BoolOption() function is deprecated;
use the BoolVariable() function instead.

appear at lines 15 and 19 in the SConstruct file.

There are two ways we can deal with deprecated code. Either we can force everyone to update by enforcing a specific SCons version in the SConstruct file by adding:

EnsureSConsVersion(1, 2)

or we can ignore it for now by running SCons with the switch:

scons -Q --warn=no-deprecated

It was decided to continue to use the Options and BoolOptions class and functions at present to maintain backwards compatibility, but it should be noted that at such time that these become more than deprecated warnings, or that something with more serious implications changes then the EnsureSConsVersion function will be used to set a minimum SCons version.

Well, it’s been a while since I updated this blog. As usual, as soon as you start on one project, several others come up which take precedence. In this case some web sites I had to write/rewrite/update.

I thought I’d restart this blog by giving a suggested plan. No dates, but I just thought I’d list a possible order of play. This isn’t a firm or exhaustive list of things to be done, but it’ll give you an idea of what to expect on this blog.

• Update at least one machine to the latest SCons (version 1.2.0), and fix any problems.
• SCons and Microsoft Visual Studio – including installing Express 2008 and testing with the current code and creating Visual Studio projects in a sensible way so we can build and debug using the Visual Studio Express 2005/2008 IDE.
• Add a second library to the code to test other SCons features such as the ‘configure’ style features allowing checks to be performed for header files, libraries and functions. Also add replacement code in our library which is included only if a particular function isn’t available.
• Add executable code that makes use of both the libraries.
• Add a third, shared library (DLL/.so) and an executable which makes use of the shared library at runtime.

• Create a version of UnitTest++ which is built using SCons, so we can build it easily on all platforms (as we have SCons installed anyway).
• Change the simple tests written in the repository code to use UnitTest++.
• Work through an example of Test Driven Development using SCons and UnitTest++.

• Install the latest version of Buildbot and set up buildmaster and buildslaves.
• Set up automatic nightly building and testing of code (using SCons and UnitTest++), and submitting results.
• Set up one machine to do a continuous build.

This is the plan so far. If you have any suggestion of other things you’d like to work through with SCons, UnitTest++ or Buildbot, then please let me know and I may add them to the list.

After updating the Fedora Linux machines from SCons 0.97 to SCons 1.0.1 (and correcting the code the update broke) I decided to do the same to the Windows box.

When the Linux machines were updated it was simply a case of running rpm (admittedly, using the -U upgrade option) with the latest package, and the new version replaced the old. The manual says this will happen. Reading around the installation part of the manual, it says that if I want to install multiple versions of SCons I have to do it manually with:

python setup.py install --version-lib

and this will install SCons in a versioned directory.

It also says that installing on Windows is extremely easy and achieved by simply downloading the installer and executing it. In my mind I put these two things together and assumed that running the Windows installer for version 1.0.1 would replace the existing version 0.97. It wouldn’t be the first installer to detect an existing version!

However, it doesn’t appear to. After running the installer, typing:

scons --version

still returned version 0.97.

It seems (in my experience) the way to update SCons under Windows (XP) is to uninstall the old version manually (through Add/Remove Programs), and then run the new installer.

It’s not a great deal of trouble, but nice to know before running the installer and finding it doesn’t do as expected.