RISC OS components have traditionally been written in either C, Assembler or BASIC. Because of the age of the components, many have not changed in the way that they have been built for some time. In general, there's usually a reticence to change a build system which works - if it is working it will continue to do so and the risk in changing the build is that components fail after the change, or that they build wrongly.

Version management

RISC OS has, historically, been maintained through CVS. Whilst being an old source control system, it's actually useful for RISC OS. Components were always committed as a single entity. If you changed (say) Filer, you would run the commit script to store the change, and that would commit all the changes to that component, together with an update to the VersionNum file which contained details about the version of the file, as C pre-processor defines, and tag the results with a version number.

As RISC OS is modular, and intentionally able to be mixed and matched in its versions of components (within certain constraints), this means that we can try out different versions more easily. There is an issue that the libraries may then differ with each component's version, but that can always happen as components move forward - the core code is always controlled. If you need to know the changes that go into any given component, the details can always be found in the history of the VersionNum file.

Within the build system, the VersionNum file is always used where any version is required in the code. The cmhg tool (or cmunge as was used later), could be configured to process through the C compiler pre-processor. The cmhg files could then use #include "VersionNum" to pull in the file, and thus embed the version strings in the module header. Any C code could obviously do the same thing.

For non-C code, sometimes you need to embed the version number (or date, or other information from the VersionNum, such as the branch name), in other files. Text based files were relatively easy - there was a simple tool called VTranslate which would substitute the defined symbols into another file (without processing any other C-like operations) if the symbol was enclosed in braces, eg '{Module_FullVersionAndDate}.'

For BASIC, either the commit tool could ensure that a VersionBas BASIC library was created which could then use LIBRARY within the program being created, or the VTranslate tool was capable of performing the {var} translations within a BASIC program.

All of this meant that the version numbers could always be propagated into the component - binary releases would always contain an explicit version number which could be reproduced. The component version would always give the base source, and the build date would further narrow the exact version of the tools and libraries that were used for the component. These days it might be easier to use a repository revision identifier, as these are provided by most source control systems, but that wouldn't be monotonically increasing.

Building components

As has been mentioned, there are a few major types of component which exist within the RISC OS build:

  • C library.
  • C module.
  • Assembler module.
  • C application.
  • Library command.
  • BASIC application.
  • Raw installation.

Originally every component had a standalone makefile which was edited to fit the requirements of the component. Most had a lot of common definitions - all the applications would apply similar rules for the !Run, !Boot, !Sprites*, !Help and Messages files. All the C modules, for example, contained the same default includes of the libraries and headers. Updating things across the system was a tedious process of changing every makefile.

To simplify the management of the components, and ensure that every one could be invoked in a consistent manner, I began to standardise the makefiles through common includes. Stewart Brodie had added a number of extensions to amu, of which including other makefiles was very useful for this use.

Initially the simple C modules and libraries were updated so that they common sections were in an include makefile - 'CModule' or 'LibExport' and only had to configure the very few things that they differed. The basic makefile became much simpler, reduced to just the fundamentals, rather than having anything specific in it. For example, an MD5 library might look thus:

# Makefile for MD5

# Program specific options:
COMPONENT  = MD5
OBJS       = o.md5
EXPORTS    = ${EXP_LIB}.${COMPONENT}.h.${COMPONENT}

include LibExport

# Exports
${EXP_LIB}.${COMPONENT}.h.${COMPONENT}: h.md5
        ${CP} $?  $@  ${CPFLAGS}

#---------------------------------------------------------------------------
# Dynamic dependencies:

The back end of LibExport (and other includes) would take care of how to generate the objects required and the different forms of targets that were necessary. For example the 'export_hdrs' target would export just the header files that were defined, whereas 'export_libs' would export the actual compiled libraries. A clean ROM build was performed by running first the 'export_hdrs' on every library component, then 'export_libs' on the library components, then a build with a target of 'rom' for all modules (which would compile the modules and partially link the component), and finally an 'install_rom' (which would link the objects at the correct base address for the module being installed). Once everything had been produced they could be (effectively) concatenated to make a ROM image - with a few padding and header bits added.

Only in very special cases was it necessary to know how the component was being built. Filer and Wimp were originally special because they included C code built into AsmModule type builds (modules primarily in assembler). There were originally some intermediate targets required to ensure that this C code built properly, but this was subsumed into the included makefile which made their use consistent across all components.

In general everything at this point was assuming the use of CMunge rather than cmhg, because it could manage all the functionality that was required of the modules, and because the source was available and flexible. So much functionality had been added that it became significantly easier to write many of the veneers that previously would have needed writing manually in assembler.

Once everything (pretty much - things like the Printer Drivers were not touched until much later, and they were quite complicated themselves) had been changed over the new style, it was simple to manipulate the builds that every component did.

LibExport was initially able to produce both application linkable objects and module linkable objects (the two being slightly different forms of the workspace access, although otherwise the same APCS variant). I added support for generating debug objects as well, which made for a total of 4 different variants of the same code being compiled. The CModule or CApp makefiles would select the correct libraries (if they existed) for use at link time.

When work was done on the 32bit build, this was added to each of the include makefiles (not each component makefile). If the BUILD32 parameter was passed, it would build a 32bit variant. The object directories changed according to the type of target being used in the form 'o[d][32][zm]'.

  • d - debug (otherwise no symbols present)
  • 32 - 32bit (otherwise 26bit)
  • zm - module target (otherwise application target)

This flexibility meant that it was simple to build a target for any variant - useful for libraries which were shared between many different components, and obviously vital when applications and modules needed to be produced which were capable of running on different systems.

The 32bit RAM builds of modules and applications always switched to linking against StubsG, rather than a 32bit-specific version of the Stubs which would have made distribution more difficult. Components such as the Toolbox and some other tools, had to run on all systems, and including a C library update was an unreasonable requirement (as discussed elsewhere).

Because all the components were being built through this build system, it meant that updating things so that components were processed in similar ways was very easy. The 'debug' option above was just one example where it became simple to update all libraries to create extra debug versions and the applications and modules to link against them.

Similarly when it came to testing components it was simple to update the system to produce 'Fortify' linked components. 'Fortify' is a freely available library by Simon Bullen which was very simple to incorporate into any C program to ensure that they did not leak, and that there were no buffer overruns or underruns. It seems a little harder to find on the Internet these days, but it appears to be on Reocities. Robin Watts wrote a similar library, which can be found in a GhostScript project, if you're willing to work with the GPL.

Specifying 'FORTIFY=yes' in the makefile or command line would ensure that the component was built with and linked against the Fortify library. Such builds were slower and used a lot more memory, but found a few small leaks and the occasional memory corruption during development.

It also helped enforce boundaries for libraries and modules. Because of how the build worked, the Fortify variant didn't use separate object files, which meant that if you tried to use components that were built Fortified with those that were not it would usually complain that invalid frees were taking place. Such complaints usually meant that an application was freeing data which was allocated by a library - something that is generally bad practice, as the library should be called to free its resources.

Because the Fortify system worked on modules as well as applications, it was easy to check the behaviour of many modules without the need more invasive tests. Obviously this way of checking the code won't catch problems in error paths unless they are exercised, but it at least provided more reassurance that in normal operation there were no problems.

ROM building

The ROM build, as mentioned above, does a relink of components at the base of the module as it will be in memory. This means that C components are linked directly against the ROM SharedCLibrary, both saving run-time space and being slightly faster. Assembler modules, which didn't refer to absolute module addresses, didn't require this relink stage. If a different APCS variant had been chosen for C modules, this might not have been necessary, but that was the way that things were.

The ROM itself is made up of a collection of modules which were originally just listed in the order they should be included. The actual dependencies for the modules weren't specified, but were generally known. It was quite easy to find that you had forgotten a dependency if you built the ROM and it failed to boot, or crashed in some way soon after start up. In some cases the dependencies were just simple problems with components being unable to cope when others they required were not present. As discussed in the Modularity ramble, the 'lobotomisation' exercises tended to weed out these cases quite well.

Component selection and ordering

One of the requests we had in the past from hardware manufacturers was for a cut down system which only included certain components. This, coupled with the need to build ROMs more flexibly for the 32bit environment work, pushed for a change in the way that the components were organised for the ROM build. Because the Select distribution was softloaded, the need to fix the modules in an order (for unplugging) was already violated, so changing the order shouldn't be a problem. Part of the intention for the future work was to update the unplug handling so that modules in the host OS would remain unplugged (by name) in the softload, but this was never completed.

Reorganising the modules on a per purpose basis was not particularly hard. The order that component were used started with a 'Groups' file which defined the groups that components were in, such that they could be included, excluded or collected together as dependencies. For example the sound system was defined something like this:

Sound0 {
:if ?RiscPC then SoundDMA
  SoundChannels
  [SoundVoices]
}

Sound1 {
  SoundScheduler
}

Sound2 {
  SharedSound2
}

SoundVoices {
  WaveSynth
  StringLib
  Percussion
}

# This just bundles the Sound system into one group
SoundSystem {
  [Sound0]
  [Sound1]
  [Sound2]
}

This meant that when the 'SoundSystem' group was included, it would add the Sound0 (core sound system), Sound1 (auxiliary sound components) and Sound2 (SharedSound and support system). The Sound0 group would itself only include the SoundDMA module if it was being built with the RiscPC option set.

To remove the sound system, you could just add a line to the ROM 'Ordering' description file like ':omit [SoundSystem]'. There were also dependencies that could be specified in addition to the conditional above, for example:

Resources {
  Messages
:require Messages before MessageTrans
}

which would ensure that MessageTrans could actually start up properly with its messages file. The files didn't declare all the dependencies, just those that were obvious and could be concisely described.

I'm pretty sure that the whole process of grouping things together was not well documented, and the usage of the files which went in to building it wasn't particularly clear - mainly because it was only me needing to update the build, but also it was a new system that was just starting to be used over the old-style fixed ROM build. Eventually I should have removed the old files and documented which bits controlled the ROM build.

The script which managed the dependency and pre-processing of the file was in Perl and ran quite slowly on RISC OS, taking about a minute to determine the order of the build. I had a choice to either recode in another language, improve the way that it worked, or find a way to run it faster. The simplest of these was to make it faster by running it on faster hardware - send the data to the Linux server, run it and collect the results back. This made the build ordering take about 6 seconds, which was far more reasonable, albeit not a good example of the system. The build could be switched back to native at any time, so would run on a standalone RISC OS system.

ROM construction

When we first received the RISC OS source, it used a BASIC tool called 'BigSplit1' to construct the ROM from a list of linked binaries. Essentially it was a tool to concatenate the modules, preceded by their length and then pad the results to the ROM image size. I believe that this was actually an older tool, and Element 14 were actually using a more advanced version, but for our release they only gave us the simple tool. It was pretty much all that we needed, at the time.

The only real 'identifier' in the ROM was the 'NCOS' word at the very end of the ROM image, along with some simple checksums for the ROM data. We updated this to 'RISC', to differentiate the ROM images from the 'NCOS' one, and created a tagged list of properties for the ROM image. The idea was that these properties would cover the ROM image as a whole, and allow them to be identified more readily.

The 'build' for the ROM was mostly done (originally) on a separate build machine, which meant that it always built from a clean source. We had created an NFS share for our 'Parts database'. Every time a component was released, it would be stored in the share, given a part number and some details about its content could be saved with it. As a process for tracking releases it wasn't too bad, and it meant that we could always pull out any released versions, or check to whom they had been released and why (assuming we actually updated the documentation!)

The ROM building tool was updated to store its results in the database automatically, and would add the part number to the ROM's properties, along with the build date. For internal builds there wouldn't be a part number, but the build date would at least identify the version. The properties were exposed in the Kernel through a new SWI OS_ReadSysInfo call.

Later, we used this feature of the properties to include user details in the flash ROMs we created. It was a bit tedious, but it did the job most of the time. For the softloads, knowing when the ROMs were built was very useful for identifying the original code that generated it.

Oh, I've just remembered that in the RISC OS 4 release, there were also the hidden cookie names at the end of the ROM image. The TaskManager module would find the end of the module chain in the ROM and immediately after the last module there would be a list of names to show when the hidden cookie was triggered. That code was stripped out later.

The ROM creation tool was also updated to support Squash compressed images later, as this saved a bit of space when we were using 6 MB images and the ends were empty. As the ROM filled up it was less effective (obviously), but still useful.

Around Select 4 time, I introduced padding of modules to 4K alignments, which gave a little extra space after each which might be useful if it were necessary to patch any code, and also made it easier to make pages as writable if necessary. It was less likely that this would be useful, but it provided a little more flexibility and wasn't that wasteful of space.

Back in Select 1 days I had written the beginnings of a replacement ROM building tool - cunningly named BuildROM - which could do the same job but was written in C, and far better structured. For a variety of reasons, this got put on hold, and was only resurrected when doing the 32 bit work, and then only to make it simpler to reconstruct modified ROMs for use within QEmu.

HdrToH

As I've mentioned, the build phases included an 'export_hdrs' phase which would place all the system headers for the components in a suitable location for use in subsequent builds. Many of the RISC OS components, being written in Assembler, only exported an assembler header detailing their SWI calls and structures.

This caused issues for the C based components which had chosen a variety of ways to address this. Some would include definitions in their own private headers. Others would use RISC_OSLib header definitions, even though they didn't use the actual library itself. A few would use the OSLib headers, and the library to go with them. The problem here was that the canonical definition - the one exported by the component itself - was not being used. This meant that there might be differences that went unnoticed, the source that was used to reference APIs could look wildly different between components, and if you updated one of the library headers you might find yourself having to duplicate the work elsewhere.

For things like RISC_OSLib that was painful, as I wanted to get rid of it, and continuing its use felt wrong (but also, stripping the components entirely was more work than was justifiable). For OSLib, there was a special version of the source which lagged behind the 'official' releases, but also had extra detail specific to the internal components. It took a while to build as well, which was always off-putting.

Part of the solution to this was to ensure that some of the Assembler headers could be used by the C components. I created a Perl script, which was pretty simple initially, to convert the assembler header definitions to C header definitions. It wasn't a perfect header parser, because the ObjAsm format was more complex than I needed to care about - only a subset of the language was needed for the header definitions. Although it understood a lot about structures, for simplicity it only exported values out of the headers.

In order to support components like !ChangeFSI, which was written in BASIC, I also created an 'export as BASIC' option as well. I'm really quite pleased with that option - the BASIC code could be written just plain format and it would be tokenised within the Perl script. So, for example, a call like 'write_basic("DEF PROCinit_$header");' in the Perl (where $header expands to the header name) would be tokenised as it was written out to the file. I'm pretty sure that the tokeniser supported all of the BASIC tokens, with the exception of the line number based tokens, as they were more complicated. It was certainly good enough for these headers.

Later still, whilst I was tinkering with the Pascal compiler tool, I added support for exporting definitions in Pascal format. It wasn't that useful, but helped to ensure that the Pascal modules that I wrote could share the same headers.

CMunge

Robin Watts wrote the CMunge tool as a response to the lack of facilities in the cmhg tool. It was intended as a free alternative which could be updated more readily and provide the extra interfaces which developers needed.

The CMHG (C Module Header Generator) tool was a companion to the compiler for use in creating RISC OS modules. It provided a means of generating the interface veneers that made C code work with the bare interfaces that RISC OS required for modules. Because the interfaces provided by the system were quite varied (SWIs and services were common, but some vectors had different interfaces, registration routines sometimes used different APIs, etc), these regularly needed different APIs in order to function.

Acorn had provided CMHG with version 5 of the compiler, allowing developers freedom to build modules more easily, without the use of assembler.

I don't remember the exact reasons that CMunge came about, but I know that there had been frustrations with having to write assembler veneers for common RISC OS APIs, which CMHG did not support. CMunge took a slightly different approach to constructing the header file to that of CMHG. Whereas CMHG had binary code fragments and a few modifications for the specific use cases, CMunge generated assembler text fragments with substitutions for the use cases. Where CMHG just wrote out the binary file, CMunge would call out to an assembler to produce the relevant object files that could be linked to provide a final module. Originally the CMunge used the objasm assembler, but later support was added for other assemblers.

I added a few features to CMunge, and eventually took over a lot of the maintenance when I made it the header generation tool of choice for RISC OS development. At that point there was nothing that the CMHG tool offered which CMunge couldn't do, and there were many features that CMunge had which CMHG could not do itself. In many cases these were accelerations to make it easier to write certain code, but also features like automated header file (ie .h) generation for SWI identifiers made it more useful in a build environment.

One problem that came up regularly in the building of components was that the exported headers needed to have access to the SWI definitions from the module. These were defined in the header written by CMunge (or CMHG) when the '-d' switch was specified, and could be used by the internal module but were unsuitable for use outside the module. As mentioned, the module building process produced headers which could be used by external components, and it is generally useful if the SWI names are exported with them. The earlier scheme, which Acorn employed, of placing all the SWIs you would ever need in '<swis.h>' wasn't really sustainable.

A little script I wrote, called 'MergeModuleSWIs' would take the CMHG header output, extract the SWI definitions, and write another file, based on a template. This meant that the export process could produce a header file which was correct for the module, including structure definitions and any constants that its API required.

ResVersions

Many of the applications were written using the Toolbox, which made their prototyping and maintenance significantly easier. During the development of the versions of Select, and through the different releases of the Toolbox that went out publicly, it was sometimes difficult to remember which Toolbox modules were actually required for a given application. In general, the list of modules needed for any given application would be loaded when the application started, using a sequence of commands to check that they were the right versions.

However, what actually tended to happen was that this list of modules and version would be created when the application was first written and then forgotten about. Because (generally) testing the application happened on the latest version of the OS, with the most recent modules present, any issues did not surface until later testing. Usually this was when the system was being tested in legacy configurations, for example running !Configure or the supplied applications on RISC OS 4, or any of prior Select releases.

In general, the applications were expected to work and the upgraded Toolbox should have been able to work with any component (assuming that other support was present). Problems lay in the particular versions that were required and whether the dependent modules were loaded at run-time.

For example, an application might start out simple enough and have its !Run file ensure that the right modules were loaded. Later it might have a Text Area added to it, but the !Run file's contents would not be updated to include the new dependency. This would mean that on systems that didn't have that module present (either because they were old or because UnplugTBox had killed the module) the user would be presented with a message something like 'Unknown gadget type &85400' (I don't remember what it was, but it wasn't very obvious). When that sort of thing happened it was confusing for an end user and frustrating for developers.

One solution was to diligently check every single application prior to release with a clean system to ensure that they started. Another was to always know which features you were using, and update the !Run file appropriately. Both are admirable and sensible solutions. However, I wasn't keen on taking on such levels of checking, and in any case even they might miss a version number as they were generally a manual process.

Instead, I adopted a solution similar to that of VTranslate. A new tool - ResVesions - would take a Resource file and would determine which objects and gadgets were present. With a mapping from the identifiers to the names of modules these could then be ensured sensibly. The command would apply dependency rules to ensure that the order in which the modules were loaded matched that for which they were required - for example if the ColourSwatch gadget was required by the Res file, the tool would know that both Window and ColourDbox were required for it to function and add those modules as dependencies.

The versions and dependencies were configured in the command at the level of Select 2's release (which included a public Toolbox release as well, so was known to be widely used), but could be overridden by a configuration setting so that a custom definition was used. The ROM and applications build used a fixed configuration file which was updated at the end of the release to ensure that, at the point of release, everything that was required was explicitly specified.

The build system itself would perform translations on the !Run file as the applications were built, substituting the relevant commands generated by ResVersions into the file that was released.

One minor downside to this way of working was that the version numbers required were always going to be higher than those that were strictly necessary - the application might not need those versions of the modules. However, it was guaranteed to work with them, which was the goal of the exercise.