Toolbox

Background

The Toolbox was produced by Acorn to try to provide an easier development environment, as well as faster turn around for interface design and prototyping using the resource tools. At the time when it was introduced - RISC OS 3.6 - it was still a little flakey, but generally worked well. However, the response from the enthusiast and commercial developers was not great. Many were still focused on supporting all versions of the Operating System from RISC OS 2 upwards, and even those who used RISC OS 3.1 and higher could not justify the 300K or so that it would take to load the Toolbox modules.

In addition there were many, many comments that the Toolbox itself could not be used from BASIC. It can, but it is not as easy. This complaint continues to this day. At the time, I believed that the reason for not making it easier was that BASIC had to stay the same so that it could be used on all systems. BASIC could not be changed to help Toolbox in that way, and the Toolbox interfaces were already in as simple a set of structures as they could be that they could not be made much easier to use from BASIC.

A few people had (and continue to) complained that there isn't an equivalent of the 'event' library for BASIC, but... that's relatively bogus. The amount of code needed for such a library is relatively trivial - BASIC not providing an easy way to register functions as callbacks (other than by the EVAL("FN"+func$+"(args$)") trick) was a greater limitation there. The lack of dynamic memory allocation was another area that might be considered a problem. Neither of these are at all insurmountable by libraries, and a few people did produce such things.

My view when doing work with RISC OS itself about making Toolbox more accessible to people were similar. Changing BASIC was, again, not an option because that wouldn't really benefit the situation - developers would then have to restrict themselves to a particular version of BASIC which wouldn't be present on earlier systems. Creating additional libraries to make it easier to use from BASIC never seemed like a good use of time. On the one hand, anyone could do that - it certainly didn't need to be anyone with direct access to the source. On the other hand, anyone who had the ability to do so would recognise that BASIC just isn't a good language to be using for such things in its current state. In some respects, I didn't want to promote the use of BASIC as a suitable way to work with the Toolbox - the result would always be second-class because of the restrictions in BASIC, and it could never be supported as well as the intended interfaces in C.

That isn't to say that interfaces couldn't be automatically created for BASIC. The !MethodGen tool could have been updated to create the necessary libraries, and much of the detail from the C headers could be parsed (in a similar way that I did with ZLib/PNG) to produce BASIC libraries. The continued feeling of wrongness which stemmed from the fact that restrictions in BASIC continued to hold back much development of APIs and the Operating System was a factor there. A bigger factor, really, was that it just took time away from other features.

It was all considered at the time, and on a few occasions I tinkered with some automatic generation tools, but they were abandoned in favour of other more important things. There was also the issue of Assembler developers. There were a few people who said that it was very hard/impossible to write using Toolbox using Assembler. In the politest of ways, I Just Didn't Care. Back then, I may have said it in a different way, but I realise that it should have been stated more clearly - get real and develop in a proper language, and keep your assembler for the things you need to do fast.

These days, I am still pretty much of the same opinion, except that it's easier to see with retrospect that the issues with BASIC should have been addressed far sooner. I had investigated alternative solutions to BASIC as the primary language - particularly JavaScript and Lua - but whilst these would have worked (to varying degrees) the general feeling that I got from most people was that acceptance would be very low. Updating BASIC would have had equally poor uptake, partially because users expect to keep systems working for so many years.

I don't know if the RISCOS Ltd or Castle developments have addressed this in subsequent releases. Various changes have always been proposed for BASIC, but I haven't (on the very limited viewing that I've seen) seen any movement on making Toolbox easier to use from it either through libraries or other improvements. I know that Steve Drain provided some support through his BASIC extension modules.

Distribution

The Toolbox modules were one of the few components which was always intended to be freely available to all, and should work on all systems from RISC OS 3.1 upwards. This was later restricted to RISC OS 3.6 in later releases - code would be written to use the facilities of RISC OS 3.6, and no attempt would be made to retain support for RISC OS 3.1.

Because of this, testing of the Toolbox was more difficult than any other stack. Bugs in earlier versions of the modules and the supporting systems (WindowManager, FilterManager, ColourTrans, SpriteExtend, Kernel, etc) had to be noted and checked when making changes. Building the modules for 32bit wasn't too hard once as most of the custom assembler veneers had been replaced by those which used CMunge which took away all the work there. Other custom veneers had been moved into a general assembler support library which meant that they were only updated once. Finally, the changes to the build system meant that just setting BUILD32=1 on the command line would produce a 26bit/32bit compatible version using StubsG which was guaranteed to run on any system without additional support modules.

I think I did 3 major releases of the Toolbox stack, together with the supporting libraries. I cannot remember how well publicised they were at the time, though. There are certainly bits in there still to support earlier Wimp by emulating the rectangle filters (or something similar - I forget exactly what it is).

The tools that come with the distribution - !ResEd, !ResTest, !ResCreate - were included in the Select 3 distribution (if I remember rightly). I am pretty sure they weren't released to the public. Although I had asked if we could do this a few times, and suggested it would be better for encouraging people to do so, we didn't do this. I think (and here my memory may be more shaky than usual) the reasons were two fold - licensing and competition. From the original license agreement RISCOS Ltd had to distribute things with the Operating System, although there were some relaxations of this agreed with Pace. Later, when dealing with Castle I don't remember much about the situation, but I think it is unlikely they'd have wanted them to be published - they had already tried to quash the public Printers+ releases. For competition, it comes down to a similar thing - it was part of RISCOS Ltd's raison d'être to not compete with other developers.

I am pretty certain that towards the end I had personally given up on trying to keep to the spirit of that when it came to Castle, after the amount of problems they had caused, their blatant disregard for licenses and bullying tactics. That said, what I felt wasn't what mattered as I didn't call the shots on what went out.

Applications

Building applications using the Toolbox was relatively easy given the libraries that existed, and the ease of writing a skeleton application which would use the Resource files that had been created. Providing a basic design of the application in !ResEd, and then testing it with !ResTest made the initial prototyping process very easy.

Some applications were designed purely through Resource files, passed to others for review, revised and redistributed without any code having been written. The Resource files that showed an expected interface - together with a text file that described the interactions that were not able to be shown with !ResTest - could very easily show both what the application could do and how the user would interact with it.

This review process could then overlap part of the code development as well - whilst the interface was being revised the code that used the Resource could easily be developed. Modular design for the functions of the application meant that attaching the functions to events from the Resource file was very easy to do. As discussed in the 'Development tools' ramble it would have been nice to incorporate the attaching of handlers at the Resource design stage, but that was still a 'nice to have' rather than a requirement for the environment.

The example applications that were supplied with the Toolbox by Acorn were still something of a frustration. They varied from simple examples which showed the handling of specific Objects, to a full blown hyperlinked application, with very little in between. This had been a criticism from a few developers, although the question of how you bridge that gap had not really been answered - maybe Castle's distribution did something more to address that.

In any case, by using these as an example it was generally very simple to get an application up and running using the Resource file. Because the Toolbox could handle many auto-open events itself (for example, a click on an IconBar icon could open a window) without any interaction from the user, such applications would not need to do much other than observe the creation of objects and record their handles. Multi-document applications were more complex, but that was only to be expected. The automatic functions of the Toolbox were still usable, but needed to be managed better by the application. In many cases of applications written with the Toolbox there wasn't much need for complex handling, but it could handle them just as well.

Most of the new applications which were developed for Select used the Toolbox rather than plain Templates and simpler libraries. Where applications already existed using Templates, they continued to do so - it wasn't worth reworking existing applications where it wasn't vital. Consequently applications like !Paint and !Draw were not updated, but new tools like the !ImgViewer were written from scratch using Toolbox. This policy had frustrations, such as when an application might benefit from features in the Toolbox but instead had to be implemented by hand. But that would be the case with any set of libraries you chose to use.

Installation

There were restrictions on what could be done with the Toolbox due to the way in which it was used. Because it could not be restarted once running (a common problem with many modules that retain state), it meant that upgrading the Toolbox could only be done when there were no Toolbox applications running. This prevented any of the 'core' desktop components from being written using it. Nothing that RISC OS starts by default should use Toolbox, as that would prevent running the installer which would install a new version of the Toolbox.

I have a vague feeling that the !HomeFiler violates this rule, though, which is frustrating. I'm pretty sure that the installer for the Toolbox modules was written so that it should be able to run with any of the Toolbox modules provided with RISC OS 4, which meant that even with the modules in use it could still install updates. The alternative there would be to rewrite the Toolbox installer to not use Toolbox itself. The Toolbox installer was just a version of the !SysMerge tool from !Configure which merged in the necessary Toolbox modules, I believe.

Again, the installer had to work on all systems so had to be built to work on any version of RISC OS from 3.6 upwards. Not hard, but an extra thing to test.

Self-extracting archives

The last version of the Toolbox updates were provided as a self-extracting Zip archive, if I remember rightly. This was a little bit of fun work that I had done to try to avoid the problem of not having an archive decompressor supplied with the Operating System. By providing a single executable and having it extract all the necessary components without the need for any third-party extraction tool, it should have made the upgrade a little more accessible and less reliant on others. I am not entirely sure it was successful, but I think the goal was right.

The self-extracting archive ('SEA') system which was used by the installer could be customised to extract any Zip archive and perform actions on the contents. At its simplest, a !AutoRun file would be run once the extraction was complete, and a license could be presented to the user for acceptance before the installation is performed. There was an SDK as well, which allowed the desktop front end to be augmented or replaced entirely. The resources used by the SEA could be internationalised as well by storing internationalised messages in the archive.

The intention was that this tool would be made available to developers to make it easier to supply applications and upgrades which didn't require that users had archive extraction tools. Having a license agreement shown before installation would also make it clearer what terms the software was supplied under. As the agreement was supplied within the Zip archive as a plain text document, you couldn't argue that by extracting the SEA manually you had not seen the agreement - it would be extracted at the same time.

In some cases it would be necessary to extract the files without going through the desktop installation process - either because the installer was broken, or because the installation was to be performed on many systems (eg a site installation). In these cases the SEA could be run with options to either just write the Zip archive to a file, or to perform the extraction to a directory without any desktop component.

Whilst the SDK supported password encrypted archives, the basic distribution of SEA did not - partly this was because there were still hoops to go through when dealing with encryption algorithms, and partly because it wasn't needed.

The actual installation was pretty easy and it worked quite well. As well as the license for acceptance, the installer also included a manifest of what files were going to be installed, which it displayed first.

Toolbox installer in use
Running the toolbox installation. (Animated version (1175K))
Spliced together from two videos, so looks a little disjointed in the middle.

Structure

The Toolbox stack is made up of two major classes of module, above which sits the Toolbox module itself, which manages the interfaces to all the modules. The primary parts of the Toolbox are 'Objects', which are usually the visualised components within the desktop - such as windows or menus. The most generic of these are the Window and Menus objects, which provide a means for controlling the native windows and menus provided by the Wimp. Most other Objects are implemented in terms of the Window or Menu objects, but that isn't required - ColourDbox is implemented as an interface to the ColourPicker, which is itself a managed form of native window, and FontMenu is implemented as an interface to the menus provided by FontManager.

The Window object is flexible, and allows a number of Gadgets to be placed within it. Each Gadget is provided either by the Window module natively, or by extension modules. The core set of Gadgets include writable fields, display fields, labels, draggable sprites, sliders, option and radio buttons and generic 'icons'.

The toolbox, stacked up
The Toolbox stack; my modules in yellow, 3rd party in grey, modules not part of Toolbox in rectangles.
(Larger version (90K))

The Gadget extension modules themselves work in a similar way to the Objects, and can be though of in the same way. Many gadgets are actually just combinations of other gadgets - the Slider gadget can include a display field to show the size of the value, and each component of the slider is made up of the other gadget types. The first extension gadget that was generally used outside of the native ones was the ToolAction gadget, which allows specific operations to be triggered for a button press - opening another window being a typical example. Its functions include more than this, but it provides greater functionality than the original Action button.

The second gadget - more well known, I think - is the TextGadgets gadget, which provides 3 different types of gadget. A ScrollList provides a way to present a number of choices without using a menu. A TextArea provides a means to present or edit sections of text. And a Scrollbar provides a way to show a scale or move through a sequence in the same way that the standard window scroll bar does - it is a naked scrollbar without any window work area.

UnplugTBox

Because the Toolbox was supplied both in ROM and with updates on disc, it was one of the more difficult systems to ensure that the correct versions were running. Originally Acorn supplied a tool - UnplugTBox which would unplug the Toolbox modules from the ROM if later versions exist on disc, which prevented them even initialising on start up. This had the problem that it relied on the position of the modules in the ROM - which certainly wasn't guaranteed on softloaded systems - and left the machine without any Toolbox modules if the machine wasn't booted properly. It did have the advantage that it ensured that any module that was ever broken in ROM could be prevented from starting. There was only one instance of this, to my knowledge.

In fact, that particular case necessitated one of the more complex changes to the 'unplug' sequence. Instead of unplugging the modules (flagging that they should never be initialised), the UnplugTBox had been changed to just kill the modules as part of the boot sequence. When the later applications were started, they should check for the modules and when they are found to not be resident the modules would be loaded from disc.

The UnplugTBox tool has to deal with the other direction as well - the version in ROM might be later than the version on disc. Normally you would not worry about this, and earlier versions of the tool didn't worry, but when dependency processing was added it was possible that a module needed to come from ROM rather than from disc.

The primary problem that caused this way of working was that the Toolbox module provided memory management for many of the modules. This has the advantage that it keeps all the memory together. The problem, though, comes when modules don't free their memory properly, or retain pointers to the old memory when Toolbox restarts. The Window module had some similar problems with its gadgets - if they didn't pay attention to the Window module being restarted they would get into a bad state.

I went through a few iterations of fixes, including a ROM patch to avoid the issue before concluding that working around the issue was impractical, and there was a sequence which would always be safe - killing everything in order and loading the right versions on start up. This had the disadvantage that the entire stack might be loaded on start up, but was safe. It would also have the nice side effect that any third-party applications which failed to RMEnsure the right modules would find that they were already present, which would prevent the 'unknown object type' error messages which would appear otherwise. All the modern Select applications used ResVersions so should always check for the right modules.

The tool was updated so that it worked out the modules that were running, those which were in ROM and those which were on disc. A table provided a set of dependencies for each module so that for any that needed to be restarted they could be done safely.

So if, for example, the ColourDbox module was newer on disc, the current version would be killed and the disc version loaded. On the other hand, if the Menu module was newer on disc, the ColourMenu module would be killed and then the Menu module - as ColourMenu depends on Menu - and then Menu would be loaded. Depending on whether there was a newer version of ColourMenu on disc or in ROM it would be loaded or restarted.

The Window module being outdated would result in all the Window gadgets, and any dependant Window objects being restarted. The Toolbox module - the one with the particular problems with memory allocations - would cause every part of the stack to be killed and restarted, in dependency order (which avoided the problem that we had seen).

Like much of the boot sequences, the tool was reasonably commented to explain what its is doing and why. It is unlikely that there were many users who would care about the reasons for the change, but the general policy that the boot sequence must be clear about its behaviour followed through to the tool.

Djinn

For all the projects that were successful, like the filer thumbnailing, or clipboard integration projects, there were many that never made it. Sometimes this was because the project weren't as important as other things; sometimes it was because they could never justify the effort that would be needed. Other times they were just plain crazy. I have mentioned some of the projects that worked out, and some of those that didn't - there are a whole slew of planned projects in the some of the later rambles.

Djinn was one of those that didn't make it, due it being too complicated for the expected gain, and I viewed the likely acceptance factor by users to be very low. The goal was relatively simple - to provide an alternative language to the ageing BBC BASIC, providing direct access to the desktop, and reducing the threshold for writing full applications.

BASIC had long been the language which was used for Acorn systems, but aside from tweaks it had not been changed in any significant way since Arthur. This left many problems with it, such as the string limitations, the lack of any structures and the lack of direct integration with the desktop.

In particular, the barrier to entry for Toolbox developers tended to be that a C compiler was required. You can write Toolbox applications as a BASIC author, but it required at least a passing understanding of C, and in any case, the toolbox headers and libraries were only available with the C compiler package. This latter issue was addressed by the release of the 'StubsG' release, which included these for free. The lack of documentation was still an issue, but the relative accessibility of the Toolbox manuals through the PRMs CD meant that this did not block such development as it had.

Uptake for Toolbox, even amongst C programmers was still low. C++ libraries existed, which tried to encapsulate the object structure that Toolbox provided, with varying degrees of success.

I had wanted to make some significant changes to BASIC, but wasn't keen on changes which might break earlier compatibility. The oft heard cries in response to just about any change was that if the feature wasn't available everywhere, you couldn't use it. This continues even still, and it remains a irritating people complain that a feature isn't available on RISC OS 3.x, so they cannot use it.

But I digress... the goal of Djinn was to try to fill a little of the gap, and make it simpler to write desktop applications, which happened to use the Toolbox. I had a good respect for JavaScript, and had turned it to many good uses internally, so wanted to use it as the language. The language lent itself reasonably well to object manipulation, and a small library could easily be written to provide interfaces to Toolbox objects.

Some of the prior objects created for the JSSL module were reused, allowing the basic system calls to be performed, and interfaces to commonly used RISC OS APIs added as objects - for example modules, graphics and sound. The Toolbox itself would have APIs defined, initially as interface descriptions (which were easily derivable from the interface definitions that !MethodGen used). Eventually, the Toolbox objects would themselves provide a introspection interface which would allow accessed objects to be defined by their implementing module.

I had previously written an interface definition processor for JavaScript when I was working for Picsel, so writing another wasn't all that hard. The initial versions worked reasonably well. Objects would load and be auto-created. Script callbacks were triggered for the object events that were received. Object specific events needed a little more complex handling at the interface level so that they were received by the script.

The JavaScript application would provide a Resource file, and it would be loaded and the script executed to handle the events. Iconbar icons could be created, windows opened, and events would be called on the objects.

There were still a lot of work to be done with it; reflecting the inheritance order in the objects was partially implemented, but functional. Event delivery was similarly functional but hardly polished. I had a lot of little things I wanted to provide as interfaces in addition to Toolbox. File access was already provided, but it would have been very useful to provide direct access to Draw primitives and some of the other more useful internal modules.

As I have suggested, though, the amount of work required didn't really justify the time. It would have been far more fun than some of the other things that needed to be done, but I didn't feel the gains from it outweighed the work needed. That said, each time I wanted to quickly throw together an application I found myself cursing that I hadn't progressed further to the point at which it would have been useful.