Filer

Working directory

Back when we were starting RISCOS Ltd, we travelled to Cambridge to collect the source code. We waited all afternoon, sitting downstairs in the Element 14 demo area. People went home and we were still waiting. Behind the scenes there was stuff going on - legal back and forth, and some other bits and pieces that needed to be sorted out. I'm pretty sure I knew some of it at the time, but it's gone from my memory now.

Finally, stuff was signed, people were happy, and we were able to take away a wodge of paper, a CD and a floppy disc which had some extra bits on it. It was very exciting, and as Andrew, Matthew and I went home it was a strange feeling that we were actually going to be doing things with RISC OS - something I had been wanting to do for so long. I remember that more sharply than I might usually because it is now inextricably linked with complete terror. Not because of the job ahead, but because of the view ahead... as we went home from Cambridge to Stratford it snowed. It snowed a lot. Outside the car was filled with white, the road ahead was barely visible. Though we weren't going that fast (well, it probably was faster than we should have been, given the weather, but that's excitement and youth, but it wasn't silly) it felt faster because of the sheer amount of snow falling.

Fear and excitement are always a good start to an evening; or a job. Leave the disappointment and bitter recriminations for later...

Anyhow, I'm pretty certain that the first thing that I got to play with - probably that evening when we got in - was Filer. Building up a module for the first time and running it was nice. RISC OS build system wasn't obvious, although we'd had a good introduction to it previously (or maybe that day - I don't remember if it was over multiple visits).

Having got the module up and running, there were a few options which were left disabled... and I can't help it, I'm a born button pusher. Some of the options were old, or very experimental, or controlled features that had been abandoned for other methods. One allowed the 'set working directory' option to appear on the menu, and it was pretty useful. Consequently it remained enabled, and we were so used to it that it became a feature. I think we may have moved its position in the menu, but that is all.

I didn't like that because turning on an option that had been previously left disabled, and which you did no work on, and then claiming it as a feature, is a lot like lying - you don't deserve it. But then - as I have said many times - we only finished RISC OS 4. Most of the stuff was already there, and we were just taking it that short distance to be usable. On the other hand, there were some larger things that got fixed.

Directory sorting failed when there were more than a thousand or so objects in a directory, which just needed a limit to be increased. As the size of the strings in the directory exceeded 64K the display would begin to corrupt, because a 16bit offset was used for the string table in directory entries. Alt-click renaming wasn't reliable in some circumstances. Lots of little things - and I'll stress that I didn't fix any of those problems (although I did hit a few others that aren't worth mentioning).

Transient selections

Once RISC OS 4 had been released, we worked on a few experimental features during the small lull before we moved on to some real development for the next release. The idea was that some of these would work out and some wouldn't, but we would get a chance to do things that were less likely to work out. Essentially this was the model that I ended up using for a lot of my own projects, as it allows you to play without worrying if it was ever going anywhere. If they did, they could be tidied up to a standard where they were usable.

One of my little things was the 'transient selection' feature in the Filer. I'm not sure that's the most friendly name for the feature, but it is the name that I gave them. As you drag a selection over objects in the window, they would highlight to indicate the state they would be in if you were to release the mouse button and complete the selection. Similar to the same operation in other systems, really. I seem to remember that the initial version was very easy to implement. It used a second flag pair on each item which indicated whether the items had been toggled selected or deselected by the temporary selection.

Whilst the selections were in progress, Filer modified the SWI Wimp_PollIdle period so that the updates happened pretty responsively. It worked rather well, and I made it a permanent option with '-TransientSelections'. Most of the responses we'd had from shows had been that people wanted to be able to turn off features if we did anything. The 3D Wimp had been successful in part because it was configurable, so it was sensible to continue in this vein.

Numerical sorting

One thing I didn't bother to make an option was that of numerical sorting - instead of sorting by the bare collation order, numbers would be evaluated (in base 10) and sorted accordingly. This made directory listings significantly more useful if you had a list of files in the directors which were numbered, as might happen if you were working on (let's say) the Wimp. That wasn't the main reason, but it helped. Really it was a way to try to improve usability by making things more obvious.

Having files that are ordered '1, 10, 100, 11, 2, 20, 3' doesn't really make anyone's life easier. I still remember one person who emailed to say that they were never going to use RISC OS again unless the option was disabled. I may be bloody minded, but whilst it made me sad, I still didn't think that it was a reason to disable it. Not sure if that was a wrong decision - I'm not sure if other systems offer the option to disable that way of sorting if they offer it. And in any case, if someone is so likely to leave because of a tiny thing like that, you've probably lost them anyhow.

The saying about pleasing all of the people all of the time is true, and you can find yourself turned about in all directions if you listen to every single demand. There's an important balance between what people want, what people say they want, what you can give, and what they need. I know that cliche is usually just the first and the latter parts, but the middle two are as important - people say different things to what they want, and it's utterly impossible to provide for some things if you have to work within other constraints.

Dynamic File Menus

After we had released RISC OS 4, I tried adding dynamic menus to the 'File' menu. The idea had been mentioned many times by people as a useful addition, but I have always been resistant to having dynamic menus. I like the idea, but the usability is generally compromised if you cannot learn actions because they change with the situation. The standard RISC OS mechanism of using shaded menu items when the operation is unavailable or inappropriate, works well for anyone who uses the system regularly. If menu items were removed, then it becomes harder to learn the sequence to follow.

The alternative is to present all items shaded, but if there are many possible actions which are just unavailable at the present time, the menu becomes large and unwieldy. Anyhow, my reservations aside I wanted to try out the behaviour to check that I wasn't being overly cautious, and to explore how such items might be handled.

The experiment was a pure hack, just to see how things might fit together, and would never see a production build. To reduce the problems with the menu changing its main operations in a way that would confuse anyone who had learnt the operations, the new items were added to the end of the Filer menu, with a divider separating the standard operations from the new operations.

The principle (which I think I based on how !FilerPro handled its dynamic menus) was to issue a service call when the menu was being created, and any clients which showed an interest would add their entries by calling a SWI. The service supplied a simple indication of the type of menu (file, directory, multiple objects, or no object selected) and the filetype to which it applied. Each SWI call added a single item to the menu, which could be faded as the client felt appropriate. This still offered the opportunity to not add items, but initially I felt it worth trying both ways.

If the menu item was selected, the code pointer registered would be called. A callback parameter could be passed, along with the workspace, so that the code would be called in the right environment. There was no handling for !Help messages or the like - it really was quite a hack.

I threw together a few modules that provided simple handling for menu items to see how it felt. A 'Share' option (this was before Sharing directories was a standard option) and a 'Zip' option were added and seemed to work relatively well, albeit a little inflexibly. Ideally there would have been a module which provided a nicer interface, so that you could define the operations in a configuration file or similar, if I'd gone further with it.

I was relatively convinced that adding items to the menu in this way could be used in a useful way, so long as people didn't add hundreds of items, or made them completely dynamic so that you had to read the menus carefully to check which items were present this time. However, I never took the work any further.

Sharing directories

When using ShareFS it was regularly useful to be able to quickly share a directory so that it was accessible on a test machine - build component, make available to test machine, load component, see that it is broken, reboot test machine and fix component on main machine whilst the test machine reboots. Similarly such things are useful if you are working with other people and you want to make something available for them. As there was no way to manually set up shares except by the command line, it seemed logical to add the option to the Filer menus so that it was available for any directory.

The Filers (ADFSFiler, RAMFSFiler, etc) supplied their own sharing options which they invoked easily - allowing the entire disc to be shared. A new 'Share...' option was added to the menu and made shaded unless the single object selected was a directory (or image). This would open a little dialogue offering the parameters for sharing.

It had been my intention to extend the API so that the sharing was extensible, allowing other clients to provide share points. However I never got around to this - it's quite a bit more work to get right. The API was simple enough, though. A simple ShareDir message broadcast by any task could be picked up by the Filer, and offer the option to share (or unshare) a given path. I had hoped to integrate Samba Server's sharing into it (somehow) - it's very complex so providing a simple interface would be handy.

At the same time, the 'directory' sprite and its 'small' and 'open' friends were copied to create 'shared' versions, which looked the same but had a little 's' swash applied to them. I had considered using overlays on the sprites, but this added complexity to the rendering (and in particular, dragging) and would have prevented any more general changes being applied by users. I did go back and forth on the overlay thing over the years, especially after talking to David Thomas about his little filetype icons that he placed on thumbnails in !PhotoFiler.

ShareFS would issue a new service to help Filer out in changing these icons as each directory was shared. The service was similar to the UpCalls for filesystem changes - in some respects it might have been useful to use an UpCall, but I didn't really think of that at the time. I'm certain now that an UpCall would really be the right solution, but oh well.

There was also a nasty collusion by calling SWI Share_IdentifyShare to explicitly check the state of a directory. That could have been handled better really.

Dragging objects

When you drag a file you would get the file's type sprite dragged. I have a recollection that I changed this so that it used DragAnObject (instead of DragASprite) which allowed an arbitrary function (or SWI) to be called to render the object being dragged. This allowed the dragged object to include the name of the file for a single object. For multiple objects, the drags had previously just shown a little package, but with the change in the way that the drag worked, this allowed me to add a count of the number of objects being dragged.

This had the fun side-effect that to count the number of objects in the window you could just select all items and then start a drag. Of course other systems had a status bar, or similar, which told you these things, but it was an amusing way of getting around the fact that we did not.

Later the DragASprite module was updated to support transparent drags, so the filer drags became transparent for free (assuming that the option was enabled in the configuration).

Thumbnailing

Thumbnailing in the Filer was a major feature that many users had requested over the years. Mostly I had held off implementing this because David Thomas' !PhotoFiler was a commercial product which provided the same functionality, and the OS was there to support such things, not take them away. I had a brief chat with David to check that he was ok with implementing the changes and he was quite happy with it.

The important thing to remember about the thumbnailing in Filer is that it's not a 'simple' change to the Filer to do so. At least not to do it right. David knew this, and had numerous workarounds for failures in the graphics system to ensure that what was done was safe. Working around, or patching, such problems wasn't an option if it was to be a feature implemented within the OS itself.

A lot of preparation had gone into this, fixing problems with the graphics system from the Kernel through ColourTrans to SpriteExtend, adding support for different JPEG rendering formats through CompressJPEG, and many other formats via the ImageFileRender interfaces. Sprite handling had to work properly for all depths of images, so that it was possible to select the images that were generated, and dragging anything around the screen had to look right.

Many of these would have been done in any case, as resilience of the graphics system, and the ability to draw images properly, was a core part of the system, but the fact that this would be a core part of the Filer's operation as well meant that it had to be solid.

There would be times when the thumbnailing wouldn't work out right, either because there was some latent issue with the components that hadn't been found yet, or because one of the ImageFileConvert/ImageFileRender modules failed. To help this, environment handlers were added so that if an error was generated through the task's Error handler it would be reported and the Filer could continue working.

Previously, such failures would mean that the task would exit and there would be no obvious way to restart it. The usual way you would restart the Filer - *RMReInit Filer then *WimpTask Desktop - isn't exactly obvious or friendly.

The thumbnailing itself was performed 'in the background', in the sense that it was done on null polls when the system wasn't busy. Because some operations were likely to take a long time whilst we generate the thumbnails, there is a limit on the amount of time that's spent generating them. If that's exceeded, we return for another poll. If we've still got time, we process more thumbnails.

As with so many of the changes, this was all done in C, which made the general algorithm far simpler to set down and maintain. The thumbnailer could be configured to generate thumbnails for the current mode, which would be faster, but might look poor if the mode was changed. It also had a configuration option to prevent very large files from being thumbnailed. The idea was that we avoid long delays when rendering - and when loading, as the disc interfaces on the systems were... slow.

Of course, what really mattered was the type of the file, rather than just its size. A 256K DrawFile could take significantly longer to render than the equivalent sprite file. I thought about making it possible to configure the size limits on a per type basis, but never progressed beyond that - there's a limit to the amount of configurability that most users want to handle.

Thumbnails which failed to render were marked with a little 'warning' icon to show that they hadn't been able to be rendered, usually because of an error. Those which were pending redraw used a 'working' sprite, which was intended to be updated to a better name which could be used more generally. I don't think I ever got around to changing it. The idea for these came partly from necessity, and partly from the prior behaviour of both !FilerPro and !PhotoFiler.

The actual thumbnailing itself isn't all that complicated, despite being managed in C. The idea of doing it in C was that the algorithm could change as needed, which made it far easier on me when maintaining the module. The rendering is (mostly) performed by plotting the image using ImageFileRender at a suitable scale to a redirected sprite. ImageFileRender is queried as to whether a mask is required (this is one of the flags that the interface provides). The sprite is plotted through a new module - ConvertSprite - which works out what the mask should be for the redirected image.

One of the fun things that we found once the thumbnailing was in place was that alpha-channel sprites didn't render quite right. That is, they did, but we didn't redraw fully so if you selected an alpha-channel sprite that had been used for an icon it would be rendered selected over the existing sprite, which made it look very wrong. If you unselected it, it would again be rendered over the top, resulting in a bit of a mess. This was very obvious with the purple ball test image.

Sadly the simplest solution to keep things quick to render was to just force a full redraw when an alpha-channel sprite was detected. Not the most elegant of solution and it's possible that if the work on external renderers had been completed it would have been possible to identify such things in a more generic way.

All the thumbnails were held in a sprite cache dynamic area, used by the Filer just for the thumbnail rendering. As sprites were thumbnailed they were added to the sprite area. When a filer window was closed, and the thumbnails were no longer required, the thumbnails for that window were deleted from the sprite area. As, in general, this meant that the sprites were deleted from the bottom upwards it would generally result in an O(n²) operation. The first sprite would be deleted, which moved all the others down, then the second, again moving all the sprites down, and so on until there were none left for that window.

This produced noticeable (and annoying) delays when the window was closed. A thought crossed my mind to defer the deletions until null polls and do them in the background but that really just moved the problem rather than addressing the fact that it was a poor way to do the deletions. Instead I flag every sprite as being deleted in the pool and then move the sprites in a single operation at the end of the window close; the result being a significantly faster response and what should be just O(n) to delete the thumbnail sprites.

Rename tracking

One small thing, that probably isn't even noticed by people, is that the rename box was updated to be tracked when a directory viewer is updated. Previously, if the directory viewer was updated or moved whilst the rename box was shown for a file, the rename box would be removed. This wasn't usually a problem because you clicked to rename a file, renamed it and then you were done - no other operations happened in between.

However, if the directory you were operating on had a file that was being updated (eg a log file) or had files created or deleted (eg a Filer_Action operation taking place), then you would lose your rename box. You had to find the file and click again to rename it. Which might be annoying as the files might keep being updated. Similarly, if you resized the window whilst the rename was shown, it would lose the rename box.

This annoyed me, partly because I often logged to the same window, and partly because if the window was shared by ShareFS it might be remotely updated if a program on another machine saved files or performed other write operations in the directory. And, of course, losing your input whilst you're doing something is not a good interface design.

I added tracking to the file being renamed such that as the window was updated, or resized, the file was remembered and the rename box was restored when the operation completed, with the contents and position of the caret preserved. It made a lot of things a lot easier whilst debugging, and made me feel a lot better.

I don't think that many people would have noticed it, but even small things like this made using the Filer less irritating, and so improved the general user experience.

Filer_Action

The Filer_Action module provides the desktop interface for performing the usual filer operations on objects - copying, renaming, moving, deleting, counting. The module itself was created for RISC OS 3 to avoid the problem of dropping to a text based, single tasking, dialogue box. Filer_Action allowed the system to multitask whilst the filing operations were being performed.

Proxying

One of the areas that was a regular frustration was the lack of any general way to replace the Filer operations with other, more powerful tools. All the operations go through the Filer_Action, but the module's implementation is limited - it's designed to be functional rather than particularly swish. Of course the entire module could be replaced by other people, but that's a reasonably large amount of work - and you have to provide all the operations that Filer_Action can perform, including any that are added in the future.

Back in the early Select days, I wanted to make it easier to have user supplied replacements for its functionality - specifically for the 'Count' and 'Find' operations. There were a couple of very good tools that had been written by people to count directories recursively and present the results in a cute way, and to find files based on type or name, or content. They would - I thought - be really cool to have properly integrated in to the Filer so that they ran seamlessly.

The 'proxying' was added to the Filer for this reason - essentially Filer_Action would start another task for the operations if there were special alias variables set. The application would then be passed messages from Filer_Action so that it could perform the task on behalf of Filer_Action. The proxied task would merely have to react to the documented Filer messages when directed at it. A message would also be broadcast to allow any existing tasks to perform the operation, rather than starting a new task.

Initially nobody took this up, but Alex Macfarlane-Smith wanted to write a 'recycle bin' to which files would be moved instead of being deleted. This would mean that it was significantly safer to delete files, knowing that they could be recovered (assuming they were deleted through the Filer interface). !Recyclone was bundled with a Select release.

The !Recyclone tool could replace all the delete operations, allowing the delete operation to be converted into a move operation.

Counting large files

Counting my !Boot. Back when we first started at RISCOS Ltd, MP3 as an audio format was beginning to take off. It offered a useful way to keep large amounts of music available for playback, without needing to hunt down CDs. I began to encode a reasonable amount of music from the CDs I owned. On occasion when it was necessary to count the music, Filer_Action wouldn't work. It only kept its counted property in 32 bits, so was limited to 4 GB.

Generally that's not a problem, but as I had quite a few tracks, and exceeded that limit easily, it annoyed me. To try to alleviate the problem, I updated the size counting in Filer_Action to be 64 bit. This should keep things going for a little while. However, then it became obvious that there was another problem - big numbers are hard for humans to read quickly.

When you saw a large number in Filer_Action, you had to count backwards from the end in groups of 3 to get the approximate size in megabytes, gigabytes (or terabytes - although we never got there for the music I had!). So I added support for separating the number into blocks of 3 digits with commas.

A little while later it annoyed me that I hadn't used the territory configured format for numbers. I never got around to addressing that, so I'm sorry if you're using European or Eastern territory which doesn't use groupings of 3, or has something other than commas as its separator. Laziness on my part not to go back and address the issue.