Heretic

Application icon for !Heretic
!Heretic

Heretic was produced by a different company to Doom. ID had licensed the engine to Relic, so that they could produce a more fantasy driven game. The game itself is a lot brighter than Doom, has a slightly more advanced engine in some places, and has a completely different premise. It's also a lot brighter due to quite different design goals.

Work on Heretic began part way through the Doom+ developments. Partly, it was a break from the Doom things, and partly it meant that I could apply some of what I'd learnt to the basic engine that we'd got so that it would work on RISC OS. Initially I started by trying to convert over the main definitions and rendering to use the same functions as Doom+ did. This didn't go well and was very quickly thrown away.

Much of the Heretic code, like the Doom code, assumed that structures would be packed in memory. In the Norcroft compiler, structures are not packed, because accesses to them become very inefficient if you cannot assume that they are word aligned. This meant that just loading the data in from disc into memory wasn't lining things up right.

Fortunately, in almost all cases you could realign the data to the memory structure without much difficulty. The side effect of aligning things differently is that they take more space, so this also means that an array of packed structures in real memory doesn't take a multiple of 'sizeof(structure)', because that value is rounded to a word boundary. This caused a few fun changes as well.

Although I hadn't made these changes to Doom+ myself (Eddie had done that work long before me), I had independently ported the Doom source release to RISC OS to produce a working version that ran reasonably well, so I had come across all these problems before.

There were other compiler differences, like the assumption that chars are signed (they're not in Norcroft by default), but generally it was up and running after just a couple of days work. The very first version used the display and keyboard input code taken from my own Doom port, as the code was less involved than that in Doom itself.

Doom used a DoomSupport module to provide its key input, bank switching, interactions with the sound system and a few other areas. The same would have needed creating and integrating early on, and I wanted to have a visibly working game sooner rather than later. The very earliest versions of the game ran at 22 frames per second, which is pretty good going for the 3 days work and a very naive graphics implementation.

First the keyboard and bank switching code came across, as these were both handled in the new HereticSupport module. The sound came next, and I quickly had something playing.

The sound was handled in almost the same way as in Doom, so adding the support for it wasn't too bad, although it did present some difficulties as the differences were important in some cases. I brought across each of the optimised sections of code in small sections. In general the volumes of the playing samples were wrong because they were represented differently in Heretic to Doom.

Bringing across the sprite and planes rendering as assembler coded routines wasn't too bad, and once that was in it was possible to start testing out the high resolution version of the game. The basic engine only supported 320x256, but the high resolution version - which needed the special routines from the Doom version - could handle anything up to about 2048x1024.

In doing so, I found that I'd broken the first level. There's an archer guy in the outer water section and he had begun walking on top of the water, rather than having his feet obscured under the water. I hunted around for a bit, diffing the code back to versions I believed work, before I realised that it had nothing to do with the high resolution code.

When I'd replaced the sprite renderers with the assembler versions, I'd missed the fact that there was a lower clipping point which was used to track where the bottom of the sprite should stop rendering at. Working the extra limit into the sprite plotters wasn't too hard once I know where the data was and how it needed to be used, but it was vaguely fun for a while.

It also showed off the water area, which had a current that pulled you in one direction. This was a feature Doom never had, but wasn't really one that you could back port usefully - it didn't really serve as much purpose in Doom.

The sound problems I mentioned above plagued the development, and whilst some sounds played they weren't necessarily the ones which you expected. Fortunately there was a 'noise' cheat which shows what sound effects should be playing. Here it was easier to see that the effects that were mostly being dealt with badly were the background effects (wind, dripping water, and the like). This turned out to be a slightly different use of a pointer to Doom.

There was a lot of fun with memory corruption whilst trying to get the high resolution version working, which was a little easier to track down by bringing the updated zone manager and enumeration code from Doom.

The configuration file changes that I'd made to Doom, to make it easier to see what the configuration meant and how it was used were brought over to Heretic as well. I've just noticed that it was still coded to write out 'v1.00β' in the file. Ah well.

I wanted to grab a screenshot in Heretic to include here and found that the the code isn't even enabled in the release version - which is a little disappointing. Fortunately it can be enabled by changing a couple of options and rebuilding (and then rebuilding again, because I had forgotten that the PCX files it writes won't be given a RISC OS file type). Small changes aside, it now captures the screenshots, and I can at least include some pictures here. And slightly more fun, I can convert the images from PCX direct to PNG with the ImgConvert - it's nice when things just come together.

Heretic, looking pretty in 640 by 480
Heretic, showing the higher resolution display, and the scaled up status bar.

Because of the higher resolution, the status bar could be displayed scaled up, so that it looked the same as it did in the lower resolutions, or could be scaled down to free up more of the screen space. Unlike Doom, I decided to leave the weapons unscaled as well. This was a conscious decision rather than just a side effect of the status bar not being scaled - if you had a small status bar but a large weapon it looked really silly. So scaling them the same way looked a lot nicer.

Don't those monsters look really blocky now?
Unscaled status bar and weapon.

By the time I was done with Doom, I'd got a pretty good set of diagnostics dumped at the point of failure. A backtrace, zone usage dump and breakdown of the statistics could be generated when a problem occurred. This made it a lot easier to see the main details that I needed at the time of the failure.

An example, below, shows what happens when an error occurs during the start up of the game. In this case it was a failed start due to the 24bit mode I was trying to use being too large for the machine it was running on.

**** ERROR: Not enough memory for screen buffers

    a1f8: C: Unnamed routine()
    b220: C: D_DoomLoop()
    c970: C: D_DoomMain()
    8984: C: main(0xf (15),0xb2004 (729092))
 386b11c: ?: SCL routine(0xb1b34 (727860),0x88b8 (35000))
    8a2c: C: Unnamed routine()
 386b514: ?: SCL routine <root call>
Z_DumpHeap: Breakdown of heap :
zone size: 51855360  location: 2bb00000
tag range: 0 to 2147483647
block:2bb00020    size:  107544    user:00000002   tag:  5 (Screen bank)
block:2bb1a438    size:    4024    user:000b77b0   tag:101 (Cache entry)
block:2bb1b3f0    size:  163864    user:00000002   tag: 28 (Openings lookup)
block:2bb43408    size:    4848    user:00000000   tag:  0 (Unused)
block:2bb446f8    size:     424    user:00000002   tag:  7 (Texture lookups)
block:2bb448a0    size:     424    user:00000002   tag:  7 (Texture lookups)
block:2bb450e8    size:     424    user:00000002   tag:  7 (Texture lookups)
block:2bb45290    size:      80    user:00000002   tag:  8 (Texture data)
block:2bb452e0    size:     280    user:00000002   tag:  9 (Texture look aside)
block:2bb453f8    size:     280    user:00000002   tag:  9 (Texture look aside)
[snip]
block:2bbc0e30    size:     156    user:00000002   tag: 16 (Sprite frames)
block:2bbc0ecc    size:     156    user:00000002   tag: 16 (Sprite frames)
block:2bbc0f68    size:     156    user:00000002   tag: 16 (Sprite frames)
block:2bbc1004    size:     156    user:00000002   tag: 16 (Sprite frames)
block:2bbc10a0    size:    1624    user:00000002   tag:  6 (Sound misc data)
block:2bbc16f8    size:     104    user:00000002   tag:  6 (Sound misc data)
block:2bbc1760    size:     412    user:000b82e4   tag: 23 (Status bar)
[snip]
block:2bbcee78    size:      96    user:000b83ac   tag: 23 (Status bar)
block:2bbceed8    size:51007784    user:00000000   tag:  0 (Unused)

Statistics:
Tag name                           Count  Memory  C %age  M %age

Sound sample                           0       0    0.00    0.00
Sound misc data                        2    1728    0.41    0.00
Music data                             0       0    0.00    0.00
Screen bank                            1  107544    0.20    0.21
Texture lookups                        7    2968    1.42    0.01
Texture data                         100    7028   20.33    0.01
Texture look aside                   200   74432   40.65    0.14
Texture temporary data                 0       0    0.00    0.00
Texture composite data                 0       0    0.00    0.00
Texture translation                    1     428    0.20    0.00
Flat translation                       1     308    0.20    0.00
Flat temporary                         0       0    0.00    0.00
Sprite lookup                          3   17952    0.61    0.03
Sprite frames                        128   40472   26.02    0.08
Sprite lumps                           1    1048    0.20    0.00
Colour data                            3  200008    0.61    0.39
Automap data                           0       0    0.00    0.00
Demo buffer                            0       0    0.00    0.00
Interlude data                         0       0    0.00    0.00
Terrain data                           1     308    0.20    0.00
Status bar                            38   55160    7.72    0.11
Temporary save game buffer             0       0    0.00    0.00
Temporary readfile buffer              0       0    0.00    0.00
Level data                             0       0    0.00    0.00
Level thinker                          0       0    0.00    0.00
Level temporary                        0       0    0.00    0.00
Planes hash table/array                2  165424    0.41    0.32
Drawsegs array blocks                  0       0    0.00    0.00
Openings lookup                        1  163864    0.20    0.32
Static data                            0       0    0.00    0.00
Cache entry                            1    4024    0.20    0.01
Unused                                 251012632    0.41   98.37
<unknown>                              0       0    0.00    0.00
                                     49251855328


ERROR: Not enough memory for screen buffers

I updated the code so that I could save a PCX file for the above screen shots, and then I wanted to do the same for 24 bit-per-pixel. The code didn't understand the PCX format, and I did try to quickly update the code so that it output a 24 bit-per-pixel PCX file, but it appears that it's a little harder - and in any case my PCX decoder doesn't understand what I wrote. So instead I've had to port the Doom screen shot code over so that it saves out sprites. It was significantly quicker than messing with PCX!

Smooth rendered 24 bit-per-pixel levels
The opening level in 24 bit-per-pixel, with some nice monsters playing with me.

The lack of the screen shot feature shows how Heretic, whilst in some areas quite proficient, was not at the same quality level as the !Doom+ work. The crash above, when the front end selected a mode that wasn't possible, is another example. It's a little disappointing that for all the work that went into it, it's still not as polished as I really wanted.

I finished Heretic in a bit of a rush, because I needed to get Hexen up and running, and I knew it would be a lot more work to make efficient because it was a far more complex game. The front end for Heretic was styled like the spell book, and used some stylised option buttons.

Black spell book for Heretic
The black spell book from Heretic provided the surround, and the option buttons were taken from the buttons from within the game.

It was whilst working with the stylised black front end that I first encountered the problems with the faded icons. The Wimp faded the icons towards the background grey, despite the fact that the window background was black. This was addressed in Select, which is why the fading doesn't look too bad here. The controls and sound configuration windows were the same as the Doom original that they were cloned from.

They're so similar in fact that it looks like I just lifted the sound icon as it stood and didn't even bother to change the outside anti-aliasing to match the black background. Look at the terrible grey dots around the edges. How ugly <sob>.

The network configuration was far simpler, as many of the new features that had been added to Doom were not present. Similar network drivers were present, but these were integrated into the front end application. Instead, a menu field offered the network options, and this configured the network more slickly than the Doom configuration.

Another area that was improved in the front end was the support for the music in the setup menu. This was really a backport from Hexen, where I'd done a lot more work in the same area. A single option button (a G-clef) enabled the menu music whilst the front end was running. This required a small amount of work to make safe to be used outside of the main game - most of the other code assumed that the other game structures would be present. It worked quite well, really, although for the amount of time you spent in the front end it may not have been worth it - it seemed to work better in Hexen.

Original Doom has used a text mode to display the status of the initialisation. This had been abandoned in favour of a progress bar on the RISC OS version - and the game start up was considerably faster anyhow. For Heretic they retained the text mode but used a simple progress bar for each of the stages.

I retained this start up, and I've got to say it's the worst part of the port by a long way. It looks very tacky, and it flickers like mad. Rather than being pre-composed and then updated to the screen, it would write the template and then update the lines that it needed to change, and it would repeat this every time that the display needed updating.

This meant that not only did the scrollbar flicker, and the text describing what it was doing appeared and disappeared as they they updated, but the version number would change. The template said 'v1.0' but the actual version replaced this and said 'v1.3', so the '3' would flick to a '0' and back again with every update. Really shoddy.

The game itself had more problems than with Doom. It was very easy to exceed the fixed array limits for sprites of visible planes, especially in the higher resolution modes. These problems most commonly appeared as crashes when playing a complex level. Increasing some of the buffers helped, but there were Heretic specific issues as well. Episode 3 map 8 in particular would crash after about half a minute of play due to a local array overrun which corrupted the stack. Finding that was fun, but that was probably the last of the major problems that the game had.

It would be wrong to say that I didn't care as much about Heretic as Doom. I really wanted to spend more time on it, but unfortunately I'd spent a lot on just getting it to work, and I didn't really have time to play through all the levels. I think it was a passable port, but not one that I'm particularly pleased with. There were so many little things that I wanted to do.

On the other hand, the level layout left quite a bit to be desired - they focused a lot on traps and hidden areas. You'd enter an area and find that collecting an object triggered a door behind you or opened the area up to find that you were surrounded. Maybe that was fine for turn based role play, but it annoyed me in Heretic.

Hexen

Application icon for !Hexen
!Hexen

I started Hexen part way through the Heretic development. Having got a reasonable understanding of Heretic, I began work on Hexen. My notes make particular mention of the fact that it was a lot easier to get to a version that displayed the first screen and began to load things as part of its initialisation, after working on Heretic. They also have a note, on the first day that there's a lot of new stuff that's been added, and things that had been reworked significantly - and a line 'WTF is a PolyObject?'.

The PolyObjects were going to be something of a pain for me. Essentially these were objects that were constructed in one location in the map and would be translated to another location, and could slide, rotate or push sideways - actions that were otherwise quite difficult in the structure of the Doom engine because of how its internal map was constructed.

Because the PolyObjects were quite complicated, some of the initial work on the game omitted them entirely - they just didn't appear, until some of the code that manipulated them was made to work properly and suddenly doors appeared where before there had been none.

After a couple of days of working on Hexen I had a game that I could play, at 20 frames per second in 320x200 (when idle) - which is very impressive given that there were no special optimisations yet. I then returned to Heretic for a couple of months to get it to a much better state, and add a decent front end; Hexen was 'working' so I felt less daunted by it.

When I came back I started looking at the speed of the game's start up and its general optimisations that could be brought in from Doom. I must have been in a good mood, as a comment in my notes says "After a long absence, with Heretic we're back and kicking arse, or at least prodding bottom".

Initially the game took about 20 seconds just to start up, which was really annoying - it displayed a nice progress bar whilst it did so, but it was quite tedious for testing, never mind if anyone was wanting to actually play. The lump caching functions from Doom reduced the start up time more noticeably (as they had more to work on!) than they ever had in Doom, reducing the loading time to under 2 seconds. I'm not sure how bad it would have been on the original PC, but the improved loading time felt good to me.

The start up screen itself was a full graphical affair, and even plotted the loading messages in its own custom font. After the mess that was the Heretic start up, I was really happy with this - even if it was just making the original code work properly.

The zone allocator was one area that had been left unchanged from the earlier games and because of this it was easy to replace the original allocator with the enhanced version that I had developed in Heretic from the Doom version. This gave a huge head start in debugging as the heap's internal consistency checks were simpler to enable, or run regularly whilst playing, and when things did go wrong, a full dump would actually be very useful to find what objects were corrupted.

Adding the support for the high resolution game wasn't too hard, but there were a lot of new types of data around which made it a little more difficult to work out exactly where the screen rendering had to be changed. Although I retained the unscaled status bar, I didn't keep the unscaled weapon. This was for a very important reason - the sprites weren't big enough for the high resolution modes.

The axe attack was the most obvious of these. When the unscaled status bar was used the axe would sweep down from the right towards the centre of the player's view - and part of the axe wouldn't be there. The top, which previously would have been cut off by the 200 pixel high screen mode, didn't have any data at all, and would therefore be cut off in a straight line which looked ugly.

I had always intended to come back and fix that, creating some new graphics for the weapons that didn't look right. I never did and I'm a little disappointed now, because it doesn't look at all as polished as I'd have liked for that one thing. <sigh> Oh well.

I had a quick return to Heretic, only to find, on one Monday in January that Raven had released the source under the GPL. That meant that we really needed to get a move on to release it properly, otherwise there would be a flood of free ports. It's a lot of work in porting the games well - and I'd already spent months on Heretic.

The notes here say that the 800x600 version of the game (presumably in 256 colours) runs at an amazing 3 frames per second. This was quickly doubled, but the speed was far worse than Heretic. I wanted to spend a lot more time on optimising some of the redraw code to improve things. A lot of the new code could be investigated for improvements.

I started using the Hierarchical Profiler (HierProf from Dr Smith's toolkit) to investigate where the time was going. This helped to focus some of the improvements, but it also reinforced my view of such profilers - they will tell an engineer who knows the code well what they already know. The most useful part of the tool is giving you some useful metrics to compare to, and some figures to back up what you know.

It's useful knowing that one routine takes 85% of the processing time, and if that routine is being called millions of times per second then the smallest improvement will have a big effect. Better still, though, is to reduce the number of calls at the same time. Better yet is to replace the algorithm that causes it to be called millions of times, so that the processing that it's doing can be achieved more efficiently. Profilers tell you where time is spent - interpreting it and choosing the right action to take is a skill.

I was reasonable at finding the right things to do, but Hexen changed the way a lot of the engine worked and so some of my experience with Doom was useful only as a guide. Still, the game was improved quite a bit and the 640x480 version runs at a respectable rate on the StrongARM. On RPCEmu on my netbook, I get 15 frames per second in 256 colours, which isn't that bad. I'm certain it could be improved though.

Startng the game as a warrior, with frame rate display
Hexen in 640x480 runs at a reasonable 15 frames per second on the first level.

As with Doom, I wanted to keep the frame rate above 12 frames per second for general play. That really was the absolutely minimum that was acceptable for playing. I had built an A5000 version of Heretic, and when I made Hexen run I also built an A5000 version. They were - to the best of my knowledge - never released. I don't remember if I ever tested them and, if I did, they would undoubtedly be cripplingly slow. Heretic might have been able to play a couple of levels, but I doubt Hexen would have worked at all sanely.

As well as the PolyObjects, which took a little bit of processing (though not as much as I originally thought) the game played up the ambient sound effects a lot more, had many more animation frames for the monsters, and other parts of the level's detail, had ambient sprites to match the ambient noises, and the terrain had more objects in it. The latter two were a bit more of a drain on the memory and processing than they had been in Heretic.

In Doom and Heretic there were objects around the place, but not as many or as varied as they were in Hexen. In Hexen there were ambient objects, to match the sounds. Leaves that would blow off the trees in the wind were a typical example of this, and they looked very nice, but they were another object that had to be tracked. There were lots of trees and the like in the external areas, and pots that could be smashed - usually revealing a health item or ammunition.

The trees, and some of the other objects reacted differently to certain attacks - and these were obvious from the first level. If you used a fire attack on a tree it would catch light and burn to the ground over a number of frames. Better still, an attack on a monster nearby using fire could cause the tree to catch light through proximity.

The level had lots of the usual triggers that Doom used, but also used a scripting language 'ACS' ('Action Control Script') which allowed the designer a lot more freedom in making the level work in different ways. These were used pretty well within Hexen, and the puzzles in the levels were far more complex than they had been in Doom or even Heretic. Hexen didn't quite lose the quality of monsters appearing from traps, but the way that it was done tended to be more satisfying than it had been in Heretic.

Adding to the rendering time, some of the close attacks used dust effects by making the attack use a transparency. As with Doom+, this effect was effective, but it slowed things down further as it meant reading and writing the screen, with a lookup in between (or scaling in the 24 bit version).

As well as objects being breakable, parts of the scenery were breakable. This was most obvious on the windows but other objects could be broken as well. Once broken, they were able to be walked through, offering either new areas or a power up.

Example of the first level using swinging doors
The first level, showing the swinging doors which are implemented by a PolyObject. (Animated version (12M))

Much of the rendering could be brought over from Heretic as it stood, but the extra processing meant that the speed up wasn't as significantly - the proportion of time spent in the redraw functions was less, so the improvement was proportionally smaller. Similarly, the performance was very poor on an ARM710 - I didn't even try the ARM600 (not sure I had one back then), but the StrongARM worked quite well.

As I mentioned for Heretic, screenshot support wasn't present in the release version, so I had to add the necessary functions just to get the pictures for this ramble. It's a bit surprising that it wasn't added, as it was really quite trivial. Maybe I just wanted to see the back of them, or more likely I didn't think that it was an important feature to have.

The front end was given a similar treatment to that of Heretic, although I feel much more comfortable with the Hexen front end. Despite the fact that it's a little too wide for the style guide (and, of course, it uses a white-on-black colour scheme) it feels more balanced than that of Heretic. It used the same network launching scheme - although as I looked at it whilst writing this ramble I noticed that some parts of the network tools still refer to Heretic. I really was trying to get it finished so that it could go out and I could be done with it.

One interesting thing about Hexen is that it supports up to 8 players. At least that's what it claims - I never added the extra support to the drivers to allow for this. That said, testing an 8 player game would have been nigh on impossible given the number of machines I had available to me.

Many of my tests at that time were varying between running on 'Ursula' (the development version of RISC OS 4) and RISC OS 3.7, so performance varied accordingly. Mostly, I tried to restrict myself to testing on RISC OS 3.7, but I'd been testing Ursula from mid-Doom+ development, as that was the only useful way to get bug reports back to Acorn so that they could fix things. After all, that's why we got the beta version.

During the time that I was doing this development work we were also trying to get together a package that would 'save' RISC OS. Writing up specifications, plans for work, estimated time and more general goals, as well as talking to a few people in Acorn and a lot of people out in the development community took up some of that time. All the while, testing and both Hexen and Heretic and keeping a bland face on the behind the scenes things that eventually became RISCOS Ltd.

I was quite pleased with Hexen in general, but it still feels too slow to me. I really wanted to get it to run faster, but I felt I'd exhausted the time I could set aside. I began work on Heretic and Hexen on 20th October 1998, and finished on 18th February 1999. There was another update in March when we isolated crashes in some areas due to an overflowed buffer (in both games). In terms of money, it probably wasn't really worth it, but it was a lot of fun.

I can't seem to find any review that talks about the port, rather than about the game. There are a few references to it being slow - and one review I saw mentioned that it was hard to play on an A7000+, which is completely understandable. I wouldn't really recommend Hexen on anything less than a StrongARM really.

Update: 2013-05-14 Gareth Moore emailed me to remind me that there was a review up at Acorn Gaming of the ports, which is pretty nice - and which gives it 5/5 which is always nice to see. He makes some fair comments about the networking for serial being pretty ropey. I think a lot of that is down to the poorer drivers that I wrote first. The IP drivers are a lot more reliable.