Brendan's Developer Diary
Contents:
30 May 2009 - The Search For A Graphics File Format
28 January 2009 - Initial Boot Loaders Done!
8 November 2008 - Build Utility Done!
1 September 2008 - The Build Utility and Other Tools Revisited
1 September 2008 - Floppy Image Utility Done
31 July 2008 - Redundant Floppies?
22 July 2008 - "DOC_GEN" Utility Done!
7 July 2008 - Technical Documentation
3 July 2008 - Website Generator
30 May 2009 - The Search For A Graphics File Format
One of the goals of BCOS is that software written for BCOS should be resolution independent and colour depth independent, caring only about aspect ratio, so that each application and GUI doesn't need code to supporting many different resolutions and colour depths. Instead they'll always generate graphics data for a virtual resolution and virtual colour depth, which is then converted into the actual resolution and colour depth by the video driver.
Another goal of BCOS is to promote interoperability between different applications (and between different operating systems). This is to be achieved by defining a standardised file format format (with open specifications) for each purpose. For example, all software that deals with 2D graphics images (pictures) will only need handle 2D graphics image files in the standard file format format, and won't need to include code to convert between the many different legacy file formats (JPG, BMP, PNG, TIFF, etc). Of course this also means that it's important to get it right from the beginning - I can't go and change the standard every time I think of a better way.
I want to create a specification for the standard file format that the OS will use for 2D graphics images. Of course resolution Independence means that 2D graphics images will almost always need to be scaled in both directions, and scaling raw pixel data is slow and produces "less than perfect" results, so the techniques used by most file formats aren't really acceptable.
My first idea was encode each horizontal picture line as a collection of "line segments"; where each line segment was encoded as a starting colour, length and an optional ending colour (used for gradients); and where the sum of the line segment widths added up to 4294967296 (so that the image can be scaled by dividing the virtual line segments widths by a scaling factor). To test this idea I wrote a utility to convert files from the "BMP" file format into this file format, and another utility to convert from this file format back into the "BMP" file format (including scaling the image). The results looked fairly good - scaling the image horizontally produced extremely nice results because it involves scaling gradients, but also because the utility did full anti-aliasing. Scaling the image vertically looked reasonable because of the anti-aliasing, but nowhere near as good as scaling the image horizontally...
Then I starting thinking about how to make vertical scaling look as good as horizontal scaling. The answer is triangles!
A triangle can be described by 3 coordinates (one for each corner) where each corner of the triangle has a colour (creating shaded triangles). In this case the triangle can be scaled by scaling the 3 coordinates, and regardless of how much you scale the triangle's coordinates the triangle itself looks exactly the same.
To test this idea I wanted to write a utility to convert lists of shaded triangles into the "BMP" file format. I finished this utility today. As an added bonus I also implemented rotation (e.g. so that the image can be rotated around it's centre). The results are insanely impressive - it's pixel perfect, regardless of how much the image is scaled in any direction and regardless of how much it's rotated.
The only thing left to do is to write a utility to do the opposite - convert files from the "BMP" file format into this file format. For this I need code to recognise shaded triangles in the original pixel data. This could take a while...
28 January 2009 - Initial Boot Loaders Done!
I've been working on four boot loaders - one for booting from floppy, one for booting from CD-ROM, one for booting from PXE/network and one for booting from a multi-boot compliant boot manager (e.g. GRUB). Yesterday I finished all 4 of them. I still need to write more boot loaders (e.g. one to boot from hard disk) but I'm not planning to worry about additional boot loaders until much later.
It's taken about 11 weeks to implement these boot loaders (almost 20 days per boot loader on average). This sounds like an extremely long time for something that's normally fairly small, even for a part-time developer. The thing is that these Boot Loaders aren't as simple as the boot loaders you might see in other OSes (they don't just load some files into RAM) - the boot loaders are packed full of features, mostly for fault tolerance.
In general the first things a boot loader does is to get the physical address space map (PASM) from the BIOS and load a Faulty RAM List into RAM, and then use these to initialise memory management. The idea is that if a computer has areas of faulty RAM then the boot loaders and everything else will avoid using the faulty RAM; so that as long as known faulty RAM hasn't been used before the boot loader has initialised memory management, everything is immune to the effects of known faulty RAM. This leads to the idea of "trusted areas" - areas of RAM that can't be faulty if immunity from known faulty RAM is guaranteed. The boot loaders all find trusted areas and check if any of these areas include known faulty RAM and refuse to boot if they do.
To maximise the OS's immunity to known faulty RAM, I did everything I could to minimise the size of the trusted area/s. For most boot loaders (excluding the multi-boot compliant boot loader) the trusted area is typically less than 64 KiB, which includes the 32 KiB area of RAM from 0x00000000 to 0x00008000 that is assumed to be relied on by the BIOS itself. Basically, you can have known faulty RAM anywhere else and the OS won't care at all.
Then I decided to make the boot loaders (and the rest of the OS) immune to "gate A20" failures. If the boot loader can't enable the A20 gate, then the entire OS can't access every odd 1 MiB area of the physical address space. This would prevent most OSes from doing anything - BCOS will just avoid using these areas and continue like normal. Note: this isn't supported by the multi-boot compliant boot loader, because the boot manager (e.g. GRUB) is unable to load the boot loader into memory if A20 isn't enabled.
Of course being immune to known faulty RAM wasn't quite good enough. Once the boot loader's memory manager is initialised all other RAM is allocated via. the memory manager, and when RAM is allocated it's also tested to see if it's faulty or not - if the allocated RAM is found to be faulty then it's not used. This means that the boot loaders (and eventually, the rest of the OS) are resistant to faulty RAM, even when the RAM isn't known to be faulty beforehand. Eventually this information will be used to regenerate a new Faulty RAM List, so that any RAM that's found to be faulty is avoided during subsequent boots.
That's not the end of it though. A lot of care was taken to avoid all known BIOS bugs, and to support the widest variety of computers, and there's a few more things these boot loaders do. For example, after a file is loaded into RAM the boot loaders do several checks to make sure the file is sane, including calculating and checking a 32-bit CRC.
All of the boot loaders also construct a Boot Log that includes all messages displayed on the screen by the boot loader. This means that once the OS is running properly administrators will be able to look back through the boot log and see every single character that any of the boot code displayed.
The floppy boot loader supports "redundant floppies". The idea is that if the floppy has 2 sides (and if the floppy is large enough) then the second side of the floppy is used as a mirror image of the first side. In this case, if there's read errors on one side of the floppy then the boot loader will automatically try to read the redundant copy of the data from the second side of the floppy, so that the OS can still boot if there's bad sectors, etc. To make this happen easily, I wrote a utility to create floppy images that automatically generates a redundant floppy if possible. This utility and the floppy boot loader will support any common floppy format - 2880 KiB floppies, 1670 KiB floppies, 1440 KiB floppies, 1200 KiB floppies, etc, all the way down to ancient 160 KiB single-sided floppies (basically, any floppy format that can be described as "X heads, Y tracks and Z sectors per track").
The CD boot loader will check to make sure the entire boot loader was loaded into RAM by the BIOS (I heard that some BIOS's are buggy and may not load the entire boot loader from CD), and if the entire boot loader wasn't loaded into RAM then there's code in the first sector to load the remainder of the boot loader, so that boot can continue as if the BIOS did things correctly. I also wrote code to create an ISO9660 CD image that includes the floppy boot images (so the same CD can be used to create bootable floppies in case the computer doesn't support booting from CD). The CD boot loader itself uses "no emulation" El Torito to avoid the need for the BIOS to handle floppy/hard disk emulation.
Now I can start working on the next piece of boot code - the Common PC BIOS Module. This shouldn't take too long...
8 November 2008 - Build Utility Done!
It took longer than I expected, but it's also more powerful than I originally planned and produces better quality results than the previous utilities it replaces - it even generates "tooltips" in assembly language when the mouse is over certain things that contain extra information (like how many times labels, etc are referenced). There's full documentation for this utility included as part of the project's documentation if anyone is interested.
The best part is I can start using the build utility now...
1 September 2008 - The Build Utility and Other Tools Revisited
With everything else out of the way it became time for me to write the fourth and final utility for generating web pages. The idea was to have one utility that generates the web site's main pages (the home page, this page, etc), one utility to generate specifications, one utility to generate web pages corresponding to assembly source code (and to build/assemble the source code, disassembly the resulting binary, do hex dumps, etc), and one utility to generate "glue" pages to make it easier for people to navigate through the other pages.
With this in mind I started the utility to generate the "glue" pages. I didn't get far.
The main goal of these utilities is to make it easy to maintain the web site by automating most of the tedious work (like cross-linking web pages, making sure the formatting is consistent, etc). The problem is that over time I ended up with slightly different markup for different utilities, and having 4 different utilities to maintain would have been a pain in the neck. The solution? A large utility that does all of it.
Well, not quite all of it. The utility that generates the web site's main pages is considered part of the web site and not part of any version of the OS. From the home page the "Current Project" link sends your browser through a symbolic link to the current project's web pages, so that (for e.g.) I can copy the current project into a new directory and update the symbolic link, and maybe one day have links to previous/older versions of the project (or maybe a "stable" and a "development" branch).
Anyway, I've been working on a system build utility that incorporates everything (except the web site's main pages). This should mean faster build times (because more can be done in parallel), no need to manually track dependencies (e.g. if one binary is included by another the build utility can figure that out and wait for the first binary to be assembled before assembling the second binary), different files can easily reference information in other files using identifiers rather than HTML links (which end up as dead links when files are moved). As a bonus, it's already generating a full "project map" and navigation links in each web page, and automatically creating directories if they don't exist (so I can add a new source code directory without also needing to add a new directory for the resulting web pages). It's also doing recursive directory searches too, which mostly means I'll never need to create or modify any scripts when new pieces are added to the project.
Basically, I'll be able to just write the source files, and I'll never need to worry about dead links, formatting, scripts, dependencies, makefiles, etc.
If you're curious, this is all "live" already - you'll be able watch my progress by viewing the resulting web pages on-line while I write the utility. Just click on the "Current Project" link at the top of this page...
1 September 2008 - Floppy Image Utility Done
Well, I've been slack - this utility was actually completed a few weeks ago...
Now I can tell the utility what format the floppy is (in KiB, from "2880" to "160") and where to find the relevant files, and it'll create a suitable floppy image. This includes automatically deciding if a redundant floppy image should be used, setting relevant fields in the BPB according to the floppy format, setting a "redundant" flag in the boot sector, calculating CRC values for any of the files that don't have one yet and doing any padding (so that different files can grow without needing later files to be relocated). The floppy boot code supports all of this too.
The end result is that now I can easily generate floppy images for lots of different floppy formats, ranging from 2880 KiB floppies down to (very obsolete) 160 KiB single sided floppies, including the standard 1440 KiB floppies and the "not quite standard" 1680 KiB floppies.
Generated on Sat Oct 10 18:35:46 2009
(UTC)