BCOS Build Utility Documentation

Contents

1                Overview
2                Command Line Syntax
3                File Types
3.1                Association Files
3.2                Document Files
3.3                Specification Files
3.4                Assembly Language Source Files
3.5                Assembly Language Include Files
3.6                Copyright Files
4                Directory Handling
5                File Headers
5.1                Association File Headers
5.2                Document File Headers
5.3                Specification File Headers
5.4                Assembly Language Source File Headers
5.5                Assembly Language Include File Headers
6                Markup
6.1                Assembly Language Source Code
6.2                Simple Markup
6.3                Complex Markup
6.3.1                Paragraphs
6.3.2                Indented Lines
6.3.3                Lists
6.3.3.1                Plain Lists
6.3.3.2                Numbered Lists
6.3.3.3                Nested Lists
6.3.3.4                Mixing List Types
6.3.4                Simple Titles
6.3.5                Simple Subtitles
6.3.6                Section Headers
6.3.7                Listings
6.3.8                Tables
6.3.9                Figures
6.3.9.1                Figure Styles 0 and 1
6.3.9.2                Figure Styles 2 and 3
6.3.9.3                Figure Styles 4 and 5

Tables

Table 6.1      Simple Markup Examples
Table 6.2      Table Style 0 Example
Table 6.3      Table Style 1 Example
Table 6.4      Table Style 2 Example
Table 6.5      Table Style 3 Example

Listings

Listing 5.1      Association File Header Example
Listing 5.2      Document File Header Example
Listing 5.3      Specification File Header Example
Listing 5.4      Assembly Language Source File Header Example
Listing 5.5      Assembly Language Include File Header Example
Listing 6.1      Paragraph Example
Listing 6.2      Indent Example
Listing 6.3      Plain List Example
Listing 6.4      Numbered List Example
Listing 6.5      Nested Lists Example
Listing 6.6      Mixed List Type Example
Listing 6.7      Listing Style 0 Example
Listing 6.8      Listing Style 1 Example
Listing 6.9      Listing Style 2 Example
Listing 6.10     Listing Style 3 Example
Listing 6.11     Table Example
Listing 6.12     Source For Figure Style 0 Example and Figure Style 1 Example
Listing 6.13     Source For Figure Style 2 Example and Figure Style 3 Example
Listing 6.14     Source For Figure Style 4 Example and Figure Style 5 Example

Figures

Figure 4.1      Example Project Directory Structure
Figure 6.1      Figure Style 0 Example
Figure 6.2      Figure Style 1 Example
Figure 6.3      Figure Style 2 Example
Figure 6.4      Figure Style 3 Example
Figure 6.5      Figure Style 4 Example
Figure 6.6      Figure Style 5 Example

1   Overview

The build utility recursively scans the current directory and all subdirectories looking for source files, then processes those source files and generates HTML pages. For assembly language source code (except include files) the utility will also execute an assembler, then (optionally) execute a disassembler, generate a hex dump of the binary and (optionally) extract a list of debugging markers.

The build utility uses file modification times to avoid doing any unnecessary work, so that if nothing has changed then nothing is done. For assembly language source code (except include files) it also takes care of dependencies between resulting binaries, such that if the resulting binary generated from one group of assembly files is included (e.g. using "%incbin") by another group of assembly files, then the utility will wait until the first binary has been updated before attempting to assemble the second binary. The utility does this automatically by parsing source files. This removes the need for additional utilities (e.g. "make"), and removes the need to maintain any form of files that control the build process (batch files, scripts, makefiles, etc).

To assist in navigating through the project, the build utility keeps track of a hierarchical tree of HTML files (roughly corresponding to the project's directory tree) and creates links to the parent file (if any) and to any children. It also creates a complete project map.

2   Command Line Syntax

The first (optional) command line argument must be either "quiet" or "verbose".

With the "verbose" command line argument, this utility will list all groups of files checked including the status (updated, failed, etc) and any error or warning messages for that group of files. It will also display some statistics (memory usage, number of threads and processes used, bytes read from disk, lines read from disk, bytes verified, bytes written, real time used and CPU time used).

Without either "quiet" or "verbose", this utility will list all groups of files checked including the status (updated, failed, etc) and any error or warning messages for that group of files.

With the "quiet" command line argument, this utility will only list groups of files if a corresponding output file was updated, or if there were any error or warning messages for that group of files.

The second (optional) command line argument specifies the name of a script to execute if any binaries were assembled and if there were no errors. The working directory for the script will be the same as the working directory for the build utility. This script can be useful for a variety of purposes (e.g. updating a TFTP directory after a successful build).

Note: It's recommended that Chapter 4: Directory Handling be read to fully understand how this utility is used.

3   File Types

The build utility is designed to process several file types. Each file type is intended for a different purpose.

3.1   Association Files

Association files show the associations between all other pieces of the project. They are arranged in a heirarchical tree to assist with describing and navigating through the project, where the deeper you get into this heirarchical tree the more detailed the information becomes until you reach specifications, documentation and/or source code. The build utility scans the directory structure to determine the reliationships between each file and generates navigation links at the top of each file based on these relationships.

Association files may exist anywhere in the project's directory structure, but must be named "index.txt".

3.2   Document Files

Documentation is an important part of any project. The build utility supports many features that assist with documentation, including the ability for other files to reference content in documents.

All document files must be within the project's "doc/" subdirectory (for e.g. the file "doc/foo/bar.txt" would be treated as a document file, but the file "mydocs/foo/bar.txt" would not).

3.3   Specification Files

Specifications are also an important part of an operating system project. The build utility supports many features that assist with specifications, including the ability for other files to reference content in specifications.

All specification files must be within the project's "spec/" subdirectory (for e.g. the file "spec/foo/bar.txt" would be treated as a specification file, but the file "myspecs/foo/bar.txt" would not).

3.4   Assembly Language Source Files

The build utility converts assembly language source files into HTML, including extensive cross-linking (e.g. so you can easily find where something is defined by clicking on text where it's used) and syntax highlighting. For assembly language source files the build utility also allows markup to be used, so that documentation and specifications can be referenced within source code comments, and so that areas within the source code can be generate normal HTML for descriptive purposes, including the ability to add things like titles, tables, lists and figures into the source code.

Unlike all other file types, assembly language source files are processed as a group of files with one main file and (optionally) many additional files that are included by the main file, where all files in the group are assembled to create the same output binary.

Assembly language source files may exist anywhere in the project's directory structure, but the main source file must be named "0index.asm". Any additional files that are included by the main file may have any name but must also exist in the same directory as the main source file (or in a subdirectory). If the main source file attempts to include a file that isn't present in the same directory or a subdirectory of that directory, then the build utility will assume it's an assembly language include file.

When an assembly language include file is included by the main source file, the build utility will search for the file in a subdirectory called "inc" in each parent subdirectory. For example, if the main source file "foo/bar/0index.asm" says "%include "hello.inc"" then the build utility will try to find the assembly source file "foo/bar/hello.inc", then try to find the assembly include files "foo/inc/hello.inc" then "inc/hello.inc".

3.5   Assembly Language Include Files

Assembly language include files are similar to assembly language source files, however they are processed individually.

An assembly language include file must exist in a subdirectory called "inc" (for e.g. "80x86/inc/foo/bar.inc" would be considered a valid assembly language include file).

3.6   Copyright Files

Copyright files aren't processed at all and aren't converted into HTML format. When a copyright file is referenced by an assembly language source file or by an assembly language include file (in the file header), the copyright file is loaded and inserted "as is" into the HTML page/s generated from the assembly language source file or assembly language include file.

4   Directory Handling

The build utility processes an entire directory tree (including subdirectories) that make up the project, and creates an equivelent directory tree containing the generated HTML files. Certain subdirectories within the project's directory have special meaning.

The "util" subdirectory contains utities that are used to manage the project (including the source code for the build utility and the build utility itself), but these utilities are not considered part of the project itself and therefore this subdirectory is ignored by the build utility.

The "www" subdirectory is where all output HTML files will be placed. For example, if the directory "/usr/myproject-1.0" is the project's main directory, then the association file "/usr/myproject-1.0/foo/index.txt" will be converted into the HTML file "/usr/myproject-1.0/www/foo/index.html". This makes it easy to have many versions of the project in different directories, where a HTTP server (e.g. Apache) has a main directory containing symbolic links to each project directory's "www" subdirectory.

The "spec" subdirectory contains all specifications, and the "doc" subdirectory contains all documentation. This makes it easy to archive these directories (e.g. so that other people can download a "*.tar.gz" containing all documentation or all specifications.

An example directory structure for a project might look like this:

Figure 4.1 - Example Project Directory Structure

5 File Headers

Each type of file processed by the build utility has a header. The format for this header depends on the type of file.

5.1 Association File Headers

The header for an association file is relatively simple, consisting of the association file's title on the first line, optionally followed by any number of subtitles preceded by a '+' character.

The Title +An optional subtitle +Another optional subtitle

Listing 5.1 - Association File Header Example

5.2 Document File Headers

The header for a document file is relatively simple, consisting of the document file's title on the first line, optionally followed by any number of subtitles preceded by a '+' character.

The Title +An optional subtitle +Another optional subtitle

Listing 5.2 - Document File Header Example

5.3 Specification File Headers

The header for a specification file is relatively simple, consisting of the specification file's title on the first line, optionally followed by any number of subtitles preceded by a '+' character.

The Title +An optional subtitle +Another optional subtitle

Listing 5.3 - Specification File Header Example

5.4 Assembly Language Source File Headers

The header for a group of assembly language source files should only exist in the main source file (the "0index.asm" file). All additional source files included by the main source file (using "%include") are considered parts of the main source file for parsing purposes (except for assembly language include files), and will inherit the same settings as the main source file (e.g. same title, same subtitle/s and same copyright).

The header for a group of assembly language source files consists of the group of assembly language source file's title on the first line, optionally followed by any number of subtitles, which is followed by the assembler invocation line, then the name of a copyright file on the last line. All of these lines must be preceded by a ';' character, except for (optional) subtitle lines which must be preceded by the characters ";+".

;The Title ;+An optional subtitle ;+Another optional subtitle ;foo/bin/output.bin ;nasm -Ox ;copying/copyright.txt

Listing 5.4 - Assembly Language Source File Header Example

The assembler invocation line provides the name of the assembler to use (typically NASM or YASM) and command line options for the assembler. The build utility automatically generates and appends a list of include directories, the name of the output binary and the name of the first source file. For example, if the assembler invocation line is "nasm -Ox" then the build utility might execute the command "nasm -Ox -I../inc -I../../inc -o ../../bin/output.bin 0index.asm".

Also note that the output binary name specified in the header is relative to the top of the project's directory. For example, if the file "foo/bar/0index.asm" has a header that says it's output file is "foo/bin/bar.bin" then the build utility will change the working directory to "foo/bar" and tell the assembler to generate the file "../bin/bar.bin" from this temporary working directory.

5.5 Assembly Language Include File Headers

The header for an assembly language include file consists of the assembly language include file's title on the first line preceded by a ';' character, optionally followed by any number of subtitles preceded by the characters ";+", which is followed by the name of a copyright file preceded by a ';' character.

;The Title ;+An optional subtitle ;+Another optional subtitle ;copying/copyright.txt

Listing 5.5 - Assembly Language Include File Header Example

6   Markup

When converting text files into HTML files the build utility recognizes certain sequences ("markup"). The markup can be categorized into 2 groups - simple markup and complex markup.

Also, assembly language source code involves some special handling. Please see Section 6.1: Assembly Language Source Code for more information.

6.1   Assembly Language Source Code

For assembly language source code (including assembly language include files) the build utility needs to be compatible with the assembler's own syntax. To do this markup is only allowed within comments, and normal source code is treated as source code where simple markup and complex markup are not allowed.

For (NASM syntax) assembly, the ';' (semi-colon) character begins a comment. If the comment begins on the start of a line and the next character is also a ';' (semi-colon) character (e.g. ";;Hello") then the entire line is ignored by the build utility. If the comment begins on the start of a line and the next character is a single-quote character (e.g. ";'Hello") then the entire line is treated as a line intended to be converted into HTML and is handled like text in other file types (association files, document files and specification files), and complex markup (as described in Section 6.3: Complex Markup) is allowed.

Other comments (comments that begin at the start of the line but don't begin with ";;" or ";'", and comments that don't begin at the start of a line) are treated as assembly source code where no complex markup is allowed; however, simple markup is allowed (for example, so that the line "label: add eax,ebx ;See [d:assembly]!" would be converted into something like "label: add eax,ebx ;See Section 6.1: Assembly Language Source Code!").

6.2   Simple Markup

Simple markup involves sequences of characters that always begin with a '[' character and end with a ']' character. Typically the characters inside the square brackets are a keyword, a colon, and some text. The easiest way to describe all available simple markup is to display them.

Markup Result

[[]

[ (left square bracket)

[]]

] (right square bracket)

[,]

, (comma, normally only used in tables)

[%:listingExample0]

Listing 6.7: Listing Style 0 Example

[#:simpleMarkupExamples]

Table 6.1: Simple Markup Examples

[b:bold text]

bold text

[d:markup]

Chapter 6: Markup (in same file)

[d:util/build:markup]

Chapter 6: Markup (in another file)

[doc:util/build]

BCOS Build Utility Documentation

[email:btrotter@gmail.com]

btrotter@gmail.com

[f:figureExample0]

Figure 6.1: Figure Style 0 Example

[i:italic text]

italic text

[inc:foo/inc/bar]

Similar to [spec:util/build], except referenced file is an assembly include file

[m:monospace text]

monospace text

[s:markup]

Same as [d:markup], except referenced section is in a specification

[s:util/build:markup]

Same as [d:util/build:markup], except referenced section is in a specification

[spec:util/build]

Same as [doc:util/build], except referenced file is a specification

[sub:Subscript]

_Subscript

[sup:Superscript]

^Superscript

[todo:Need to fix that]

[Need to fix that]

[u:underlined text]

underlined text

[url:http://bcos.hopto.org]

http://bcos.hopto.org

[b:bold [i:italic [u:and underlined]] text]

bold italic and underlined text

Table 6.1 - Simple Markup Examples

All simple markup can be nested (although nesting simple markup doesn't always make sense). Simple markup can appear almost anywhere, including within any comments in assembly language source code.

6.3 Complex Markup

Complex markup involves sequences of characters (potentially on many lines) that always begin with a '*' character (or for assembly language source code, with the characters ";'*"). Additional lines always begin with '+' character (or for assembly language source code, with the characters ";'+").

6.3.1 Paragraphs

A new paragraph begins at the start of the line with the character '!' which is followed the pragraph's text. More text can be added to the same paragraph on subsequent lines. If the paragraph follows any other type of complex markup, then the initial '!' character is optional.

Listing 6.1: Paragraph Example is a copy of the source text for this description.

!A new paragraph begins at the start of the line with the character '[i:!]' which is followed the pragraph's text. More text can be added to the same paragraph on subsequent lines. If the paragraph follows any other type of complex markup, then the initial '[i:!]' character is optional. ![%:paragraphExample] is a copy of the source text for this description.

Listing 6.1 - Paragraph Example

6.3.2 Indented Lines

A line of text that begins with a tab character will be indented. A single indented line will look like this:

An indented line

Multiple indented lines will be combined, like this:

	First indented line
	Second indented line
	Third indented line

Listing 6.2: Indent Example is a copy of the source text for this description.

!A line of text that begins with a tab character will be indented. A single indented line will look like this: An indented line Multiple indented lines will be combined, like this: First indented line Second indented line Third indented line ![%:indentExample] is a copy of the source text for this description.

Listing 6.2 - Indent Example

6.3.3 Lists

Lists work in a similar way to indented lines. There's 2 types of lists - plain lists and numbered lists.

6.3.3.1 Plain Lists

A line of text that begins with a '-' character will be treated as a line in a plain list. A single plain list line will look like this:

•	A list line

Multiple plain list lines are combined, like this:

•	First list line
•	Second list line
•	Third list line

- First list line - Second list line - Third list line

Listing 6.3 - Plain List Example

6.3.3.2 Numbered Lists

A line of text that begins with a '%' character will be treated as a line in a numbered list. A single numbered list line will look like this:

1.	A list line

Multiple numbered list lines are combined, like this:

1.	First list line
2.	Second list line
3.	Third list line

% First list line % Second list line % Third list line

Listing 6.4 - Numbered List Example

6.3.3.3 Nested Lists

Lists can be nested up to the the maximum nesting depth. If you reach the maximum nesting depth you'll get an error (but this is unlikely, as you'll probably run out of screen width before you reach the maximum nesting depth). Here's an example:

First list line

Second list line

2.1.

Third list line

2.2.

Fourth list line

2.2.1.

Fifth list line

Sixth list line

Seventh list line

% First list line % Second list line %% Third list line %% Fourth list line %%% Fifth list line % Sixth list line % Seventh list line

Listing 6.5 - Nested Lists Example

6.3.3.4 Mixing List Types

List types can be mixed. Here's an example:

First list line

Second list line

•

Third list line

•

Fourth list line

1.	Fifth list line

Sixth list line

Seventh list line

% First list line % Second list line %- Third list line %- Fourth list line %-% Fifth list line % Sixth list line % Seventh list line

Listing 6.6 - Mixed List Type Example

List types can also be mixed inappropriately, where different list types are used at the same depth. For example:

1.	First list line
2.	Second list line
•	Third list line
1.	Fourth list line
2.	Fifth list line

6.3.4 Simple Titles

Simple titles may only be used in association files, assembly source files and assembly include files. A simple title begins at the start of the line with the characters "*T", followed the title's text.

Example:

*T Hello

6.3.5 Simple Subtitles

Simple subtitles may only be used in association files, assembly source files and assembly include files. A simple subtitle begins at the start of the line with the characters "*t", followed the subtitle's text.

Example:

*t Hello

6.3.6 Section Headers

Section headers may only be used in document files and specification files. A section header begins at the start of the line with the characters "*s", followed by a digit from 0 to 3 that determines the type of section (3 = chapter, 2 = section, 1 = subsection, 1 = paragraph), a comma, an optional reference name (that can by used to reference the section with "[d:<reference_name>]" or "[s:<reference_name>]"), another comma, then the section's title.

Examples:

	`*s1, , Section Headers`
	`*s2, assembly, Assembly Language Source Code`

Note: If three or more section headers are used, then a table of contents will be automatically generated from the section headers.

6.3.7 Listings

A listing consists of a listing header and listing lines. A listing header begins at the start of the line with the characters "*%" followed by a digit from 0 to 3 that determines the style of the listing, a comma, an optional reference name (that can by used to reference the listing with "[%:<reference_name>]"), another comma, then the listing's title.

Listing lines begin with the character '+' followed by the listing line's text.

*%0, listingExample0, Listing Style 0 Example +Hello + Goodbye

Listing 6.7 - Listing Style 0 Example

*%1, , Listing Style 1 Example +Hello + Goodbye

Listing 6.8 - Listing Style 1 Example

1: *%2, , Listing Style 2 Example 2: +Hello 3: + Goodbye

Listing 6.9 - Listing Style 2 Example

1: *%3, , Listing Style 3 Example 2: +Hello 3: + Goodbye

Listing 6.10 - Listing Style 3 Example

Note: If three or more listing headers are used within a document file or a specification file, then a table of listings will be automatically generated from the listing headers.

6.3.8 Tables

A table consists of a table header, table rows, and table fields. A table header begins at the start of the line with the characters "*#" followed by a digit from 0 to 3 that determines the style of the table, a comma, an optional reference name (that can by used to reference the table with "[#:<reference_name>]"), another comma, then the table's title. A table row begins with the '+' character and consists of table fields seperated by a comma, where table fields contain nothing or normal text (potentially with simple markup). Within table fields the simple markup "[,]" is used for a comma that doesn't seperate table fields.

*#0, , Table Style 0 Example +first, second, third +foo, bar +,hello[,] goodbye

Listing 6.11 - Table Example

All of the following table style examples use the same source text (shown in Listing 6.11: Table Example), except for the different style character.

first	second	third
foo	bar
	hello, goodbye

Table 6.2 - Table Style 0 Example

first	second	third
foo	bar
	hello, goodbye

Table 6.3 - Table Style 1 Example

first

second

third

foo

bar

hello, goodbye

Table 6.4 - Table Style 2 Example

first

second

third

foo

bar

hello, goodbye

Table 6.5 - Table Style 3 Example

Note: If three or more table headers are used within a document file or a specification file, then a table of tables will be automatically generated from the table headers.

6.3.9 Figures

A figure consists of a figure header and figure lines. A figure header begins at the start of the line with the characters "*f" followed by a digit from 0 to 3 that determines the style of the figure, a comma, an optional reference name (that can by used to reference the figure with "[f:<reference_name>]"), another comma, then the figure's title.

Figure lines begin with the character '+' followed by the figure line's text. The format and of the figure line's text depends on the figure style.

Note: If three or more figure headers are used within a document file or a specification file, then a table of figures will be automatically generated from the figure headers.

6.3.9.1 Figure Styles 0 and 1

Figure styles 0 and 1 work the same as a listing, where text is display "as is". It's intended for simple figures, where a graphics image isn't necessary.

*f0, figureExample0, Figure Style 0 Example + ~~~ + ~~~00 + @ > + \ _/ + || ASCII Art...

Listing 6.12 - Source For Figure Style 0 Example and Figure Style 1 Example

~~~ ~~~00 @ > \ _/ || ASCII Art...

Figure 6.1 - Figure Style 0 Example

~~~ ~~~00 @ > \ _/ || ASCII Art...

Figure 6.2 - Figure Style 1 Example

6.3.9.2 Figure Styles 2 and 3

Figure styles 2 and 3 are intended to display lists of ranges. Each line contains one range, and begins with a '+' character, an optional relative size of the range, and optional range description, a comma, and an optional "bottom of range" note. Any line that doesn't have a relative size or a comma is treated as an additional line of text (not part of any range).

*f2, , Figure Style 2 Example +,Top of RAM +5 RAM, 0x00100000 +2 ROMs, 0x000C0000 +2 Video display memory, 0x000A0000 +1 Extended BIOS Data Area +3 RAM, 0x00000500 +1 BIOS Data Area, 0x00000000 +Note: Address ranges not to scale.

Listing 6.13 - Source For Figure Style 2 Example and Figure Style 3 Example

	_Top of RAM
RAM	_0x00100000
ROMs	_0x000C0000
Video display memory	_0x000A0000
Extended BIOS Data Area
RAM	_0x00000500
BIOS Data Area	_0x00000000

Note: Address ranges not to scale.

Figure 6.3 - Figure Style 2 Example

	_Top of RAM
RAM	_0x00100000
ROMs	_0x000C0000
Video display memory	_0x000A0000
Extended BIOS Data Area
RAM	_0x00000500
BIOS Data Area	_0x00000000

Note: Address ranges not to scale.

Figure 6.4 - Figure Style 3 Example

6.3.9.3 Figure Styles 4 and 5

Figure styles 4 and 5 are for including images. Each line begins with a '+' character and contains the URLs for one or more images, seperated by commas.

*f4, , Figure Style 4 Example +http://bcos.hopto.org/pic/icon_biggrin.gif,http://bcos.hopto.org/pic/icon_biggrin.gif +,http://bcos.hopto.org/pic/icon_biggrin.gif +,http://bcos.hopto.org/pic/icon_biggrin.gif,http://bcos.hopto.org/pic/icon_biggrin.gif

Listing 6.14 - Source For Figure Style 4 Example and Figure Style 5 Example

Figure 6.5 - Figure Style 4 Example

Figure 6.6 - Figure Style 5 Example

Generated on Sun Aug 23 08:08:26 2009