%==============================================================================% % Start of Ch1.tex % %==============================================================================% % % Copyright % --------- % Copyright (C) 1992 Ross N. Williams. % This file contains a chapter of the FunnelWeb Hacker's Manual. % See the main TeX file for this manual for further information. % %==============================================================================% \chapter{FunnelWeb Design} \label{chapdesign}\xx{FunnelWeb}{design} This chapter contains notes on the design of FunnelWeb. These notes were not created \dq{from scratch}, as the original version of FunnelWeb (\newterm{FunnelWeb V1}) was designed and constructed in a hurry late in 1986, and no design notes were ever recorded. These notes are in fact the result of a complete review of the FunnelWeb design that took place in late 1991 as part of the process of upgrading it for public release. Throughout the design process I tried to stick to the principles of simplicity and clarity. As a rule, it was considered more important that a feature be simple and not allow the user to outsmart himself than it was for the feature to be particularly crisp. For example, the FunnelWeb macro calling syntax is not as crisp as the C syntax, but is more visible. To some extent the design review was influenced by the requirements of backward compatibility.\xx{backwards}{compatibility} During the review it was hard not to think about all the source files in FunnelWeb~V1 format that I have written over the years that would have to be converted were I to significantly deviate from the old input language. Luckily it turned out that there was little clash between these interests, and the resulting design is both clean and does not require much conversion of old source files. The main revision is in the document structuring facility. FunnelWeb~V1 had just two levels of heading indicated by \p{@*@*} (for major headings) and \p{@*} (for minor headings). This scheme (which was copied from Knuth's Web) has been replaced by a hierarchical scheme of five heading levels levels (\p{@A}$\ldots$\p{@E}). This chapter is rather unstructured, acting as it has, mainly as a dumping ground for random ideas about FunnelWeb. \section{Motivation for FunnelWeb} \xx{FunnelWeb}{motivation} During 1986, I was exposed to Donald Knuth's\xn{Donald}{Knuth} \p{WEB} literate programming system in the form of Jon Bentley's\xn{Jon}{Bentley} \i{Programming Pearls}\x{programming pearls} column in \i{Communications\x{Communications of the ACM} of the ACM}\/\paper{Bentley86}.\footnote{As it happens, this was somewhat of a lucky encounter. I had only just joined the ACM and the May 1986 issue of CACM in which the column appeared was the first issue I received.} This prompted me to obtain a copy of the report on the Web system\paper{Knuth83} and to try out the program which had been installed on the local Vaxen. Web was the best system that I had seen for producing printed and online, inline documentation. To me the most extraordinary aspect of the system was its success despite the fact that it had been built into the horribly antiquated file/batch processing systems that we all know and love (and use). I had imagined sophisticated documentation systems before this time, but had always assumed that they would be parts of complex programming environments. Knuth showed that, to some extent, it can be done using 1960s software technology (excluding the 1980s typesetting technology). This was exciting. The Web system was enticing and promising but to me suffered from many drawbacks, many of which Knuth had promoted as advantages. The following highly subjective list of disadvantages formed a springboard for the construction of FunnelWeb. \begin{itemize} \item Web can only process Pascal\x{Pascal} programs. \item Web can produce only one output file. In many instances it is desirable to generate more than one output file. For example, when programming in Ada, it is desirable to write a package specification and a package body together in the same file.\xx{number}{output files} \item Web enforces Knuth's individual style of indentation. Web supplies commands to over-ride the automatic indentation but it is an uphill battle and the code becomes clogged up with format directives.\x{indentation} \item Web does not cater for non-standard Pascal programs. In particular, all identifiers are truncated to about eight characters. \item Web formats the program output file into a form that is unreadable to humans. \item Web does not provide an include facility. This was considered a feature essential for supporting macro libraries. \item Web provides macros with at most one parameter. Knuth describes a hack that can extract a multiple parameter macro facility from a single parameter one, but it is hardly satisfactory. \item Web does not provide conditionals. \end{itemize} Most of these objections boiled down to two points: that Web is far too specialized, and that Knuth's \dq{Occam's Razor}\x{Occam's razor} had cut too far. What I wanted was a documentation system that employed all the same principles as Web but was far more general. The result was FunnelWeb~V1, which can process programs in any language or any combination of languages at the cost of typesetting the text in \p{tt font}. Originally, it was intended that FunnelWeb would be typesetter independent as well as language independent. It was intended that a format file consisting of a set of productions describing how the document file was to be formatted would be handed to FunnelWeb along with the input file. In the end, time pressures forced me to take the back door and hack up a \TeX{} document file generator. This compromise has found its way into FunnelWeb~V3.0 which is still reliant on \TeX{}, although V3.0 at least encourages input files to be typesetter independent. It is hoped that future versions of FunnelWeb will include more sophisticated typesetting facilities. \section{Indentation} \x{indentation}\xx{indentation}{no}% \xx{indentation}{blank}\xx{indentation}{text} A macro call that does not appear at the left margin is called an \newterm{indented macro call} and seems to lead to three different alternatives for its expansion: \newterm{no indentation}, \newterm{blank indentation}, and \newterm{text indentation}. Here are examples of each kind of indentation. First the example problem. \begin{verbatim} @$@==@{@- Aardvark Walrus@} @O@==@{@- Zebra@ Giraffe @} \end{verbatim} There are three ways that the second line of the \p{Sloth} macro can be indented. \thing{No indentation:} \begin{verbatim} ZebraAardvark Walrus Giraffe \end{verbatim} \thing{Blank indentation:} \begin{verbatim} ZebraAardvark Walrus Giraffe \end{verbatim} \thing{Text indentation:} \begin{verbatim} ZebraAardvark ZebraWalrus Giraffe \end{verbatim} No indentation is useful where the user wishes to deal with the output stream as a pure output stream. Blank indentation is useful when the user wishes to generate indented computer programs. Text indentation is useful where the user wishes to prefix each line of an entire macro invocation with a string. This can be useful for commenting out code (\eg{}in Ada using \p{--}), and for prepending things like a dollar sign at the start of each line an a VAX VMS DCL script command file.\x{DCL} FunnelWeb~V1 provided a choice of no indentation or blank indentation. The choice was made in the command line and could not be overridden. The design questions are as follows: \begin{enumerate} \item Which of the three kinds of indentation should FunnelWeb support? \item What should be the granularity of swapping between indentation modes? \item Are particular indentation modes dangerous? \item Is the presence of particular combinations of indentation modes confusing to the user? \item How and when should the choice of indentation be specified? \end{enumerate} All sorts of schemes were considered, including a finely grained system in which the user could specify at the point of call which indentation mode should be used for the called macro expansion. After a lot of thought, the dominant factor that should affect the design was decided to be the \i{clarity} in the user's mind of the indentation facility and the \i{danger} associated with misunderstanding it. Here are two examples that show how easily a confusion or misunderstanding of the indenting being used can cause danger. The first example shows how blank or no indentation might be misused. \begin{verbatim} --Misuse of blank (and no) indentation. --@ \end{verbatim} Here the user has assumed that text indentation is in action and has placed an Ada comment symbol \dqp{--} before the invocation of the macro \p{@} in the hope that the entire text of the macro would be prefixed by \dqp{--}. The result could be passed by the compiler which would activate all but the first statement in the expansion of macro \p{@}. The next example demonstrates how text indentation could be misused.\xx{indentation}{dangers} \begin{verbatim} --Misuse of text indentation: a++; @ \end{verbatim} Here the user has placed the call to \p{@} after the incrementing of variable \p{a}. The result is that there is a good chance that the \dqp{a++;} prepended to each line of the expansion of \p{@} will be passed by the compiler and will cause \p{a} to be overincremented. These examples are not to be laughed at. It is possible that FunnelWeb will be used widely, and the problems above may cause problems in critical systems. The examples above are particularly scary because they are reflexive. One cannot simply pin the blame on one particular indentation form. A little thought reveals that the greatest danger lies in \i{confusion} in the user's mind. If the user is confused between text or blank indenting, problems will arise. There seem to be two ways to solve the problem. The first is to ban all macro calls that are preceded by non-blank text. This is not a good option because there are so many cases where it is desirable to expand more than one single line macros on the same line. A second option is to eliminate one of the two forms so as to reduce the potential for ambiguity in the user's mind. I choose the latter option. Of the two forms, the clear choice for elimination is text indenting for the following reasons: \begin{enumerate} \item It actually introduces extra text which gives it an a priori potential for problems. \item It is harder to implement and would slow down Tangle. \item It would not be compatible with FunnelWeb~V1 which uses blank indentation. \end{enumerate} The only other decision is the level of granularity of choice between the remaining options: no indentation and blank indentation. FunnelWeb~V1 allowed this choice to be made in the command line. In retrospect, this was bad design because the user might unwittingly code certain macro calls relying on one or the other mode. A better system is to allow the user to specify which mode in the input file itself. This has been done in FunnelWeb~V3. Again, to avoid confusion, it seems sensible to allow the user only one indentation mode per FunnelWeb input file. In most cases, the user will be happy with blank indentation (the default) and there will be no need for change anyway. \thing{Decision:} Implement only \dq{no indentation} and \dq{blank indentation}. Make the choice of indentation a static attribute of a particular FunnelWeb run that is specified in the input file. This solution is the same as FunnelWeb~V1 except that the choice has been moved from the command line to the input file. \section{Review of FunnelWeb Syntax} \xx{FunnelWeb}{syntax} One of the distressing aspects of FunnelWeb~V1 was its clumsy macro definition and calling syntax. Compared to (say) the C preprocessor, FunnelWeb's macro call syntax is like a freight train in a china show. During the FunnelWeb redesign, a complete review of this syntax took place to try to neaten it up. Surprisingly, the V1 syntax survived unscathed with the exception that \dqp{==}\x{==} in macro definitions was made optional. The survival of this clumsy syntax was a product of the design goal of making everything simple, explicit, and not subject to subtle errors. \subsection{Review of Macro Definition Syntax} \xx{macro definition}{syntax} FunnelWeb~V1 used a macro definition syntax that resulted in macro definitions such as the following. \begin{verbatim} @$@==@{@- Open the door Say out Close the door@} \end{verbatim} This is messy, but I couldn't think of anything better at the time. The \p{@\$} is necessary to cue a definition. Without it, the definition might somehow be mistaken for an invocation. The \p{@<} and \p{@>} delimit the name. The \p{@$\{$} and \p{@$\}$} delimit the text. The \p{@-} is a product of the rule \dq{exactly the text between the \p{@$\{$} and \p{@$\}$}}. The only real target is the \dqp{+=} and \dqp{==} which really break all the rules and should be changed. Unfortunately I couldn't think of anything to change them to. If there was no \p{+=} mechanism, we could use: \begin{verbatim} @$@@{ Open the door Say out Close the door@} \end{verbatim} In fact, eliminating \p{+=} is thinkable because it does not appear in many of my existing FunnelWeb source files. This indicates at least that it was not needed much by myself. A minimalist construct could be \begin{verbatim} @ Open the door Say out Close the door@} \end{verbatim} but this is too dangerous for my tastes. \thing{Decision:} For compatibility reasons, retain the old \p{+=} and \p{==} constructs, but make them optional. The new syntax for defining macros is: \begin{verbatim} @$@@{ Open the door Say out Close the door@} \end{verbatim} Next we turn to parameterized macro definitions.\xx{parameterized macro definitions}{syntax} A conventional FunnelWeb parameterized macro definition looks like this: \begin{verbatim} @$@@(@3@)==@{@- Open the door Say out Close the door@} \end{verbatim} which is a bit messy. The natural alternative is even worse: \begin{verbatim} @$@@(@1@,@2@,@3@)==@{@- Open the door Say out Close the door@} \end{verbatim} Thus, just specifying the number of parameters seems sensible. However, perhaps the syntax could be trimmed to \begin{verbatim} @$@@3==@{@- Open the door Say out Close the door@} \end{verbatim} I decided to reject this in favour of the old syntax. \thing{Decision:} Retain the old syntax of $\ldots$\p{@(@3@)}$\ldots$ \subsection{Review of Macro Call Syntax} \xx{macro call}{syntax} Here are some ideas for alternatives to the FunnelWeb~V1 macro call syntax. \begin{verbatim} Open the door @ @! FunnelWeb~V1 style. Close the door Open the door @@ Close the door Open the door @"Say out@" Close the door Open the door @(Say out@) Close the door Open the door @ Close the door \end{verbatim} \thing{Decision:} Continue with the old notation. It may not be neat, but at least it is clear and consistent. The main temptation is the format \p{@@} which looks rather good. However, it breaks the special sequence rational and hence is too confusing. \subsection{Review of Parameterized Macro Call Syntax} \xx{parameterized macro call}{syntax} FunnelWeb~V1 provided a messy parameterized macro call syntax: \begin{verbatim} @@(@"firstparam@" @, @"Secondparam@" @, @"thirdparam@" @) \end{verbatim} This syntax can be cleaned up considerably by making the \p{@"} symbols optional. This results in calls such as the following: \begin{verbatim} @@(firstparam@,Secondparam@,thirdparam@) \end{verbatim} As the first form allows the alignment of complicated parameters by allowing white space to be inserted outside the \p{@"}, and the second form is cleaner, both are retained simply by making the quotes optional. \thing{Decision:} Make the double quotes optional. \section{Document Structuring} \xx{document}{structure}\x{headings}\x{sections} Experience with FunnelWeb~V1, which provided only two levels of headings (major and minor) proved that there was a strong need for fully hierarchical multiple-level headings. The only question was how it should be done. Here are some ideas that were considered. \begin{verbatim} @*@
@**@ @***@ @*@
@*@*@ @*@*@*@ @s@
@ss@ @sss@ @s@
@s@s@ @s@s@s@ @S@
@SS@ @SSS@ @S@
@S@S@ @S@S@S@ @A@
-- The syntax finally chosen. @B@ @C@ @A Main Program @B Read the Message @C Encrypt the Buffer @*A Main Program @*B Read the Message @*C Encrypt the Buffer @1@
@2@ @3@ (using @A..@I@ as macro parameters or overload @1..@9) @*@1@
@*@2@ @*@3@ (using @A..@I@ as macro parameters or overload @1..@9) \end{verbatim} Choosing between these alternatives was not easy. The following thoughts contributed to the decision. \begin{itemize} \item Syntaxes that require visual counting are probably not a good idea. \item Syntaxes that do not delimit the heading name somehow are likely to cause problems where heading names are omitted. Users will be tempted to start paragraphs after the start of heading symbol and the result is that the first line of the paragraph will be sucked into the heading. \item Overloading the \p{@1}, $\ldots$, \p{@9} sequences is undesirable. \end{itemize} \thing{Decision:} Use \p{@A}$\ldots$\p{@E} with optional following macro name syntax for the section name. Note: We stop at \p{@E} because five levels is probably sufficient, and we may wish later to use \p{@F} for \b{F}ile (to augment or replace \p{@O}). \section{Discussion of Some Miscellaneous Issues} \xx{miscellaneous}{issues} \thing{Comment duplication:} If\xx{comment}{duplication} the FunnelWeb user inserts comments into the target code (in the \p{.fw} file) as well as into the documentation (free text) part of the \p{.fw} file, then it is possible for the situation to get a bit silly. \b{Decision:} This is a problem for the programmer, not for FunnelWeb. \thing{Out-of-date documentation:} Sometimes\xx{out of date}{documentation} it is all too easy for the programmer to modify the code without updating the surrounding documentation. \b{Decision:} This is a serious and major problem. In an automated environment, it may be possible to create a system of dependencies between scraps of code and scraps of documentation. However, it is hard to see how a tool such as FunnelWeb could provide support for prevention of this sort of error. \thing{Meta-macro-level parameterization:} Sometimes, when using FunnelWeb, the facility to use one macro to construct the name of another has been needed. \b{Decision:} Allowing macro names to be constructed would lose the simple nature of the preprocessor and so this suggestion is rejected. \thing{Clumsy notation:} The \p{@} notation can be clumsy. \b{Decision:} This is necessary to maintain the simplicity of the translation. \section{Automated Regression Testing} \xx{regression}{testing} Automated regression testing is extremely important for two reasons: \begin{enumerate} \item It provides confidence that changes made to the program have not introduced bugs. \item It allows portability problems to be pinpointed when the program is moved to a new machine. \end{enumerate} The simplest way to set up automated regression testing is to construct a suite of test cases (and their solutions) and then write a script in the target machine's command language to run through the suite. Unfortunately, there is no command language that is shared among the machines to which FunnelWeb must be ported. These machines are at least: Macintosh, IBM-PC, Sun, VMS Vax. One option is simply to rewrite the script in each machine's particular command language. This would be a feasible option were it not for the fact that the Macintosh (the machine upon which FunnelWeb was developed) does not have a command language! After some thought, I decided that the best solution to the problem was to create a command language \i{within FunnelWeb}. FunnelWeb could then be invoked in two modes, one-shot command line or interactive/script. This approach had the benefit of providing total control over the command language and its complete portability. The result is described in the \i{FunnelWeb User's Manual}. \section{Command Line Interface} \xx{command line}{interface} FunnelWeb~V1 was implemented in Ada\x{Ada} and runs on a VMS VAX.\xx{vms}{vax} As such it has a full VMS DCL command line interface. Here is the \dqp{.CLD} file\xx{.cld}{file} for the DCL command line interface. \beginsmall \begin{verbatim} ! FUNNELWEB Command Definition ! ============================ ! Ross Williams. 28 April 1987. module command_table define verb dummy_command parameter p1 , label=input_file ,value(required,type=$file) qualifier include_files, label=include_files,value(type=$file) ,default qualifier output_files , label=output_files ,value(type=$file) ,default qualifier delete , label=delete ,default qualifier tex_file , label=tex_file ,value(type=$file) ,default qualifier listing_file , label=listing_file ,value(type=$file) ,default qualifier brief , label=brief,value(type=$number,default=5),default qualifier screen , label=screen ,value(type=$number,default=0) qualifier trace , label=trace qualifier file_spec , label=file_spec qualifier compare \end{verbatim} \endsmall As portability was a key goal of FunnelWeb~V3, it was obvious that the command line interface would have to be redesigned. The design goals for the new command line interface were:\xx{design goals}{command line interface} \begin{enumerate} \item The interface should not \i{depend} on case. However, it must allow case-sensitive filenames to be transmitted on systems that have case-sensitive filenames. \item Each option must have a symmetric positive and negative form. For example, it is confusing for \p{-X} to turn on a feature and \p{-Q} to turn it off. \item The interface must be extensible to allow inclusion of more features at a later date. \item Options should be consistent and memorable. \end{enumerate} The result is described in the \i{FunnelWeb User's Manual}. \section{File Name Management} \xx{file}{names} File names present a host of problems for a program like FunnelWeb. First, FunnelWeb can generate so many different kinds of files that conventions must be adopted to prevent them from becoming unmanageable. Second, the constraints on file names, and even the structure of file names themselves varies considerably from machine to machine. These two problems have combined to result in the sophisticated and rather complicated way in which FunnelWeb~V3 handles filenames. To summarize, the three problems are: \begin{enumerate} \item What filename extensions should be chosen for various kinds of file? \item What filename inheritance should take place? \item How should FunnelWeb cope with the variations in filename structure between machines? \end{enumerate} The following three sections address these questions. \subsection{Filename Extensions} \xx{filename}{extensions} FunnelWeb is capable of reading and writing a variety of different kinds of files. In particular, FunnelWeb must often operates in an environment where the same information is stored in many forms (\eg{}prog.fw, prog.c, proc.exe). File extensions are an essential tool in managing this situation. The filename extensions chosen for FunnelWeb are: \begin{verbatim} FunnelWeb : .fw Product : None. Documentation : .tex Listing : .lis Journal : .jrn \end{verbatim} Lowercase will be used in systems that are case sensitive. Readers who are wondering how FunnelWeb copes in environments such as UNIX where there are no file extensions should refer to Section~\ref{portablefilenames}. \subsection{Filename Inheritance} \xx{filename}{inheritance} Inheritance in filenames refers to how input and output files inherit parts of their name from other filenames and their environment. For example if the command \begin{verbatim} fw sloth +J +L +T \end{verbatim} was issued, you would expect to see output files \p{sloth.jrn}, \p{sloth.lis}, and \p{sloth.tex}. The output file names have inherited the \dqp{sloth}. The following table gives the hierarchy devised for FunnelWeb. \begin{center} \begin{tabular}{|l|l|l|l|l|l|l|l|} \hline & Script & Input & Include & Journal & List & Document & Product \\ \hline 1 & & & \p{@i} & & & & \p{@o} \\ 2 & \p{+x} & \p{+f} & \p{+i} & \p{+j} & \p{+l} & \p{+t} & \p{+o} \\ 3 & \dqp{.fws} & \dqp{.fw} & \dqp{.fwi} & \dqp{.jrn} & \dqp{.lis} & \dqp{.tex} & \\ 4 & & & \p{+f} & \p{+f} & \p{+f} & \p{+f} & \\ 5 & DefDir & Defdir & Defdir & Defdir & Defdir & Defdir & Defdir \\ \hline \end{tabular} \end{center} The following notes explain the table. \begin{enumerate} \item This scheme is similar to that used in FunnelWeb~V1. \item The journal, list, and documentation files all fall in the same pattern and can be considered as a single case. \item Level~1 has the highest priority because it is a direct specification by the user in the input file. \item Level~2 comes next because this is also a direct specification from the user on the command line. \item Level~3 provides the default file extensions. Product files do not inherit an extension as they could be of any type. \item Level~5 is built into most operating systems' file specification systems. If I specify file \dqp{x.y}, it is taken to mean on the default disk in the default directory. \item Level~4 looks straightforward, but secretly conceals a difficult design decision. By the time we get down to this level of inheritance, we know for sure that the filename has already picked up a file extension. So all that is left to inherit is the path and the filename proper. Obviously we have to inherit the filename proper (e.g. \p{sloth} in \p{sloth.tex}), but should we inherit the input file path? If we do inherit the input file path, files will be placed in the same directory as the input file. If we don't inherit the input file path, files will be placed in the current directory. The choice I have made is to send all the logging type files into the same directory as the input file. This means, for example, that \p{sloth.lis} and \p{sloth.tex} will generally land in the same directory as \p{sloth.fw}. However, I have decided that output files should be sent to the default directory (if not earlier specified) as this is where the action is. In normal use, the main product of FunnelWeb will be product files and so the user will expect them to appear in the current directory by default. \end{enumerate} \subsection{Portable Structure of File Names} \label{portablefilenames}\xx{file}{names}\xx{portable}{filenames} Another problem with file names is the variation of their structure between environments. Here are examples of some of the formats that prevail: \begin{verbatim} UNIX /device/dir1/dir2/name VMS node::device:[dir1.dir2]name.ext;vn MSDOS device:\dir1\dir2\name.ext MAC device:dir1:dir2:name \end{verbatim} Isn't it amazing that none of these popular systems use the same format? The solution to dealing with these different formats is to classify them as non-portable and hide the functions that manipulate them in the machine-specific module of FunnelWeb. Luckily there are not many such functions. The main problem is coping with file systems that do not explicitly support file extensions. With so many possible input and output files, FunnelWeb all but needs such extensions. Machines that do not support them pose difficult design decisions. If the user specifies \dqp{sloth} as an input file on such a non-extension-supporting system, should FunnelWeb look for \p{sloth} or \p{sloth.fw}? If the user specifies \p{walrus} as a listing file, should it generate \p{walrus} or \p{walrus.lis}? Some possible solutions are: \begin{enumerate} \item Regard the filename \p{sloth} as having an empty extension. It will then default to \p{sloth.fw}. \item Regard the filename \p{sloth} as having a blank but full extension. That is, it cannot be overwritten by inheritance, but it remains blank. \item Provide an extra syntactic mechanism to allow the user to specify one or other of the two options above. \end{enumerate} My solution was to choose the first option. Use of FunnelWeb results in lots of files lying around (\eg{}\p{sloth.lis}) and it is hard to see how the user will cope with them all without some kind of naming discipline. If a naming discipline has to be used, it might as well be the FunnelWeb one. Thus the names of all files read and written by FunnelWeb will have a file extension of from zero to three letters separated from the rest of the filename by a \dqp{.}. The only exception is product files whose extension is left undefined. Product files need not contain a \dqp{.} and a file extension, although they can inherit one if the user wishes. \section{Specifying Constraints on the Number of Instantiations} \xx{invocations}{number} Experience with FunnelWeb~V1 demonstrated the need to be able to specify in macro definitions how many times it was expected that the macro would be used. FunnelWeb~V1 generates an error if a macro is not used, but permits macros to be called more than once. This caused problems for macro libraries, which would be included, but whose macros were often not called. By default, FunnelWeb~V3 requires that each macro (except for the ones attached to output files) be called exactly once. However, it also provides syntax that allows the user to specify that a macro be allowed to be called zero times or many times. This allows a macro to be specified with the following permissible ranges of numbers of calls depending on the presence or absence of \dqp{@Z} and \dqp{@M}: \begin{verbatim} 0..1 @$@@Z==... 1 @$@==... 1..n @$@@M==... 0.....n @$@@Z@M==... \end{verbatim} The only two problems with this scheme are: \begin{enumerate} \item It is incompatible with FunnelWeb~V1 files, as the default in FunnelWeb~V1 is $1 \ldots n$ whereas the default in new FunnelWeb is $1$. This is not a big problem because most macros in the old files were used exactly once. Only a few macros will have to be changed. \item What should the syntax be? (above is a sneak preview only!) \end{enumerate} The initial proposal for syntax was to allow the user to insert zero, one, or both of \p{@?} and \p{@M} just after the \p{@\$} of a macro definition. However, this has the following drawbacks: \begin{enumerate} \item It uses two sequences that are desirable to reserve (\p{@?} for conditionals and \p{@M} for macro). \item It stops the user from searching for the string \p{@\$@@(@5@)==@{@- This is a short macro. With only a line or two@} @$@@?@M@(@5@)==@{@- This is a short macro. With only a line or two@} @$@@(@5@)@?@M==@{@- This is a short macro. With only a line or two@ \end{verbatim} The first form puts me off because I think that it is a good visual rule to start all the macros with \p{@\$}. The second form puts me off because it detaches the macro name from the parameter list, thus making it look less like a call, which is desirable syntactic resonance. The third form is messy but probably workable. Because we are right next to the tested constant string (either \p{==} or \p{+=}) we could augment it further. For example, \p{01==} could allow a macro to be called from 0 to 1 times. The main problem with this is that we are trying to phase out \p{==} anyway! Nevertheless, all the logic points to after the parameter list as the best place to locate this information. After some thought, it was decided that the \dqp{@?} sequence be reserved for a possible future conditional facility, and so \p{@Z} was used instead. \thing{Decision:} The position is after the parameter list. The notation is \p{@Z} for zero, and \p{@M} for many.\x{@Z}\x{@M} Example of final syntax: \begin{verbatim} @$@@(@5@)@Z@M+=@{@- This is a short macro. With only a line or two@} \end{verbatim} \section{The Relationship Between Document Structure and Macro Structure} \xx{document structure}{macro structure}% \xx{document}{structure}\xx{macro}{structure}\note{01-Dec-1991}% \xx{hierarchical}{structure} Having already decided upon a fully hierarchical document structure, I determined to refine the details. The issues to be addressed were as follows: \begin{itemize} \item How should the hierarchical structure connect to the macro structure? \item How can backwards compatibility be achieved? Should it? \item Should the macros be cross referenced by section or by definition? \item Should nameless sections inherit macro names as headings? \item Should we simply use \TeX{} macros to structure the document? \end{itemize} After some thought, I arrived at the following thoughts: \thing{Basically a \TeX{} file:} One option is simply to treat each \p{.fw} file as a \TeX{}\x{TeX} file laced with macros. That is, a \p{.fw} file could be structured as a real \TeX{} file from which FunnelWeb could extract macro definitions. This approach is feasible because \TeX{} could be programmed to respond to the \p{@} sequences in the same way that FunnelWeb responds to them. Thus, there would be no need for Weave. I rejected this approach, first because it is too typesetter-dependent, and second because it complicates the inclusion of any sort of complicated post-processing in the documentation file. This would have to be implemented in \TeX{}. A better approach is to use an invented section notation (e.g. \p{@A @B @C}). This maintains FunnelWeb's typesetter independence and can easily be converted into \TeX{} sectioning macros by Weave. In the same spirit, it might be worth introducing a few \p{@} sequences for certain general typesetting operations such as italics and program code. \thing{Confusion in FunnelWeb~V1 Heading Numbering:} FunnelWeb~V1 uses two levels of section headings, but numbers all the headings sequentially. In this, it is a little confused. Clearly with a fully hierarchical document structure, the headings cannot be numbered sequentially --- the numbering must reflect the structure (\eg{}3.2.1). \thing{Hierarchical numbering is messy for macros:} Unfortunately, hierarchical numbering\xx{section}{numbering} is messy and confusing when applied to macro names. In FunnelWeb~V1's typeset output, each macro call has appended in square brackets the number of the section in which the macro is defined. Use of hierarchical numbering would be somewhat messy. For example, a macro call might look like. \begin{verbatim} Write out the output[6.7.4.3] \end{verbatim} Similarly, cross reference lists would be messy: \begin{verbatim} This macro is used in 3.4.5, 1.2, 7.8.9, 7.4, 2.2.1.1. \end{verbatim} \thing{Separate numbering for macros and headings:} One idea is to use hierarchical numbering for the sections, but to number the macros sequentially. This could be a little confusing in documents without much structure, but would be very much less messy than cross referencing using hierarchical numbering. Also, it will be easier to find macros indexed by a sequential number than by section, which has a less direct relationship with page bulk and number. By macro numbering is meant the sequential numbering of each macro body part through the whole document. \thing{Input format matters more than output format:} At this point we realize that a distinction should be made between the \i{input format} and the \i{typeset output}. The critical issue here is not how the program should be formatted for printing, but rather the format of its \p{.fw} file. The typeset output can always be changed simply by fiddling with Weave. However, as soon as the document structuring features of FunnelWeb are fixed, they will be used in dozens or hundreds of documents and it will be very difficult indeed to change them. Therefore, the important thing is to provide as sensible and expressive a \p{.fw} format as possible. It is therefore a separate decision as to whether we should number macros by section or by sequence number. The important thing is to address the format and rules for the expression of structure. \thing{Naming sections:} The naming of sections requires some thought. In many cases (especially in the case of high-level sections) the writer will provide an explicit name for a section. In other cases, provision of such a name will merely duplicate the name of the macro contained within the section. It therefore makes sense to allow the user to omit the name from a section, with Weave naming the section after the first macro definition in the section. If a macro is unnamed and there is no macro in the section, an error can be generated. All these thoughts lead to the following scheme: \begin{itemize} \item Documents will be hierarchically structured using \p{@A}, \p{@B} etc. \item Each section can be given a name delimited by \p{@<@>}. \item Sections that do not have names inherit the name of their first macro. \item If a section does not have a name or a macro, it is erroneous. \item Sections will be numbered hierarchically either by FunnelWeb or by \TeX{}. \item Macro body parts will be numbered sequentially by FunnelWeb and cross referenced by these numbers. \end{itemize} All this results in a system which: \begin{itemize} \item Provides a hierarchical document structuring capability. \item Is typesetter independent. \item Does not require duplication between heading and macro names. \item Separates the heading and macro systems so that Weave can be configured at a later date to cross reference in different ways without requiring input files to be reworked. \end{itemize} \section{Diagnostic Messages} \xx{diagnostic}{messages} In FunnelWeb, all error messages commence with an indicator indicating the severity of the error message. Here are some of the formats that I investigated before settling on the final format: \begin{verbatim} W--Error creating sloth. E--Error opening output file. S--I'm a teapot. F--Can't open output file. W-Error creating sloth. E-Error opening output file. S-I'm a teapot. F-Can't open output file. W:Error creating sloth. E:Error opening output file. S:I'm a teapot. F:Can't open output file. W: Error creating sloth. -- Format chosen. E: Error opening output file. S: I'm a teapot. F: Can't open output file. War-Error creating sloth. Err-Error opening output file. Sev-I'm a teapot. Fat-Can't open output file. W-Old fashioned feature. W-Old fashioned feature. W--Old fashioned feature. W: Old fashioned feature. W:Old fashioned feature. \end{verbatim} \section{Summary} This rather unstructured chapter has addressed some of the key design decisions of FunnelWeb. In many cases, the alternatives have been unpleasant, but I am confident that in all cases, a fully workable solution has been found. %==============================================================================% % Start of Ch1.tex % %==============================================================================%