The content extractor

The engine consumes records; the code and the tests they cover are C files in another repository. affirmatrix.sources.content.CSourceExtractor sits between the two: from the need exports, the Doxygen output and the source files it supplies one Implementation or TestSpecification record per need, each with two content hashes, and one edge per declared link. The command line composes it, with the other configured readers, into the current stream (see Command line and configuration). The Python binding of the same principle is not built (see the last section).

What it reads

extractor = CSourceExtractor(
    Path("checkout"),
    repository="toolbox",
    implementations=config.ImplementationInputs(
        export=Path("needs/api-traceability/needs.json"),
        doxygen=Path("xml/dox-safe-data-api"),
    ),
    specifications=config.SpecificationInputs(
        export=Path("needs/test-specification/needs.json"),
        doxygen=Path("xml/dox-safe-data-testspec"),
        types=frozenset({"test_case"}),
        doxygen_prefix="checkout/",
    ),
    specification_placement=Placement("tests-repository", Path("tests-checkout")),
)

Three kinds of input, each for one purpose:

the need exports give the structure: which nodes exist, their identity and their links (SEG-SREQ-153). Where need types are configured for an export, only the needs of those types supply a record, and a need of another type is never refused (SEG-SREQ-275, SEG-SREQ-277). The loader refuses a list of types that holds an empty text (SEG-SREQ-341), and a null list means none are configured (SEG-SREQ-369). Where none are configured, every need in the implementation export is an Implementation and every need in the test-case export a TestSpecification (SEG-SREQ-276). The symbol a need names is its title on an implementation need and its test_function on a test-case need;
the Doxygen output gives the location: the file and the lines of a symbol’s declaration and body. Only names, kinds and <location> attributes are read from it; none of its text reaches a hash (SEG-SREQ-170);
the source files under the stream’s root give the bytes that are hashed.

Each stream has its own place: the configured name of the repository its files belong to, never a path (SEG-SREQ-134), and the path of that repository. A stream without a Placement of its own takes repository and root as its place. A stream set to None supplies nothing, and with both None the extractor supplies nothing at all.

The paths in a Doxygen output come from Doxygen’s own base directory. A stream may name a prefix, the text that each of those paths begins with (SEG-SREQ-279). The extractor removes the prefix as written, and reads the rest within the stream’s repository. A path that does not begin with the prefix is an error for its node and is not read (SEG-SREQ-280). So is a path whose remainder after the prefix is empty or begins with a separator. The error names the path and the prefix (SEG-SREQ-342). A prefix that ends with the separator avoids that error. With no prefix, the path is used as it stands. A prefix is text, not a path, so the configuration file’s directory plays no part in it. It should end with the path separator.

A stream may also name a path root: a directory inside the repository that is put in front of each path after the prefix is removed (SEG-SREQ-348, SEG-SREQ-349). Doxygen often names a header by the path that a programmer writes in an include line, which is not the path in the repository. The root bridges the two. The prefix is removed first, with the errors above, and then the root is put in front, with one separator between them whether or not the root ends with one. An empty root and an absent root put nothing in front. The joined path is the path inside the repository everywhere below: it is the file that is read, the path of the anchor, and the path that a test’s module is compared with. The text of a node (SEG-SREQ-311) is read from it too. The extractor does not clean the joined path. A root that is absolute or leads out of the repository gives a path that the next step refuses, naming the node.

What it supplies

Each node is identified by its need identifier, verbatim (SEG-SREQ-154, SEG-SREQ-155; ADR-0007): a renamed symbol does not rename the node. Each link in satisfies becomes an Implements edge from the implementation to the requirement, each link in verifies a Verifies edge from the test specification to the requirement, in the pending state (SEG-SREQ-156, SEG-SREQ-157). Edges need neither the Doxygen output nor a source file. On the toolbox evidence fixture that is 12 Implementations, 19 TestSpecifications and 16 + 24 edges.

Finding the lines

Each Doxygen tree is read once, when the extractor is built. Its members are indexed by name, each tree separately, so an implementation’s symbol is never looked up in the test tree. A node is located through the one member whose name is its symbol (SEG-SREQ-160). “One” means one distinct definition: a member listed in two compounds with the same id and location counts once, but two id values, or one id at two locations, are two. Where several members share the symbol and a test-case need names a test_module, the members whose file lies within that directory are the candidates (SEG-SREQ-278). A test_module that is absent, null or empty text names no module (SEG-SREQ-343). One that is present and is neither null nor text is an error for its node, found before the members are looked up (SEG-SREQ-344). The file is compared as a path inside the repository, after the prefix is removed and the path root is put in front, and by whole path components: tests/a does not contain tests/ab. A member whose remainder after the prefix is no relative path is an error for the node, not a member outside the module. After that choice, a symbol with none or with several members is an error for its node (SEG-SREQ-161). Members no need names are never consulted, which is why the test tree’s duplicate SAFE_CONTAINER_DEFINE costs nothing. The index of Doxygen’s own listing is not followed: it lists members repeatedly and gives no location.

Before a file is opened, every path the node’s location names has its prefix removed, its path root put in front, and is resolved under the root of the node’s stream (links included). It must stay inside that root; .., an absolute path and a link that leads out are errors, and the file outside is not read (SEG-SREQ-162). A path inside the root of another stream is outside this one. Then the line each location names must contain the symbol as a whole identifier: line and declline in the declaration file, and bodystart in the body file, never bodyend (SEG-SREQ-175). A location that Doxygen left stale, pointing at another function’s lines, is refused instead of hashed.

The four hashes

A hash is the SHA-256 of a run of whole lines of one source file, each with its terminator (SEG-SREQ-168); nothing is trimmed, so a CR before the line feed is content. ADR-0011 fixes where each run starts and ends:

Node and hash	First line	Last line
Implementation, `apiHash`, function declared in a header	the opener of the comment above the declaration	the line of the first `;` at parenthesis depth zero
Implementation, `apiHash`, function defined where it is declared	the opener of the comment above the definition	the line before the one that opens the body with `{`
Implementation, `apiHash`, macro	the opener of the comment above the `#define`	the `#define` line
Implementation, `bodyHash`	`bodystart`	`bodyend`
TestSpecification, `specHash`	the opener of the comment the search for a test finds above it	the closer of that comment
TestSpecification, `implHash`	`bodystart`	`bodyend`

The documentation comment of an Implementation is found from the line above the declaration, stepping over #if, #ifdef and #ifndef lines only. The next line must end the comment with */, and its opener must be /** or /*!. A blank line, a plain comment or code there means the node has no documentation comment, an error and never an empty hash (SEG-SREQ-169). A guard line between the comment and the declaration lies inside the span; one above the comment lies outside.

The search for a test is wider, as Doxygen attaches a comment to the next member (SEG-SREQ-166). From the line above the test, it steps over blank lines, over conditional lines (#if, #ifdef, #ifndef, #elif, #else and #endif) and over plain comments. A plain comment is a block comment that is not a documentation comment, with only white space before its opener on its line. The first other line must end a documentation comment. Code, any other directive, a line comment or the top of the file there means the test has no documentation comment, an error and never an empty hash (SEG-SREQ-283). The span runs from the opener to the closer, so the lines stepped over lie outside it and changing them changes no hash. The search does not read the text of the comment. A test that loses its own comment can therefore take a group comment above it, when only such lines lie between. A test whose member Doxygen finds in one branch of a conditional, with its comment above the other branch, is still an error, because code lies between. When the end of a declaration is sought, text in comments is skipped, and a { at parenthesis depth zero before the ;, an unbalanced ) or no ; at all is an error: such a declaration is not one this rule can end. A body that Doxygen records as ending at -1 has no lines and is an error too.

Anyone can reproduce a hash without the tool:

sed -n '247,263p' include/safe_data/safe_data.h | sha256sum

prints the apiHash of safe_data_init, dcfd58ce00f23973afff2aa76f3b11cc422aad373981488c22b1745eb325ba9c: the comment opens at line 247 and the semicolon that ends the declaration is on line 263.

The anchors

Each hash is supplied with an anchor of three parts, naming where its bytes came from: the configured name of the stream’s repository; the path of the file inside that repository (SEG-SREQ-281), which is the file as Doxygen names it with the prefix removed and the path root put in front, and which for an apiHash is the declaration file and for a bodyHash or implHash the body file; and the locator symbol:<name>#api, #body, #spec or #impl (SEG-SREQ-171 to SEG-SREQ-174). No line number is recorded, so moving a function within its file changes no anchor.

The bytes behind a hash

The extractor meets the protocol ContentSource (SEG-SREQ-311). The method content(local_id, hash_name) gives the span that the hash covers. It finds the need in the stream. It locates the member in the Doxygen index, which exists since the extractor was built. Then it cuts the span from the source files as they are now. nodes() and content() read through one function, so a hash and its bytes cannot follow two rules. A call reads at most three source files and keeps nothing. The answer is None for an unknown identifier and for a name that is not one of the stream’s two hashes.

Errors

ExtractorError is raised in three places. When the extractor is built it refuses what the exports and the Doxygen trees alone show, before any record is supplied: an export that cannot be read, that holds no or several versions or no needs, or that carries a build timestamp (SEG-SREQ-158); a need with no symbol, or an identifier other than its key; a Doxygen directory that is missing or a file of it that does not parse. The export checks are the requirements reader’s, shared through one private helper.

The input of a stream of implementations may carry a list of need identifiers (SEG-SREQ-357). When it does, a need is read when its key is in the list and, where types are configured, when its type is one of them (SEG-SREQ-358). Both filters apply before any need is checked, so a need outside them is never refused for its shape (SEG-SREQ-374); a need that the list names keeps every check. Every listed identifier that no admitted need carries is named in one error, one line for each, when the extractor is built (SEG-SREQ-360). A list that is null means no list. The same filters are one function that the outcome extractor shares.

The need checks do not stop at the first fault. For each export, the extractor gives one error that names every misshapen need (SEG-SREQ-351). The first line of the error gives the count, and each next line holds one need and its reason, in the order of the export, so the same export always gives the same text. A need with an empty symbol is named in the same list.

The repository paths are checked first, before any export is read (SEG-SREQ-373). Each repository that a stream reads through must be a directory. A path that does not exist or is a file is refused, once for each configured name, whatever number of streams read through it. The error starts with a line that counts the repositories. Each next line holds the configured name, the path and the cause, sorted by the configured name. No need is named for this cause, because a repository that cannot be used would give the same fault to every need that reads through it. A plain directory that holds the files is valid. The repository of the requirements reader is not checked, because that reader opens no file.

Everything that needs a location or a source is checked when nodes() reaches the node. A node that fails does not stop the pass and is never skipped. The records of the other nodes are supplied as they are reached. At the end of the stream the extractor raises one UnsuppliedNodesError that names every node it cannot supply (SEG-SREQ-282). The first line of its message gives the count. Each next line holds one need, its symbol and its reason, so a reader finds one need on one line. The same facts are in the failures attribute. The stream is never silently short. A consumer that writes as it reads could leave half a graph behind, so it must consume the whole stream before it writes, as the drift derivation consumes both record streams completely.

The third place is content(). A node that cannot be located or cut raises an UnsuppliedNodesError that holds this one failure.

Reading the sources

Each source file is read once for one pass of nodes(), so every node of that pass sees the same bytes; a second pass reads again, and so does each call of content(). The functions that cut the spans are public so that an auditor can call them without building an extractor: split_lines, span, find_comment, find_test_comment, declaration_end and head_end.

The Python binding

The other binding of the same principle, docstring markers located by ast, is not built. Its design is stated in the module docstring of affirmatrix.sources.content: the marker declares identity and edges, the parser only locates, and the hash is over the verbatim byte span.