Outcome Extractor
The outcome extractor reads one run artifact per run bundle it is given and supplies one TestOutcome for each test result the artifact records. The artifact is the authority on what a test did; the test-case export is the authority on which specification a result belongs to, so a result reaches its specification by an exact match of a test identifier, never by a name alone. A run artifact’s format is the runner’s own, a twister run, say.
A run bundle is the unit of test evidence. It holds the run artifact (the runner’s report), a revision and a dirty flag for each checkout the run used, the name of the run and the command. The bundle’s identity is its digest. The command line names each bundle by its path. The extractor computes the digest of every bundle it reads. No digest is expected beforehand. The proof records the digest of each bundle it used, and from then on the digest is the identity of the bundle. The extractor reads a run from its bundle and from no other record.
A TestOutcome’s content hash covers the record read from the artifact’s entry for the test, projected onto the specification identity, the run identifier and the result. Its result and revision are claims recorded beside the hash. The projection is not the entry’s bytes, so recomputing an outcome’s hash needs the run’s name and the specification export as well as the artifact. Because a run’s scenarios share platform and tests, and scenario names can nest, a test identifier or an outcome identity is never split to recover its parts.
The outcome extractor shall supply a TestOutcome record for every test result a run artifact records. |
The outcome extractor shall identify each TestOutcome record by its run identifier and its specification identifier, joined by a slash. |
The outcome extractor shall form each run identifier from the run’s name, the platform and the scenario the run artifact records for the result, in that order, joined by hyphens. |
The outcome extractor shall take a TestOutcome’s specification identifier from the identifier of the test-case need the result maps to. |
The outcome extractor shall map each result to the test-case need whose suite and test function, joined to the scenario the run artifact records for the result, form the result’s test identifier, and to no other need. |
If a result maps to no test-case need, or to more than one, then the outcome extractor shall report an error for that result instead of dropping it. |
Where need types are configured for the test-case export, the outcome extractor shall take a need of that export as a test-case need when, and only when, the need’s type is one of those types. |
While no need type is configured for the test-case export, the outcome extractor shall take every need of that export as a test-case need. |
Where need types are configured for the test-case export or for the implementation export, the outcome extractor shall not refuse the export because of a need whose type is not one of those types. |
If one or more results of a run artifact map to no test-case need or to more than one, then the outcome extractor shall report every such result in one error, each with its reason. |
The outcome extractor shall compute each TestOutcome’s content hash from the canonical serialization of the specification identifier, the run identifier and the result taken from the run artifact’s record of that test, and from no other field of that record. |
Results
The outcome extractor shall record the result of each TestOutcome as the member of the closed result set that the run artifact’s status for that test corresponds to. |
If the run artifact records a status to which no member of the closed result set corresponds, then the outcome extractor shall report an error for that result instead of recording it. |
The outcome extractor shall record each skipped result as a TestOutcome whose result is skipped. |
Revision
Software Requirement: A TestOutcome's revision is the implementation checkout's revision in the run bundle SEG-SREQ-186
|
The outcome extractor shall record as each TestOutcome’s revision the revision that the run bundle records for the implementation repository’s checkout, exactly as recorded. |
If the run bundle records no revision for the implementation repository’s checkout, then the outcome extractor shall refuse the run instead of supplying outcomes for it. |
Edges and anchor
The outcome extractor shall derive a Confirms edge from each TestOutcome to the TestSpecification its result maps to. |
Software Requirement: Witnesses edges run to the implementations of what the specification verifies SEG-SREQ-189
|
The outcome extractor shall derive a Witnesses edge from each TestOutcome to every Implementation whose implementation need satisfies a requirement that the outcome’s test-case need verifies. |
Where need types are configured for the implementation export and no list of need identifiers is configured for it, the outcome extractor shall take a need of that export as an implementation need when, and only when, the need’s type is one of those types. |
While neither need types nor a list of need identifiers is configured for the implementation export, the outcome extractor shall take every need of that export as an implementation need. |
Where a list of need identifiers is configured for the implementation export, the outcome extractor shall take a need of that export as an implementation need when, and only when, the need’s identifier is in the list and, where need types are configured for that export, the need’s type is one of those types. |
Software Requirement: A listed identifier that names no need is refused by the Witnesses read SEG-SREQ-361
|
If a need identifier in the list configured for the implementation export names no need of that export that the configured need types admit, then the outcome extractor shall refuse the configuration when it reads the export, naming every such identifier. |
Software Requirement: A need outside the identifier list is not refused by the outcome extractor SEG-SREQ-375
|
Where a list of need identifiers is configured for the implementation export, the outcome extractor shall not refuse the export because of a need whose identifier is not in the list. |
The outcome extractor shall anchor each TestOutcome’s content hash at the path of the run artifact within its run bundle, naming the run bundle by its digest, with the locator nodeid:<the result’s test identifier>. |
Run bundle
The outcome extractor shall read each run from the run’s bundle and from no other record of that run. |
The outcome extractor shall compute the digest of a run bundle as the SHA-256 of the list of the bundle’s files, in the order of their paths, each given by its path in the bundle and the SHA-256 of its bytes. |
If a run bundle records that the checkout of the implementation repository was dirty, then the outcome extractor shall refuse the run instead of supplying outcomes for it. |
If a run bundle records no name for the run, then the outcome extractor shall refuse the run instead of supplying outcomes for it. |
If a path named as a run bundle is not a directory that holds a run artifact the outcome extractor can read, then the outcome extractor shall refuse the run instead of supplying outcomes for it. |
Software Requirement: An export with build timestamps is refused by the outcome extractor SEG-SREQ-225
|
If a need export the outcome extractor reads carries a build timestamp, then the outcome extractor shall refuse the export instead of supplying records from it. |
Software Requirement: Every misshapen need of an outcome-extractor export is reported in one error SEG-SREQ-353
|
If one or more needs of an export the outcome extractor reads lack a text field it reads, declare an id other than their key, or carry a link field that is not a list of identifiers, then the outcome extractor shall report every such need in one error, each with its reason. |