Test Summary

These charts summarize the results. Reducing the wealth of information to a set of numbers necessarily loses information, so take them with a grain of salt. Nevertheless, these number provide an indication to what degree an implementation agrees with the expectations of this test suite.

The first chart shows the percentage of tests where an implementation agrees with the test suite's expectations on all individual test vectors.

The second chart shows the percentage of individual test vectors where an implementation agrees with the test suite's expectations.

How to read the test results

Tests are loosely coupled in categories. Both tests and categories have anchors and can be linked to. The anchors should be stable enough to be included in commit messages and documentation. Every test describes the setup, and may introduce terminology used in the test results. Additional resources (e.g. certificates) required by the test can be inspected by clicking on the inspect button (🔎). The results are in tabular form. The producers are on the left going down, the consumers on the top going right.

There are two kinds of tests. In producer-consumer tests, the OpenPGP implementations being tested produce an artifact (e.g. they encrypt a message), and every implementation is used to consume the artifact (e.g. to decrypt the encrypted message). In consumer tests, the artifacts are produced by the test suite, and consumed by every OpenPGP implementation. In either case, the artifact that is consumed can be inspected by clicking on the inspect button (🔎) in the second column in every row. If a producer failed to produce an artifact, or the artifact did not conform to the expectation, a cross mark (✗) is displayed. Hovering over it with the mouse pointer reveals the error message in a tooltip.

Each row now contains the result of consuming the row's artifact using the different OpenPGP implementations. Here, a check mark (✓) indicates that the operation was successful. The resulting output (e.g. the decrypted message) can be found in the tooltip. Like before, a cross mark (✗) indicates that the operation was not successful, or the produced artifact did not meet expectations. Again, details can be found in the tooltip.

Up to this point, we did not judge whether or not a operation should be successful or not, we merely recorded the facts. This answers the question of how implementations react to certain inputs, and we can quantify that and have an informed conversation about the consequences. But, we observed that the bare results were hard to interpret, a problem exacerbated by the vastness of the results due to combinatorial effects.

To address this, most tests now have an expectation for the outcome, and an explanation for the expected outcome. (If one of these expectations disagree with you, please get in touch!) If the result of an operation agrees with the expectation, the result has a green background and has a diagonal line in the top-left corner. If they disagree, the background is red and the line is in the top-right corner.