OpenPGP interoperability test suite

These are the results of running the OpenPGP interoperability test suite version 0.1.0 (1faad1f) on 2023-05-11T10:37.

This test suite has been very successful in identifying problems in many OpenPGP implementations. If you want to see your implementation included in these results, please implement the Stateless OpenPGP Command Line Interface and open an issue in our tracker. Note: The implementation doesn't have to be complete to be useful.

Test Summary

These charts summarize the results. Reducing the wealth of information to a set of numbers necessarily loses information, so take them with a grain of salt. Nevertheless, these number provide an indication to what degree an implementation agrees with the expectations of this test suite.

The first chart shows the percentage of tests where an implementation agrees with the test suite's expectations on all individual test vectors.

100% 75% 50% 25% 0%

GopenPGP 2.7.1
Sequoia 1.14+crypto-refresh
PGPainless 1.5.2-bc174
GnuPG 2.4.0
OpenPGP.js 5.5.0
RNP 0.16.2
LibTMCG 1.4.0
PGPy 0.6.0
GnuPG 2.2.40
GnuPG 1.4.23

The second chart shows the percentage of individual test vectors where an implementation agrees with the test suite's expectations.

100% 75% 50% 25% 0%

GopenPGP 2.7.1
Sequoia 1.14+crypto-refresh
PGPainless 1.5.2-bc174
GnuPG 2.4.0
OpenPGP.js 5.5.0
RNP 0.16.2
LibTMCG 1.4.0
PGPy 0.6.0
GnuPG 2.2.40
GnuPG 1.4.23

How to read the test results

Tests are loosely coupled in categories. Both tests and categories have anchors and can be linked to. The anchors should be stable enough to be included in commit messages and documentation. Every test describes the setup, and may introduce terminology used in the test results. Additional resources (e.g. certificates) required by the test can be inspected by clicking on the inspect button (). The results are in tabular form. The producers are on the left going down, the consumers on the top going right.

There are two kinds of tests. In producer-consumer tests, the OpenPGP implementations being tested produce an artifact (e.g. they encrypt a message), and every implementation is used to consume the artifact (e.g. to decrypt the encrypted message). In consumer tests, the artifacts are produced by the test suite, and consumed by every OpenPGP implementation. In either case, the artifact that is consumed can be inspected by clicking on the inspect button () in the second column in every row. If a producer failed to produce an artifact, or the artifact did not conform to the expectation, a cross mark (✗) is displayed. Hovering over it with the mouse pointer reveals the error message in a tooltip.

Each row now contains the result of consuming the row's artifact using the different OpenPGP implementations. Here, a check mark (✓) indicates that the operation was successful. The resulting output (e.g. the decrypted message) can be found in the tooltip. Like before, a cross mark (✗) indicates that the operation was not successful, or the produced artifact did not meet expectations. Again, details can be found in the tooltip.

If an operation is not supported by an implementation, the result is marked as failing with a cross mark (✗) with a light-yellow background and a diagonal line in the bottom-left corner. Details can be found in the tooltip.

If an operation is not done correctly by an implementation, i.e. the operation returned success, some data was produced, but the test suite found an issue with the produced data, or the operation failed in any other way (say communication with the backend failed), the result is marked as failing with a cross mark (✗) with a light-red background and a diagonal line in the bottom-right corner. Details can be found in the tooltip.

Up to this point, we did not judge whether or not a operation should be successful or not, we merely recorded the facts. This answers the question of how implementations react to certain inputs, and we can quantify that and have an informed conversation about the consequences. But, we observed that the bare results were hard to interpret, a problem exacerbated by the vastness of the results due to combinatorial effects.

To address this, most tests now have an expectation for the outcome, and an explanation for the expected outcome. (If one of these expectations disagree with you, please get in touch!) If the result of an operation agrees with the expectation, the result has a green background and has a diagonal line in the top-left corner. If they disagree, the background is red and the line is in the top-right corner.

Example test

This is an example.

Additional artifacts:

Certificate

	Consumer	FooPGP 1	BarPGP 2	BazPGP 3	PartialPGP 4	Expectation	Comment
Producer	Artifact
Base case		✓	✓	✓	✓	✓	Interoperability concern.
Well-formed variant		✓	✓	✗	✗	✓	Interoperability concern.
Malformed variant		✗	✓	✗	✗	✗	Message is malformed.
Weird variant		✗	✓	✓	✗
Producer failure	✗					✓	Should work (TM).

An example consumer test result

Test Results

Asymmetric Encryption

Encrypt-Decrypt roundtrip with minimal key from RFC9760

Encrypt-Decrypt roundtrip with minimal key from Appendix A.3 of RFC9760.

Additional artifacts:

Certificate
Key

	Consumer	Sequoia 1.14+crypto-refresh	LibTMCG 1.4.0	GopenPGP 2.7.1	OpenPGP.js 5.5.0	PGPainless 1.5.2-bc174	RNP 0.16.2	PGPy 0.6.0	GnuPG 2.4.0	GnuPG 2.2.40	GnuPG 1.4.23	Expectation	Comment
Producer	Artifact
Sequoia 1.14+crypto-refresh		✓ 00000000 48 65 6c 6c 6f 20 57 6f 72 6c 64 20 3a 29 Hello World :)	✗ 00000000	✓ 00000000 48 65 6c 6c 6f 20 57 6f 72 6c 64 20 3a 29 Hello World :)	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✓	Interoperability concern.
LibTMCG 1.4.0	✗											✓	Interoperability concern.
GopenPGP 2.7.1		✓ 00000000 48 65 6c 6c 6f 20 57 6f 72 6c 64 20 3a 29 Hello World :)	✗ 00000000	✓ 00000000 48 65 6c 6c 6f 20 57 6f 72 6c 64 20 3a 29 Hello World :)	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✓	Interoperability concern.
OpenPGP.js 5.5.0	✗											✓	Interoperability concern.
PGPainless 1.5.2-bc174	✗											✓	Interoperability concern.
RNP 0.16.2	✗											✓	Interoperability concern.
PGPy 0.6.0	✗											✓	Interoperability concern.
GnuPG 2.4.0	✗											✓	Interoperability concern.
GnuPG 2.2.40	✗											✓	Interoperability concern.
GnuPG 1.4.23	✗											✓	Interoperability concern.

Symmetric Encryption

Argon2 encrypted SEIPDv1 message

Tests support for Argon2 in the context of symmetrically encrypted messages. These are the test vectors from Appendix A.8 of RFC9760. The plaintext is "Hello, world!". The the password is "password".

	Consumer	Sequoia 1.14+crypto-refresh	LibTMCG 1.4.0	GopenPGP 2.7.1	OpenPGP.js 5.5.0	PGPainless 1.5.2-bc174	RNP 0.16.2	PGPy 0.6.0	GnuPG 2.4.0	GnuPG 2.2.40	GnuPG 1.4.23	Expectation	Comment
Producer	Artifact
AES-128		✓ 00000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 Hello, world!	✗ 00000000	✓ 00000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 Hello, world!	✗ 00000000	✓ 00000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 Hello, world!	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✓	Interoperability concern
AES-192		✓ 00000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 Hello, world!	✗ 00000000	✓ 00000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 Hello, world!	✗ 00000000	✓ 00000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 Hello, world!	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✓	Interoperability concern
AES-256		✓ 00000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 Hello, world!	✗ 00000000	✓ 00000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 Hello, world!	✗ 00000000	✓ 00000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 Hello, world!	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✓	Interoperability concern

SKESKv6 & SEIPDv2 encrypted message

Tests support for SKESKv6 and SEIPDv2. These are the test vectors from Appendix A.5 through A.7 of RFC9760. The plaintext is "Hello, world!". The the password is "password".

	Consumer	Sequoia 1.14+crypto-refresh	LibTMCG 1.4.0	GopenPGP 2.7.1	OpenPGP.js 5.5.0	PGPainless 1.5.2-bc174	RNP 0.16.2	PGPy 0.6.0	GnuPG 2.4.0	GnuPG 2.2.40	GnuPG 1.4.23	Expectation	Comment
Producer	Artifact
AES128-EAX		✓ 00000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 Hello, world!	✗ 00000000	✓ 00000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 Hello, world!	✗ 00000000	✓ 00000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 Hello, world!	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✓	Interoperability concern
AES128-OCB		✓ 00000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 Hello, world!	✗ 00000000	✓ 00000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 Hello, world!	✗ 00000000	✓ 00000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 Hello, world!	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✓	Interoperability concern
AES128-GCM		✓ 00000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 Hello, world!	✗ 00000000	✓ 00000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 Hello, world!	✗ 00000000	✓ 00000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 Hello, world!	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✓	Interoperability concern

Detached Signatures

Detached Sign-Verify roundtrip with minimal key from RFC9760

Detached Sign-Verify roundtrip with minimal key from Appendix A.3 of RFC9760.

Additional artifacts:

Key
Certificate

	Consumer	Sequoia 1.14+crypto-refresh	LibTMCG 1.4.0	GopenPGP 2.7.1	OpenPGP.js 5.5.0	PGPainless 1.5.2-bc174	RNP 0.16.2	PGPy 0.6.0	GnuPG 2.4.0	GnuPG 2.2.40	GnuPG 1.4.23	Expectation	Comment
Producer	Artifact
Sequoia 1.14+crypto-refresh		✓ 00000000 32 30 32 33 2d 30 35 2d 31 31 54 31 30 3a 33 36 2023-05-11T10:36 00000010 3a 35 33 5a 20 43 42 31 38 36 43 34 46 30 36 30 :53Z CB186C4F060 00000020 39 41 36 39 37 45 34 44 35 32 44 46 41 36 43 37 9A697E4D52DFA6C7 00000030 32 32 42 30 43 31 46 31 45 32 37 43 31 38 41 35 22B0C1F1E27C18A5 00000040 36 37 30 38 46 36 35 32 35 45 43 32 37 42 41 44 6708F6525EC27BAD 00000050 39 41 43 43 39 20 43 42 31 38 36 43 34 46 30 36 9ACC9 CB186C4F06 00000060 30 39 41 36 39 37 45 34 44 35 32 44 46 41 36 43 09A697E4D52DFA6C 00000070 37 32 32 42 30 43 31 46 31 45 32 37 43 31 38 41 722B0C1F1E27C18A 00000080 35 36 37 30 38 46 36 35 32 35 45 43 32 37 42 41 56708F6525EC27BA 00000090 44 39 41 43 43 39 20 6d 6f 64 65 3a 62 69 6e 61 D9ACC9 mode:bina 000000a0 72 79 0a ry.	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✓	Interoperability concern.
LibTMCG 1.4.0	✗											✓	Interoperability concern.
GopenPGP 2.7.1		✗ 00000000	✗ 00000000	✓ 00000000 32 30 32 33 2d 30 35 2d 31 31 54 31 30 3a 33 36 2023-05-11T10:36 00000010 3a 35 34 5a 20 43 42 31 38 36 43 34 46 30 36 30 :54Z CB186C4F060 00000020 39 41 36 39 37 45 34 44 35 32 44 46 41 36 43 37 9A697E4D52DFA6C7 00000030 32 32 42 30 43 31 46 31 45 32 37 43 31 38 41 35 22B0C1F1E27C18A5 00000040 36 37 30 38 46 36 35 32 35 45 43 32 37 42 41 44 6708F6525EC27BAD 00000050 39 41 43 43 39 20 43 42 31 38 36 43 34 46 30 36 9ACC9 CB186C4F06 00000060 30 39 41 36 39 37 45 34 44 35 32 44 46 41 36 43 09A697E4D52DFA6C 00000070 37 32 32 42 30 43 31 46 31 45 32 37 43 31 38 41 722B0C1F1E27C18A 00000080 35 36 37 30 38 46 36 35 32 35 45 43 32 37 42 41 56708F6525EC27BA 00000090 44 39 41 43 43 39 0a D9ACC9.	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✗ 00000000	✓	Interoperability concern.
OpenPGP.js 5.5.0	✗											✓	Interoperability concern.
PGPainless 1.5.2-bc174	✗											✓	Interoperability concern.
RNP 0.16.2	✗											✓	Interoperability concern.
PGPy 0.6.0	✗											✓	Interoperability concern.
GnuPG 2.4.0	✗											✓	Interoperability concern.
GnuPG 2.2.40	✗											✓	Interoperability concern.
GnuPG 1.4.23	✗											✓	Interoperability concern.

Versions

These are the SOP implementations used in this report:

```
sqop 0.28.0
sop-rs 0.5.0
Nettle 3.9 (Cv448: true, OCB: true)
Sequoia 1.14.0
```
Implementing SOP version ~draft-dkg-openpgp-stateless-cli-06.
Profiles for the encrypt subcommand

rfc4880

use algorithms from RFC 4880

Profiles for the generate-key subcommand

draft-koch-eddsa-for-openpgp-00

use EdDSA & ECDH over Cv25519
draft-ietf-openpgp-crypto-refresh-08

use MTI algorithms from RFC 9760
rfc4880

use algorithms from RFC 4880
```
LibTMCG 1.4.0
```

gosop 0.0.0
GopenPGP 2.7.1
ProtonMail/go-crypto a79e0d555d24
x/crypto v0.7.0
Compiled using go1.19.8

sop-openpgp.js 1.0.0
OpenPGP.js 5.5.0
Running on Node.js v18.13.0

PGPainless-SOP 1.5.2-bc174-SNAPSHOT
https://codeberg.org/PGPainless/pgpainless/src/branch/master/pgpainless-sop

Implementation of the Stateless OpenPGP Protocol Version 06
https://datatracker.ietf.org/doc/html/draft-dkg-openpgp-stateless-cli-06

Based on pgpainless-core 1.5.2-bc174-SNAPSHOT
https://pgpainless.org

Using Bouncy Castle 1.74
https://www.bouncycastle.org/java.html

Implementing SOP version draft-dkg-openpgp-stateless-cli-06.

Profiles for the encrypt subcommand

rfc4880: Follow the packet format of rfc4880

Profiles for the generate-key subcommand

draft-koch-eddsa-for-openpgp-00: Generate EdDSA / ECDH keys using Curve25519
rfc4880: Generate 4096-bit RSA keys

```
RNP-sop 0.2.0
RNP 0.16.2
```

SOPGPy 0.3.0
python-sop 0.4.0
PGPy 0.6.0

```
gpgme-sop 0.1.0
GnuPG 2.4.0
```
```
gpgme-sop 0.1.0
GnuPG 2.2.40
```
```
gpgme-sop 0.1.0
GnuPG 1.4.23
```

Configuration

This is the configuration used to produce this report:

{
  "drivers": [
    {
      "env": {},
      "id": "Sequoia 1.14+crypto-refresh",
      "path": "/home/teythoon/.local/bin/sqop-v6",
      "profile": null
    },
    {
      "env": {},
      "id": null,
      "path": "/home/teythoon/repos/pep/pgpzoo/bin/dkg-sop",
      "profile": null
    },
    {
      "env": {},
      "id": null,
      "path": "/home/teythoon/repos/pep/openpgp-interoperability-test-suite/sops/gosop/gosop",
      "profile": null
    },
    {
      "env": {},
      "id": null,
      "path": "sops/sop-openpgpjs/sop-openpgp",
      "profile": null
    },
    {
      "env": {
        "JAR": "/home/teythoon/repos/pep/openpgp-interoperability-test-suite/sops/pgpainless-cli-1.5.2-bc174-SNAPSHOT-all.jar"
      },
      "id": "PGPainless 1.5.2-bc174",
      "path": "/home/teythoon/repos/pep/pgpzoo/bin/pgpainless-cli",
      "profile": null
    },
    {
      "env": {},
      "id": null,
      "path": "/home/teythoon/repos/pep/rnp-sop/target/debug/rnp-sop",
      "profile": null
    },
    {
      "env": {},
      "id": null,
      "path": "glue/sopgpy",
      "profile": null
    },
    {
      "env": {
        "GNUPG_BIN": "/opt/gnupg-2.4.0/bin/gpg"
      },
      "id": null,
      "path": "/home/teythoon/repos/pep/gpgme-sop/target/debug/gpgme-sop",
      "profile": null
    },
    {
      "env": {
        "GNUPG_BIN": "/usr/bin/gpg"
      },
      "id": null,
      "path": "/home/teythoon/repos/pep/gpgme-sop/target/debug/gpgme-sop",
      "profile": null
    },
    {
      "env": {
        "GNUPG_BIN": "/usr/bin/gpg1"
      },
      "id": null,
      "path": "/home/teythoon/repos/pep/gpgme-sop/target/debug/gpgme-sop",
      "profile": null
    }
  ],
  "rlimits": {
    "DATA": 34359738368
  }
}