OpenPGP interoperability test suite

These are the results of running the OpenPGP interoperability test suite version 0.1.0 (5f824fc) on 2020-11-26T10:58.

This test suite has been very successful in identifying problems in many OpenPGP implementations. If you want to see your implementation included in these results, please implement the Stateless OpenPGP Command Line Interface and open an issue in our tracker. Note: The implementation doesn't have to be complete to be useful.

How to read the test results

Tests are loosely coupled in categories. Both tests and categories have anchors and can be linked to. The anchors should be stable enough to be included in commit messages and documentation. Every test describes the setup, and may introduce terminology used in the test results. Additional resources (e.g. certificates) required by the test can be inspected by clicking on the inspect button (🔎). The results are in tabular form. The producers are on the left going down, the consumers on the top going right.

There are two kinds of tests. In producer-consumer tests, the OpenPGP implementations being tested produce an artifact (e.g. they encrypt a message), and every implementation is used to consume the artifact (e.g. to decrypt the encrypted message). In consumer tests, the artifacts are produced by the test suite, and consumed by every OpenPGP implementation. In either case, the artifact that is consumed can be inspected by clicking on the inspect button (🔎) in the second column in every row. If a producer failed to produce an artifact, or the artifact did not conform to the expectation, a cross mark (✗) is displayed. Hovering over it with the mouse pointer reveals the error message in a tooltip.

Each row now contains the result of consuming the row's artifact using the different OpenPGP implementations. Here, a check mark (✓) indicates that the operation was successful. The resulting output (e.g. the decrypted message) can be found in the tooltip. Like before, a cross mark (✗) indicates that the operation was not successful, or the produced artifact did not meet expectations. Again, details can be found in the tooltip.

Up to this point, we did not judge whether or not a operation should be successful or not, we merely recorded the facts. This answers the question of how implementations react to certain inputs, and we can quantify that and have an informed conversation about the consequences. But, we observed that the bare results were hard to interpret, a problem exacerbated by the vastness of the results due to combinatorial effects.

To address this, most tests now have an expectation for the outcome, and an explanation for the expected outcome. (If one of these expectations disagree with you, please get in touch!) If the result of an operation agrees with the expectation, the result has a green background and has a diagonal line in the top-left corner. If they disagree, the background is red and the line is in the top-right corner.

Table of Contents

1. Test Results
   1. Asymmetric Encryption
      1. Encrypt-Decrypt roundtrip with key 'Alice'
      2. Encrypt-Decrypt roundtrip with key 'Bob'
      3. Recipient IDs
   2. Symmetric Encryption
      1. Symmetric Encryption Algorithm support
      2. Encrypt-Decrypt roundtrip with key 'Bob', IDEA
      3. Encrypt-Decrypt roundtrip with key 'Bob', TripleDES
      4. Encrypt-Decrypt roundtrip with key 'Bob', CAST5
      5. Encrypt-Decrypt roundtrip with key 'Bob', Blowfish
      6. Encrypt-Decrypt roundtrip with key 'Bob', AES128
      7. Encrypt-Decrypt roundtrip with key 'Bob', AES192
      8. Encrypt-Decrypt roundtrip with key 'Bob', AES256
      9. Encrypt-Decrypt roundtrip with key 'Bob', Twofish
      10. Encrypt-Decrypt roundtrip with key 'Bob', Camellia128
      11. Encrypt-Decrypt roundtrip with key 'Bob', Camellia192
      12. Encrypt-Decrypt roundtrip with key 'Bob', Camellia256
      13. Encrypt-Decrypt roundtrip with key 'Bob', EAX
      14. Encrypt-Decrypt roundtrip with key 'Bob', OCB
      15. SEIP packet support
   3. Detached Signatures
      1. Detached Sign-Verify roundtrip with key 'Alice'
      2. Detached Sign-Verify roundtrip with key 'Bob'
      3. Detached signature with Subpackets
      4. Detached signatures: Linebreak normalization
      5. Detached signatures with unknown packets
   4. Hash Algorithms
      1. Detached Sign-Verify roundtrip with key 'Bob', MD5
      2. Detached Sign-Verify roundtrip with key 'Bob', SHA1
      3. Detached Sign-Verify roundtrip with key 'Bob', RipeMD
      4. Detached Sign-Verify roundtrip with key 'Bob', SHA256
      5. Detached Sign-Verify roundtrip with key 'Bob', SHA384
      6. Detached Sign-Verify roundtrip with key 'Bob', SHA512
      7. Detached Sign-Verify roundtrip with key 'Bob', SHA224
      8. Signature over the shattered collision
   5. Compression Algorithms
      1. Compression Algorithm support
   6. Key Generation
      1. Default key generation, encrypt-decrypt roundtrip
      2. Default key generation, encrypt-decrypt roundtrip, 2 UIDs
      3. Default key generation, encrypt-decrypt roundtrip, no UIDs
   7. Certificates
      1. Interpretation of encryption keyflags
      2. Interpretation of primary key flags
      3. Primary key binding signatures
      4. Key Flags Composition
      5. Perturbed certificates
      6. Certificate expiration
      7. Detached primary key
      8. Binding signature subpackets
   8. Revocations
      1. Key revocation test: primary key signs and is not revoked (base case)
      2. Key revocation test: subkey signs, primary key is not revoked (base case)
      3. Key revocation test: primary key signs and is revoked; revoked: no subpacket
      4. Key revocation test: subkey signs, primary key is revoked; revoked: no subpacket
      5. Key revocation test: subkey signs, subkey is revoked; revoked: no subpacket
      6. Key revocation test: primary key signs and is revoked; revoked: unspecified
      7. Key revocation test: subkey signs, primary key is revoked; revoked: unspecified
      8. Key revocation test: subkey signs, subkey is revoked; revoked: unspecified
      9. Key revocation test: primary key signs and is revoked; revoked: compromised
      10. Key revocation test: subkey signs, primary key is revoked; revoked: compromised
      11. Key revocation test: subkey signs, subkey is revoked; revoked: compromised
      12. Key revocation test: primary key signs and is revoked; revoked: private
      13. Key revocation test: subkey signs, primary key is revoked; revoked: private
      14. Key revocation test: subkey signs, subkey is revoked; revoked: private
      15. Key revocation test: primary key signs and is revoked; revoked: unknown
      16. Key revocation test: subkey signs, primary key is revoked; revoked: unknown
      17. Key revocation test: subkey signs, subkey is revoked; revoked: unknown
      18. Key revocation test: primary key signs and is revoked; revoked: superseded
      19. Key revocation test: subkey signs, primary key is revoked; revoked: superseded
      20. Key revocation test: subkey signs, subkey is revoked; revoked: superseded
      21. Key revocation test: primary key signs and is revoked; revoked: key retired
      22. Key revocation test: subkey signs, primary key is revoked; revoked: key retired
      23. Key revocation test: subkey signs, subkey is revoked; revoked: key retired
      24. Key revocation test: primary key signs and is revoked; revoked: uid retired
      25. Key revocation test: subkey signs, primary key is revoked; revoked: uid retired
      26. Key revocation test: subkey signs, subkey is revoked; revoked: uid retired
   9. Message structure
      1. Unusual Message Structure
      2. Maximum recursion depth
      3. Marker Packet
      4. Messages with unknown packets
   10. ASCII Armor
       1. Concatenated ASCII Armor Keyring
       2. Mangled ASCII Armor
   11. Elliptic Curve Cryptography
       1. EdDSA signature encodings
   12. Packet parser
       1. Packet boundaries
2. Hall of Fame
3. Configuration

Asymmetric Encryption

Encrypt-Decrypt roundtrip with key 'Alice'

Encrypt-Decrypt roundtrip using the 'Alice' key from draft-bre-openpgp-samples-00.

Additional artifacts:

• Certificate 🔎

Encrypt-Decrypt roundtrip with key 'Bob'

Encrypt-Decrypt roundtrip using the 'Bob' key from draft-bre-openpgp-samples-00.

Additional artifacts:

• Certificate 🔎

Recipient IDs

Tests variations of recipient ids.

Additional artifacts:

• TSK 🔎

Symmetric Encryption

Symmetric Encryption Algorithm support

This tests support for the different symmetric encryption algorithms using Sequoia to generate the artifacts.

Additional artifacts:

• Certificate 🔎

Encrypt-Decrypt roundtrip with key 'Bob', IDEA

Encrypt-Decrypt roundtrip using the 'Bob' key from draft-bre-openpgp-samples-00, modified with the symmetric algorithm preference [IDEA].

Additional artifacts:

• Certificate 🔎

Encrypt-Decrypt roundtrip with key 'Bob', TripleDES

Encrypt-Decrypt roundtrip using the 'Bob' key from draft-bre-openpgp-samples-00, modified with the symmetric algorithm preference [TripleDES].

Additional artifacts:

• Certificate 🔎

Encrypt-Decrypt roundtrip with key 'Bob', CAST5

Encrypt-Decrypt roundtrip using the 'Bob' key from draft-bre-openpgp-samples-00, modified with the symmetric algorithm preference [CAST5].

Additional artifacts:

• Certificate 🔎

Encrypt-Decrypt roundtrip with key 'Bob', Blowfish

Encrypt-Decrypt roundtrip using the 'Bob' key from draft-bre-openpgp-samples-00, modified with the symmetric algorithm preference [Blowfish].

Additional artifacts:

• Certificate 🔎

Encrypt-Decrypt roundtrip with key 'Bob', AES128

Encrypt-Decrypt roundtrip using the 'Bob' key from draft-bre-openpgp-samples-00, modified with the symmetric algorithm preference [AES128].

Additional artifacts:

• Certificate 🔎

Encrypt-Decrypt roundtrip with key 'Bob', AES192

Encrypt-Decrypt roundtrip using the 'Bob' key from draft-bre-openpgp-samples-00, modified with the symmetric algorithm preference [AES192].

Additional artifacts:

• Certificate 🔎

Encrypt-Decrypt roundtrip with key 'Bob', AES256

Encrypt-Decrypt roundtrip using the 'Bob' key from draft-bre-openpgp-samples-00, modified with the symmetric algorithm preference [AES256].

Additional artifacts:

• Certificate 🔎

Encrypt-Decrypt roundtrip with key 'Bob', Twofish

Encrypt-Decrypt roundtrip using the 'Bob' key from draft-bre-openpgp-samples-00, modified with the symmetric algorithm preference [Twofish].

Additional artifacts:

• Certificate 🔎

Encrypt-Decrypt roundtrip with key 'Bob', Camellia128

Encrypt-Decrypt roundtrip using the 'Bob' key from draft-bre-openpgp-samples-00, modified with the symmetric algorithm preference [Camellia128].

Additional artifacts:

• Certificate 🔎

Encrypt-Decrypt roundtrip with key 'Bob', Camellia192

Encrypt-Decrypt roundtrip using the 'Bob' key from draft-bre-openpgp-samples-00, modified with the symmetric algorithm preference [Camellia192].

Additional artifacts:

• Certificate 🔎

Encrypt-Decrypt roundtrip with key 'Bob', Camellia256

Encrypt-Decrypt roundtrip using the 'Bob' key from draft-bre-openpgp-samples-00, modified with the symmetric algorithm preference [Camellia256].

Additional artifacts:

• Certificate 🔎

Encrypt-Decrypt roundtrip with key 'Bob', EAX

Encrypt-Decrypt roundtrip using the 'Bob' key from draft-bre-openpgp-samples-00, modified with the symmetric algorithm preference [AES256], AEAD algorithm preference [EAX].

Additional artifacts:

• Certificate 🔎

Encrypt-Decrypt roundtrip with key 'Bob', OCB

Encrypt-Decrypt roundtrip using the 'Bob' key from draft-bre-openpgp-samples-00, modified with the symmetric algorithm preference [AES256], AEAD algorithm preference [OCB].

Additional artifacts:

• Certificate 🔎

SEIP packet support

This tests support for the Symmetrically Encrypted Integrity Protected Data Packet (Tag 18) and verifies that modifications to the ciphertext are detected. It uses Sequoia to generate the artifacts.

Additional artifacts:

• Certificate 🔎

Detached Signatures

Detached Sign-Verify roundtrip with key 'Alice'

Detached Sign-Verify roundtrip using the 'Alice' key from draft-bre-openpgp-samples-00.

Additional artifacts:

• Certificate 🔎

Detached Sign-Verify roundtrip with key 'Bob'

Detached Sign-Verify roundtrip using the 'Bob' key from draft-bre-openpgp-samples-00.

Additional artifacts:

• Certificate 🔎

Detached signature with Subpackets

Tests how implementations constrain the validity of signatures depending on the given subpackets.

Additional artifacts:

• Certificate 🔎

Detached signatures: Linebreak normalization

Tests how implementations normalize line breaks when verifying text signatures. Section 5.2.1 of RFC4880 says: The signature is calculated over the text data with its line endings converted to <CR><LF>.

This test creates two signatures, a binary and a text signature, over the message one\r\ntwo\r\nthree, and checks whether variants of the message with different line endings can be verified using these signatures.

Additional artifacts:

• Certificate 🔎

Detached signatures with unknown packets

This tests whether detached signatures with unknown versions of Signature packets are still verified. This is important for the evolution of the message format.

Additional artifacts:

• Certificate 🔎

Hash Algorithms

Detached Sign-Verify roundtrip with key 'Bob', MD5

Detached Sign-Verify roundtrip using the 'Bob' key from draft-bre-openpgp-samples-00, modified with the hash algorithm preference [MD5].

Additional artifacts:

• Certificate 🔎