17. Security Analysis

This chapter analyses the design's security properties. It is organised around the trust relationships the provider mechanism creates, because those are what the design changes; the security of the underlying primitives is assumed and is not the subject.

17.1 The provider is in the trusted computing base

Constraint C5 stated it plainly: a provider executes with the full privileges of the process that loads it. There is no sandbox, no capability restriction and no memory isolation between the provider and the application. A provider can read any memory the process can, including private keys held by other parts of the application.

This is not a weakness of the design; it is the nature of dynamic linking, and the ENGINE mechanism it replaced had the same property. But it must be stated, because the packaging invites a different intuition: a provider looks like a plug-in, and plug-in architectures in other domains often carry isolation guarantees. This one does not.

Three consequences follow for deployment. The module must be treated as privileged code and protected accordingly (§13.5). The supply chain that delivers it is as security-critical as the one delivering OpenSSL itself. And a compromise of the module is a compromise of every process that loads it, not a degradation of the algorithms it supplies.

17.2 Threat model

The design assumes a network attacker with the standard capabilities: observing, modifying, injecting and replaying traffic, and initiating connections. It assumes the endpoints are not compromised and the module is authentic. It assumes the underlying primitives are secure against the attacker's computational resources.

Out of scope: an attacker with code execution on an endpoint (against whom the design offers nothing, per §17.1); physical attacks; and attacks on the primitives themselves.

17.3 Attack surface introduced by each component

ComponentAttacker-controlled inputPrincipal risk
AEAD decryptEvery ciphertext recordReleasing unverified plaintext; nonce misuse; timing in tag comparison
Hash updateHandshake messagesBuffer handling at block boundaries; state confusion via duplication
Key exchangePeer key shareInvalid-curve and small-subgroup attacks
Signature verifyPeer signature and certificateMalleability; accepting malformed encodings

Each row names an operation processing data from an unauthenticated party, which is the definition of attack surface. The key exchange row is the most serious because its failure mode recovers a private key rather than breaking a single connection.

17.4 The three failures that matter most

17.4.1 Missing peer-share validation

Specified as a requirement in §10.4 and repeated here because it is the design's most dangerous single omission. An implementation that performs a scalar multiplication with the private key against an attacker-chosen point that is not on the intended curve leaks information about the private key, and a sequence of such handshakes recovers it. The attack requires only the ability to connect, and its traffic is indistinguishable from ordinary failed handshakes.

17.4.2 Releasing unverified plaintext

Specified in §8.6. An AEAD that returns plaintext before or despite tag verification removes integrity protection from the connection while leaving it apparently functional. The corresponding requirement is that decryption failure is reported as failure and no output is used.

17.4.3 Silent fallback

Specified in §6.4 and §14.2. This is not a cryptographic failure but a policy failure, and it is included among the three because it is the most likely of them to occur in a real deployment: every component works correctly, the connection is secure by ordinary standards, and the deployment's actual requirement is unmet without any signal.

17.5 Side channels

Requirement N2 asks for constant-time behaviour on secret data. The obligations are the standard ones — no secret-dependent branches, no secret-indexed memory access, constant-time comparison of authentication tags and of any value derived from a key.

The honest qualification is that these cannot be guaranteed at the C level. A compiler may introduce a branch where the source has none, and a processor may introduce timing variation the source cannot control. Implementations that take this seriously verify the property on the compiled artefact, with tooling that examines the binary or measures timing distributions directly. A design that claims constant-time behaviour without such verification is claiming an intention rather than a property, and this design claims the intention.

17.6 What the design does not weaken

It is worth stating the negative result as well. Because a provider cannot alter protocol sequencing (constraint C1), the design cannot introduce a downgrade, cannot suppress authentication, and cannot change what the handshake binds. The protocol-level security properties of TLS 1.3 are preserved by construction, and the analysis reduces to the correctness of the algorithms supplied and the policy governing their selection.

That reduction is the architecture's principal security contribution, and it is worth appreciating: an extension mechanism that could alter sequencing would require a security analysis of every possible provider, rather than of the algorithms one provider supplies.

17.7 Validation status

The provider is not a FIPS-validated module and declares no property claiming otherwise (§14.3). A deployment with validation obligations cannot discharge them with this design; it would require the module to undergo the CMVP process, with the algorithm testing, the integrity self-test and the security policy that entails. The relationship between this design and a validated module is that they use the same loading mechanism and nothing else.