5. The libssl / libcrypto Boundary

OpenSSL is two libraries. libcrypto implements cryptography and hosts the provider machinery; libssl implements TLS and consumes libcrypto. Where a given responsibility falls determines whether provider code can reach it, and much confusion about what providers can do dissolves once the division is stated.

5.1 The division

ConcernLibraryProvider-reachable
Algorithm implementationslibcryptoDirectly — this is what providers are
Algorithm selection by name and propertylibcryptoDirectly
Capability advertisement (groups, sigalgs)Provider, consumed by libsslDirectly
Cipher suite listlibsslSee §4.4
Handshake state machinelibsslNo
Record layer framinglibsslNo
Key schedule sequencinglibsslNo — but it consumes provider algorithms at every step
Extension encodinglibsslNo

The pattern is that libssl owns protocol and libcrypto owns computation. A provider changes what is computed, never how the protocol is sequenced. This is a sound separation and not an oversight: a module that could alter handshake sequencing would be able to weaken the protocol invisibly, and the security analysis in Chapter 17 would be considerably harder to write.

5.2 How libssl consumes an algorithm

When libssl needs an algorithm it performs an ordinary fetch against the library context associated with the SSL_CTX. The name it fetches is determined by the negotiated parameters — the suite entry names an AEAD and a handshake MAC, the group entry names a key-management algorithm — and the property query is whatever the context's configuration established.

This has an important corollary for deployment. Because the fetch is ordinary, a provider-supplied implementation is selected by the ordinary rules: activation, property query, and provider ordering. There is no TLS-specific registration for path 1. An operator who has activated a provider that offers AES-256-GCM with a matching property has already changed which code protects their records, whether or not they intended to.

Deployment hazard. The invisibility cuts both ways. Because path-1 substitution requires no protocol change and no application change, there is no handshake artefact recording that it happened. Chapter 16 recommends that a deployment which substitutes implementations record the fact out of band, since the connection itself will not show it.

5.3 Where the key schedule meets the provider

The TLS 1.3 key schedule is sequenced by libssl but computed by libcrypto. Each Extract and Expand step is an HKDF operation keyed by the suite's hash. Consequently a provider that supplies the hash is invoked many times per handshake, in a pattern fixed by the protocol, on inputs whose lengths derive from the hash's own output size.

Two properties of the hash implementation therefore matter more than they would in a general-purpose setting. It must report its output length accurately through the parameter mechanism, because the schedule sizes its buffers from that value. And it must support duplication of a partially updated context, because the transcript hash is used at several points in the handshake without being finalised — the running transcript is duplicated, the copy finalised for one derivation, and the original continued. A hash implementation lacking a working duplication operation will fail in the handshake even though it computes correct digests in isolation.

5.4 Where the record layer meets the provider

Record protection fetches the AEAD once per traffic-key epoch and uses it for every record until keys are updated. The provider's cipher implementation is therefore on the hot path of all bulk data transfer, and its per-operation overhead — not its per-fetch overhead — dominates throughput. Chapter 18 separates these two costs, which are frequently conflated when provider performance is discussed.

The record layer also requires that the cipher accept an externally supplied IV and AAD, as described in §2.4. In provider terms this means the cipher must implement the parameter-setting entry points for these values and must not attempt to manage nonces itself.

5.5 Consequences for the design

The boundary yields four rules that the design chapters observe:

  1. Everything the protocol must know is a parameter. Lengths, identifiers and capabilities cross the boundary as OSSL_PARAM entries; nothing crosses as a struct field. A design that omits a parameter creates a failure at the consuming site, not at the provider.
  2. Nothing the provider does can change protocol sequencing. An algorithm that requires a different message flow cannot be accommodated by a provider at all, regardless of how it is implemented.
  3. Selection happens in a context. The library context the SSL_CTX uses is the one whose activated providers matter; the application's default context is irrelevant if it is not the same one.
  4. Advertisement and implementation are separate obligations. Supplying a group implementation without advertising the capability yields an algorithm that works when fetched directly and never appears in a handshake — a failure mode that looks like a negotiation bug and is really a missing capability entry.

5.6 A note on what "without modifying OpenSSL" means

The phrase is used loosely in practitioner writing, and precision helps. Three distinct situations are commonly described the same way:

No modification at all
A provider is built separately and activated by configuration. The OpenSSL binaries are those the distribution shipped. Paths 1–3 achieve this.
No modification to the application
The application is unchanged, but the OpenSSL installation has been configured — possibly substantially — to load and prefer the new module. This is the usual deployment situation and is a weaker claim than the first.
No modification to the protocol
The wire format is unchanged and peers are unaffected. True of path 1; false of paths 2, 3 and 4, each of which introduces a code point that a peer must recognise.

A claim to have added an algorithm "without modifying OpenSSL" should be read against these three meanings, and this paper states which it intends wherever the question arises.