Inside TLS 1.3: How a Secure Handshake Works

TCP delivers packets reliably but without any confidentiality — every byte you send passes through routers and switches where anyone with a network tap can read it. TLS (Transport Layer Security) sits between TCP and the application layer, encrypting and authenticating the data stream so that an observer on the path sees only ciphertext.

The interesting engineering problem TLS solves isn’t encryption itself — AES is fast and well-understood. The problem is key exchange: how do a client and server that have never communicated establish a shared secret over a network that a third party is actively monitoring? The answer TLS 1.3 uses is ECDHE, and the reason the protocol was substantially redesigned from 1.2 is that the older answer (RSA key transport) had a fundamental property that made historical traffic retroactively vulnerable.

This article walks through each phase of a TLS 1.3 handshake, the mathematics behind each step, and what the protocol looks like at the byte level.

Why TLS 1.3 Dropped RSA Key Exchange

In TLS 1.2 and earlier, one common mode worked like this: the client generated a random session key, encrypted it with the server’s RSA public key, and sent it. The server decrypted it with its private key. Both sides now had the session key.

The problem: if someone recorded that encrypted traffic and later obtained the server’s private RSA key — through a breach, a court order, or a cryptographic break — they could decrypt everything retroactively. Every session from years prior becomes readable.

TLS 1.3 eliminated RSA key exchange entirely. It mandates ephemeral key exchange, where the session key material is generated fresh for each connection and destroyed immediately after. The server’s certificate private key is used only for authentication, never for encrypting key material. Even if the private key is compromised years later, past sessions remain protected. This property is called Perfect Forward Secrecy (PFS).

Phase 1: Key Exchange with ECDHE

The key exchange mechanism TLS 1.3 uses is Elliptic Curve Diffie-Hellman Ephemeral (ECDHE). The “Ephemeral” means both sides generate fresh key pairs for each session.

Elliptic Curve Basics

An elliptic curve over a prime field is defined by:

$y^2 \equiv x^3 + ax + b \pmod{p}$

The curve has a designated generator point $G$ with a known order $n$ (the number of distinct points reachable by repeatedly adding $G$ to itself). The security of the system relies on the Elliptic Curve Discrete Logarithm Problem (ECDLP): given a point $Q = k \cdot G$ , recovering $k$ is computationally infeasible for properly sized curves. TLS 1.3 mandates curves from a specific set — x25519 and secp256r1 are the most common.

The Exchange

The client and server each generate a random private scalar and compute the corresponding public point:

Client: picks random $c$ , computes $C = c \cdot G$ , sends $C$
Server: picks random $s$ , computes $S = s \cdot G$ , sends $S$

Both then compute the shared secret independently:

$\text{Client: } c \cdot S = c \cdot (s \cdot G) = c \cdot s \cdot G$ $\text{Server: } s \cdot C = s \cdot (c \cdot G) = s \cdot c \cdot G$

Since scalar multiplication commutes, both arrive at the same point. An observer who saw $C$ and $S$ in transit cannot compute $c \cdot s \cdot G$ without solving the ECDLP.

In the TLS 1.3 ClientHello, the client’s public key $C$ is sent in the key_share extension. The server responds with $S$ in the ServerHello’s key_share extension. The shared point’s x-coordinate becomes the input to key derivation.

Phase 2: Authentication via Certificate

ECDHE establishes a shared secret, but it doesn’t tell the client who they’re talking to. A person-in-the-middle could intercept the connection, run separate ECDHE exchanges with both parties, and relay traffic — seeing everything while both sides believe they’re communicating directly.

The defense is certificate-based authentication. The server holds a private key and a corresponding X.509 certificate signed by a Certificate Authority (CA) the client trusts. The certificate contains the server’s public key, the domain name it’s valid for, a validity window, and the CA’s signature over all of this.

TLS 1.3 Authentication Flow

After the key exchange, the server signs the entire handshake transcript with its certificate private key. TLS 1.3 requires RSA-PSS or ECDSA for this — the older PKCS#1 v1.5 signature scheme was removed from TLS 1.3 because its deterministic padding is vulnerable to certain attacks.

For ECDSA, the signature over a hash $H$ of the transcript:

$\text{Signature} = \text{ECDSA-Sign}(d_{cert},\ H(\text{transcript}))$

The client verifies this signature against the server’s certificate public key, then verifies the certificate chain up to a root CA embedded in its trust store. If either check fails, the handshake aborts.

The transcript hash binds the authentication to the specific key exchange that happened. Even if an attacker intercepted and modified the key_share values, the server couldn’t produce a valid signature over the altered transcript.

Certificate Chain and SNI

A practical detail: the certificate the server sends is typically not signed directly by a root CA. It’s signed by an intermediate CA, which is signed by a root. The client must validate the full chain. This also means the server must send the intermediate certificates alongside its leaf certificate in the Certificate handshake message.

One important privacy consideration: the Server Name Indication (SNI) extension, sent in the ClientHello before encryption, reveals which hostname the client is connecting to. It’s visible to any network observer, even though the subsequent traffic is encrypted. Encrypted Client Hello (ECH), a draft extension, addresses this by encrypting the ClientHello using the server’s public key obtained from DNS — but ECH is not yet universally deployed.

Phase 3: Key Derivation with HKDF

The shared secret from ECDHE is a point coordinate — it has algebraic structure and non-uniform distribution. It can’t be used directly as a cipher key. TLS 1.3 runs it through HKDF (HMAC-based Key Derivation Function, RFC 5869) to produce the actual symmetric keys.

HKDF operates in two steps:

Extract: Mix the shared secret with a salt to produce a uniformly distributed pseudorandom key (PRK):

$\text{PRK} = \text{HMAC-SHA256}(\text{salt},\ \text{shared\_secret})$

Expand: Derive specific keys from the PRK using labeled context strings. The label strings prevent different derived keys from having any mathematical relationship to each other. From RFC 8446, the application traffic secret labels are:

$\text{client\_secret} = \text{HKDF-Expand-Label}(\text{PRK},\ \texttt{"c ap traffic"},\ H(\text{transcript}),\ 32)$ $\text{server\_secret} = \text{HKDF-Expand-Label}(\text{PRK},\ \texttt{"s ap traffic"},\ H(\text{transcript}),\ 32)$

From each traffic secret, the actual write key and IV are derived:

$\text{write\_key} = \text{HKDF-Expand-Label}(\text{secret},\ \texttt{"key"},\ \texttt{""},\ 16)$ $\text{write\_iv} = \text{HKDF-Expand-Label}(\text{secret},\ \texttt{"iv"},\ \texttt{""},\ 12)$

The client and server use separate secrets — and therefore separate keys and IVs — for their respective write directions. A compromise of one direction’s key doesn’t affect the other.

TLS 1.3’s key schedule is a layered sequence of HKDF operations covering early secret, handshake secret, and master secret stages. The version described above covers the application traffic keys; the handshake itself uses keys derived from an intermediate stage.

Phase 4: Record Encryption with AES-GCM

Once keys are derived, all subsequent data is encrypted with AES-GCM (Galois/Counter Mode). AES-GCM is an AEAD (Authenticated Encryption with Associated Data) cipher — it provides both confidentiality and integrity in a single operation.

The integrity guarantee matters: encryption alone doesn’t prevent an attacker from flipping bits in ciphertext. Decrypting flipped ciphertext produces garbled plaintext, which might be a valid but altered message. AES-GCM prevents this by computing a 128-bit authentication tag using GHASH — polynomial multiplication over $GF(2^{128})$ — over the ciphertext and associated data. The tag is appended to each TLS record.

On receipt, the decryptor recomputes the tag. If it doesn’t match, the record is rejected and the connection is terminated. Modification of any byte — including the associated data like the record header — is detectable.

The IV for each record is constructed by XORing the static write_iv with the 64-bit sequence number (zero-padded to 12 bytes). This ensures IV uniqueness per record without transmitting the IV — both sides maintain their own sequence counters.

Wire-Level: Parsing a TLS Record in C

All TLS data — handshake messages and encrypted application data — is wrapped in TLS Records. Each record starts with a 5-byte header:

Offset	Size	Field
0	1 byte	Content type
1	2 bytes	Legacy version (always `0x0303` for TLS 1.2 compat)
3	2 bytes	Payload length

TLS 1.3 uses a compatibility trick: the record layer version field is always 0x0303 (TLS 1.2) even in TLS 1.3 sessions, to avoid breaking middleboxes that filter on this field. The actual version is negotiated through the supported_versions extension in the ClientHello and ServerHello.

Content type values: 0x14 = ChangeCipherSpec, 0x15 = Alert, 0x16 = Handshake, 0x17 = ApplicationData. After the handshake, all records use type 0x17; the real inner content type is encrypted inside the payload.

#include <stdio.h>
#include <stdint.h>
#include <string.h>
#include <arpa/inet.h>

#define TLS_CHANGE_CIPHER_SPEC  0x14
#define TLS_ALERT               0x15
#define TLS_HANDSHAKE           0x16
#define TLS_APPLICATION_DATA    0x17

#define TLS_HANDSHAKE_CLIENT_HELLO  1
#define TLS_HANDSHAKE_SERVER_HELLO  2
#define TLS_HANDSHAKE_CERTIFICATE   11
#define TLS_HANDSHAKE_FINISHED      20

/*
 * Use a packed struct to match the on-wire layout exactly.
 * Access fields via memcpy to avoid unaligned access UB on strict-alignment
 * architectures (ARM etc.). On x86 this is safe to dereference directly,
 * but memcpy is portable.
 */
struct tls_record_hdr {
    uint8_t  content_type;
    uint8_t  version_major;
    uint8_t  version_minor;
    uint8_t  length_hi;
    uint8_t  length_lo;
} __attribute__((packed));

static void parse_tls_record(const uint8_t *buf, size_t buf_len)
{
    if (buf_len < sizeof(struct tls_record_hdr)) {
        printf("[!] Buffer too short for TLS record header (%zu bytes)\n", buf_len);
        return;
    }

    struct tls_record_hdr hdr;
    memcpy(&hdr, buf, sizeof(hdr));

    /* Reconstruct length without assuming alignment */
    uint16_t payload_len = ((uint16_t)hdr.length_hi << 8) | hdr.length_lo;
    uint16_t version     = ((uint16_t)hdr.version_major << 8) | hdr.version_minor;

    printf("=== TLS Record ===\n");
    printf("Content-Type : 0x%02X", hdr.content_type);

    switch (hdr.content_type) {
    case TLS_HANDSHAKE:
        printf(" (Handshake)\n");
        if (buf_len > sizeof(hdr)) {
            uint8_t hs_type = buf[sizeof(hdr)];
            switch (hs_type) {
            case TLS_HANDSHAKE_CLIENT_HELLO: printf("Handshake    : ClientHello\n");   break;
            case TLS_HANDSHAKE_SERVER_HELLO: printf("Handshake    : ServerHello\n");   break;
            case TLS_HANDSHAKE_CERTIFICATE:  printf("Handshake    : Certificate\n");   break;
            case TLS_HANDSHAKE_FINISHED:     printf("Handshake    : Finished\n");      break;
            default:                         printf("Handshake    : type=0x%02X\n", hs_type); break;
            }
        }
        break;
    case TLS_APPLICATION_DATA:
        printf(" (ApplicationData — encrypted)\n");
        break;
    case TLS_CHANGE_CIPHER_SPEC:
        printf(" (ChangeCipherSpec — compatibility only in TLS 1.3)\n");
        break;
    case TLS_ALERT:
        printf(" (Alert)\n");
        break;
    default:
        printf(" (unknown)\n");
        break;
    }

    /* TLS 1.3 always sends 0x0303 in the record layer for middlebox compat.
     * Real version negotiation happens in supported_versions extension. */
    printf("Record ver.  : 0x%04X %s\n", version,
           version == 0x0303 ? "(TLS 1.2 compat field)" :
           version == 0x0301 ? "(TLS 1.0 compat field)" : "");
    printf("Payload len  : %u bytes\n", payload_len);
    printf("==================\n\n");
}

int main(void)
{
    /*
     * Minimal TLS 1.3 ClientHello as it appears on the wire.
     * Record layer uses 0x0303 (TLS 1.2) for middlebox compatibility.
     * The ClientHello body starts at byte 5; byte 9-10 carry 0x0303
     * as legacy_version inside the handshake message itself.
     * The real version appears in the supported_versions extension (not shown).
     */
    uint8_t client_hello[] = {
        0x16,        /* content_type: Handshake */
        0x03, 0x01,  /* record layer version: TLS 1.0 (some clients still use this) */
        0x00, 0x2A,  /* payload length: 42 bytes */
        0x01,        /* handshake type: ClientHello */
        0x00, 0x00, 0x26, /* handshake length: 38 bytes */
        0x03, 0x03,  /* legacy_version: TLS 1.2 */
        /* 32 bytes of client random would follow, then session_id,
           cipher_suites, compression_methods, extensions (including
           key_share and supported_versions) */
    };

    uint8_t app_data[] = {
        0x17,        /* content_type: ApplicationData (encrypted payload follows) */
        0x03, 0x03,  /* version: 0x0303 always for TLS 1.3 records */
        0x00, 0x10,  /* payload length: 16 bytes (ciphertext + 16-byte GCM tag) */
    };

    parse_tls_record(client_hello, sizeof(client_hello));
    parse_tls_record(app_data, sizeof(app_data));

    return 0;
}

=== TLS Record ===
Content-Type : 0x16 (Handshake)
Handshake    : ClientHello
Record ver.  : 0x0301 (TLS 1.0 compat field)
Payload len  : 42 bytes
==================

=== TLS Record ===
Content-Type : 0x17 (ApplicationData — encrypted)
Record ver.  : 0x0303 (TLS 1.2 compat field)
Payload len  : 16 bytes
==================

The version field in the record header doesn’t identify the TLS version in use — it’s a legacy compatibility artifact. TLS 1.3 negotiation happens inside the handshake through the supported_versions extension: the client lists versions it supports (including 0x0304 for TLS 1.3), the server selects one. If you’re parsing TLS 1.3 traffic and looking for version confirmation, check the ServerHello’s supported_versions extension, not the record header.

TLS 1.3 Handshake Summary

The full 1-RTT handshake in sequence:

Client                                          Server
──────                                          ──────
ClientHello
  + key_share (C = c·G)
  + supported_versions ([TLS 1.3])
  + supported_groups, cipher_suites
  ─────────────────────────────────────────────►

                                           ServerHello
                                             + key_share (S = s·G)
                                             + supported_versions (TLS 1.3)
                                           {EncryptedExtensions}
                                           {Certificate}
                                           {CertificateVerify}  ← ECDSA/RSA-PSS sig
                                           {Finished}           ← HMAC of transcript
                                      ◄────────────────────────────────────────────

{Finished}                             ← client verifies, sends its own Finished
Application Data (encrypted)
─────────────────────────────────────►

                                           Application Data (encrypted)
                                      ◄────────────────────────────────────────────

After the ServerHello, all subsequent messages are encrypted using handshake traffic keys. Application data uses separate keys derived after both Finished messages are exchanged and verified.

TLS also supports 0-RTT (session resumption), where a returning client can send application data in its first flight using a pre-shared key from a previous session. This saves a round trip at the cost of reduced forward secrecy — 0-RTT data is replayable under certain conditions, which is why servers must implement replay protection for it.

Practical Inspection

To examine a real TLS 1.3 handshake, openssl s_client decodes the record layer and prints human-readable output:

openssl s_client -connect example.com:443 -tls1_3 -msg 2>&1 | head -60

The -msg flag prints each handshake message as it’s exchanged. To capture and decode at the packet level, Wireshark with a pre-master secret log file (SSLKEYLOGFILE=./keys.log openssl s_client ...) can decrypt the session and show individual record contents, including the encrypted inner content type bytes.