Adversarial Machine Learning: Evasion, Poisoning, and Model Extraction

Classical security vulnerabilities are syntactic: a missing bounds check, an unescaped input, a logic error in a state machine. Adversarial machine learning is different. The model’s code is correct. The training procedure ran as intended. The weights represent exactly what they were trained to represent. The vulnerability is in what the model learned to represent — and the gap between that and what we assumed it learned.

When a neural network classifies an image, it’s not “seeing” the object in any human sense. It’s computing a function over pixel values learned to correlate with labels in the training distribution. That function is surprisingly brittle: small, carefully chosen perturbations to the input can cause confident misclassification while remaining imperceptible to humans. This isn’t a bug in the code. It’s a property of how high-dimensional learning works.

This article covers the three major adversarial ML attack categories, the mathematics of the two most important evasion attacks (FGSM and PGD), practical implementations in PyTorch, and the defenses that actually work.

Attack Taxonomy

Adversarial attacks target different phases of the ML pipeline:

Evasion attacks happen at inference time. The model is trained and deployed; the attacker crafts inputs that cause misclassification. The model weights are not modified. This is the most studied category and the focus of most published research.

Poisoning attacks happen before or during training. The attacker corrupts the training data, either to degrade overall performance or to implant a backdoor that triggers specific behavior on specially crafted inputs.

Model extraction attacks happen against deployed APIs. The attacker queries a black-box model to reconstruct a functionally equivalent copy, bypassing whatever IP protection the model owner intended.

Evasion: FGSM

The Fast Gradient Sign Method (Goodfellow et al., 2014) remains the foundation of evasion attack research. Understanding it requires thinking about what gradient descent does during training, then running it in reverse.

During training, backpropagation computes the gradient of the loss with respect to the model’s weights, and the optimizer adjusts the weights to reduce the loss. The input data is fixed; the weights move.

An evasion attack flips this: the weights are frozen, and the attacker asks what change to the input would maximally increase the loss. That’s the gradient of the loss with respect to the input:

$\nabla_x J(\theta, x, y)$

Adding this gradient to the original input would increase the loss — making the model more wrong — but the magnitude might be visually obvious. FGSM constrains the perturbation to be imperceptible by taking only the sign of the gradient and scaling by a small $\epsilon$:

$x_{adv} = x + \epsilon \cdot \text{sign}(\nabla_x J(\theta, x, y))$

The sign operation means every pixel is perturbed by exactly $\pm\epsilon$, regardless of the gradient’s magnitude. This is an $L_\infty$ norm constraint: no single pixel changes by more than $\epsilon$. At $\epsilon = 0.007$ on a $[0,1]$-normalized image, the perturbation is smaller than the visual noise in a JPEG artifact.

Why does it work at such small magnitude? Goodfellow’s linearity hypothesis: in high-dimensional space, many small aligned perturbations accumulate. If an image has $n$ pixels and each is shifted by $\epsilon$ in the direction that increases the loss, the total effect on the activations is proportional to $n \cdot \epsilon$. Even when $\epsilon$ is tiny, $n$ is large enough (ImageNet images are 150,000+ values) that the accumulated effect crosses decision boundaries. The model’s learned function is locally linear enough that this works reliably.

PyTorch Implementation

import torch
import torch.nn as nn
import torchvision.models as models
import torchvision.transforms as transforms
from PIL import Image

def fgsm_attack(image: torch.Tensor,
                epsilon: float,
                data_grad: torch.Tensor) -> torch.Tensor:
    """
    Apply FGSM perturbation.
    image:     [1, C, H, W] input tensor with requires_grad=True
    epsilon:   perturbation magnitude (e.g. 0.007 to 0.05)
    data_grad: gradient of loss w.r.t. image
    """
    perturbed = image + epsilon * data_grad.sign()
    # Clamp to valid pixel range
    return torch.clamp(perturbed, 0, 1)


def run_fgsm(model: nn.Module,
             image: torch.Tensor,
             true_label: torch.Tensor,
             epsilon: float = 0.05) -> torch.Tensor | None:
    """
    Full FGSM pipeline: forward pass → loss → gradient → perturb → verify.
    """
    model.eval()
    image = image.clone().requires_grad_(True)

    # Forward pass on clean image
    output = model(image)
    pred_clean = output.argmax(dim=1)

    if pred_clean.item() != true_label.item():
        print("Model misclassifies clean image — attack not meaningful.")
        return None

    # Compute loss and backpropagate through the INPUT, not the weights
    loss = nn.CrossEntropyLoss()(output, true_label)
    model.zero_grad()
    loss.backward()

    # image.grad.data is ∇_x J(θ, x, y)
    adv_image = fgsm_attack(image, epsilon, image.grad.data)

    # Verify misclassification
    with torch.no_grad():
        adv_output = model(adv_image)
        pred_adv = adv_output.argmax(dim=1)

    if pred_adv.item() != true_label.item():
        conf = torch.softmax(adv_output, dim=1).max().item()
        print(f"Attack succeeded: true={true_label.item()} → predicted={pred_adv.item()} ({conf:.1%} confidence)")
    else:
        print(f"Attack failed at ε={epsilon}. Try increasing epsilon.")

    return adv_image


# Example usage with a pretrained ResNet-50
if __name__ == "__main__":
    model = models.resnet50(weights=models.ResNet50_Weights.DEFAULT)

    transform = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
    ])

    # Load any image and provide its ImageNet class index as the label
    img = transform(Image.open("panda.jpg")).unsqueeze(0)  # [1, 3, 224, 224]
    label = torch.tensor([388])  # ImageNet class 388 = giant panda

    adv = run_fgsm(model, img, label, epsilon=0.05)

The key line is image.grad.data — PyTorch accumulates gradients with respect to any tensor that has requires_grad=True. By setting that on the input image rather than on the model parameters, loss.backward() computes $\nabla_x J$ instead of $\nabla_\theta J$.

Evasion: PGD (Projected Gradient Descent)

FGSM is a single-step attack. Madry et al. (2018) showed that multi-step iterative attacks are significantly stronger: Projected Gradient Descent (PGD).

PGD applies FGSM repeatedly with a smaller step size $\alpha$, projecting the result back into the $\epsilon$-ball after each step:

$x^{t+1} = \text{Proj}_\epsilon\left(x^t + \alpha \cdot \text{sign}(\nabla_x J(\theta, x^t, y))\right)$

The projection $\text{Proj}_\epsilon$ clips the perturbation so that $\|x^{t+1} - x_{orig}\|_\infty \leq \epsilon$. Starting from a random point within the $\epsilon$-ball (random initialization) and running many steps gives a much stronger attack than the single FGSM step.

Madry et al. argued that PGD represents approximately the strongest first-order attack under the $L_\infty$ constraint, making it the standard benchmark for evaluating adversarial defenses. A defense that doesn’t hold up to PGD is generally not considered robust.

Norm choices matter for the threat model:

Physical-world attacks (adversarial patches on stop signs, adversarial glasses for face recognition) are closer to $L_0$ — a small region of the image is heavily modified. Eykholt et al. (2018) demonstrated reliable stop sign misclassification in real driving conditions using printed stickers, with additional constraints for lighting variation and camera angle changes.

Transferability: Black-Box Attacks

A significant practical concern: adversarial examples generated against one model often transfer to different models trained on the same task — even models with different architectures.

If an attacker trains a local “substitute” model on a similar dataset, generates adversarial examples against it, and submits those examples to a black-box target model (querying only output probabilities, not gradients), the attack often succeeds. Transfer rates vary widely by model family and attack method, but the phenomenon is robust enough that it’s a first-order concern in deploying models to adversarial environments.

The transferability mechanism isn’t fully understood, but it’s related to shared decision boundaries for natural data distributions — models trained on the same task learn similar representations near the data manifold.

Poisoning and Backdoor Attacks

In a backdoor attack, the attacker corrupts a subset of training data by adding a specific trigger pattern to inputs and mislabeling them. The model learns to associate the trigger with the target class. On clean inputs, the model behaves normally. On inputs containing the trigger, it reliably misclassifies to the attacker’s target class.

The threat is realistic in scenarios where training data is sourced from the internet (image datasets scraped from public sources), where pre-trained models are fine-tuned on third-party datasets, or where ML-as-a-service providers train models on customer-provided data.

Detection methods include:

• Neural Cleanse (Wang et al., 2019): Identifies potential triggers by finding minimal perturbations that cause all inputs to be classified as each possible target class
• STRIP (Gao et al., 2019): Detects backdoor inputs at inference time by observing that superimposing random patterns on backdoor inputs doesn’t change the prediction (the trigger dominates), while clean inputs are affected
• Fine-pruning: Prunes neurons that are dormant on clean data but active on backdoor inputs

Model Extraction

A deployed model exposed through an API can be reconstructed by querying it with many inputs and training a surrogate on the input-output pairs. Neural networks are universal function approximators — given enough query-response pairs, a surrogate model can match the decision boundaries of the target.

The economics: if the query API charges per request and the value of the reconstructed model is high, even millions of queries may be worth it. Tramèr et al. (2016) demonstrated extraction of models with near-original accuracy using far fewer queries than one might expect.

Defenses:

• Prediction throttling and rate limiting: Limits the number of queries an entity can make
• Output perturbation: Rounding confidence scores, returning only top-1 labels instead of full probability distributions, adding calibrated noise — all reduce the information available per query
• Differential privacy in training: Provides theoretical bounds on how much information about any training point can be extracted, which partially bounds model extraction as well
• Watermarking: Embedding a statistical signature in model behavior that can be detected in a suspected copy, providing forensic evidence of extraction

Defenses That Actually Work

Adversarial Training

The most empirically reliable defense: include adversarial examples in the training data. Madry et al. formulate this as a min-max problem:

$\min_\theta \mathbb{E}_{(x,y)} \left[\max_{\delta \in \Delta} J(\theta, x+\delta, y)\right]$

The inner maximization finds the worst-case perturbation; the outer minimization trains the model to be correct even under that perturbation. In practice, PGD-generated adversarial examples are added to each training batch.

Adversarial training consistently provides the best empirical robustness against $L_\infty$ attacks but comes with costs: longer training time, and a reduction in accuracy on clean inputs (a robust accuracy vs. clean accuracy tradeoff that hasn’t been fully closed).

Certified Robustness: Randomized Smoothing

Adversarial training is empirically strong but provides no guarantees — a stronger attack might break it. Randomized smoothing (Cohen et al., 2019) is the primary technique for provable robustness.

The idea: build a “smoothed classifier” $g$ from a base classifier $f$ by classifying inputs based on the majority vote of $f$ evaluated on many Gaussian noise corruptions of the input:

$g(x) = \arg\max_c\ \Pr_{\delta \sim \mathcal{N}(0, \sigma^2 I)}[f(x + \delta) = c]$

If class $c_A$ has sufficiently higher probability than the next class under noise, Cohen et al. proved that $g(x) = c_A$ holds for all perturbations within an $L_2$ radius $R$, where $R$ is a function of the probability margin. This is a mathematical guarantee, not an empirical one.

The tradeoff: accuracy on clean inputs drops because the classifier is inherently averaging over noisy inputs, and the $L_2$ certificate doesn’t directly translate to $L_\infty$ guarantees.

Input Preprocessing

Techniques like feature squeezing (reducing color depth, applying smoothing filters), JPEG compression, and pixel discretization can reduce the effectiveness of adversarial perturbations by removing high-frequency noise. These are generally weaker defenses — adaptive attacks can be constructed that survive preprocessing — but they raise the cost of attacks and are useful as one layer of a defense-in-depth approach.

The Fundamental Constraint

Why do adversarial examples exist at all? The current best explanation is the combination of high dimensionality and the fact that models learn the training distribution, not the true underlying concept.

In high-dimensional space, the “natural data manifold” — where real images live — is a tiny fraction of the full pixel space. The model’s decision boundaries are defined by the training data’s distribution, not by any formal understanding of the concept. Points slightly off the manifold can land in regions the model has never seen, where its learned boundaries become arbitrary.

This isn’t patchable by fixing a bug. It’s a consequence of learning from finite data in high-dimensional spaces. The research directions that address it most directly — certified robustness with formal guarantees, training on distributions that better cover the input space, formal verification of network properties — are active areas but don’t have complete solutions at the scale of production models.

The practical implication for deploying models in adversarial environments: treat model outputs as probabilistic estimates with known failure modes, apply defense-in-depth (adversarial training + input preprocessing + output monitoring), and design systems so that model misclassification has bounded consequences rather than assuming the model will be correct on all inputs.