This is my first blog in a series dedicated to learning about AI Red Teaming. I am focusing on getting as much hands on as I can with dedicated local models that I will be using for testing out my methodology and improving my pattern recognition skills.
[!INFO] Here are my system specifications in case anyone would like to follow along:
- The system that I am utilizing is a Lenovo LOQ laptop with 16 GB of RAM and i5-13450HX processor with RTX-3050 (6 GB VRAM) which is running Windows 11. And I have tested this setup to be sufficient for running upto 8B parameter models with ollama without spilling into system RAM under load and causing noticeable latency.
- The content material I am using for this is given in these github repositories: repository 1 repository 2
- And along with this, I also be using the OWASP AI Red Teaming Guide. Link
AI Red Teaming: Foundations and Frameworks
Transitioning into AI red teaming requires a fundamental shift in methodology. While traditional pentesting targets system access and disk persistence, AI red teaming targets model behavior, data integrity, and algorithmic decision-making.
This guide outlines the foundational concepts of machine learning and the primary attack vectors used to compromise them.
Part 1: Machine Learning Foundations
The AI Hierarchy
Artificial Intelligence terminology is strictly hierarchical:
-
Artificial Intelligence (AI): The overarching category of intelligent machines and systems.
-
Machine Learning (ML): A subset of AI where algorithms identify patterns and learn from data.
-
Deep Learning (DL): A specialized subset of ML utilizing multi-layered artificial neural networks.
Training Methodologies
Machine learning models require specific training mechanisms to process data.
| Learning Type | Mechanism | Analogy |
|---|---|---|
| Supervised Learning | Learns from labeled datasets. | Showing labeled flashcards to a student to identify objects. |
| Unsupervised Learning | Identifies hidden structures in unlabeled data. | Sorting a mixed box of blocks by shape or color without prior instructions. |
| Reinforcement Learning | Learns via trial and error utilizing a reward/penalty system. | Navigating a maze where dead ends trigger penalties and the exit triggers a reward. |
Foundational Algorithms
Understanding core mathematical models is required for effective vulnerability research.
| Algorithm | Primary Function | Concept / Application |
|---|---|---|
| Linear Regression | Predicts continuous numerical values. | Drawing a line of best fit through a scatterplot (e.g., forecasting network traffic volume). |
| Logistic Regression | Performs binary classification by mapping data to probabilities (0 to 1). | Separating spam from benign emails utilizing a defined decision boundary threshold. |
| Decision Trees | Executes logic via sequential binary (Yes/No) nodes. | Firewall rules (e.g., If packet > 500kb → check IP → Block or Allow). |
| Support Vector Machines (SVM) | Establishes an optimal hyperplane to separate distinct data classes. | The “Kernel Trick”: Mapping complex 2D data into 3D space to slide a flat boundary between intertwined data points. |
Part 2: AI Red Teaming Methodologies
Threat Modeling Frameworks
Structured frameworks are necessary for executing attacks and communicating risk.
-
MITRE ATLAS: The adversarial threat landscape for AI systems, functioning similarly to the MITRE ATT&CK framework.
-
Google SAIF: Secures AI ecosystems by mapping trust zones across the Model, Data, Application, and System infrastructure.
-
NVIDIA AI Kill Chain: Defines the operational lifecycle of an AI attack:
Reconnaissance→Poisoning→Hijacking→Persistence→Impact
Primary Attack Vectors (OWASP ML Examples)
The following vectors map traditional web vulnerabilities to their machine learning equivalents.
| ID | Attack Vector | Mechanism | Practical Example |
|---|---|---|---|
| ML01 | Input Manipulation (Evasion) | Modifying inputs at runtime to deceive the model’s classification. | Placing black tape on a Stop sign. A human recognizes the sign; an autonomous vehicle’s classifier identifies a 70 km/h speed limit sign. |
| ML02 | Data Poisoning | Injecting malicious data into the training set to establish a backdoor. | Submitting benign CVs containing a hidden trigger word (“Synergy”) to an HR screening tool. The model learns to automatically approve future CVs containing the word. |
| ML03 | Model Inversion | Reconstructing private training data directly from model outputs. | Querying a medical diagnostic model repeatedly to reverse-engineer the patient Personally Identifiable Information (PII) it was trained on. |
| ML04 | Membership Inference | Determining if specific data was utilized during model training. | Exploiting model confidence metrics to mathematically verify that an executive’s private emails were included in a public chatbot’s training dataset. |
| ML05 | Model Theft | Exfiltrating proprietary model architecture, weights, or parameters. | Programmatically querying a competitor’s paid API, logging the inputs/outputs, and utilizing the dataset to train a functional clone. |
| ML06 | AI Supply Chain Attacks | Compromising upstream AI components, dependencies, or repositories. | Uploading a pre-trained model to a repository containing a reverse shell hidden via Pickle deserialization within the model weights. Execution occurs immediately upon load. |