Required reading:
Building Intelligent Systems: A Guide to Machine Learning Engineering, G. Hulten (2018), Chapter 25: Adversaries and Abuse.
The Top 10 Risks of Machine Learning Security, G. McGraw et al., IEEE Computer (2020).
Learning Goals
Explain key concerns in security (in general and with regard to ML models)
Analyze a system with regard to attacker goals, attack surface, attacker capabilities
Describe common attacks against ML models, including poisoning and evasion attacks
Understand design opportunities to address security threats at the system level
Identify security requirements with threat modeling
Apply key design principles for secure system design
Security
Elements of Security
Security requirements (policies)
What does it mean for my system to be secure?
Threat model
What are the attacker's goal, capability, and incentive?
Attack surface
Which parts of the system are exposed to the attacker?
Protection mechanisms
How do we prevent the attacker from compromising a security requirement?
Security Requirements
"CIA triad" of information security
Confidentiality: Sensitive data must be accessed by authorized users only
Integrity: Sensitive data must be modifiable by authorized users only
Availability: Critical services must be available when needed by clients
Example: College Admission System
Confidentiality, integrity, or availability?
Applications to the program can only be viewed by staff and faculty
in the department.
The application site should be able to handle requests on the
day of the application deadline.
Application decisions are recorded only by the faculty and staff.
The acceptance notices can only be sent out by the program director.
Other Security Requirements
Authentication (no spoofing): Users are who they say they are
Non-repudiation: Every change can be traced to who was responsible for it
Authorization (no escalation of privilege): Only users with the right permissions can access a resource/perform an action
Threat Modeling
Why Threat Model?
What is Threat Modeling?
Threat model: A profile of an attacker
Goal: What is the attacker trying to achieve?
Capability:
Knowledge: What does the attacker know?
Actions: What can the attacker do?
Resources: How much effort can it spend?
Incentive: Why does the attacker want to do this?
Attacker Goal
What is the attacker trying to achieve?
Undermine one or more security requirements
Example: College admission
Access other applicants info without being authorized
Modify application status to “accepted”
Cause website shutdown to sabotage other applicants
Attacker Capability
What are the attacker’s actions?
Depends on system boundary & its exposed interfaces
Use an architecture diagram to identify attack surface & actions
Example: College admission
Physical: Break into building & access server
Cyber: Send malicious HTTP requests for SQL injection,
DoS attack
Social: Send phishing e-mail, bribe an insider for access
STRIDE Threat Modeling
A systematic approach to identifying threats & attacker actions
For each component, enumerate & identify potential threats
e.g., Admission Server & DoS: Applicant may flood it with requests
Tool available (Microsoft Threat Modeling Tool)
Limitations:
May end up with a long list of threats, not all of them relevant
False sense of security: STRIDE does not imply completeness!
Open Web Application Security Project
OWASP: Community-driven source of knowledge & tools for web security
Threat Modeling for ML
ML Attacker Goal
Confidentiality attacks: Exposure of sensitive data
Infer a sensitive label for a data point (e.g., hospital record)
Integrity attacks: Unauthorized modification of data
Induce a model to misclassify data points from one class to another
e.g., Spam filter: Classify a spam as a non-spam
Availability attacks: Disruption to critical services
Reduce the accuracy of a model
Induce a model to misclassify many data points
Attacker Capability
Knowledge: Does the attacker have access to the model?
Training data? Learning algorithm used? Parameters?
Attacker actions:
Training time: Poisoning attacks
Inference time: Evasion attacks, model inversion attacks
Understanding Machine Learning, Bhogavalli (2019)
Poisoning Attacks: Availability
Availability: Inject mislabeled training data to damage model
quality
3% poisoning => 11% decrease in accuracy (Steinhardt, 2017)
Attacker must have some access to the training set
e.g., models trained on public data set (e.g., ImageNet)
Example: Anti-virus (AV) scanner
Online platform for submission of potentially malicious code
Some AV company (allegedly) poisoned competitor's model
Poisoning Attacks: Integrity
Insert training data with seemingly correct labels
More targeted than availability attacks
Cause misclassification from one specific class to another
Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural
Networks, Shafahi et al. (2018)
Example: Home Assistant Robot
Dialogue system to interact with family members
Use perception & speech to identify the person
Log & upload interactions; re-train & update models for all robots
Threat modeling to identify security requirements & attacker capabilities
ML-specific attacks on training data, telemetry, or the model
Poisoning attack on training data to influence predictions
Evasion attacks to shape input data to achieve intended
predictions (adversarial learning)
Model inversion attacks for privacy violations
Security design at the system level
Principle of least privilege
Isolation & compartmentalization
AI can be used for defense (e.g. anomaly detection)
Key takeaway: Adopt a security mindset! Assume all components may be vulnerable in one way or another. Design your system to explicitly reduce the impact of potential attacks
Security
Eunsuk Kang
Required reading:
Building Intelligent Systems: A Guide to Machine Learning Engineering, G. Hulten (2018), Chapter 25: Adversaries and Abuse.
The Top 10 Risks of Machine Learning Security, G. McGraw et al., IEEE Computer (2020).