ML Attacker Goal

• Confidentiality attacks: Exposure of sensitive data
  • Infer a sensitive label for a data point (e.g., hospital record)
• Integrity attacks: Unauthorized modification of data
  • Induce a model to misclassify data points from one class to another
  • e.g., Spam filter: Classify a spam as a non-spam
• Availability attacks: Disruption to critical services
  • Reduce the accuracy of a model
  • Induce a model to misclassify many data points

ML Attacks

• Attacker knowledge: Does the attacker have access to the model?
  • Training data? Learning algorithm used? Parameters?
• Attacker actions:
  • Training time: Poisoning attacks
  • Inference time: Evasion attacks, model inversion attacks

Model Inversion: Confidentiality

• Given a model output (e.g., name of a person), infer the corresponding, potentially sensitive input (facial image of the person)
• One method: Gradient descent on input space
  • Assumes that the model produces a confidence score for prediction
  • Start with a random input vector & iterate towards input values with higher confidence level

Defense against Model Inversion Attacks

• Limit attacker access to confidence scores
  • e.g., reduce the precision of the scores by rounding them off
  • But also reduces the utility of legitimate use of these scores!
• Differential privacy in ML
  • Limit what attacker can learn about the model (e.g., parameters) based on an individual training sample
  • Achieved by adding noise to input or output (e.g., DP-SGD)
  • More noise => higher privacy, but also lower model accuracy!

State of ML Security

• On-going arms race (mostly among researchers)
  • Defenses proposed & quickly broken by noble attacks
• Assume ML component is likely vulnerable
  • Design your system to minimize impact of an attack
• Remember: There may be easier ways to compromise system
  • e.g., poor security misconfiguration (default password), lack of encryption, code vulnerabilities, etc.,

Security Mindset

• Assume that all components may be compromised at one point or another
• Don't assume users will behave as expected; assume all inputs to the system as potentially malicious
• Aim for risk minimization, not perfect security; reduce the chance of catastrophic failures from attacks

Secure Design Principles for ML

• Principle of least privilege
  • Who has access to training data, model internal, system input & output, etc.,?
  • Does any user/stakeholder have more access than necessary?
    • If so, limit access by using authentication mechanisms

Secure Design Principles for ML

• Principle of least privilege
  • Who has access to training data, model internal, system input & output, etc.,?
  • Does any user/stakeholder have more access than necessary?
    • If so, limit access by using authentication mechanisms
• Isolation & compartmentalization
  • Can a security attack on one ML component (e.g., misclassification) adversely affect other parts of the system?
    • If so, compartmentalize or build in mechanisms to limit impact (see risk mitigation strategies)
• Monitoring & detection:
  • Look for odd shifts in the dataset and clean the data if needed (for poisoning attacks)
  • Assume all system input as potentially malicious & sanitize (evasion attacks)

Defining Safety

• Prevention of a system failure or malfunction that results in:
  • Death or serious injury to people
  • Loss or severe damage to equipment/property
  • Harm to the environment or society
• Safety is a system concept
  • Can't talk about software being "safe"/"unsafe" on its own
  • Safety is defined in terms of its effect on the environment
• Safety != Reliability
  • Can build safe systems from unreliable components (e.g. redundancies)
  • Reliable components may be unsafe (e.g. stronger gas tank causes more severe damage in incident)

Safety of AI-Enabled Systems

Safety of AI-Enabled Systems

Safety is a broad concept

• Not just physical harms/injuries to people
• Includes harm to mental health
• Includes polluting the environment, including noise pollution
• Includes harm to society, e.g. poverty, polarization

Case Study: Self-Driving Car

How did traditional vehicles become safe?

• National Traffic & Motor Safety Act (1966): Mandatory design changes (head rests, shatter-resistant windshields, safety belts); road improvements (center lines, reflectors, guardrails)

Autonomous Vehicles: What's different?

• In traditional vehicles, humans ultimately responsible for safety
  • Some safety features (lane keeping, emergency braking) designed to help & reduce risks
  • i.e., safety = human control + safety mechanisms
• Use of AI in autonomous vehicles: Perception, control, routing, etc.,
  • Inductive training: No explicit requirements or design insights
  • Can ML achieve safe design solely through lots of data?

Demonstrating Safety

More miles tested => safer?

Challenge: Edge/Unknown Cases

• Gaps in training data; ML will unlikely be able to cover all unknown cases
• Why is this a unique problem for AI? What about humans?

Approach for Demonstrating Safety

• Safety Engineering: An engineering discipline which assures that engineered systems provide acceptable levels of safety.
• Typical safety engineering process:
  • Identify relevant hazards & safety requirements
  • Identify potential root causes for hazards
  • For each hazard, develop a mitigation strategy
  • Provide evidence that mitigations are properly implemented

What is Hazard Analysis?

• Hazard: A condition or event that may result in undesirable outcome
  • e.g., "Ego vehicle is in risk of a collision with another vehicle."
• Safety requirement: Intended to eliminate or reduce one or more hazards
  • "Ego vehicle must always maintain some minimum safe distance to the leading vehicle."
• Hazard analysis: Methods for identifying hazards & potential root causes

Recall: World vs Machine

Software is not unsafe on its own; the control signals it generates may be

Root of unsafety usually in wrong requirements & environmental assumptions

Recall: Requirement vs Specification

• REQ: Ego vehicle must always maintain some minimum safe distance to the leading vehicle.
• ENV: Engine is working as intended; sensors are providing accurate information about the leading car (current speed, distance...)
• SPEC: Depending on the sensor readings, the controller must issue an actuator command to accelerate/decelerate the vehicle as needed.

Review: Fault Tree Analysis (FTA)

• Top-down, backward search method for root cause analysis
  • Start with a given hazard (top event), derive a set of components faults (basic events)
  • Compute minimum cutsets as potential root causes
  • Q. But how do we identify relevant hazards in the first place?

Forward vs Backward Search

Failure Mode and Effects Analysis (FMEA)

• A forward search technique to identify potential hazards
• For each function, (1) enumerate possible failure modes (2) possible safety impact (effects) and (3) mitigation strategies.
• Widely used in aeronautics, automotive, healthcare, food services, semiconductor processing, and (to some extent) software

FMEA Example: Autonomous Vehicles

• Architecture of the Apollo autonomous driving platform

FMEA Example: Autonomous Vehicles

FMEA Example: Autonomous Vehicles

Hazard and Operability Study (HAZOP)

• A forward search method to identify potential hazards
• For each component, use a set of guide words to generate possible deviations from expected behavior
• Consider the impact of each generated deviation: Can it result in a system-level hazard?

HAZOP Example: Emergency Braking (EB)

• Specification: EB must apply a maximum braking command to the engine.
  • NO OR NOT: EB does not generate any braking command.
  • LESS: EB applies less than max. braking.
  • LATE: EB applies max. braking but after a delay of 2 seconds.
  • REVERSE: EB generates an acceleration command instead of braking.
  • BEFORE: EB applies max. braking before a possible crash is detected.

HAZOP Exercise: Autonomous Vehicles

• Architecture of the Apollo autonomous driving platform

HAZOP Exercise: Perception

• What is the specification of the perception component?
• Use HAZOP to answer:
  • What are possible deviations from the specification?
  • What are potential hazards resulting from these deviations?

HAZOP: Benefits & Limitations

• Easy to use; encourages systematic reasoning about component faults
• Can be combined with FTA/FMEA to generate faults (i.e., basic events in FTA)
• Potentially labor-intensive; relies on engineer's judgement
• Does not guarantee to find all hazards (but also true for other techniques)

Remarks: Hazard Analysis

• None of these methods guarantee completeness
  • You may still be missing important hazards, failure modes
• Intended as structured approaches to thinking about failures
  • But cannot replace human expertise and experience
• When available, leverage prior domain knowledge
  • Safety standards: A set of design and process guidelines for establishing safety
  • ISO 26262, ISO 21448, IEEE P700x, etc.,
  • Most do not consider AI; new standards being developed (e.g., UL 4600)

Defining Robustness:

• A prediction for $x$ is robust if the outcome is stable under minor perturbations of the input
  • $\forall x'. d(x, x')<\epsilon \Rightarrow f(x) = f(x')$
  • distance function $d$ and permissible distance $\epsilon$ depends on problem
• A model is robust if most predictions are robust

Robustness and Distance for Images

• slight rotation, stretching, or other transformations
• change many pixels minimally (below human perception)
• change only few pixels
• change most pixels mostly uniformly, e.g., brightness

Robustness in a Safety Setting

• Does the model reliably detect stop signs?
• Also in poor lighting? In fog? With a tilted camera? Sensor noise?
• With stickers taped to the sign? (adversarial attacks)

No Model is Fully Robust

• Every useful model has at least one decision boundary (ideally at the real task decision boundary)
• Predictions near that boundary are not (and should not) be robust

Evaluating Robustness

• Lots of on-going research (especially for DNNs)
• Formal verification
  • Constraint solving or abstract interpretation over computations in neuron activations
  • Conservative abstraction, may label robust inputs as not robust
  • Currently not very scalable
  • Example: An abstract domain for certifying neural networks. Gagandeep et al., POPL (2019).
• Sampling
  • Sample within distance, compare prediction to majority prediction
  • Probabilistic guarantees possible (with many queries, e.g., 100k)
  • Example: Certified adversarial robustness via randomized smoothing. Cohen, Rosenfeld, and Kolter, ICML (2019).

Improving Robustness for Safety

• Robustness checking at Inference time
  • Handle inputs with non-robust predictions differently (e.g. discard or output low confidence score)
  • Downside: Significantly raises cost of prediction; may not be suitable for time-sensitive applications (e.g., self-driving cars)
• Design mechanisms
  • Deploy redundant components for critical tasks
  • Ensemble learning: Combine models with different biases
  • Multiple, independent sensors (e.g., lidar + radar + cameras)

Improving Robustness for Safety

• Learning more robust models
  • Curate data for abnormal scenarios (e.g., fogs, snow, sensor noise)
  • Augment training data with transformed versions (but same label)
• Testing and debugging
  • Identify training data near model's decision boundary (i.e., is the model robust around all training data?)
  • Check robustness on test data

Demonstrating Safety

How do we demonstrate to a third-party that our system is safe?

Safety & Certification Standards

• Guidelines & recommendations for achieving an acceptable level of safety
• Examples: DO-178C (airborne systems), ISO 26262 (automotive), IEC 62304 (medical software), Common Criteria (security)
• Typically, prescriptive & process-oriented
  • Recommends use of certain development processes
  • Requirements specification, design, hazard analysis, testing, verification, configuration management, etc.,
• Limitations
  • Most not designed to handle ML systems (exception: UL 4600)
  • Costly to satisfy & certify, but effectiveness unclear (e.g., many FDA-certified products recalled due to safety incidents)
• Good processes are important, but not sufficient; provides only indirect evidence for system safety

Safety Cases

• An explicit argument that a system achieves a desired safety requirement, along with supporting evidence
• Structure:
  • Argument: A top-level claim decomposed into multiple sub-claims
  • Evidence: Testing, software analysis, formal verification, inspection, expert opinions, design mechanisms...

Safety Cases: Example

• Questions to think about:
  • Do sub-claims imply the parent claim?
  • Am I missing any sub-claims?
  • Is the evidence strong enough to discharge a leaf claim?

Safety Cases: Example

Uber Safety Case

Safety Cases: Breakout

Build a safety case to argue that your movie recommendation system provides at least 80% availability. Include evidence to support your argument.

Safety Cases: Benefits & Limitations

• Provides an explicit structure to the safety argument
  • Easier to navigate, inspect, and refute for third-party auditors
  • Provides traceability between system-level claims & low-level evidence
  • Can also be used for other types of system quality (security, reliabiility, etc.,)
• Challnges and pitfalls
  • Informal links between claims & evidence
    • e.g., Does the sub-claims actually imply the top-level claim?
  • Effort in constructing the case & evidence
    • How much evidence is enough?
  • System evolution
    • If system changes, must reproduce the case & evidence
• Tools for building & analyzing safety cases available
  • e.g., ASCE/GSN from Adelard
  • But ultimately, can't replace domain knowledge & critical thinking

Review: Elements of Safe Design

(See Mitigation Strategies from the Lecture on Risks)

• Assume: Components will fail at some point
• Goal: Minimize the impact of failures
• Detection
  • Monitoring
• Response
  • Graceful degradation (fail-safe)
  • Redundancy (fail over)
• Containment
  • Decoupling & isolation

Safety Assurance with ML Components

• Consider ML components as unreliable, at most probabilistic guarantees
• Testing, testing, testing (+ simulation)
  • Focus on data quality & robustness
• Adopt a system-level perspective!
• Consider safe system design with unreliable components
  • Traditional systems and safety engineering
  • Assurance cases
• Understand the problem and the hazards
  • System level, goals, hazard analysis, world vs machine
  • Specify end-to-end system behavior if feasible
• Recent research on adversarial learning and safety in reinforcement learning

Negative Side Effects

• AI is optimized for a specific objective/cost function
  • Inadvertently cause undesirable effects on the environment
  • e.g., Transport robot: Move a box to a specific destination
    • Side effects: Scratch furniture, bump into humans, etc.,
• Side effects may cause ethical/safety issues (e.g., social media example from the Ethics lecture)
• Again, requirements problem!
  • Recall: "World vs. machine"
  • Identify stakeholders in the environment & possible effects on them
• Modify the AI goal from "Perform Task X" to:
  • Perform X subject to common-sense constraints on the environment
  • Perform X but avoid side effects to the extent possible

Reward Hacking

PlayFun algorithm pauses the game of Tetris indefinitely to avoid losing

When about to lose a hockey game, the PlayFun algorithm exploits a bug to make one of the players on the opposing team disappear from the map, thus forcing a draw.

Self-driving car rewarded for speed learns to spin in circles

Example: Coast Runner

Reward Hacking

• AI can be good at finding loopholes to achieve a goal in unintended ways
• Technically correct, but does not follow designer's informal intent
• Many possible causes, incl. partially observed goals, abstract rewards, feedback loops
• In general, a very challenging problem!
  • Difficult to specify goal & reward function to avoid all possible hacks
  • Requires careful engineering and iterative reward design

Reward Hacking -- Many Examples

Other Challenges

• Safe Exploration
  • Exploratory actions "in production" may have consequences
  • e.g., trap robots, crash drones
  • -> Safety envelopes and other strategies to explore only in safe bounds (see also chaos engineering)
• Robustness to Drift
  • Drift may lead to poor performance that may not even be recognized
  • -> Check training vs production distribution (see data quality lecture), change detection, anomaly detection
• Scalable Oversight
  • Cannot provide human oversight over every action (or label all possible training data)
  • Use indirect proxies in telemetry to assess success/satisfaction
  • -> Semi-supervised learning? Distant supervision?

Beyond Traditional Safety Critical Systems

• Recall: Legal vs ethical
• Safety analysis not only for regulated domains (nuclear power plants, medical devices, planes, cars, ...)
• Many end-user applications have a safety component

Examples?

Twitter

Mental Health

IoT

Addiction

Addiction

Society: Unemployment Engineering / Deskilling

Society: Polarization

Environmental: Energy Consumption

Exercise

Look at apps on your phone. Which apps have a safety risk and use machine learning?

Consider safety broadly: including stress, mental health, discrimination, and environment pollution

Takeaway

• Many systems have safety concerns
• ... not just nuclear power plants, planes, cars, and medical devices
• Do the right thing, even without regulation
• Consider safety broadly: including stress, mental health, discrimination, and environment pollution
• Start with requirements and hazard analysis