17-445 Software Engineering for AI-Enabled Systems, Christian Kaestner
Software Architecture of AI-enabled Systems
Christian Kaestner
Required reading:
- đź•® Hulten, Geoff. "Building Intelligent Systems: A Guide to Machine Learning Engineering." Apress, 2018, Chapter 13 (Where Intelligence Lives).
- đź“° Daniel Smith. "Exploring Development Patterns in Data Science." TheoryLane Blog Post. 2017.
Recommended reading: Rick Kazman, Paul Clements, and Len Bass. Software architecture in practice. Addison-Wesley Professional, 2012, Chapter 1
Learning Goals
- Understand important quality considerations when using ML components
- Follow a design process to explicitly reason about alternative designs and their quality tradeoffs
- Gather data to make informed decisions about what ML technique to use and where and how to deploy it
- Create architectural models to reason about relevant characteristics
- Critique the decision of where an AI model lives (e.g., cloud vs edge vs hybrid), considering the relevant tradeoffs
- Deliberate how and when to update models and how to collect telemetry
Software Architecture
Software Architecture
Focused on reasoning about tradeoffs and desired qualities
Software Architecture
The software architecture of a program or computing system is the structure or structures of the system, which comprise software elements, the externally visible properties of those elements, and the relationships among them. -- Kazman et al. 2012
Why Architecture? (Kazman et al. 2012)
- Represents earliest design decisions.
- Aids in communication with stakeholders
- Shows them “how” at a level they can understand, raising questions about whether it meets their needs
- Defines constraints on implementation
- Design decisions form “load-bearing walls” of application
- Dictates organizational structure
- Teams work on different components
- Inhibits or enables quality attributes
- Similar to design patterns
- Supports predicting cost, quality, and schedule
- Typically by predicting information for each component
- Aids in software evolution
- Reason about cost, design, and effect of changes
- Aids in prototyping
- Can implement architectural skeleton early
Case Study: Twitter
Twitter - Caching Architecture
Twitter's Redesign Goals
- Performance
- Improve median latency; lower outliers
- Reduce number of machines 10x
- Reliability
- Isolate failures
- Maintainability
- "We wanted cleaner boundaries with “related” logic being in one place": encapsulation and modularity at the systems level (rather than at the class, module, or package level)
- Modifiability
- Quicker release of new features: "run small and empowered engineering teams that could make local decisions and ship user-facing changes, independent of other teams"
Raffi. New Tweets per second record, and how! Twitter Blog, 2013
Twitter: Redesign Decisions
- Ruby on Rails -> JVM/Scala
- Monolith -> Microservices
- RPC framework with monitoring, connection pooling, failover strategies, loadbalancing, ... built in
- New storage solution, temporal clustering, "roughly sortable ids"
- Data driven decision making
Twitter Case Study: Key Insights
- Architectural decisions affect entire systems, not only individual modules
- Abstract, different abstractions for different scenarios
- Reason about quality attributes early
- Make architectural decisions explicit
- Question: Did the original architect make poor decisions?
Architectural Modeling and Reasoning
Analysis-Specific Abstractions
- All maps were abstractions of the same real-world construct
- All maps were created with different goals in mind
- Different relevant abstractions
- Different reasoning opportunities
- Architectural models are specific system abstractions, for reasoning about specific qualities
- No uniform notation
What can we reason about?
What can we reason about?
Ghemawat, Sanjay, Howard Gobioff, and Shun-Tak Leung. "The Google file system." ACM SIGOPS operating systems review. Vol. 37. No. 5. ACM, 2003.
Modeling Recommendations
- Use notation suitable for analysis
- Document meaning of boxes and edges in legend
- Graphical or textual both okay; whiteboard sketches often sufficient
- Formal notations available
Case Study: Augmented Reality Translation
Case Study: Augmented Reality Translation
Case Study: Augmented Reality Translation
Qualities of Interest?
Architectural Decision: Selecting AI Techniques
What AI techniques to use and why? Tradeoffs?
Architectural Decision: Where Should the Model Live?
Where Should the Model Live?
- Glasses
- Phone
- Cloud
What qualities are relevant for the decision?
Considerations
- How much data is needed as input for the model?
- How much output data is produced by the model?
- How fast/energy consuming is model execution?
- What latency is needed for the application?
- How big is the model? How often does it need to be updated?
- Cost of operating the model? (distribution + execution)
- Opportunities for telemetry?
- What happens if users are offline?
Exercise: Latency and Bandwidth Analysis of AR Translation
Identify key components of a solution and their interactions
Estimate latency and bandwidth requirements between components
Discuss tradeoffs among different deployment models
When would one use the following designs?
- Static intelligence in the product
- Client-side intelligence
- Server-centric intelligence
- Back-end cached intelligence
- Hybrid models
More Considerations
- Coupling of ML pipeline parts
- Coupling with other parts of the system
- Ability for different developers and analysts to collaborate
- Support online experiments
- Ability to monitor
Architectural Decision: Telemetry Requirements
Telemetry Design
How to evaluate mistakes in production?
The Right and Right Amount of Telemetry
Purpose:
- Monitor operation
- Monitor mistakes (e.g., accuracy)
- Improve models over time (e.g., detect new features)
Challenges:
- too much data
- no/not enough data
- hard to measure, poor proxy measures
- rare events
- cost
- privacy
Telemetry Tradeoffs
What data to collect? How much? When?
Estimate data volume and possible bottlenecks in system.
Related: Cost of Data and Feature Engineering
- How much data do we acquire for training and evaluating models?
- What data sources at what scale and latency (considering engineering cost, storage cost, processing cost, license cost, ...)
- Is it worth investing more time in feature engineering? What if additional data sources are needed?
- What is the cost for cleaning, preprocessing the data and the value of the additional accuracy?
Architectural Decision: Independent Model Service
Microservice architecture:
Model Inference and Model Learning as a RESTful Service?
Coupling and Changeability
What's the interface between the AI component and the rest of the system?
- Learning data and process
- Inference API
- Where does feature extraction happen?
- Provide raw data (images, user profile, all past purchases) to service, grant access to shared database, or provide feature vector?
- Cost of feature extraction? Who bears the cost?
- Versioned interface?
- Coupling to other models? Direct coupling to data sources (e.g., files, databases)? Expected formats for raw data (e.g., image resolution)?
- Coupling to telemetry?
Model Service API
Consider encapsulating the model as a microservice. Sketch a (REST) API.
Future-Proofing an API
- Anticipating and encapsulating change
- What parts around the model service are likely to change?
- Rigid vs flexible data formats?
- Versioning of APIs
- Version numbers vs immutable services?
- Expecting to run multiple versions in parallel? Implications for learning and evolution?
Robustness
- Redundancy for availability?
- Load balancer for scalability?
- Can mistakes be isolated?
- Local error handling?
- Telemetry to isolate errors to component?
- Logging and log analysis for what qualities?
Architectural Decision: Updating Models
- Design for change!
- Models are rarely static outside the lab
- Data drift, feedback loops, new features, new requirements
- When and how to update models?
- How to version? How to avoid mistakes?
Risk of Stale Models
What could happen if models become stale?
Risk: Discuss drift, adversarial interactions, feedback loops
Update Requirements or Goals
Estimate the required update frequency and the related cost regarding training, data transfer, etc.
Outlook: Big Data Designs
Stream + Batch Processing
Architectural Styles / Tactics / Design Patterns for AI Enabled Systems
(no standardization, yet)
Architectures and Patterns
- The Big Ass Script Architecture
- Decoupled multi-tiered architecture (data vs data analysis vs reporting; separate business logic from ML)
- Microservice architecture (multiple learning and inference services)
- Gateway Routing Architecture
- Pipelines
- Data lake, lambda architecture
- Reuse between training and serving pipelines
- Continuous deployment, ML versioning, pipeline testing
- Daniel Smith. "Exploring Development Patterns in Data Science." TheoryLane Blog Post. 2017.
- Washizaki, Hironori, Hiromu Uchida, Foutse Khomh, and Yann-Gaël Guéhéneuc. "Machine Learning Architecture and Design Patterns." Draft, 2019
Anti-Patterns
- Big Ass Script Architecture
- Dead Experimental Code Paths
- Glue code
- Multiple Language Smell
- Pipeline Jungles
- Plain-Old Datatype Smell
- Undeclared Consumers
- Washizaki, Hironori, Hiromu Uchida, Foutse Khomh, and Yann-Gaël Guéhéneuc. "Machine Learning Architecture and Design Patterns." Draft, 2019
- Sculley, David, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. "Hidden technical debt in machine learning systems." In Advances in neural information processing systems, pp. 2503-2511. 2015.
AI as a Service
Third-Party AI Components in the Cloud
AI Components as Microservices
Readymade AI Components in the Cloud
- Data Infrastructure
- Large scale data storage, databases, stream (MongoDB, Bigtable, Kafka)
- Data Processing
- Massively parallel stream and batch processing (Sparks, Hadoop, ...)
- Elastic containers, virtual machines (docker, AWS lambda, ...)
- AI Tools
- Notebooks, IDEs, Visualization
- Learning Libraries, Frameworks (tensorflow, torch, keras, ...)
- Models
- Image, face, and speech recognition, translation
- Chatbots, spell checking, text analytics
- Recommendations, knowledge bases
Build vs Buy
Hardware, software, models?
Reflection
Qualities of interest? Important design tradeoffs? Decisions?
Summary
- Software architecture is an established discipline to reason about design alternatives
- Understand relevant quality goals
- Problem-specific modeling and analysis: Gather estimates, consider design alternatives, make tradeoffs explicit
- Examples of important design decision:
- modeling technique to use
- where to deploy the model
- how and how much telemetry to collect
- whether and how to modularize the model service
- when and how to update models
- build vs buy, cloud resources
Software Architecture of AI-enabled Systems
Christian Kaestner
Required reading:
đź•® Hulten, Geoff. "Building Intelligent Systems: A Guide to Machine Learning Engineering." Apress, 2018, Chapter 13 (Where Intelligence Lives).
đź“° Daniel Smith. "Exploring Development Patterns in Data Science." TheoryLane Blog Post. 2017.
Recommended reading: Rick Kazman, Paul Clements, and Len Bass. Software architecture in practice. Addison-Wesley Professional, 2012, Chapter 1
