ML Component: Predicting Real Estate Value

Given a large database of house sales and statistical/demographic data from public records, predict the sales price of a house.

$f(size, rooms, tax, neighborhood, ...) \rightarrow price$

Data Science is Iterative and Exploratory

(Source: Guo. "Data Science Workflow: Overview and Challenges." Blog@CACM, Oct 2013)

Data Science is Iterative and Exploratory

Martínez-Plumed et al. "CRISP-DM Twenty Years Later: From Data Mining Processes to Data Science Trajectories." IEEE Transactions on Knowledge and Data Engineering (2019).

Data Science is Iterative and Exploratory

(Microsoft Azure Team, "What is the Team Data Science Process?" Microsoft Documentation, Jan 2020)

Data Science is Iterative and Exploratory

Source: Patel, Kayur, James Fogarty, James A. Landay, and Beverly Harrison. "Investigating statistical machine learning as a tool for software development." In Proc. CHI, 2008.

Data Science is Iterative and Exploratory

• Science mindset: start with rough goal, no clear specification, unclear whether possible
• Heuristics and experience to guide the process
• Try and error, refine iteratively, hypothesis testing
• Go back to data collection and cleaning if needed, revise goals

Share Experience?

Different Trajectories

Martínez-Plumed et al. "CRISP-DM Twenty Years Later: From Data Mining Processes to Data Science Trajectories." IEEE Transactions on Knowledge and Data Engineering (2019).

Different Trajectories

From: Martínez-Plumed et al. "CRISP-DM Twenty Years Later: From Data Mining Processes to Data Science Trajectories." IEEE Transactions on Knowledge and Data Engineering (2019).

Computational Notebooks

Notebooks Support Iteration and Exploration

• Quick feedback, similar to REPL
• Visual feedback including figures and tables
• Incremental computation: reexecuting individual cells
• Quick and easy: copy paste, no abstraction needed
• Easy to share: document includes text, code, and results

Brief Discussion: Notebook Limitations and Drawbacks?

Innovative vs Routine Projects

• Like data science tasks, most software projects are innovative
  • Google, Amazon, Ebay, Netflix
  • Vehicles and robotics
  • Language processing, Graphics, AI
• Routine (now, not 20 years ago)
  • E-commerce websites?
  • Product recommendation? Voice recognition?
  • Routine gets automated -> innovation cycle

A Simple Process

1. Discuss the software that needs to be written
2. Write some code
3. Test the code to identify the defects
4. Debug to find causes of defects
5. Fix the defects
6. If not done, return to step 1

Software Process

“The set of activities and associated results that produce a software product”

Examples?

Example of Process Problems?

Typical Process Steps (not necessarily in this order)

• Understand customers, identify what to build, by when, budget
• Identify relevant qualities, plan/design system accordingly
• Test, deploy, maintain, evolve
• Plan, staff, workaround

Survival Mode

• Missed deadlines -> "solo development mode" to meet own deadlines
• Ignore integration work
• Stop interacting with testers, technical writers, managers, ...

Hypothesis: Process increases flexibility and efficiency + Upfront investment for later greater returns

Ad-hoc Processes

1. Discuss the software that needs to be written
2. Write some code
3. Test the code to identify the defects
4. Debug to find causes of defects
5. Fix the defects
6. If not done, return to step 1

Waterfall Model

taming the chaos, understand requirements, plan before coding, remember testing

(CC-BY-SA-2.5)

Risk First: Spiral Model

incremental prototypes, starting with most risky components

Constant iteration: Agile

working with customers, constant replanning

(CC BY-SA 4.0, Lakeworks)

Contrasting Process Models

Ad-hoc -- Waterfall -- Spiral -- Agile

Discussion: Iteration in Notebook vs Agile?

Poor Software Engineering Practices in Notebooks?

Understanding Data Scientist Workflows

• Instead of blindly recommended "SE Best Practices" understand context
• Documentation and testing not a priority in exploratory phase
• Help with transitioning into practice
  • From notebooks to pipelines
  • Support maintenance and iteration once deployed
  • Provide infrastructure and tools

Data Science Practices by Software Engineers

• Many software engineers get involved in data science without explicit training
• Copying from public examples, little reading of documentation
• Lack of data visualization/exploration/understanding, no focus on data quality
• Strong preference for code editors, non-GUI tools
• Improve model by adding more data or changing models, rarely feature engineering or debugging
• Lack of awareness about overfitting/bias problems, single focus on accuracy, no monitoring
• More system thinking about the product and its needs

Process for AI-Enabled Systems

• Integrate Software Engineering and Data Science processes
• Establish system-level requirements (e.g., user needs, safety, fairness)
• Inform data science modeling with system requirements (e.g., privacy, fairness)
• Try risky parts first (most likely include ML components; ~spiral)
• Incrementally develop prototypes, incorporate user feedback (~agile)
• Provide flexibility to iterate and improve
• Design system with characteristics of AI component (e.g., UI design, safeguards)
• Plan for testing throughout the process and in production
• Manage project understanding both software engineering and data science workflows
• No existing "best practices" or workflow models

Trajectories

• Not every project follows the same development process, e.g.
  • Small ML addition: Product first, add ML feature later
  • Research only: Explore feasibility before thinking about a product
  • Data science first: Model as central component of potential product, build system around it
• Different focus on system requirements, qualities, and upfront planning
• Manage interdisciplinary teams and different expectations

Technical Debt Metaphor

• Analogy to financial debt
  • Have a benefit now (e.g., progress quickly, release now)
  • accepting later cost (loss of productivity, e.g., higher maintenance/operating cost, rework)
  • debt accumulates and can suffocate project
• Ideally a deliberate decision (short term tactical or long term strategic)
• Ideally track debt and plan for paying it down

Examples?

Source: Martin Fowler 2009, https://martinfowler.com/bliki/TechnicalDebtQuadrant.html

Technical Debt from ML Components?

(see reading)

The Notebook

Jupyter Notebooks are a gift from God to those who work with data. They allow us to do quick experiments with Julia, Python, R, and more -- John Paul Ada

ML and Technical Debt

• Often reckless and inadvertent in inexperienced teams
• ML can seem like an easy addition, but it may cause long-term costs
• Needs to be maintained, evolved, and debugged
• Goals may change, environment may change, some changes are subtle
• Example problems
  • Systems and models are tangled and changing one has cascading effects on the other
  • Untested, brittle infrastructure; manual deployment
  • Unstable data dependencies, replication crisis
  • Data drift and feedback loops
  • Magic constants and dead experimental code paths

Controlling Technical Debt from ML Components

Controlling Technical Debt from ML Components

• Avoid AI when not needed
• Understand and document requirements, design for mistakes
• Build reliable and maintainable pipelines, infrastructure, good engineering practices
• Test infrastructure, system testing, testing and monitoring in production
• Test and monitor data quality
• Understand and model data dependencies, feedback loops, ...
• Document design intent and system architecture
• Strong interdisciplinary teams with joint responsibilities
• Document and track technical debt
• ...

Further Reading 2

• 🗎 Patel, Kayur, James Fogarty, James A. Landay, and Beverly Harrison. "Investigating statistical machine learning as a tool for software development." In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 667-676. 2008.
• 🗎 Yang, Qian, Jina Suh, Nan-Chen Chen, and Gonzalo Ramos. "Grounding interactive machine learning tool design in how non-experts actually build models." In Proceedings of the 2018 Designing Interactive Systems Conference, pp. 573-584. 2018.
• 📰 Fowler and Highsmith. The Agile Manifesto
• 🕮 Steve McConnell. Software project survival guide. Chapter 3
• 🕮 Pfleeger and Atlee. Software Engineering: Theory and Practice. Chapter 2
• 🗎 Kruchten, Philippe, Robert L. Nord, and Ipek Ozkaya. "Technical debt: From metaphor to theory and practice." IEEE Software 29, no. 6 (2012): 18-21.