Download

Explainable Artificial Intelligence (XAI) - DARPA

Upload
0
Embed
Share
Upload and view presentations on any device and embed the player to your website! --- > >
Upload PPT
Upload PPT

Download this Presentation

christine

2021-8-24 published at Business&Finance category

Presentation Transcript

1.1 Explainable Artificial Intelligence (XAI) David Gunning DARPA/I2O Proposers Day 11 AUG 2016 Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
2.Distribution Statement "A" (Approved for Public Release, Distribution Unlimited) 2 A. Introduction B. Program Scope 1. Explainable Models 2. Explanation Interface 3. Psychology of Explanation 4. Emphasis and Scope of XAI Research C. Challenge Problems and Evaluation 1. Overview 2. Data Analysis 3. Autonomy 4. Evaluation D. Technical Areas 1. Explainable Learners 2. Psychological Model of Explanation E. Schedule and Milestones F. Deliverables XAI BAA Outline
3.3 Fill out a question card Send an email to: XAI@darpa.mil Questions Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
4.Watson AlphaGo 4 A. Introduction - The Need for Explainable AI User AI System Sensemaking Operations Why did you do that? Why not something else? When do you succeed? When do you fail? When can I trust you? How do I correct an error? We are entering a new age of AI applications Machine learning is the core technology Machine learning models are opaque, non-intuitive, and difficult for people to understand ©IBM Distribution Statement "A" (Approved for Public Release, Distribution Unlimited) ©Marcin Bajer/Flickr ©Eric Keenan, U.S. Marine Corps ©NASA.gov The current generation of AI systems offer tremendous benefits, but their effectiveness will be limited by the machine’s inability to explain its decisions and actions to users. Explainable AI will be essential if users are to understand, appropriately trust, and effectively manage this incoming generation of artificially intelligent partners.
5.Distribution Statement "A" (Approved for Public Release, Distribution Unlimited) 5 B. Program Scope – XAI Concept Machine Learning Process Training Data Learned Function Today Why did you do that? Why not something else? When do you succeed? When do you fail? When can I trust you? How do I correct an error? Decision or Recommendation Task User New Machine Learning Process Training Data Explainable Model XAI Explanation Interface I understand why I understand why not I know when you succeed I know when you fail I know when to trust you I know why you erred Task User
6.6 The target of XAI is an end user who: depends on decisions, recommendations, or actions of the system needs to understand the rationale for the system’s decisions to understand, appropriately trust, and effectively manage the system The XAI concept is to: provide an explanation of individual decisions enable understanding of overall strengths & weaknesses convey an understanding of how the system will behave in the future convey how to correct the system’s mistakes (perhaps) B. Program Scope – XAI Concept New Machine Learning Process Training Data Explainable Model XAI Explanation Interface I understand why I understand why not I know when you succeed I know when you fail I know when to trust you I know why you erred Task User Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
7.7 B. Program Scope – XAI Development Challenges New Machine Learning Process Training Data XAI I understand why I understand why not I know when you succeed I know when you fail I know when to trust you I know why you erred Task User Explainable Model Explanation Interface develop a range of new or modified machine learning techniques to produce more explainable models integrate state-of-the-art HCI with new principles, strategies, and techniques to generate effective explanations Explainable Models Explanation Interface summarize, extend, and apply current psychological theories of explanation to develop a computational theory Psychology of Explanation Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
8.8 B. Program Scope – XAI Development Challenges New Machine Learning Process Training Data XAI I understand why I understand why not I know when you succeed I know when you fail I know when to trust you I know why you erred Task User Explainable Model Explanation Interface develop a range of new or modified machine learning techniques to produce more explainable models integrate state-of-the-art HCI with new principles, strategies, and techniques to generate effective explanations Explainable Models Explanation Interface summarize, extend, and apply current psychological theories of explanation to develop a computational theory Psychology of Explanation TA 1: Explainable Learners TA 2: Psychological Models Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
9.9 B.1 Explainable Models Prediction Accuracy Explainability Learning Techniques (today) Neural Nets Statistical Models Ensemble Methods Decision Trees Deep Learning SVMs AOGs Bayesian Belief Nets Markov Models HBNs MLNs New Approach Create a suite of machine learning techniques that produce more explainable models, while maintaining a high level of learning performance SRL CRFs Random Forests Graphical Models Explainability (notional) Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
10.10 B.1 Explainable Models Prediction Accuracy Explainability Learning Techniques (today) Neural Nets Statistical Models Ensemble Methods Decision Trees Deep Learning SVMs AOGs Bayesian Belief Nets Markov Models HBNs MLNs Deep Explanation Modified deep learning techniques to learn explainable features New Approach Create a suite of machine learning techniques that produce more explainable models, while maintaining a high level of learning performance SRL CRFs Random Forests Graphical Models Explainability (notional) Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
11.11 B.1 Explainable Models Prediction Accuracy Explainability Learning Techniques (today) Neural Nets Statistical Models Ensemble Methods Decision Trees Deep Learning SVMs AOGs Bayesian Belief Nets Markov Models HBNs MLNs Deep Explanation Modified deep learning techniques to learn explainable features New Approach Create a suite of machine learning techniques that produce more explainable models, while maintaining a high level of learning performance SRL Interpretable Models Techniques to learn more structured, interpretable, causal models CRFs Random Forests Graphical Models Explainability (notional) Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
12.12 B.1 Explainable Models Prediction Accuracy Explainability Learning Techniques (today) Explainability (notional) Neural Nets Statistical Models Ensemble Methods Decision Trees Deep Learning SVMs AOGs Bayesian Belief Nets Markov Models HBNs MLNs Model Induction Techniques to infer an explainable model from any model as a black box Deep Explanation Modified deep learning techniques to learn explainable features New Approach Create a suite of machine learning techniques that produce more explainable models, while maintaining a high level of learning performance SRL Interpretable Models Techniques to learn more structured, interpretable, causal models CRFs Random Forests Graphical Models Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
13.13 State of the Art Human Computer Interaction (HCI) UX design Visualization Language understanding & generation New Principles and Strategies Explanation principles Explanation strategies Explanation dialogs HCI in the Broadest Sense Cognitive science Mental models Joint Development as an Integrated System In conjunction with the Explainable Models Existing Machine Learning Techniques Also consider explaining existing ML techniques B.2 Explanation Interface Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
14.14 Psychology Theories of Explanation Structure and function of explanation Role of explanation in reasoning and learning Explanation quality and utility Theory Summarization Summarize existing theories of explanation Organize and consolidate theories most useful for XAI Provide advice and consultation to XAI developers and evaluator Computational Model Develop computational model of theory Generate predictions of explanation quality and effectiveness Model Testing and Validation Test model against Phase 2 evaluation results B.3 Psychology of Explanation Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
15.15 B.4 Emphasis and Scope of XAI Research Machine Learning Human Computer Interaction End User Explanation XAI Emphasis Question Answering Dialogs Visual Analytics Interactive ML Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
16.16 DoD Funding Categories XAI Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
17.17 Explainable AI – Challenge Problem Areas Learn a model to perform the task Data Analytics Classification Learning Task Explainable Model Recommend Explanation Explainable Model Explanation Interface An analyst is looking for items of interest in massive multimedia data sets Explainable Model Explanation Interface Autonomy Reinforcement Learning Task Actions Explanation ArduPilot & SITL Simulation An operator is directing autonomous systems to accomplish a series of missions Classifies items of interest in large data set Learns decision policies for simulated missions Explains why/why not for recommended items Analyst decides which items to report, pursue Explains behavior in an after-action review Operator decides which future tasks to delegate Use the explanation to perform a task Explain decisions, actions to the user Multimedia Data Two trucks performing a loading activity Distribution Statement "A" (Approved for Public Release, Distribution Unlimited) ©Getty Images ©US Army ©Air Force Research Lab ©ArduPikot.org
18.18 Developers propose their own Phase 1 problems Within one or both of the two general categories (Data Analytics and Autonomy) During Phase 1, the XAI evaluator will work with developers Define a set of common test problems in each category Define a set of metrics and evaluation methods During Phase 2, the XAI developers will demonstrate their XAI systems against the common test problems defined by the XAI evaluator Proposers should suggest creative and compelling test problems Productive drivers of XAI research and development Sufficiently general and compelling to be useful for multiple XAI approaches Avoid unique, tailored problems for each research project Consider problems that might be extended to become an open, international competition C. Challenge Problems and Evaluation Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
19.19 Machine learning to classify items, events, or patterns of interest In heterogeneous, multimedia data Include structured/semi-structured data in addition to images and video Require meaningful explanations that are not obvious in video alone Proposers should describe: Data sets and training data (including background knowledge sources) Classification function to be learned Types of explanations to be provided User decisions to be supported Challenge problem progression Describe an appropriate progression of test problems to support your development strategy C.1 Data Analysis Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
20.20 Reinforcement learning to learn sequential decision policies For a simulated autonomous agent (e.g., UAV) Explanations may cover other needed planning, decision, or control modules, as well as decision policies learned through reinforcement learning Explain high level decisions that would be meaningful to the end user (i.e., not low level motor control) Proposers should describe: Simulation environment Types of missions to be covered Decision policies and mission tasks to be learned Types of explanations to be provided User decisions to be supported Challenge problem progression Describe an appropriate progression of test problems to support your development strategy C.2 Autonomy Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
21.21 XAI developers are presented with a problem domain Apply machine learning techniques to learn an explainable model Combine with the explanation interface to construct an explainable system The explainable system delivers and explains decisions or actions to a user who is performing domain tasks The system’s decisions and explanations contribute (positively or negatively) to the user’s performance of the domain tasks The evaluator measures the learning performance and explanation effectiveness The evaluator also conducts evaluations of existing machine learning techniques to establish baseline measures for learning performance and explanation effectiveness C.3 Evaluation – Evaluation Sequence Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
22.Distribution Statement "A" (Approved for Public Release, Distribution Unlimited) 22 C.3 Evaluation – Evaluation Framework Explanation Framework Task Decision Recommendation, Decision or Action Explanation The system takes input from the current task and makes a recommendation, decision, or action The system provides an explanation to the user that justifies its recommendation, decision, or action The user makes a decision based on the explanation Explainable Model Explanation Interface XAI System
23.Distribution Statement "A" (Approved for Public Release, Distribution Unlimited) 23 TA1: Explainable Learners Multiple TA1 teams will develop prototype explainable learning systems that include both an explainable model and an explanation interface TA2: Psychological Model of Explanation At least one TA2 team will summarize current psychological theories of explanation and develop a computational model of explanation from those theories D. Technical Areas Challenge Problem Areas Evaluation Framework Evaluator TA 2: Psychological Model of Explanation TA 1: Explainable Learners Autonomy ArduPilot & SITL Simulation Data Analytics Multimedia Data Explanation Measures User Satisfaction Mental Model Task Performance Trust Assessment Correctability Learning Performance Explanation Effectiveness Psych. Theory of Explanation Computational Model Consulting Teams that provide prototype systems with both components: Explainable Model Explanation Interface Deep Learning Teams Interpretable Model Teams Model Induction Teams
24.24 TA1: Explainable Learners Each team consists of a machine learning and a HCI PI/group Teams may represent one institution or a partnership Teams may represent any combination of university and industry researchers Multiple teams (approximately 8-12 teams) expected Team size ~ $800K-$2M per year TA2: Psychological Model of Explanation This work is primarily theoretical (including the development of a computational model of the theory) Primarily university teams are expected (but not mandated) One team expected Expected Team Characteristics Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
25.25 Challenge Problem Area Select one or both of the challenge problems areas: data analytics or autonomy Describe the proposed test problem(s) you will work on in Phase 1 Explainable Model Describe the proposed machine learning approach(s) for learning explainable models Explanation Interface Describe your approach for designing and developing the explanation interface Development Progression Describe the development sequence you intend to follow Test and Evaluation Plan Describe how you will evaluate your work in the first phase of the program Describe how you will measure learning performance and explanation effectiveness D.1 Technical Area 1 – Explainable Learners Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
26.Distribution Statement "A" (Approved for Public Release, Distribution Unlimited) 26 Theories of Explanation Describe how you will summarize the current psychological theories of explanation Describe how this work will inform the development of the TA1 XAI systems Describe how this work will inform the definition of the evaluation framework for measuring explanation effectiveness by the XAI evaluator Computational Model Describe how you will develop and implement a computational model of explanation Identify predictions that might be tested with the computational model Explain how you will test and refine the model Model Validation Describe how you will validate the computational model against the TA1 evaluation results in Phase 2 of the XAI program The government evaluator will not conduct evaluation of TA2 models D.2 Technical Area 2 – Psychological Model
27.27 E. Schedule and Milestones Technical Area 1 Milestones: Demonstrate the explainable learners against problems proposed by the developers (Phase 1) Demonstrate the explainable learners against common problems (Phase 2) Deliver software libraries and toolkits (at the end of Phase 2) Technical Area 2 Milestones: Deliver an interim report on psychological theories (after 6 months during Phase 1) Deliver a final report on psychological theories (after 12 months, during Phase 1) Deliver a computational model of explanation (after 24 months, during Phase 2) Deliver the computational model software (at the end of Phase 2) Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
28.28 E. Schedule and Milestones – Phase 1 Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
29.29 E. Schedule and Milestones – Phase 2 Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
30.30 Slide Presentations XAI Project Webpage Monthly Coordination Reports Monthly expenditure reports in TFIMS Software Software Documentation Final Technical Report F. Deliverables Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
31.31 Goal: to create a suite of new or modified machine learning techniques to produce explainable models that when combined with effective explanation techniques enable end users to understand, appropriately trust, and effectively manage the emerging generation of AI systems XAI is seeking the most interesting and compelling ideas to accomplish this goal Summary Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
32.32 Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)

Latest Release