Teaching Guidance in Interactive Machine Learning systems

Context

Machine Learning (ML) has become ubiquitous in modern interactive technology because of the wide range of complex tasks it can handle without requiring explicit programming but providing data instead. However, most ML systems offer users little agency in the way models are trained from data: the training and testing are mostly performed offline by ML practitioners and remain opaque to end users, especially with deep and non-interpretable model architectures.
The growing field of Interactive Machine Learning (IML) (Fails & Olsen, 2013) field tries to understand how the development of ML can be more interactive and accessible to end users. This field focuses on building human-centered interaction to allow end-users to personalize the training and evaluate the model (Gillies et al., 2016). With novice users, we found that the way participants organize training examples both in term of order and complexity greatly affect the performances and their understanding of the system (Sanchez et al.). As a guideline for design, we suggested the use of guidance for building the training curriculum (Cakmak & Thomaz, 2014; Wall et al., 2019) i.e. a strategy to organize the training examples in a meaningful way, that can both improve the performances and increase the users understanding of the system.

As an example of guidance, we investigated a technique called Active Learning (Settles, 2009) that estimates the most informative element of unlabeled data to be given to the model. This technique relies on the uncertainty estimation of Neural Networks (Gal et al., 2017). In theory, the Active Learning framework is expected to improve the performance but is unevenly applicable depending on the interaction scenarios.

Objectives

The goal of this internship is to explore and compare guidance for teaching interactively Machine Learning algorithms. We want to investigate how guidance strategy affects user understanding and model performance in a chosen interaction scenario.

The intern can choose a relevant interaction scenario that can cover various:

• Tasks, involving image (or sound) classification or generation;
• Input modalities: image, sound, or movement data.

The scenario will be decided collegially with the supervisors. Particular care will put on the fact that teaching guidance has a critical role in the chosen scenario.

Specific Activities

During this internship, the student will be expected to:

• Study the different types of guidance from the literature (algorithmic, user-centered or other) and reflect on their properties and application scope.
• Choose a relevant Interactive Machine Learning scenario on which to apply one or several guidance.
• Design and implement a prototype application. For instance, the prototype can be developed with a framework Marcelle (https://marcelle.netlify.app/) dedicated to Interactive Machine Learning application and developed by Jules Françoise (LIMSI, Université Paris Saclay), Baptiste Caramiaux and Téo Sanchez.
• Evaluate the prototype and understand how different guidance affect both the performance and the user behavior and understanding.

The internship will last from four to six months, starting March 1st, 2021.

Expected Results

We anticipate that this work will lead to a submission to a top-tier research conference, such as ACM CHI (deadline mid-September). It could also serve as the foundation for a Ph.D. thesis.

Required Skills

We are looking for motivated students who are enthusiastic about Human-Computer Interaction (HCI) research and particularly about Interactive Machine Learning. Candidates may have skills in Machine Learning (ML) and interests in HCI or skills in HCI and interests in ML.

Programming skills are required, preferably Javascript (or/and optionally Python).