What we do
Challenges
Recent advancements in Artificial Intelligence (AI) have improved robot autonomy and manipulation tasks but fall short in enabling sophisticated interactions with humans and adapting to new environments. While robots can now operate closer to humans, they lack the necessary cognitive capabilities to understand and execute tasks in varied domains, similar to human adaptability and problem-solving.
The slow progress in robotic autonomy and the immature AI technologies make it difficult for robots to apply learned tasks to new, even slightly altered, contexts. The challenge extends to replicating human-like cognitive processes such as task inversion and contingency reaction, requiring a significant overhaul of the current robot learning paradigm. This necessitates a novel approach that integrates multiple levels of abstraction from physical interactions to decision-making, aiming for robots to anticipate and adapt like humans.
Vision
The scientific vision of INVERSE is to endow robots with the cognitive capabilities needed to synthesise, monitor, and execute inverse plans from direct tasks defined in terms of human-understandable instructions and procedures.
INVERSE’s approach to synthesise reusable and robust plans implies an enhanced ability of the robot
- to understand its surroundings, including human intentions and needs, in order to determine what action the robot has to perform and how the environment is expected to change as a result;
- to represent robot knowledge in a flexible structure, specifically designed to facilitate execution monitoring, recovery from failures and task inversion;
- to adapt robot knowledge to different domains in order to synthesise effective solutions for direct and inverse tasks and to robustly react to the intrinsic variability of non-repetitive tasks.
The framework envisioned in INVERSE will result in substantial advances in long-term robot autonomy, enhancing the robot’s ability to solve complex manipulation tasks across different domains. Most learning algorithms fail to revert a learned task because they are not designed to adapt when training data are scarce or missing.
Solution
The INVERSE project aims to provide robots with these essential cognitive abilities by adopting a continual learning approach. After an initial bootstrap phase, used to create initial knowledge from human-level specifications, the robot refines its repertoire by capitalising on its own experience and on human feedback.
This experience-driven strategy permits to frame different problems, like performing a task in a different domain, as a problem of fault detection and recovery. Humans have a central role in INVERSE, since their supervision helps limit the complexity of the refinement loop, making the solution suitable for deployment in production scenarios. The effectiveness of developed solutions will be demonstrated in two complementary use cases designed to be a realistic instantiation of the actual work environments.
The robot is presented with high-level instructions of the direct task (e.g., CAD models, human observation, intuitive programming interfaces, etc.);
The robot learns a representation of its operating space consisting of Environment, Task and Product models. The environment model contains information on the robot workspace (e.g., number of workcells) and trained models of human activities. The product (e.g., a battery or a gear) model contains information like object classes and geometric relationships. The task model contains actions needed to perform the task. An action (or skill) consists of motor data (e.g., trajectories) and task parameters (e.g., tightening forces), and it is logically related to the state of the environment (preconditions) and expected outcomes (effects). Models are enriched with information on correct executions;
The knowledge is sufficient to synthesise and execute direct task plans. The robot can reuse the knowledge gained in a domain to plan actions and tasks in similar domains, assuming that similar actions executed on similar components produce similar effects. The robot can detect failures up to a certain extent, but it lacks the knowledge to recover from failures (and, therefore, to perform task inversion and quality control);
The robot iterates the MoCLE loop (steps 4 to 6)) to enrich its knowledge. A planner synthesises a task plan for the robot, for the human operator, and/or for both, leveraging available knowledge. In case of task inversion, the plan could be as simple as “execute the actions in the reversed order”; however, reasoning on the knowledge base will make the process smoother. From the knowledge base, it is possible to infer the expected outcome of each action (for the inverse task by backtracking the direct task);
We use an active strategy to improve perception. The robot is presented with the planned action and it seeks to interpret the scene leveraging available knowledge. Should the resulting accuracy be too low, the system tries to update its models using unsupervised techniques. Should the unsupervised update also fail (low accuracy), the system falls back to a supervised technique based on human intervention. Data from failure/successful execution is used to improve the system models for anomaly detection.
We use an active strategy to improve execution and verification. The robot tries to execute/verify the planned action (for the inverse task going from the goal to the start), matching observed effects with expected ones. Should the robot fail, a self-supervised learning strategy is triggered to update the task model. Should this also fail, for example if an unknown action is needed before performing the planned one, the robot falls back on human intervention to demonstrate the correct strategy. Data from failure/successful executions are also used to learn fault-recovery strategies.
timeline
Kick-off event INVERSE
INVERSE coordinator University of Trento (UNITN) invited the whole consortium to kick-off the INVERSE project.