Research
Learning to assist with AI and robots in the real world.
Background in embodied AI, AI-based decision making, language understanding, assistive technologies, multimodal AI and human-centred AI. Applications in healthcare, manufacturing and agriculture.
Communicating with humans
Bridging communication gaps with common-sense, personalisation and training support.
- Multimodal content generation for people with language impairment (aphasia), with a user study with speech-and-language therapists.
- Prediction of child–caregiver interaction with multimodal AI.
- Virtual role simulator (ongoing): a research platform for studying and training empathy in meaningful conversations, combining a large language model, an empathy-assessment framework (EPITOME), speech and 3D avatars, in workplace, education and healthcare scenarios.
- Empathic Home (ongoing): AI models of a home or building that aim to improve the well-being of its occupants.
- Adaptive social interaction with the MiRo-E robot; human–drone interfaces for post-disaster response; outdoor robot guides for visually impaired people.
- Earlier work at NICT: generating natural-language descriptions and instructions for domestic service robots.
Publications (8)
- 2026A Multimodal Adaptive Framework for Social Interaction with the MiRo-E Robot
Sensors 26(4), 1209, 2026. DOI - 2026Applying a Transformer-based machine-learning model to classify caregiver and infant behaviours during dyadic interactions
Infant Behavior and Development 82, 102175, 2026. DOI - 2026Designing Intuitive Human–Drone Interfaces for Post-disaster Response
Companion Proceedings of the 21st ACM/IEEE International Conference on Human-Robot Interaction (HRI), 1245–1248, 2026. DOI - 2026From Terrain to Wind Noise: Contextual Factors Shaping Outdoor Robot Guide Design for Visually Impaired People
Companion Proceedings of the 21st ACM/IEEE International Conference on Human-Robot Interaction (HRI), 838–843, 2026. DOI - 2025An Ethical Risk Assessment of a Social Robot in the Workplace
34th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), 1859–1864, 2025. DOI - 2025Beyond Detection – Orchestrating Human-Robot-Robot Assistance via an Internet of Robotic Things Paradigm
International Conference on Social Robotics + AI (ICSR), Lecture Notes in Computer Science, 516–523, 2025. DOI - 2020Alleviating the Burden of Labeling: Sentence Generation by Attention Branch Encoder–Decoder Network
IEEE Robotics and Automation Letters 5(4), 5945–5952, 2020. DOI - 2020Multimodal Attention Branch Network for Perspective-Free Sentence Generation
Conference on Robot Learning (CoRL), PMLR 100, 76–85, 2020. Paper
Perceiving and understanding the world
Solving the ambiguities of the world with multimodal perception.
IROS 2026
PhyPush: one push is all you need for sensorless physical property estimation with physics-guided transformers. Estimating the physical properties of an object from a single push, without force sensing. arXiv
- Multimodal language understanding for domestic service robots (NICT, 2017–2021). Disambiguating instructions such as "Put away the milk and cereal" from language, vision and context (MMC-GAN); reducing the labelling burden with latent-space data augmentation and automatically generated instructions; explaining predictions with attention, for example the risk of collision when placing an object (PonNet); vision-and-language navigation (CrossMap Transformer).
- Robot audition and aural servo (PhD, Inria/IRISA, 2013–2016). Controlling the robot's motion directly from auditory cues such as interaural level and time differences, instead of localising the sound source first. Because the control relies on the variation of the cues rather than on their absolute values, it needs no model of the room acoustics or of the robot's head.
Publications (16)
- 2026PhyPush: One Push is All You Need for Sensorless Physical Property Estimation with Physics-Guided Transformers
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2026. arXiv - 2021Predicting and attending to damaging collisions for placing everyday objects in photo-realistic simulations
Advanced Robotics 35(12), 787–799, 2021. DOI - 2021CrossMap Transformer: A Crossmodal Masked Path Transformer Using Double Back-Translation for Vision-and-Language Navigation
IEEE Robotics and Automation Letters 6(4), 6258–6265, 2021. DOI - 2020A Multimodal Target-Source Classifier With Attention Branches to Understand Ambiguous Instructions for Fetching Daily Objects
IEEE Robotics and Automation Letters 5(2), 532–539, 2020. DOI - 2020Latent-Space Data Augmentation for Visually-Grounded Language Understanding
Advances in Artificial Intelligence: Selected Papers from the Annual Conference of the Japanese Society for Artificial Intelligence (JSAI 2019), Advances in Intelligent Systems and Computing, 179–187, 2020. DOI - 2019A multimodal target-source classifier model for object fetching from natural language instructions
Proceedings of the 33rd Annual Conference of the Japanese Society for Artificial Intelligence (JSAI), 2D3E403, 2019. - 2019Understanding Natural Language Instructions for Fetching Daily Objects Using GAN-Based Multimodal Target–Source Classification
IEEE Robotics and Automation Letters 4(4), 3884–3891, 2019. DOI - 2018A Multimodal Classifier Generative Adversarial Network for Carry and Place Tasks From Ambiguous Language Instructions
IEEE Robotics and Automation Letters 3(4), 3113–3120, 2018. Presented at IROS 2018; RoboCup Best Paper Award DOI - 2018Aural Servo: Sensor-Based Control From Robot Audition
IEEE Transactions on Robotics 34(3), 572–585, 2018. DOI - 2018Exploiting the Distance Information of the Interaural Level Difference for Binaural Robot Motion Control
IEEE Robotics and Automation Letters 3(3), 2048–2055, 2018. DOI - 2016Audio-based robot control from interchannel level difference and absolute sound energy
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 1992–1999, 2016. DOI - 2016Aural servo: towards an alternative approach to sound localization for robot motion control
PhD thesis, Université Rennes 1, 2016. - 2016Binaural auditory interaction without HRTF for humanoid robots: A sensor-based control approach
IROS 2016 Workshop on Multimodal Sensor-based Control for HRI and Soft Manipulation, 2016. - 2016First applications of sound-based control on a mobile robot equipped with two microphones
IEEE International Conference on Robotics and Automation (ICRA), 2557–2562, 2016. DOI - 2015Sound-based control with two microphones
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 5568–5573, 2015. DOI - 2011A comparison of several approaches to perform a vision-based long range navigation
10th International Workshop on Electronics, Control, Measurement and Signals (ECMS), 1–10, 2011.
Physically interacting with the world
Interacting safely with complex objects in complex environments.
- Deep reinforcement learning for mobile robots: whole-body control for mobile manipulation in complex environments, and precise, safe manoeuvring of double-Ackermann-steering robots (ManeuverNet, ICRA 2026).
- Deformable linear object manipulation. Controlling large 3D deformations of cables, hoses and plant stems with deep reinforcement learning, for example MultiAC6, a multi actor-critic framework that decomposes the action space. This line of work started in the EU H2020 project ACROBA.
- Physics-structured policies. Screw Attention, a transformer layer in which the relation between two bodies is a spatial transform, so that the policy does not depend on how reference frames are attached to the robot's links.
- Assistive caregiving. Robot-assisted bed bathing with the RoboNotts team, winner of the ICRA 2025 Physical Robotic Caregiving Challenge.
Publications (5)
- 2026DRL-Based Pose Control for Double-Ackermann Robots Under Actuation Uncertainties
ICRA 2026 Workshop on Uncertainty in Open-World Robotics, 2026. arXiv - 2026ManeuverNet: A Soft Actor-Critic Framework for Precise Maneuvering of Double-Ackermann-Steering Robots with Optimized Reward Functions
IEEE International Conference on Robotics and Automation (ICRA), 2026. arXiv - 2026PhyPush: One Push is All You Need for Sensorless Physical Property Estimation with Physics-Guided Transformers
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2026. arXiv - 2025Towards Safe Maneuvering of Double-Ackermann-Steering Robots with a Soft Actor-Critic Framework
IROS 2025 Workshop on Safety of Intelligent and Autonomous Vehicles (SIAV-FM2L), 2025. arXiv - 2024Multi Actor-Critic DDPG for Robot Action Space Decomposition: A Framework to Control Large 3D Deformation of Soft Linear Objects
IEEE Robotics and Automation Letters 9(2), 1318–1325, 2024. DOI