Simon Stepputtis

Postdoctoral Fellow | Carnegie Mellon University | Robotics Institute

Paper: A System for Imitation Learning of Contact-Rich Bimanual Manipulation Policies

Paper: A System for Imitation Learning of Contact-Rich Bimanual Manipulation Policies

[July 2022]: Our paper in collaboration with Intrinsic got accepted to IROS 2022, which we will be presenting in Kyoto, Japan.

Workshop: RSS Pioneers

Workshop: RSS Pioneers

[June 2022]: I was accepted to the RSS Pioneers Workshop 2022 with my work on Language-Conditioned Human-Agent Teaming.

IROS 2022 "TOM4HAT" Workshop Accepted

IROS 2022 "TOM4HAT" Workshop Accepted

[May 2022]: I am organizing the Workshop on Human Theory of Machines and Machine Theory of Mind for Human-Agent Teams at IROS 2022 in Kyoto, Japan!

Postdoctoral Fellow at Carnegie Mellon University

Postdoctoral Fellow at Carnegie Mellon University

[January 2022]: I started as a postdoctoral fellow at Carnegie Mellon University (CMU) with Prof. Katia Sycara.

Graduation: Ph.D. in Computer Science

Graduation: Ph.D. in Computer Science

[December 2021]: I completed my Ph.D. in Computer Science at Arizona State University with Prof. Heni Ben Amor!

Imperial College London: Invited Talk

Imperial College London: Invited Talk

[July 2021]: I will be giving brief summary and outlook of my work presented in our NeurIPS 2020 paper at the Imperial College London!

Resident @ X, The Moonshot Factory

Resident @ X, The Moonshot Factory

[Summer 2021]: Over the summer, I will be a resident at X, The Moonshot Factory, where I will be working on industrial manipulation tasks for Intrinsic, a robotics software and AI project at X .

Video: Language Conditioned Imitation Learning

Video: Language Conditioned Imitation Learning

[January 2021]: We contributed a video to the robot expo at IJCAI 2021 that is a direct extension to our NeurIPS 2020 paper. You can check out the video here!

Paper: Language-Conditioned Imitation Learning for Robot Manipulation Tasks

Paper: Language-Conditioned Imitation Learning for Robot Manipulation Tasks

[December 2020]: We published a new paper at NeurIPS 2020! Our paper got accepted as a spotlight presentation (top ~4% of accepted papers). Please check out my blog post about our work!

Intel AI Labs: Invited Talk

Intel AI Labs: Invited Talk

[November 2020]: I am excited to give a talk, Language for Robotics at Intel AI Labs summarizing our efforts on learning robot policies from natural language instructions.

Teaching <em>Introduction to Theoretical Computer Science</em> at ASU

Teaching Introduction to Theoretical Computer Science at ASU

[Summer 2020]: I will be teaching CSE 355: Introduction to Theoretical Computer Science at Arizona State University as the main instructor during the upcoming Summer 2020 semester!

Intel AI: Talk at the Deep Learning Community

Intel AI: Talk at the Deep Learning Community

[March 2020]: I will be giving a talk at the Deep Learning Community of Practice titled Imitation Learning for Adaptive Robot Control Policies from Language, Vision, and Motion

Workshop Paper: Imitation Learning of Robot Policies by Combining Language, Vision and Demonstration

Workshop Paper: Imitation Learning of Robot Policies by Combining Language, Vision and Demonstration

[December 2019]: We contributed a workshop paper to the Workshop on Robot Learning at NeurIPS 2019!

Best Poster Award

Best Poster Award

[January 2019]: I received the Best Poster Award by Nvidia at the Southwest Robotics Symposium for my work on Neural Policy Translation for Robot Control!

Robotics Intern @ BOSCH

Robotics Intern @ BOSCH

[Summer 2018]: I will be joining Robot BOSCH in Sunnyvale for an internship to work on semantic data analysis with a focus on time series segmentation.

Paper: Extrinsic Dexterity through Active Slip Control using Deep Predictive Models

Paper: Extrinsic Dexterity through Active Slip Control using Deep Predictive Models

[May 2018]: We got our paper accepted to ICRA 2018, and I will be presenting our work in Brisbane, Australia!

Best Video Award

Best Video Award

[November 2016]: Awarded at the International Conference on Humanoid Robots (Humanoids) 2016 for our work on Learning human-robot interactions from human-human demonstrations

About Me

Short Introduction

I am a Postdoctoral Fellow at Carnegie Mellon University's Robotics Institute, where I am part of the Advanced Agent-Robotics Technology Lab led by Prof. Katia Sycara. As part of the lab, I mainly work on Human-Agent Teaming and Machine Learning, investigating how humans and robots can efficiently collaborate in complex scenarios by analyzing the underlying concepts of trust and coordination. Before this, I received a B.Sc and M.Sc. in Engineering and Computing from the TU Bergakademie Freiberg in Germany and a Ph.D. in Computer Science from Arizona State University. During my Ph.D., I worked in the Interactive Robotics Lab led by Dr. Heni Ben Amor.

Throughout my academic career, I have worked on bridging the gap between cognition and robotics by using modern approaches to artificial intelligence. More specifically, I have implemented natural language processing methodologies to enhance imitation learning, which received the best poster award at the Southwest Robotics Symposium 2019, award by NVIDIA, as well as a spotlight presentation at NeurIPS 2020. Recently, I am expanding on this work by incorporating Theory of Mind, modeling a user's mental state for better collaboration. Further, I am also interested in creating trust and competence in the system by investigating human Theory of Machines, capturing how users perceive the robot they interact with.

Education

2017 - 2021
Arizona State University, USA

Ph.D. Computer Science

Thesis: Multimodal Robot Learning for Grasping and Manipulation

2015 - 2016
Technische Universität Freiberg, Germany

M.Sc. Engineering & Computing

Thesis: A data-driven approach for triadic interactions in human-robot interaction

2011 - 2015
Technische Universität Freiberg, Germany

B.Sc. Engineering & Computing

Thesis: Upper body tracking for avatar visualization in HMD-based virtual reality

Experience

January 2022 - Present
Carnegie Mellon University

Postdoctoral Fellow

I am working in the Advanced Agent-Robotics Technology Lab with Prof. Katia Sycara on efficient Human-Agent Teaming.

Summer 2021
X, The Moonshot Factory

Resident @ X

As a resident at X, the moonshot factory, I worked on industrial manipulation tasks for Intrinsic, a robotics software and AI project at X.

2017 - 2021
Arizona State University

Graduate Service Assistant

During my Ph.D., I worked as a teaching and research assistant on various occasions in the Interactive Robotics Lab led by Dr. Heni Ben Amor.

Awards

Best Poster Award

Best Poster Award

Awarded by NVIDIA at the Southwest Robotics Symposium 2019 for my work on “Neural Policy Translation for Robot Control”

CIDSE PhD Fellowship

CIDSE PhD Fellowship

Awarded 2017, 2018, 2019, 2020, and 2021 for excellent research progress and strong academic work by the “School of Computing, Informatics, and Decision Systems Engineering”

Best Video Award

Best Video Award

Awarded at the International Conference on Humanoid Robots (Humanoids) 2016 for my work on “Learning human-robot interactions from human-human demonstrations”

Publications

Find all my workshop, conference, and journal papers here
Language-Conditioned Imitation Learning for Robot Manipulation Tasks

Language-Conditioned Imitation Learning for Robot Manipulation Tasks

Simon Stepputtis, Joseph Campbell, Mariano Phielipp, Stefan Lee, Chitta Baral, Heni Ben Amor
Neural Information Processing Systems (NeurIPS), 2020
Imitation Learning of Robot Policies by Combining Language, Vision, and Demonstration

Imitation Learning of Robot Policies by Combining Language, Vision, and Demonstration

Simon Stepputtis, Joseph Campbell, Mariano Phielipp, Chitta Baral, Heni Ben Amor
NeurIPS Workshop on Robot Learning: Control and Interaction in the Real World, 2019
Improved Exploration Through Latent Trajectory Optimization in Deep Deterministic Policy Gradient

Improved Exploration Through Latent Trajectory Optimization in Deep Deterministic Policy Gradient

Kevin Sebastian Luck, Mel Vecerik, Simon Stepputtis, Heni Ben Amor, Jonathan Scholz
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019
Learning Interactive Behaviors for Musculoskeletal Robots Using Bayesian Interaction Primitives

Learning Interactive Behaviors for Musculoskeletal Robots Using Bayesian Interaction Primitives

Joseph Campbell, Arne Hitzmann, Simon Stepputtis, Shuhei Ikemoto, Koh Hosoda, Heni Ben Amor
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019
Neural Policy Translation for Robot Control

Neural Policy Translation for Robot Control

Simon Stepputtis, Chitta Baral, Heni Ben Amor
Southwest Robotics Symposium, 2019
One-Shot Learning of Human–Robot Handovers with Triadic Interaction Meshes

One-Shot Learning of Human–Robot Handovers with Triadic Interaction Meshes

David Vogt, Simon Stepputtis, Bernhard Jung, Heni Ben Amor
Autonomous Robots, 2018
Extrinsic Dexterity Through Active Slip Control Using Deep Predictive Models

Extrinsic Dexterity Through Active Slip Control Using Deep Predictive Models

Simon Stepputtis, Yezhou Yang, Heni Ben Amor
IEEE International Conference on Robotics and Automation (ICRA), 2018
Towards Semantic Policies for Human-Robot Collaboration

Towards Semantic Policies for Human-Robot Collaboration

Simon Stepputtis, Chitta Baral, Heni Ben Amor
Southwest Robotics Symposium, 2018
Probabilistic Multimodal Modeling for Human-Robot Interaction Tasks

Probabilistic Multimodal Modeling for Human-Robot Interaction Tasks

Joseph Campbell, Simon Stepputtis, Heni Ben Amor
Conference on Robot Learning (CoRL), 2017
Speech Enhanced Imitation Learning and Task Abstraction for Human-Robot Interaction

Speech Enhanced Imitation Learning and Task Abstraction for Human-Robot Interaction

Simon Stepputtis, Chitta Baral, Heni Ben Amor
IROS 2017: Workshop on Synergies Between Learning and Interaction, 2017
Active Slip Control for In-Hand Object Manipulation using Deep Predictive Models

Active Slip Control for In-Hand Object Manipulation using Deep Predictive Models

Simon Stepputtis, Heni Ben Amor
RSS 2017: Workshop on Tactile Sensing for Manipulation: Hardware, Modeling, and Learning, 2017
Deep Predictive Models for Active Slip Control

Deep Predictive Models for Active Slip Control

Simon Stepputtis, Heni Ben Amor
RSS 2017: Workshop on (Empirically) Data-Driven Robotic Manipulation, 2017
A System for Learning Continuous Human-Robot Interactions from Human-Human Demonstrations

A System for Learning Continuous Human-Robot Interactions from Human-Human Demonstrations

David Vogt, Simon Stepputtis, Steve Grehl, Bernhard Jung, Heni Ben Amor
IEEE International Conference on Robotics and Automation (ICRA), 2017
Learning Human-Robot Interactions from Human-Human Demonstrations (With Applications in Lego Rocket Assembly)

Learning Human-Robot Interactions from Human-Human Demonstrations (With Applications in Lego Rocket Assembly)

David Vogt, Simon Stepputtis, Richard Weinhold, Bernhard Jung, Heni Ben Amor
IEEE-RAS International Conference on Humanoid Robots (Humanoids), 2016

Videos / Projects

Here you will find videos and project overviews
Language-Conditioned Imitation Learning for Robot Manipulation Tasks

Language-Conditioned Imitation Learning for Robot Manipulation Tasks

Video
Learning Interactive Behaviors for Musculoskeletal Robots Using Bayesian Interaction Primitives

Learning Interactive Behaviors for Musculoskeletal Robots Using Bayesian Interaction Primitives

Video
Probabilistic Multimodal Modeling for Human-Robot Interaction Tasks

Probabilistic Multimodal Modeling for Human-Robot Interaction Tasks

Video
Language Conditioned Imitation Learning

Language Conditioned Imitation Learning

Research Projects
Interplanetary Initiative

Interplanetary Initiative

Research Projects
Tactile Sensing

Tactile Sensing

Research Projects
A Robot Learns To Jointly Assemble A Lego-Rocket With A User

A Robot Learns To Jointly Assemble A Lego-Rocket With A User

Video
Learning Continuous Human-Robot Interactions from Human-Human Demonstrations

Learning Continuous Human-Robot Interactions from Human-Human Demonstrations

Video
wp-content/themes/kerge-child