Simon Stepputtis

About

I am an Assistant Professor in the Department of Mechanical Engineering at Virginia Tech, with a courtesy appointment in the Department of Computer Science, where I lead the Thinking Embodied Agents (TEA) Lab. I build robots that operate in human environments rather than around them. In the TEA Lab, we study how robots acquire and retain manipulation skills across their lifetime in unstructured settings, and how they model and adapt to the people working alongside them. We pair learned policies, such as language-conditioned, generative, and vision-language-driven approaches, with symbolic structure such as knowledge graphs, planners, and movement primitives, so that behavior generalizes from few demonstrations and stays inspectable enough to reason about safety and contact.

Before joining Virginia Tech, I was a Postdoctoral Fellow and Project Scientist at Carnegie Mellon’s Robotics Institute, where I worked with Prof. Katia Sycara on efficient human-agent teaming. I earned my Ph.D. from Arizona State University under Prof. Heni Ben Amor in the Interactive Robotics Lab, where I specialized in physical human-robot interaction.

Experience

  • Assistant Professor (2025 - Present)

    • Virginia Tech
    • I lead the Thinking Embodied Agents (TEA) Lab in the Department of Mechanical Engineering, with a courtesy appointment in the Department of Computer Science.

  • Postdoctoral Fellow / Project Scientist (2022 - 2025)

  • Resident @ Google X (May to October 2021)

    • X, The Moonshot Factory
    • As a resident, I worked on industrial manipulation tasks for Intrinsic, a robotics software and AI project at Google X.

  • Robotics Intern (Summer 2018)

    • Robert Bosch LLC
    • During my internship, I segmented time series data into semantic sections and validated user behaviors based on usage patterns and global goal constraints.

Education

  • Ph.D. Computer Science (2017 - 2021)

    • Arizona State University, USA
    • Thesis: Multimodal Robot Learning for Grasping and Manipulation

  • M.Sc. Engineering & Computing (2015 - 2016)

    • TU Bergakademie Freiberg, Germany
    • Thesis: A data-driven approach for triadic interactions in human-robot interaction

  • B.Sc. Engineering & Computing (2011 - 2015)

    • TU Bergakademie Freiberg, Germany
    • Thesis: Upper body tracking for avatar visualization in HMD-based virtual reality

Teaching

  • ME 3534 - Controls Engineering I (Spring 2026, Spring 2027)

    • Virginia Tech
    • A required junior-level course on the fundamentals of feedback control. We build up time-domain and frequency-domain analysis, then use it to synthesize controllers that meet performance and stability requirements. The accompanying lab moves those designs onto hardware through numerical simulation and discrete real-time implementation on microcontrollers.

  • CSE 355 - Introduction to Theoretical Computer Science (Summer 2020)

    • Arizona State University, sole instructor
    • A core undergraduate course on the theory of computation, taught without reference to any one programming language. We work up the hierarchy from finite automata and regular languages through context-free grammars to Turing machines, then turn that machinery on itself to ask which problems a computer can solve at all, and how much time and space the solvable ones demand.

Awards

  • Best Workshop Paper Award (2025)

    • Generative Models x HRI Workshop at RSS 2025

  • Best Workshop Paper Award (2025)

    • Workshop on Nonverbal Cues for Human-Robot Cooperative Intelligence at ICRA 2025

  • Best Poster Award (2019)

    • Southwest Robotics Symposium, awarded by NVIDIA

  • CIDSE Doctoral Fellowship (2017 - 2021)

    • Awarded annually by Arizona State University’s School of Computing, Informatics, and Decision Systems Engineering

  • Best Video Award (2016)

    • IEEE-RAS International Conference on Humanoid Robots

Invited Talks

  • University of Washington (2024)

    • Neuro-Symbolic Robot Intelligence

  • Imperial College London (2021)

    • An overview of our NeurIPS 2020 work on language-conditioned imitation learning.

  • Intel AI Labs (2020)

    • Language for Robotics

  • Intel Deep Learning Community of Practice (2020)

    • Imitation Learning for Adaptive Robot Control Policies from Language, Vision, and Motion

Service

  • Co-Chair, Pathways @ RSS (2025)

    • Organized fellowships and mentorship for students starting a career in robotics.

  • Advising Faculty, RSS Pioneers (2025)

    • Previously a member of the program committee in 2023.

  • Workshop Organizer (2023)

    • Articulate Robots: Utilizing Language for Robot Learning at RSS 2023

  • Workshop Organizer (2022)

    • Human Theory of Machines and Machine Theory of Mind for Human-Agent Teams at IROS 2022

  • RSS Pioneers (2022)

    • Selected participant, presenting work on Language-Conditioned Human-Agent Teaming.

  • Reviewing

    • Outstanding Reviewer at CVPR, as well as reviewing for NeurIPS, ICLR, RSS, ICRA, IROS, HRI, T-RO, and RA-L.