Continuous Wrist Control on the Hannes Prosthesis:
a Vision-based Shared Autonomy Framework

Federico Vasile, Elisa Maiettini, Giulia Pasquale, Nicolò Boccardo, Lorenzo Natale

TL;DR: Simulation environment for wrist control from an eye-in-hand camera

(check out also our dinoDet for instance segmentation, the simulation environment for synthetic data generation and the iHannes dataset used for evaluation)

Getting started

The project uses Unity 2021.3.28.f1. Find the version here and click on the Unity Hub button to download.

Installation

• Install Git LFS.
• Open a Command Prompt and run git lfs install to initialize it.
  Then, clone the repository: git clone --single-branch --branch main --recursive https://github.com/hsp-iit/hannes-wrist-control
• Go on the Unity Hub, click on Open and locate the downloaded repository.
• This repository has a submodule located at Python/libs/visual-servoing, follow the instructions to set up the Python environment.

Running the Simulation

This repository provides a simulation environment for the Hannes prosthesis in Unity. The wrist is controlled using Image-Based Visual Servoing (IBVS) from an eye-in-hand camera.

Steps to Run

0. Set up the Visual Servoing

   Navigate to Python/libs/visual-servoing/configs/conf.yaml and set:

   vservo_name: IBVSFEPropPS
   lmbda_ps: 0.01

1. Start the Python server

   cd <path_to_repo>/Python
   python src/socket/server/launch.py

2. Open the Unity project and press the Play button.

3. Control the arm

   • Use WASD, IJKL, and the arrow keys to translate and rotate the arm in space.
   • The wrist will automatically adjust to point at the target object.

Implementation Details

Communication between Unity and Python is handled through a TCP socket:

• Unity sends data (e.g., camera images, joint states, object poses) to the Python server.
• The Python server computes the control action and sends it back to Unity.

Key files:

• Unity client: Assets/Scripts/Socket/Client/SocketClient.cs
• Python server: Python/src/socket/server/launch.py

Citation

If you find our work useful, please consider citing our paper as follows:

@inproceedings{vasile2025continuous,
  title={Continuous Wrist Control on the Hannes Prosthesis: a Vision-based Shared Autonomy Framework},
  author={Vasile, Federico and Maiettini, Elisa and Pasquale, Giulia and Boccardo, Nicol{\`o} and Natale, Lorenzo},
  booktitle={2025 IEEE International Conference on Robotics and Automation (ICRA)},
  pages={},
  year={2025},
}

Mantainer

This repository is mantained by:

	@FedericoVasile1