| | --- |
| | license: apache-2.0 |
| | datasets: |
| | - ShadiAbpeikar/HandGesture2Robot |
| | language: |
| | - en |
| | base_model: |
| | - google/siglip2-base-patch16-224 |
| | pipeline_tag: image-classification |
| | library_name: transformers |
| | tags: |
| | - Robot |
| | - Hand-Gesture |
| | - SigLIP2 |
| | - code |
| | - Sign |
| | --- |
| | |
| |  |
| |
|
| | # **Hand-Gesture-2-Robot** |
| |
|
| | > **Hand-Gesture-2-Robot** is an image classification vision-language encoder model fine-tuned from **google/siglip2-base-patch16-224** for a single-label classification task. It is designed to recognize hand gestures and map them to specific robot commands using the **SiglipForImageClassification** architecture. |
| |
|
| | ```py |
| | Classification Report: |
| | precision recall f1-score support |
| | |
| | "rotate anticlockwise" 0.9926 0.9958 0.9942 944 |
| | "increase" 0.9975 0.9975 0.9975 789 |
| | "release" 0.9941 1.0000 0.9970 670 |
| | "switch" 1.0000 0.9986 0.9993 728 |
| | "look up" 0.9984 0.9984 0.9984 635 |
| | "Terminate" 0.9983 1.0000 0.9991 580 |
| | "decrease" 0.9942 1.0000 0.9971 684 |
| | "move backward" 0.9986 0.9972 0.9979 725 |
| | "point" 0.9965 0.9913 0.9939 1716 |
| | "rotate clockwise" 1.0000 1.0000 1.0000 868 |
| | "grasp" 0.9922 0.9961 0.9941 767 |
| | "pause" 0.9991 1.0000 0.9995 1079 |
| | "move forward" 1.0000 0.9944 0.9972 886 |
| | "Confirm" 0.9983 0.9983 0.9983 573 |
| | "look down" 0.9985 0.9970 0.9977 664 |
| | "move left" 0.9952 0.9968 0.9960 622 |
| | "move right" 1.0000 1.0000 1.0000 622 |
| | |
| | accuracy 0.9972 13552 |
| | macro avg 0.9973 0.9977 0.9975 13552 |
| | weighted avg 0.9972 0.9972 0.9972 13552 |
| | ``` |
| |
|
| |  |
| |
|
| | The model categorizes hand gestures into 17 different robot commands: |
| | - **Class 0:** "rotate anticlockwise" |
| | - **Class 1:** "increase" |
| | - **Class 2:** "release" |
| | - **Class 3:** "switch" |
| | - **Class 4:** "look up" |
| | - **Class 5:** "Terminate" |
| | - **Class 6:** "decrease" |
| | - **Class 7:** "move backward" |
| | - **Class 8:** "point" |
| | - **Class 9:** "rotate clockwise" |
| | - **Class 10:** "grasp" |
| | - **Class 11:** "pause" |
| | - **Class 12:** "move forward" |
| | - **Class 13:** "Confirm" |
| | - **Class 14:** "look down" |
| | - **Class 15:** "move left" |
| | - **Class 16:** "move right" |
| |
|
| | # **Run with Transformers🤗** |
| |
|
| | ```python |
| | !pip install -q transformers torch pillow gradio |
| | ``` |
| |
|
| | ```python |
| | import gradio as gr |
| | from transformers import AutoImageProcessor |
| | from transformers import SiglipForImageClassification |
| | from transformers.image_utils import load_image |
| | from PIL import Image |
| | import torch |
| | |
| | # Load model and processor |
| | model_name = "prithivMLmods/Hand-Gesture-2-Robot" |
| | model = SiglipForImageClassification.from_pretrained(model_name) |
| | processor = AutoImageProcessor.from_pretrained(model_name) |
| | |
| | def gesture_classification(image): |
| | """Predicts the robot command from a hand gesture image.""" |
| | image = Image.fromarray(image).convert("RGB") |
| | inputs = processor(images=image, return_tensors="pt") |
| | |
| | with torch.no_grad(): |
| | outputs = model(**inputs) |
| | logits = outputs.logits |
| | probs = torch.nn.functional.softmax(logits, dim=1).squeeze().tolist() |
| | |
| | labels = { |
| | "0": "rotate anticlockwise", |
| | "1": "increase", |
| | "2": "release", |
| | "3": "switch", |
| | "4": "look up", |
| | "5": "Terminate", |
| | "6": "decrease", |
| | "7": "move backward", |
| | "8": "point", |
| | "9": "rotate clockwise", |
| | "10": "grasp", |
| | "11": "pause", |
| | "12": "move forward", |
| | "13": "Confirm", |
| | "14": "look down", |
| | "15": "move left", |
| | "16": "move right" |
| | } |
| | predictions = {labels[str(i)]: round(probs[i], 3) for i in range(len(probs))} |
| | |
| | return predictions |
| | |
| | # Create Gradio interface |
| | iface = gr.Interface( |
| | fn=gesture_classification, |
| | inputs=gr.Image(type="numpy"), |
| | outputs=gr.Label(label="Prediction Scores"), |
| | title="Hand Gesture to Robot Command", |
| | description="Upload an image of a hand gesture to predict the corresponding robot command." |
| | ) |
| | |
| | # Launch the app |
| | if __name__ == "__main__": |
| | iface.launch() |
| | ``` |
| |
|
| | # **Intended Use:** |
| |
|
| | The **Hand-Gesture-2-Robot** model is designed to classify hand gestures into corresponding robot commands. Potential use cases include: |
| |
|
| | - **Human-Robot Interaction:** Enabling intuitive control of robots using hand gestures. |
| | - **Assistive Technology:** Helping individuals with disabilities communicate commands. |
| | - **Industrial Automation:** Enhancing robotic operations in manufacturing. |
| | - **Gaming & VR:** Providing gesture-based controls for immersive experiences. |
| | - **Security & Surveillance:** Implementing gesture-based access control. |
