LLaSA-7B

LLaSA-7B is a large language and sensor assistant that can interpret IMU data for human activities.

Abstract

Wearable systems can recognize activities from IMU data but often fail to explain their underlying causes or contextual significance. To address this limitation, we introduce two large-scale resources: SensorCap, comprising 35,960 IMU--caption pairs, and OpenSQA, with 199,701 question--answer pairs designed for causal and explanatory reasoning. OpenSQA includes a curated tuning split (Tune-OpenSQA) optimized for scientific accuracy, narrative clarity, and diagnostic insight. Leveraging these datasets, we develop LLaSA (Large Language and Sensor Assistant), a family of compact sensor-aware language models (7B and 13B) that generate interpretable, context-rich responses to open-ended questions grounded in raw IMU data. LLaSA outperforms commercial LLMs, including GPT-3.5 and GPT-4o-mini, on benchmark and real-world tasks, demonstrating the effectiveness of domain supervision and model alignment for sensor reasoning.

Model Summary

• Developed by: BASH Lab, WPI
• Model type: sensor-text-to-text
• Language(s) (NLP): English
• Finetuned from model: lmsys/vicuna-7b-v1.5

Model Sources

• Repository: https://github.com/BASHLab/LLaSA
• Paper: https://arxiv.org/abs/2406.14498
• Project Website: https://bashlab.github.io/llasa_project/

Usage

git clone https://github.com/BASHLab/LLaSA.git
cd LLaSA/LLaSA
pip install -e .
hf download BASH-Lab/LLaSA-7B

You can run any of the inference scripts (zero-shot classification or question-answering) following the scripts in the eval subdirectory of the LLaSA GitHub repository, or you can run one sample as follows.

from llava.eval.run_llava import eval_model
from llava.mm_utils import get_model_name_from_path


sensor_reading = "imu.npy" # 20Hz, 2 sec (shape: (120,6))
prompt = "Narrate this activity by analyzing the data."
model_path = "LLaSA-7B"
args = type('Args', (), {
    "model_path": model_path,
    "model_base": None,
    "model_name": get_model_name_from_path(model_path),
    "query": prompt,
    "conv_mode": None,
    "image_file": sensor_reading,
    "sep": ",",
    "temperature": 0,
    "top_p": None,
    "num_beams": 1,
    "max_new_tokens": 300
})()
llasa_answer = eval_model(args)
print(llasa_answer)

Citation

BibTeX:

@article{imran2024llasa,
    title={LLaSA: A Sensor-Aware LLM for Natural Language Reasoning of Human Activity from IMU Data},
    author={Imran, Sheikh Asif and Khan, Mohammad Nur Hossain and Biswas, Subrata and Islam, Bashima},
    journal={arXiv preprint arXiv:2406.14498},
    year={2024}
}