Speech Models Excel in Wearable Health Tasks

Historically, multi-modal large language models (MLLMs) have achieved significant progress in domains like vision, enabling advanced understanding and reasoning capabilities. However, this broad success has not extended to time-series data, where prior works on time-series MLLMs have primarily focused on forecasting tasks. Few studies have demonstrated how large language models could be effectively applied to time-series reasoning in natural language, leaving a gap in handling diverse sensor-based applications.

This new research reveals that speech foundation models, such as HuBERT and wav2vec 2.0, learn representations that generalize beyond their original audio domain to achieve state-of-the-art performance on time-series tasks from wearable sensors. The study shows that both speech and sensor data encode information in time- and frequency-domains, including spectral powers and waveform shapelets, allowing these models to adapt seamlessly. By leveraging pre-trained speech models, the approach addresses the limitations of modality-specific datasets and enhances capabilities in data-scarce scenarios.

Ology involves extracting features from speech foundation models and training simple probes on these features for various health-related tasks. Specifically, the researchers used HuBERT and wav2vec 2.0 to generate representations, which were then applied to mood classification, arrhythmia detection, and activity classification using wearable sensor data. This process highlights the relevance of convolutional feature encoders in speech models for capturing temporal patterns in sensor signals, without requiring extensive retraining or large datasets.

From the study demonstrate that probes trained on features from speech models consistently outperform those from self-supervised models trained directly on modality-specific datasets. In tasks like mood classification and arrhythmia detection, the speech-based features led to higher accuracy and robustness, establishing new benchmarks in these areas. indicate that the shared properties of time-series data across speech and sensors enable effective transfer learning, even with minimal probing s.

Of this work are significant for developing generalized time-series models that unify speech and sensor modalities, potentially streamlining AI applications in healthcare and beyond. By reducing the need for large, labeled datasets, this approach could accelerate innovations in wearable technology and remote monitoring. It represents a step toward more efficient and adaptable AI systems that leverage cross-domain insights for improved performance.

Limitations noted in the research include the reliance on specific speech models and tasks, which may not cover all time-series applications. The authors acknowledge that further validation is needed across diverse datasets and real-world conditions to ensure broad applicability. Despite these constraints, the study provides a strong foundation for future explorations into unified time-series modeling.