Wearable gadgets, from smartwatches to health trackers, have change into ubiquitous, constantly capturing a wealthy stream of knowledge about our lives. They document our coronary heart fee, rely our steps, observe our health and sleep, and way more. This deluge of data holds immense potential for personalised well being and wellness. Nevertheless, whereas we will simply see what our physique is doing (e.g., a coronary heart fee of 150 bpm), the essential context of why (say, “a brisk uphill run” vs. “a worrying public talking occasion”) is usually lacking. This hole between uncooked sensor information and its real-world that means has been a serious barrier to unlocking the total potential of those gadgets.
The first problem lies within the shortage of large-scale datasets that pair sensor recordings with wealthy, descriptive textual content. Manually annotating thousands and thousands of hours of knowledge is prohibitively costly and time-consuming. To resolve this, and to actually let wearable information “communicate for itself”, we’d like fashions that may study the intricate connections between sensor indicators and human language straight from the information.
In “SensorLM: Studying the Language of Wearable Sensors”, we introduce SensorLM, a household of sensor–language basis fashions that bridges this hole. Pre-trained on an unprecedented 59.7 million hours of multimodal sensor information from over 103,000 people, SensorLM learns to interpret and generate nuanced, human-readable descriptions from high-dimensional wearable information, setting a brand new state-of-the-art in sensor information understanding.
