Address
N4-B2a-01, School of Computer Science and Engineering (SCSE)
Nanyang Technological University, Singapore 639798
sijie001@e.ntu.edu.sg
1Nanyang Technological University, 2The Hong Kong Polytechnic University, 3The Hong Kong University of Science and Technology
This paper presents HandFi, which constructs hand skeletons with practical WiFi devices. Unlike previous WiFi hand sensing systems that primarily employ predefined gestures for pattern matching, by constructing the hand skeleton, HandFi can enable a variety of downstream WiFi-based hand sensing applications in gaming, healthcare, and smart homes. Deriving the skeleton from WiFi signals is challenging, especially because the palm is a dominant reflector compared with fingers. HandFi develops a novel multi-task learning neural network with a series of customized loss functions to capture the low-level hand information from WiFi signals. During offline training, HandFi takes raw WiFi signals as input and uses the leap motion to provide supervision. During online use, only with commercial WiFi, HandFi is capable of producing 2D hand masks as well as 3D hand poses. We demonstrate that HandFi can serve as a foundation model to enable developers to build various applications such as finger tracking and sign language recognition, and outperform existing WiFi-based solutions.
@inproceedings{HandFi_SenSys23,
author = {Ji, Sijie and Zhang, Yuanye and Zheng, Yuanqing and Li, Mo},
title = {Construct 3D Hand Skeleton with Commercial WiFi},
year = {2023},
publisher = {Association for Computing Machinery},
booktitle = {Proceedings of the 21th ACM Conference on Embedded Networked Sensor Systems},
series = {SenSys '23}
}
N4-B2a-01, School of Computer Science and Engineering (SCSE)
Nanyang Technological University, Singapore 639798
sijie001@e.ntu.edu.sg