【AIOT】智能感知--人

From: https://liudongdong1.github.io/

1. 人体存在感知

• 目标：检测环境中的所有人体，标记出每个人体的坐标位置；不限人体数量，适应中低空斜拍、人体轻度遮挡、截断等场景

.1. WAYV AIR

• WAYV AIR 智能人体存在感知雷达目前已成功应用于多个智能卫生间项目中，实现厕位的占位及人流量统计

  • 检测准确率高，不管是静止、微运动还是运动人员都可以实现准确检测；

  • 无隐私、敏感问题，无镜头设计；

  • 超低功耗，辐射量仅为蓝牙十分之一，对人体安全无害；

  • 检测距离远，适配各种安装高度；

  • 可自由设定检测范围，适用于不同大小和形状的空间区域；

  • 不受环境障碍物影响，如烟雾，污垢遮挡、低光照，热源等，不需要任何维护；

  • 美观、可隐藏在木材或塑料天花板等非金属材料后。

Shuai X, Shen Y, Tang Y, et al. milliEye: A Lightweight mmWave Radar and Camera Fusion System for Robust Object Detection[C]//Proceedings of the International Conference on Internet-of-Things Design and Implementation. 2021: 145-157. [pdf]

Bhatia J, Dayal A, Jha A, et al. Object Classification Technique for mmWave FMCW Radars using Range-FFT Features[C]//2021 International Conference on COMmunication Systems & NETworkS (COMSNETS). IEEE, 2021: 111-115.

Lu, Chris Xiaoxuan, et al. "See through smoke: robust indoor mapping with low-cost mmWave radar." Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services. 2020. [pdf]

Devoti, Francesco, et al. "PASID: Exploiting Indoor mmWave Deployments for Passive Intrusion Detection." IEEE INFOCOM 2020-IEEE Conference on Computer Communications. IEEE, 2020. [pdf]

Gu T, Fang Z, Yang Z, et al. Mmsense: Multi-person detection and identification via mmwave sensing[C]//Proceedings of the 3rd ACM Workshop on Millimeter-wave Networks and Sensing Systems. 2019: 45-50. [pdf]

J. Yan, G. Zhang, H. Hong, H. Chu, C. Li, and X. Zhu, “Phase-basedhuman target 2-D identification with a mobile FMCW radar platform,”IEEE Trans. Microw. Theory Techn., vol. 67, no. 12, pp. 5348–5359,Dec. 2019. [pdf]

M. Zhaoet al., “Through-wall human mesh recovery using radio signals,” inProc. IEEE Int. Conf. Comput. Vis., Oct. 2019,pp. 10112–10121. [pdf]

Zhang Y, Zhang J, Chu X, et al. Effects of Wall Reflection on the Per-Antenna Power Distribution of ZF-Precoded ULA for Indoor mmWave MU-MIMO Transmissions[J]. IEEE Communications Letters, 2020. [pdf]

J. Yan et al., "The Development of Vital-SAR-Imaging with an FMCW Radar System," 2019 IEEE MTT-S International Microwave Biomedical Conference (IMBioC), 2019, pp. 1-4, doi: 10.1109/IMBIOC.2019.8777881. [pdf]

Hicheri R, Pätzold M, Youssef N. Estimation of the velocity of a walking person in indoor environments from mmWave signals[C]//2018 IEEE Globecom Workshops (GC Wkshps). IEEE, 2018: 1-7. [pdf]

Huang X, Cheena H, Thomas A, et al. Indoor Detection and Tracking of People Using mmWave Sensor[J]. Journal of Sensors, 2021, 2021. [pdf]

2. 人员计数

• 静态人数统计：中远距离俯拍，以头部为识别目标统计图片中的瞬时人数；无人数上限，广泛适用于机场、车站、商场、展会、景区等人群密集场所
• 动态人流量统计：面向门店、通道等出入口场景，以头肩为识别目标，进行人体检测和追踪，根据目标轨迹判断进出区域方向，实现动态人数统计，返回区域进出人数

• 安防监控：实时监测机场、车站、展会、展馆、景区、学校、体育场等公共场所的人流量，及时导流、限流，预警核心区域人群过于密集等安全隐患
• 驾驶检测：针对客运车辆，实时监控上下车和车内乘客数量，分析站点客流量、车内超载情况，为线路规划、站台设计提供精准参考依据

. Yavari, X. Gao, and O. Boric-Lubecke, “Subject count estimation by using Doppler radar occupancy sensor,” inProc. Annu. Int. Conf.IEEE Eng. Med. Biol. Soc., Oct. 2018, pp. 4428–4431

<<<<<<< HEAD

Qi, Delong, et al. "YOLO5Face: Why Reinventing a Face Detector." arXiv preprint arXiv:2105.12931 (2021). [pdf] [code]

.1. NanoDet

.2. Ultra-Light-Fast-Generic-Face-Detector-1MB

3. 人像分割

• 将人体轮廓与图像背景进行分离，返回分割后的二值图、灰度图、透明背景人像前景图，多人体、复杂背景、遮挡、背面、侧面等各类人体姿态
• 证件照片：针对自拍类单人图片，基于人脸检测、人体关键点先裁剪出符合证件照场景的人像图片，对裁剪后的图片进行发丝级精细化分割，一键制作证件照
• 视频人像：可对实时视频流中的人像背景进行分割，支持背景图自定义及3D背景图定制，适用于视频会议、短视频及直播场景

• 人像抠图美化：将原始图片中的人像分离出来，选择新的背景图像进行替换、合成；同时可以对背景进行虚化处理，突出人像，实现大光圈人像拍照效果
• 人体特效：识别用户的人体轮廓，为人像实时增加各种设定的背景特效、贴纸道具，提供更加丰富的娱乐体验
• 影视后期处理：识别影视作品中的人像区域，进行一键抠像、背景替换、人像虚化等后期处理

Salehi B, Belgiovine M, Sanchez S G, et al. Machine Learning on Camera Images for Fast mmWave Beamforming[C]//2020 IEEE 17th International Conference on Mobile Ad Hoc and Sensor Systems (MASS). IEEE, 2020: 338-346. [pdf]

P. Nallabolu and C. Li, "A Novel Radar Imaging Method Based on Random Illuminations Using FMCW Radar," 2020 IEEE Topical Conference on Wireless Sensors and Sensor Networks (WiSNeT), 2020, pp. 27-29, doi: 10.1109/WiSNeT46826.2020.9037583. [pdf]

4. 关键点检测

• 体育锻炼：根据人体关键点信息，分析人体姿态、运动轨迹、动作角度等，辅助运动员进行体育训练，分析健身锻炼效果，提升教学效率；
• 娱乐互动： 视频直播平台、线下互动屏幕等场景，可基于人体检测和关键点分析，增加身体道具、体感游戏等互动形式，丰富娱乐体验
• 安防监控：实时监测定位人体，判断特殊时段、核心区域是否有人员入侵；基于人体关键点信息，进行二次开发，识别特定的异常行为，及时预警管控

S. Li, X. Li, Q. Lv, G. Tian, and D. Zhang, “WiFit: Ubiquitous body weight exercise monitoring with commodity wi-fi devices,” inProc. IEEE SmartWorld, Ubiquitous Intell. Comput., Adv. TrustedComput., Scalable Comput. Commun., Cloud Big Data Comput., Inter-net People Smart City Innovations, SmartWorld/UIC/ATC/ScalCom/CBDCom/IoP/SCI, Dec. 2018, pp. 530–537

.1. 人体

• 通过摄像头捕捉追踪人体在一段时间内的姿势变化，检测人体姿态是否达到预期的角度、幅度、速度，辅助健身锻炼、体育训练、康复训练等应用

.2. 人脸

.3. 手部

.1. 关键点检测

精准定位手部的21个主要骨节点，包括指尖、各节指骨连接处等，返回每个骨节点的坐标信息

• AR特效: 短视频、直播等娱乐交互场景中，基于指尖点检测和指骨关键点检测，可实现手部特效、空间作画等多种创意玩法，丰富交互体验
• 自定义手势识别: 根据手部骨节坐标信息，可灵活定义业务场景中需要用到的手势，例如面向智能家电、可穿戴等硬件设备的操控类手势，面向内容审核场景的特殊手势

.2. 手势识别

识别24种常见手势，支持单手手势和双手手势，包括拳头、OK、比心、作揖、作别、祈祷、我爱你、点赞、Diss、Rock、竖中指、数字等

• 智能家居：智能家电、家用机器人、可穿戴、儿童教具等硬件设备，通过用户的手势控制对应的功能，人机交互方式更加智能化、自然化
• 视频直播：视频直播或者拍照过程中，结合用户的手势（如点赞、比心），实时增加相应的贴纸或特效，丰富交互体验
• 智能驾驶：将手势识别应用到驾驶辅助系统中，使用手势来控制车内的各种功能、参数，一定程度上解放双眼，将更多的注意力放在道路上，提升驾车安全性

Ren Y, Lu J, Beletchi A, et al. Hand gesture recognition using 802.11 ad mmWave sensor in the mobile device[C]//2021 IEEE Wireless Communications and Networking Conference Workshops (WCNCW). IEEE, 2021: 1-6.

Wang S, Song J, Lien J, et al. Interacting with soli: Exploring fine-grained dynamic gesture recognition in the radio-frequency spectrum[C]//Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 2016: 851-860. [pdf]

.3. 指尖

.1. 指尖检测

检测图像中的手部位置，精准定位食指指尖，返回手部、食指指尖的坐标信息，尤其适用于儿童学习机点读场景

.2. 定位追踪

• 键盘输入
• 智能点读

4. 属性识别

识别人体的20余类通用属性，包含性别年龄、服饰类别、服饰颜色、戴帽子（可区分安全帽/普通帽）、戴口罩、背包、手提物、抽烟、使用手机，戴手套等

• Scenarios

• 安防监控：识别人体的性别年龄、衣着外观等特征，辅助定位追踪特定人员；监测预警各类危险、违规行为（如公共场所跑跳、抽烟、未佩戴口罩），减少安全隐患
• 人群属性，广告投放： 楼宇、户外等广告屏智能化升级，采集人体信息，分析人群属性，定向投放广告物料，提升用户体验和商业效率

5. 粗粒度行为感知

.1. 驾驶行为

识别图像中的所有人体，将目标最大的人体作为驾驶员，返回坐标位置，同时返回总人数（含驾驶员和乘客）；支持夜间红外场景

• 营运车辆驾驶监测： 针对出租车、客车、公交车、货车等各类营运车辆，实时监控车内情况，识别驾驶员抽烟、使用手机、未系安全带、未佩戴口罩、疲劳、视线偏离等违规行为，及时预警，降低事故发生率，保障人身财产安全
• 社交内容分析审核: 汽车类论坛、社区平台，对配图库以及用户上传的UGC图片进行分析识别，自动过滤出涉及危险驾驶行为的不良图片，有效减少人力成本并降低业务违规风险

C. Dinget al., “Inattentive driving behavior detection based onportable FMCW radar,”IEEE Trans. Microw. Theory Techn., vol. 67,no. 10, pp. 4031–4041, Oct. 2019

.2. 危险行为

• 单人场景行为识别： 针对单人监控视频片段，可识别4类常见危险行为，包括：情绪性指人、摔倒、激烈抱怨、砸东西 、高空抛物 触摸去静电装置（某些工厂如燃气场进入前）
• 双人场景行为识别： 针对双人监控视频片段，识别是否有危险行为，如出拳、拉扯、推搡、激烈搂抱、踢踹、砸按等

• 安防监控: 社区、园区、厂房、门店、楼道、电梯等重点区域，检测人员摔倒、砸按、打斗、肢体冲突等行为，及时预警、管控，避免安全事故
• 智能看护: 家庭、医院、养老院、幼儿园等场所，实时监控分析人员行为，及时发现老人摔倒、病患摔倒、幼儿摔倒等危险情况，保障人身安全

Y. Tang, Z. Peng, L. Ran and C. Li, "iPrevent: A novel wearable radio frequency range detector for fall prevention," 2016 IEEE International Symposium on Radio-Frequency Integration Technology (RFIT), 2016, pp. 1-3, doi: 10.1109/RFIT.2016.7578162. [pdf]

.3. 多人活动

D. V. Q. Rodrigues and C. Li, "Noncontact Exercise Monitoring in Multi-Person Scenario With Frequency-Modulated Continuous-Wave Radar," 2020 IEEE MTT-S International Microwave Biomedical Conference (IMBioC), 2020, pp. 1-3, doi: 10.1109/IMBIoC47321.2020.9385031. [pdf]

6. 细粒度行为感知

.1. 眼部

• 眨眼转头检测

E. Cardillo, G. Sapienza, C. Li and A. Caddemi, "Head Motion and Eyes Blinking Detection: a mm-Wave Radar for Assisting People with Neurodegenerative Disorders," 2020 50th European Microwave Conference (EuMC), 2021, pp. 925-928, doi: 10.23919/EuMC48046.2021.9338116.

• aid for people with neurodegenerative disorder.
• silicon Radar TRX_120_002 on-chip frontend

• 瞳孔转动检测

.2. 喉咙

• 声音识别
• 声纹识别

Li, Huining, et al. "VocalPrint: exploring a resilient and secure voice authentication via mmWave biometric interrogation." Proceedings of the 18th Conference on Embedded Networked Sensor Systems. 2020. [pdf]

Xu, Chenhan, et al. "Waveear: Exploring a mmwave-based noise-resistant speech sensing for voice-user interface." Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services. 2019. [pdf]

.3. vital sign

L. Zhang, C. Ding, X. Zhou, H. Hong, C. Li and X. Zhu, "Body movement cancellation using adaptive filtering technology for radar-based vital sign monitoring," 2020 IEEE Radar Conference (RadarConf20), 2020, pp. 1-5, doi: 10.1109/RadarConf2043947.2020.9266671.

.1.呼吸

H. Zhao et al., "A Noncontact Breathing Disorder Recognition System Using 2.4-GHz Digital-IF Doppler Radar," in IEEE Journal of Biomedical and Health Informatics, vol. 23, no. 1, pp. 208-217, Jan. 2019, doi: 10.1109/JBHI.2018.2817258. [pdf]

. M. M. Islam, A. Sylvester, G. Orpilla, and V. M. Lubecke, “Respiratory feature extraction for radar-based continuous identity authentication,” inProc. IEEE Radio Wireless Symp., Jan. 2020, pp. 119–122.

X. Ma, Y. Wang, X. You, J. Lin, and L. Li, “Respiratory pattern recognition of an adult bullfrog using a 100-GHz CW Doppler radar transceiver,” inProc. IEEE MTT-S Int. Microw. Biomed. Conf., 2019,pp. 1–3.

Q. Lvet al., “Doppler vital signs detection in the presence of large-scale random body movements,”IEEE Trans. Microw. Theory Techn.,vol. 66, no. 9, pp. 4261–4270, Sep. 2018.

J. Tu, T. Hwang, and J. Lin, “Respiration rate measurement under 1-D body motion using single continuous-wave Doppler radar vital sign detection system,”IEEE Trans. Microw. Theory Techn., vol. 64, no. 6,pp. 1937–1946, Jun. 2016.

S. M. M. Islam, E. Yavari, A. Rahman, V. M. Lubecke, and O.Boric-Lubecke, “Multiple subject respiratory pattern recognition and estimation of direction of arrival using phase-comparison mono-pulse radar,” inProc. IEEE Radio Wireless Symp., 2019, pp. 1–4.

Cardillo, Emanuele, Changzhi Li, and Alina Caddemi. "Vital Sign Detection and Radar Self-Motion Cancellation Through Clutter Identification." IEEE Transactions on Microwave Theory and Techniques 69.3 (2021): 1932-1942. [pdf] [todo]

• remove a novel technique to remove the radar self-motion effects(RSMs) for accurate detection of human vital signs;
• extracts the RSM from the signals reflected by stationary clutters, and propose two procedures to automatic identification for detecting both small and large radar motions.
  • the autocorrelation applied to the received phase histories for each measured range bin based on the inherent periodicity.
  • the autocorrelation on the cross correlation between the measured range-Doppler pro-files.

.2. Blood pressure

. Hui, T. B. Conroy, and E. C. Kan, “Multi-point near-field RF sensing of blood pressures and heartbeat dynamics,”IEEE Access,vol. 8, pp. 89935–89945, 2020.

.3. Cardiac motion

H. Zhao, X. Gu, H. Hong, Y. Li, X. Zhu, and C. Li, “Non-contact beat-to-beat blood pressure measurement using continuous waveDoppler radar,” inIEEE MTT-S Int. Microw. Symp. Dig., Jun. 2018,pp. 1413–1415.

. Saluja, J. Casanova, and J. Lin, “A supervised machine learning al-gorithm for heart-rate detection using Doppler motion-sensing radar,”IEEE J. Electromagn. RF Microw. Med. Biol., vol. 4, no. 1, pp. 45–51,Mar. 2020.VOLUME 1, NO. 1, JANUARY 202177

F. Lin, C. Song, Y. Zhuang, W. Xu, C. Li, and K. Ren, “Cardiacscan: A non-contact and continuous heart-based user authentication system,” inProc. Annu. Int. Conf. Mobile Comput. Netw., Oct. 2017,pp. 315–328.

.4. 步态

. S. Koo, L. Ren, Y. Wang, and A. E. Fathy, “UWB micro doppler radar for human gait analysis, tracking more than one person, and vital sign detection of moving persons,” inIEEE MTT-S Int. Microw. Symp.Dig., 2013, pp. 1–4.

Y. Tang, L. Ran and C. Li, "A feasibility study on human gait monitoring using a wearable K-band radar," 2016 46th European Microwave Conference (EuMC), 2016, pp. 918-921, doi: 10.1109/EuMC.2016.7824494. [pdf]

.5. 书写

. Lienet al., “Soli: Ubiquitous gesture sensing with millimeter waveradar,”ACM Trans. Graph., vol. 35, no. 10, pp. 1–19, 2016

.6. 睡眠

H. Hong et al., "Microwave Sensing and Sleep: Non contact Sleep-Monitoring Technology With Microwave Biomedical Radar," in IEEE Microwave Magazine, vol. 20, no. 8, pp. 18-29, Aug. 2019, doi: 10.1109/MMM.2019.2915469.

L. Zhang, J. Xiong, H. Zhao, H. Hong, X. Zhu and C. Li, "Sleep stages classification by CW Doppler radar using bagged trees algorithm," 2017 IEEE Radar Conference (RadarConf), 2017, pp. 0788-0791, doi: 10.1109/RADAR.2017.7944310. [pdf]

H. Honget al., “Microwave sensing and sleep,”IEEE Microw. Mag.,vol. 20, no. 8, pp. 18–29, Aug. 2019.

. Baboli, A. Singh, B. Soll, O. Boric-Lubecke, and V. M. Lubecke,“Good night: Sleep monitoring using a physiological radar monitoring system integrated with a polysomnography system,”IEEE Microw.Mag., vol. 16, no. 6, pp. 34–41, Jul. 2015.

. Baboli, A. Singh, B. Soll, O. Boric-Lubecke, and V. M. Lubecke,“Wireless sleep apnea detection using continuous wave quadrature Doppler radar,”IEEE Sensors J., vol. 20, no. 1, pp. 538–545,Jan. 2020

H. Hong, L. Zhang, C. Gu, Y. Li, G. Zhou, and X. Zhu, “Noncontact sleep stage estimation using a CW Doppler radar,”IEEE J. Emerg. Sel.Topics Circuits Syst., vol. 8, no. 2, pp. 260–270, Jun. 2018.

F. Linet al., “SleepSense: A noncontact and cost-effective sleep monitoring system,”IEEE Trans. Biomed. Circuits Syst., vol. 11, no. 1,pp. 189–202, Feb. 2017.

.7. 摔倒

Y. Tang, Z. Peng, L. Ran and C. Li, "iPrevent: A novel wearable radio frequency range detector for fall prevention," 2016 IEEE International Symposium on Radio-Frequency Integration Technology (RFIT), 2016, pp. 1-3, doi: 10.1109/RFIT.2016.7578162. [pdf]

Jin F, Sengupta A, Cao S. mmFall: Fall Detection using 4D MmWave Radar and Variational Recurrent Autoencoder[J]. arXiv preprint arXiv:2003.02386, 2020. [pdf]

Sun Y, Hang R, Li Z, et al. Privacy-Preserving Fall Detection with Deep Learning on mmWave Radar Signal[C]//2019 IEEE Visual Communications and Image Processing (VCIP). IEEE, 2019: 1-4.

Wang K, Zhan G, Chen W. A New Approach for IoT-based Fall Detection System using Commodity mmWave Sensors[C]//Proceedings of the 2019 7th International Conference on Information Technology: IoT and Smart City. 2019: 197-201.

7. 追踪轨迹

Palacios, Joan, et al. "LEAP: Location estimation and predictive handover with consumer-grade mmWave devices." IEEE INFOCOM 2019-IEEE Conference on Computer Communications. IEEE, 2019. [pdf]

J. Wang, D. Nolte, K. Tanja, J. Muñoz-Ferreras, R. Gómez-García and C. Li, "Trade-off on Detection Range and Channel Usage for Moving Target Tracking using FSK Radar," 2020 IEEE Topical Conference on Wireless Sensors and Sensor Networks (WiSNeT), 2020, pp. 38-41, doi: 10.1109/WiSNeT46826.2020.9037618. [pdf]

Zeng, Yunze, et al. "Human tracking and activity monitoring using 60 GHz mmWave." 2016 15th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN). IEEE, 2016. [pdf]

8. HotTopic

.1. MOTION SEPARATION&CLASSIFICATION IN DYNAMIC ENVIRONMENT

• random motions of both human subject&radar platform

. Mercuri, I. R. Lorato, Y. H. Liu, F. Wieringa, C. Van Hoof, and T.Torfs, “Vital-sign monitoring and spatial tracking of multiple people using a contactless radar-based sensor,”Nature Electron., vol. 2, no. 6,pp. 252–262, Jun. 2019

Z. Guet al., “Blind separation of Doppler human gesture signals based on continuous-wave radar sensors,”IEEE Trans. Instrum. Meas.,vol. 68, no. 7, pp. 2659–2661, Jul. 2019.

S. M. M. Islam, E. Yavari, A. Rahman, V. M. Lubecke, and O. Boric-Lubecke, “Separation of respiratory signatures for multiple subjectsusing independent component analysis with the JADE algorithm,”inProc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., Oct. 2018,pp. 1234–1237.

A high-dynamic-range radar can be aided with algorithmssuch as matched filters to retrieve signals concealed by bodymotion noise [107].

In [108], the direction of body motion is extracted alongwith the new position of the respiration peaks in the frequencyspectrum and respiration rate was calculated

a low-IFSIMO system employed a two-step blind motion separation tosequentially tackles signal separation and nonlinear demodu-lation [109

[111], the vital signs signal fidelitywas improved using RSS indicator and Direction of Arrival(DOA) to compensate for the platform motion via a closedloop control system that modulates the UAV electronic speedcontroller. In addition, an optical tracking system [112] or anRF tag [113] can be used to achieve adaptive platform motionnoise cancellation.

a precise phase-based humantarget 2-D SAR imaging and recognition system based onvital sign tracking was demonstrated [114]. It first relies onFMCW phase detection to extract the vital signs of multiplehuman targets, then applies a SAR algorithm to obtain the 2-Dimaging of the scene and labels human targets.

.2. CROWD DETECTION AND SIGNAL-OF-INTEREST EXTRACTION

. An SNR-basedintelligent decision algorithm integrated two different ap-proaches to isolate respiratory signatures of two subjectswithin the radar beamwidth [115]: Independent ComponentAnalysis with the JADE algorithm (ICA-JADE) [116] andDOA [117],

.3. interaction of microwave technology &artificial intelligence

. Li and J. Lin, “Wavelet-transform-based data-length-variation technique for fast heart rate detection using 5.8-GHz CW Doppler radar,”IEEE Trans. Microw. Theory Techn., vol. 66, no. 1, pp. 568–576,Jan. 2018

. Tu and J. Lin, “Fast acquisition of heart rate in non-contact vital sign radar measurement using time-window-variation technique,”IEEE Trans. Instrum. Meas., vol. 65, no. 1, pp. 112–122, Jan. 2016.

C. Dinget al., “Continuous human motion recognition with a dynamic range-Doppler trajectory method based on FMCW radar,”IEEE Trans.Geosci. Remote Sens., vol. 57, no. 9, pp. 6821–6831, Sep. 2019.