The Prospects of Multimodal Artificial Intelligence Identification in Forensic Pathology

doi:10.3969/j.issn.1671-2072.2024.06.004

Abstract

Abstract: Traditional forensic pathological identification has primarily relied on methods such as on-site inspections, case analysis, autopsies, and histopathological examinations, all of which highly depend on the personal experience and subjective judgement of the examiners. After centuries of practice, this mode of identification has been the cornerstone of the forensic pathology. However, with the rapid advancement of artificial intelligence （AI） technologies, particularly the emergence of generative AI technologies, the application of AI in forensic pathology has evolved from analyzing single-modal data to processing multimodal data. This paper aims to delve into multimodal AI technologies and their data integration strategies in forensic pathological identification, especially the potential application scenarios and developmental prospects in forensic pathological identification. Additionally, this paper discusses the challenges and ethical issues that might arise with the implementation of multimodal AI technologies in forensic pathological identification practice and offers insights on how to overcome these obstacles.

Key words: forensic pathology, artificial intelligence （AI）, multimodal, data fusion

摘要： 传统法医病理学鉴定主要依赖于现场勘查、案情分析、尸体解剖和组织病理学检验等方法，这一过程高度依赖鉴定人的个人经验和主观判断。历经数百年的鉴定实践，该模式一直是法医病理学领域的基石。然而，随着人工智能（artifical intelligence，AI）技术的突飞猛进，特别是生成式AI技术的兴起，AI在法医病理学中的应用已经实现了从单一模态数据分析到多模态数据综合处理的转变条件。深入探讨了多模态AI技术及其数据融合策略在法医病理学鉴定中的应用展望，尤其是潜在应用场景和发展前景。同时，还审视了多模态AI技术在法医病理鉴定实践中可能遇到的挑战和伦理问题，并对如何克服这些“瓶颈”问题提出了见解。

关键词: 法医病理学, 人工智能, 多模态, 数据融合

CLC Number:

DF793.1
TP18

HUANG Ping. The Prospects of Multimodal Artificial Intelligence Identification in Forensic Pathology[J]. Chinese Journal of Forensic Sciences, 2024(6): 36-43.

黄平. 多模态法医病理学智能化鉴定的应用展望[J]. 中国司法鉴定, 2024(6): 36-43.

References

[ 1 ] GIUSEPPE L R，ANTONINA A，MASSIMO M，et al. Radiology in forensic medicine[M]. Berlin，Germany：Springer，2019：3-14.
[ 2 ] CASTIGLIONE V，MODENA M，AIMO A，et al. Molecular autopsy of sudden cardiac death in the genomics era[J]. Diagnostics（Basel），2021，11（8）：1378.
[ 3 ] DAWIDOWSKA J，KRZYŻANOWSKA M，MARKU-SZEWSKI M J，et al. The application of metabolomics in forensic science with focus on forensic toxicology and time-of-death estimation[J]. Metabolites，2021，11（12）：801.
[ 4 ] DUONG V A，PARK J M，LIM H J，et al. Proteomics in forensic analysis：Applications for human samples[J]. Applied Sciences，2021，11（8）：3393.
[ 5 ] HAAS C，NEUBAUER J，SALZMANN A P，et al. Forensic transcriptome analysis using massively parallel sequencing[J]. Forensic Science International：Genetics，2021，52：102486.
[ 6 ] ROS A C，BACCI S，LUNA A，et al. Forensic impact of the omics science involved in the wound：A systematic review[J]. Frontiers in Medicine，2022，6（8）：786798.
[ 7 ] SZEREMETA M，PIETROWSKA K，NIEMCUNOWI-CZ-JANICA A，et al. Applications of metabolomics in forensic toxicology and forensic medicine [J]. International Journal of Molecular Sciences，2021，22（6）：3010.
[ 8 ] ACOSTA J N，FALCONE G J，RAJPURKAR P，et al. Multimodal biomedical AI[J].Nature Medicine，2022，28（9）：1773-1784.
[ 9 ] GAO J，LI P，CHEN Z K，et al. A survey on deep learning for multimodal data fusion[J]. Neural Computation，2020，32（5）：829-864.
[10] BALTRUSAITIS T，AHUJA C，MORENCY L P. Multimodal machine learning：A survey and taxonomy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence，2019，41（2）：423-443.
[11] SHARMA K，GIANNAKOS M. Multimodal data capabilities for learning：What can multimodal data tell us about learning?[J]. British Journal of Educational Technology，2020，51（5）：1450-1484.
[12] BORNIK A, URSCHLER M, SCHMALSTIEG D, et al. Integrated computer-aided forensic case analysis, presentation, and documentation based on multimodal 3D data[J]. Forensic Science International, 2018, 287:12-24.
[13] STAHLSCHMIDT S R，ULFENBORG B，SYNNERG-REN J. Multimodal deep learning for biomedical data fusion：A review[J]. Briefings in Bioinformatics，2022，23（2）：bbab569.
[14] 黄平，张吉，邹冬华，等. 数字化与智能化技术在法医病理学鉴定中的研究现状与挑战[J]. 中国司法鉴定，2023（2）：35-47.
[15] 赵亮，马文静，赵旭舒，等. 人工智能技术在法医学中的应用研究进展[J]. 刑事技术，2024，49（2）：185-189.
[16] 刘蓬然，霍彤彤，陆林，等. 人工智能在医学中的应用现状与展望[J]. 中华医学杂志，2021，101（44）：3677-3683.
[17] GORE J C. Artificial intelligence in medical imaging[J]. Magnetic Resonance Imaging，2020，68：A1-A4.
[18] JIANG T，GRADUS J L，ROSELLINI A J. Supervised machine learning：A brief primer[J]. Behavior Therapy，2020，51（5）：675-687.
[19] SEN P C，HAJRA M，GHOSH M. Supervised classification algorithms in machine learning：A survey and review[C]//Proceedings of IEM Graph 2018. Berlin，Germany：Springer，2020：99-111.
[20] THURZO A，KOSNÁČOVÁ H S，KURILOVÁ V，et al. Use of advanced artificial intelligence in forensic medicine，forensic anthropology and clinical anatomy[J].Healthcare，2021，9（11）：1545.
[21] GARLAND J，ONDRUSCHKA B，STABLES S，et al. Identifying fatal head injuries on postmortem computed tomography using convolutional neural network/deep learning：A feasibility study[J]. Journal of Forensic Sciences，2020，65（6）：2019-2022.
[22] FANNI S C，FEBI M，AGHAKHANYAN G，et al. Natural language processing[C]//Proceedings of the Introduction to Artificial Intelligence. Berlin，Germany：Springer，2023：87-99.
[23] PAOLANTI M，FRONTONI E. Multidisciplinary pattern recognition applications：A review[J]. Computer Science Review，2020，37：100276.
[24] XU P，ZHU X，CLIFTON D A. Multimodal learning with transformers：A survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence，2023，45（10）：12113-12132.
[25] RAMESH A，PAVLOV M，GOH G，et al. Zero-shot text-to-image generation[EB/OL].[2024-05-02].https：//arxiv.org/pdf/2102.12092.
[26] NICHOL A，DHARIWAL P，RAMESH A，et al. GLIDE：Towards photorealistic image generation and editing with text-guided diffusion models[J]. International Conference on Machine Learning，2021，2112：10741.
[27] WANG J M，LI Z D，YING F，et al. Reconstruction of a real-world car-to-pedestrian collision using geomatics techniques and numerical simulations[J]. Journal of Forensic and Legal Medicine，2022，91：102433.
[28] BAYLY P V，ALSHAREEF A，KNUTSEN A K，et al. MR imaging of human brain mechanics in vivo：New measurements to facilitate the development of computational models of brain injury[J]. Annals of Biomedical Engineering，2021，49（10）：2677-2692.
[29] OURA P，JUNNO A，JUNNO J A. Deep learning in forensic gunshot wound interpretation—a proof-of-concept study[J]. International Journal of Legal Medicine，2021，135（5）：2101-2106.
[30] ZIMMERMANN N，SIEBERTH T，DOBAY A. Automated wound segmentation and classification of seven common injuries in forensic medicine[J]. Forensic Science，Medicine and Pathology，2024，20（2）：443-451.
[31] CHEN Y J. Current state and progress of research on forensic biomechanics in China[J]. Forensic Sciences Research，2021，6（1）：1-12.
[32] ALBEEDAN M，KOLIVAND H，HAMMADY R，et al. Seamless crime scene reconstruction in mixed reality for investigation training：A design and evaluation study[J]. IEEE Transactions on Learning Technologies，2024，17：856-873.
[33] VILLA C，LYNNERUP N，JACOBSEN C. A virtual，3D multimodal approach to victim and crime scene reconstruction[J]. Diagnostics，2023，13（17）：2764.
[34] LIU R N，GU Y X，SHEN M W，et al. Predicting postmortem interval based on microbial community sequences and machine learning algorithms[J]. Environmental Microbiology，2020，22（6）：2273-2291.
[35] MADEA B. Estimation of the time since death[M]. 4th ed. Boca Raton，USA：CRC Press，2023：187-281.
[36] ZISSLER A，STOIBER W，GEISSENBERGER J，et al. Influencing factors on postmortem protein degradation for PMI estimation：A systematic review[J]. Diagnostics，2021，11（7）：1146.
[37] GUEBELIN D L C，DOBAY A，EBERT L，et al. Correlation of age，sex and season with the state of human decomposition as quantified by postmortem computed tomography[J]. Forensic Science，Medicine and Pathology，2021，17（2）：185-191.
[38] BUONASERA T，EERKENS J，DE FLAMINGH A，et al. A comparison of proteomic，genomic，and osteological methods of archaeological sex estimation[J]. Scientific Reports，2020，10（1）：11897.
[39] LI Y，WANG J，LIANG W B，et al. CR-GAN：Automatic craniofacial reconstruction for personal identification[J]. Pattern Recognition，2022，124（1/2/3）：108400.
[40] REMYA S，DWAZ C G，PILLAI P G，et al. A novel framework for forensic bone age estimation using CNN architecture and multimodal fusion[C]//Proceedings of the 7th International Conference on Electronics，Materials Engineering & Nano-Technology （IEMENTech）. Washington D.C.，USA：IEEE Press，2023：1-6.
[41] TESI C，RICCI S，CREZZINI J，et al. Wounded to death. Holistic，multimodal reconstruction of the dynamics in a case of multiple perimortem cranial injuries from a medieval site in northern Italy[J]. Journal of Archaeological Science，2022，46：103643.
[42] BYAHATTI P，SHETTAR M S. Fusion strategies for multimodal biometric system using face and voice cues[C]// Proceedings of Computational Engineering and Material Science （ICCEMS-2020）.[s.l.]：IOP Publishing，2020：012031.
[43] BALA N，GUPTA R，KUMAR A. Multimodal biometric system based on fusion techniques：A review[J]. Information Security Journal：A Global Perspective，2022，31（3）：289-337.
[44] TALREJA V，VALENTI M C，NASRABADI N M. Deep hashing for secure multimodal biometrics[J]. IEEE Transactions on Information Forensics and Security，2020，16：1306-1321.
[45] RAJASEKAR V，SARACEVIC M，HASSABALLAH M，et al. Efficient multimodal biometric recognition for secure authentication based on deep learning approach[J]. International Journal on Artificial Intelligence Tools，2023，32（3）：2340017.
[46] SIDDIQUI M F H，JAVAID A Y. A multimodal facial emotion recognition framework through the fusion of speech with visible and infrared images[J]. Multimodal Technologies and Interaction，2020，4（3）：46.
[47] DERAKHSHAN A，MIKAEILI M，GEDEON T，et al. Identifying the optimal features in multimodal deception detection[J]. Multimodal Technologies and Interaction，2020，4（2）：1-14.
[48] BONCI A，FIORI S，HIGASHI H，et al. An introductory tutorial on brain-computer interfaces and their applications[J]. Electronics，2021，10（5）：560.
[49] BERISHA V，KRANTSEVICH C，HAHN P R，et al. Digital medicine and the curse of dimensionality[J]. NPJ Digital Medicine，2021，4（1）：1-8.
[50] LIANG P P，ZADEH A，MORENCY L P. Foundations and trends in multimodal machine learning：Principles，challenges，and open questions[J]. ACM Computing Surveys，2024，56（10）：1-42.
[51] ESTEVA M，XU W，SIMONE N，et al. Modeling data curation to scientific inquiry：A case study for multimodal data integration[C]//Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020. New York，USA：Association for Computing Machinery，2020：235-242.
[52] ONDRUSCHKA B，SEIFERT D，ROTERMUND N，et al. A critical review of “artificial intelligence in the practice of forensic medicine：A scoping review”[J]. International Journal of Legal Medicine，2024，138（4）：1667-1668.

[1]	FENG Hao, WU Hejian, CHEN Feng, LIU Jiaqi. Exploration of Multimodal Information Fusion in Traffic Accident Appraisal of Intelligent and Connected Vehicles [J]. Chinese Journal of Forensic Sciences, 2024, 0(5): 18-26.
[2]	WEI Mingfei, LÜ Xiaoyu, MA Xiuwei, SHI Junzhan, QIN Haojie, LI Fan. Research Progress on Time Series of Blood Stain [J]. Chinese Journal of Forensic Sciences, 2024, 0(5): 50-58.
[3]	JIA Yuqing, YAO Chengyu, MAO Ruiming, WANG Tianqi, ZHANG Mengzhou, LIU Jihui, ZHU Baoli, CAO Zhipeng. Advances in the Forensic Pathology of Traumatic Pulmonary Fat Embolism and Its Identification Application [J]. Chinese Journal of Forensic Sciences, 2024, 0(4): 49-53.
[4]	LI Na, ZHANG Shurui, SUN Junhong. Strategies Driven by Artificial Intelligence Technology for Time Problems in Forensic Pathology [J]. Chinese Journal of Forensic Sciences, 2024, 0(3): 41-48.
[5]	SUN Daming. The Function, Characteristics, and Refference of the British Coroner System [J]. Chinese Journal of Forensic Sciences, 2024, 0(2): 25-32.
[6]	LI Liliang. Current Status and Forensic Appraisal Progress of Sudden Unexplained Death in Individuals with Mental Disorders [J]. Chinese Journal of Forensic Sciences, 2024, 0(2): 47-54.
[7]	ZHANG Ping, XU Hui, GAO Jue, JIANG Shihao, ZHANG Zhixiang, SHI Xiaoling. Retrospective Analysis: The Medical Dispute Appraisal of 78 Cases of Sudden Cardiac Death [J]. Chinese Journal of Forensic Sciences, 2023, 0(5): 57-63.
[8]	HUANG Ping, ZHANG Ji, ZOU Donghua, LIU Ningguo, LI Zhengdong, WANG Jinming, QIN Zhiqiang, ZHANG Jianhua. Frontiers and Challenges for Digital and Intelligent Technology in Forensic Pathology [J]. Chinese Journal of Forensic Sciences, 2023, 0(2): 35-47.
[9]	YE Weiquan, WU Zhaobin, CAI Liuxin, YANG Bofan, QIAO Zhi. Analysis of Characteristics of 182 Death Cases in Traffic Accidents in Nansha District of Guangzhou [J]. Chinese Journal of Forensic Sciences, 2022, 0(6): 61-66.
[10]	CHEN Ji, ZHANG Kaiqiao, ZHANG Zhiwei, ZHOU Shiyi, TANG Wei, LI Rong, CHEN Feng. Forensic Pathological Appraisal of Death by Vomit Inhalation [J]. Chinese Journal of Forensic Sciences, 2022, 0(3): 44-50.
[11]	DONG Tingting, XIAO Zuorun, LIU Zengjia, WANG Yequan, ZHANG Guoan , CUI Wen. Preliminary Screening and Verification of Skeletal Muscle circRNA in PMI Esitimation [J]. Chinese Journal of Forensic Sciences, 2022, 0(1): 85-92.
[12]	WEI Haipeng. Research Progress on Forensic Application of Blister Skin Injury [J]. Chinese Journal of Forensic Sciences, 2022, 0(1): 60-65.
[13]	ZHOU Yuxiang. Adverse Effects of Propofol and Current Status of Forensic Pathology Studies [J]. Chinese Journal of Forensic Sciences, 2020, 0(6): 28-34.
[14]	LIU Liang-Cai, HUANG Xiao-Jun, CUI Yu-Bao. Analysis of Behavior Ability after Fatal Trauma in 3 Cases [J]. , 2014, 0(2): 56-57.
[15]	HU Dong-yang, SUN Yi, HUANG Cui, HUANG Lei, LU Si-ya, CHENG Jian-ding, TANG Shuang-bo. Forensic Analysis of 16 Cases of Sudden Death from Hypokalemic Periodic Paralysis [J]. Chinese Journal of Forensic Sciences, 2013, 0(4): 52-54.