Getting Meta: A Multimodal Approach for Detecting Unsafe Conversations within Instagram Direct Messages of Youth

被引：7
作者：
Ali S. [1 ]
Razi A. [2 ]
Kim S. [3 ]
Alsoubai A. [4 ]
Ling C. [1 ]
De Choudhury M. [3 ]
Wisniewski P.J. [4 ]
Stringhini G. [1 ]
机构：
[1] Boston University, Boston, MA
[2] Drexel University, Philadelphia, PA
[3] Georgia Institute of Technology, Atlanta, GA
[4] Vanderbilt University, Nashville, TN
来源：
Proceedings of the ACM on Human-Computer Interaction | 2023年 / 7卷 / CSCW1期
基金：
美国国家科学基金会;
关键词：
end-to-end encryption; ensemble models; instagram; machine learning; online risk detection; social media;
D O I：
10.1145/3579608
中图分类号：
学科分类号：
摘要：
Instagram, one of the most popular social media platforms among youth, has recently come under scrutiny for potentially being harmful to the safety and well-being of our younger generations. Automated approaches for risk detection may be one way to help mitigate some of these risks if such algorithms are both accurate and contextual to the types of online harms youth face on social media platforms. However, the imminent switch by Instagram to end-to-end encryption for private conversations will limit the type of data that will be available to the platform to detect and mitigate such risks. In this paper, we investigate which indicators are most helpful in automatically detecting risk in Instagram private conversations, with an eye on high-level metadata, which will still be available in the scenario of end-to-end encryption. Toward this end, we collected Instagram data from 172 youth (ages 13-21) and asked them to identify private message conversations that made them feel uncomfortable or unsafe. Our participants risk-flagged 28,725 conversations that contained 4,181,970 direct messages, including textual posts and images. Based on this rich and multimodal dataset, we tested multiple feature sets (metadata, linguistic cues, and image features) and trained classifiers to detect risky conversations. Overall, we found that the metadata features (e.g., conversation length, a proxy for participant engagement) were the best predictors of risky conversations. However, for distinguishing between risk types, the different linguistic and media cues were the best predictors. Based on our findings, we provide design implications for AI risk detection systems in the presence of end-to-end encryption. More broadly, our work contributes to the literature on adolescent online safety by moving toward more robust solutions for risk detection that directly takes into account the lived risk experiences of youth. © 2023 ACM.
引用
收藏
相关论文
共 134 条
[11]  
Baron J., The key to gen Z is video content, Forbes, (2019)
[12]  
Barrett-Maitland N., Lynch J., Social media, ethics and the privacy paradox, Security and privacy from a legal, ethical, and technical perspective, (2020)
[13]  
Bond S., Allyn B., Facebook whistleblower tells Congress products hurt kids and weaken democracy NPR, (2021)
[14]  
Buck T., Updates to end-to-end encrypted chats on Messenger, Meta, (2022)
[15]  
Caddle X.V., Razi A., Kim S., Ali S., Popo T., Stringhini G., De Choudhury M., Wisniewski P.J., MOSafely: Building an Open-Source HCAI Community to Make the Internet a Safer Place for Youth, pp. 315-318, (2021)
[16]  
Isabel Canhoto A., Padmanabhan Y., We (don't) know how you feel'-a comparative study of automated vs. manual analysis of social media conversations, Journal of Marketing Management, 31, 9-10, pp. 1141-1157, (2015)
[17]  
Cecillon N., Labatut V., Dufour R., Linares G., Abusive language detection in online conversations by combining content-and graph-based features, Frontiers in big Data, 2, (2019)
[18]  
Chatzakou D., Leontiadis I., Blackburn J., De Cristofaro E., Stringhini G., Vakali A., Kourtellis N., Detecting Cyberbullying and Cyberaggression in Social Media, ACM Transactions on the Web (TWEB), 13, pp. 1-51, (2019)
[19]  
Chavan V.S., Shylaja S.S., Machine learning approach for detection of cyber-aggressive comments by peers on social media network, pp. 2354-2358, (2015)
[20]  
Chen Y., Hsieh S., An Analysis of Multimodal Document Intent in Instagram Posts, (2020)
12345678910