Multimodal packaging waste brand identification approach for extended producer responsibility traceability

Extended Producer Responsibility (EPR) policies in packaging wastes are challenging due to waste traceability in their post-consumer stage. Tracking packages after disposal involves identifying their producers under extreme conditions. Several Computer Vision (CV) approaches for waste material recognition have been successfully tested. However, the identification of waste producers remains unexplored mainly due to difficult conditions for brand recognition and the requirement of large datasets that vary from place to place and over time. We propose a multimodal approach for waste brand identification that utilizes only one "real" image per product for each brand, achieving a macro F1-score of 0.75 with 23 brands and 38 products. The approach leverages package texts and visual features extracted with pre-trained models and predicts the brand using a KNN model with a custom distance based on the Levenshtein distance. Our method employs data augmentation and random word sampling to create synthetic samples from each product image. The KNN model uses random words and a vector of visual features extracted with a previously trained CNN model for prediction. During prediction, the distance of the K nearest neighbors is computed as the weighted sum of the L 2 visual features distance and the sum of the minimum words Levenshtein distances. This study demonstrates the feasibility of brand identification on packaging waste for EPR traceability without the burden of large dataset acquisition.