Standardization of Dialect Comments in Social Networks in View of Sentiment Analysis : Case of Tunisian Dialect

With the growing access to the internet, the spoken Arabic dialect language becomes an informal languages written in social media. Most users post comments using their own dialect. This linguistic situation inhibits mutual understanding between internet users and makes difficult to use computational approaches since most Arabic resources are intended for the formal language: Modern Standard Arabic (MSA). In this paper, we present a pipeline to standardize the written texts in social networks by translating them to the MSA. We fine-tune at first an identification bert-based model to select Tunisian Dialect (TD) comments from MSA and other dialects. Then, the resulting comments are translated using a neural translation model. Each of these steps was evaluated on the same test corpus. In order to test the effectiveness of the approach, we compared two opinion analysis models, the first is intended for the Sentiment Analysis (SA) of dialect texts and the second is for the MSA texts. We concluded that through standardization we obtain the best score.