Low-resource YouTube comment encoding for Luganda sentiment classification performance

2-1 Determining Luganda YouTube comment collection

There exists many YouTube music channels for/from Ugandan audiences. These channels tend to have generally positive emotion or opinions and our study requires a scope, which is representative of both positive (pos), negative (neg) as well as neutral (neu) comments. Similarly, Ugandan artist channels in existence largely suffer from the same scenario above. This is so because the channels tend to align more to the artist’s supporters or fun base. Others like disco jokers (DJs) and music promoter channels etc. we find that generally comments there also tend to be positively inclined. In view of the above, we propose Ugandan channels criticizing artists' music videos. These tend to carry relatively well-balanced emotionally oriented comments appropriate for this study.

Our desire is to harness as many Luganda comments as possible but we have to remember that comments from these channels are mixed i.e. English and Luganda in one sentence and sometimes with possibility of finding additional local languages in use too. In such circumstances, there is need to come up with assumptions for sorting or filtering and finally selecting Luganda comments to building our dataset for sentiment analysis modelling.

Assumptions for Luganda comments collection

• ■ Identify appropriate YouTube channel criticizing artists’ music videos.
• ■ Identify if a channel’s given video has more than 5,000 views with at least 5 comments.
• ■ Use channel YouTube video link to download all comments.
• ■ Remove wholly English sentence comments since our target is Luganda comments.
• ■ Keep a mixed sentence comment if it has more than 2 Luganda words.

2-2 Luganda YouTube dataset

We follow the proposed guidelines to identify our channel for target Luganda comments, and we identified a TV show called “Akasengejja” broadcasted on Bukedde TV a local channel in Uganda [15]. The station broadcasts mostly in Luganda and the TV show chosen for our comment gathering focuses on assessing Ugandan artists' music videos where later the aired show is posted on their Akasengejja YouTube channel. Using a YouTube comment scraper [16], which is based on Google YouTube API [17], we download 294 mixed comments using 18 YouTube video links from our chosen channel satisfying our assumptions. Fig 1. shows an example of Luganda comments in YouTube. After, we manually remove wholly English comments and finally left with 158 mostly Luganda comments as per set guidelines. Since one of the authors is a native Luganda speaker, to reduce subjectivity during comment label assignment or annotation, we adopt one additional native through online collaboration and together we assign each of the 158 Luganda comments with a general sentiment label. A comment is read and determination is made as to whether it’s a positive (1), negative (-1) or neural (0). The Table 1, shows the total number of comments after general sentiment assignment (label) in our Luganda dataset and in Table 2, we also show a sample of 3 comments from our Luganda dataset.

Fig. 1.

Some example of Luganda YouTube Comment

Table 1.

Total number of Luganda dataset comments

Table 2.

Some example of labelled Luganda YouTube comments

2-3 Proposed token processing and normalization

Our aim is to utilize the collected Luganda comments in order to determine the most appropriate technique(s) to build a sentiment model while working in a low-resource setting. Firstly, Luganda comment data under goes through various preprocessing to make it fit for intended training experiments at each stage. For example, we tokenize and lowercase all comments, remove all punctuation marks and non-alphabetic characters. After this process we are left with 964 clean tokens and 641 unique vocabulary. During cleaning we do not use any stopword list since at the time of this writing, no known agreed upon list for Luganda language could be found.

Secondly, we normalize tokens in aspects of artists’ most common name in comments, repeating words like “hahahahaha” are shortened to common phrase, names misspelt are also normalized to one common name style in comments. After normalization process, 961 clean tokens and 622 unique vocabulary is left. Table 3 shows some example of comment token normalization.

Table 3.

Some example of extracted Token normalization

Thirdly, because sentiment analysis cannot be conducted directly on text tokens using machine learning or deep learning techniques, we transform our comments using selected text encoding techniques based on Bag of Word (BoW) model to enable comparison analysis. The main text encoding used include, binary, count, tf-idf and frequency representation or feature extraction methods.

2-4 Proposed transformations with comment segmentation (GandaKIT)

In a recent machine translation (MT) research for English to Luganda, the authors designed GandaKIT as a tool for morphological segmentation of Luganda sentences [18]. As a result of their tool, they demonstrated improved performance in MT output using Statistical Machine Translation with Moses toolkit.

In similar way, we propose to adopt GandaKIT segmentation function to our normalized Luganda YouTube comments for further comparison of comment sentiment analyses. We believe the additional process here too, is capable of improving our classifier training models and subsequent prediction on tests.

3-1 Dataset

As in Table 1, the dataset is composed of 158 labeled Luganda comments with positive (1), negative (-1) and neural (0) labels. We conduct experiments using normalized clean comments after all preprocessing steps described before.

3-2 Classifier modelling

■ Machine Learning Model (MLM): From sklearn machine learning library, we utilize Gaussian Naive Bayes (GNB), LinearSVC from Support Vector Machines (SVM) and Random Forest (RFC) classifiers to train encoded vector comments. Then we evaluate each classifier model’s prediction accuracy for test set based on comparison between text encoding technique used (i.e. binary, count, tf-idf, and frequency). The chosen classifiers inherently can support multi-class classification which is the case for our sentiment problem at hand.
■ Neural Network Model (NNM): From Keras deep learning library, we utilize Multilayer Perceptron (MLP) for multi-class softmax classification based on sequential model. The input layer for this model is guided by the number of words (622 normalized comments), one hidden layer shaped at 100 with 'relu' as activation function and final output layer shaped at 3 because of 3 classes in our sentiment prediction with 'softmax' as activation function. We compile our NNM model with 'categorical_crossentropy' as loss function, optimized with 'adagrad' and 'accuracy' for generating metric scores. The model is also trained at 20 epoch and evaluated for test set split at 10%, 20%, and 30% for each encoding. The whole process is automatically repeated 10 times where we report the average scores for each encoding method.

4-1 MLM Results with cross validation

We use 10 fold cross validation and report mean accuracy scores for each machine learning classifier based on encoding technique used. In Table 4, evaluation results based on test set splitting at 10%, 20% and 30% with cross validation is presented.

Table 4.

MLM Classifier mean score (%) per encoding on Test set at 10%, 20% and 30% with CV

At a glance, we can notice that at 10% test set split, the GNB classifier performed generally better and thus leading in the 4 encoding methods. This could probably be that GNB benefits from having more training data (90%). At 20% split, we can see a general reduction in GNB classifier's sentiment prediction performance though at 30% split it tries to gain a little ground. SVM notably gains ground in test set splitting at 20% and 30% exhibiting its highest score of 50.8 using frequency encoding. For the RFC classifier, it scored better at 20% test set split with encoding techniques count (42.5) and tf-idf (47.5) taking lead but registering poor results for 10% and 30% test set splits. We can notice some general performance instability in classifier scores when we increase test set split to 20% and 30%. This behaviour is expected for low-resource environment like ours because of decrease in number of training samples as more samples are availed for the test set.

In Table 5, MLM classifier with comment segmentation shows a consistent improvement in performance especially for Test split at 10%. Under this, the GNB classifier reported a 5.0 overall best increase in results across the four text encoding methods.

Table 5.

MLM Classifier mean score (%) per encoding on Test set at 10%, 20% and 30% with CV with comment segmentation

4-2 Neural Network Model (NNM) Results

Having trained our Multilayer Perceptron (MLP) based on sequential model, Table 6 shows results for each encoding technique evaluated for test set split at 10%, 20%, and 30%.

Table 6.

MLP Classifier mean score (%) per encoding on Test set at 10%, 20% and 30% without cross validation

From Table 5 our neural nework MLP classifier model on test set split at 10% scored highest (average score 66.3 for 'Tf-Idf' text encoding) followed by 59.4 for 'Count' text encoding method. However, classifier performance scores drop for test set split at 20% registering 56.9 as the highest average score based on 'Count' text encoding. The same trend is observed for test set split at 30% though the average scores drop below the 50% mark for all text encoding techniques. Similar to the MLM classifier, our MLP of NNM also suffers in performance as more samples are retained for the test set at 20% and 30% split. In addition it is known that deep learning algorithms heavily require an appropriate volume of training data in order to predict well on a test set and therefore a reduction in training data in this case does not benefit the model further at all.

After training the MLP Classifier model on the same dataset with comments segmented using the GandaKIT tool [17], the model results in Table 7 show an improvement in general classification performance especially for Test set split at 10%. Here Tf-Idf scored 68.8, representing a 2.5 increase in comparison to Table 6 results without comment segmentation. The reported increment in performance in segmented models is because the segmentation process assists to increase Luganda word token count which contributes to more availed data for model training thus better performance.

Table 7.

MLP Classifier mean score (%) per encoding on Test set at 10%, 20% and 30% with comment segmentation without cross validation