Multi-Class Malware Detection

Using the ranking algorithm, we now have a set of APIs that we know are unique to benign apps. We wanted to see the effect on the classification output of adding these known benign APIs to malware apps. From the graph, we can see that for the first 100 odd APIs we were barely seeing any significant results, but after that we see a large drop in the p-value. A possible reason for that could be that these benign APIs need to co-occur with others to classify an app as benign.

Looking at the classification report, we see that the F1 score of Adware has actually increased, unlike all the other categories. The main reason for this is that adware and benign apps are similar. Analysing the SVM weights and tSNE will help improve our understanding of the same.

We wanted to closely analyse the support vectors and SVM weights in our model. We are using scikit learn Linear Support Vector Classifier as our model. Additionally, since we have five unique classes, we will have 5 decision boundaries. Each decision boundary is composed of both negative and positive weights. The positive weights will correspond to the class of category, for instance, Benign, while the negative weights will correspond to all other classes in training. The two images on the right show the top 5% of positive weights in each SVM. These weights are ordered in descending order with the top of the columns corresponding to the highest weights. These weights will have the most influence on the positive decision that pulls an app towards that particular category. There are 55 vectors (or apps) shown for each category and these apps are responsible for approximately 30-45% of the sum of the total positive weights for their respective category.

As shown, we see that AA^T has significantly less incorrect app categories misplaced into the correct corresponding category as opposed to APA^T. This is one important reason why AAt continuously performs better than APA^T. This is likely due to the influence of incorrect categories of apps in the positive weights of the SVM. We will further be able to observe the reason for this occurrence by examining tSNE.

tSNE is a method used to visualize high dimensional data in low dimensions. In tSNE plots, data points that are more clustered together are more similar than those that are not. We created a plot for each kernel. The x and y axes refer to tSNE dimensions 1 and 2, while the data points represent individual apps in our training dataset.

For the tSNE plot of the AA^T kernel, you can see that benign and ransomware apps are most cluttered within their own categories, while there is a big cluster of adware, trojan, and backdoor apps in the middle of the plot. This means that the AA-transpose kernel perceives those adware, trojan, and backdoor apps to be similar. We investigated this and found that the trojan apps in that cluster contain a lot of APIs that are rare to trojans but really common to adware apps. The backdoor apps located there not only contain common adware apis but also contain common trojan apis. This is a possible reason why the AA-transpose kernel perceives those adware, trojan and backdoor apps to be similar to each other.

A common technique used is to analyse the SHAP values for a given classification. These values leverage the idea of shapley values for model feature influence scoring by calculating the average marginal contribution of feature values over all possible coalitions; because of this, these values are incredibly expensive and time consuming to compute. Thus, we will have used Local Interpretable Model agnostic Explanations or LIME to find similar values at the expense of some accuracy. LIME provides a local subset of SHAP which will allow us to understand the important features when classifying an app.

The bar on the left of the scene depicts the proper classification of an app given by the model. In this case, the number 4 corresponds to the category Ransomware. The Horizontal bar graph shows various app indexes and their respective feature importance in the classification of that app. This app was misclassified as Ransomware but was actually Trojan.