Comparative Analysis of Machine Learning Techniques for Network Intrusion Detection in Cyber Security with a Diverse Metric-Based Performance Assessment

Abstract

Network intrusion detection is one of the major recurring problems in today's cyber security landscape, with attackers continually finding new ways to mimic legitimate traffic with malicious activity. A comparative study of eight supervised machine learning algorithms, namely: Logistic Regression, Decision Tree, Random Forest, Gradient Boosting, Support Vector Machine (SVM), K-Nearest Neighbors (KNN), Naive Bayes and a Multi-Layer Perceptron (MLP) neural network, for evaluation using a synthetic network intrusion dataset that contains 300 labelled traffic samples belonging to five classes: Normal, Denial of Service (DoS), Probe, Remote-to-Local (R2L), User-to-Root (U2R). We do not use any single measure of accuracy, but rather a comprehensive assessment framework based on multiple measures of accuracy: accuracy, precision, recall, F1-score, Matthews Correlation Coefficient (MCC), specificity, false positive rate and Area Under the ROC Curve (AUC), which have been validated using stratified 5-fold cross validation. All classifiers perform in a very tight range, around 47% to 54% accuracy, with the MCC values around 0 and the AUC values around 0.48 – 0.52, as shown in our results. Overall these findings suggest that none of the algorithms successfully learned a discriminative signal to a level of significance above random guessing on this specific data set. A supporting correlation analysis verifies that the numeric traffic features (duration, source bytes, destination bytes, failed logins and packet count) have near zero linear inter-correlations with the intrusion label, and this is why the models perform uniformly badly across all families. We believe this is a valuable result as it shows why it is unwise to judge an intrusion detection system by its accuracy alone: several of the models studied in this paper achieved an accuracy above 50% and their MCC and AUC values indicate that they are performing no better than random guessing. Finally, the paper provides some practical considerations for dataset design, feature engineering, and evaluation methodology in the context of applied problem solving in ID research, and gives concrete suggestions for future research with a larger, richer, and more realistically correlated network traffic datasets like NSL-KDD, CICIDS2017, or UNSW-NB15.