An Efficient system based on Artificial Intelligence for the Detection and Mitigation of network Intrusion using encrypted traffic protocols: A Systematic Approach

https://doi.org/10.5281/zenodo.17551668

Abstract

Intrusion detection is a crucial aspect of cybersecurity, as attackers increasingly exploit encrypted traffic to conceal their malicious activities. While encryption enhances privacy and confidentiality, it also limits the effectiveness of traditional Intrusion Detection Systems (IDS), which primarily rely on inspecting the payload. Yet, these cut-edge technologies come with daily disastrous, ever-increasing cyberattacks on sensitive data in the IoT-based environment. Hence, there is a continued need for groundbreaking strengths of AI-based models to develop and implement intrusion detection systems (IDSs) to arras and mitigate these ugly cyber-threats with IoT-based systems. Therefore, this chapter discusses the security issues within IoT-based environments and the application of AI models for security and privacy in IoT-based for a secure network. The article proposes a hybrid AI-model framework for intrusion detection in an IoT-based environment using CIC-IDS2017and UNSW-NB15 to test the proposed model's performance. The model performed better with an accuracy of 99.45%, with a detection rate of 99.75%. The results from the proposed model show that the classifier performs far better when compared with existing work using the same datasets, thus proving more effective in the classification of intruders and attackers on IoT-based systems. Artificial Intelligence (AI)-based approaches to intrusion detection in encrypted network traffic, utilizing Machine Learning (ML) and Deep Learning (DL) methods that learn behavioral and statistical patterns, rather than relying solely on packet contents. Models such as Support Vector Machines (SVM), Random Forest (RF), Convolutional Neural Networks (CNN), and Recurrent Neural Networks (RNN) are examined in the context using standard datasets, such as CICIDS2017, NSL-KDD, and UNSW-NB15. The article shows the the strengths of these models, with a particular focus on scalability challenges, false positive rates, and adaptability in real-time encrypted environments. Furthermore, the study identifies critical research gaps, including the scarcity of updated encrypted datasets, the need for explainable AI (XAI) to enhance trust and transparency, and the potential of hybrid detection frameworks that combine host- and network-level perspectives. With the rise of cloud services, Internet of Things (IoT) devices, and virtualization technologies, the cyberattack surface has expanded significantly, creating more risks for organizations Overall, this article consolidates current approaches, emphasizes open challenges, and outlines future directions for developing efficient, scalable, and intelligent IDS capable of securing modern encrypted networks.