Leveraging Large Language Models for Network Traffic Analysis : Design, Implementation, and Evaluation of an LLM-Powered System for Cyber Incident Reconstruction

Abstract

Cyberthreats are evolving, and becoming more sophisticated. Hence, there is a growing need for advanced security analysis tools. Traditional approaches for analyzing network traffic and reconstructing cyber incidents struggle to efficiently process and analyse large amounts of network data in real-time. The rapid advancement of Generative AI, particularly large language models (LLMs), opens up new possibilities for extending current digital forensics and incident response capabilities. LLMs can play an important role in cybersecurity, but their adoption has been limited due to reliance on commercial models, data privacy concerns, and a lack of domain-specific fine-tuning.

This thesis proposes a new type of framework that uses locally run open-source LLMs and vector databases to perform network traffic analytics to reconstruct a cyber incident and identify the attacker. The architecture of the proposed system includes five major components: (1) a data preprocessing pipeline that ingests the network packet capture (PCAP) files and extracts relevant features while also using external threat intelligence data source, such as "VirusTotal", to provide a context-aware response; (2) a vector database (ChromaDB) that stores the preprocessed data for fast similarity search and retrieval; (3) Locally run open-source LLMs (LLAMA, Falcon, and Mistral) which analyze the retrieved data and generate human-like responses to the security queries; and (4) a user interface that allows security analysts to interact with the system, to gather insights.

The proposed system was tested using real-world network traffic data and offers a practical framework for using advanced LLM in network security analysis while ensuring data privacy and system efficiency. The comparison of LLMs like LLAMA, Falcon, and Mistral with different parameter sizes (7B,13B) provides useful insights for selecting and optimizing models for real-world use. The results indicate that these open-source LLMs have great potential to provide accurate and smart responses to security queries. The results indicate that larger models (large parameters) generally perform better than smaller ones in accuracy. Llama7-13b stood out as the top performer across all metrics. Furthermore, designing a detailed form with sufficient information and context is important to guide the output of LLMs and obtain relevant insights.