AUTHOR=Sammour Mahmoud , Othman Mohd Fairuz Iskandar , Hassan Aslinda , Bhais Omar , Talib Mohammed Saad 
  
TITLE=Advanced DNS tunneling detection: a hybrid reinforcement learning and metaheuristic approach
  
JOURNAL=Frontiers in Computer Science
  
VOLUME=Volume 7 - 2025
  
YEAR=2026
  
URL=https://www.frontiersin.org/journals/computer-science/articles/10.3389/fcomp.2025.1728980
  
DOI=10.3389/fcomp.2025.1728980
  
ISSN=2624-9898
  
ABSTRACT=IntroductionDNS tunneling remains a critical network threat, exploiting the inherent trust in the DNS protocol for unauthorized communication, data exfiltration, and firewall evasion.MethodsAddressing this challenge, this paper introduces a novel, hybrid feature selection framework that integrates the Random Forest classifier with an Enhanced Reinforcement Learning-Guided Grey Wolf Optimizer (EnhancedRLGWO). The EnhancedRLGWO employs a Dueling Deep Q-Network and strategic Opposition-Based Learning to intelligently navigate the feature space and identify an optimal, minimal subset.ResultsEvaluated against the benchmark CIRA-CIC-DoHBrw-2020 dataset, the proposed approach achieved a state-of-the-art accuracy of 99.82% and a weighted F1-score of 99.79% using a highly compact subset of only 12 features. This performance significantly outperforms existing machine learning-based DNS tunneling detection systems, such as a hybrid feature selection model achieving 98.3% accuracy and a full 28-feature Random Forest baseline (98.50% accuracy). The experimental results showed the robustness of this method in identifying various types of DNS tunneling attacks, including Iodine, DNS2TCP, and DNScat2, while maintaining performance and accuracy.