Optimizing Log Parsing: Template Refinement and Semantic Generalization

Abstract

Logs are a primary source of runtime information in modern software systems and are fun- damental to tasks such as anomaly detection, failure diagnosis, performance monitoring, and reliability engineering. Log parsing, the process of converting unstructured log messages into structured templates, serves as the foundation of these downstream analyses. To assist in this, many novel approaches have been proposed. This dissertation contains three sections: 1) an empirical study of existing log parsers, 2) SynLog+, and 3) GeLT. In the empirical study, we evaluate syntax-based, semantic-based, and hybrid log parses. The observations from the study indicates two issues. Firstly, there is a gap between the pars- ing accuracy of the syntax- and semantic-based parsers that hybrid parsers aim to bridge. However, semantic-based and hybrid parsers leverage PLMs or LLMs, requiring heavy run- time costs. Secondly, semantic-based parsers lack in generalization ability over unseen logs. With SynLog+, we propose a wrapper to existing log parsers that acts as a template refinement module aiming to improve the parsing accuracy without added runtime cost. With GeLT, we aim to re-evaluate the existing generalizable log parser, Log3T, and identify its design flaws and propose an optimized generalizable semantic-based log parser.