LogBatcher Parser

LLM-based log parser that infers event templates from raw log messages using any OpenAI-compatible model. No training data or labeled examples are required.

In/out

Input and output schemas in the pipeline

	Schema	Description
Input	LogSchema	Raw log string
Output	ParserSchema	Structured log with template and variables

Installation

LogBatcherParser requires the llm optional extra:

pip install "detectmatelibrary[llm]"
# or with uv
uv sync --extra llm

Overview

LogBatcherParser wraps the LogBatcher engine (MIT, LogIntelligence 2024) as a CoreParser. Parsing proceeds in two phases:

1. Cache lookup: the incoming log is matched against previously seen templates using a hash-based exact match followed by a tree-based similarity check. If a match is found, no LLM call is made.
2. LLM query: on a cache miss, the log is submitted to the configured model. The returned template is stored in the cache for future reuse.

Variable slots in templates use the <*> wildcard notation (e.g. User <*> logged in from <*>). Extracted variables are written to output_["variables"] in order of appearance.

Examples

Set up the parser (parsing requires a valid API key):

import yaml
from detectmatelibrary.parsers.logbatcher import LogBatcherParser

# LogBatcher sends logs to an LLM: put a real api_key in the YAML to parse.
# Without one, this only builds the parser.
with open("docs/examples/parsers/logbatcher_parser.yaml") as f:
    config = yaml.safe_load(f)
parser = LogBatcherParser(name="LogBatcherParser", config=config)

# with a real api_key:
#   output = parser.process(schemas.LogSchema({"logID": "1", "log": "User admin logged in from 192.168.1.10"}))
#   print(output["template"])   # e.g. "User <*> logged in from <*>"

Using a local Ollama instance:

from detectmatelibrary.parsers.logbatcher import LogBatcherParser, LogBatcherParserConfig  # noqa: E402

# a local Ollama instance speaks the same API; here the config is built in Python
config = LogBatcherParserConfig(
    api_key="ollama",
    model="llama3",
    base_url="http://localhost:11434/v1",
)
parser = LogBatcherParser(name="LogBatcherParser", config=config)
# with Ollama running, parse exactly as above:
#   print(output["template"])   # e.g. "User <*> logged in from <*>"
#   print(output["variables"])  # e.g. ["admin", "192.168.1.10"]

Configuration file

The configuration used by the first example above (the Ollama example builds its configuration in Python instead). It sets only what this use case needs; every other parameter keeps its default (see Configuration arguments).

parsers:
  LogBatcherParser:
    method_type: logbatcher_parser
    params:
      api_key: <YOUR_API_KEY>   # key for the LLM endpoint
      model: gpt-4o-mini

The same file works unchanged in both places a parser runs:

• Library: load it with yaml.safe_load and pass the dict as config=, as in the example. The key under parsers: must match the parser's name.
• DetectMateService: use it as the service's parser configuration.

Configuration arguments

All parameters this parser accepts, grouped by the YAML block they go in. Scope tells whether a parameter is specific to this parser or shared with other parsers (see the Parsers overview).

Top level

Field	Type	Default	Scope	Description
method_type	string	logbatcher_parser	shared	Indicates what type of method it is.
auto_config	boolean	False	shared	Runs the configuration step before the training process.

params

Field	Type	Default	Scope	Description
model	string	gpt-4o-mini	specific	fitting description yet to find
api_key	string		specific	fitting description yet to find
base_url	string		specific	fitting description yet to find
batch_size	integer	10	specific	fitting description yet to find
start_id	integer	10	shared	Number used to start the unique ID generator.
data_use_training	integer, null	None	shared	Number of logs used for training. None skips training.
data_use_configure	integer, null	None	shared	Number of logs used for the configuration step. None skips it.
use_config_data_as_training	boolean	True	shared	Combine the configured data in the training process if True.
train_buffer_max_records	integer	100000	shared	Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records.
train_buffer_dir	string, null	None	shared	Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind.
log_format	string, null	None	shared	fitting description yet to find
time_format	string, null	None	shared	fitting description yet to find