LogBatcher Parser

LLM-based log parser that infers event templates from raw log messages using any OpenAI-compatible model. No training data or labeled examples are required.

Schema Description
Input LogSchema Raw log string
Output ParserSchema Structured log with template and variables

Installation

LogBatcherParser requires the llm optional extra:

pip install "detectmatelibrary[llm]"
# or with uv
uv sync --extra llm

Overview

LogBatcherParser wraps the LogBatcher engine (MIT, LogIntelligence 2024) as a CoreParser. Parsing proceeds in two phases:

  1. Cache lookup — the incoming log is matched against previously seen templates using a hash-based exact match followed by a tree-based similarity check. If a match is found, no LLM call is made.
  2. LLM query — on a cache miss, the log is submitted to the configured model. The returned template is stored in the cache for future reuse.

Variable slots in templates use the <*> wildcard notation (e.g. User <*> logged in from <*>). Extracted variables are written to output_["variables"] in order of appearance.

Configuration

Field Type Default Description
method_type string "logbatcher_parser" Parser type identifier
model string "gpt-4o-mini" Model name passed to the OpenAI-compatible endpoint
api_key string "" API key for the chosen provider
base_url string "" Base URL of the OpenAI-compatible endpoint. Leave empty to use the default OpenAI endpoint
batch_size int 10 Maximum number of logs submitted per LLM call

Example YAML fragment (OpenAI):

parsers:
  LogBatcherParser:
    method_type: logbatcher_parser
    params:
      model: "gpt-4o-mini"
      api_key: "<YOUR_API_KEY>"
      batch_size: 10

Example YAML fragment (local Ollama):

parsers:
  LogBatcherParser:
    method_type: logbatcher_parser
    params:
      model: "llama3"
      api_key: "ollama"
      base_url: "http://localhost:11434/v1"
      batch_size: 10

Usage examples

Basic usage — set up the parser (parsing requires a valid API key):

from detectmatelibrary.parsers.logbatcher import LogBatcherParser, LogBatcherParserConfig

# instantiate the parser (config as a typed config object)
# note: LogBatcher sends logs to an LLM, so a real api_key is required
#       to actually parse. Here we only build the parser.
config = LogBatcherParserConfig(
    api_key="<YOUR_API_KEY>",
    model="gpt-4o-mini",
    batch_size=10,
)

parser = LogBatcherParser(name="LogBatcherParser", config=config)

# to parse a log, provide a real api_key above and call:
#   output = schemas.ParserSchema()
#   parser.parse(input_log, output)
#   print(output["template"])   # e.g. "User <*> logged in from <*>"

Using a local Ollama instance:

from detectmatelibrary.parsers.logbatcher import LogBatcherParser, LogBatcherParserConfig

config = LogBatcherParserConfig(
    api_key="ollama",
    model="llama3",
    base_url="http://localhost:11434/v1",
    batch_size=10,
)
parser = LogBatcherParser(name="LogBatcherParser", config=config)
# with a running Ollama instance, parse a log like this:
#   input_log = schemas.LogSchema({
#       "logID": "1",
#       "log": "User admin logged in from 192.168.1.10",
#   })
#   output = schemas.ParserSchema()
#   parser.parse(input_log, output)
#   print(output["template"])   # e.g. "User <*> logged in from <*>"
#   print(output["variables"])  # e.g. ["admin", "192.168.1.10"]

Passing config as a dict:

from detectmatelibrary.parsers.logbatcher import LogBatcherParser

# same parser, but configured from a plain dict instead of a config object
cfg = {
    "parsers": {
        "LogBatcher": {
            "method_type": "logbatcher_parser",
            "params": {
                "api_key": "<YOUR_API_KEY>",
                "model": "gpt-4o-mini",
                "batch_size": 10,
            },
        }
    }
}

parser = LogBatcherParser(name="LogBatcher", config=cfg)

Go back to Index