Template Tree Matcher

The Template Tree Matcher is a parser that matches incoming logs against a set of templates using a tree-based depth-first search. It provides faster matching for batch workloads compared with the streaming Template Matcher.

This parser wraps functionality from the DetectMatePerformance project: https://github.com/ait-detectmate/DetectMatePerformance. Prefer the performance implementation when parsing many log lines in non-stream (batch) mode.

In/out

Input and output schemas in the pipeline

	Schema	Description
Input	LogSchema	Unstructured log
Output	ParserSchema	Structured log

Not yet stable

This parser is not yet in a stable release and may behave differently across platforms or hardware.

Template format

• Templates are plain text lines in a template file.
• Use the token <*> to mark wildcard slots.

Example template file (templates.txt):

pid=<*> uid=<*> auid=<*> ses=<*> msg='op=PAM:<*> acct=<*>
login success: user=<*> source=<*>

Example

import yaml
from detectmatelibrary.parsers.tree_matcher import TemplateCppTreeMatcher
from detectmatelibrary import schemas

with open("docs/examples/parsers/template_tree_matcher.yaml") as f:
    config = yaml.safe_load(f)
parser = TemplateCppTreeMatcher(name="TreeMatcher", config=config)

parsed = parser.process(schemas.LogSchema({
    "logID": "0",
    "log": "pid=9699 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting acct=\"root\"'",
}))
print(parsed.eventID)    # 0: the first template in the file
print(parsed.template)   # pid <*> uid <*> auid <*> ses <*> msg op PAM <*> acct <*>
print(parsed.variables)  # ['9699', '0', '4294967295', '4294967295', 'accounting', 'root']

Configuration file

The configuration used by the example above. It sets only what this use case needs; every other parameter keeps its default (see Configuration arguments).

parsers:
  TreeMatcher:
    method_type: tree_matcher
    params:
      path_templates: tests/test_data/test_templates.txt   # one template per line

The same file works unchanged in both places a parser runs:

• Library: load it with yaml.safe_load and pass the dict as config=, as in the example. The key under parsers: must match the parser's name.
• DetectMateService: use it as the service's parser configuration.

Configuration arguments

All parameters this parser accepts, grouped by the YAML block they go in. Scope tells whether a parameter is specific to this parser or shared with other parsers (see the Parsers overview).

Top level

Field	Type	Default	Scope	Description
method_type	string	tree_matcher	shared	Indicates what type of method it is.
auto_config	boolean	False	shared	Runs the configuration step before the training process.

params

Field	Type	Default	Scope	Description
path_templates	string, null	None	specific	fitting description yet to find
start_id	integer	10	shared	Number used to start the unique ID generator.
data_use_training	integer, null	None	shared	Number of logs used for training. None skips training.
data_use_configure	integer, null	None	shared	Number of logs used for the configuration step. None skips it.
use_config_data_as_training	boolean	True	shared	Combine the configured data in the training process if True.
train_buffer_max_records	integer	100000	shared	Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records.
train_buffer_dir	string, null	None	shared	Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind.
log_format	string, null	None	shared	fitting description yet to find
time_format	string, null	None	shared	fitting description yet to find

Note

This matcher removes non-alphanumeric characters from logs and templates before matching, except for the <*> token. Ensure your templates are compatible with that normalization.