LogBERT Detector

The LogBERT Detector is inspired by the LogBERT paper.

In/out

Input and output schemas in the pipeline

	Schema	Description
Input	ParserSchema	Structured log
Output	DetectorSchema	Combined alert / finding

At a glance

Learns from training data	Auto-configuration	Needs `events`	Federation
✅	✅ fine-tunes the model	❌	❌

Description

Deep learning method that looks at the event ID sequence.

Example

import yaml
from detectmatelibrary import schemas
from detectmatelibrary.detectors.logbert_detector import LogBertDetector

with open("docs/examples/detectors/logbert.yaml") as f:
    config = yaml.safe_load(f)
detector = LogBertDetector(name="LogBertDetector", config=config)


def event(event_id: int) -> schemas.ParserSchema:
    """A parsed log; only its EventID (the matched template) matters here."""
    return schemas.ParserSchema({"eventID": event_id})


# a repeating workflow of 4 events; the model trains once 60 windows are collected
for log in [event(i) for i in [0, 1, 2, 3] * 15]:
    detector.process(log)

# the same workflow, but with events 2 and 3 swapped in the middle
stream = [0, 1, 2, 3, 0, 1, 3, 2, 0, 1, 2, 3]
flagged = [i for i, e in enumerate(stream) if detector.process(event(e))]
# Positions of the flagged windows. On a stream this small LogBERT also flags
# some normal windows; in practice, train it on thousands of logs.
print(flagged)

Configuration file

The configuration used by the example above. It sets only what this use case needs; every other parameter keeps its default (see Configuration arguments).

detectors:
  LogBertDetector:
    method_type: logbert_detector
    params:
      data_use_training: 60     # train the model on the first 60 windows, then detect
      window_size: 4            # length of the EventID sequences the model learns

The same file works unchanged in both places a detector runs:

Library: load it with yaml.safe_load and pass the dict as config=, as in the example. The key under detectors: must match the detector's name.
DetectMateService: use it as the service's detector configuration.

Configuration arguments

All parameters this detector accepts, grouped by the YAML block they go in. Scope tells whether a parameter is specific to this detector or shared with other detectors (see the Detectors overview). Where this detector changes a shared default, the shared value is shown in brackets.

Top level

Field	Type	Default	Scope	Description
`method_type`	string	logbert_detector	shared	Indicates what type of method it is.
`auto_config`	boolean	True	shared	Runs the configuration step before the training process. Has no effect unless data_use_configure is set.
`events`	object	{}	shared	Events configuration dict keyed by event_id.
`global`	object	{}	shared	Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML.

params

Field	Type	Default	Scope	Description
`start_id`	integer	10	shared	Number used to start the unique ID generator.
`data_use_training`	integer, null	None	shared	Number of logs used for training. None skips training.
`data_use_configure`	integer, null	None	shared	Number of logs used for the configuration step. None skips it.
`use_config_data_as_training`	boolean	True	shared	Combine the configured data in the training process if True.
`train_buffer_max_records`	integer	100000	shared	Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records.
`train_buffer_dir`	string, null	None	shared	Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind.
`parser`	string	PARSER	shared	Name of the parser used.
`window_size`	integer	10	shared	Number of consecutive events used as one training/detection sequence.
`validation_per`	number	0.2	shared	Fraction of data held out for validation during (fine)training.
`finetune_epochs`	integer	2	shared	Number of epochs used when finetuning during the configuration phase.
`hyperparameters`	object	{'Model': {'n_embed': 10, 'hidden': 32, 'num_heads': 2, 'n_layers': 1, 'dropout': 0.0, 'max_len': 1000}, 'Train': {'seed': 0, 'batch_size': 256, 'learning_rate': 0.01, 'epochs': 10, 'mask_per': 0.4, 'alpha': 0.0, 'patience': 3}, 'Finetune': [['Model', 'hidden', [64, 128, 256]], ['Model', 'n_layers', [1, 2, 3]], ['Train', 'learning_rate', [0.002, 0.001, 0.005]]]}	shared	Model, training and hyperparameter-search settings for the LogBERT model.