DeepLog Detector

The DeepLog Detector is inspired by the DeepLog paper.

In/out

Input and output schemas in the pipeline

	Schema	Description
Input	ParserSchema	Structured log
Output	DetectorSchema	Combined alert / finding

At a glance

Learns from training data	Auto-configuration	Needs events	Federation
✅	✅ fine-tunes the model	❌	❌

Description

Deep learning method that looks at the event ID sequence.

Example

import yaml
from detectmatelibrary import schemas
from detectmatelibrary.detectors.deeplog_detector import DeeplogDetector

with open("docs/examples/detectors/deeplog.yaml") as f:
    config = yaml.safe_load(f)
detector = DeeplogDetector(name="DeeplogDetector", config=config)


def event(event_id: int) -> schemas.ParserSchema:
    """A parsed log; only its EventID (the matched template) matters here."""
    return schemas.ParserSchema({"eventID": event_id})


# a repeating workflow of 4 events; the model trains once 60 windows are collected
for log in [event(i) for i in [0, 1, 2, 3] * 15]:
    detector.process(log)

# the same workflow, but with events 2 and 3 swapped in the middle
stream = [0, 1, 2, 3, 0, 1, 3, 2, 0, 1, 2, 3]
flagged = [i for i, e in enumerate(stream) if detector.process(event(e))]
print(flagged)  # [6, 7]: the windows ending on the swapped events

Configuration file

The configuration used by the example above. It sets only what this use case needs; every other parameter keeps its default (see Configuration arguments).

detectors:
  DeeplogDetector:
    method_type: deeplog_detector
    params:
      data_use_training: 60     # train the model on the first 60 windows, then detect
      window_size: 4            # length of the EventID sequences the model learns

The same file works unchanged in both places a detector runs:

• Library: load it with yaml.safe_load and pass the dict as config=, as in the example. The key under detectors: must match the detector's name.
• DetectMateService: use it as the service's detector configuration.

Configuration arguments

All parameters this detector accepts, grouped by the YAML block they go in. Scope tells whether a parameter is specific to this detector or shared with other detectors (see the Detectors overview). Where this detector changes a shared default, the shared value is shown in brackets.

Top level

Field	Type	Default	Scope	Description
method_type	string	deeplog_detector	shared	Indicates what type of method it is.
auto_config	boolean	True	shared	Runs the configuration step before the training process. Has no effect unless data_use_configure is set.
events	object	{}	shared	Events configuration dict keyed by event_id.
global	object	{}	shared	Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as global in YAML.

params

Field	Type	Default	Scope	Description
start_id	integer	10	shared	Number used to start the unique ID generator.
data_use_training	integer, null	None	shared	Number of logs used for training. None skips training.
data_use_configure	integer, null	None	shared	Number of logs used for the configuration step. None skips it.
use_config_data_as_training	boolean	True	shared	Combine the configured data in the training process if True.
train_buffer_max_records	integer	100000	shared	Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records.
train_buffer_dir	string, null	None	shared	Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind.
parser	string	PARSER	shared	Name of the parser used.
window_size	integer	10	shared	Number of consecutive events used as one training/detection sequence.
validation_per	number	0.2	shared	Fraction of data held out for validation during (fine)training.
finetune_epochs	integer	2	shared	Number of epochs used when finetuning during the configuration phase.
hyperparameters	object	{'Model': {'hidden_dim': 64, 'n_layers': 2}, 'Train': {'seed': 0, 'batch_size': 2048, 'learning_rate': 0.01, 'epochs': 10, 'patience': 3}, 'Finetune': [['Model', 'hidden_dim', [128, 256, 512]], ['Model', 'n_layers', [1, 2, 3]], ['Train', 'learning_rate', [0.01, 0.02, 0.03]]]}	shared	Model, training and hyperparameter-search settings for the DeepLog model.