New Event Detector

The New Event Detector raises alerts when previously unseen log templates, distinguished by event IDs, appear in log data. It is useful to detect unexpected types of events in the environment.

In/out

Input and output schemas in the pipeline

	Schema	Description
Input	ParserSchema	Structured log
Output	DetectorSchema	Alert / finding

At a glance

Learns from training data	Auto-configuration	Needs `events`	Federation
✅	❌	❌	✅

Description

This detector keeps the set of event IDs observed during training and emits an alert whenever an event ID that is not in the set appears. Alerts don't add the event ID to the set, so the same unseen event ID raises an alert every time it appears.

Example

import yaml
from detectmatelibrary import schemas
from detectmatelibrary.detectors.new_event_detector import NewEventDetector

with open("docs/examples/detectors/new_event.yaml") as f:
    config = yaml.safe_load(f)
detector = NewEventDetector(name="NewEventDetector", config=config)


def event(event_id: int) -> schemas.ParserSchema:
    """A parsed log; only its EventID (the matched template) matters here."""
    return schemas.ParserSchema({"eventID": event_id})


# the first 3 logs train the detector (data_use_training: 3):
# EventID 0 = "Accepted password ...", EventID 1 = "Failed password ..."
for log in [event(0), event(1), event(0)]:
    detector.process(log)

print(detector.process(event(1)))  # None: known event
alert = detector.process(event(2))  # a template never seen during training
print(dict(alert["alertDetails"]))  # {'EventID 2 - {}': "Unknown event ID: '2'"}

Configuration file

The configuration used by the example above. It sets only what this use case needs; every other parameter keeps its default (see Configuration arguments).

detectors:
  NewEventDetector:
    method_type: new_event_detector
    params:
      data_use_training: 3      # learn the known EventIDs from the first 3 logs, then detect

The same file works unchanged in both places a detector runs:

Library: load it with yaml.safe_load and pass the dict as config=, as in the example. The key under detectors: must match the detector's name.
DetectMateService: use it as the service's detector configuration.

Configuration arguments

All parameters this detector accepts, grouped by the YAML block they go in. Scope tells whether a parameter is specific to this detector or shared with other detectors (see the Detectors overview). Where this detector changes a shared default, the shared value is shown in brackets.

Top level

Field	Type	Default	Scope	Description
`method_type`	string	new_event_detector	shared	Indicates what type of method it is.
`auto_config`	boolean	True	shared	Runs the configuration step before the training process. Has no effect unless data_use_configure is set.
`events`	object	{}	shared	Events configuration dict keyed by event_id.
`global`	object	{}	shared	Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML.
`persist`	object, null	None	shared	Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page.

params

Field	Type	Default	Scope	Description
`start_id`	integer	10	shared	Number used to start the unique ID generator.
`data_use_training`	integer, null	None	shared	Number of logs used for training. None skips training.
`data_use_configure`	integer, null	None	shared	Number of logs used for the configuration step. None skips it.
`use_config_data_as_training`	boolean	True	shared	Combine the configured data in the training process if True.
`train_buffer_max_records`	integer	100000	shared	Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records.
`train_buffer_dir`	string, null	None	shared	Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind.
`parser`	string	PARSER	shared	Name of the parser used.
`allow_fed`	boolean	False	shared	Builds the stores needed to take part in federation.