ECVC Detector

The Event Count Vector Clustering Detector (ECVC) detects anomalies by calculating the distance between the count vectors from training and new ones. The method can be found in this publication.

In/out

Input and output schemas in the pipeline

	Schema	Description
Input	ParserSchema	Structured log
Output	DetectorSchema	Alert / finding

At a glance

Learns from training data	Auto-configuration	Needs `events`	Federation
✅	❌	❌	✅

Description

A count vector is formed by counting the number of appearances of each event ID in a sequence of a specific window size.

Count vectors learned during training are stored via persistency, so a trained model can be saved and restored with a persist: block. A count vector is only comparable within the window it was counted over, so restoring state at a different window_size logs a warning, because the restored vectors cannot match and every window would alert.

Example

import yaml
from detectmatelibrary import schemas
from detectmatelibrary.detectors.ecvc_detector import ECVCDetector

with open("docs/examples/detectors/ecvc_detector.yaml") as f:
    config = yaml.safe_load(f)
detector = ECVCDetector(name="ECVCDetector", config=config)


def event(event_id: int) -> schemas.ParserSchema:
    """A parsed log; only its EventID (the matched template) matters here."""
    return schemas.ParserSchema({"eventID": event_id})


# normal session: 0 = login accepted, 1 = session opened, 2 = session closed.
# The detector works on sliding windows of 3 logs (window_size: 3).
for log in [event(i) for i in [0, 1, 2] * 8]:
    detector.process(log)

# a burst of logins: windows whose EventID counts are far from anything learned
alerts = [detector.process(event(0)) for _ in range(4)]
print([alert["score"] if alert else None for alert in alerts])  # [None, 0.5, 0.8, 0.8]

Configuration file

The configuration used by the example above. It sets only what this use case needs; every other parameter keeps its default (see Configuration arguments).

detectors:
  ECVCDetector:
    method_type: ecvc_detector
    params:
      data_use_training: 20     # learn count vectors from the first 20 windows, then detect
      window_size: 3            # count EventIDs over windows of 3 consecutive logs

The same file works unchanged in both places a detector runs:

Library: load it with yaml.safe_load and pass the dict as config=, as in the example. The key under detectors: must match the detector's name.
DetectMateService: use it as the service's detector configuration.

Configuration arguments

All parameters this detector accepts, grouped by the YAML block they go in. Scope tells whether a parameter is specific to this detector or shared with other detectors (see the Detectors overview). Where this detector changes a shared default, the shared value is shown in brackets.

Top level

Field	Type	Default	Scope	Description
`method_type`	string	ecvc_detector	shared	Indicates what type of method it is.
`auto_config`	boolean	True	shared	Runs the configuration step before the training process. Has no effect unless data_use_configure is set.
`events`	object	{}	shared	Events configuration dict keyed by event_id.
`global`	object	{}	shared	Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML.
`persist`	object, null	None	shared	Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page.

params

Field	Type	Default	Scope	Description
`window_size`	integer	10	specific	Length of the event-ID window a count vector is built over.
`validation_per`	number	0.2	specific	Fraction of the learned count vectors held out as a validation split and used to derive the anomaly threshold.
`seed`	integer	0	specific	Random seed used to shuffle count vectors into train/validation splits.
`threshold_method`	string	mean	specific	Method used to derive the anomaly threshold from the validation split: 'mean' averages the distance scores, 'default' uses a fixed threshold of 0.
`start_id`	integer	10	shared	Number used to start the unique ID generator.
`data_use_training`	integer, null	None	shared	Number of logs used for training. None skips training.
`data_use_configure`	integer, null	None	shared	Number of logs used for the configuration step. None skips it.
`use_config_data_as_training`	boolean	True	shared	Combine the configured data in the training process if True.
`train_buffer_max_records`	integer	100000	shared	Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records.
`train_buffer_dir`	string, null	None	shared	Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind.
`parser`	string	PARSER	shared	Name of the parser used.
`allow_fed`	boolean	False	shared	Builds the stores needed to take part in federation.