SCVS Detector

The Sequence Count Vector Set Detector (SCVS) detects anomalies by finding count vectors that were not present in the training dataset.

In/out

Input and output schemas in the pipeline

	Schema	Description
Input	ParserSchema	Structured log
Output	DetectorSchema	Alert / finding

At a glance

Learns from training data	Auto-configuration	Needs `events`	Federation
✅	❌	❌	✅

Description

A count vector is formed by counting the number of appearances of each event ID in a sequence of a specific window size.

Count vectors learned during training are stored via persistency, so a trained model can be saved and restored with a persist: block. A count vector is only comparable within the window it was counted over, so restoring state at a different window_size logs a warning, because the restored vectors cannot match and every window would alert.

Example

import yaml
from detectmatelibrary import schemas
from detectmatelibrary.detectors.scvs_detector import SCVSDetector

with open("docs/examples/detectors/scvs_detector.yaml") as f:
    config = yaml.safe_load(f)
detector = SCVSDetector(name="SCVSDetector", config=config)


def event(event_id: int) -> schemas.ParserSchema:
    """A parsed log; only its EventID (the matched template) matters here."""
    return schemas.ParserSchema({"eventID": event_id})


# normal session: 0 = login accepted, 1 = session opened, 2 = session closed.
# The detector works on sliding windows of 3 logs (window_size: 3).
for log in [event(i) for i in [0, 1, 2] * 3]:
    detector.process(log)

# three logins in a row: a window whose EventID counts were never seen
alerts = [detector.process(event(0)) for _ in range(3)]
print([alert["score"] if alert else None for alert in alerts])  # [None, 1.0, 1.0]

Configuration file

The configuration used by the example above. It sets only what this use case needs; every other parameter keeps its default (see Configuration arguments).

detectors:
  SCVSDetector:
    method_type: scvs_detector
    params:
      data_use_training: 6      # learn count vectors from the first 6 windows, then detect
      window_size: 3            # count EventIDs over windows of 3 consecutive logs

The same file works unchanged in both places a detector runs:

Library: load it with yaml.safe_load and pass the dict as config=, as in the example. The key under detectors: must match the detector's name.
DetectMateService: use it as the service's detector configuration.

Configuration arguments

All parameters this detector accepts, grouped by the YAML block they go in. Scope tells whether a parameter is specific to this detector or shared with other detectors (see the Detectors overview). Where this detector changes a shared default, the shared value is shown in brackets.

Top level

Field	Type	Default	Scope	Description
`method_type`	string	scvs_detector	shared	Indicates what type of method it is.
`auto_config`	boolean	True	shared	Runs the configuration step before the training process. Has no effect unless data_use_configure is set.
`events`	object	{}	shared	Events configuration dict keyed by event_id.
`global`	object	{}	shared	Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML.
`persist`	object, null	None	shared	Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page.

params

Field	Type	Default	Scope	Description
`window_size`	integer	10	specific	Length of the event-ID window a count vector is built over.
`start_id`	integer	10	shared	Number used to start the unique ID generator.
`data_use_training`	integer, null	None	shared	Number of logs used for training. None skips training.
`data_use_configure`	integer, null	None	shared	Number of logs used for the configuration step. None skips it.
`use_config_data_as_training`	boolean	True	shared	Combine the configured data in the training process if True.
`train_buffer_max_records`	integer	100000	shared	Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records.
`train_buffer_dir`	string, null	None	shared	Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind.
`parser`	string	PARSER	shared	Name of the parser used.
`allow_fed`	boolean	False	shared	Builds the stores needed to take part in federation.