Value Range Detector

The Value Range Detector raises alerts when numerical values outside of known ranges appear in configured fields. It is useful to detect unexpected changes in quantities such as sizes, counts, or durations.

In/out

Input and output schemas in the pipeline

	Schema	Description
Input	ParserSchema	Structured log
Output	DetectorSchema	Alert / finding

At a glance

Learns from training data	Auto-configuration	Needs `events`	Federation
✅	✅ picks the variables to monitor	✅ unless `auto_config: true`	✅

Description

This detector keeps only the minimum and maximum value seen during training for each monitored field, so its memory use stays constant however many values it sees. It emits an alert when a value falls outside that range (subject to configuration).

Example

import yaml
from detectmatelibrary import schemas
from detectmatelibrary.detectors.value_range_detector import ValueRangeDetector

with open("docs/examples/detectors/value_range.yaml") as f:
    config = yaml.safe_load(f)
detector = ValueRangeDetector(name="DownloadSizeDetector", config=config)


def download(path: str, bytes_read: int) -> schemas.ParserSchema:
    """What a parser emits for 'close <path> bytes read <bytes_read> written
    0'."""
    return schemas.ParserSchema({"eventID": 2, "variables": [path, str(bytes_read), "0"]})


# the first 3 logs train the detector (data_use_training: 3)
for log in [download("/srv/a.pdf", 100), download("/srv/b.pdf", 250), download("/srv/c.pdf", 180)]:
    detector.process(log)

print(detector.process(download("/srv/d.pdf", 200)))  # None: inside the learned range
alert = detector.process(download("/srv/db.dump", 99999))
print(dict(alert["alertDetails"]))  # {'EventID 2 - bytes_read': "Out of range value: '99999' (100 - 250)"}

Configuration file

The configuration used by the example above. It sets only what this use case needs; every other parameter keeps its default (see Configuration arguments).

detectors:
  DownloadSizeDetector:
    method_type: value_range_detector
    auto_config: false          # we pick the variables ourselves (see `events`)
    params:
      data_use_training: 3      # learn the range from the first 3 logs, then detect
    events:
      2:                        # EventID: "close <*> bytes read <*> written <*>"
        download:
          variables:
            - pos: 1
              name: bytes_read

The same file works unchanged in both places a detector runs:

Library: load it with yaml.safe_load and pass the dict as config=, as in the example. The key under detectors: must match the detector's name.
DetectMateService: use it as the service's detector configuration.

Configuration arguments

All parameters this detector accepts, grouped by the YAML block they go in. Scope tells whether a parameter is specific to this detector or shared with other detectors (see the Detectors overview). Where this detector changes a shared default, the shared value is shown in brackets.

Top level

Field	Type	Default	Scope	Description
`method_type`	string	value_range_detector	shared	Indicates what type of method it is.
`auto_config`	boolean	True	shared	Runs the configuration step before the training process. Has no effect unless data_use_configure is set.
`events`	object	{}	shared	Events configuration dict keyed by event_id.
`global`	object	{}	shared	Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as `global` in YAML.
`persist`	object, null	None	shared	Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page.

params

Field	Type	Default	Scope	Description
`ignore_non_numerical_val`	boolean	True	specific	Drop non-numeric values instead of raising when a configured variable cannot be cast to a number.
`start_id`	integer	10	shared	Number used to start the unique ID generator.
`data_use_training`	integer, null	None	shared	Number of logs used for training. None skips training.
`data_use_configure`	integer, null	None	shared	Number of logs used for the configuration step. None skips it.
`use_config_data_as_training`	boolean	True	shared	Combine the configured data in the training process if True.
`train_buffer_max_records`	integer	100000	shared	Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records.
`train_buffer_dir`	string, null	None	shared	Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind.
`parser`	string	PARSER	shared	Name of the parser used.
`allow_fed`	boolean	False	shared	Builds the stores needed to take part in federation.

auto_config_params (read only while auto_config is true)

Field	Type	Default	Scope	Description
`classification.index`	boolean	True	shared	Segment-mean test over equal-count segments.
`classification.time`	boolean	False	shared	Segment-mean test over equal-duration segments. Needs timestamp_variable.
`classification.segment_thresholds`	array	[1.1, 0.3, 0.1, 0.01]	shared	Upper bound on the mean change rate, one per segment; the list length is the segment count. Used by index and time.
`classification.slope_index`	boolean	False	shared	Change-centroid test on the index axis.
`classification.slope_time`	boolean	False	shared	Change-centroid test on the time axis. Needs timestamp_variable.
`classification.slope_threshold`	number	-0.05	shared	A variable is STABLE when its change centroid (-0.5 to +0.5) is at or below this. Used by slope_index and slope_time.
`classification.decision`	string	consensus	shared	How the enabled methods' verdicts combine: consensus needs all of them, majority needs more than half.
`timestamp_variable`	string, null	None	shared	Header variable (from the parser's log_format) holding each event's time. Required by the time and slope_time classification methods.
`timestamp_format`	string, null	None	shared	Format of timestamp_variable. None detects it automatically.
`use_stable_vars`	boolean	True	shared	Monitor the variables the configure phase classifies as STABLE.
`use_static_vars`	boolean	True	shared	Monitor the variables the configure phase classifies as STATIC (a single value).