Skip to content

Persistency

The persistency module gives a detector a place to remember things about the events it sees. State is keyed by eventID and survives across training, the detection loop, and (optionally) restarts on disk.

This page is structured to read top-to-bottom: first the mental model, then a quick start, then the API surface.

Mental model

Persistency has three moving parts. Understanding what each one does makes the rest of the page much easier to follow.

1. Events

Logs are grouped by eventID. Two events with the same ID share a template but have their own variable values. Persistency stores one independent state object per event ID, so an EventStabilityTracker for eventID=4733 does not interfere with one for eventID=4624.

2. Backends (EventDataStructure)

A backend is the thing that actually stores the per-event state. Persistency owns the dict {event_id: backend}; the backend itself decides how data is kept.

Two families ship today:

  • DataFrame backends (EventDataFrame, ChunkedEventDataFrame) keeps the raw rows. Very storage heavy and not recommended for production-ready detectors.
  • Tracker backends (EventStabilityTracker) keep only derived features (e.g. "this variable has been constant for the last 10k events") that are relevant for the detector. Use these when you only need a summary or a subset of the log's information, not the raw history: they cost a fraction of the memory.

All backends implement the same four-method contract: add_data, get_data, dump, load. That contract is what EventPersistency and PersistencySaver rely on, so anything you add later only has to follow it.

3. Saver lifecycle (PersistencySaver)

EventPersistency itself is in-memory. To survive a process restart, the state has to be written somewhere. PersistencySaver wraps an EventPersistency and:

  • writes to disk (or any fsspec URI) on two triggers: a wall-clock interval and an event-count threshold;
  • optionally auto_loads previously saved state during construction;
  • exposes start() / stop() so the background timer can be torn down cleanly. stop() is idempotent and is called automatically when a tracker detector is used as a context manager.

In practice a detector never instantiates PersistencySaver directly: a tracker detector (a subclass of base_detectors.TrackerDetector) takes a persist: block in its config and TrackerDetector starts the saver.

Quick start

from detectmatelibrary.utils import persistency

ep = persistency.EventPersistency()

ep.ingest_event(
    event_id="4624",
    event_template="An account was successfully logged on.",
    named_variables={"AccountName": "alice", "LogonType": "3"},
)

tracker = ep.get_event_data("4624")  # or ep["4624"]

That snippet covers the whole in-memory API: pick a backend class, ingest events, query state.

API reference

EventPersistency

Parameter Description
variable_blacklist Variable names to skip when ingesting. Defaults to ["Content"].
event_data_kwargs Extra kwargs forwarded to each backend instance.

Common methods:

ep.ingest_event(event_id, event_template, variables=..., named_variables=...)

ep.get_event_data(event_id)        # backend for a single event
ep.get_events_data()               # dict[event_id -> backend]
ep.get_event_template(event_id)
ep.get_event_templates()
ep.get_events_seen()               # all event IDs ever ingested
ep[event_id]                       # alias for get_event_data

Available backends

Class Use when
persistency.EventStabilityTracker You only care about how variables behave over time (STATIC / STABLE / UNSTABLE / RANDOM). Cheapest memory footprint.

All three are re-exported from the top of the package: persistency.X is the canonical import; the deeply nested submodules are an implementation detail.

Persisting to disk

saver = persistency.PersistencySaver(
    ep,
    persistency.PersistencySaverConfig(
        path="./state/my-detector",
        save_interval_seconds=300,
        events_until_save=10_000,     # save after this many ingests, too
        auto_load=False,
        storage_options={},           # forwarded to fsspec
    ),
)
saver.start()
# ... detector runs ...
saver.stop()    # final flush, stops the background timer

PersistencySaver.save() is thread-safe, and stop() is idempotent. The two save triggers (save_interval_seconds and events_until_save) are independent: whichever fires first wins.

If writing to storage fails (unwritable path, full disk, lost credentials), save() and stop() raise persistency.PersistencySaveError, and events_since_save keeps counting the unsaved events. Saves fired by the two triggers cannot raise to a caller, so they log the failure at ERROR instead; the timer keeps running, and after a failed save the event-count trigger waits another events_until_save events before retrying.

Restoring state

saver = persistency.PersistencySaver(
    ep,
    persistency.PersistencySaverConfig(path="./state/my-detector", auto_load=True),
)
# ep is now pre-populated from disk

If auto_load=True and no saved state exists, the constructor raises persistency.PersistencyLoadError immediately: it fails fast rather than silently starting empty.

Exporting and importing state on demand

For one-shot transfers (e.g. moving trained state to a new environment, or taking a manual snapshot), use the standalone functions directly:

from detectmatelibrary.utils import persistency

# Export to a file URI
persistency.save(ep, "./snapshots/trained-state")

# Export to bytes (no disk I/O; useful when sending state over a network API)
data: bytes = persistency.save(ep)

# Import from a file URI
persistency.load(ep, "./snapshots/trained-state")

# Import from bytes
persistency.load(ep, data)

save writes the same format as PersistencySaver and resets ep._events_since_save. When called without a path it returns the state as a zip archive in memory. load accepts either a path string or the bytes returned by save; it raises PersistencyLoadError if no state exists at the path, restores event_data_class and event_data_kwargs from metadata, and clears any existing state in ep before loading.

Both functions are not thread-safe when called concurrently with a running PersistencySaver on the same ep. Use the detector-level wrappers below when a saver is active.

Detector-level export and import

When working through a detector (the typical path for DetectMateService), use the methods on the detector object directly, with no need to access EventPersistency internals:

# Export to a file URI
detector.export_state("./snapshots/my-detector")

# Export to bytes (e.g. for an API response)
data: bytes = detector.export_state()

# Import from a file URI
detector.import_state("./snapshots/my-detector")

# Import from bytes (e.g. from an API request body)
detector.import_state(data)

Both methods are thread-safe: they hold the saver lock while a PersistencySaver is running. Only tracker detectors have state; on any other component export_state() returns None and import_state() does nothing.

Storage backends (fsspec)

PersistencySaverConfig.path accepts any URI fsspec understands: a local path (./state), s3://bucket/key, gs://..., az://..., and so on. Provider credentials and tuning knobs go in storage_options.

Using persistency inside a detector

Detectors do not import utils.persistency. A detector that keeps state subclasses base_detectors.TrackerDetector, which owns the stores, the persist: saver, export_state() / import_state() and federation. The tracker detectors are New Event, New Value, New Value Combo, Value Range, Charset, Event Sequence, Bigram Frequency, SCVS and ECVC; only they accept a persist: block. See Saving state (persist) for the config schema.

In detector code, the pattern is:

from detectmatelibrary.base_detectors import TrackerDetector, TrackerDetectorConfig

class MyDetectorConfig(TrackerDetectorConfig):
    method_type: str = "my_detector"

class MyDetector(TrackerDetector):
    def __init__(self, name="MyDetector", config=MyDetectorConfig()):
        super().__init__(name=name, config=config)

    def train(self, input_):
        self._ingest(input_, variables={...}, event_id=input_["eventID"])

    def detect(self, input_, output_):
        tracker = self.persistency.get_events_data().get(input_["eventID"])
        # compare against tracker to produce alerts

TrackerDetector builds self.persistency, the store training and detection use, and self.auto_conf_persistency, the configure-phase store. Override _event_data_kwargs() / _auto_conf_kwargs() to pass tracker kwargs, and call self._new_store() for any extra store. Without a persist: block, self.saver stays None.

Rebuild fields derived from the stores (a count vector, a window length) in _sync_from_state(). TrackerDetector calls it at the end of __init__ (after a possible auto_load), after import_state(), on the detector from_binary() returns, and on every component after aggregate_strategy(). The first call runs inside TrackerDetector.__init__, so anything it reads or writes must exist before super().__init__() returns: declare it as a class attribute or assign it before the call.