Detectors
Detectors process structured logs from Parsers and emit alerts when anomalies are detected.
| Schema | Description | |
|---|---|---|
| Input | ParserSchema | Structured log |
| Output | DetectorSchema | Alert / finding |
This document describes the minimal API, implementation guidance and a short example detector.
CoreDetector: minimal API
class CoreDetectorConfig(CoreConfig):
component_type: str = "detectors"
method_type: str = "core_detector"
parser: str = "PARSER"
auto_config: bool = True
events: EventsConfig | dict[str, Any] = {}
global_instances: dict[str, Any] = {} # written as `global` in YAML
class CoreDetector(CoreComponent):
def __init__(
self,
name: str = "CoreDetector",
buffer_mode: BufferMode = BufferMode.NO_BUF,
buffer_size: int | None = None,
config: CoreDetectorConfig | dict[str, Any] | None = CoreDetectorConfig(),
) -> None:
"""buffer_mode and buffer_size set how many logs detect() receives per call."""
def run(
self, input_: List[ParserSchema] | ParserSchema, output_: DetectorSchema
) -> bool:
"""Define in the Core detector"""
def detect(
self,
input_: List[ParserSchema] | ParserSchema,
output_: DetectorSchema,
) -> bool:
"""Empty, must be define in the specific detector"""
def train(self, input_: ParserSchema | list[ParserSchema]) -> None:
"""Empty, can be define in the detector. It trains the detector"""
Implementing a detector: example
Simple detector that raises an alert when a numeric variable exceeds a threshold.
class SimpleThresholdConfig(CoreDetectorConfig):
method_type: str = "simple_threshold"
threshold: float = 0.0
class SimpleThresholdDetector(CoreDetector):
def __init__(
self,
name: str = "SimpleThreshold",
config: SimpleThresholdConfig | dict[str, Any] = SimpleThresholdConfig(),
):
if isinstance(config, dict):
config = SimpleThresholdConfig.from_dict(config, name)
super().__init__(name=name, buffer_mode=BufferMode.NO_BUF, config=config)
def detect(
self, input_: schemas.ParserSchema, output_: schemas.DetectorSchema
) -> bool:
# calculate is a dummy method
if calculate(input_) > self.config.threshold:
output_["alertID"] = f"{self.name}-{int(time.time())}"
output_["logIDs"].extend([ev.logID] if ev.logID else [])
output_["score"] = float(value)
output_["description"] = (
f"Value {value} > threshold {self.config.threshold}"
)
return True
return False
Detectors methods
The detectors are numbered from simplest to most complex, and the sidebar lists them in the same order. The simplest ones need no training or learn a plain set of values; the most complex are neural networks that need a lot of training data. If you are new to DetectMate, start at the top.
| # | Detector | What it detects |
|---|---|---|
| 00 | Random Detector | Raises random alerts, for testing a pipeline. No training. |
| 01 | Rule Detector | Logs that match fixed rules (unknown template, keyword, exception, error level). No training. |
| 02 | New Event Detector | Event IDs (log templates) never seen in training. |
| 03 | New Value Detector | Values of a variable never seen in training. |
| 04 | New Value Combo Detector | Combinations of values never seen together in training. |
| 05 | Value Range Detector | Numeric values outside the range seen in training. |
| 06 | Charset Detector | Characters in a variable never seen in training. |
| 07 | Event Sequence Detector | Orders of consecutive event IDs never seen in training. |
| 08 | Bigram Frequency Detector | Values whose character pairs are improbable under a learned frequency model. |
| 09 | SCVS Detector | Sequence Count Vector Set: windows whose event counts never occurred in training. |
| 10 | ECVC Detector | Event Count Vector Clustering: windows whose event counts are far from those seen in training. |
| 11 | DeepLog Detector | Unexpected next events in a sequence, predicted by an LSTM. |
| 12 | LogBERT Detector | Unexpected events in a sequence, predicted by a Transformer. |
Configuration
Tell the detector how much data to learn from
data_use_training (logs used for training) and data_use_configure (logs used for
auto-configuration) both default to null, which skips that phase. Without
data_use_training a detector starts detecting with nothing learned; with
auto_config: true but no data_use_configure, it never picks anything to monitor.
Write null, not None, in YAML: None is read as a string.
Long configure phases spill to disk
With use_config_data_as_training: true (the default) every log of the configure phase
is kept until training starts, then replayed into training. Up to
train_buffer_max_records logs (default 100000) stay in memory; beyond that the buffer
is written as Parquet files to a private detectmate-train-* directory under
train_buffer_dir (a local path; default: the system temp directory, or TMPDIR) and
read back when training starts. A warning in the log marks the first spill. The
directory is removed once training has read it; a process that is killed leaves it
behind, and you can delete it by hand. To keep it off disk, lower data_use_configure
or set use_config_data_as_training: false. In a container, the default directory is
in the container's writable layer: point train_buffer_dir at a mounted volume when
the configure phase is large. Where /tmp is a RAM-backed tmpfs (common on Fedora,
Arch and Debian 13) the default spill uses RAM, so point train_buffer_dir at real
disk. NewValueComboDetector reads its configure logs a second time, to learn which
combinations are stable, so it keeps a copy of its own, which spills the same way.
Every detector page shows a minimal, working configuration file next to its example. The reference below explains the blocks those files use.
auto_config defaults to True for detectors, but the configure phase only runs when
data_use_configure is set. A detector with an events block and no
data_use_configure therefore uses that block even if auto_config is left out. The
example configurations set auto_config: false anyway, to make this explicit. If
auto_config is True and data_use_configure is set, the configure phase replaces the
events block with its own selection.
When auto_config is set to False, the detector expects an explicit events or global block that specifies exactly which variables to monitor. events refers to event-specific variables while global refers to variables that are not bound to events (header_variables can but don't have to be event bound):
detectors:
NewValueDetector:
method_type: new_value_detector
auto_config: False
params: # detector-wide parameters
data_use_training: 1000 # the first 1000 logs train the detector
events: # event-specific configuration
1: # event_id
instance1: # name of instance (arbitrary)
params: {} # additional params
variables:
- pos: 0 # location of an unnamed variable from the log message
name: var1 # name of variable (arbitrary)
header_variables:
- pos: Level # name of a field in the parser's log_format (case-sensitive: <Level> -> Level)
global: # define global instance for new_value_detector similar to "events"
global_instance1: # define instance name
header_variables: # same logic as header_variables in "events"
- pos: Status
Common parameters (all detectors)
Some parameters are inherited by every detector from CoreDetectorConfig/CoreConfig/BasicConfig, regardless of what it does. The parameters that are specific to a detector are explained on that detector's own page.
Top level
| Field | Type | Default | Description |
|---|---|---|---|
auto_config |
boolean | True | Runs the configuration step before the training process. Has no effect unless data_use_configure is set. |
events |
object | {} | Events configuration dict keyed by event_id. |
global |
object | {} | Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as global in YAML. |
params
| Field | Type | Default | Description |
|---|---|---|---|
start_id |
integer | 10 | Number used to start the unique ID generator. |
data_use_training |
integer, null | None | Number of logs used for training. None skips training. |
data_use_configure |
integer, null | None | Number of logs used for the configuration step. None skips it. |
use_config_data_as_training |
boolean | True | Combine the configured data in the training process if True. |
train_buffer_max_records |
integer | 100000 | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. |
train_buffer_dir |
string, null | None | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. |
parser |
string | PARSER | Name of the parser used. |
Beyond the common parameters, detectors inherit the parameters of the group they belong to.
Tracker detectors
The detectors that keep their model in persistency stores (New Event, New Value, New Value Combo, Value Range, Charset, Event Sequence, Bigram Frequency, SCVS, ECVC) share the following parameters, inherited from TrackerDetectorConfig. Only these detectors accept a persist: block (see Saving state (persist)).
Top level
| Field | Type | Default | Description |
|---|---|---|---|
persist |
object, null | None | Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page. |
params
| Field | Type | Default | Description |
|---|---|---|---|
allow_fed |
boolean | False | Builds the stores needed to take part in federation. |
Per-variable model detectors
The tracker detectors that learn a per-variable model (Bigram Frequency, Charset, New Value Combo, New Value, Value Range) also share the following parameters, inherited from VariableDetectorConfig.
auto_config_params (read only while auto_config is true)
| Field | Type | Default | Description |
|---|---|---|---|
classification.index |
boolean | True | Segment-mean test over equal-count segments. |
classification.time |
boolean | False | Segment-mean test over equal-duration segments. Needs timestamp_variable. |
classification.segment_thresholds |
array | [1.1, 0.3, 0.1, 0.01] | Upper bound on the mean change rate, one per segment; the list length is the segment count. Used by index and time. |
classification.slope_index |
boolean | False | Change-centroid test on the index axis. |
classification.slope_time |
boolean | False | Change-centroid test on the time axis. Needs timestamp_variable. |
classification.slope_threshold |
number | -0.05 | A variable is STABLE when its change centroid (-0.5 to +0.5) is at or below this. Used by slope_index and slope_time. |
classification.decision |
string | consensus | How the enabled methods' verdicts combine: consensus needs all of them, majority needs more than half. |
timestamp_variable |
string, null | None | Header variable (from the parser's log_format) holding each event's time. Required by the time and slope_time classification methods. |
timestamp_format |
string, null | None | Format of timestamp_variable. None detects it automatically. |
use_stable_vars |
boolean | True | Monitor the variables the configure phase classifies as STABLE. |
use_static_vars |
boolean | True | Monitor the variables the configure phase classifies as STATIC (a single value). |
Deep learning detectors
The two neural detectors (DeepLog, LogBERT) share the following parameters, inherited from DeepLearningDetectorConfig.
params
| Field | Type | Default | Description |
|---|---|---|---|
window_size |
integer | 10 | Number of consecutive events used as one training/detection sequence. |
validation_per |
number | 0.2 | Fraction of data held out for validation during (fine)training. |
finetune_epochs |
integer | 2 | Number of epochs used when finetuning during the configuration phase. |
hyperparameters |
object | {'Model': {}, 'Train': {}, 'Finetune': []} | Model, training and hyperparameter-search settings passed to the underlying deep learning model. |
Configuration semantics (preliminary)
events key: the key is the eventID (or event_id) to monitor: an integer as the parser emits it (see the Template Matcher docs for how the EventID is assigned), or a name (see Named EventIDs and variables below).
global key - This one has a similar functionality as the events key but refers to variables, that are not bound to events (thus can only contain header_variables).
variables[].pos: the 0-indexed position of the <*> wildcard in the matched template, counting from left to right starting at 0. For example, given:
pid=<*> uid=<*> auid=<*> ses=<*> msg='op=<*> acct=<*> exe=<*> hostname=<*> addr=<*> terminal=<*> res=<*>'
pos: 0 captures the value after pid= (for example 10125), pos: 6 the value after
exe= (for example "/usr/sbin/cron"), and so on.
header_variables[].pos: a named field from the log format string (e.g., Type, Time, Content) rather than a wildcard position.
Named EventIDs and variables
Instead of numbers, the events block can use names: a name for the eventID, and
the name of a wildcard for variables[].pos. The names come from the
Template Matcher's template file:
- Named variables: write
<user>instead of<*>in a template. A template uses either named wildcards or<*>, not both. This works in.txtand.csvtemplate files. -
Named EventIDs: use a
.csvtemplate file with anEventIdcolumn next to theEventTemplatecolumn:EventId,EventTemplate login_ok,Accepted password for <user> from <ip> port <port> login_failure,Failed password for <user> from <ip> port <port>
The parser still emits numbers (eventID 1 and position 0 for <user> in the second
template), so the names must be translated before the detector runs. Do this once,
after creating both components, with compile_detector_config(). Without this step
the detector monitors nothing; it only logs a warning that the named eventID was
never observed in training.
from detectmatelibrary import schemas
from detectmatelibrary.parsers.template_matcher import MatcherParser
from detectmatelibrary.detectors.new_value_detector import NewValueDetector
# Templates from a CSV file: the EventId column names each template, and
# named wildcards (<user>, <ip>, <port>) name its variables.
parser = MatcherParser(name="MatcherParser", config={"parsers": {"MatcherParser": {
"method_type": "matcher_parser",
"path_templates": "docs/examples/data/named_templates.csv",
}}})
# The detector configuration uses those names instead of numbers.
detector = NewValueDetector(name="NewUserDetector", config={"detectors": {"NewUserDetector": {
"method_type": "new_value_detector",
"auto_config": False,
"params": {"data_use_training": 2},
"events": {
"login_failure": { # EventId from the CSV
"failed_login": {
"variables": [{"pos": "user", "name": "user"}], # named wildcard <user>
},
},
},
}}})
# Resolve the names to the numbers the parser emits (login_failure -> 1, user -> 0).
# Without this step the detector monitors nothing.
detector.config.events = parser.template_matcher.compile_detector_config(detector.config.events)
lines = [
"Failed password for alice from 10.0.0.1 port 22", # training
"Failed password for alice from 10.0.0.2 port 22", # training
"Failed password for mallory from 10.0.0.3 port 22", # new user
]
for line in lines:
alert = detector.process(parser.process(schemas.LogSchema({"log": line})))
print(dict(alert["alertDetails"]) if alert else None)
# None
# None
# {'EventID 1 - user': "Unknown value: 'mallory'"}
Auto-configuration (optional)
Detectors can optionally support auto-configuration: a process where the detector automatically discovers which variables are worth monitoring, instead of requiring the user to specify them manually.
Auto-configuration is controlled by the auto_config flag in the pipeline config (e.g. config/pipeline_config_default.yaml):
detectors:
NewValueDetector:
method_type: new_value_detector
auto_config: True # enable auto-configuration
params:
data_use_configure: 1000 # logs used to pick the variables
data_use_training: 1000 # logs used to train on them afterwards
# no "events" block needed: it is generated automatically
How it works
When auto-configuration is enabled, the detector goes through two extra phases before training:
Phase 1, configure(input_): The detector ingests events into an EventPersistency instance that uses a tracker backend to analyze variable behavior, for example whether each variable is stable, random, or still has insufficient data. This instance is typically separate from the one used for training, because the configuration phase needs to observe all variables to decide which ones are worth monitoring, while training only tracks the variables that were selected as a result.
Phase 2, set_configuration(): After enough data has been ingested, the detector queries the tracker to select variables that meet its criteria (e.g. only stable variables). It then generates a full events configuration from those results and updates its own config. At this point auto_config is set to False in the generated config, since the configuration is now explicit.
After these two phases, the detector proceeds with the normal train() and detect() lifecycle using the generated configuration.
To support auto-configuration in your own detector, see Development.
Full lifecycle with auto-configuration
1. configure(input_) # call for each event in the dataset
2. set_configuration() # finalize which variables to monitor
3. train(input_) # call for each event in the dataset
4. detect(input_, output_) # call for each event to detect anomalies
When auto_config is False, steps 1 and 2 are skipped entirely. data_use_configure
still reserves its records, and with use_config_data_as_training they still go to
training, so a config rerun with auto_config: False trains on exactly the data the
configuring run saw. This holds for every component with a configure phase, including
the hyperparameter searches of DrainParser and the deep-learning detectors.
That distinction is visible in the config. A detector's settings live in two blocks:
auto_config_params: inputs to the configure phase. They pick which variables the phase selects and are read only whileauto_configisTrue.params: operational settings, read during training and detection on every run.
The configure phase writes its results into the top-level events block (and,
for EventSequenceDetector, into fixed_window_size) and then sets
auto_config to False. It never modifies either input block, so a config can
be rerun with auto_config: False and reproduce the same detector.
Stability classification (optional)
Stability classification decides whether a variable's change history counts as
STABLE by running one or more classification methods against it and combining
their verdicts. There are four independent methods, over two primitives and two
axes:
| method | what it thresholds | axis |
|---|---|---|
index |
segment-mean thresholds | equal-count boundaries |
time |
segment-mean thresholds | equal-duration boundaries |
slope_index |
change centroid vs. slope_threshold |
index positions |
slope_time |
change centroid vs. slope_threshold |
normalized timestamps |
Any subset of the four may be enabled, and any single one may stand alone. By
default only index is enabled: each segment's mean rate
of change is compared against its entry in segment_thresholds (four segments
by default, one threshold per segment), and the segments are equal-count:
each holds the same number of observations, regardless of how much time they
cover. For bursty log sources that is misleading: a variable that changed
constantly during a quiet night and then went silent under a flood of daytime
traffic looks stable, because the flood supplies enough samples to dominate the
later segments. Enabling time cuts the same segments at equal
durations instead, so each segment covers the same amount of wall-clock time;
the detector then needs an event time per record, which it reads from the log's
named variables (headerVariables, i.e. the fields declared in the parser's
log_format) under the name given by timestamp_variable. slope_index and
slope_time ask a different question (whether the change centroid sits early
or late in the series), on the index axis and the time axis respectively.
These parameters live on every VariableDetector subclass (NewValueDetector,
NewValueComboDetector, ValueRangeDetector, CharsetDetector, BigramDetector, …)
and go in the detector's auto_config_params block: they are inputs to the
auto-configuration phase, read only while auto_config is True, and never
consulted at detection time.
detectors:
NewValueDetector:
method_type: new_value_detector
auto_config: True
auto_config_params:
use_stable_vars: True
use_static_vars: True
classification:
index: True # segment-mean thresholds, equal-count cuts
time: False # segment-mean thresholds, equal-duration cuts
segment_thresholds: [1.1, 0.3, 0.1, 0.01] # one per segment; the length is the segment count
slope_index: False # change centroid over index positions
slope_time: False # change centroid over normalized time
slope_threshold: -0.05 # shared by both slope methods
decision: consensus # consensus | majority
timestamp_variable: Time
timestamp_format: "%y%m%d %H%M%S"
The defaults are index: True, the other three False,
segment_thresholds: [1.1, 0.3, 0.1, 0.01], decision: consensus and
slope_threshold: -0.05. A config that sets nothing under classification
uses exactly these. A variable with fewer observations than segments is
classified INSUFFICIENT_DATA; see the segment floor under "Fields" below.
The decision rule
When more than one method is enabled, decision picks how their verdicts
combine. consensus requires every enabled method to return stable; majority
requires strictly more than half of them to.
| enabled | consensus needs |
majority needs |
differ? |
|---|---|---|---|
| 1 | 1/1 | 1/1 | no |
| 2 | 2/2 | 2/2 (a 1–1 tie is UNSTABLE) | no |
| 3 | 3/3 | 2/3 | yes |
| 4 | 4/4 | 3/4 (a 2–2 tie is UNSTABLE) | yes |
Ties resolve to UNSTABLE. That keeps majority from ever being more lenient
than a coin-flip, and makes it collapse onto consensus at one and two enabled
methods. Turning a third method on is the only place the rule starts to matter.
All four methods false is a config error, rejected by a pydantic validator.
It is not a harmless no-op: classification decides INSUFFICIENT_DATA,
STATIC and RANDOM before any method is consulted, so a method-less config
would silently classify every remaining variable STABLE.
Fields
All of these live in the detector's auto_config_params block.
| Field | Type | Default | Description |
|---|---|---|---|
use_stable_vars |
bool |
true |
Include variables classified STABLE in the generated configuration. |
use_static_vars |
bool |
true |
Include variables classified STATIC. Defaults to false on NewValueComboDetector. |
classification |
ClassificationMethods |
see below | Which classification methods run and how their verdicts combine. |
timestamp_variable |
str \| null |
null |
Name of the field in headerVariables holding the record's event time. Required for time and slope_time to have any effect. Only named log-format fields are consulted, never the positional variables list. |
timestamp_format |
str \| null |
null |
Explicit strftime pattern for parsing that field. When unset, TimeFormatHandler auto-detects the format (ISO 8601, Apache, syslog, numeric epoch seconds/milliseconds, and other common layouts). |
Set timestamp_format when the source uses a layout the auto-detection does not
know. The HDFS loghub corpus, for example, stamps records as 081109 203615, which
only parses with an explicit "%y%m%d %H%M%S".
classification's seven fields:
| Field | Type | Default | Description |
|---|---|---|---|
index |
bool |
true |
Segment-mean thresholds, equal-count boundaries. |
time |
bool |
false |
Segment-mean thresholds, equal-duration boundaries. Needs timestamp_variable. |
segment_thresholds |
list[float] |
[1.1, 0.3, 0.1, 0.01] |
Per-segment upper bounds on the mean change rate for index and time. The list's length is the segment count. Entries must be positive and finite and the list must not be empty; an entry above one exempts its segment. |
slope_index |
bool |
false |
Change centroid vs. slope_threshold, measured on index positions. |
slope_time |
bool |
false |
Change centroid vs. slope_threshold, measured on normalized timestamps. Needs timestamp_variable. |
slope_threshold |
float |
-0.05 |
The change-centroid cut-off both slope methods compare against, on a shared [-0.5, +0.5] scale. A variable passes when its centroid is at or below this value. |
decision |
"consensus" \| "majority" |
"consensus" |
How verdicts from more than one enabled method combine; see above. |
The segment methods need at least one observation per segment. A variable with
fewer observations than segments is classified INSUFFICIENT_DATA (with a reason
naming the segment count) rather than scored over empty segments. The tracker's
own min_samples stays the floor for STATIC and RANDOM, which are decided
before any segment is cut; the effective minimum for a STABLE / UNSTABLE
verdict is the larger of the two. A block with only slope methods enabled cuts no
segments and has no such floor.
Fallback behaviour
Time-aware classification is best-effort and never fails a run:
- If
timeorslope_timeis enabled buttimestamp_variableis unset, or the named field is absent from a record, or its value cannot be parsed, the detector logs a single warning (once per detector, so a bad config cannot flood the log) and falls back to the index axis. - If timestamps stop lining up with the recorded observations, or the observed time
span is zero, or they arrive out of order,
timesilently reuses the equal-index cuts, andslope_timecomputes its centroid on the index axis instead, so it degrades toslope_index. - Under
majority, a method that fell back still casts its own vote. If both methods of a pair (slope_index/slope_timeorindex/time) are enabled and timestamps are unusable, they compute the same verdict, which then counts twice.
In every fallback case classification still runs and produces a result; only the axis behind it changes back to index.
A segment with no observations in it is not a fallback: it scores a mean of 0.0,
because nothing observed means nothing changed. Once the segment floor above is
met, equal-index cuts never leave a segment empty; equal-duration cuts of a bursty
variable still do, routinely, so time on its own is lenient towards a burst of
churn followed by silence. Enable index and time together when that leniency
matters: the index pass keeps every segment populated.
Saving state (persist)
Tracker detectors can persist their training state to disk
(or cloud storage) so it can be restored in a later session. Configure this with a
top-level persist: block in the detector config:
detectors:
NewValueDetector:
method_type: new_value_detector
persist:
path: ./state # base path; detector name is appended automatically
interval_seconds: 300 # save every N seconds (default: 300)
events_until_save: null # also save after N ingested events (default: disabled)
auto_load: false # restore saved state on startup (default: false)
storage_options: {} # backend credentials (see below)
events:
...
All fields are optional: persist: {} uses all defaults. Omitting persist: entirely
disables saving (backward compatible). The other detectors (Random, Rule, DeepLog,
LogBERT) have no state to save: a persist: block on them fails at config load with
persist: Extra inputs are not permitted.
The detector name is automatically appended to path, so path: ./state for a detector
named NewValueDetector writes to ./state/NewValueDetector/.
Running under systemd
The default path is CWD-relative. systemd services usually run with CWD /,
so ./state would resolve to /state (wrong location, needs root). To avoid
this, set StateDirectory= in your unit file: systemd creates /var/lib/<dir>
with the right ownership and exports $STATE_DIRECTORY, which the default path
reads automatically. No explicit path: needed:
[Service]
User=detectmate
StateDirectory=detectmate # → state at /var/lib/detectmate/<detector>/
Setting path: explicitly (e.g. an s3:// URL) always overrides $STATE_DIRECTORY.
Fields
| Field | Type | Default | Description |
|---|---|---|---|
path |
str |
$STATE_DIRECTORY or "./state" |
Base directory or cloud URL. Detector name is appended. Defaults to systemd's $STATE_DIRECTORY if set, else ./state (see note above). |
interval_seconds |
int |
300 |
Background save interval in seconds. |
events_until_save |
int \| null |
null |
Save after this many ingested events. null disables event-count triggering. |
auto_load |
bool |
false |
Load saved state on construction. Raises PersistencyLoadError if no state exists. |
storage_options |
dict |
{} |
Credentials and options forwarded to fsspec. |
Storage options examples
Local filesystem (no storage_options needed):
persist:
path: ./state
S3:
persist:
path: s3://my-bucket/detector-state
storage_options:
key: AKIAIOSFODNN7EXAMPLE
secret: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
region_name: eu-west-1
S3-compatible storage (MinIO, etc.):
persist:
path: s3://my-bucket/detector-state
storage_options:
endpoint_url: http://minio:9000
key: minioadmin
secret: minioadmin
Azure Blob Storage:
persist:
path: az://my-container/detector-state
storage_options:
account_name: mystorageaccount
account_key: base64encodedkey==
GCS:
persist:
path: gs://my-bucket/detector-state
storage_options:
project: my-gcp-project
token: /path/to/service-account.json
In practice, credentials are usually supplied via environment variables
(AWS_ACCESS_KEY_ID, etc.) or instance roles, in which case storage_options
stays empty or is omitted.