Charset Detector

The Charset Detector raises alerts when previously unseen characters appear in configured fields. It is useful to detect malformed or injected values in fields with a predictable format, for example shell characters such as ; in a user name.

In/out

Input and output schemas in the pipeline

	Schema	Description
Input	ParserSchema	Structured log
Output	DetectorSchema	Alert / finding

At a glance

Learns from training data	Auto-configuration	Needs events	Federation
✅	✅ picks the variables to monitor	✅ unless auto_config: true	✅

Description

This detector keeps the set of characters observed during training for each monitored field and emits an alert whenever a value contains a character that is not in the set. Alerts don't add the character to the set, so the same unseen character raises an alert every time it appears.

Example

import yaml
from detectmatelibrary import schemas
from detectmatelibrary.detectors.charset_detector import CharsetDetector

with open("docs/examples/detectors/charset.yaml") as f:
    config = yaml.safe_load(f)
detector = CharsetDetector(name="UserCharsetDetector", config=config)


def failed_login(user: str) -> schemas.ParserSchema:
    """What a parser emits for 'Failed password for <user> from 10.0.0.1 port
    22'."""
    return schemas.ParserSchema({"eventID": 1, "variables": [user, "10.0.0.1", "22"]})


# the first 3 logs train the detector (data_use_training: 3)
for log in [failed_login("alice"), failed_login("bob"), failed_login("carol")]:
    detector.process(log)

print(detector.process(failed_login("carla")))  # None: only known characters
alert = detector.process(failed_login("bob;rm -rf"))
print(dict(alert["alertDetails"]))
# {'EventID 1 - user': "Unknown character(s): ' ', '-', ';', 'f', 'm'"}

Configuration file

The configuration used by the example above. It sets only what this use case needs; every other parameter keeps its default (see Configuration arguments).

detectors:
  UserCharsetDetector:
    method_type: charset_detector
    auto_config: false          # we pick the variables ourselves (see `events`)
    params:
      data_use_training: 3      # learn from the first 3 logs, then detect
    events:
      1:                        # EventID: "Failed password for <*> from <*> port <*>"
        failed_login:
          variables:
            - pos: 0
              name: user

The same file works unchanged in both places a detector runs:

• Library: load it with yaml.safe_load and pass the dict as config=, as in the example. The key under detectors: must match the detector's name.
• DetectMateService: use it as the service's detector configuration.

Configuration arguments

All parameters this detector accepts, grouped by the YAML block they go in. Scope tells whether a parameter is specific to this detector or shared with other detectors (see the Detectors overview). Where this detector changes a shared default, the shared value is shown in brackets.

Top level

Field	Type	Default	Scope	Description
method_type	string	charset_detector	shared	Indicates what type of method it is.
auto_config	boolean	True	shared	Runs the configuration step before the training process. Has no effect unless data_use_configure is set.
events	object	{}	shared	Events configuration dict keyed by event_id.
global	object	{}	shared	Instances monitoring event-independent header variables (e.g. hostname, level), keyed by instance name. Written as global in YAML.
persist	object, null	None	shared	Periodic state saving (path, interval_seconds, events_until_save, auto_load, storage_options). None disables it. See Saving state (persist) on the Detectors page.

params

Field	Type	Default	Scope	Description
start_id	integer	10	shared	Number used to start the unique ID generator.
data_use_training	integer, null	None	shared	Number of logs used for training. None skips training.
data_use_configure	integer, null	None	shared	Number of logs used for the configuration step. None skips it.
use_config_data_as_training	boolean	True	shared	Combine the configured data in the training process if True.
train_buffer_max_records	integer	100000	shared	Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records.
train_buffer_dir	string, null	None	shared	Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind.
parser	string	PARSER	shared	Name of the parser used.
allow_fed	boolean	False	shared	Builds the stores needed to take part in federation.

auto_config_params (read only while auto_config is true)

Field	Type	Default	Scope	Description
classification.index	boolean	True	shared	Segment-mean test over equal-count segments.
classification.time	boolean	False	shared	Segment-mean test over equal-duration segments. Needs timestamp_variable.
classification.segment_thresholds	array	[1.1, 0.3, 0.1, 0.01]	shared	Upper bound on the mean change rate, one per segment; the list length is the segment count. Used by index and time.
classification.slope_index	boolean	False	shared	Change-centroid test on the index axis.
classification.slope_time	boolean	False	shared	Change-centroid test on the time axis. Needs timestamp_variable.
classification.slope_threshold	number	-0.05	shared	A variable is STABLE when its change centroid (-0.5 to +0.5) is at or below this. Used by slope_index and slope_time.
classification.decision	string	consensus	shared	How the enabled methods' verdicts combine: consensus needs all of them, majority needs more than half.
timestamp_variable	string, null	None	shared	Header variable (from the parser's log_format) holding each event's time. Required by the time and slope_time classification methods.
timestamp_format	string, null	None	shared	Format of timestamp_variable. None detects it automatically.
use_stable_vars	boolean	True	shared	Monitor the variables the configure phase classifies as STABLE.
use_static_vars	boolean	True	shared	Monitor the variables the configure phase classifies as STATIC (a single value).