JSON Parser

Extracts structured information from JSON-formatted logs. Optionally delegates parsing of a specific JSON field (the "content") to a sibling Template Matcher parser. Nested JSON objects are always flattened to dot-separated keys in headerVariables.

In/out

Input and output schemas in the pipeline

	Schema	Description
Input	LogSchema	Raw log (JSON string)
Output	ParserSchema	Structured log with extracted fields

Examples

import json
from detectmatelibrary.parsers.json_parser import JsonParser
from detectmatelibrary import schemas

parser = JsonParser(name="JsonParser")  # all defaults: flattens the JSON fields

json_log = {
    "time": "2023-11-18 10:30:00",
    "request": {"method": "GET", "path": "/api/users"},
}
output = schemas.ParserSchema()
parser.parse(schemas.LogSchema({"logID": "1", "log": json.dumps(json_log)}), output)

print(output.headerVariables["request.method"])  # GET
print(output.headerVariables["request.path"])    # /api/users

With the configuration file below, the message field is also matched against templates:

import yaml  # noqa: E402

with open("docs/examples/parsers/json_parser.yaml") as f:
    config = yaml.safe_load(f)
parser = JsonParser(name="JsonParser", config=config)

json_log = {
    "time": "2023-11-18 10:30:00",
    "message": "pid=9699 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting acct=\"root\"",
    "level": "INFO",
}
output = schemas.ParserSchema()
parser.parse(schemas.LogSchema({"logID": "1", "log": json.dumps(json_log)}), output)

print(output.headerVariables["level"])  # INFO
print(output.template)                     # pid=<*> uid=<*> auid=<*> ses=<*> msg='op=PAM:<*> acct=<*>

Event IDs

Log	Event ID	Template
message matches a template	that template's ID	the template
message matches no template	-1	<Not Found>
no message field	-2	empty

If the content parser has no templates (for example with the defaults, as in the first example), every message gets -1. The parser then logs a warning at the first message.

Configuration file

The configuration used by the second example above (the first one runs with all defaults). It sets only what this use case needs; every other parameter keeps its default (see Configuration arguments).

parsers:
  JsonParser:
    method_type: json_parser
    params:
      timestamp_name: time            # JSON key holding the timestamp
      content_name: message           # JSON key holding the free-text message
      content_parser: JsonMatcherParser   # parser (defined below) applied to the message
  JsonMatcherParser:
    method_type: matcher_parser
    params:
      path_templates: tests/test_data/test_templates.txt

JsonParser hands the message field to the parser named in content_parser, so the file defines two entries. The same file works unchanged in both places a parser runs:

• Library: load it with yaml.safe_load and pass the dict as config=, as in the example. The key under parsers: must match the parser's name.
• DetectMateService: use it as the service's parser configuration.

Configuration arguments

All parameters this parser accepts, grouped by the YAML block they go in. Scope tells whether a parameter is specific to this parser or shared with other parsers (see the Parsers overview).

Top level

Field	Type	Default	Scope	Description
method_type	string	json_parser	shared	Indicates what type of method it is.
auto_config	boolean	False	shared	Runs the configuration step before the training process.

params

Field	Type	Default	Scope	Description
timestamp_name	string	time	specific	fitting description yet to find
content_name	string	message	specific	fitting description yet to find
content_parser	string	JsonMatcherParser	specific	fitting description yet to find
start_id	integer	10	shared	Number used to start the unique ID generator.
data_use_training	integer, null	None	shared	Number of logs used for training. None skips training.
data_use_configure	integer, null	None	shared	Number of logs used for the configuration step. None skips it.
use_config_data_as_training	boolean	True	shared	Combine the configured data in the training process if True.
train_buffer_max_records	integer	100000	shared	Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records.
train_buffer_dir	string, null	None	shared	Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind.
log_format	string, null	None	shared	fitting description yet to find
time_format	string, null	None	shared	fitting description yet to find

The content_parser value is a name, not an inline config. The referenced parser must be defined as a separate sibling entry at the same level as JsonParser.