Docker Compose Reference

DetectMate Service ships a comprehensive Docker Compose configuration that includes all services required to consume and process logs and send alerts. This reference explains the settings of the docker-compose.yaml. The following image illustrates the log data pipeline deployed by Docker Compose:

Log file ingestion is handled by Fluentd. It supports reading from various systems and can convert the data to a specific format before sending it to any other target. Using the default setup, fluentd reads lines from a file and sends them to the DetectMate parser. The parser processes and forwards this data to the detector. If the detector finds an anomaly, it will send it to another Fluentd process that can communicate with various targets, such as Elasticsearch, Kafka, or a log file.

The following sections will explain the docker-compose.yml service by service.

Fluentin

The first component in this configuration is the service fluentin, which is a fluentd container. It reads data from a file, formats it to the proper detectmate protobuf format, and then forwards it via a nanomsg queue socket to the parser service.

While reading from a file is just an example in this setup, you can refer to the Fluentd documentation for guidance on reading from other sources.

Building the image

Docker Compose builds the fluentin image using the configuration in the file container/Dockerfile_fluentd. This Dockerfile uses the official Fluentd image for the base and adds the necessary Fluentd plugins.

Volumes

fluentin mounts the following volumes using bind-mount.

• container/fluentin:/fluentd/etc provides the fluentd configuration. The file container/fluentin/fluent.conf can be used to reconfigure fluentd to read from other log sources.

• container/fluentlogs:/fluentd/log exposes the directory containing logfiles to read. fluentin reads all logs from the file some.log by default.

• container/run:/run is mounted to hold the socket files for inter process communication. In this configuration, fluentin and parser are communicating via unix domain sockets using NNG (Nano Message Next Generation) for maximum performance.

Depends

This services depends on the parser service, because parser creates the socket file to establish communication.

Parser

The second component of the logdata analysis pipeline is the service parser. This DetectMate service listens for logs received from the fluentin service on a Nano Message socket. It expects the data in the LogSchema format used by DetectMate and sends the parsed data using the ParserSchema to another Nano Message socket provided by a DetectMate detector service.

Building the image

Docker compose builds the detectmate image using the Dockerfile in the root of the DetectMate repository. This image is the same for parser and detector.

Volumes

All DetectMate container(parser and detector) need the following volumes mounted:

• container/config:/config provides the DetectMate configuration. Each DetectMate Service needs one settings-file for the general service configuration and one config file for the specific DetectMate component. For parser these files are parser_settings.yaml for service settings and parser_config.yaml that holds the component-specific parameters.

• container/logs:/logs exposes all log files that are generated by DetectMate services. Mostly needed for debugging.

• container/run:/run is mounted to hold the socket files for inter process communication. fluentin and parser communicate via the file ipc:///run/parser.engine.ipc. parser and detector communicate via the file ipc:///run/detector.ipc.

Ports

The parser service exposes port 8001 (internally the service runs on port 8000). The management API server runs on this port. This service exposes metrics for the prometheus service and allows to start, stop and reconfigure the service.

Depends

This services depends on the detector service, because detector creates the socket file to establish communication.

Detector

The detector service listens for parsed logs received from the parser service on a Nano Message socket. It expects the data in the ParserSchema format used by DetectMate and sends anomalies using the DetectorSchema to another Nano Message socket provided by the fluentout service.

Building the image

Docker compose builds the detectmate image using the Dockerfile in the root of the DetectMate repository. This image is the same for parser and detector.

Volumes

All DetectMate container(parser and detector) need the following volumes mounted:

• container/config:/config provides the DetectMate configuration. Each DetectMate Service needs one settings file for the general service configuration and one config file for the DetectMate component-specific parameters. For detector these files are detector_settings.yaml and detector_config.yaml respectively.

• container/logs:/logs exposes all log files that are generated by DetectMate services. Mostly needed for debugging.

• container/run:/run is mounted to hold the socket files for inter process communication. parser and detector communicate via the file ipc:///run/detector.ipc and detector and fluentout use the socket ipc:///run/output.ipc.

Ports

The detector service exposes port 8002 (internally the service runs on port 8000). The management API server runs on this port. This service exposes metrics for the prometheus service and allows to start, stop and reconfigure the service.

Depends

This services depends on the fluentout service, because fluentout creates the socket file to establish communication.

Fluentout

The fluentoutservice receives data from the detector, parses it from the proper detectmate protobuf format, and then writes the alerts to the logfiles output.%Y%m%d.

While writing to a log file is just an example in this setup, you can refer to the Fluentd documentation for guidance on writing alerts to other targets.

Building the image

Docker Compose builds the fluentout image using the configuration in the file container/Dockerfile_fluentd. This Dockerfile uses the official Fluentd image for the base and adds the necessary Fluentd plugins.

Volumes

fluentout mounts the following volumes using bind-mount.

• container/fluentin:/fluentd/etc provides the fluentd configuration. The file container/fluentin/fluent.conf can be used to reconfigure fluentd to write to other targets.

• container/fluentlogs:/fluentd/log exposes the directory containing written log files. By default, fluentout writes all alerts to files named output.%Y%m%d.

• container/run:/run is mounted to hold the socket files for inter process communication. fluentout and detector are talking in this configuration using Nano Message Next Generation via unix domain sockets for maximum performance.

Prometheus

Prometheus is the monitoring server that collects performance metrics from the management api that runs on the parser and detector on port 8000.

Volumes

prometheus mounts the following volumes:

• container/prometheus.yml:/etc/prometheus/prometheus.yml is the single prometheus configuration file, that tells prometheus to collect the metrics data from parserand detector at port 8000*.

• prometheus_data:/prometheus holds the prometheus database.

Ports

This service exposes port 9090

Grafana

Grafana allows to visualize the metrics data that is stored in the prometheus service. Per default the username and password are admin/admin.

Volumes

grafana mounts the following volumes:

• container/grafana/prometheus.yml:/etc/grafana/provisioning/datasources/prometheus.ymlis the config file that preconfigures grafana to use prometheus as a datasource.
• grafana_data:/var/lib/grafana is the database of grafana

Kafka

The Apache Kafka service is a message queue that can be used to read logdata from or send alerts to. It is not needed by the logpipeline and is therefore not active(commented out).