Alternative way to integrate custom logstash filters/plugins

Saturday, February 4 2023 · Lesezeit: 2 Minuten · 420 Wörter

Logstash plays an important part in the Elastic Stack environment. The built-in filters already provide a wide range of functions to manipulate events. In case you need something special you’ve a few choices. You could for example write inline ruby code or package it to a gem and make it an official plugin. Do you want to write ruby code? I for sure don’t.
The approach I want to talk about is using the http filter (🖇️ 🔐) . With that you can send primitive data types (regular strings, integers, etc) or complex data structures (as JSON) to your custom filter providing a HTTP API. It can then enrich the data and return the results back to the pipeline. By doing it this way you gain a few advantages:

• A logstash independant microservice
• A service which can be scaled easily with modern toolchains
• Using your language of choice
• Endless independence how your data gets enriched

There is just one downside: You can only send string values to your service. Previously converted fields (for example to integer) are lost in transit due to the nature how logstash interpretes fields.

The service

This example service is REST based and provides two routes:

• /plain -> Receives a plain string and echoes it back with a “Received:” prefix
• /json -> Receives JSON in the body with an animal and a number. The animal gets repeated and the number doubled and sent back as JSON in the response

The webserver is listening loally on port 8080.

The logstash pipeline

There are four log lines to process:

Sample dog 1
Sample cat 2
Sample fox 3
Sample snake 4

First of all grok parses the lines into fields:

grok {
    match => {
        "message" => "Sample %{WORD:animal} %{NUMBER:number}"
    }
}

After that we send the data to our custom filter via HTTP:

http {
    body => {
        "number" => "%{number}"
        "animal" => "%{animal}"
    }
    body_format => "json"
    target_body => "response_json"
    url => "http://[::1]:8080/json"
    verb => "GET"
}

The response is stored in the field response_json. Here is an example output:

logstash-http-plugin-logstash-1 | {
logstash-http-plugin-logstash-1 | “number” => “3”,
logstash-http-plugin-logstash-1 | “animal” => “fox”,
logstash-http-plugin-logstash-1 | “message” => “Sample fox 3”,
logstash-http-plugin-logstash-1 | “response_json” => {
logstash-http-plugin-logstash-1 | “number_doubled” => 6,
logstash-http-plugin-logstash-1 | “animal_repeated” => “foxfox”
logstash-http-plugin-logstash-1 | } logstash-http-plugin-logstash-1 | }

The full example using a go application, sample log lines and a logstash pipeline can be found on my GitHub (🖇️ 🔐) . If you have docker compose installed it’s usable out of the box.

Du hast einen Kommentar, einen Wunsch oder eine Verbesserung? Schreib mir doch eine E-Mail! Die Infos dazu stehen hier.

🖇️ = Link zu anderer Webseite
🔐 = Webseite nutzt HTTPS (verschlüsselter Transportweg) Zurück