Knowledge Required: Little

Tools required: Elasticsearch, Logstash

If you’re running Elasticsearch, there are many ways to get data in. One popular option is Filebeat.

By default, Filebeat will submit logs to an index which contains the phrase ‘Filebeat’ and also includes the verion number. Similar to:

list Screenshot of Kibana (Data -> Index Management)

The default behaviour is that Filebeat will create an index name specified in the config if it doesn’t exist, so we don’t need to worry about manually creating the index in Elastcisearch!

Custom configration

Sending logs to a custom index can be achieved by modifying the config file at /etc/filebeat/filebeat.yml

Under the output.elasticsearch directive, we’re going configure the index by adding the following:

output.elasticsearch:
   index: '<yourindexname>-%{[agent.version]}-%{+dd.MM.yyyy}'

Note the indentation

As we’ve specified a custom index, we must also provide Filebeat a template name and template pattern. Like indexes, these are created automatically after we’ve specified them in the config file. At the end of your filebeat.yml and outside any indentation, add the following:

setup.template.name: "<yourindexname>"
setup.template.pattern: "<yourindexname>-*"

For example, a valid configuration is shown below:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["192.168.0.21:443"]
  ssl.verification_mode: none
  path: /elastic
  protocol: "https"
  index: 'myindex-%{[agent.version]}-%{+dd.MM.yyyy}'

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: elastic
setup.ilm.enabled: false 
setup.template.name: "myindex"
setup.template.pattern: "myindex-*"

for a valid configuration, you should have configured some of above settings already

Note:

When specifing the index we make use of variables such as agent.version and the date. This means the result index woud look something like myindex-17.16.2-02.01.2022. Including the agent version and time in the index is useful; it allows us to clean up old indexes by date or by agent version later on, if needed.

To search logs in Kibana regardless of the date or agent version in the index, create an index pattern matching myindex-*. You can read about setting up index patterns with the API here

EOF break