f5elastic

A fast system tool for receiving high volumes of traffic logs sent from F5 LTM load balancers and indexes these requests inside ES.

It is today battle tested in a production environment handling loads of 500k req/min on a single host.

Features

Works with Elasticsearch 7.x
Support for clusters of ES nodes, for distribution of indexing load.
Health checks for each ES node is enabled by default.
Uses bulk indexing for better performance with large amount of requests.
GeoIP lookup of each request, adds long/lat, city, and country for each client.
Allows to obfuscate client ip if needed.
...

Getting started

Create a Request Logging profile on your BigIP, and add the following Response Settings template (set HSL Protocol to TCP):

<34> $CLIENT_IP || $HTTP_METHOD || ${Host} || $HTTP_URI || $HTTP_STATCODE || $RESPONSE_SIZE || ${Referer} || ${User-Agent} || $SERVER_IP:$SERVER_PORT || $VIRTUAL_POOL_NAME || $VIRTUAL_NAME

Download compiled binary from https://github.com/martensson/f5elastic/releases or go get github.com/martensson/f5elastic
Download the latest Maxmind GeoLite2-City or GeoIP2-City db.
Edit f5elastic.toml (check example in repo)
Apply the ES template inside the repo. (optional but recommended). Create a corresponding ILM policy named f5elastic with the retention rules of your liking.
Run f5elastic -f /path/to/f5elastic.toml
Take a cup of coffee and make some nice dashboards inside Kibana :)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!