Install filebeat
on RPM Based:
sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
create /etc/yum.repos.d/elastic.repo
[elastic-6.x] name=Elastic repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md
sudo yum install filebeat
on deb based:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
sudo apt-get update && sudo apt-get install filebeat
Configure
/etc/filebeat/filebeat.yml:
#============================== Dashboards ===================================== # These settings control loading the sample dashboards to the Kibana index. Loading # the dashboards is disabled by default and can be enabled either by setting the # options here, or by using the `-setup` CLI flag or the `setup` command. setup.dashboards.enabled: true
setup.kibana: # Kibana Host # Scheme and port can be left out and will be set to the default (http and 5601) # In case you specify and additional path, the scheme is required: http://localhost:5601/path # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601 host: "elkhost:5601"
#-------------------------- Elasticsearch output ------------------------------ output.elasticsearch: # Array of hosts to connect to. hosts: ["elkhost:9200"]
optionally, change the name
edit filebeat.yml, change “name:” to set the ‘beta.name’ on kibana
Modify pattern:
/usr/share/filebeat/module/nginx/access/ingest/default.json
"patterns":[
"\"?%{IP_LIST:nginx.access.remote_ip_list} - %{DATA:nginx.access.user_name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{WORD:nginx.access.method} %{DATA:nginx.access.url} HTTP/%{NUMBER:nginx.access.http_version}\" %{NUMBER:nginx.access.response_code} %{NUMBER:nginx.access.body_sent.bytes} \"%{DATA:nginx.access.referrer}\" \"%{DATA:nginx.access.agent}\" \"%{DATA:nginx.access.http_x_forwarded_for}\" %{NUMBER:nginx.access.request_time}"
],
Create request_time Field
/etc/filebeat/fields.yml:
- name: request_time
type: float
description: >
processing time in seconds with a milliseconds resolution
Change the template
Add nginx.access.request_time field.
/usr/share/filebeat/kibana/default/index-pattern/filebeat.json
{\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"nginx.access.request_time\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"number\", \"scripted\": false},
Update NGINX log format
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $request_time';
access_log /var/log/nginx/access.log main;
for EB use $http_x_forwarded_for instead of $remote_addr:
log_format main '$http_x_forwarded_for - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $request_time';
Enable nginx module on filebeat
filebeat modules enable nginx
Set for auto start on boot
rpm:
sudo update-rc.d filebeat defaults 95 10
or (deb)
sudo chkconfig --add filebeat
Sample Config Files
https://github.com/crochik/tldr
0 Comments