ELK+redis+filebeat收集Apache紀錄檔

準備環境：兩臺centos7
規劃如下：
ip地址：
伺服器端：192.168.232.135：
jdk
kibana
logstash
elasticsearch
redis
192.168.232.136：
jdk
elasticsearch
filebeat
Apache

1.關閉防火牆：

systemctl stop firewalld
setenforce 0

2.上傳ELK壓縮包並解壓：

3.安裝jdk（兩臺都安裝）

[root@localhost ELK]# rpm -ivh jdk-8u131-linux-x64_.rpm

4.安裝elasticsearch（兩臺都裝）

rpm -ivh elasticsearch-6.6.2.rpm

5.設定192.168.232.135的elasticsearch組態檔

[root@localhost ELK]# vim /etc/elasticsearch/elasticsearch.yml

開啟elasticsearch

[root@localhost ELK]# systemctl start elasticsearch

驗證：

5.1設定192.168.232.136的elasticsearch組態檔


開啟elasticsearch

[root@localhost ELK]# systemctl start elasticsearch

驗證：

6.伺服器端安裝redis

[root@localhost ~]# tar zxf redis-5.0.0.tar.gz
[root@localhost ~]# cp -r redis-5.0.0 /usr/local/redis
[root@localhost ~]# cd /usr/local/redis/
[root@localhost redis]# yum -y install gcc-c++
[root@localhost redis]# make
[root@localhost redis]# make MALLOC=libc
1.給redis做軟連結
	[root@localhost redis]# ln -sv /usr/local/redis/src/redis-server /usr/bin/redis-	server
	"/usr/bin/redis-server" -> "/usr/local/redis/src/redis-server"
	[root@localhost redis]# ln -sv /usr/local/redis/src/redis-cli /usr/bin/redis-cli
	"/usr/bin/redis-cli" -> "/usr/local/redis/src/redis-cli"
2.修改組態檔
	[root@localhost redis]# vim /usr/local/redis/redis.conf
	69行修改：
 	bind 192.168.232.135
 	508行新增：
 	requirepass 123321
 3.啟動redis：
 	[root@localhost redis]# redis-server ./redis.conf
 4.把511追加到/proc/sys/net/core/somaxconn：
 	[root@localhost redis]# echo 511 >> /proc/sys/net/core/somaxconn
 5.去/etc/sysctl.conf最後一行新增下面命令：
 	[root@localhost redis]# vim /etc/sysctl.conf
 		vm.overcommit_memory = 1
 6.在/etc/rc.local裡面新增：
 	echo never > /sys/kernel/mm/transparent_hugepage/enabled
 7.進入redis組態檔136行將daemonize no修改為yes
 	[root@localhost redis]# vim /usr/local/redis/redis.conf
 		136行將daemonize no修改為daemonize yes
 8.重新啟動redis：
 	[root@localhost redis]# redis-server ./redis.conf
 9.測試：
 192.168.232.135:6379> auth 123321
 OK
 192.168.232.135:6379> get *
 (nil)
 

7.使用者端安裝httpd並啟動：

[root@localhost ELK]# yum -y install httpd
[root@localhost ELK]# systemctl start httpd

8.使用者端安裝filebeat：

[root@localhost ~]# rpm -ivh filebeat-6.8.1-x86_64.rpm

9.設定filebeat.yml檔案收集httpd的紀錄檔

[root@localhost ~]# cd /etc/filebeat/
備份一份原組態檔，以防改錯：
[root@localhost filebeat]# cp filebeat.yml filebeat.yml.bak
[root@localhost filebeat]# vim filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/httpd/access_log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
  setup.ilm.enabled: false
  setup.template.name: "filebeat-httpd"
  setup.template.pattern: "filebeat-httpd-*"
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
output.redis:
  hosts: ["192.168.232.135:6379"]
  key: "filebeat-httpd"
  db: 1
  timeout: 5
  password: 123321

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

10.伺服器端安裝logstash

[root@localhost ELK]# rpm -ivh logstash-6.6.0.rpm

11.設定logstash檔案新增httpd的索引

[root@localhost ELK]# cd /etc/logstash/conf.d/
[root@localhost conf.d]# vim httpd.conf
input {
        redis {
                data_type => "list"
                host => "192.168.232.135"
                password => "123321"
                port => "6379"
                db => "1"
                key => "filebeat-httpd"
        }
}

output {

         elasticsearch {
                hosts => ["192.168.232.135:9200"]
                index => "redis-httpdlog-%{+YYYY.MM.dd}"




         }
}

12.開啟logstash並驗證：

1.[root@localhost conf.d]# systemctl start logstash
2.[root@localhost conf.d]# netstat -nltp |grep 9600

13.安裝kibana：

1.[root@localhost ELK]# rpm -ivh kibana-6.6.2-x86_64.rpm

13.1.修改kibana組態檔：

[root@localhost ELK]# vim /etc/kibana/kibana.yml

13.2.開啟kibana服務並驗證：

[root@localhost ELK]# systemctl start kibana
[root@localhost ELK]# netstat -nltp |grep 5601

14.http://192.168.232.135:5601/瀏覽器存取kibana頁面：




附加：如果進入頁面之後索引出不來就進入httpd網頁重新整理頁面httpd索引就會出來了