ELK8.8部署安裝並設定xpark認證

ELK8.8部署安裝並設定xpark認證

• 介紹

  主要記錄下filebeat+logstash+elasticsearch+kibana抽取過濾儲存展示應用紀錄檔檔案的方式；版本基於8.8，並開啟xpack安全認證。由於從7.X開始就自帶JDK，故這裡也不展示環境設定等步驟。

• 下載服務

elasticsearch：https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.8.1-linux-x86_64.tar.gz
kibana:https://artifacts.elastic.co/downloads/kibana/kibana-8.8.1-linux-x86_64.tar.gz
filebeat:https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.8.1-linux-x86_64.tar.gz
logstash:https://artifacts.elastic.co/downloads/logstash/logstash-8.8.1-linux-x86_64.tar.gz

• 環境介紹

• 部署elasticsearch

1. 由於elasticsearch無法用root使用者啟動，這裡建立一個普通使用者elk，後續操作將在此使用者下進行；

useradd -d /home/elk -m elk
echo '123@qwe'|passwd elk --stdin

2. 安裝es

- 建立es資料目錄和log目錄；
mkdir /data/elk/elasticsearch/{data,log}

- 解壓安裝包
tar -zxvf elasticsearch-8.8.1-linux-x86_64.tar.gz
- 進入config目錄，修改組態檔elasticsearch.yml；
cd elasticsearch-8.8.1/config

- 修改組態檔取消以下注釋並設定；
vim elasticsearch.yml
cluster.name: my-application
node.name: node-1
path.data: /data/elk/elasticsearch/data
path.logs: /data/elk/elasticsearch/logs
network.host: 0.0.0.0
http.port: 9200

- 首次啟動不要後臺啟動；
pwd
/home/elk/elasticsearch-8.8.1/config
cd /home/elk/elasticsearch-8.8.1/bin
./elasticsearch
**前臺紀錄檔輸出最後內容<要記錄下來>**：
✅ Elasticsearch security features have been automatically configured!
✅ Authentication is enabled and cluster connections are encrypted.

ℹ️  Password for the elastic user (reset with `bin/elasticsearch-reset-password -u elastic`):
  2j6qweqeRqnAnPGU61

ℹ️  HTTP CA certificate SHA-256 fingerprint:
  09189c0bb24353451b32f603d509272d591sad123815b1233d7ae

ℹ️  Configure Kibana to use this cluster:
• Run Kibana and click the configuration link in the terminal when Kibana starts.
• Copy the following enrollment token and paste it into Kibana in your browser (valid for the next 30 minutes):
  eyJ2ZXIiOiI4LjguMSIsImFkciI6WyIxMC4yNTMuMTc3LjkyOjkyMDAiXSwiZmdyIjoiMDkxODljMGJiMjc4NDE4YTIyNjE4YjBlN2M5OGIzMmY2MDNkNTA5MjcyZDU5MWZiNzkwMDQzODE1YjY3ZDdhZSIsImtleSI6Im02ckE5WWdCUEJtZ2J3czVUWU14OjRUYVliMi1SUWFHSlVlRWJaYk5NUVEifQ==

ℹ️ Configure other nodes to join this cluster:
• Copy the following enrollment token and start new Elasticsearch nodes with `bin/elasticsearch --enrollment-token <token>` (valid for the next 30 minutes):
  eyJ2ZXIiOiI4LjguMSIsImFkciI6WyIxMC4yNTMuMTc3LjkyOjkyMDAiXSwiZmdyIjoiMDkxODljMGJiMjc4NDE4YTIyNjE4YjBlN2M5OGIzMmY2MDNkNTA5MjcyZDU5MWZiNzkwMDQzODE1YjY3ZDdhZSIsImtleSI6Im1xckE5WWdCUEJtZ2J3czVUWU12Omt1aEdkVXAzUTA2LUpqOVNmMWkweEEifQ==

  If you're running in Docker, copy the enrollment token and run:
  `docker run -e "ENROLLMENT_TOKEN=<token>" docker.elastic.co/elasticsearch/elasticsearch:8.8.1`


- 重新開啟一個對談視窗，再次檢視elasticsearch.yml設定，會發現多了xpack安全認證；

    ```
    # Enable security features
    xpack.security.enabled: true
    
    xpack.security.enrollment.enabled: true
    
    xpack.monitoring.collection.enabled: true
    
    # Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
    xpack.security.http.ssl:
      enabled: true
      keystore.path: certs/http.p12
    
    # Enable encryption and mutual authentication between cluster nodes
    xpack.security.transport.ssl:
      enabled: true
      verification_mode: certificate
      keystore.path: certs/transport.p12
      truststore.path: certs/transport.p12
    # Create a new cluster with the current node only
    # Additional nodes can still join the cluster later
    cluster.initial_master_nodes: ["node-1"]
    ```
    同時/home/elk/elasticsearch-8.8.1/config目錄下增加一個certs目錄，裡面有以下內容；
    ll certs/
    總用量 24
    -rw-rw---- 1 elk elk 1915 6月  26 11:29 http_ca.crt
    -rw-rw---- 1 elk elk 9997 6月  26 11:29 http.p12
    -rw-rw---- 1 elk elk 5822 6月  26 11:29 transport.p12
    
- 關閉elasticsearch服務，並後臺啟動；
[elk@host-172-16-0-1 config]$ ps -ef|grep elasticsearch|grep -v grep|awk '{print $2}'|xargs kill
[elk@host-172-16-0-1 config]$ cd  ../bin/
[elk@host-172-16-0-1 bin]$ ./elasticsearch -d 

- 初始化elasticsearch內建kibana使用者密碼；
./elasticsearch-reset-password -u kibaina

- 瀏覽器存取https://172.16.0.1:9200，輸入elastic密碼2j6qweqeRqnAnPGU61：

3. 安裝kibana

- 解壓安裝包；
tar -zxvf kibana-8.8.1-linux-x86_64.tar.gz
cd  kibana-8.8.1/config

- 拷貝es目錄下certs資料夾到config下；
\cp  -rf /home/elk/elasticsearch-8.8.1/config/certs ./

- 修改kibana.yml，取消以下注釋並設定；
vim kibana.yml
server.port: 5601
server.host: "172.16.0.1"
elasticsearch.hosts: ["https://172.16.0.1:9200"]
elasticsearch.username: "kibana"    # es內建使用者；
elasticsearch.password: "pkRqnAnPGU61123"  # es初始化的的密碼；
elasticsearch.ssl.certificateAuthorities: [ "/home/elk/kibana-8.8.1/config/certs/http_ca.crt" ]
i18n.locale: "zh-CN"

- 啟動kibana
cd  /home/elk/kibana-8.8.1/
nohup ./bin/kibana &

- 存取http://10.253.177.92:5601/login，輸入elastic密碼2j6qweqeRqnAnPGU61：

4. 安裝logstash

- 解壓安裝包；
tar -zxvf logstash-8.8.1-linux-x86_64.tar.gz
cd logstash-8.8.1/config/

- 拷貝es目錄下certs資料夾到config下；
\cp  -rf /home/elk/elasticsearch-8.8.1/config/certs ./

- 修改組態檔logstash.yml；
vim logstash.yml
http.host: "0.0.0.0"
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: 2j6qweqeRqnAnPGU61
xpack.monitoring.elasticsearch.hosts: ["https://172.16.0.1:9200"]
xpack.monitoring.elasticsearch.ssl.certificate_authority: "/home/elk/logstash-8.8.1/config/certs/http_ca.crt"
xpack.monitoring.elasticsearch.ssl.ca_trusted_fingerprint: 09189c0bb278418a22618b0e7c98b32f603d509272d591fb790043815b67d7ae 

- 修改logstash-sample.conf設定；
vim logstash-sample.conf
input {
  beats {
    port => 5041
  }
}
output {
  elasticsearch {
    hosts => ["https://172.16.0.1:9200"]
    #index => "%{[fields][service_name]}-%{+YYYY.MM.dd}"
    ssl => true
    ssl_certificate_verification => false
    cacert => "/home/elk/logstash-8.8.1/config/certs/http_ca.crt"
    ca_trusted_fingerprint => "09189c0bb278418a22618b0e7c98b32f603d509272d591fb790043815b67d7ae"
    user => "elastic"
    password => "2j6qweqeRqnAnPGU61"
  }
  stdout {codec => rubydebug}
}

- 啟動logstash；
nohup ./bin/logstash -f /home/elk/logstash-8.8.1/config/logstash-sample.conf &

5. 安裝filebeat

- 分別登入172.16.0.2/3兩臺應用伺服器/data目錄；
mkdir filebeat
tar -zxvf filebeat-8.8.1-linux-x86_64.tar.gz
cd filebeat-8.8.1-linux-x86_64

- 修改filebeat.yml組態檔
vim filebeat.yml
filebeat.inputs:
 - type: log
 id: 1
 enable: true
 paths: 
   - /data/app/ap/logs/*.log  # 要採集的紀錄檔檔案或路徑
# output.elasticsearch:  # 由於本文架構是filebeat的output是到logstash，故關閉預設output.elasticsearch；
output.logstash:
    hosts: ["172.16.0.1:5041"]  # 這裡的埠要與logstash-sample.conf設定裡的一致；

- 設定完成，臨時啟動filebeat；
nohup ./filebeat -e -c filebeat.yml > /dev/null 2>&1

- 由於通過nohub方式啟動filebeat，執行一段時間後filebeat自動退出；原因是filebeat預設會定期檢測檔案是否有新的內容，如果超過一定時間檢測的檔案沒有新資料寫入，那麼filebeat會自動退出，解決辦法就是將filebeat通過系統後臺的方式長期執行；
    - 新增systemctl服務啟動設定
    vim  /etc/systemd/system/filebeat.service
    
    [Unit]
    Description=Filebeat is a lightweight shipper for metrics.
    Documentation=https://www.elastic.co/products/beats/filebeat
    Wants=network-online.target
    After=network-online.target
    
    [Service]
    Environment="LOG_OPTS=-e"
    Environment="CONFIG_OPTS=-c /data/filebeat/filebeat-8.8.1-linux-x86_64/filebeat.yml"
    Environment="PATH_OPTS=-path.home /data/filebeat/filebeat-8.8.1-linux-x86_64/filebeat -path.config /data/filebeat/fileb
    eat-8.8.1-linux-x86_64 -path.data /data/filebeat/filebeat-8.8.1-linux-x86_64/data -path.logs /data/filebeat/filebeat-8.
    8.1-linux-x86_64/logs"
    ExecStart=/data/filebeat/filebeat-8.8.1-linux-x86_64/filebeat $LOG_OPTS $CONFIG_OPTS $PATH_OPTS
    Restart=always
    
    [Install]
    WantedBy=multi-user.target
    
    - 授予可執行許可權
    chmod +x /etc/systemd/system/filebeat.service
    
    - 設定開機啟動等
    systemctl daemon-reload
    systemctl enable filebeat
    systemctl start filebeat

• 登入kibana檢視