AWS Elasticsearch Service初探

创建Elasticsearch域

通过VPC内网访问ES的VPC地址,方法有很多种可以使用负载均衡进行转发访问

访问成功看到熟悉的You Know, for Search

导入日志数据

使用filebeat和elasticsearch的pipeline进行数据导入,注意AWS提供的VPC地址实际上是个负载均衡其elasticsearch所在端口为443,filebeat必须使用filebeat-oss不然会出现x_pack认证问题

使用kibana console创建pipeline

access log pipeline

PUT _ingest/pipeline/weblog_combined
{
    "description": "Ingest pipeline for Combined Log Format",
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            """%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}"""
          ]
        }
      },
      {
        "date": {
          "field": "timestamp",
          "formats": [
            "dd/MMM/YYYY:HH:mm:ss Z"
          ]
        }
      },
      {
        "user_agent": {
          "field": "agent"
        }
      }
    ]
}

error log pipeline

PUT _ingest/pipeline/weblog_nginx_error
{
  "description": "Ingest pipeline Nginx error logs",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          """^(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER:threadid}\:( \*%{NUMBER:connectionid})? %{DATA:message}(,|$)( client: %{IPORHOST:client})?(, server: %{IPORHOST:server})?(, request: "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion}))")?(, upstream: "%{DATA:upstream}")?(, host: "%{IPORHOST:vhost}")?"""
        ]
      }
    },
    {
      "date": {
        "field": "timestamp",
        "formats": [
          "YYYY/MM/dd HH:mm:ss"
        ]
      }
    }
  ]
}

修改filebeat配置文件

---
filebeat.inputs:
  -
    enabled: true
    exclude_lines:
      - GET.*ELB-HealthChecker\/2.0
    fields:
      index_name: weblog_access
    paths:
      - /home/wwwlogs/access.log
    pipeline: weblog_combined
    tags:
      - weblogs
      - nginx
    type: log
  -
    enabled: true
    exclude_lines:
      - "newsyslog\\[.*\\]: logfile turned over"
    fields:
      index_name: weblog_nginx_error
    paths:
      - /home/wwwlogs/nginx_error.log
    pipeline: weblog_nginx_error
    tags:
      - weblogs
      - nginx
    type: log
logging.to_files: false
logging.to_syslog: true
logging.level: debug
output.elasticsearch:
  hosts:
    - "https://:443"
  index: "%{[fields.index_name]:logs}-%{+YYYY.MM.dd}-fbt_%{[agent.version]}"
  protocol: https
  ssl.verification_mode: none
setup.template.enabled: false
setup.ilm.enabled: false

启动filebeat观察输出如有错误需要清洗log或者更改pipeline patterns

filebeat -e

from Nginx Logs to Elasticsearch (in AWS) Using Pipelines and Filebeat (no Logstash)

发表评论

电子邮件地址不会被公开。 必填项已用*标注