AWS Elasticsearch Service初探

创建Elasticsearch域

通过VPC内网访问ES的VPC地址,方法有很多种可以使用负载均衡进行转发访问

访问成功看到熟悉的You Know, for Search

导入日志数据

使用filebeat和elasticsearch的pipeline进行数据导入,注意AWS提供的VPC地址实际上是个负载均衡其elasticsearch所在端口为443,filebeat必须使用filebeat-oss不然会出现x_pack认证问题

使用kibana console创建pipeline

access log pipeline

PUT _ingest/pipeline/weblog_combined
{
    "description": "Ingest pipeline for Combined Log Format",
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            """%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}"""
          ]
        }
      },
      {
        "date": {
          "field": "timestamp",
          "formats": [
            "dd/MMM/YYYY:HH:mm:ss Z"
          ]
        }
      },
      {
        "user_agent": {
          "field": "agent"
        }
      }
    ]
}

error log pipeline

PUT _ingest/pipeline/weblog_nginx_error
{
  "description": "Ingest pipeline Nginx error logs",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          """^(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER:threadid}\:( \*%{NUMBER:connectionid})? %{DATA:message}(,|$)( client: %{IPORHOST:client})?(, server: %{IPORHOST:server})?(, request: "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion}))")?(, upstream: "%{DATA:upstream}")?(, host: "%{IPORHOST:vhost}")?"""
        ]
      }
    },
    {
      "date": {
        "field": "timestamp",
        "formats": [
          "YYYY/MM/dd HH:mm:ss"
        ]
      }
    }
  ]
}

修改filebeat配置文件

---
filebeat.inputs:
  -
    enabled: true
    exclude_lines:
      - GET.*ELB-HealthChecker\/2.0
    fields:
      index_name: weblog_access
    paths:
      - /home/wwwlogs/access.log
    pipeline: weblog_combined
    tags:
      - weblogs
      - nginx
    type: log
  -
    enabled: true
    exclude_lines:
      - "newsyslog\\[.*\\]: logfile turned over"
    fields:
      index_name: weblog_nginx_error
    paths:
      - /home/wwwlogs/nginx_error.log
    pipeline: weblog_nginx_error
    tags:
      - weblogs
      - nginx
    type: log
logging.to_files: false
logging.to_syslog: true
logging.level: debug
output.elasticsearch:
  hosts:
    - "https://:443"
  index: "%{[fields.index_name]:logs}-%{+YYYY.MM.dd}-fbt_%{[agent.version]}"
  protocol: https
  ssl.verification_mode: none
setup.template.enabled: false
setup.ilm.enabled: false

启动filebeat观察输出如有错误需要清洗log或者更改pipeline patterns

filebeat -e

from Nginx Logs to Elasticsearch (in AWS) Using Pipelines and Filebeat (no Logstash)

删除文件中多匹配字符串行

grep -F -v test example.txt > example.txt.tmp && mv example.txt.tmp example.txt

其思路为通过字符串查找匹配行然后输出不匹配的行到临时文件,然后再用临时文件覆盖原文件达到删除匹配字符串行的效果.

替换war包中的文件

  • jar tvf 查找war包中的文件
    jar tvf sw-V2.0.1.war | grep AppValues
    
    13908 Tue Jun 30 10:27:42 CST 2020 WEB-INF/classes/cn/com/xx/xx/common/constant/AppValues.class
  • jar xvf 解压完整路径
jar xvf sw-V2.0.1.war WEB-INF/classes/cn/com/xx/xx/common/constant/AppValues.class

inflated: WEB-INF/classes/cn/com/xx/xx/common/constant/AppValues.class
  • 替换文件
  • jar uvf 压缩至war包
    jar uvf sw-V2.0.1.war WEB-INF/classes/cn/com/xx/xx/common/constant/AppValues.class
    
    adding: WEB-INF/classes/cn/com/xx/xx/common/constant/AppValues.class(in = 13894) (out= 5232)(deflated 62%)