AWS Elasticsearch Service初探

创建Elasticsearch域

通过VPC内网访问ES的VPC地址,方法有很多种可以使用负载均衡进行转发访问

访问成功看到熟悉的You Know, for Search

导入日志数据

使用filebeat和elasticsearch的pipeline进行数据导入,注意AWS提供的VPC地址实际上是个负载均衡其elasticsearch所在端口为443,filebeat必须使用filebeat-oss不然会出现x_pack认证问题

使用kibana console创建pipeline

access log pipeline

PUT _ingest/pipeline/weblog_combined
{
    "description": "Ingest pipeline for Combined Log Format",
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            """%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}"""
          ]
        }
      },
      {
        "date": {
          "field": "timestamp",
          "formats": [
            "dd/MMM/YYYY:HH:mm:ss Z"
          ]
        }
      },
      {
        "user_agent": {
          "field": "agent"
        }
      }
    ]
}

error log pipeline

PUT _ingest/pipeline/weblog_nginx_error
{
  "description": "Ingest pipeline Nginx error logs",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          """^(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER:threadid}\:( \*%{NUMBER:connectionid})? %{DATA:message}(,|$)( client: %{IPORHOST:client})?(, server: %{IPORHOST:server})?(, request: "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion}))")?(, upstream: "%{DATA:upstream}")?(, host: "%{IPORHOST:vhost}")?"""
        ]
      }
    },
    {
      "date": {
        "field": "timestamp",
        "formats": [
          "YYYY/MM/dd HH:mm:ss"
        ]
      }
    }
  ]
}

修改filebeat配置文件

---
filebeat.inputs:
  -
    enabled: true
    exclude_lines:
      - GET.*ELB-HealthChecker\/2.0
    fields:
      index_name: weblog_access
    paths:
      - /home/wwwlogs/access.log
    pipeline: weblog_combined
    tags:
      - weblogs
      - nginx
    type: log
  -
    enabled: true
    exclude_lines:
      - "newsyslog\\[.*\\]: logfile turned over"
    fields:
      index_name: weblog_nginx_error
    paths:
      - /home/wwwlogs/nginx_error.log
    pipeline: weblog_nginx_error
    tags:
      - weblogs
      - nginx
    type: log
logging.to_files: false
logging.to_syslog: true
logging.level: debug
output.elasticsearch:
  hosts:
    - "https://:443"
  index: "%{[fields.index_name]:logs}-%{+YYYY.MM.dd}-fbt_%{[agent.version]}"
  protocol: https
  ssl.verification_mode: none
setup.template.enabled: false
setup.ilm.enabled: false

启动filebeat观察输出如有错误需要清洗log或者更改pipeline patterns

filebeat -e

from Nginx Logs to Elasticsearch (in AWS) Using Pipelines and Filebeat (no Logstash)

删除文件中多匹配字符串行

grep -F -v test example.txt > example.txt.tmp && mv example.txt.tmp example.txt

其思路为通过字符串查找匹配行然后输出不匹配的行到临时文件,然后再用临时文件覆盖原文件达到删除匹配字符串行的效果.

替换war包中的文件

  • jar tvf 查找war包中的文件
    jar tvf sw-V2.0.1.war | grep AppValues
    
    13908 Tue Jun 30 10:27:42 CST 2020 WEB-INF/classes/cn/com/xx/xx/common/constant/AppValues.class
  • jar xvf 解压完整路径
jar xvf sw-V2.0.1.war WEB-INF/classes/cn/com/xx/xx/common/constant/AppValues.class

inflated: WEB-INF/classes/cn/com/xx/xx/common/constant/AppValues.class
  • 替换文件
  • jar uvf 压缩至war包
    jar uvf sw-V2.0.1.war WEB-INF/classes/cn/com/xx/xx/common/constant/AppValues.class
    
    adding: WEB-INF/classes/cn/com/xx/xx/common/constant/AppValues.class(in = 13894) (out= 5232)(deflated 62%)

在内网如何知道其他ip的MAC地址

 arp -a
? (192.168.1.169) at 48:a1:95:c3:d8:14 [ether] on eno1
? (172.17.0.2) at 02:42:ac:11:00:02 [ether] on docker0
? (192.168.1.243) at 50:7b:9d:0b:4c:52 [ether] on eno1
? (192.168.1.193) at 64:b0:a6:ca:85:d0 [ether] on eno1
? (192.168.1.190) at 50:e5:49:c6:fe:3e [ether] on eno1
? (192.168.1.234) at e0:89:7e:b3:7c:3f [ether] on eno1
gateway (192.168.1.1) at 08:57:00:a9:06:d7 [ether] on eno1
? (192.168.1.241) at 54:bf:64:40:11:c7 [ether] on eno1
? (192.168.1.185) at 80:ad:16:47:a9:01 [ether] on eno1
? (192.168.1.212) at 4a:1d:9b:f9:41:df [ether] on eno1
? (192.168.1.78) at 80:18:44:e8:91:fc [ether] on eno1
? (192.168.1.240) at 7c:76:68:ce:eb:ec [ether] on eno1

注意缓存

Auto start service on Boot

CentOS or RHEL 6.x

chkconfig --add httpd
chkconfig httpd on
chkconfig --list 可以看到其他启动服务和启动级别

/etc/rc.d/init.d下可观察启动脚本

/etc/centos-release 是否存在观察centOS版本

RHEL or CentOS 7.x

systemctl enable php-fpm