【优化篇】telegraf+shell脚本实现秒级的服务状态异常监测与告警

一、场景

在《【shell脚本】Graylog服务状态异常监测与告警》文章脚本的基础上，修改成一个检测GrayLog5.1版本opensearch状态的脚本

脚本如下

vim /opt/check_opensearch_status.sh

#!/bin/bashLOCK_FILE=/var/log/opensearch_record.log#钉钉机器人WebhookURLWEBHOOK_URL="https://oapi.dingtalk.com/robot/send?access_token=XXXXXXXXXXXXXXXXXXXXXXX"#记录异常状态的文件路径STATUS_FILE="/tmp/opensearch_status.txt"NORMAL_STATUS_FILE="/opt/opensearch_normal_status.txt"#echo"Active:active(running)">/opt/opensearch_normal_status.txt#获取当前时间current_time=$(date+"%Y-%m-%d%H:%M:%S")#获取当前opensearch状态current_status=$(/usr/bin/systemctlstatusopensearch|grepActive|awk-F"since"'{print$1}'|sed's/^*//;s/*$//')#读取上次记录的状态previous_status=$(cat"$STATUS_FILE")#正常opensearch状态#Active:active(running)normal_status=$(cat"$NORMAL_STATUS_FILE")#判断当前状态是否与上次记录的状态不一致if[["$current_status"=="$normal_status"&&"$previous_status"!="$normal_status"]];then#发送恢复正常告警到钉钉机器人echo"--------------------------------">>${LOCK_FILE}2>&1echo`date+"%Y-%m-%d%H:%M:%S"`>>${LOCK_FILE}2>&1echo"服务状态已恢复运行，发送dingding告警">>${LOCK_FILE}2>&1recovery_message="【告警通知】：$(hostname-I)的opensearch 状态已恢复正常。nn【时间】：$current_timenn【opensearch状态】：<font color=#67C23A>$current_status</font>"echo$recovery_message>>${LOCK_FILE}2>&1curl-s-H"Content-Type:application/json"-d"{"msgtype":"markdown","markdown":{"title":"告警通知","text":"$recovery_message"}}""$WEBHOOK_URL"#将当前状态记录到文件echo"$current_status">"$STATUS_FILE"echo"--------------------------------">>${LOCK_FILE}2>&1echo"$current_status"elif[["$current_status"!="$previous_status"]];then#发送异常告警到钉钉机器人echo"--------------------------------">>${LOCK_FILE}2>&1echo`date+"%Y-%m-%d%H:%M:%S"`>>${LOCK_FILE}2>&1echo"服务状态异常，发送dingding告警">>${LOCK_FILE}2>&1alert_message="【告警通知】：$(hostname-I)的opensearch 状态异常告警！nn【时间】：$current_timenn 【opensearch状态】：<font color=#FF0000>$current_status</font>"echo$alert_message>>${LOCK_FILE}2>&1curl-s-H"Content-Type:application/json"-d"{"msgtype":"markdown","markdown":{"title":"告警通知","text":"$alert_message"}}""$WEBHOOK_URL"#将当前状态记录到文件echo"$current_status">"$STATUS_FILE"echo"--------------------------------">>${LOCK_FILE}2>&1echo"$current_status"fi

需求：

原由于Crontab定时任务最多能实现1分钟执行一次的频率

(图片点击放大查看)

如果在1分钟内服务重启后又恢复时这种情况很大的可能是无法监测到状态异常变化的情况

下面借助telegraf的inputs.exec模块实现秒级的状态监测

具体操作如下

1、下载telegraf的rpm包并安装

https://github.com/influxdata/telegraf/releasesrpm-ivhtelegraf-1.27.3-1.x86_64.rpm

(图片点击放大查看)

2、生成配置文件样本并替换修改

telegraf--sample-config--input-filterexec--output-filtergraylog>telegraf.confmv/etc/telegraf/telegraf.conf/etc/telegraf/telegraf.conf_defaultmv./telegraf.conf/etc/telegraf/telegraf.confvim/etc/telegraf/telegraf.conf

(图片点击放大查看)

(图片点击放大查看)

最终的配置文件如下

[root@centosopt]#cat/etc/telegraf/telegraf.conf|grep-v^#|grep-v^$|grep-v^.*##|grep-v^.*#[global_tags][agent]interval="10s"round_interval=truemetric_batch_size=1000metric_buffer_limit=10000collection_jitter="0s"flush_interval="10s"flush_jitter="0s"precision="0s"hostname=""omit_hostname=false[[outputs.graylog]]servers=["udp://127.0.0.1:12201"][[inputs.exec]]commands=["/opt/check_opensearch_status.sh"]timeout="5s"name_override="sshd_service_status_check"data_format="value"data_type="string"interval="10s"

(图片点击放大查看)

3、测试

记得将/tmp/opensearch_status.txt 状态记录文件设置成可写

并且执行

echo "Active: active (running)" > /opt/opensearch_normal_status.txt

chmod777/tmp/opensearch_status.txtsystemctlstopopensearch.servicesystemctlstartopensearch.service

(图片点击放大查看)

效果如下

(图片点击放大查看)

上面的interval = "10s"为10秒，也就是10秒执行一次状态监测脚本，当然你可以设置得更低。例如5s