目录
-
0x01 目的
-
0x02 配置文件
-
0x03 审计级别-level
-
0x04 审计阶段-stage
-
0x05 审计策略
-
0x06 审计日志样例
-
0x07 告警策略
-
7.1 对容器执行命令时告警
-
7.2 APIServer异常IP访问失败告警
-
参考文档
0x01 目的
通过日志可以达到以下目标:
-
集群里发生的活动。 -
活动的发生时间及发生对象。 -
活动的触发时间、触发位置及观察点。 -
活动的结果以及后续处理行为。
0x02 配置文件
Master节点开启日志审计,API Server配置文件的目录是/etc/kubernetes/manifests/kube-apiserver.yaml
配置 | 说明 |
---|---|
audit-policy-file | 审计日志配置策略文件,文件路径为:/etc/kubernetes/audit-policy.yml。 |
audit-log-path | 审计日志输出路径为/var/log/kubernetes/kubernetes.audit。 |
audit-log-maxsize | 单个审计日志最大内存为100 MB。 |
audit-log-maxbackup | 审计日志最大分片存储10个日志文件。 |
audit-log-maxage | 审计日志最多保存期为7天。 |
0x03 审计级别-level
参数 | 说明 |
---|---|
None | 不记录。 |
Metadata | 记录请求的元数据(例如:用户、时间、资源、操作等),不包括请求和响应的消息体。 |
Request | 除了元数据外,还包括请求消息体,不包括响应消息体。 |
RequestResponse | 记录所有信息,包括元数据以及请求、响应的消息体。 |
0x04 审计阶段-stage
记录日志可以发生在不同的阶段,参考以下表格内容:
参数 | 说明 |
---|---|
RequestReceived | 一收到请求就记录。 |
ResponseStarted | 返回消息头发送完毕后记录,只针对 watch 之类的长连接请求。 |
ResponseComplete | 返回消息全部发送完毕后记录。 |
Panic | 内部服务器出错,请求未完成。 |
0x05 审计策略
Master节点机器,审计配置策略文件的目录是/etc/kubernetes/audit-policy.yml
,内容如下
apiVersion: audit.k8s.io/v1beta1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "RequestReceived"
rules:
# The following requests were manually identified as high-volume and low-risk,
# so drop them.
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
resources:
- group: "" # core
resources: ["endpoints", "services"]
- level: None
users: ["system:unsecured"]
namespaces: ["kube-system"]
verbs: ["get"]
resources:
- group: "" # core
resources: ["configmaps"]
- level: None
users: ["kubelet"] # legacy kubelet identity
verbs: ["get"]
resources:
- group: "" # core
resources: ["nodes"]
- level: None
userGroups: ["system:nodes"]
verbs: ["get"]
resources:
- group: "" # core
resources: ["nodes"]
- level: None
users:
- system:kube-controller-manager
- system:kube-scheduler
- system:serviceaccount:kube-system:endpoint-controller
verbs: ["get", "update"]
namespaces: ["kube-system"]
resources:
- group: "" # core
resources: ["endpoints"]
- level: None
users: ["system:apiserver"]
verbs: ["get"]
resources:
- group: "" # core
resources: ["namespaces"]
# Don't log these read-only URLs.
- level: None
nonResourceURLs:
- /healthz*
- /version
- /swagger*
# Don't log events requests.
- level: None
resources:
- group: "" # core
resources: ["events"]
# Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
# so only log at the Metadata level.
- level: Metadata
resources:
- group: "" # core
resources: ["secrets", "configmaps"]
- group: authentication.k8s.io
resources: ["tokenreviews"]
# Get repsonses can be large; skip them.
- level: Request
verbs: ["get", "list", "watch"]
resources:
- group: "" # core
- group: "admissionregistration.k8s.io"
- group: "apps"
- group: "authentication.k8s.io"
- group: "authorization.k8s.io"
- group: "autoscaling"
- group: "batch"
- group: "certificates.k8s.io"
- group: "extensions"
- group: "networking.k8s.io"
- group: "policy"
- group: "rbac.authorization.k8s.io"
- group: "settings.k8s.io"
- group: "storage.k8s.io"
# Default level for known APIs
- level: RequestResponse
resources:
- group: "" # core
- group: "admissionregistration.k8s.io"
- group: "apps"
- group: "authentication.k8s.io"
- group: "authorization.k8s.io"
- group: "autoscaling"
- group: "batch"
- group: "certificates.k8s.io"
- group: "extensions"
- group: "networking.k8s.io"
- group: "policy"
- group: "rbac.authorization.k8s.io"
- group: "settings.k8s.io"
- group: "storage.k8s.io"
# Default level for all other requests.
- level: Metadata
说明
-
在收到请求后不立即记录日志,当返回体Header发送后才开始记录。 -
对于大量冗余的 kube-proxy
watch请求,kubelet
和system:nodes
对于节点的get请求,kube组件在kube-system
下对于endpoint的操作,以及API Server
对于Namespaces
的get请求等不作审计。 -
对于 /healthz*
,/version*
及/swagger*
等只读URL不作审计。 -
对于可能包含敏感信息或二进制文件的Secrets,ConfigMaps,tokenreviews接口的日志等级设为metadata,该level只记录请求事件的用户、时间戳、请求资源和动作,而不包含请求体和返回体。 -
对于一些如authentication、rbac、certificates、autoscaling、storage等敏感接口,根据读写记录相应的请求体和返回体。
0x06 审计日志样例
{
"kind":"Event",
"apiVersion":"audit.k8s.io/v1",
"level":"RequestResponse",
"auditID":0a4376d5-307a-4e16-a049-24e017******,
"stage":"ResponseComplete",
// 发生了什么
"requestURI":"/apis/apps/v1/namespaces/default/deployments",
"verb":"create",
// 谁发起的
"user":{
"username":"admin",
"uid":"admin",
"groups":[
"system:masters",
"system:authenticated"
]
},
// 从哪里发起
"sourceIPs":[
"10.0.6.68"
],
"userAgent":"kubectl/v1.16.3 (linux/amd64) kubernetes/ald64d8",
// 发生了什么
"objectRef":{
"resource":"deployments",
"namespace":"default",
"name":"nginx-deployment",
"apiGroup":"apps",
"apiVersion":"v1"
},
// 结果是什么
"responseStatus":{
"metadata":{
},
"code":201
},
// 请求及返回具体信息
"requestObject":Object{...},
"responseObject":Object{...},
// 什么时候开始/结束
"requestReceivedTimestamp":"2020-04-10T10:47:34.315746Z",
"stageTimestamp":"2020-04-10T10:47:34.328942Z",
// 请求被接收/拒绝的原因是什么
"annotations":{
"authorization.k8s.io/decision":"allow",
"authorization.k8s.io/reason":""
}
}
0x07 告警策略
7.1 对容器执行命令时告警
目前公司对于Kubernetes集群使用有严格限制,不允许用户登录容器或对容器执行命令,如果有用户执行命令时需要立即给出告警,并希望告警时能够显示用户登录的具体容器、执行的命令、操作人、事件ID、时间、操作源IP等信息
verb : create and objectRef.subresource:exec and stage: ResponseStarted | SELECT auditID as "事件ID", date_format(from_unixtime(__time__), '%Y-%m-%d %T' ) as "操作时间", regexp_extract("requestURI", '([^?]*)/exec?.*', 1)as "资源", regexp_extract("requestURI", '?(.*)', 1)as "命令" ,"responseStatus.code" as "状态码",
CASE
WHEN "user.username" != 'kubernetes-admin' then "user.username"
WHEN "user.username" = 'kubernetes-admin' and regexp_like("annotations.authorization.k8s.io/reason", 'RoleBinding') then regexp_extract("annotations.authorization.k8s.io/reason", ' to User "(w+)"', 1)
ELSE 'kubernetes-admin' END
as "操作账号",
CASE WHEN json_array_length(sourceIPs) = 1 then json_format(json_array_get(sourceIPs, 0)) ELSE sourceIPs END
as "源地址" limit 100
#条件表达式为:操作事件 =~ ".*"
7.2 APIServer异常IP访问失败告警
为防止恶意攻击,需要监控内网访问的次数以及失败率,若访问次数到达一定阈值(10次)且失败率高于一定阈值(50%)则立即告警,并希望告警时能够显示用户的IP所属区域、操作源IP、是否高危IP等信息。
* | select ip as "源地址", total as "访问次数", round(rate * 100, 2) as "失败率%", failCount as "非法访问次数", CASE when security_check_ip(ip) = 1 then 'yes' else 'no' end as "是否高危IP", ip_to_country(ip) as "国家", ip_to_province(ip) as "省", ip_to_city(ip) as "市", ip_to_provider(ip) as "运营商" from (select CASE WHEN json_array_length(sourceIPs) = 1 then json_format(json_array_get(sourceIPs, 0)) ELSE sourceIPs END
as ip, count(1) as total,
sum(CASE WHEN "responseStatus.code" < 400 then 0
ELSE 1 END) * 1.0 / count(1) as rate,
count_if("responseStatus.code" = 403) as failCount
from log group by ip limit 10000) where ip_to_domain(ip) != 'intranet' having "访问次数" > 10 and "失败率%" > 50 ORDER by "访问次数" desc limit 100
#条件表达式为:源地址 =~ ".*"
参考文档
-
腾讯云:https://cloud.tencent.com/document/product/457/48346 -
阿里云:https://help.aliyun.com/document_detail/91406.html
原文始发于微信公众号(小宝的安全学习笔记):企业安全建设SIEM之K8s日志审计策略
免责声明:文章中涉及的程序(方法)可能带有攻击性,仅供安全研究与教学之用,读者将其信息做其他用途,由读者承担全部法律及连带责任,本站不承担任何法律及连带责任;如有问题可邮件联系(建议使用企业邮箱或有效邮箱,避免邮件被拦截,联系方式见首页),望知悉。
- 左青龙
- 微信扫一扫
-
- 右白虎
- 微信扫一扫
-
评论