CVE漏洞信息与poc链接爬取

admin 2024年5月21日19:57:23评论2 views字数 4291阅读14分18秒阅读模式

前言

之前业务上有个要求,对cve信息和漏洞利用脚本进行爬取,然后再检测目标机器Windows上是否缺少对应的补丁

在这个功能基础上我拓展成了爬取CVE漏洞信息和poc链接的脚本,爬取信息会存进数据库中。

功能

对漏洞产品cve信息的爬取链接为:
https://nvd.nist.gov/vuln/search/results?form_type=Advanced&results_type=overview&search_type=all&isCpeNameSearch=false&cpe_product='           'cpe:/:{}:{}

CVE漏洞信息与poc链接爬取

其中两个{},分别填入厂商和产品名,比如微软的Windows系统,就填入:Microsoft、Windows

之后爬取所有信息(包含漏洞信息和poc),部分代码如下:

def fetch_all_cves(producer, software):    global cve_all    cve_all = []    global herfs_all    herfs_all = []    url = 'https://nvd.nist.gov/vuln/search/results?form_type=Advanced&results_type=overview&search_type=all&isCpeNameSearch=false&cpe_product='           'cpe:/:{}:{}'.format(producer, software)    print(url)    # to get cve number    try:        response = requests.get(url, timeout=60, headers=headers)        if response.status_code == 200:            num = re.findall('"vuln-matching-records-count">(.*)?</strong>', response.text)[0]            msg = 'There are {} cves with {} {}...'.format(num, software, banner)            print(msg)    except:        pass    # fetch all cve no    start_index = index = 0    while start_index < int(num):        url = 'https://nvd.nist.gov/vuln/search/results?form_type=Advanced&results_type=overview&search_type=all&isCpeNameSearch=false&cpe_product='               'cpe:/:{}:{}&startIndex={}'.format(producer, software, start_index)        msg = 'processing page {}/{}...'.format(index + 1, math.ceil(int(num) / 20))        print(msg)        index += 1        start_index = index * 20        try:            response = requests.get(url, timeout=60, headers=headers)            if response.status_code == 200:                cves = re.findall('"vuln-detail-link-d+">(.*)?</a>', response.text)                cve_all.extend(cves)                herfs = 'https://nvd.nist.gov/vuln/detail/' + re.findall('"vuln-detail-link-d+">(.*)?</a>',                                                                         response.text)                herfs_all.extend(herfs)        except:            pass    return url

之后新建数据库pymysql,新建数据库用户名为test,密码为yxc113108

相关信息存进数据库中

CVE漏洞信息与poc链接爬取

之后就是爬取对应的漏洞补丁编号信息,这里使用微软官方的api接口https://api.msrc.microsoft.com/爬取

CVE漏洞信息与poc链接爬取

部分代码如下:

def get_csrf_json(this_month_id, year, date):  # url = "{}cvrf/{}?api-Version={}".format(base_url, THIS_MONTH_ID, YEAR)  url = "{}cvrf/{}?api-Version={}".format(base_url, this_month_id, year)  headers = {'api-key': api_key, 'Accept': 'application/json'}  response = requests.get(url, headers = headers)  if response.content:    try:      data = json.loads(response.content)      for each_product in data["ProductTree"]["FullProductName"]:        productid = each_product['ProductID']        product_name = each_product['Value']        search_productid_sql = "SELECT * FROM win_product_name WHERE product_id='" + str(productid) + "'"        cur.execute(search_productid_sql)        if cur.rowcount > 0:          pass        else:          insert_product_sql = "INSERT IGNORE INTO win_product_name (product_id,product_name,dt) VALUES('" + str(productid) + "','" + str(product_name) + "','" + TODAY + "')"          cur.execute(insert_product_sql)          db.commit()      for each_cve in data["Vulnerability"]:        cve = each_cve["CVE"]        if re.search('ADV',cve):          pass        else:          cve = each_cve["CVE"]          kblist = []          scorelist = []          product_id_list = str(each_cve["ProductStatuses"][0]["ProductID"]).replace("'","")          product_id_list = product_id_list.replace("[","")          product_id_list = product_id_list.replace("]","")          product_id_list = product_id_list.replace(" ","")          for each_kb in each_cve["Remediations"]:            try:              kb_num = each_kb["Description"]["Value"]              if re.search('Click to Run',kb_num) or re.search('Release Notes',kb_num):                pass              else:                kblist.append('KB{}'.format(kb_num))            except Exception as e:              print("kb",e)          kblist=list(set(kblist))          kblist=str(kblist).replace("'","")          kblist=kblist.replace("[","")          kblist=kblist.replace("]","")          kblist=kblist.replace(" ","")          for each_score in each_cve["CVSSScoreSets"]:            try:              scorelist.append(each_score["BaseScore"])            except Exception as e:              print("score",e)          try:            score_mean = format(sum(scorelist)/len(scorelist),'.1f')          except Exception as e:            print("score_mean",e)            score_mean = 0          # insert_cve_sql = "INSERT INTO win_cve_db (cve,score,product_id_list,kb_list,cvrf_id,dt) VALUES("" + str(cve) + "","" + str(score_mean) + "","" + str(product_id_list) + "","" + str(kblist) + "","" + str(THIS_MONTH_ID) + "","" + TODAY + "")"          # print(insert_cve_sql)          insert_cve_sql = "INSERT IGNORE INTO win_cve_db (cve, score, product_id_list, kb_list, cvrf_id, dt) VALUES (%s, %s, %s, %s, %s, %s)"          cve_data = (cve, score_mean, product_id_list, kblist, THIS_MONTH_ID, TODAY)          cur.execute(insert_cve_sql, cve_data)          # cur.execute(insert_cve_sql)          db.commit()      # db.close()    except json.decoder.JSONDecodeError as e:      print("JSON 解析错误:", e)  else:    print("Response content is empty")

补丁也会存入对应的表中,方便后续关联查询

CVE漏洞信息与poc链接爬取

脚本最终的运行结果:

CVE漏洞信息与poc链接爬取

为了方便查看结果,我便用了flask来启动页面,运行main.py,之后访问对应网页链接即可

在网页端显示结果如下:

CVE漏洞信息与poc链接爬取

前端页面参考了开源项目:

https://github.com/Donvink/Spider.BC

只要改改图标就行,多余的页面可以删除

最终代码我整合成了main.zip,获取下载链接~

 

链接:https://pan.baidu.com/s/1Ad-S8f6JEUJWvf-soqotxA 提取码:k67e

原文始发于微信公众号(HackingWiki漏洞感知):漏洞监测 | CVE漏洞信息与poc链接爬取

  • 左青龙
  • 微信扫一扫
  • weinxin
  • 右白虎
  • 微信扫一扫
  • weinxin
admin
  • 本文由 发表于 2024年5月21日19:57:23
  • 转载请保留本文链接(CN-SEC中文网:感谢原作者辛苦付出):
                   CVE漏洞信息与poc链接爬取https://cn-sec.com/archives/2763716.html

发表评论

匿名网友 填写信息