projectdiscovery之nuclei源码阅读

2021年9月8日11:12:08评论293 views字数 11842阅读39分28秒阅读模式

简介

Nuclei is a fast tool for configurable targeted vulnerability scanning based on templates offering massive extensibility and ease of use.

Github: https://github.com/projectdiscovery/nuclei

原理上和以前基于python的POC-T类似，不过nuclei是用Go编写的，并且基于yaml编写模板。

这类的工具挺多的，流程也都大同小异，重要的想让人使用的动力，主要还是来自于生态吧。

nuclei基于社区提供了很多可以白嫖的模板，本着这一点，本文就是记录一下如何在自己扫描器中调用nuclei的模板，以及记录一些有趣的、以及以后可能也会用到的技术细节。

有趣的细节

相同的请求

相同的请求可以合并，就不需要发送两次啦

v2pkgprotocolshttpcluster.go

package http
import (  "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/compare")
// CanCluster returns true if the request can be clustered.//// This used by the clustering engine to decide whether two requests// are similar enough to be considered one and can be checked by// just adding the matcher/extractors for the request and the correct IDs.func (r *Request) CanCluster(other *Request) bool {  if len(r.Payloads) > 0 || len(r.Raw) > 0 || len(r.Body) > 0 || r.Unsafe {    return false  }  if r.Method != other.Method ||    r.MaxRedirects != other.MaxRedirects ||    r.CookieReuse != other.CookieReuse ||    r.Redirects != other.Redirects {    return false  }  if !compare.StringSlice(r.Path, other.Path) {    return false  }  if !compare.StringMap(r.Headers, other.Headers) {    return false  }  return true}

比较模板请求中的method，最大重定向数，是否共享cookie请求，是否重定向
比较请求的path
比较请求的header

compare的细节函数

package compare
import "strings"
// StringSlice 比较两个字符串切片是否相等func StringSlice(a, b []string) bool {	// If one is nil, the other must also be nil.	if (a == nil) != (b == nil) {		return false	}	if len(a) != len(b) {		return false	}	for i := range a {		if !strings.EqualFold(a[i], b[i]) {			return false		}	}	return true}
// StringMap 比较两个字符串map是否相同func StringMap(a, b map[string]string) bool {	// If one is nil, the other must also be nil.	if (a == nil) != (b == nil) {		return false	}	if len(a) != len(b) {		return false	}	for k, v := range a {		if w, ok := b[k]; !ok || !strings.EqualFold(v, w) {			return false		}	}	return true}

client报告

nuclei支持github、gitlab、jira、markdown好几种报告模式，刚开始以为是只报告bug呢，后面知道，发现新的结果也会报告的。

看一下生成markdown的描述

projectdiscovery之nuclei源码阅读

报告的细节很详细，请求细节和返回细节都会报告出来。

headless模拟

nuclei的最新版本支持基于chromium的headless访问，用于直接模拟浏览器访问，在v2pkgprotocolsheadless

使用的库是https://github.com/go-rod/rod

我看源码结构里面定义了很多事件,后面应该是想基于yaml来模拟操作浏览器吧？

没有细看实现的完整度有多少，如果这个实现了，就太厉害了 - =

interface转换

go类型中的interface可以看成是任意类型，但是在使用时需要将他转换成我们指定的类型，nuclei实现了这个方法。未来可能也会用到记录下。

// Taken from https://github.com/spf13/cast.
package types
import (	"fmt"	"strconv"	"strings")
// ToString converts an interface to string in a quick wayfunc ToString(data interface{}) string {	switch s := data.(type) {	case nil:		return ""	case string:		return s	case bool:		return strconv.FormatBool(s)	case float64:		return strconv.FormatFloat(s, 'f', -1, 64)	case float32:		return strconv.FormatFloat(float64(s), 'f', -1, 32)	case int:		return strconv.Itoa(s)	case int64:		return strconv.FormatInt(s, 10)	case int32:		return strconv.Itoa(int(s))	case int16:		return strconv.FormatInt(int64(s), 10)	case int8:		return strconv.FormatInt(int64(s), 10)	case uint:		return strconv.FormatUint(uint64(s), 10)	case uint64:		return strconv.FormatUint(s, 10)	case uint32:		return strconv.FormatUint(uint64(s), 10)	case uint16:		return strconv.FormatUint(uint64(s), 10)	case uint8:		return strconv.FormatUint(uint64(s), 10)	case []byte:		return string(s)	case fmt.Stringer:		return s.String()	case error:		return s.Error()	default:		return fmt.Sprintf("%v", data)	}}
// ToStringSlice casts an interface to a []string type.func ToStringSlice(i interface{}) []string {	var a []string
	switch v := i.(type) {	case []interface{}:		for _, u := range v {			a = append(a, ToString(u))		}		return a	case []string:		return v	case string:		return strings.Fields(v)	case interface{}:		return []string{ToString(v)}	default:		return nil	}}
// ToStringMap casts an interface to a map[string]interface{} type.func ToStringMap(i interface{}) map[string]interface{} {	var m = map[string]interface{}{}
	switch v := i.(type) {	case map[interface{}]interface{}:		for k, val := range v {			m[ToString(k)] = val		}		return m	case map[string]interface{}:		return v	default:		return nil	}}

DSL语法

nuclei的模板语法支持很多静态的匹配条件，regx，word等等，同时也引入了dsl语法，让静态的yaml文件具备了调用函数的特性。

一个nuclei模板

id: CVE-2018-18069
info:  name: Wordpress unauthenticated stored xss  author: nadino  severity: medium  description: process_forms in the WPML (aka sitepress-multilingual-cms) plugin through 3.6.3 for WordPress has XSS via any locale_file_name_ parameter (such as locale_file_name_en) in an authenticated theme-localization.php request to wp-admin/admin.php.  tags: cve,cve2018,wordpress,xss
requests:  - method: POST    path:      - "{{BaseURL}}/wp-admin/admin.php"    body: 'icl_post_action=save_theme_localization&locale_file_name_en=EN"><html xmlns="hacked'
    matchers:      - type: dsl        dsl:          - 'status_code==302 && contains(set_cookie, "_icl_current_admin_language")'

可以看到dsl是一个表达式。

v2pkgoperatorscommondsldsl.go 展现了实现dsl语法的函数细节

projectdiscovery之nuclei源码阅读

匹配模式

识别不同的类型进行不同类型的规则匹配

nuclei使用的是https://github.com/Knetic/govaluate 这个库，上面有基本用法

expression, err := govaluate.NewEvaluableExpression("(mem_used / total_mem) * 100");
parameters := make(map[string]interface{}, 8)parameters["total_mem"] = 1024;parameters["mem_used"] = 512;
result, err := expression.Evaluate(parameters);// result is now set to "50.0", the float64 value.

这个库已经3年没有更新了。后面我在用这个库的时候发现一个bug。。就是dsl的函数参数会与自带的语法冲突，官方方案是使用转义，但是这个对于dsl的人来说太痛苦，连-都要转义是什么滋味？

后面我fork了一份解决了，在使用参数的时候不用管转义的问题了。

https://github.com/boy-hack/govaluate

官方太久没更新，所以也没提pull request

projectfile

projectfile是nuclei提供了可以保存项目的选项。

内部实现是通过一个map保存了所有请求的包以及返回结果，key是对请求体(request struct)序列化后进行sha256运算。

再次读取时初始化这个就好了，其中用到了gob对数据结构进行序列化和反序列化操作。

v2pkgprojectfilehttputil.go

package projectfile
import (	"bytes"	"crypto/sha256"	"encoding/gob"	"encoding/hex"	"io"	"io/ioutil"	"net/http")
func hash(v interface{}) (string, error) {	data, err := marshal(v)	if err != nil {		return "", err	}
	sh := sha256.New()
	_, err = io.WriteString(sh, string(data))	if err != nil {		return "", err	}	return hex.EncodeToString(sh.Sum(nil)), nil}
func marshal(data interface{}) ([]byte, error) {	var b bytes.Buffer	enc := gob.NewEncoder(&b)	err := enc.Encode(data)	if err != nil {		return nil, err	}
	return b.Bytes(), nil}
func unmarshal(data []byte, obj interface{}) error {	dec := gob.NewDecoder(bytes.NewBuffer(data))	err := dec.Decode(obj)	if err != nil {		return err	}
	return nil}
type HTTPRecord struct {	Request  []byte	Response *InternalResponse}
type InternalRequest struct {	Target    string	HTTPMajor int	HTTPMinor int	Method    string	Headers   map[string][]string	Body      []byte}
type InternalResponse struct {	HTTPMajor    int	HTTPMinor    int	StatusCode   int	StatusReason string	Headers      map[string][]string	Body         []byte}
// Unused// func newInternalRequest() *InternalRequest {// 	return &InternalRequest{// 		Headers: make(map[string][]string),// 	}// }
func newInternalResponse() *InternalResponse {	return &InternalResponse{		Headers: make(map[string][]string),	}}
// Unused// func toInternalRequest(req *http.Request, target string, body []byte) *InternalRequest {// 	intReq := newInternalRquest()
// 	intReq.Target = target// 	intReq.HTTPMajor = req.ProtoMajor// 	intReq.HTTPMinor = req.ProtoMinor// 	for k, v := range req.Header {// 		intReq.Headers[k] = v// 	}// 	intReq.Headers = req.Header// 	intReq.Method = req.Method// 	intReq.Body = body
// 	return intReq// }
func toInternalResponse(resp *http.Response, body []byte) *InternalResponse {	intResp := newInternalResponse()
	intResp.HTTPMajor = resp.ProtoMajor	intResp.HTTPMinor = resp.ProtoMinor	intResp.StatusCode = resp.StatusCode	intResp.StatusReason = resp.Status	for k, v := range resp.Header {		intResp.Headers[k] = v	}	intResp.Body = body	return intResp}
func fromInternalResponse(intResp *InternalResponse) *http.Response {	var contentLength int64	if intResp.Body != nil {		contentLength = int64(len(intResp.Body))	}	return &http.Response{		ProtoMinor:    intResp.HTTPMinor,		ProtoMajor:    intResp.HTTPMajor,		Status:        intResp.StatusReason,		StatusCode:    intResp.StatusCode,		Header:        intResp.Headers,		ContentLength: contentLength,		Body:          ioutil.NopCloser(bytes.NewReader(intResp.Body)),	}}
// Unused// func fromInternalRequest(intReq *InternalRequest) *http.Request {// 	return &http.Request{// 		ProtoMinor:    intReq.HTTPMinor,// 		ProtoMajor:    intReq.HTTPMajor,// 		Header:        intReq.Headers,// 		ContentLength: int64(len(intReq.Body)),// 		Body:          ioutil.NopCloser(bytes.NewReader(intReq.Body)),// 	}// }

集成nuclei

为了白嫖nuclei的poc，我们准备在自己的扫描器中集成nuclei，或者兼容它的语法。

以前版本想这么做，要深入到很底层的代码去改（因为很多底层接口都是内部的，外部提供的参数我们不需要），一个文件一个文件去扣，很麻烦。

新版的nuclei好多了，不仅包结构调整为go包的形式，很多类都是interface类型，我们只需要根据interface实现那几个函数就能模拟一个mock的类传入。

而且nuclei的测试用例页提供了参考,如果也想调用nuclei，可以看下面代码的例子。

v2internaltestutilstestutils.go

提供很多mock struct

package testutils
import (	"github.com/logrusorgru/aurora"	"github.com/projectdiscovery/gologger/levels"	"github.com/projectdiscovery/nuclei/v2/pkg/catalog"	"github.com/projectdiscovery/nuclei/v2/pkg/output"	"github.com/projectdiscovery/nuclei/v2/pkg/progress"	"github.com/projectdiscovery/nuclei/v2/pkg/protocols"	"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/protocolinit"	"github.com/projectdiscovery/nuclei/v2/pkg/types"	"go.uber.org/ratelimit")
// Init initializes the protocols and their configurationsfunc Init(options *types.Options) {	_ = protocolinit.Init(options)}
// DefaultOptions is the default options structure for nuclei during mocking.var DefaultOptions = &types.Options{	RandomAgent:          false,	Metrics:              false,	Debug:                false,	DebugRequests:        false,	DebugResponse:        false,	Silent:               false,	Version:              false,	Verbose:              false,	NoColor:              true,	UpdateTemplates:      false,	JSON:                 false,	JSONRequests:         false,	EnableProgressBar:    false,	TemplatesVersion:     false,	TemplateList:         false,	Stdin:                false,	StopAtFirstMatch:     false,	NoMeta:               false,	Project:              false,	MetricsPort:          0,	BulkSize:             25,	TemplateThreads:      10,	Timeout:              5,	Retries:              1,	RateLimit:            150,	BurpCollaboratorBiid: "",	ProjectPath:          "",	Severity:             []string{},	Target:               "",	Targets:              "",	Output:               "",	ProxyURL:             "",	ProxySocksURL:        "",	TemplatesDirectory:   "",	TraceLogFile:         "",	Templates:            []string{},	ExcludedTemplates:    []string{},	CustomHeaders:        []string{},}
// MockOutputWriter is a mocked output writer.type MockOutputWriter struct {	aurora          aurora.Aurora	RequestCallback func(templateID, url, requestType string, err error)	WriteCallback   func(o *output.ResultEvent)}
// NewMockOutputWriter creates a new mock output writerfunc NewMockOutputWriter() *MockOutputWriter {	return &MockOutputWriter{aurora: aurora.NewAurora(false)}}
// Close closes the output writer interfacefunc (m *MockOutputWriter) Close() {}
// Colorizer returns the colorizer instance for writerfunc (m *MockOutputWriter) Colorizer() aurora.Aurora {	return m.aurora}
// Write writes the event to file and/or screen.func (m *MockOutputWriter) Write(result *output.ResultEvent) error {	if m.WriteCallback != nil {		m.WriteCallback(result)	}	return nil}
// Request writes a log the requests trace logfunc (m *MockOutputWriter) Request(templateID, url, requestType string, err error) {	if m.RequestCallback != nil {		m.RequestCallback(templateID, url, requestType, err)	}}
// TemplateInfo contains info for a mock executed template.type TemplateInfo struct {	ID   string	Info map[string]interface{}	Path string}
// NewMockExecuterOptions creates a new mock executeroptions structfunc NewMockExecuterOptions(options *types.Options, info *TemplateInfo) *protocols.ExecuterOptions {	progressImpl, _ := progress.NewStatsTicker(0, false, false, 0)	executerOpts := &protocols.ExecuterOptions{		TemplateID:   info.ID,		TemplateInfo: info.Info,		TemplatePath: info.Path,		Output:       NewMockOutputWriter(),		Options:      options,		Progress:     progressImpl,		ProjectFile:  nil,		IssuesClient: nil,		Browser:      nil,		Catalog:      catalog.New(options.TemplatesDirectory),		RateLimiter:  ratelimit.New(options.RateLimit),	}	return executerOpts}
// NoopWriter is a NooP gologger writer.type NoopWriter struct{}
// Write writes the data to an output writer.func (n *NoopWriter) Write(data []byte, level levels.Level) {}

v2pkgprotocolshttpbuild_request_test.go

一个例子。

func TestMakeRequestFromModal(t *testing.T) {	options := testutils.DefaultOptions
	testutils.Init(options)	templateID := "testing-http"	request := &Request{		ID:     templateID,		Name:   "testing",		Path:   []string{"{{BaseURL}}/login.php"},		Method: "POST",		Body:   "username=test&password=pass",		Headers: map[string]string{			"Content-Type":   "application/x-www-form-urlencoded",			"Content-Length": "1",		},	}	executerOpts := testutils.NewMockExecuterOptions(options, &testutils.TemplateInfo{		ID:   templateID,		Info: map[string]interface{}{"severity": "low", "name": "test"},	})	err := request.Compile(executerOpts)	require.Nil(t, err, "could not compile http request")
	generator := request.newGenerator()	req, err := generator.Make("https://example.com", map[string]interface{}{})	require.Nil(t, err, "could not make http request")
	bodyBytes, _ := req.request.BodyBytes()	require.Equal(t, "/login.php", req.request.URL.Path, "could not get correct request path")	require.Equal(t, "username=test&password=pass", string(bodyBytes), "could not get correct request body")}

最后

我对于yaml的poc始终感觉怪怪的，但也渐渐明白一个运营安全社区的道理。想让别人接受，得要先把工具和生态做好，此时不要想着别人回赠。等别人用得舒服了，自然就会回赠了，这是一个自然而然的过程，但是需要时间去累积吧。

微信公众号的排版有点问题，可以点击阅读全文到我博客上看详细内容。

本文始发于微信公众号（Hacking就是好玩）：projectdiscovery之nuclei源码阅读

左青龙
微信扫一扫

右白虎
微信扫一扫

projectdiscovery之nuclei源码阅读

简介

有趣的细节

相同的请求

client报告

headless模拟

interface转换

DSL语法

projectfile

集成nuclei

最后

开源SOAR的初探

pbootcms V3.1.2 "虚假的无文件落地RCE"

cc链之cc3分析

写的第二款工具 > DLL_Finder 白加黑_DLL挖掘辅助工具

基于 dind 开发一个远程代码运行器

如何理解进程调度机制？

利用GPT进行代码审计-Beelt模板注入

必学|并发保姆级教程（Turbo Intruder）

用Coze搞一个简单的安全AI助手

POC&EXP编写—常用模块

发表评论

在线咨询

微信