ResponseBody Not Populated Correctly When Using HTTPX as a Library

[c0r1]

httpx version:

v1.6.10

Current Behavior:

When using HTTPX as a library with the -mdc (Match Condition) option enabled, the ResponseBody attribute of the Result object remains empty, even though the actual HTTP response contains a body. I have confirmed that the server is indeed sending a non-empty response body.

Expected Behavior:

When the HTTP response contains a body, the ResponseBody attribute should not be empty and should reflect the actual content of the response.

Steps To Reproduce:

package main

import (
	"log"

	"github.com/projectdiscovery/goflags"
	"github.com/projectdiscovery/gologger"
	"github.com/projectdiscovery/gologger/levels"
	"github.com/projectdiscovery/httpx/runner"
)

func main() {
	gologger.DefaultLogger.SetMaxLevel(levels.LevelVerbose) // increase the verbosity (optional)

	options := runner.Options{
		Methods:              "GET",
		InputTargetHost:      goflags.StringSlice{"https://projectdiscovery.io"},
		DisableStdin:         true,
		OutputMatchCondition: `status_code == 200 && regex("html", body)`,
	}

	if err := options.ValidateOptions(); err != nil {
		log.Fatal(err)
	}

	httpxRunner, err := runner.New(&options)
	if err != nil {
		log.Fatal(err)
	}
	defer httpxRunner.Close()

	httpxRunner.RunEnumeration()
}

Running the above code does not yield any matching results, even though the response body should match the specified condition (status_code == 200 && regex("html", body)).
However, executing the equivalent HTTPX command-line command correctly produces the expected results.
httpx -u https://projectdiscovery.io -mdc 'status_code == 200 && regex("html", body)'

This command correctly returns the expected results, indicating that the issue is specific to the library usage.

[dogancanbakir]

Do you want to see the result in the output? If yes, you can try using OnResult fn:

package main

import (
	"fmt"
	"log"

	"github.com/projectdiscovery/goflags"
	"github.com/projectdiscovery/gologger"
	"github.com/projectdiscovery/gologger/levels"
	"github.com/projectdiscovery/httpx/runner"
)

func main() {
	gologger.DefaultLogger.SetMaxLevel(levels.LevelVerbose)

	options := runner.Options{
		Methods:              "GET",
		InputTargetHost:      goflags.StringSlice{"https://projectdiscovery.io"},
		DisableStdin:         true,
		OutputMatchCondition: `status_code == 200 && regex("html", body)`,
		OnResult: func(r runner.Result) {
			if r.Err != nil {
				fmt.Printf("[Err] %s: %s\n", r.Input, r.Err)
				return
			}
			fmt.Printf("%s %s %d\n", r.Input, r.Host, r.StatusCode)
		},
	}

	if err := options.ValidateOptions(); err != nil {
		log.Fatal(err)
	}

	httpxRunner, err := runner.New(&options)
	if err != nil {
		log.Fatal(err)
	}
	defer httpxRunner.Close()

	httpxRunner.RunEnumeration()
}

[c0r1]

I'm filtering a batch of URL requests to find those with a 200 status code and a specific string in the response body. The problem is that the regex("html", body) DSL expression always returns an empty body.

[c0r1]

The DSL expression status_code == 200 && regex("html", body) is being compared to the following alternative configuration:

options := runner.Options{
		Methods:         "GET",
		InputTargetHost: goflags.StringSlice{"https://projectdiscovery.io"},
		DisableStdin:    true,
		OutputMatchStatusCode: "200",
		OutputMatchRegex:      goflags.StringSlice{"html"},
		//OutputMatchCondition: `status_code == 200 && regex("html", body)`,

		OnResult: func(r runner.Result) {
			if r.Err != nil {
				fmt.Printf("[Err] %s: %s\n", r.Input, r.Err)
				return
			}
			fmt.Printf("%s %s %d\n", r.Input, r.Host, r.StatusCode)
		},
	}

Using this alternative configuration, the requests are matched correctly

In contrast, using the DSL expression results in an empty match set

options := runner.Options{
		Methods:         "GET",
		InputTargetHost: goflags.StringSlice{"https://projectdiscovery.io"},
		DisableStdin:    true,
		//OutputMatchStatusCode: "200",
		//OutputMatchRegex:      goflags.StringSlice{"html"},
		OutputMatchCondition: `status_code == 200 && regex("html", body)`,

		OnResult: func(r runner.Result) {
			if r.Err != nil {
				fmt.Printf("[Err] %s: %s\n", r.Input, r.Err)
				return
			}
			fmt.Printf("%s %s %d\n", r.Input, r.Host, r.StatusCode)
		},
	}

Investigation revealed that the body parameter, which represents the response body, is always empty within the regex("html", body) DSL expression. This explains why the DSL expression fails to match.

[dogancanbakir]

I couldn't repro the second issue you mentioned with the code I've shared.

[jimen0]

@c0r1 the problem is the current configuration of the runner sets max response body size to read to 0 (int zero value).

Adding the below to your options does the trick. It's not ideal but it seems to be what the current client requires.

MaxResponseBodySizeToRead: int(httpx.DefaultOptions.MaxResponseBodySizeToRead)

[c0r1]

@c0r1 the problem is the current configuration of the runner sets max response body size to read to 0 (int zero value).问题是 Runner 的当前配置将 Max Response Body Size to Read 设置为 0（int 零值）。

Adding the below to your options does the trick. It's not ideal but it seems to be what the current client requires.将以下内容添加到您的选项中可以解决问题。这并不理想，但似乎是当前客户所需要的。

MaxResponseBodySizeToRead: int(httpx.DefaultOptions.MaxResponseBodySizeToRead)

Thanks! I've set it up with the configuration you suggested (MaxResponseBodySizeToRead: int(httpx.DefaultOptions.MaxResponseBodySizeToRead)), and it's working as expected now – I'm getting the desired results.
I really appreciate your help!

[ehsandeep]

Moving this to discussion as we are not tracking SDK/Library usage-related implementations as issues.