Web Reconnaissance with httpx

Rasoul

Rasoul

— 5 min read
Web Reconnaissance with httpx

In the fast-paced world of web application security, reconnaissance is the cornerstone of any successful penetration test or bug bounty hunt. It’s the phase where you gather intelligence, map out the attack surface, and pinpoint potential vulnerabilities. Enter httpx, a lightning-fast, feature-packed tool developed by ProjectDiscovery. Written in Go, httpx excels at manipulating HTTP requests and filtering responses, making it an indispensable asset for hackers, programmers, and cybersecurity professionals. While tools like curl can perform similar tasks, httpx shines with its speed, simplicity, and extensive feature set. This guide transforms a raw user-provided overview into a polished, in-depth exploration of httpx, complete with practical examples and technical precision.

Installation

To harness the power of httpx, you’ll need Go version 1.21 or later installed, as it’s a dependency for building and running the tool.

Setting Up Go

Here’s how to install Go on a Linux system (e.g., Kali on amd64 architecture):

Configure Environment Variables:
Add these lines to your ~/.zshrc (or ~/.bashrc if using Bash):

# Go environment variables
export GOPATH=/root/go-workspace
export GOROOT=/usr/local/go
export GOBIN=$GOPATH/bin
export PATH=$GOBIN:$GOROOT/bin:$PATH

Apply the changes:

  source ~/.zshrc

Verify the installation:

go version

Expected output: go version go1.24.0 linux/amd64.

Download Go :

wget https://go.dev/dl/go1.24.0.linux-amd64.tar.gz

Extract it to /usr/local/:

tar -C /usr/local/ -xzf go1.24.0.linux-amd64.tar.gz

Note: Adjust the download link and paths based on your OS and architecture. Visit go.dev/dl for alternatives.

Installing httpx

With Go ready, install httpx using this simple command:

go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest

Check if it’s installed correctly by accessing the help menu:

httpx --help

Basic Usage

httpx accepts input via STDIN, making it perfect for piping in URLs or domains. Start with a basic scan:

echo "http://testphp.vulnweb.com" | httpx

This performs a minimal probe. Enhance it with options for richer output:

echo "http://testphp.vulnweb.com" | httpx -title -status-code -tech-detect -follow-redirects
  • -title: Extracts the webpage title.
  • -status-code (-sc): Displays the HTTP response code (e.g., 200 OK, 404 Not Found).
  • -tech-detect: Identifies technologies powering the site.
  • -follow-redirects: Follows redirects to the final URL.

To scan multiple targets, use the -l flag with a file:

httpx -l list.txt -title -status-code -tech-detect -follow-redirects

Subdomain Enumeration with subfinder

Pair httpx with subfinder (another ProjectDiscovery gem) to enumerate and scan subdomains:

subfinder -d vulnweb.com | httpx -title -status-code -tech-detect -follow-redirects

This workflow discovers subdomains and immediately probes them, revealing live hosts and their details, which are crucial for expanding your attack surface.

Content Probes

httpx provides a suite of probes to dissect HTTP responses. Let’s break them into categories.

Basic Probes

  • -sc: Shows the HTTP status code.
  • -path: Tests if a specific path exists.

Example: Check for /robots.txt:

httpx -l list.txt -path /robots.txt -sc

httpx can run using docker as well. Here, we feed a list of all subdomains as STDIN to httpx:

cat list.txt | docker run -i projectdiscovery/httpx -title -status-code -tech-detect -follow-redirects

Advanced Probes

  • -location: Reveals redirect destinations.
  • -cl: Displays content length.
  • -ct: Shows content type (e.g., text/html).

Example: Analyze redirects and content:

echo "http://google.com" | httpx -sc -cl -ct -location

Content Analysis Probes

  • -favicon: Computes the mmh3 hash of /favicon.ico (handy for identifying frameworks).
  • -rt: Measures response time (useful for spotting server load).
  • -server: Shows the server header (e.g., Apache, Nginx).
  • -hash: Generates a content hash (e.g., SHA256) to detect changes.
  • -probe: Indicates scan status (success/failed).
  • -ip: Lists the server’s IP address.
  • -cdn: Detects CDNs or WAFs.
  • -lc: Counts lines in the response body.
  • -wc: Counts words in the response body.

Example: Deep dive into a target:

echo "http://testphp.vulnweb.com" | httpx -favicon -rt -server -hash sha256 -probe -ip -cdn -lc -wc

Content Comparers and Filters

Narrow down results with comparers and filters for precision targeting.

Matching Filters

  • -mc: Matches specific status codes.
  • -mlc: Matches line counts.
  • -ml: Matches content lengths.
  • -mwc: Matches word counts.
  • -ms: Matches strings in the response.
  • -er: Extracts content via regex.

Example: Find pages with status 200 and “login” text:

cat list.txt | httpx -mc 200 -ms "login"

Example: Extract regex matches:

echo "http://testphp.vulnweb.com" | httpx -er "\w test"

Excluding Filters

  • -fc: Excludes specific status codes.
  • -fl: Excludes certain content lengths.
  • -fwc: Excludes specific word counts.
  • -flc: Excludes certain line counts.
  • -fs: Excludes responses with a specific string.
  • -ffc: Excludes specific favicon hashes.

Example: Skip 404s and pages with “test”:

cat list.txt | httpx -fc 404 -fs "test"

Example: Filter out a specific favicon hash:

cat list.txt | httpx -favicon -ffc -215994923

Rates and Timeouts

Control scan intensity to avoid overwhelming targets:

  • -t: Sets thread count (default: 50, max: 150).
  • -rl: Limits requests per second.
  • -rlm: Limits requests per minute.
  • -timeout: Aborts after specified seconds.
  • -retries: Sets retry attempts.

Example: Gentle scan with 10 threads:

cat list.txt | httpx -sc -probe -t 10 -rl 1 -rlm 600

Tip: Adjust these settings carefully; nobody wants to accidentally DDoS a server!

Show Responses and Requests

Peek under the hood with debugging options:

  • -debug: Displays full request and response.
  • -debug-req: Shows the sent HTTP request.
  • -debug-resp: Shows the received response.
  • -stats: Provides scan progress stats.

Example: Debug a request:

echo "http://testphp.vulnweb.com" | httpx -debug

Filtering for Vulnerabilities

Though not a full vulnerability scanner, httpx can flag potential issues.

SQL Injections

Detect error-based SQLi by matching error messages:

echo "http://testphp.vulnweb.com" | httpx -path "/listproducts.php?cat=1'" -ms "Error: You have an error in your SQL syntax;"

XSS Reflections

Spot reflected XSS payloads:

echo "http://testphp.vulnweb.com" | httpx -path "/listproducts.php?cat=<script>alert(1)</script>" -ms "<script>alert(1)</script>"

Web Page Fuzzing

Fuzz for hidden resources with -path:

echo "http://testphp.vulnweb.com" | httpx -probe -sc -path "/login.php"

File Output

Save results for documentation:

  • -o: Outputs to a text file.
  • -csv: Saves as CSV.
  • -json: Saves as JSON.
  • -srd: Stores responses in a directory.

Example: Save to a text file:

cat list.txt | httpx -sc -o results.txt

Example: Save responses:

cat list.txt | httpx -sc -o results.txt -srd /root/responses

TCP/IP Customizations

Go beyond HTTP with network-level probes:

  • -pa: Probes all IPs tied to a host.
  • -p: Scans specific ports.

Example: Check common ports:

echo "http://hackerone.com" | httpx -p 22,25,80,443,3306 -probe

POST Login

Test authenticated areas with POST requests:

echo "http://testphp.vulnweb.com" | httpx -x post -path "/userinfo.php" -H "Cookie: login=test%2Ftest" -body "uname=test&pass=test" -debug-resp

This logs in and displays the profile page, which is ideal for testing post-authentication vulnerabilities.

HTTP Methods Probe

Identify allowed HTTP methods:

echo "http://testphp.vulnweb.com" | httpx -x all -probe

Routing Through Proxy

Integrate with tools like Burp Suite:

echo "http://testphp.vulnweb.com" | httpx -x all -probe -http-proxy http://127.0.0.1:8080

This routes traffic for real-time analysis.

Conclusion

httpx is a powerhouse for web reconnaissance, blending speed, versatility, and depth. From basic probes to advanced filtering, fuzzing, and authenticated testing, it equips you to tackle diverse security challenges. Whether you’re mapping subdomains, hunting vulnerabilities, or analyzing responses, httpx it delivers. Dive into the official repository for the latest features, and happy hunting in the wild world of web security!

Read more