Lots of information can be gathered from a webpage to help with OSINT investigations. Here are a couple of scripts to help !
# un-shorten.sh - This bash script allows you to expand short URLs created by URL shortening services such as bit.ly, tinyurl.com, or goo.gl, providing the original URL before clicking on the shortened link.
# webpage-parse.sh - searches the webpage for links and email addresses
# HREF-Link-Extractor - Searches the webpage and extracts all HREF links
# get-headers.sh - grabs the http headers
# get-securitytxt.sh - grabs the info in the security.txt files and displays it on the command line
# check-http-status-code.sh - Simple script that returns the description for a http status code