grep

Сбор ip адресов:

egrep -o '([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}' all.txt

Сбор ссылок

wget -q http://example.com -O - | \
    tr "\t\r\n'" '   "' | \
    grep -i -o ']\+href[ ]*=[ \t]*"\(ht\|f\)tps\?:[^"]\+"' | \
    sed -e 's/^.*"\([^"]\+\)".*$/\1/g'

Второй вариант:

wget --user-agent="Opera/9.60 (J2ME/MIDP; Opera Mini/4.2.14912/812; U; ru) Presto/2.4.15" -T 10 --spider -H --force-html -r -l5 http://start_url.ru 2>&1 | grep '^--' | awk '{ print $3 }' | grep -v '\.\(css\|js\|png\|gif\|jpg\)$' | egrep "^http://[a-zA-Z0-9\._\-]+/$" > urls.txt &

Сбор доменов

wget http://www.who.is/whois_index/index.php  -o /dev/null -O - | grep -oh "index\.[0-9a-zA-Z\-]*\.php" | sed "s/^/http:\/\/www.who.is\/whois_index\//p" | wget -i - -o /dev/null  --user-agent="Opera/9.60 (J2ME/MIDP; Opera Mini/4.2.14912/812; U; ru) Presto/2.4.15" -O -  | grep -oh "[a-z0-9\-]*\/domain_list\.[a-z0-9\-]*\.[0-9]*\.php" | sed "s/^/http:\/\/www.who.is\/whois_index\//p"  |wget -i - -o /dev/null  --user-agent="Opera/9.60 (J2ME/MIDP; Opera Mini/4.2.14912/812; U; ru) Presto/2.4.15" -O -  | grep -oh "\/whois\-com\/ip\-address\/[a-z0-z\-]*\.[a-z]*" | grep -oh "[a-z0-z\-]*\.[a-z]*" | sort -u