First of a big thanks to @achillean and his awesome website over at http://www.shodanhq.com, the amount of information that gets collected and stored is mind-blowing. I had a brief email conversation with John when I decided to write this blog and at the time there were over 70 million records stored in ShodanHQ.

So to the point of this blog post, in my current job I work a lot on e-commerce type stuff, mostly because I’m responsible for the load balancers we use (if you’ve read this blog before you might be able to guess what they are..). Part of that work means every now and again I get sent the output of our regular pen tests to answer questions or fix “holes”.

One of the most common “holes” I fix is what the external pen testers call “Environment Disclosure Information“, which in layman’s terms means you are giving out more information that you should to external people when they visit your websites.

This is an example HTTP header extract from a website, which will highlight the sort of stuff I mean:

Access-Control-Allow-Origin: *
Cache-Control: no-cache, no-store, max-age=0, no-transform, private
Content-Length: 43
Content-Type: image/gif
Date: Sun, 13 May 2012 10:35:11 GMT
Etag: “4FAF8E5F-48B6-0D239661”
Expires: Sat, 12 May 2012 10:35:11 GMT
Last-Modified: Mon, 14 May 2012 10:35:11 GMT
Pragma: no-cache
Server: Omniture DC/2.0.0
Vary: *
X-C: ms-4.4.5
p3p: policyref=”/w3c/p3p.xml”, CP=”NOI DSP COR NID PSA OUR IND COM NAV STA”
xserver: www4

Now remember I’m no security expert but to me this amount of “free” information about your web environment is both unnecessary and well to be fair a bit sloppy.

Looking at the HTTP header above an unethical type of person can determine the type of server you are running (Server: Omniture DC/2.0.0) and the version its running. Which would make it easier when looking for known vulnerabilities, and you can tell that they have at least 4 web servers (xserver: www4) providing this content (which means some sort of load balancing).

This is another HTTP header from a rather “large” software company that like Marmite you either love or hate..

Cache-Control: max-age=0
Connection: close
Content-Length: 12941
Content-Type: text/html; charset=utf-8
Date: Sun, 13 May 2012 10:40:24 GMT
Expires: Sun, 13 May 2012 10:39:24 GMT
PPServer: PPV: 30 H: BAYIDSLGN1H57 V: 0
Server: Microsoft-IIS/7.5
Set-Cookie: MSPRequ=lt=1336905624&co=1&id=64855; path=/;version=1
MSPOK=$uuid-b9356970-ea8a-491c-8c62-f367d9460ca3;
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-XSS-Protection: 0
p3p: CP=”DSP CUR OTPi IND OTRi ONL FIN”

Again you will see that the Server: HTTP header is still there, so is this really a security concern? Do pen testers just highlight it as something to put in a report??

Now onto the cool stuff (well it’s cool to me), if you have ever used ShodanHQ you will know that there is an API available, and if you pay a small amount of $$ you can get a lot of functionality. I decided to use that API and write a ruby script that would look through the 70 million records and give me the total number of results that matched some of the most popular HTTP server headers.

This is my code (I have compared the numbers against individual searches with the same server header).

#!/usr/bin/env ruby
require 'rubygems'
require 'shodan'

#Set your Shodan API Key
SHODAN_API_KEY = "enteryourapihere"

#Create the API object
api = Shodan::WebAPI.new(SHODAN_API_KEY)

#Define the array of Server headers you want to search for
array = ["Apache/2.4","Apache/2.3","Apache/2.2.21", "Apache/2.2.20", "Apache/2.2.19", "Apache/2.2.18", "Apache/2.2.17", "Apache/2.2.16", "Apache/2.2.15", "Apache/2.2.14", "Apache/2.2.13", "Apache/2.2.12", "Apache/2.2.11", "Apache/2.2.10", "Apache/2.2.9", "Apache/2.2.8", "Apache/2.2.6", "Apache/2.2.5", "Apache/2.2.4", "Apache/2.2.3", "Apache/2.2.2", "Apache/2.2.0", "Microsoft-IIS/7.5", "Microsoft-IIS/7.0", "Microsoft-IIS/6.0", "Microsoft-IIS/5.0", "Microsoft-IIS/4.0", "Microsoft-IIS/3.0", "Microsoft-IIS/2.0", "Microsoft-IIS/1.0", "nginx", "squid", "lighttpd"]
begin
#For each value in array, search through Shodan
array.each_index {|s| d = api.search("#{array[s]}")
#Print the array value and the total number of matches against the array value
puts "#{array[s]}: #{d['total']}"}
end

I know it’s nothing flash, but it works.. 🙂

Now the results (drum roll please)…Bear in mind this isn’t all the web server versions, just the ones I could think of or find without spending hours crawling through the internet.

Results:

Apache/2.4: 465
Apache/2.3: 531
Apache/2.2.21: 229250
Apache/2.2.20: 72756
Apache/2.2.19: 72666
Apache/2.2.18: 4048
Apache/2.2.17: 351696
Apache/2.2.16: 444607
Apache/2.2.15: 328945
Apache/2.2.14: 517311
Apache/2.2.13: 141590
Apache/2.2.12: 81345
Apache/2.2.11: 346329
Apache/2.2.10: 89642
Apache/2.2.9: 743891
Apache/2.2.8: 420166
Apache/2.2.6: 97186
Apache/2.2.5: 63
Apache/2.2.4: 131883
Apache/2.2.3: 2854600
Apache/2.2.2: 28955
Apache/2.2.0: 65168
Microsoft-IIS/7.5: 681421
Microsoft-IIS/7.0: 749303
Microsoft-IIS/6.0: 3932895
Microsoft-IIS/5.0: 506169
Microsoft-IIS/4.0: 14731
Microsoft-IIS/3.0: 603
Microsoft-IIS/2.0: 37
Microsoft-IIS/1.0: 31
nginx: 1299084
squid: 192084
lighttpd: 503577

Yes yes I know, surely someone can’t be using IIS/1.0 but I did triple check that result.. 🙂 To me that’s a lot of people who either don’t care about hiding this information, or like I said earlier it’s not really a big issue.

So lets take it one step further, ShodanHQ also lets you search the exploitdb using the API. Using the ruby script available from the documentation I ran it against Microsoft IIS/6.0 (the most popular IIS version from my research). Using the script I got 6 “known” exploits back (see below).

Results found: 6
3965: Microsoft IIS 6.0 (/AUX/.aspx) Remote Denial of Service Exploit
8704: Microsoft IIS 6.0 WebDAV Remote Authentication Bypass Vulnerability
8754: Microsoft IIS 6.0 WebDAV Remote Authentication Bypass Exploit (patch)
8765: Microsoft IIS 6.0 WebDAV Remote Authentication Bypass Exploit (php)
8806: Microsoft IIS 6.0 WebDAV Remote Authentication Bypass Exploit (pl)
15167: Microsoft IIS 6.0 ASP Stack Overflow (Stack Exhaustion) Denial of Service (MS10-065)

Now most of these might not be valid because of patching, but out of the 3,932,895 results there might be one or two that hasn’t been patched??

I know that realistically you will never be able to hide everything that might or might not give unethical people an advantage if you become a target, but why make it easy for them??

So is this kind of free information really an issue? If you are pen tester does this kind of information help you when running a test or is it just accepted that it’s out there and available??

Let me know what you think.