So a while back I blogged about the future of sniffMyPackets and how I was looking at building components for it that would make better use of existing systems you may be using (not over tooling things). Gobbler is the first step of that journey and is now available for your amusement..
Gobbler can be found HERE:
Essentially you feed Gobbler a pcap file and tell it how you want the data outputted. The current supported outputs are:
- Splunk (via TCP)
- Splunk (via UDP)
- JSON (just prints to screen at the moment).
The tool is based on Scapy (as always) and once the initial file read is done it’s actually quite quick (well I think it is).
Moving forward I am going to add dumping into databases (mongodb to start with) and Elastic search. On top of this I’m going to start writing protocol parsers for Scapy that aren’t “available”. I’ve found an HTTP one written by @steevebarbeau that is included in Gobbler so you can now get better Scapy/HTTP dumps.
To use with Splunk is easy, first in your Splunk web console create a new TCP or UDP listener (see screenshots below).
NOTE: UDP Listeners are better for any pcap with over 10,000 packets. I’ve had issues with TCP listeners past that point.
Once your Splunk Listener is ready, just “tweak” the gobbler.conf file to meet your needs:
[splunk]
server = ‘localhost’
port = 10000
protocol = ‘udp’
If you created a TCP listener, change the protocol to ‘tcp’ and gobbler will work out the rest for you.
To run gobbler is easy (as it should be), from the gobbler directory run this command:
./gobbler.py -p [pcapfile] -u [upload type]
I’ve included a 1 packet pcap in the repo so you can straight away test using:
./gobbler.py -p pcaps/test.pcap -u splunk
or if you want to see the results straight away try the json option:
./gobbler.py - pcaps/test.pcap -u json
If you load the packets into Splunk, then they will look something like this:
This means if you were to search Splunk for any packet from a Source IP (of your choice) you can just use (test_index is where I am storing them):
index="test_index" "ip_src=12.129.199.110"
This means you can then use the built-in GeoIP lookup to find out the City etc using this Splunk Search:
index="test_index" | iplocation ip_src
Which would give you back the additional fields so you can do stuff like this:
The benefit of using Gobbler is that I’ve done all the hard work on the formatting, I was going to write a Splunk app but the people at Splunk have been ignoring me so…
I’ve successfully managed to import about 40,000 packets via a UDP listening in just a few minutes, it’s not lightning quick but it’s a start.
Enjoy..
PS. A “rebranding” of my core projects will be underway soon so stay tuned for more updates.. 🙂
The Cyber Nomad 
Not sure who at Splunk would be ignoring you about an app, I think any method for packet consumption, is one we should share with other folks. I like your approach, reach out, i would like to help get this on apps.splunk.com.
Hi Fred,
I requested a Dev license so I don’t have to worry about the daily 500MB limit (which I managed to hit today indexing just 2 pcap files) and had no response, so I emailed them about it and never got a reply to that either. There are a few open source alternatives to Splunk out there so I will probably focus on integrating with them.
Adam, Having met Fred IRL before and knowing his role I hope he can sort this “issue” out and keep encouraging you to develop for the Splunk platform
Adam,
I get the following error using the pcap from SEC503 day6 challenge
Loading pcap file: ../Training/GCIA/SEC503/Exercises/Day6/challenge.pcap
Number of packets: 36286
Uploading to Splunk via UDP
Traceback (most recent call last):
File “./gobbler.py”, line 67, in
main()
File “./gobbler.py”, line 56, in main
for s in x:
File “/home//Downloads/gobbler-master/parsers/packetParser.py”, line 47, in parsePacket
“dns_tc”: p[UDP].tc, “dns_rd”: p[UDP].rd, “dns_ra”: p[UDP].ra, “dns_z”: p[UDP].z, “dns_ad”: p[UDP].ad,
File “/usr/lib/python2.7/dist-packages/scapy/packet.py”, line 176, in __getattr__
fld,v = self.getfield_and_val(attr)
File “/usr/lib/python2.7/dist-packages/scapy/packet.py”, line 172, in getfield_and_val
return self.payload.getfield_and_val(attr)
File “/usr/lib/python2.7/dist-packages/scapy/packet.py”, line 172, in getfield_and_val
return self.payload.getfield_and_val(attr)
File “/usr/lib/python2.7/dist-packages/scapy/packet.py”, line 1057, in getfield_and_val
raise AttributeError(attr)
AttributeError: ad
It seems to have inserted 29 lines but it has also tried to put them in one event as below
timestamp=2003-09-04 19:54:27.680255, pcapfile=../Training/GCIA/SEC503/Exercises/Day6/challenge.pcap, packetnumber=1,ip_dst=255.255.255.255, ip_flags=0L, ip_version=4L, ip_ihl=5L, ip_id=28248, ip_frag=0L, ip_chksum=2646, ip_len=576, ip_src=0.0.0.0, ip_options=[], ip_ttl=64, ip_proto=17, ip_tos=0,udp_dport=67, udp_sport=68, udp_len=556, udp_chksum=64983,ether_src=00:50:56:6b:00:76, ether_dst=ff:ff:ff:ff:ff:ff, ether_type=2048 timestamp=2003-09-04 19:54:31.870594, pcapfile=../Training/GCIA/SEC503/Exercises/Day6/challenge.pcap, packetnumber=2,ip_dst=255.255.255.255, ip_flags=0L, ip_version=4L, ip_ihl=5L, ip_id=28249, ip_frag=0L, ip_chksum=2645, ip_len=576, ip_src=0.0.0.0, ip_options=[], ip_ttl=64, ip_proto=17, ip_tos=0,udp_dport=67, udp_sport=68, udp_len=556, udp_chksum=64983,ether_src=00:50:56:6b:00:76, ether_dst=ff:ff:ff:ff:ff:ff, ether_type=2048 timestamp=2003-09-04 19:54:40.190604, pcapfile=../Training/GCIA/SEC503/Exercises/Day6/challenge.pcap, packetnumber=3,ip_dst=255.255.255.255, ip_flags=0L, ip_version=4L, ip_ihl=5L, ip_id=28250, ip_frag=0L, ip_chksum=2644, ip_len=576, ip_src=0.0.0.0, ip_options=[], ip_ttl=64, ip_proto=17, ip_tos=0,udp_dport=67, udp_sport=68, udp_len=556, udp_chksum=64983,ether_src=00:50:56:6b:00:76, ether_dst=ff:ff:ff:ff:ff:ff, ether_type=2048 timestamp=2003-09-04 19:54:56.362741, pcapfile=../Training/GCIA/SEC503/Exercises/Day6/challenge.pcap, packetnumber=4,ip_dst=255.255.255.255, ip_flags=0L, ip_version=4L, ip_ihl=5L, ip_id=28251, ip_frag=0L, ip_chksum=2643, ip_len=576, ip_src=0.0.0.0, ip_options=[], ip_ttl=64, ip_proto=17, ip_tos=0,udp_dport=67, udp_sport=68, udp_len=556, udp_chksum=64983,ether_src=00:50:56:6b:00:76, ether_dst=ff:ff:ff:ff:ff:ff, ether_type=2048 timestamp=2003-09-04 19:55:28.320661, pcapfile=../Training/GCIA/SEC503/Exercises/Day6/challenge.pcap, packetnumber=5,ip_dst=255.255.255.255, ip_flags=0L, ip_version=4L, ip_ihl=5L, ip_id=28252, ip_frag=0L, ip_chksum=2642, ip_len=576, ip_src=0.0.0.0, ip_options=[], ip_ttl=64, ip_proto=17, ip_tos=0,udp_dport=67, udp_sport=68, udp_len=556, udp_chksum=50545,ether_src=00:50:56:6b:00:76, ether_dst=ff:ff:ff:ff:ff:ff, ether_type=2048
Show all 29 lines
Hi Jon, can you just do a git pull and try a tcp listener. I made some changes yesterday afternoon to fix a couple of issues including the clumping results together. Thanks.
I did svn co which I think should have pulled down the latest version (I’m not sure as anything with git didn’t seem to work) With a TCP listener the results aren’t clumped but the SEC503 day 6 challenge still fails after packet 29 and the Day 5 phishing fails after 299 of 858 packets. Both the tests in the PCAPs sub directory from github work fine.
Ah ok I will dig those pcaps out and give it a go myself.
Adam, Doing some digging and in all cases it seems to choke on DNS packets since pulling the latest version i just get
Loading pcap file: test.pcap
Number of packets: 1000
Uploading to Splunk via TCP – Full Details
ad
Upload Complete
But upload seems to stop at 1st DNS request as per the SEC503 files
Hi Jon,
Yeah I found the same problem, I think it’s because they are slightly “dodgy” packets. I’ve got the pcap at home and I’ve got a couple of tweaks to make to the code. Will do that tonight and push it up to the repo.
Thanks for your patience and help.
Adam
Strange thing is the one i tried today wasn’t a dodgy pcap straight capture from my own personal machine with the same issue. Will try the code again later or in the morning and let you know.
Hi Jon I think I’ve fixed the issue and made it quicker. Do a git pull and try out the Splunk tcp upload again. Wasn’t an issue with dodgy packets, was the way that Scapy handles DNS packets. Tested with the 503 phishing pcap file and it doesn’t error anymore.. Let me know how u get on.
That seems a lot better but it still chokes on the day6 challenge file
Loading pcap file: ../../Training/GCIA/SEC503/Exercises/Day6/challenge.pcap
Number of packets: 36286
Uploading to Splunk via TCP – Full Details
invalid syntax (, line 1)
Upload Complete
Hi Jon,
I’ve fixed the issues (sort of), I’ve removed some of the DNS and LLNMR layers out of the pcap due to the way that Scapy stacks them. I’ve pushed it up to the repo and tested on the challenge.pcap and the phishing-attack.pcap and both complete with no errors. I will have a look at sorting the “dodgy” packets out over the weekend.
The issue with the UDP upload is because of the way that Splunk handles the UDP traffic, there is a fix for that which will get rolled out when I write the Splunk App (just reading the docs at the moment).
The mongodb upload function is nearly complete as well, there are some issues with it not liking the Raw layer when it contains Hex in it, so I either have to encode them as BSON binary or strip them out.
Other than a few small formatting issues I think it’s a much better process now (and hopefully quicker) and I will work through the bugs I know about. Give me a shout if you find anymore.
Thanks again for being the unofficial BETA tester.. 🙂
Looks good. Would it be possible to keep track of the streams like wireshark so that you can pull out all the bits of a conversation.
Also would it be possible to have someway to set the initial timestamp and do the rest based of this as i’m tthinking of a use case wherre the capture maybe got later and wanting to effectively overlay it on other data.
Don’t mind being a beta tester as I can see real potential for this.