Maltego: Creating entities..

So far in this blog series we have talked about how you can create new Maltego transforms and machines easily. Today we are going to cover creating new entities for you to use and/or abuse.

Creating new entities is (like creating transforms) quite easy and painless, there are a couple of things you need to keep in mind but we will cover those as we go.

To create a new entity within Maltego click:

Manage – Manage Entities

Screen Shot 2014-09-29 at 12.55.55

You will now get the Entity Manager window.

Screen Shot 2014-09-29 at 12.56.36

Once again (like most things in Maltego) you get a ‘New Entity Wizard‘. Lets have a look at the options you have to supply.

  • Display name – This is the name of your entity as it will appear in the ‘Palette‘ within Maltego, try to keep it short and sweet
  • Short description – A quick one liner of what the entity is for
  • Unique type name – Again like your transform make it unique and relevant to the entity type
  • Inheritance
    • Base Entity – Inheritance allows you to create a new entity but ‘inherit‘ the properties of an existing entity. This means if you want to create a new entity for ‘IPv4 Address‘ which has additional fields for your needs you can ‘inherit‘ the base properties and the transforms that can be run against the base entity into your new entity (it’s actually a really cool feature).
  • Icons – For each new entity you create you have to set an icon. You can either use a built-in one or use a new one. For our purpose we are going to use an existing one.

Click on Browse

Screen Shot 2014-09-29 at 12.56.49

Make sure you have selected the ‘Default’ tab at the top, you should now see a whole stack of icons you can pick.

Screen Shot 2014-09-29 at 13.06.17

Pick an icon, any icon and then click on ‘OK’. You should hop back to the Entity Wizard and your Icon image should now be populated with your choice.

Screen Shot 2014-09-29 at 13.12.17

Still with me?? Cool right-click ‘Next’ and we progress onto the next page. This is where we have to define the properties for the entity. If you use Inheritance this is typically grayed out as it will want to use the existing properties from the entity you are inheriting from (but you can add additional ones if you like). Here we need to define the main property for the entity, this is the one that we will set by default each time we return this entity type.

Screen Shot 2014-09-29 at 13.14.37

A lot of the fields are set by default but we can go through them just so you know what they are for.

  • Property display name – This is the name of the property when you open the entity. I tend to keep it the same as the Display name as I set as this is the main property and it just makes sense to me to have them the same.
  • Short description – This is a short description of the entity property (rather than the description of the entity)
  • Unique property name – I have a habit of setting this to the same as the entity unique type name (when using Canari) but this is what you will need to call in your transform code when setting the value.
  • Data type – By default this is a ‘string’ value, however you can set the following types:
    • string
    • int
    • date
    • double
  • Sample value – If you drag your entity into a graph, this is the value that will get set and display for this property.

We will just tweak a couple of values before moving on.

Screen Shot 2014-09-29 at 13.24.01

Now we are done, click on ‘Next’ and lets carry on. We can now add our new entity into an existing or new category.

Screen Shot 2014-09-29 at 13.24.27

You can add your new entity into one of the existing categories which are listed below (and are the ones listed in your palette within Maltego).

  • Devices
  • Infrastructure
  • Locations
  • Penetration Testing
  • Personal
  • Social Network

For our purpose we are going to create a new category called ‘MyFirstTransform‘. If you want to create a new category just type it into the box.

Screen Shot 2014-09-29 at 13.26.50

Click Finish.

All being well in your ‘Palette‘ pane on the right you should see the new category called ‘MyFirstTransform‘ and in the entity list you should see the ‘Robot Entry‘ entity.

Screen Shot 2014-09-29 at 13.27.41

Before we move on I just want to cover creating additional properties for our new entity. This is handy if you want to use more than one property in executing a transform or storing in your entity. From within the Entity Manager click on the 3 little blue dots on the right of the new entity.

Screen Shot 2014-09-29 at 13.30.57

Click on the ‘Additional Properties‘ tab at the top, you will see the original property that we created so we are going to add another (just for giggles). Click on ‘Add Property‘ in the top right.

Screen Shot 2014-09-29 at 14.20.09

We are going to add a new property for the dynamic property we called in the origin GetRobots-mk2 transform. Type the following into the available fields.

Name – properties.orgurl
Display Name – Original URL
Type – String

Screen Shot 2014-09-29 at 14.21.59

The field name is used for the unique name, and display name is the “friendly description”.

You may notice that the list of available types is a lot longer then when we created the entity. I’m not sure why but it is, ‘string’ for example has to options:

  • string
  • string[]

The ‘string’ type is for a single string value, while the ‘string[]’ allows you to provide a range of string values (like the ‘ports’ property in the Website entity).

Click OK to add the new property and then click OK again to return to the Entity Manager.

Screen Shot 2014-09-29 at 14.26.11

Awesome, isn’t this fun. Now we need to tweak our existing transform to make use of the new entity. Lets have a look at the code again to refresh our memory.

#!/usr/bin/env python

# Maltego transform for getting the robots.txt file from websites

from MaltegoTransform import *
import requests

m = MaltegoTransform()
m.parseArguments(sys.argv)

website = m.getVar('fqdn')
port = m.getVar('ports')
port = port.split(',')
ssl = m.getVar('website.ssl-enabled')
robots = []

try:
  for c in port:
    if ssl == 'true':
      url = 'https://' + website + ':' + str(c) + '/robots.txt'
      r = requests.get(url)
      if r.status_code == 200:
        robots = str(r.text).split('\n')
        for i in robots:
          ent = m.addEntity('maltego.Phrase', i)
          ent.addAdditionalFields("url","Original URL",True,url)
      else:
        m.addUIMessage("No Robots.txt found..")
    else:
      url = 'http://' + website + ':' + str(c) + '/robots.txt'
      r = requests.get(url)
      if r.status_code == 200:
        robots = str(r.text).split('\n')
        for i in robots:
          ent = m.addEntity('maltego.Phrase', i)
          ent.addAdditionalFields("url","Original URL",True,url)
      else:
        m.addUIMessage("No Robots.txt found..")
except Exception as e:
  m.addUIMessage(str(e))

m.returnOutput()

To make use of the new entity we just need to change two lines of code (there are two instances of this though). The original lines code of code that define your entity returned is below:

ent = m.addEntity('maltego.Phrase', i)
ent.addAdditionalFields("url","Original URL",True,url)

We simply need to change this to our new entity type and property values.

ent = m.addEntity('adammax.robotentry', i)
ent.addAdditionalFields("property.orgurl","Original URL",True,url)

This is what the code should look like now.

#!/usr/bin/env python

# Maltego transform for getting the robots.txt file from websites

from MaltegoTransform import *
import requests

m = MaltegoTransform()
m.parseArguments(sys.argv)

website = m.getVar('fqdn')
port = m.getVar('ports')
port = port.split(',')
ssl = m.getVar('website.ssl-enabled')
robots = []

try:
  for c in port:
    if ssl == 'true':
      url = 'https://' + website + ':' + str(c) + '/robots.txt'
      r = requests.get(url)
      if r.status_code == 200:
        robots = str(r.text).split('\n')
        for i in robots:
          ent = m.addEntity('adammax.robotentry', i)
          ent.addAdditionalFields("property.orgurl","Original URL",True,url)
      else:
        m.addUIMessage("No Robots.txt found..")
    else:
      url = 'http://' + website + ':' + str(c) + '/robots.txt'
      r = requests.get(url)
      if r.status_code == 200:
        robots = str(r.text).split('\n')
        for i in robots:
          ent = m.addEntity('adammax.robotentry', i)
          ent.addAdditionalFields("property.orgurl","Original URL",True,url)
      else:
        m.addUIMessage("No Robots.txt found..")
except Exception as e:
  m.addUIMessage(str(e))

m.returnOutput()

Save the changes and then within Maltego run the GetRobots transform again, and you should see your new entity spring into life (fingers crossed).

Screen Shot 2014-09-29 at 14.35.50

So there you go, the basics behind creating your own entities. Next time we are going to talk about exporting all the Maltego goodness we have created.

Maltego: Rise of the Machines…

In this episode of Maltego loving (or learning if you prefer that) we are going to talk about Maltego Machines. Maltego machines in the simplest form are a collection of transforms that execute in order, each one feeding off the entities generated from the last transform.

Maltego Machines to start with may seem complicated especially when you are trying to get them to run in parallel, but they are incredibly useful and powerful so it seemed worthy to give them their own blog post.

The official documentation for Maltego Machines can be found HERE.

To start with we are going to create a simple Maltego Machine to run the ‘GetRobots-mk2’ transform we created in the earlier blog posts. In reality a Machine to run one transform is a bit of a waste but it’s a good place to start. To find the Machines within Maltego click.

Machines – New Machine

This will launch a wizard that we can use to create our first Machine.

Screen Shot 2014-09-25 at 07.18.06

Lets fill in the details;

Display Name – The display name for our Machine.
Unique ID – This is the same principle as with Transforms, make it unique and relevant.
Author – That will be you.
Description – Write something in that will make sense if anyone else uses your Machine.

Screen Shot 2014-09-25 at 07.20.39

Click Next….

Now we get a choice of the type of Machine we want to create. There are two main types (excluding blank, which we will come back to).

Macro – These Machines will execute once (and only once) when you start them.
Timer – These are my favourite Machines, once started they execute in a timed loop (defined by you) and will keep running until you stop them.

Screen Shot 2014-09-25 at 07.21.01

Select Macro and click Next…

Screen Shot 2014-09-25 at 07.24.38

Now we get to the good part, the next window is the main Machine ‘editor’, this is where you can add your transforms and make sure the Machine compiles properly. You will also notice that they have pre-loaded the Machine with some example transforms so you have an idea of what it should look like.

There are three options along the top which we will talk about first.

Format – This essentially formats your code so it looks pretty.
Toggle Comment – Will either comment a line or un-comment it (saves you typing 2 //’s).
Compile – Checks your code and makes sure that it is all happy.

Screen Shot 2014-09-25 at 07.26.13

On the right hand side of the window is the list of transforms available for your Machine, you can add local transforms or remote transforms (TDS), but bear in mind if you want to share your Machines you need to make sure that everyone has the transforms you used available (more on exporting stuff from Maltego in the next post).

Screen Shot 2014-09-25 at 07.48.14

Right let’s get on with the actual coding of the Machine shall we, first off we are going to remove all the stuff that Maltego puts in there by default (that’s not actually needed).

//Welcome to Maltego Machines!

//Each machine starts with a statement like this
machine("adammax.macgetrobots",
displayName:"Get Robots.txt",
author:"@catalyst256",
description: "A simple 1 transform Maltego machine to uses \
the GetRobots-mk2 transform.") {

//A macro machine has a start function like this
start {

}
}
//Of course there is much more you can do with machines… Have fun!

Right so we now have a basic (yet empty) Machine so lets add in our transform. In the space between ‘start {‘ and the following ‘}’ expand ‘Local’ on the right hand side and then double-click on GetRobots-mk2. Your code should now look like this.

//Welcome to Maltego Machines!

//Each machine starts with a statement like this
machine("adammax.macgetrobots",
displayName:"Get Robots.txt",
author:"@catalyst256",
description: "A simple 1 transform Maltego machine to uses \
the GetRobots transform.") {

//A macro machine has a start function like this
start {
run(“adammax.GetRobotsMk2”)

}
}
//Of course there is much more you can do with machines… Have fun!

Notice the lack of formatting?? Click on the Format button at the top of the window and magically your code will now look like this.

//Welcome to Maltego Machines!

//Each machine starts with a statement like this
machine("adammax.macgetrobots",
displayName:"Get Robots.txt",
author:"@catalyst256",
description: "A simple 1 transform Maltego machine to uses \
the GetRobots transform.") {

//A macro machine has a start function like this
start {
run(“adammax.GetRobotsMk2”)

}
}
//Of course there is much more you can do with machines… Have fun!

Ok so we’ve added the transform, we sorted the formatting so now we just need to check to make sure it will compile. Click the green tick at the top of the window and the compile debug screen will appear at the bottom of the window. If we haven’t made any horrible mistakes (just copy and paste the code if you have a problem), you should get this wonderful message.

Compiling machine...
Success

OK so now we have the Machine written (yes thats it), click on Save….

We can now run the Machine, create a new graph and add a Website onto it, right-click and under Machines you should see our newly created Machine. But wait what black magic is this? How does the Machine know it can run on that entity type?? So I have no fancy answer for this, other than it takes the first transform in the Machine and based on what entity it is designed to run on will then appear when you right-click on that entity type.

Screen Shot 2014-09-25 at 08.15.08

When a Machine is running you get a window in the top right of your Maltego screen that will show the status of the Machine and once it has completed will update and tell you how many entities returned.

Screen Shot 2014-09-25 at 08.15.30

Lets now have a look at some of the more “advanced” Machines that are available. We will look at some of the built-in Machines and work through how they work. Click on

Machines – Manage Machines

Screen Shot 2014-09-25 at 08.23.46

As you can see there are already some pretty cool machines available by default. Lets open one up and see what makes it tick.

Double click on the ‘Footprint L1’ Machine, and the Machine editor will open up.

Screen Shot 2014-09-25 at 08.25.20

Now these Machines are read-only so we can’t make changes to them, we can however (if so inclined) copy and paste them into a new Machine.

Lets work through the code and see what this Machine does.

machine("paterva.footprint.level1",
displayName:"Footprint L1",
author:"Roelof Temmingh",
description: "This performs a level 1 (fast, basic) footprint of a domain.") {

start {
//do all the DNS enumeration
log("Performing DNS enumeration",showEntities:false)
status("Phase 1 - DNS enumeration")
paths {
run("paterva.v2.DomainToWebsite_DNS")
run("paterva.v2.DomainToDNSName_DNSBrute",slider:500)
run("paterva.v2.DomainToDNSName_ZT",slider:10000)
run("paterva.v2.DomainToSOAInformation")
run("paterva.v2.DomainToWebsiteDNS_SE",slider:255)
}

//here we end up with DNS names (and MX,NS,websites)
//take it to IP address and Netblock
log(“Resolving to IP”,showEntities:false)
status(“Phase 2 – Resolve DNS names”)
run(“paterva.v2.DNSNameToIPAddress_DNS”)

//we now have IP adddresses
status(“Phase 3 – Netblocks and AS”)
log(“Computing netblocks”,showEntities:false)
run(“paterva.v2.IPAddressToNetblock_Cuts”)
log(“Looking up AS”,showEntities:false)
run(“paterva.v2.NetblockToAS_SS”)

}
}

Right you ready for this..

So the first few lines of this Machine detail the information about the Machine, which is the same as we have in ours.

machine("paterva.footprint.level1",
displayName:"Footprint L1",
author:"Roelof Temmingh",
description: "This performs a level 1 (fast, basic) footprint of a domain.")

We then move into the core of the Machine, the first function is a ‘log’. Log is used send a message to the GUI within the Machine window.

log("Performing DNS enumeration",showEntities:false)

With this particular ‘log’ function we post the message “Performing DNS enumeration” to the GUI, the ‘showEntities:false’ means that once run it won’t log the list of entities to the GUI. If you set this to ‘showEntities:true’ then it will list the entities that the transform will run against.

Next we can see a ‘status’ function.

status("Phase 1 - DNS enumeration")

The ‘status’ function within a Maltego Machine sets the contents of the status function (“Phase 1 – DNS enumeration”) as the label on the Machine window. This is useful if you are running lots of different stages of transforms and want to be able to track the current running set of transforms.

We can now move onto the main body of this Machine.

paths {
run("paterva.v2.DomainToWebsite_DNS")
run("paterva.v2.DomainToDNSName_DNSBrute",slider:500)
run("paterva.v2.DomainToDNSName_ZT",slider:10000)
run("paterva.v2.DomainToSOAInformation")
run("paterva.v2.DomainToWebsiteDNS_SE",slider:255)
}

The are two ways you can run transforms in a Machine, you can either run in parallel or in series. The use of ‘paths’ means that these transforms will run in parallel. Basically this means that each of the 5 will run at once against the base entity (so that’s 5 transforms run from within 1 Machine). and each transform will return their own entity type into the graph.

The ‘slider’ function within the ‘run’ sets the maximum number of entities that can be returned by the transform. If you are using the community edition of Maltego like me (still looking for donations of a full license) that is 12. The transform below when run will only return 500 entities.

run("paterva.v2.DomainToDNSName_DNSBrute",slider:500)

We can now move onto the second block of code.

//here we end up with DNS names (and MX,NS,websites)
//take it to IP address and Netblock
log("Resolving to IP",showEntities:false)
status("Phase 2 - Resolve DNS names")
run("paterva.v2.DNSNameToIPAddress_DNS")

Again we use ‘log’ and ‘status’ to provide output to the Machine window, we then run another transform which will take the DNS entity returned from the first block of code and run this transform against them.

run("paterva.v2.DNSNameToIPAddress_DNS")

The last block of code is below.

//we now have IP adddresses
status("Phase 3 - Netblocks and AS")
log("Computing netblocks",showEntities:false)
run("paterva.v2.IPAddressToNetblock_Cuts")
log("Looking up AS",showEntities:false)
run("paterva.v2.NetblockToAS_SS")

This takes the IP addresses from the last transform and then runs two more transforms (with ‘log’ and ‘status’ messages) in sequence.

So now we know some more functions of Machines, lets update ours to do a little bit more. We are going to add some ‘log’ and ‘status’ messages. Have a look at the code below (which you can copy and paste over the original Machine we did).

//Welcome to Maltego Machines!

//Each machine starts with a statement like this
machine("adammax.macgetrobots",
displayName:"Get Robots.txt",
author:"@catalyst256",
description: "A simple 1 transform Maltego machine to uses \
the GetRobots transform.") {

//A macro machine has a start function like this
start {
log(“Hunting for Robots.txt”,showEntities:true)
status(“Connecting to Website”)
run(“adammax.GetRobotsMk2”)

}
}
//Of course there is much more you can do with machines… Have fun!

Below is a screenshot of what the Machine window now shows.

Screen Shot 2014-09-25 at 12.06.30

There is a lot more to Maltego Machines that I haven’t covered but hopefully you will have a good idea of what they can do and how to go about coding them. Have a look at the other built-in Machines and the documentation I’ve linked at the top of the post is very good.

Enjoy

Maltego: Enhance your….entity

So over the weekend I started off my series on how you can create your own Maltego transforms, quickly and without much effort. Today I wanted to expand on that and look at how we can expand on the original ‘GetRobots’ transform to make use of some of the additional properties within an entity.

If you remember our GetRobots transform made use of two builtin Maltego entities. The starting entity (the one you run the transform on) was the Website entity. The Website entity has two additional properties that can be set.

These are:

  • Ports
  • SSL Enabled

Screen Shot 2014-09-21 at 09.04.00

By default these are set to Port 80 and SSL isn’t enabled. The ‘Ports’ property allows you to enter an array of different ports that your target website might be running on, the ‘SSL Enabled’  is essentially a True or False option. In order to make use of these values (or allow for them to be changed) we need to change our original transform slightly.

NOTE: The python code is this transform might not be pretty but its more to show you how to call the additional properties rather than impress you with my Python skills.

OK, so in the GitHub repo (found HERE), you will see I’ve created a new file called GetRobots-mk2.py this is the same base code as the original transform but with some tweaks. So lets go through them;

from MaltegoTransform import *
import requests

m = MaltegoTransform()
m.parseArguments(sys.argv)

So first off we removed the 'website = sys.argv[1]' line and replaced it with 'm.parseArguments(sys.argv)'. This is essentially telling the transform to make use of all the entity properties (regardless of how they are set). We now need to call these properties and store them in variables so we can use them.

website = m.getVar('fqdn')
port = m.getVar('ports')
port = port.split(',')
ssl = m.getVar('website.ssl-enabled')
robots = []

Now you may be wondering where I got the value names from, there are two places you can look. The first is this PDF from Paterva which explains the core entities and the values available which you can find HERE.

The second is from within Maltego (this is the way I tend to do it), click on Manage – Manage Entities scroll down to the Website entity, click on the three little blue dots on the right, then click on the ‘Additional Properties’ tab on the top.

On the left you will see a list of properties, if you select one the top right window will change to show you the details for that property. The ‘Name’  value is the important value and when you are pulling out these properties in your transforms it’s the one that will case you headaches (well it does me anyway).

Screen Shot 2014-09-22 at 20.59.29

So back to the code. We use the ‘getVar’  and the associated entity name (for that property) to store the values in a variable. The ‘ports’  properties is stored as a list so we can iterate over it later, the ‘ssl-enabled’  is a boolean so will return either True or False.

The rest of the code is tweaked slightly to make use of these new variables, which you can see below.

try:
for c in port:
if ssl == 'true':
url = 'https://' + website + ':' + str(c) + '/robots.txt'
r = requests.get(url)
if r.status_code == 200:
robots = str(r.text).split('\n')
for i in robots:
ent = m.addEntity('maltego.Phrase', i)
ent.addAdditionalFields("url","Original URL",True,url)
else:
m.addUIMessage("No Robots.txt found..")
else:
url = 'http://' + website + ':' + str(c) + '/robots.txt'
r = requests.get(url)
if r.status_code == 200:
robots = str(r.text).split('\n')
for i in robots:
ent = m.addEntity('maltego.Phrase', i)
ent.addAdditionalFields("url","Original URL",True,url)
else:
m.addUIMessage("No Robots.txt found..")
except Exception as e:
m.addUIMessage(str(e))

We’ve added two extra components, first we create a ‘for’ loop and iterate through each value stored in ‘ports’. This allows our transform to cater for multiple (or single) ports. We then throw in some ‘if’  logic to see if the ‘ssl-enabled’  property is set. If it is we change the url to be ‘https’ and chuck in the port number for good measure. Like I said, it’s not pretty but it works.. 🙂

The final addition piece to the puzzle is we have declared some additional properties in the ‘Phrase’  which if you check yourself will see don’t exist. The ‘Phrase’  entity has only one value, but Maltego allows you to create dynamic values which you can then use later on in other transforms. To do this, we changed the way we create the entity, in order to use some of the more ‘advanced’ features of ‘addEntity’ we created a variable called ‘ent’  which then allows use to add additional properties among other things.

ent = m.addEntity('maltego.Phrase', i)
ent.addAdditionalFields("url","Original URL",True,url)

The ‘addAdditionalFields’ function has these options (taken from Paterva’s documentation).

addAdditionalFields(fieldName=None,displayName=None,matchingRule=False,value=None)
Set additional fields for the entity
fieldName: Name used on the code side, eg displayName may be “Age of Person”, but the app and your transform will see it as the fieldName variable
displayName: display name of the field shown within the entity properties
matchingRule: either “strict” for strict matching on this specific field or false
value: The additional fields value

If you install the new transform following the previous posts instructions and run it you will see that the new additional property is created as a ‘Dynamic Property’.

Screen Shot 2014-09-22 at 21.26.12

Screen Shot 2014-09-22 at 21.33.21

If you were to then write a new transform for our new ‘Phrase’ entity you would call the dynamic property using.

ent.addAdditionalFields("url","Original URL",True,url)
url = m.getVar('url')

Hope that all makes sense?? Next time we are going to look at Maltego Machines

Maltego: My First Transform

On Friday I posted a challenge on twitter called “Transform Friday”, you suggest a Maltego transform and I would have a go at writing it. There is no real reason behind the challenge other than I like writing Maltego transform and its a nice way to write something different to the normal packet related ones I do. My plan was to write them without using Canari (native Maltego transforms) and make them all available as TDS transforms (no installation required).

Screen Shot 2014-09-21 at 08.41.28

The first challenger soon steeped up and much to my surprise it was Paterva (yes thats right the people that make Maltego).

Screen Shot 2014-09-21 at 08.41.44

However it was then suggested to me that it would be quite cool if I wrote up how you (yes you) could create your own Maltego transforms. Now this is where I need to apologise, I’ve spent the last two years telling you all about the awesome (well I think they are awesome) Maltego transforms I’ve created but I’ve never told you how. Part of the reason behind this blog was to share information and try in some way to save you the pain I’ve suffered, that’s why I wrote the Scapy guide after all.

So this will be the first in a series of blog posts about creating Maltego transforms. Today we are going to cover some of the basics and write a nice and simple transform (that’s useful at the same time), and then over the coming weeks we will cover more advanced topics, like setting additional properties, changing link colour and finally creating TDS transforms.

There is a GitHub repo available with the code snippets in so you can follow along or just download the completed ones (link available towards the bottom).

The transforms will be written in Python (as that’s the only one I can do) but Paterva do cater for a range of languages to be used as well as having some documentation available. Links for all the resources will be available at the bottom of this post and in the GitHub repo. I should at this point say the documentation for developing with Maltego is actually pretty good and this is not supposed to be a replacement, but more a compliment to what the guys at Paterva already provide.

For the purpose of these blog posts, I’m going to assume that you know what Maltego is and how to use it (even the basics is fine). So before we start writing the transform we are going to talk a little about the two key Maltego things you need for a transform.

Entities & Transforms

Entities

These the building blocks of a Maltego graph, they are both the start and the finish of a transform. The type of entity (e.g. Website) defines what transforms can be run against, once a transform has run then you typically get a different entity type as a result (and the cycle repeats).

For your first transform we are going to use two entities.

  • Website
  • Phrase

The website entity by defaults contains ‘www.paterva.com’ as the default value. You can change this to the website of your choice and you have the option of marking whether the website is “SSL Enabled” or not and the ports it’s listening on.

Screen Shot 2014-09-21 at 09.04.00

The Phrase entity is really simple, it literally a piece of text. It has no additional properties available.

Screen Shot 2014-09-21 at 09.13.28

Transforms

Transforms are where the magic on Maltego occurs, they take your starting entity (in our case website) run some magic (well code) and return the results as another entity.

There are two types of transforms,

  • Local
  • Remote

Local transforms are ones that you have to install locally on your machine. All the components required to run that transform have to be installed on your machine (Python libraries for example). One of the benefits of using local transforms is that it widens your scope for what you can do (I’ll explain that later), but it does limit the portability of those transforms (to a degree) as each person would need to install them (and the dependencies) before they can use them. There are a couple of ways to ease that pain but it’s still a manual process.

Remote transforms run on TDS servers, the code for TDS transforms are executed on the TDS server so you remove yourself from having to worry about dependencies and libraries. The main benefit of TDS transforms is that they are truely portable, however they are limited to what you can do with them (well to my knowledge). For example if you have a custom transform that wants to read a file on your machine (or a local server) there isn’t an easy way to execute that transform as TDS transform because the remote TDS server won’t have access to that file. You would have to somehow get that file to a location that the TDS server could access it.

Your First Transform

So enough about the boring stuff lets get started. Your first transform is nice and simple we are going to use a Website entity and then write a transform to check for the robots.txt file, open it and then return each entry as a Phrase entity.

So the first thing you need is the Maltego Python library which can be found HERE:

Extract the MaltegoTransform.py to your working directory (where you are going to write the transform), for example mine is:

/Users/Adam/Coding/Maltego/MyFirstTransform/

Now create a new file in that directory called GetRobots.py

pwd
/Users/Adam/Coding/Maltego/MyFirstTransform
touch GetRobots.py

Using your editor of choice lets open the file and start coding (I use Sublime Text 3).

First off lets add the Python sheebang and a comment about what this code is for.

#!/usr/bin/env python

# Maltego transform for getting the robots.txt file from websites

Now we need to import the Maltego transform library and the other Python libraries we need.

from MaltegoTransform import *
import sys
import os
import requests

We will be using the ‘requests’  library to make the call to the website to retrieve the robots.txt file. We need the ‘sys’  library as the sys.argv command is used to pass the value of the starting Maltego entity to the transform. So in our case the value of website (e.g. http://www.paterva.com) will be sent to our transform as sys.argv[1].

Our next job is to create a variable to store that entity value in and something to store the output in.

website = sys.argv[1]
robots = []

Now we are going to initialise the Maltego library so we can use it later on.

m = MaltegoTransform()

Still with me??

Ok lets write the code for using ‘requests’ to go and retrieve the robots.txt file (if it exists).

try:
r = requests.get('http://' + website + '/robots.txt')
if r.status_code == 200:
robots = str(r.text).split('\n')
for i in robots:
m.addEntity('maltego.Phrase', i)
else:
m.addUIMessage("No Robots.txt found..")
except Exception as e:
m.addUIMessage(str(e))

m.returnOutput()

Now this isn’t as bad as it looks (promise). Firstly we put everything in a ‘try/except’  bubble so that we have some error handling.

Next we make the initial GET request.

r = requests.get('http://' + website + '/robots.txt')

This adds the value we set on the Website entity and passes into the ‘request.get’  call with the ‘robots.txt’ added on the end.

We then add some logic in place to see if the robots.txt exists or not.

if r.status_code == 200:
robots = str(r.text).split('\n')
for i in robots:
m.addEntity('maltego.Phrase', i)
else:
m.addUIMessage("No Robots.txt found..")

If the status code is 200 (that’s a HTTP OK response), we store the output in the list variable ‘robots’ as text (instead of unicode) and then split into separate list entries based on each new line.

We then iterate through the list and for each line we create a new Maltego entity.

m.addEntity('maltego.Phrase', i)

This is where the magic happens. We initialise the Maltego Transform library earlier as ‘m’, we then use the function ‘addEntity’  to create an entity and set is as ‘maltego.Phrase’. We can then pass ‘i’ (which is from our for loop) as the value to write to that entity.

There are two other statements that use part of the Maltego Transform library in our code.

else:
m.addUIMessage("No Robots.txt found..")
except Exception as e:
m.addUIMessage(str(e))

The ‘addUIMessage’  function allows you to pass text back to the Maltego GUI when the transform is run (or running), it’s ideal for error handling messages which is how we used it here. The ‘else’  statement is if the status.code isn’t 200 (which is usually a bad thing) and would return the message to the Maltego GUI and the ‘except’  statement will return any other error message that occurs (such as the website not being available etc.).

We finish the whole thing off my then calling the Maltego Transform library and telling it to return the output from our transform back into the GUI.

m.returnOutput()

Before we install this transform into Maltego we can actually check to make sure it works as expected. From your working directory simply run:

python GetRobots.py www[dot]paterva[dot]com

If it’s all worked you should see something like this.

Screen Shot 2014-09-21 at 11.14.50

What you might not have realised is that Maltego uses XML a lot of it’s transforms and entities, the Maltego Transform library will automatically wrap your output up in the correct format so that displays properly within Maltego (after all who likes messing with XML).

Now that we have working code, we can add it into Maltego as our own transform. This is acutally really easy. Within the Maltego GUI, click

Manage – Local Transforms

You should get a popup window like the one below.

Screen Shot 2014-09-21 at 10.33.54

On this first screen you need to enter some details about your transform.

Display name – This is how your transform appears in Maltego
Description – A brief description about what your transform does
Transform ID – A unique reference that Maltego uses (you can’t change it once you’ve created it)
Author – That’s you..
Input entity type – We need to choose ‘Website’, this means that our entity will only run on Website entities (which is good for us)
Transform set – Transform sets are a collection of transforms that relate to an entity type. I’ve chosen ‘none’ just for this example. You can create your own transform sets which we will cover in later posts

Screen Shot 2014-09-21 at 10.34.53

Click Next….

Screen Shot 2014-09-21 at 10.38.51

This is where we set the meat of the transform, as in actually point it to your code.

Command – This is the path to your interpretor so in my case ‘/usr/bin/python’ (I’m using a Mac though)
Parameters – This is where you point it to your actual GetRobots.py file
Working Directory – Set this to your working directory.

Screen Shot 2014-09-21 at 10.40.48

Click Finish….

So you’ve now added your transform into Maltego.. shall we run it??

Drag a website entity onto an empty graph in Maltego. Leave the default http://www.paterva.com address in there for now. Right click on the entity, then select

Run Transform – Other Transforms – GetRobots

Screen Shot 2014-09-21 at 10.45.44

All being well you should get a number of Phrase entities returned containing the same output as when you run your script manually??? (fingers crossed)

Screen Shot 2014-09-21 at 10.48.00

Add some other websites and see what you get back. If like me you are only using the Community Edition of Maltego then don’t be worried if you only get 12 entities returned.

Congratulations you’ve just created your first Maltego Transform.. it wasn’t that hard now was it, so what you waiting for?? 🙂

In the next post in this series we will explore some more of the features available when returning entites and look at how to use additional properties on an entity when running a transform and how to return additional properties once a transform is complete.

Link to GitHub repo:

https://github.com/catalyst256/MyFirstTransform

Links to Maltego Developer documentation:

http://www.paterva.com/web6/documentation/developer.php – Main Developer Site

http://www.paterva.com/web6/documentation/developer-local.php? – Local Transforms
http://www.paterva.com/web6/documentation/localTransforms-SpecIII.pdf – Local Transform Specifications
http://www.paterva.com/web6/documentation/MaltegoTransform-Python.zip – Python Library

sniffmypackets v2 – Sneak Peak

So this week I started the long awaited (well on my part) rewrite of sniffmypackets. My initial release was more a voyage of discovery rather than a well thought out application but it did teach me a lot about how to write transforms for Maltego (using Canari Framework) so I felt it was best to start from scratch and just reuse chunks of code where it made sense.

I’ve also decided to add more functionality that will make using SmP (sniffmypackets) more than just a Maltego based exercise. There are 3 key features I will be adding.

  1. Database Support
  2. Web Interface
  3. Netflow support

The decision to use a database (MongoDB at the moment) was to allow for the database to exist after Maltego is closed. Every pcap will have a unique Session ID which means that not only can you load a pcap into SmP you can (on a different machine and without the original pcap) then rebuild the contents of the pcap based on the Session ID.

The web interface will mean that other people (who may or may not have Maltego) can then view key information about a pcap or you can just use it as another source of reference.

Both the database and web interface will be able to run outside of the machine running Maltego (so you can centralise it) and I might even include Vagrant machines if you don’t have the capacity to do that.

It does mean that SmP will need a MongoDB backend to run but trust me it will be worth it.

After a conversation I had a few weeks ago with some very clever people I decided to add Netflow capabilities into SmP to give you even more bang for your buck (so to speak).

Below are some screenshots of what it looks like at the minute.

SmP - Overview

SmP - Session Import

SmP-DNS-Screenshot

HoneyMalt – Maltego for Honeypots

So in my normal fashion, the other week I came up for another Maltego/Canari project while still not having completed most of the other projects I’ve started. That being said I like to keep things interesting so today I give you HoneyMalt.

The love child of the Canari Framework, Maltego and Kippo (SSH Honeypot), HoneyMalt allows you to pull data out of your Kippo honeypot (as long as you are using MySQL) and display it in a graph. A bit like this one below.

HoneyMaltScreenshot

Currently HoneyMalt allows you to display the following information:

“Evil” IP address
Geo IP lookup for country code
Session ID
Username/Password combinations
Input (the stuff they type in)
File download info

I’ve also managed to get the hang of Maltego Machines (finally) and all of these transforms run from one machine (who knew that path & paths would give me so much grief). The sneaky use of a Maltego machines and a database means, that if like me you don’t have a full version of Maltego (donations accepted) you can still get the all the entities returned instead of hitting the 15 entity limit on a transform run, it just takes a few iterations of the machine running.

There is a quick Youtube video of what that looks like below.

I’m also in the process of adding some search functions so you can find specific sessions based on IP address, or keyword or url. After that I will be moving onto more honeypots (Dionaea next) to add into HoneyMalt with the sole intent of trying to breaking your machine while Maltego tries to graph it all (only kidding).

With HoneyMalt I’ve also resisted the urge to create loads of new entities so where possible I’ve used the Maltego builtin ones so that you can make use of the existing transform sets which is good for you and means I have less to code (I’m not lazy honest..). Likewise I’ve also resisted the urge to use lots of Python modules so you only need 3 (currently) and 1 of those is Canari.

The GitHub repo has the install instructions and if you get any problems raise an Issue Ticket on the repo. Repo is HERE

Have fun!!!

Scapy: Heartbleed

So I might be a bit late to the game but I post this code on Twitter a while back but then forgot to blog about so here you go…

I’ve written a little snippet of Python code that uses Scapy to search through a pcap file looking for Heartbleed requests and responses. Due to Scapy not having a layer for TLS connections this have been done by slicing and dicing the RAW layer and pulling out the information we need. A lot of the time that I’m writing Scapy code to analyse pcap files I use Wireshark to look at the packets and match that up in Scapy.

This is the output from Wireshark;

Screen Shot 2014-07-25 at 07.24.03

And this is what Scapy sees;

Raw load='\x18\x03\x02\x00\x03\x01@\x00'

Currently the script just looks for traffic on port 443, but I will tweak it over the next week or two to handle any port.

I’ve removed the port restrictions so it should be protocol agnostic.. (hopefully)..

You can find the code HERE:

To run it, just type

./pcap-heartbleed.py [pcapfile]

So for example:

./pcap-heartbleed.py heartbleed.pcap

If it finds anything it thinks is a Heartbleed packet you get an output similar to the one below (I’ve changed the IP addresses):

Heartbleed Request: src: 1.1.1.1 dst: 2.2.2.2
Heartbleed Response: src: 2.2.2.2 dst: 1.1.1.1
Heartbleed Request: src: 1.1.1.1 dst: 2.2.2.2
Heartbleed Response: src: 2.2.2.2 dst: 1.1.1.1
Heartbleed Request: src: 1.1.1.1 dst: 2.2.2.2
Heartbleed Response: src: 2.2.2.2 dst: 1.1.1.1

Enjoy!!

Packet Addict: IPv4 Packets

So I’ve decided to finally getting around to revising (and then taking) my SANS 503 exam (or GCIA). It’s been a while since I’ve spent anytime looking at packets up close and personal so rather than me suffering alone I thought I would blog as I go (plus it’s a good way to remember stuff).

This is the first in a new series called “Packet Addict” (that’s me by the way).

Today we are going to have a look at IPv4 packets, what makes up a IP packet and how you can read these things in their “naked” version (or Hex if you want to be precise). I’m going to be honest you can find all of this on Google, so go ahead or just carry on reading.

DISCLAIMER: If I get something wrong, please tell me as I’m revising for an exam and you wouldn’t want to feel guilty if I fail because you didn’t tell me something was wrong.. now would YOU.

So lets just cover some key terms before we start.

Bit – Smallest unit of a packet, has the value of 0 or 1 (so either on or off).
Nibble – Made up of 4 bits and has is one hexadecimal value.
Byte – This is 8 bits, or 2 nibbles and is 2 hexadecimal values.

So to put some context (we all love context), a TCP flag (covered later but used here for reference) is made up of 1 bit which means it’s either on or off (set or unset).

An example of a nibble (4 bits) would be the IP version number (currently you either get 4 or 6 but you know).

A byte is something like the IP Time to Live (amount of hops before the packet dies a slow and painful death in the darkness that is the internet), working on the basis that it’s 2 hexadecimal values then the maximum it can be (or should be) is 255 (0xFF).

To convert hex to decimal in python you can use this snippet of code:

>>> print int("5", 16)

Hexadecimal is a base 16 numbering system, so the python code takes your value (5 in this case) and prints the int value based on it being base 16.

All make sense?? Awesome lets carry on.

So an IPv4 packet is a minimum of 20 bytes with a maximum of 60 bytes, most IPv4 packets are 20 bytes and I’ve not come across one that is larger than that but I’ve led a settled life.

So in order to work out from hex (that’s hexadecimal in big word speak) what a IPv4 header is really saying (without using Wireshark) you need one of these things below (not mine, found it on the internet)..

IPv4Header

This is a graphical representation of an IPv4 header and it’s associated bits (and bytes), so now with the magic of a calculator and Google for when I get stuck, lets make some packet magic.

OK so here is the hex dump of a IPv4 header (taken from a random pcap file).

4500 003c 2d70 4000 4006 d7f6 0a14 9096 0a14 9097

So lets work through the first few bits together. Every IPv4 packet starts (typically) with 4500 which makes it a bit easier to spot when looking at lots of raw packets. So remembering that each hex character is a nibble (2 bits) and using the IPv4 Header image from above we can begin to break out the individual parts of the packet.

Version -> 4 bits (1 hex) = 4
IHL -> 4 bits (1 hex) = 5

Now this is the bit that confuses me, how does a 5 in hex equal 20 bytes?? (the standard length of an IPv4 header)… Google to the rescue..

Brain Meltdown:
The IHL (Internet Header Length) field is 4 bits long and specifies the header length in 32-bit word (4 bytes).

So there are two ways to do this, my way and the right way but mine is quicker. 4 bits for IHL, multiple that by your IHL value (5) and you get 20 bytes.

The correct way is to do it like this:

5 * 32 / 8 = 20 (bytes)

5 is the IHL value, multiple by 32 (32 bit word), divide by 8 (number of bits in a byte).

So confused yet?? I know I am (trying writing a blog about this stuff), but lets continue. Next we will break down each chunk of the packet and then we can convert it back to decimal (i.e. readable format) later. Using the table above and the raw packet I will map each value into the diagram below with the hex values.

decodeipv4

So I’ve tried to match the colouring coding across the original IPv4 Header picture a bit further up. Lets work through each value, convert it to decimal and then we should be able to see it make a bit more sense.

IP Version = 4 – Makes sense it’s an IPv4 packet after all
IHL = 5 – So we know that we need to multiple this by 4 to get the number of bytes = 20 bytes
Type of Service = 00 – So this is a result of zero, not uncommon to see in IP headers
Total Length = 003c – If we convert this to decimal we get 60, which is the size of the packet in bytes
IP Identification Number = 2d70 – This converts to 11632, this value is usually used for identifing the group of fragments of a single IP datagram
Flags = 4 – These flags are used for determining if a packet is fragmented or not and each of the 3 flags is a bit value (either 1 or a 0). The flags are:
bit 0: Reserved, must be zero
bit 1: Don’t Fragment (DF)
bit 2: More Fragments (MF)
So if we look at this from a binary point of view (as we are dealing with bits), we would get 0 1 0 0 which is hexadecimal for 4 (Don’t Fragment). If it was a fragmented packet it would be binary of 0 0 1 0 which is hexadecimal of 2.
Fragment Offset = 000 – A value of zero means it’s either the first fragment, but in this case because the Don’t Fragment bit isn’t set then there is no need for an offset.
Time to Live = 40 – This converts back to 64 decimal which is a “standard” TTL for a packet
Protocol = 06 – This converts to 6 decimal which is the protocol number for a TCP packet (the next layer)
Header Checksum = d7f6 – These are used for error checking, this value converts to 55286
Source IP = 0a149096 – So we know that the maximum value for an IPv4 Address is 255, so each octet of the IP address will be 2 hex values seperated by a “.”, so
if we break each down in 2 hex value pairs we get 0a 14 90 96, which converts to 10.20.144.150.
Destination IP = 0a149097 – Same principal as before, we turn this value into 0a 14 90 97 which in turns converts to 10.20.144.151

So there you go, a fully decoded IPv4 packet.. trust me it was as painful to write as it was for you to read.. 🙂

More information about the protocol numbers can be found here:

http://en.wikipedia.org/wiki/List_of_IP_protocol_numbers

and if you want to see more details about the IPv4 structure you can read this:

http://en.wikipedia.org/wiki/IPv4

Gobbler: Eating its way through your pcap files…

So a while back I blogged about the future of sniffMyPackets and how I was looking at building components for it that would make better use of existing systems you may be using (not over tooling things). Gobbler is the first step of that journey and is now available for your amusement..

Gobbler can be found HERE:

Essentially you feed Gobbler a pcap file and tell it how you want the data outputted. The current supported outputs are:

  1. Splunk (via TCP)
  2. Splunk (via UDP)
  3. JSON (just prints to screen at the moment).

The tool is based on Scapy (as always) and once the initial file read is done it’s actually quite quick (well I think it is).

Moving forward I am going to add dumping into databases (mongodb to start with) and Elastic search. On top of this I’m going to start writing protocol parsers for Scapy that aren’t “available”. I’ve found an HTTP one written by @steevebarbeau that is included in Gobbler so you can now get better Scapy/HTTP dumps.

To use with Splunk is easy, first in your Splunk web console create a new TCP or UDP listener (see screenshots below).

SplunkDataInputs

SplunkUDPListener

NOTE: UDP Listeners are better for any pcap with over 10,000 packets. I’ve had issues with TCP listeners past that point.

Once your Splunk Listener is ready, just “tweak” the gobbler.conf file to meet your needs:

[splunk]
server = ‘localhost’
port = 10000
protocol = ‘udp’

If you created a TCP listener, change the protocol to ‘tcp’ and gobbler will work out the rest for you.

To run gobbler is easy (as it should be), from the gobbler directory run this command:

./gobbler.py -p [pcapfile] -u [upload type]

I’ve included a 1 packet pcap in the repo so you can straight away test using:

./gobbler.py -p pcaps/test.pcap -u splunk

or if you want to see the results straight away try the json option:

./gobbler.py - pcaps/test.pcap -u json

If you load the packets into Splunk, then they will look something like this:

SplunkGobblerPacket

This means if you were to search Splunk for any packet from a Source IP (of your choice) you can just use (test_index is where I am storing them):

index="test_index" "ip_src=12.129.199.110"

This means you can then use the built-in GeoIP lookup to find out the City etc using this Splunk Search:

index="test_index" | iplocation ip_src

Which would give you back the additional fields so you can do stuff like this:

SplunkIPLocate

The benefit of using Gobbler is that I’ve done all the hard work on the formatting, I was going to write a Splunk app but the people at Splunk have been ignoring me so…

I’ve successfully managed to import about 40,000 packets via a UDP listening in just a few minutes, it’s not lightning quick but it’s a start.

Enjoy..

PS. A “rebranding” of my core projects will be underway soon so stay tuned for more updates.. 🙂

sniffmypackets – The Future..

So about a year ago I started work on “sniffMyPackets”, the Maltego transform set (using Canari Framework) for analysing pcap files. I started it for 3 reasons;

  1. I’m obsessed with pcap files (I admit it, I’m an addict)
  2. I wanted to start writing Python code
  3. It sounded like fun

At the end of last year I hit a wall, 68 transforms in with what I think is a pretty cool “flow” to it and I ran out of ideas, direction, steam, whatever you want to call it. So I haven’t really done much with it since.

Then over the last few months, with the Scapy workshop at BSides and a mental kick up the ass I decided that I should give it some love (virtual of cause). The next question was “OK so how do I make it better??”, I had an idea, I thought it was awesome that was going to be the next big release for SmP (sniffMyPackets), but as the weeks have gone on I’m starting to wonder if I”m looking at this wrong. What benefit would this give the people that use it??

Let me try and explain… (bear with me)

People write tools to help other people, to enable them to do the same task over and over again with little effort and to ensure the results are predictable (within a small margin of error). So I had written a tool (SmP), within another tool (Maltego), using another tool (Canari Framework) to enable me to do that. To be honest when you write it down like that it seems a little crazy. In order to use my tool you need another 2 in place already!!

One of the things that bugs me at work, is when people say “Sorry can’t do that, I need this tool and it costs $$$$”, honestly the amount of times I want to kick them in the nuts and say “Open your eyes, we have loads of tools already, just make the most of them”. The reality is that in IT (especially Operations), we like off the shelf tools, because we’re not developers, we don’t code, we just use the same tools as everyone else does.

So why do I tell you this?? Well so my “awesome” idea for SmP was to start integrating pcaps into a database, with a web front end and then rewrite the Maltego transforms to suck out data from the database. I honestly thought that would be cool. Then while researching I discovered there are lots of “tools” out there already. Things like Splunk, LogStash, ElasticSearch, MongoDB (for databases), so why I am retooling something just for the sake of it (other than learning how to do it)??

So from now on I’m going to write tools that let you integrate into existing toolsets, I want you to use my code to get more out of what you already have rather than forcing you to get another toolset to manage and support. You want pcap files in Splunk??? Right I’m going to write you a App for that, you want to use ElasticSearch?? Awesome I’ll do that as well. You want pcap files in json? Easy..

The existing Maltego set for SmP will stay (as I have at least 1 user) but I will rewrite them to plug into existing toolset and suck the same data out, that way you don’t need more tools, you can just get more of the ones you’ve already committed to.

Stay tuned…

PS sorry for any spelling/grammar issues, I’m not wearing my glasses..