Python ipaddr performance

Last weekend while I was cleaning up my IP address summarization script (I added a setup.py, created a Cheese Shop entry and a downloadable archive) I had a look at the state of IP address manipulation in Python and found a new module called ipaddr. What sparked my interest in this module was that it had already been integrated into upcoming Python 2.7 and 3.1 as a standard library.

It seemed to contain all the functionality of IPy but with much cleaner code. It also has a function called collapse_address_list() that is similar to my summarize function but it can handle a list of non-contiguous networks and IP addresses and collapse them down.

When I wrote my summarize() function I spent quite a while optimizing the algorithm so that it can handle large address ranges in a reasonable amount of time. For example, on my development box it can summarize the two worst case ranges: IPv4 0.0.0.0 to 255.255.255.254 in 0.1s and IPv6 :: to ffff:ffff:ffff:ffff:ffff:ffff:ffff:fffe in 0.2s.

I had to see how well ipaddr performed so I wrote a little benchmarking script that summarized a /24. The results were very poor with it taking 64.27 seconds to perform 1000 runs. I then ported my summarize() code to use ipaddr and modified collapse_address_list() to use it. The result was a 50 times improvement in performance (1000 runs took 1.21 seconds).

ipaddr is developed by Google and the coders use Reitveld for reviewing patches. So I've submitted my patches there. Feel free to review and comment on the code.

Here's hoping that they are accepted and benefit everyone by making it into the next release(s) of Python!

Using Splunk's iplocation search command behind a proxy

Splunk has an iplocation search command that will add City and Country fields to your search results. It does this by looking up the IP addresses it finds using the hostip.info API. Unfortunately if your Splunk server doesn't have direct Internet access then this script will fail.

The script itself is a very simple Python script that use the module urllib.urlopen to make the API call. To get it to use your proxy server is easy.

Make a backup of the original script:

$ cd $SPLUNK_HOME/etc/searchscripts
$ cp iplocation.py iplocation.py.bak

Edit iplocation.py and add the following line below the LOCATION_URL definition:

PROXIES = {'http':'http://proxy.example.com:8080'}

Then find the line that reads:

location = urllib.urlopen( LOCATION_URL + ip )

and change it to:

location = urllib.urlopen( LOCATION_URL + ip, proxies=PROXIES )

Then perform your search and pipe it to iplocation. Make sure to limit your search as the script will do a HTTP request for every IP address it finds.

IPv4 Exhaustion Counter

I've added the IPv4 Exhaustion Counter to my side bar. It uses data provided by Geoff Huston in his IPv4 Address Report. His models currently predict that IANA will run out of addresses on 24th January 2011 with the RIR's running out by 11th December 2011.

Summarizing an IP address range

As I've mentioned previously, I've found IPy python module to be extremely useful for manipulating IP addresses. One such use is a script I've written to summarize an IP address range into the networks that make it up. The script supports both IPv4 and IPv6 addresses.

Example usage:

$ ./summarize.py 192.168.1.0 192.168.1.8
192.168.1.0/29
192.168.1.8

or alternatively as a python module:

>>> from summarize import summarize
>>> summarize('192.168.1.0', '192.168.1.254')
['192.168.1.0/25', '192.168.1.128/26', '192.168.1.192/27', '192.168.1.224/28', '192.168.1.240/29', '192.168.1.248/30', '192.168.1.252/31', '192.168.1.254']

The source for this script is available for download from the Wad of Stuff repository.

Sorting IP addresses in python

Here's a quick example of a function to sort a list of IP addresses in python using the decorate-sort-undecorate idiom (aka Schwartzian Transform) and the IPy module.

def sort_ip_list(ip_list):
    """Sort an IP address list."""
    from IPy import IP
    ipl = [(IP(ip).int(), ip) for ip in ip_list]
    ipl.sort()
    return [ip[1] for ip in ipl]

And here is an example of it in use:

>>> l = ['127.0.0.1', '192.168.0.10', '192.168.0.1', '10.1.1.1', '172.16.255.254']
>>> sort_ip_list(l)
['10.1.1.1', '127.0.0.1', '172.16.255.254', '192.168.0.1', '192.168.0.10']

Manipulating IPv4 and IPv6 addresses in python

Eighteen months ago, after many years of programming in perl for most of my systems programming needs, I decided to give python a go after much coaxing by a colleague, Alec Thomas. At the time I had started developing an IP address management system, designed for ISPs/Telcos who need to manage hundreds of address blocks and allocations to customers and internal infrastructure. I did a quick evaluation of both Ruby on Rails and Django and decided on Django for a few reasons:

• I didn't have to manage my database schema and models separately. Django allowed me to define my data models in a single place and it handled the job of creating the database tables. (This was before Rails had migrations).
  
• Django's built in administrative interface was a huge time saver and allowed me to focus on developing my application rather than designing forms.
• After programming in perl for so long the cleanliness of the python language really appealed to me. Ruby to me just seemed like OO-Perl done properly but with all the $@!#{} perlisms left in.
  

While developing the application I found an extremely useful python module called IPy (originally developed here), that handles IPv4 and IPv6 addresses and networks.

Here's a sample of how you can use it:

>>> from IPy import IP
>>> ip = IP('127.0.0.0/30')
>>> for x in ip:
...  print x
...
127.0.0.0
127.0.0.1
127.0.0.2
127.0.0.3
>>> ip2 = IP('0x7f000000/30')
>>> ip == ip2
1
>>> ip.reverseNames()
['0.0.0.127.in-addr.arpa.', '1.0.0.127.in-addr.arpa.',
'2.0.0.127.in-addr.arpa.', '3.0.0.127.in-addr.arpa.']

I'll be discussing some tools I've developed with this module at a later date.