Project Repositories Have Moved to GitHub

With Google announcing last month that they are shutting down Google Code I have moved my old and unloved code repositories to GitHub. The main code base that seems to still be in use by others is the Django Full Serializer which I may split out into its own repository one day.

Django gets support for IPv6 fields

My favorite web development framework recently committed some code to add support for IPv6 addresses in data models. The commit closes a ticket I opened 6 years ago. Congratulations to Erik Romijn for finally closing this off. Better late than never :)

The new code differs from my original code in a number of ways. First of all it has better testing and documentation. Second, it does IPv6 address validation programmatically rather than using a regular expression.

The code also borrows from ipaddr-py, which I also contribute to.

A variation on my original code is still running and has been used to manage IPv6 addresses so I can vouch for it. Most of my code doesn't actually use IPAddressFields in Django because of this even older issue which was fixed 3 years ago.

Now both of them are fixed I might look at using the new fields.

Requiring at least one inline FormSet

Last month I posted an article about my Improved Django FormWizard, well this month I've release a simple subclass of Django's BaseInlineFormSet that demonstrates how you can require a user to enter at least one entry in an inline formset.

After updating to wadofstuff.django.forms 1.1.0 you can use the RequireOneFormSet class as the formset argument to inlineformset_factory().

When the formset is validated and it does not contain one or more entries, then a ValidationError is raised which gets put into formset.non_form_errors. You will need to check this in your templates if you wish to display the error message to your users.

Inlines support for Django generic views

Django's excellent Generic Views provide developers with most of what they need to get a site up and running (if they aren't using the Admin of course). The flexibility of these views is such that for most sites you don't need much else. Extending these views is also well documented and probably covers off 95% of the situations where the plain generic views fall short.

In a recent project I found another area where the generic views fall short: inline formsets.

I have added the ability to use inline formsets in generic views. The new view functions are drop in replacements for Django's with the addition of a new inlines argument.

Installation

From Source

Download wadofstuff-django-views.

To install it, run the following command inside the unpacked source directory:

python setup.py install

From pypi

If you have the Python easy_install utility available, you can
also type the following to download and install in one step:

easy_install wadofstuff-django-views

Or if you're using pip:

pip install wadofstuff-django-views

Or if you'd prefer you can simply place the included wadofstuff
directory somewhere on your Python path, or symlink to it from
somewhere on your Python path; this is useful if you're working from a
Subversion checkout.

Note that this application requires Python 2.4 or later. You can obtain
Python from http://www.python.org/.

Usage

wadofstuff.django.views.create_object(..., inlines=None)
wadofstuff.django.views.update_object(..., inlines=None)

These functions are identical to the Django ones except for the addition of the
inlines argument. This argument consists of a list of dictionaries that will
be passed as arguments after the parent_model argument to
inlineformset_factory(parent_model, ...).

For example, arguments to a generic view might typically look like:

crud_dict = {
   'model':Author
   'inlines':[{
       'model':Book,
       'extra':2,
       'form':BookForm,
   },{
       'model':Article,
   }],
   # ... other generic view arguments
}

would translate to calls to inlineformset_factory() like:

inlineformset_factory(Author, model=Book, extra=2, form=BookForm)

and

inlineformset_factory(Author, model=Article)

The view function will create a formset for each inline model and add them to the template context. In the example above the context variables would be named book_formset and article_formset.

Update: A quick change to allow the views to be imported from wadofstuff.django.views. Bumped version to 1.0.1.

Improved Django FormWizard

A few months back I had a project that I thought needed a wizard-style interface for one of its forms. For a while now Django has included the FormWizard class in django.contrib.formtools.wizard so I decided to use that. However, I immediately hit a couple of issues with it.

FormWizard requires you to output the previous_fields context variable in each of the form's step templates. Django's FormWizard previous_fields is just a string of raw HTML. If you are using some other kind of markup to do your forms e.g. XForms, or even javascript widgets from dojo or extjs, then this is no good.

I quickly came up with a fix for this and opened #10557 which has now been slated to be included in Django 1.2.....

Wait a minute! Django 1.1 isn't even out yet!

The second issue was that FormWizard didn't handle FormSets.

Who knows when 1.2 will be out so I've bundled up my code and released it as wadofstuff.django.forms 1.0.0.

From the README:

Functions

security_hash(request, form, exclude=None, *args)

Calculates a security hash for the given Form/FormSet instance.

This creates a list of the form field names/values in a deterministic
order, pickles the result with the SECRET_KEY setting, then takes an md5
hash of that.

Allows a list of form fields to be excluded from the hash calculation. This
is useful form fields that may have their values set programmatically.

Classes

BoundFormWizard

A subclass of Django's FormWizard that adds the following functionality:

• Renders previous_fields as a list of bound form fields in the template context rather than as raw html.
• Can handle FormSets.

The usage of this class is identical to that documented here with the exception that when rendering previous_fields you should change your wizard step templates from:

{{ previous_fields|safe }}

to:

{% for f in previous_fields %}{{ f.as_hidden }}{% endfor %}

The latest stable release for the forms module can be obtained by:

• Running easy_install wadofstuff-django-forms
• Downloading the 1.0.0 release and running python setup.py install.

Note: The BoundFormWizard class can only handle regular FormSets and not ModelFormSets. I have ModelFormWizard class that I'll be adding to a future release that can handle simple Models.

Django Serializer Updates

I've had a couple of emails and forum posts where users were having difficulty with my serialization module. The problems mostly centered around installing it correctly.

I did a little work tonight to clean up the installation side of things. As a result you can now find the module in the Cheese Shop.

The latest stable release for the serialization module can also be obtained by:

• Running easy_install wadofstuff-django-serializers
• Downloading the 1.0.0 release and running python setup.py install.

The source tree has been moved around a little to accommodate these changes and can be found here.

There were no new features or bug fixes made but I have a few interesting tickets from users that I'll be reviewing over the coming weeks.

Full Django Serializers - Part II

In the first part of this article I covered the excludes and extras options to the Wad of Stuff serializer. In this article I introduce the relations option.

Relations

The Wad of Stuff serializers allow you to follow and serialize related fields of a model to any depth you wish. This is why it is considered a "full serializer" as opposed to Django's built-in serializers that only return the related fields primary key value.

When using the relations argument to the serializer you may specify either a list of fields to be serialized or a dictionary of key/value pairs. The dictionary keys are the field names of the related fields to be serialized and the values are the arguments to pass to the serializer when processing that related field.

This first example shows the simplest way of serializing a related field. The Group model has a ManyToManyField to the Permission model. In this case there is only one permission but if there were more then each would be serialized.

>>> print serializers.serialize('json', Group.objects.all(), indent=4, relations=('permissions',))
[
    {
        "pk": 2,
        "model": "auth.group",
        "fields": {
            "name": "session",
            "permissions": [
                {
                    "pk": 19,
                    "model": "auth.permission",
                    "fields": {
                        "codename": "add_session",
                        "name": "Can add session",
                        "content_type": 7
                    }
                }
            ]
        }
    }
]

The simple case may be all you need but if you want more control over exactly which fields or extras are included, excluded, and the depth of relations to follow then you need to pass a dictionary in the relations option. This dictionary is a series of nested dictionaries that are unrolled and passed as arguments when serializing each related field.

>>> print serializers.serialize('json', Group.objects.all(), indent=4, relations={'permissions':{'fields':('codename',)}})
[
    {
        "pk": 2,
        "model": "auth.group",
        "fields": {
            "name": "session",
            "permissions": [
                {
                    "pk": 19,
                    "model": "auth.permission",
                    "fields": {
                        "codename": "add_session"
                    }
                }
            ]
        }
    }
]

The relations option in this example roughly translates to a call to serialize('json', permissions_queryset, fields=('codename',)) when the permissions field is serialized.

Serializing deeper relations

The power of the relations option becomes obvious when you see it in action serializing related fields that are 2 or more levels deep. Below the content_type ForeignKey field on the Permission model is also serialized.

>>> print serializers.serialize('json', Group.objects.all(), indent=4, relations={'permissions':{'relations':('content_type',)}})
[
    {
        "pk": 2,
        "model": "auth.group",
        "fields": {
            "name": "session",
            "permissions": [
                {
                    "pk": 19,
                    "model": "auth.permission",
                    "fields": {
                        "codename": "add_session",
                        "name": "Can add session",
                        "content_type": {
                            "pk": 7,
                            "model": "contenttypes.contenttype",
                            "fields": {
                                "model": "session",
                                "name": "session",
                                "app_label": "sessions"
                            }
                        }
                    }
                }
            ]
        }
    }
]

Combining options

You may also combine the other options when serializing related fields. In the example below I am excluding the content_type.app_label field from being serialized.

>>> print serializers.serialize('json', Group.objects.all(), indent=4, relations={'permissions':{'relations':{'content_type':{'excludes':('app_label',)}}}})
[
    {
        "pk": 2,
        "model": "auth.group",
        "fields": {
            "name": "session",
            "permissions": [
                {
                    "pk": 19,
                    "model": "auth.permission",
                    "fields": {
                        "codename": "add_session",
                        "name": "Can add session",
                        "content_type": {
                            "pk": 7,
                            "model": "contenttypes.contenttype",
                            "fields": {
                                "model": "session",
                                "name": "session"
                            }
                        }
                    }
                }
            ]
        }
    }
]

That wraps up this series of articles. Head over to Wad of Stuff at Google Code to grab the source and don't hesitate to open a ticket if you find any bugs or have any enhancement requests.

Django Full Serializers - Part I

Introduction

The wadofstuff.django.serializers python module extends Django's built-in serializers, adding 3 new capabilities inspired by the Ruby on Rails JSON serializer. These parameters allow the developer more control over how their models are serialized. The additional capabilities are:

• excludes - a list of fields to be excluded from serialization. The excludes list takes precedence over the fields argument.
• extras - a list of non-model field properties or callables to be serialized.
• relations - a list or dictionary of model related fields to be followed and serialized.

De/Serialization Formats

At the moment the module only supports serializing to JSON and Python. It will also only deserialize data that is in the original Django format. i.e. it won't deserialize the results of using the excludes, extras, or relations options.

Source

The source for the serialization module can be obtained here.

Examples

Project Settings

You must add the following to your project's settings.py to be able to use the JSON serializer.

SERIALIZATION_MODULES = {
 'json': 'wadofstuff.django.serializers.json'
}

Backwards Compatibility

The Wad of Stuff serializers are 100% compatible with the Django serializers when serializing a model.

>>> from django.contrib.auth.models import Group
>>> from django.core import serializers
>>> print serializers.serialize('json', Group.objects.all(), indent=4)
[
    {
        "pk": 2,
        "model": "auth.group",
        "fields": {
            "name": "session",
            "permissions": [
                19
            ]
        }
    }
]

Excludes

>>> print serializers.serialize('json', Group.objects.all(), indent=4, excludes=('permissions',))
[
 {
     "pk": 2,
     "model": "auth.group",
     "fields": {
         "name": "session"
     }
 }
]

Extras

The extras option allows the developer to serialize properties of a model that are not fields. These properties may be almost any standard python attribute or method. The only limitation being that you may only serialize methods that do not require any arguments.

For demonstration purposes in this example I monkey patch the Group model to have a get_absolute_url() method.

>>> def get_absolute_url(self):
...     return u'/group/%s' % self.name
...
>>> Group.get_absolute_url = get_absolute_url
>>> print serializers.serialize('json', Group.objects.all(), indent=4, extras=('__unicode__','get_absolute_url'))
[
 {
     "pk": 2,
     "model": "auth.group",
     "extras": {
         "get_absolute_url": "/group/session",
         "__unicode__": "session"
     },
     "fields": {
         "name": "session",
         "permissions": [
             19
         ]
     }
 }
]

Stay tuned for the second part of this article where I demonstrate the ability to serialize related fields such as ForeignKeys and ManyToManys.

Django: DRY Custom Model Forms and Fields

With the release of Django 1.0, the ability to add a list of validators to a model field was removed. This was replaced with the clean*() methods in the new forms classes. Unfortunately when you write your own form subclasses you may end up repeating a lot of the parameters in your form field definitions that you had already declared in your model fields such as max_length, required, help_text, etc. So in the spirit of DRY I worked out a way to avoid this.

A few months back I was updating an old Django site to use new the latest 1.0 release and needed to re-implement the functionality of the following, slightly contrived, pre-1.0 model:

from django.db import models
from django.core.validators import MatchesRegularExpression, isNotOnlyDigits

class Acronym(models.Model):
 acronym = models.CharField(maxlength=3, unique=True, db_index=True,
     validator_list=[isNotOnlyDigits,
         MatchesRegularExpression(r'^[A-Z0-9]+$',
         error_message='This field may only contain uppercase letters and' \
         ' numbers.')],
     help_text='A three letter acronym.')
 definition = models.CharField(maxlength=64)

The new model looked like this after I fixed up all of the parameters:

from django.db import models

class Acronym(models.Model):
 acronym = models.CharField(max_length=3, unique=True, db_index=True,
     help_text='A three letter acronym.')
 definition = models.CharField(max_length=64)

The problem above is that the extra validation is now not being done and needs to be reimplemented either in a new form or form field class.

My first attempt probably looked like this:

import re
from django import forms
from models import Acronym

TLA_RE = re.compile(r'^[A-Z0-9]+$')

class NaiveAcronymForm(forms.ModelForm):
 acronym = forms.RegexField(TLA_RE, max_length=3, required=True,
     label='Code', help_text='A three letter acronym.')
 class Meta:
     model = Acronym

The amount of repetition is already becoming obvious and the code above only provides half the solution. We also need to implement the isNotOnlyDigits validation.

import re
from django import forms
from models import Acronym

TLA_RE = re.compile(r'^[A-Z0-9]+$')

class NaiveAcronymForm(forms.ModelForm):
 acronym = forms.RegexField(TLA_RE, max_length=3, required=True,
     label='Code', help_text='A three letter acronym.')

 def clean_acronym(self):
     if self.cleaned_data['acronym'].isdigit():
         raise forms.ValidationError(u"This value can't be comprised soley of digits."

 class Meta:
     model = Acronym

I got to this point in my own code and thought that there must be a way to leverage all the work already being done by Django to generate a form field from a model field. When I make a change to my model fields I want those changes to automatically flow on to my forms.

The solution I came up with was this:

import re
from django import forms
from models import Acronym

TLA_RE = re.compile(r'^[A-Z0-9]+$')

class AcronymField(forms.RegexField):
 """Form field for three letter acronym."""
 default_error_messages = {
     'invalid': u'This field may only contain uppercase letters and ' \
         'numbers.',
     'notonlydigits': u'''This value can't be comprised solely of digits.'''
 }

 def __init__(self, *args, **kwargs):
     """Initialize the field with the acronym regex."""
     super(AcronymField, self).__init__(TLA_RE, min_length=3, *args,
         **kwargs)

 def clean(self, value):
     """Ensure acronym matches regex and is not only digits."""
     value = super(AcronymField, self).clean(value)
     if value.isdigit():
         raise forms.ValidationError(self.error_messages['notonlydigits'])
     return value

class AcronymModelForm(forms.ModelForm):
 """Form for Acronym model."""
 def __init__(self, *args, **kwargs):
     """Programmatically declare fields."""
     super(AcronymModelForm, self).__init__(*args, **kwargs)
     field = self.Meta.model._meta.get_field('acronym')
     self.fields['acronym'] = field.formfield(form_class=AcronymField)

 class Meta:
     model = Acronym

How does this work? First of all, AcronymField is subclassed from forms.RegexField and the __init__() method overriden. It then calls the parent class's __init__() with my extra parameters, the regexp and min_length, as well as passing the original positional and keyword arguments. This ensures that any extra parameters from the model such as required, label, max_length, and help_text are preserved. I also override the clean() method and again call the superclass's clean() method before checking that the value is not only digits.

Next I subclass AcronymForm from forms.ModelForm and override the __init__() method again. Here the superclass's __init__() method is called then the acronym form field is replaced with one that uses the new AcronymField class. It is within the call to field.formfield(...) (see django.db.models.fields.Field class for the gory details) that all the values for required, label, and help_text are taken from the model field and the form field instance created.

While this results in more lines of code for a model form definition it ultimately results in less presentation and validation errors due to model fields and form fields getting out of sync. This approach could probably be generalized even further to a subclass of ModelForm but I've only had to use it for a handful of models so I leave that as an exercise for the reader.

Update: fixed formatting mistake so you can more clearly see the before and after models.

One rule to bind them - part II

In part I of this article I tried reducing the number of steps a user was required to perform to get a system installed via jumpstart and addressed a couple of the goals I mentioned in Building a manageable jumpstart infrastructure using the Solaris Security Toolkit.

My ultimate goal was to reduce steps 1 through 5 and selecting from the build menu to a single interaction such as entering the jumpstart client details into a web page or a configuration file. To eliminate the build menu I had to find a way to inform the client which build it was to install onto itself. The /etc/bootparams file doesn't allow custom parameters to be passed to the client. So after some research and experimentation I switched to DHCP with my own custom client options.

I started by working out the minimum amount of information a client required. I settled on the following DHCP option definitions in dhcp_inittab(4) format:

JSBuild         SITE,           128,    ASCII,     1,      0,      sdmi
JSRevision      SITE,           130,    ASCII,     1,      0,      sdmi
JSOSVersion     SITE,           131,    ASCII,     1,      0,      sdmi

JSBuild

A single word such as FIREWALL, DNS, WEBSERVER etc.

JSRevision

The version of the jumpstart environment. I have all of my finish scripts and configurations under source control using subversion. This option is so that I can build hosts from a specific version of my jumpstart environment.

JSOSVersion

The OS version e.g. Solaris_8, Solaris_9, Solaris_10.

Both JSRevision and JSOSVersion are used on the DHCP server side to craft the paths of a number of jumpstart specific DHCP client options.

These new DHCP options were added to the dhcp inittab in the OS install miniroot:

/srv/install/OS/Solaris_10/Solaris_10/Tools/Boot/etc/dhcp/inittab

This was necessary to allow me to run '/sbin/dhcpinfo JSBuild' or '/sbin/dhcpinfo 128' in my begin script.

I chose the ISC DHCPD for my DHCP server as it is extremely easy to configure and very flexible. The dhcpd.conf is:

#
# DHCP Configuration
#

pid-file-name "/var/run/dhcpd.pid";
lease-file-name "/srv/dhcp/dhcpd.leases";

ping-check true;
ddns-update-style none;
authoritative;

default-lease-time 86400;
max-lease-time 86400;

# MY Custom Options
option space MY;
option MY.jumpstart-build code 128 = text;
option MY.jumpstart-revision code 130 = text;
option MY.jumpstart-osversion code 131 = text;

# SUN's Jumpstart DHCP Vendor options
option space SUNW;
option SUNW.root-mount-options code 1 = text;
option SUNW.root-server-address code 2 = ip-address;
option SUNW.root-server-hostname code 3 = text;
option SUNW.root-path-name code 4 = text;
option SUNW.swap-server-address code 5 = ip-address;
option SUNW.swap-file-path code 6 = text;
option SUNW.boot-file-path code 7 = text;
option SUNW.posix-timezone-string code 8 = text;
option SUNW.boot-read-size code 9 = unsigned integer 16;
option SUNW.install-server-address code 10 = ip-address;
option SUNW.install-server-hostname code 11 = text;
option SUNW.install-path code 12 = text;
option SUNW.sysidcfg-path code 13 = text;
option SUNW.jumpstart-cfg-path code 14 = text;
option SUNW.terminal-type code 15 = text;
option SUNW.boot-uri code 16 = text;
option SUNW.http-proxy code 17 = text;

subnet 192.168.1.0 netmask 255.255.255.0 {
   authoritative;
   deny unknown-clients;
   next-server jumpserver;
   server-name "jumpserver";
   option tftp-server-name "jumpserver";
   option domain-name "example.com";
   option routers 192.168.1.254;
   option subnet-mask 255.255.255.0;
   option broadcast-address 192.168.1.255;
   option domain-name-servers 192.168.1.1, 192.168.1.2;
   option ntp-servers 192.168.1.1, 192.168.1.2;

   option SUNW.root-server-address         jumpserver;
   option SUNW.root-server-hostname        "jumpserver";
   option SUNW.posix-timezone-string       "Australia/NSW";

   # default OS is Solaris 10
   option MY.jumpstart-osversion "Solaris_10";
}
#
# Solaris
#
class "SUNW" {
   match if substring(option vendor-class-identifier, 0, 4) = "SUNW";

   #
   # Serve the correct inetboot file to sun4v hardware platforms.   
   #
   # Note: T2000 is actually SUNW.Sun-Fire-T200
   #
   if option vendor-class-identifier = "SUNW.Sun-Fire-T1000" or
       option vendor-class-identifier = "SUNW.Sun-Fire-T200" {
       filename = concat ("inetboot.SUN4V.",
           config-option MY.jumpstart-osversion, "-1");
   } else {
       filename = concat ("inetboot.SUN4U.",
           config-option MY.jumpstart-osversion, "-1");
   }

   option dhcp-parameter-request-list 1,3,6,12,15,42,43,128,129,130;

   site-option-space "MY";

   vendor-option-space SUNW;

   option SUNW.terminal-type       "vt100";
   option SUNW.root-mount-options  "rsize=32768";
   option SUNW.install-path = concat("/srv/install/OS/",
       config-option MY.jumpstart-osversion);
   option SUNW.install-server-address = config-option SUNW.root-server-address;
   option SUNW.install-server-hostname = config-option SUNW.root-server-hostname;

   # the path to the miniroot
   option SUNW.root-path-name = concat(config-option SUNW.install-path,
       "/", config-option MY.jumpstart-osversion, "/Tools/Boot");

   # the path to correct the version of the jumpstart scripts
   option SUNW.jumpstart-cfg-path = concat(config-option SUNW.root-server-hostname,
       ":/srv/jass/", config-option MY.jumpstart-revision);

   # the path to the OS specific sysidcfg file
   option SUNW.sysidcfg-path = concat(config-option SUNW.jumpstart-cfg-path,
       "/Sysidcfg/", config-option MY.jumpstart-osversion);

   # there is always a symlink in /srv/jass to the latest release.
   option MY.jumpstart-revision "latest";
}

# Solaris host declarations
include "/srv/dhcp/hosts/solaris";

A host declaration typically looks like this:

host testfw01 {
       hardware ethernet 8:0:20:ab:cd:e1;
       fixed-address 192.168.1.10;
       option MY.jumpstart-build "FIREWALL";
       option MY.jumpstart-revision "2.555";
       option MY.jumpstart-osversion "Solaris_9";
}

I include the host declarations from a separate file so that I can assign group privileges to users to edit that file only or generate it from the information the user enters into a Django web front-end to a database.

Now on the client I can drop down to the ok prompt and type:

ok boot net:dhcp -s

and when it gets to the root prompt run:

# /sbin/dhcpinfo JSBuild
FIREWALL
# /sbin/dhcpinfo JSRevision
2.555

There you have it. The client now knows which build it is meant to be. You'll notice that I haven't actually saved the user any work yet. In the next part of this series I'll demonstrate how I use this information in the begin script to select a jumpstart profile and pass this information to the JASS driver for post-installation processing.

Building a manageable jumpstart infrastructure using the Solaris Security Toolkit

Luke Kanies has been stirring the hornet's nest lately about why system administration isn't evolving despite the huge growth in the development communities around the applications we manage and use every day. The tools system administrators use get nowhere near the attention, if any, that the applications do. Nor do system administrators often publish the tools they use because they feel they are too specific to the environment they were written for, are one-off hacks, they don't have time, etc, etc. The main aim of this blog is to address my lack of sharing the sysadmin love over the years.

When I started working for my current employer a few years ago, one of my first tasks was to deploy around 20 Check Point firewalls. I had used the Solaris Security Toolkit (aka JASS) many times before as a post-installation hardening tool and set forth building a jumpstart server using JASS to automate the build of these systems. After a couple of weeks I had a build system that could have firewall up and running from bare metal in 20 minutes.

It wasn't long before a colleague had to do a similar roll out of another application so I quickly added it. He was soon to be followed by another colleague...and then another.

Cut to a year later and there were 20 different system builds being done with JASS and it was starting to become a little difficult to manage. The kinds of issues I started facing were:

1. Modularity - the JASS drivers could be modularized to a degree but it was error prone and difficult for new developers to grasp.
   
2. One rule to bind them - having to manage the rules file was a pain. A single rule like "any - Begin/soe.beg = Drivers/soe.driver" that would allow a client system to boot, automatically select the build it was meant to have, and install would be ideal.
3. Flexibility - varying disk layouts, users, third-party packages, etc on a per build basis.
4. Testing - When there were 5 individual system builds it was easy for one person to test each one and make sure they built correctly. Once it got to 20 and beyond (now closer to 40) it became very time consuming and error prone. Add to that a few developers checking in changes and you end up with bugs being introduced that could potentially impact many systems.
   
5. Builds were slow - initial installation and applying recommended patches were painfully slow.
   

To solve 1 and 3, I developed a simple configuration layer on top of JASS called slant. Slant removes the need to edit JASS drivers directly. Instead a simple configuration file is created and from it a driver file is generated from a template.

For 2, DHCP based booting rather than bootparamd was needed. A custom DHCP option is provided to the jumpstart client during installation so that it could determine which system build it was installing.

For testing, I discovered a great tool called buildbot that is normally used to automate the compile/test phase of software projects. I've developed a buildbot configuration that allows me to test all of my system builds from bare metal to login prompt and report any errors, warnings, core dumps and more that could possibly happen during an installation.

To address the slow build times I developed an automated flash archive building system that would check for new recommended patch bundles, download them, and trigger a rebuild of the flash archives with the new patches applied. New systems being built look for the presence of a flash archive and use it, otherwise they do a standard (slow!) installation. One of additional problems faced here was making a flash archive that could be built on a V210 and deployed onto a V440 with all the necessary drivers.

All of these developments have culminated in a system that is now used by dozens of people every day with nice a django web front end for provisioning that also generates our DHCP configurations and DNS zones. There is also a jumpstart "appliance" build that is put onto loan or permanent servers for other groups in the organisation to use.

I plan to discuss each of these topics in much greater detail in later articles so stick around.

Manipulating IPv4 and IPv6 addresses in python

Eighteen months ago, after many years of programming in perl for most of my systems programming needs, I decided to give python a go after much coaxing by a colleague, Alec Thomas. At the time I had started developing an IP address management system, designed for ISPs/Telcos who need to manage hundreds of address blocks and allocations to customers and internal infrastructure. I did a quick evaluation of both Ruby on Rails and Django and decided on Django for a few reasons:

• I didn't have to manage my database schema and models separately. Django allowed me to define my data models in a single place and it handled the job of creating the database tables. (This was before Rails had migrations).
  
• Django's built in administrative interface was a huge time saver and allowed me to focus on developing my application rather than designing forms.
• After programming in perl for so long the cleanliness of the python language really appealed to me. Ruby to me just seemed like OO-Perl done properly but with all the $@!#{} perlisms left in.
  

While developing the application I found an extremely useful python module called IPy (originally developed here), that handles IPv4 and IPv6 addresses and networks.

Here's a sample of how you can use it:

>>> from IPy import IP
>>> ip = IP('127.0.0.0/30')
>>> for x in ip:
...  print x
...
127.0.0.0
127.0.0.1
127.0.0.2
127.0.0.3
>>> ip2 = IP('0x7f000000/30')
>>> ip == ip2
1
>>> ip.reverseNames()
['0.0.0.127.in-addr.arpa.', '1.0.0.127.in-addr.arpa.',
'2.0.0.127.in-addr.arpa.', '3.0.0.127.in-addr.arpa.']

I'll be discussing some tools I've developed with this module at a later date.