Full Django Serializers - Part II

In the first part of this article I covered the excludes and extras options to the Wad of Stuff serializer. In this article I introduce the relations option.

Relations

The Wad of Stuff serializers allow you to follow and serialize related fields of a model to any depth you wish. This is why it is considered a "full serializer" as opposed to Django's built-in serializers that only return the related fields primary key value.

When using the relations argument to the serializer you may specify either a list of fields to be serialized or a dictionary of key/value pairs. The dictionary keys are the field names of the related fields to be serialized and the values are the arguments to pass to the serializer when processing that related field.

This first example shows the simplest way of serializing a related field. The Group model has a ManyToManyField to the Permission model. In this case there is only one permission but if there were more then each would be serialized.

>>> print serializers.serialize('json', Group.objects.all(), indent=4, relations=('permissions',))
[
    {
        "pk": 2,
        "model": "auth.group",
        "fields": {
            "name": "session",
            "permissions": [
                {
                    "pk": 19,
                    "model": "auth.permission",
                    "fields": {
                        "codename": "add_session",
                        "name": "Can add session",
                        "content_type": 7
                    }
                }
            ]
        }
    }
]

The simple case may be all you need but if you want more control over exactly which fields or extras are included, excluded, and the depth of relations to follow then you need to pass a dictionary in the relations option. This dictionary is a series of nested dictionaries that are unrolled and passed as arguments when serializing each related field.

>>> print serializers.serialize('json', Group.objects.all(), indent=4, relations={'permissions':{'fields':('codename',)}})
[
    {
        "pk": 2,
        "model": "auth.group",
        "fields": {
            "name": "session",
            "permissions": [
                {
                    "pk": 19,
                    "model": "auth.permission",
                    "fields": {
                        "codename": "add_session"
                    }
                }
            ]
        }
    }
]

The relations option in this example roughly translates to a call to serialize('json', permissions_queryset, fields=('codename',)) when the permissions field is serialized.

Serializing deeper relations

The power of the relations option becomes obvious when you see it in action serializing related fields that are 2 or more levels deep. Below the content_type ForeignKey field on the Permission model is also serialized.

>>> print serializers.serialize('json', Group.objects.all(), indent=4, relations={'permissions':{'relations':('content_type',)}})
[
    {
        "pk": 2,
        "model": "auth.group",
        "fields": {
            "name": "session",
            "permissions": [
                {
                    "pk": 19,
                    "model": "auth.permission",
                    "fields": {
                        "codename": "add_session",
                        "name": "Can add session",
                        "content_type": {
                            "pk": 7,
                            "model": "contenttypes.contenttype",
                            "fields": {
                                "model": "session",
                                "name": "session",
                                "app_label": "sessions"
                            }
                        }
                    }
                }
            ]
        }
    }
]

Combining options

You may also combine the other options when serializing related fields. In the example below I am excluding the content_type.app_label field from being serialized.

>>> print serializers.serialize('json', Group.objects.all(), indent=4, relations={'permissions':{'relations':{'content_type':{'excludes':('app_label',)}}}})
[
    {
        "pk": 2,
        "model": "auth.group",
        "fields": {
            "name": "session",
            "permissions": [
                {
                    "pk": 19,
                    "model": "auth.permission",
                    "fields": {
                        "codename": "add_session",
                        "name": "Can add session",
                        "content_type": {
                            "pk": 7,
                            "model": "contenttypes.contenttype",
                            "fields": {
                                "model": "session",
                                "name": "session"
                            }
                        }
                    }
                }
            ]
        }
    }
]

That wraps up this series of articles. Head over to Wad of Stuff at Google Code to grab the source and don't hesitate to open a ticket if you find any bugs or have any enhancement requests.

Feed URL change

Quite a while back the feed URL for this blog changed back to the default Blogger one. I've been using FeedBurner from day one but Blogger initially didn't have the ability to redirect to a different URL for your feed.

I've noticed that few subscribers are still directly using the old FeedBurner URL. When you find the time please update your reader to use the canonical URL:

http://wadofstuff.blogspot.com/feeds/posts/default

I don't expect the old one to go away any time soon but better to be safe than sorry.

Django Full Serializers - Part I

Introduction

The wadofstuff.django.serializers python module extends Django's built-in serializers, adding 3 new capabilities inspired by the Ruby on Rails JSON serializer. These parameters allow the developer more control over how their models are serialized. The additional capabilities are:

• excludes - a list of fields to be excluded from serialization. The excludes list takes precedence over the fields argument.
• extras - a list of non-model field properties or callables to be serialized.
• relations - a list or dictionary of model related fields to be followed and serialized.

De/Serialization Formats

At the moment the module only supports serializing to JSON and Python. It will also only deserialize data that is in the original Django format. i.e. it won't deserialize the results of using the excludes, extras, or relations options.

Source

The source for the serialization module can be obtained here.

Examples

Project Settings

You must add the following to your project's settings.py to be able to use the JSON serializer.

SERIALIZATION_MODULES = {
 'json': 'wadofstuff.django.serializers.json'
}

Backwards Compatibility

The Wad of Stuff serializers are 100% compatible with the Django serializers when serializing a model.

>>> from django.contrib.auth.models import Group
>>> from django.core import serializers
>>> print serializers.serialize('json', Group.objects.all(), indent=4)
[
    {
        "pk": 2,
        "model": "auth.group",
        "fields": {
            "name": "session",
            "permissions": [
                19
            ]
        }
    }
]

Excludes

>>> print serializers.serialize('json', Group.objects.all(), indent=4, excludes=('permissions',))
[
 {
     "pk": 2,
     "model": "auth.group",
     "fields": {
         "name": "session"
     }
 }
]

Extras

The extras option allows the developer to serialize properties of a model that are not fields. These properties may be almost any standard python attribute or method. The only limitation being that you may only serialize methods that do not require any arguments.

For demonstration purposes in this example I monkey patch the Group model to have a get_absolute_url() method.

>>> def get_absolute_url(self):
...     return u'/group/%s' % self.name
...
>>> Group.get_absolute_url = get_absolute_url
>>> print serializers.serialize('json', Group.objects.all(), indent=4, extras=('__unicode__','get_absolute_url'))
[
 {
     "pk": 2,
     "model": "auth.group",
     "extras": {
         "get_absolute_url": "/group/session",
         "__unicode__": "session"
     },
     "fields": {
         "name": "session",
         "permissions": [
             19
         ]
     }
 }
]

Stay tuned for the second part of this article where I demonstrate the ability to serialize related fields such as ForeignKeys and ManyToManys.

Django: DRY Custom Model Forms and Fields

With the release of Django 1.0, the ability to add a list of validators to a model field was removed. This was replaced with the clean*() methods in the new forms classes. Unfortunately when you write your own form subclasses you may end up repeating a lot of the parameters in your form field definitions that you had already declared in your model fields such as max_length, required, help_text, etc. So in the spirit of DRY I worked out a way to avoid this.

A few months back I was updating an old Django site to use new the latest 1.0 release and needed to re-implement the functionality of the following, slightly contrived, pre-1.0 model:

from django.db import models
from django.core.validators import MatchesRegularExpression, isNotOnlyDigits

class Acronym(models.Model):
 acronym = models.CharField(maxlength=3, unique=True, db_index=True,
     validator_list=[isNotOnlyDigits,
         MatchesRegularExpression(r'^[A-Z0-9]+$',
         error_message='This field may only contain uppercase letters and' \
         ' numbers.')],
     help_text='A three letter acronym.')
 definition = models.CharField(maxlength=64)

The new model looked like this after I fixed up all of the parameters:

from django.db import models

class Acronym(models.Model):
 acronym = models.CharField(max_length=3, unique=True, db_index=True,
     help_text='A three letter acronym.')
 definition = models.CharField(max_length=64)

The problem above is that the extra validation is now not being done and needs to be reimplemented either in a new form or form field class.

My first attempt probably looked like this:

import re
from django import forms
from models import Acronym

TLA_RE = re.compile(r'^[A-Z0-9]+$')

class NaiveAcronymForm(forms.ModelForm):
 acronym = forms.RegexField(TLA_RE, max_length=3, required=True,
     label='Code', help_text='A three letter acronym.')
 class Meta:
     model = Acronym

The amount of repetition is already becoming obvious and the code above only provides half the solution. We also need to implement the isNotOnlyDigits validation.

import re
from django import forms
from models import Acronym

TLA_RE = re.compile(r'^[A-Z0-9]+$')

class NaiveAcronymForm(forms.ModelForm):
 acronym = forms.RegexField(TLA_RE, max_length=3, required=True,
     label='Code', help_text='A three letter acronym.')

 def clean_acronym(self):
     if self.cleaned_data['acronym'].isdigit():
         raise forms.ValidationError(u"This value can't be comprised soley of digits."

 class Meta:
     model = Acronym

I got to this point in my own code and thought that there must be a way to leverage all the work already being done by Django to generate a form field from a model field. When I make a change to my model fields I want those changes to automatically flow on to my forms.

The solution I came up with was this:

import re
from django import forms
from models import Acronym

TLA_RE = re.compile(r'^[A-Z0-9]+$')

class AcronymField(forms.RegexField):
 """Form field for three letter acronym."""
 default_error_messages = {
     'invalid': u'This field may only contain uppercase letters and ' \
         'numbers.',
     'notonlydigits': u'''This value can't be comprised solely of digits.'''
 }

 def __init__(self, *args, **kwargs):
     """Initialize the field with the acronym regex."""
     super(AcronymField, self).__init__(TLA_RE, min_length=3, *args,
         **kwargs)

 def clean(self, value):
     """Ensure acronym matches regex and is not only digits."""
     value = super(AcronymField, self).clean(value)
     if value.isdigit():
         raise forms.ValidationError(self.error_messages['notonlydigits'])
     return value

class AcronymModelForm(forms.ModelForm):
 """Form for Acronym model."""
 def __init__(self, *args, **kwargs):
     """Programmatically declare fields."""
     super(AcronymModelForm, self).__init__(*args, **kwargs)
     field = self.Meta.model._meta.get_field('acronym')
     self.fields['acronym'] = field.formfield(form_class=AcronymField)

 class Meta:
     model = Acronym

How does this work? First of all, AcronymField is subclassed from forms.RegexField and the __init__() method overriden. It then calls the parent class's __init__() with my extra parameters, the regexp and min_length, as well as passing the original positional and keyword arguments. This ensures that any extra parameters from the model such as required, label, max_length, and help_text are preserved. I also override the clean() method and again call the superclass's clean() method before checking that the value is not only digits.

Next I subclass AcronymForm from forms.ModelForm and override the __init__() method again. Here the superclass's __init__() method is called then the acronym form field is replaced with one that uses the new AcronymField class. It is within the call to field.formfield(...) (see django.db.models.fields.Field class for the gory details) that all the values for required, label, and help_text are taken from the model field and the form field instance created.

While this results in more lines of code for a model form definition it ultimately results in less presentation and validation errors due to model fields and form fields getting out of sync. This approach could probably be generalized even further to a subclass of ModelForm but I've only had to use it for a handful of models so I leave that as an exercise for the reader.

Update: fixed formatting mistake so you can more clearly see the before and after models.

Using Splunk's iplocation search command behind a proxy

Splunk has an iplocation search command that will add City and Country fields to your search results. It does this by looking up the IP addresses it finds using the hostip.info API. Unfortunately if your Splunk server doesn't have direct Internet access then this script will fail.

The script itself is a very simple Python script that use the module urllib.urlopen to make the API call. To get it to use your proxy server is easy.

Make a backup of the original script:

$ cd $SPLUNK_HOME/etc/searchscripts
$ cp iplocation.py iplocation.py.bak

Edit iplocation.py and add the following line below the LOCATION_URL definition:

PROXIES = {'http':'http://proxy.example.com:8080'}

Then find the line that reads:

location = urllib.urlopen( LOCATION_URL + ip )

and change it to:

location = urllib.urlopen( LOCATION_URL + ip, proxies=PROXIES )

Then perform your search and pipe it to iplocation. Make sure to limit your search as the script will do a HTTP request for every IP address it finds.