So what has changed? The most visible change is that there are now at least two separate API’s that you have to work with. You have the File API and the Storage API. The File API which exposes the File class provides a thin wrapper around Python file objects. The Storage API, on the other hand, exposes a base class Storage that you can use to implement custom storage facilities. There is another API that provides FileUploadHandler. This will allow you to customize the way Django handles the uploaded files in their “raw” form. For most purposes, the File and Storage API will suffice.

This post is meant to supplement the information found in the “Handling Uploaded Files” section of the Django File Uploads documentation. You will still need to refer to documentation.

The examples I will be using here implement basic file uploads handling in Django 1.0. The examples provided in the official Django documentation, as of 18 Feb 2009, do not give you an idea of how the File and Storage can be used together. Please note that these examples will not work with Django versions before 1.0.

The Model

Here is the model that we will be using for this example:

from django.db import models

class Attachment(models.Model):
    attached_file = models.FileField(upload_to='attachments')
    mimetype = models.CharField(max_length=64, editable=False)
    created = models.DateTimeField(auto_now_add=True, editable=False)
    updated = models.DateTimeField(auto_now=True, auto_now_add=True, editable=False)

We have marked the 3 fields that are not editable so that the corresponding form fields will not be created for them by Django’s automatic admin and django.forms.ModelForm.

The Form

Normally you would just use django.forms.ModelForm to create the form. Like so:

from django.forms import ModelForm

class AttachmentForm(ModelForm):
    class Meta:
        model = Attachment

This will take care of most use-cases where you have a simple app with one form per model. Taking it a bit further, you could just make use of Django 1.0’s excellent “automatic admin interface”. This is what you should normally do when you are just starting out with Django. However, there are times when you just want a bit more control over the way things are done. In this context, we will create a form class manually to illustrate how you would use Django’s File and Storage API.

from django import forms

class AttachmentForm(forms.Form):
    attached_file = forms.FileField()

    def __init__(self, bound_object=None, *args, **kwargs):
        super(AttachmentForm, self).__init__(*args, **kwargs)
        self.bound_object = bound_object
        self.is_updating = False
        if self.bound_object:
            self.is_updating = True

    def save(self):
        if not self.is_updating:
            self.bound_object = Attachment()
        # Retrieve the UploadedFile object for the attached_file field.
        uploaded_file = self.cleaned_data['attached_file']
        # Clean up the filename before storing it.
        import re
        stored_name = re.sub(r'[^a-zA-Z0-9._]+', '-', uploaded_file.name)
        # Save the file and its metadata.
        self.bound_object.attached_file.save(stored_name, uploaded_file)
        self.bound_object.mimetype = uploaded_file.content_type

In the example above, we used the save() method of the Attachment object’s attached_file field. The save() method’s second argument takes any object that implements the File class’ methods. All file uploads handled by Django’s default upload handlers are UploadedFile objects. Since UploadedFile is a subclass of File we can use it directly in the save() method of our file field.

By default the uploaded file will be saved on the file system under the directory structure specified by settings.MEDIA_ROOT plus the upload_to. So if you have settings.MEDIA_ROOT set to /webapp/media_root adding our upload_to setting for the model above, the file for attached_file will be saved under the directory /webapp/media_root/attached_file. Normally this directory will also be mapped to the appropriate public URL on your webserver’s configuration.

There are times however when you want to change the default storage behavior. This is where the Storage API comes in. For example we want to be able to save files in a different root directory. The problem is that you can only specify one settings.MEDIA_ROOT for your entire app. To do this, we need to pass the parameter storage to our attached_file field.

Here is our model again, this time we specify a custom storage location other than MEDIA_ROOT.

from django.core.files.storage import FileSystemStorage
from django.db import models

attachment_file_storage = FileSystemStorage(location='/webapp/attachments_root', base_url='/attachments')

class Attachment(models.Model):
    attached_file = models.FileField(upload_to='attachments', storage=attached_file_storage)
    mimetype = models.CharField(max_length=64, editable=False)
    created = models.DateTimeField(auto_now_add=True, editable=False)
    updated = models.DateTimeField(auto_now=True, auto_now_add=True, editable=False)

With our modified model, calling the save() method of the attached_file field will save the file under /webapp/attachments_root/attachments.

It does not end with customizing storage locations on the filesystem however. With the Storage API you can customize how files are actually saved by creating a subclass of Storage that overrides the appropriate API methods as documented here and here.

Django FileUploadHandler API

By default Django 1.0 handles file uploads using either django.core.files.uploadhandler.TemporaryFileUploadHandler or django.core.files.uploadhandler.MemoryFileUploadHandler. TemporaryFileUploadHandler is the default for files larger than settings.FILE_UPLOAD_MAX_MEMORY_SIZE. This handler streams the uploaded file to a temporary file saved under settings.FILE_UPLOAD_TEMP_DIR. It then wraps this file with a TemporaryUploadFile object. For smaller files, Django uses the MemoryUploadFileHandler. This handler wraps the uploaded file with a StringIO object. Either way, you get an object with a file-like interface. The only difference is where the file’s data is actually stored.

The FileUploadHandler allows you to customize the default behavior by using subclasses of FileUploadHandler and inserting them into the settings.FILE_UPLOAD_HANDLERS list. This is documented in the “Upload Handlers” section of the Django manual.

Performing Additional Processing

There are cases when you need to do more than just save an uploaded file. For instance, in a photo album application, you might want to use PIL to resize uploaded images or create thumbnails before you save them. In the examples above, we did not do any post-processing of the uploaded files and just saved them right away. It really depends on what you intend to do with the uploaded file. The most common way is to just read the file into a variable using the read() method. You must be careful when you do this however since you will overload your server if the uploaded file is too large. Use the size propery of the File object to test if a file is too large for your system to handle. You would normally do this in your form’s clean() method.

A more efficient way of applying post-processing is to stream the file and do it lazily. PIL’s Image.open() is a good example. Using PIL as an example, here is how you would apply post-processing to uploaded images. This example also shows how to use ContentFile.

from PIL import Image

def postprocess_image(uploaded_file):
    img = Image.open(uploaded_file)
    # ... do image post-processing and manipulation ...
    return img

Elsewhere, in one of your form’s methods (implemented with self.bound_object as the example above)…

def save_uploaded_photo(self):
    uploaded_image = self.cleaned_data['uploaded_image']
    # We assume that uploaded_image.size has been checked in
    # the form's clean() method and that we are good to go.
    img = postprocess_image(uploaded_image)
    if (img.format == 'JPEG'):
        processed_image = ContentFile(img.tostring('jpeg', img.mode))
    elif (img.format == 'PNG'):
        # PIL Image.tostring() does not support PNG encoding for some reason.
        imgstr = StringIO()
        img.save(imgstr, 'PNG')
        imgstr.reset()
        processed_image = ContentFile(imgstr.read())
    elif (img.format == 'GIF'):
        processed_image = ContentFile(img.tostring('gif', img.mode))

    # Clean up the filename before storing it.
    import re
    stored_name = re.sub(r'[^a-zA-Z0-9._]+', '-', uploaded_image.name)
    self.bound_object.image.save(stored_name, processed_image)

The above code is overly simplified and does not perform any additional checking of the file data or any exception handling. Do note that in the versions of PIL I am using, Image.tostring() does not support PNG encoding. This is why some additional code was needed.

At first glance, it may be a bit overwhelming. But the new API allows for greater flexibility in handling file uploads. It allows you to implement anything from custom storage to a CDN to upload progress meters. So learning how to use the new API is well worth the effort.

Please post any corrections in the comments below. Thanks.

a

Facebook, being a PHP-powered site, likes to make the assumption that you will also be writing your app using PHP. They don’t even make any efforts to hide their bias towards PHP. The “official” client library is for PHP. The example program they give to you the first time you sign up is written in PHP. All the relevant starting points all lead towards PHP-centric development.

Thankfully, there are those who realize that PHP is not the be-all and end-all to programming web applications. You can find unofficial client libraries for other programming languages. Download and install the client library for your programming language of choice and follow the documentation and hopefully you get to have everything working on your first try.

If you are programming a web-based Facebook app in Python, you will have two choices:

1. PyFacebook – a little bit on the heavy side since it comes with a framework-neutral core plus Django integration and Google App Engine support. It seems to be more actively maintained. The latest revision was on December 2, 2008 at the time of writing.

2. minifb – small. However it seems that activity has stopped since February 2008.

The choice of client library was a no-brainer for me since the site I am working on was written using Django. After downloading and installing PyFacebook I was able to get a minimal Facebook app up and running. By “up and running” I mean I can view the app’s canvas page on Facebook. However updates to my profile via profile.setFBML() do not seem to be working. After a bit of digging, I was able to find the culprit in a bug report. The modifications suggested by the issue reporter fixed the problem.

The app I was making was supposed to post updates to a Facebook Page. Facebook Pages are a relatively new feature and as a result documentation for it is a bit lacking. The IRC transcript in the Wiki page for Facebook Pages provided some useful clues. There are two things you need to ensure if you need to write a Django view that has to update a Facebook Page:

1. Do not use @facebook.require_login() decorator.

2. The uid parameter for the profile.setFBML() call must be taken from the fb_page_id query parameter.

For example:

from django.template import loader, RequestContext

def update_page_profile(request):
    # facebook attribute in request is added by FacebookMiddleware
    uid = request.GET.get('fb_page_id')
    if uid:
        ctx = RequestContext(request, {'page_id': uid})
        # Do whatever you need to do here...
        t = loader.get_template('facebook/page_fbml.xml')
        # Render the template into the profile parameter for setFBML.
        # We do not use the deprecated markup parameter.
        request.facebook.profile.setFBML(profile=t.render(ctx), uid=uid)
        # Return a redirect to the profile URL for the page.
        # Note that we do not use Django HttpResponseRedirect here.
        return request.facebook.redirect(request.facebook.get_url('profile', id=uid))
    else:
        # Do something else here if the view did not get called
        # with the fb_page_id parameter. One possibility is to check if we got
        # called with a valid user ID and do something completely different.
        pass

One other thing to note about “profile boxes” in Facebook Pages is that they do not automatically update every time you view the Facebook Page. That is, you will need to provide a way of calling the update view manually. What I did was set the update view URL as the “Page Edit URL” for the app when embedded in a Facebook Page. Ideally you should set “Page Edit URL” to output a canvas page where the page admin can update settings for the app. Because the Facebook app I was making was mainly intended for internal use, I opted out of having a dedicated “Page Edit” canvas page and instead linked it directly to the profile update view URL.

a

a

from django.db import models

class MyModel(models.Model):
    file = models.FileField(upload_to=’some/path’)

If you want to to retrieve the URL for the file you would use:

m = MyModel().objects.get(id=1)
m.get_file_url()

Notice that your class automagically gets a method called “get_file_url()”. This is the “magic” part and all of this is done behind the scenes for you. While this is really convenient for getting things done most of the time, there are some times where you need to have more control over what’s returned by these magic methods.

You’re not in OOP Kansas anymore…

All of the _get_FIELD_ methods are defined in the Model class. Normally when you want to replace logic for a method in a subclass, you would override the method in the subclass. So the most straightforward approach would be to create a subclass of Model and override the _get_FIELD_ methods you need. Then you would use the subclass for your own models where you need _get_FIELD_* overridden. This approach does not work (try it if you don’t take my word for it). It will take too long to explain why this won’t work but suffice to say, the current implementation does not allow for subclassing a subclass of Model. The Django team is working on getting around this limitation.

Why would you want to replace them in the first place? In my app, I had the need for “protected” and “unprotected” downloads. I needed the URL’s for “protected” files to be rooted to /protected and “unprotected” files to /media This meant that I had to use a different MEDIA_ROOT and MEDIA_URL for the “protected” files and a different one for “unprotected” files. Because of the way things are set up by default in Django, you can only have one MEDIA_ROOT and MEDIA_URL for all your models. Plus having to embed the URL root in your templates would be tedious to maintain.

So what do you do when you want to replace the logic for _get_FIELD_? Since subclassing won’t work, what other options do you have? One solution, and the one I used for my app, is to use mix-ins. We still won’t be able to override the existing _get_FIELD_ methods in the Model class but using a ProtectedDownloadModelMixIn we will be able to provide alternate implementations under different names. For instance, we only need to override the _get_FIELD_url() method or provide a suitable replacement for it in the mix-in. Let’s call the new method, _get_PROTECTED_FIELD_url(). This is straight out of the django/db/models/base.py module with a few modifications.

class ProtectedDownloadModelMixIn(object):
    def _get_PROTECTED_FIELD_url(self, field):
        if getattr(self, field.attname): # value is not blank
            import urlparse
            return urlparse.urljoin(settings.PROTECTED_MEDIA_URL, getattr(self, field.attname)).replace(’\\’, ‘/’)
        return ”

The _get_FIELD_* methods are automatically added to the fields through the contribute_to_class() method of the field. To make our model use our custom _get_PROTECTED_FIELD_url() we need to create a subclass of either a FileField or ImageField and override the contribute_to_class method. In my app, I needed to subclass the FileField class.

from django.utils.functional import curry
from django.db import get_creation_module

class ProtectedFileField(models.FileField):
    def contribute_to_class(self, cls, name):
        super(ProtectedFileField, self).contribute_to_class(cls, name)
        setattr(cls, ‘get_%s_url’ % self.name, curry(cls._get_PROTECTED_FIELD_url, field=self))

# Register our new field.
data_types = get_creation_module().DATA_TYPES
data_types['ProtectedFileField'] = data_types['FileField']

The last bit will register our new field with Django so that the database backend will know what SQL to use when the field is created. Now to use our custom classes:

from django.db import models

class ProtectedFile(models.Model, ProtectedDownloadModelMixIn):
    file = ProtectedFileField(upload_to=’protected/’
    mime = models.CharField(max_length=64)

That’s pretty much it! Now if you need to override the other _get_FIELD_*, for instance you might want to override _save_FIELD_file() you need to define that method in the mix-in and add the method to the class in contribute_to_class. There are a lot of things that I have left unexplained here. This is because most of this is “magic” and Django is going through “magic removal” so all of this will become irrelevant eventually. If you really need to understand what’s going on, try looking at the code in django/db/models/base.py and in django/db/fields/init.py.

a

A colleague brought up one web hosting company the other day. Ultra-cheap packages and impressive numbers make it very attractive to those who are looking to host their site for the first time. I don’t know what bothers me more. Is it the fact that people buy into these bullshit packages? Or the fact that some unscrupulous web hosting companies take advantage of numbers to market their crappy service. To be fair to them (and so you don’t fall prey to them as well), I won’t mention that company here. However, I will take a look at the packages they offer.

Marketing

Here are some of the interesting bits on their hosting packages page:

• 99.9% Uptime Guarantee
• 27/7 Support
• Lowest Latency
• Highest Reliability
• Powerful Servers

Our servers are packed with powerful Dual Quad Core Clovertown Xeon (8 logical CPU’s) machines with 4GB of memory, SCSI drives, and being monitored by sytem administrator 24/7.

Our network having the best network equipment will give you the hightest reliability, lowest latency and 99.9% uptime guarantee.

Cisco Routers & Switches – 6500, 3750, 2900 Series routers and switches with gigabit speeds available from the individual server to the Internet.

Redundant GigE Backbone – Multi-router gigabit connections to the Internap, SAVVIS, VERIO/NTT, Abovenet, and Global Crossing core backbone exchanges located in the INFOMART in Dallas, Texas

Traffic Management & Analysis – Utilizing Peakflow from Arbor Networks coupled with Netflow data from Cisco routers and switches, network engineers continuously monitor the health, reliability, and performance of the network. Historical analysis is critical for demand and capacity planning.

Network IDS, IPS & DDOS – Industry best solutions from Tipping Point are utilized to guard against denial of service, malicious traffic, viruses, malware, phishing, spyware and other types of Internet threats via wire speed detection and remediation coupled with automatic and zero day response.

The datacenter which our servers are housed in is a tier one datacenter facility located inside the well-known INFOMART telecom hotel near downtown Dallas. Specializing in datacenter and hosting facilities, INFOMART is home to multiple datacenters including MCI, Level 3, Equinix, Verio, Switch & Data, Verizon, DataSides, and more. The building sits atop three redundant electrical grids from TXU delivering diverse power to each quadrant of the building. HVAC needs are supplied via five one hundred ton onsite water chillers to deliver N+1 cooling requirements. Home to over twelve switch sites and thirty-five different carriers, INFOMART houses technology and telecom related businesses in a complex communications ecosystem.

• 2000AMPS 480V Input Power
• Parallel 500Kva UPS Battery Backup Units
• 2000Kw Diesel Generator with Onsite Fuel Storage
• Redundant Liebert 20 Ton HVAC Units
• Pre-Action Dry Pipe Fire Suppression
• Digital Security Video Surveillance

At first glance it’s all pretty impressive huh? If you’re impressed then they almost have you at their mercy at this point. Don’t fall for it though. It’s all smoke and mirrors. A company putting up an infrastructure such as this will have spent millions of dollars to set it up. Every web hosting company I have reviewed has drivel like this in one form or another. Surely if they all had “data centers” as they claim on their marketing page, Dallas would be full of these concrete bunkers by now.

The truth of the matter is that these web hosting companies are just copy/pasting specs from the company doing the actual web hosting. More likely these web hosting companies are co-located or have managed servers under companies like RackSpace, Serverbeach, or Softlayer.

It’s not that they’re lying or anything. Since they are co-located or managed by another company, then the drivel also applies to them as well.

The Numbers! They Lie!

Web hosting companies will usually offer several packages with varying price points and features. Usually it all boils down to the following:

• Disk quota
• Bandwidth
• Number of email accounts
• Number of FTP accounts
• Number of subdomains
• Number of databases
• Occasionally, number of support tickets

Here’s a very interesting set of packages from one web hosting company:

Package 1

• Monthly rate – $3.95
• Disk Quota – 1GB
• Bandwidth – 10GB/Month
• Email Accounts – Unlimited
• FTP Accounts – Unlimited
• Subdomains – 10
• Databases – 10 MySQL

Package 2

• Monthly rate – $7.95
• Disk Quota – 5GB
• Bandwidth – 50GB/Month
• Email Accounts – Unlimited
• FTP Accounts – Unlimited
• Subdomains – 20
• Databases – 20 MySQL

Package 3

• Monthly rate – $14.95
• Disk Quota – 10GB
• Bandwidth – 100GB/Month
• Email Accounts – Unlimited
• FTP Accounts – Unlimited
• Subdomains – 40
• Databases – 40 MySQL

Pretty impressive huh? But let’s take a closer look and put ourselves in the shoes of someone who wants to run a web hosting business. First let’s deal with the claim of “Our servers are packed with powerful Dual Quad Core Clovertown Xeon (8 logical CPU’s) machines with 4GB of memory, SCSI drives”. So far I’ve found that the cheapest Dual Quad Core Xeon servers cost around $259 (not including taxes) for a managed server at Softlayer (Rackspace and Serverbeach have them but at $300+). The cheap setup I’m going with does not have any add-ons for storage, CPanel with WHM at additional $25. The Linux distribution is not a factor as all of the offerings are the free distros anyway. Total cost each month to keep the server running is $284.

The server setup is like this:

• Processor – Dual Processor Quad Core Xeon 5310
• Memory – 2 GB FB-DIMM Registered 533/667
• Uplink Port Speed – 10 Mbps Public & Private Networks
• Public Bandwidth – 2000 GB Bandwidth
• Disk – 250GB SATA II

OK. So it takes $284 to run the server each month. Let’s take the most expensive package offering, Package 3. To break even, you need 284 / 14.55 = 18.99 hosting clients. Let’s round it off to 19. That gives you $284.05 per month, with a profit of $0.05. Not enough profit if you ask me. What’s the point of running a business if you don’t turn any profits?!? Let’s up our clients to 25! We get $373.75 a month. That’s $89.75 in monthly profits.

Now let’s turn our attention to resource usage. First off, disk space usage. Remember we have a 250GB disk. The OS will use around 10GB of that for a typical server installation. For 25 clients each requiring 10GB disk quota, we would need 25 * 10GB = 250GB. That would not leave us enough room for our operating system! Let’s say we adjust our purchase and instead of the 250GB disk, we get the 500GB disk. This will cost us an additional $30 each month. So we are left with $59.75 in monthly profits. But now we have a bigger disk, we can accommodate more clients. So let’s assume we have 40 clients. That’s 40 * $14.95 = $598 per month. We have a monthly profit of $284 now and we have 100GB disk space to spare as well.

Now let’s look at bandwidth usage. Let’s say all 40 clients soak up their 100GB per month. That’s already 4000GB, twice as much bandwidth that was allocated for our server. Say we purchase more bandwidth 4000GB is $200 extra. But that would be cutting it really close. So now our profits are cut down to $84 per month putting us back where we started!

I’ve been in the web hosting business once and quite frankly tech support is a nightmare. Especially if you make the mistake of enabling phone-in tech support. Phone in tech support is something that you would outsource and call centers don’t come cheap. So for a budget outfit like the one I’m describing here, it really makes you wonder if it’s not just some fly-by-night company.

Afterthoughts

I have been contemplating on running a small “custom web hosting” shop for 2 years now. I planned to focus on catering to Django-powered sites. But with the recent release of Google App Engine I think it really hit the nail on the coffin for that idea. It’s really not worth the expense and the tech support nightmare.

So far SoftLayer is the cheapest managed hosting I’ve found. I’ve looked at DimeHost but I hear a lot of their IP’s are black listed because spammers used them in the past.

a

For instance, viewing a post on one site would require a URL like /blog/posts/1/view/ and the other site it would be /diary/posts/1/view/. For blog archives you would have URLs like /blog/archives/ and /diary/archives/. You would use a URLconf pattern like this:

(r&#8217;^posts/(?P<post_id>\d+)/view/$&#8217;, &#8216;post_view&#8217;),
(r&#8217;^archives/$&#8217;, &#8216;post_archive&#8217;),

You can hard-code the URLs in your views and templates, but that would be less than ideal since you now have to maintain two slightly different versions of what is essentially the same code base. What you need to do is somehow obtain the URL properly rooted to either /blogs/ or /diary/.

Django provides an API for doing just that, django.core.urlresolvers.reverse() will give you the URL for the view you specify. To retrieve the URL for archives, you do the following:

from django.core import urlresolvers

archives_url = urlresolvers.reverse(’Blog.views.post_archive’)

Retrieving the URL for posts is a bit different and this is where I found a gotcha:

from django.core import urlresolvers

posts_url = urlresolvers.reverse(’Blog.views.post_view’)

Will give you NoReverseMatch exception. The problem here is that the URLconf for posts requires a post_id parameter and the resolver will not be able to return the reverse resolution to a URL without it. So you need to do:

from django.core import urlresolvers

posts_url = urlresolvers.reverse(’Blog.views.post_view’, None, (’0′))

Note that the '0' is just a dummy value that we use so that reverse() will work properly. It’s easy overlook this important detail especially when you have been using reverse() with views that don’t take arguments for a while.

There’s another gotcha with urlresolvers.reverse() and this time it involves its use in decorator functions. Put simply, you cannot use urlresolvers.reverse() from within a decorator function such as user_passes_test. This is because decorators are parsed and evaluated before the URLconfs are processed. Because the decorator creates a closure, it ends up with an invalid URLconf.¹ There is a way to use urlresolvers.reverse() inside a decorator but you need to ensure that the lookup is triggered later, rather than upon evaluation in the decorator (lazy evaluation).

There are two solutions. One found here uses a wrapper method around the reverse() call. The other one found here wraps the call with a “lazy string” class. Both solutions seem to work rather well, but I particularly like the second implementation.

¹ As far as I understand it, this is what happens. Please correct me if I’m wrong. Thanks.

a

First up are newforms. Older code based on 0.95 and earlier use the older “Manipulators” system. The replacement system is “newforms” which, until 0.96.1, had only been available from the SVN versions. Newer code should make use of the “newforms” system as Django team has already announced that they will remove the old Manipulators system eventually.

newforms, New Code

Like the rest of Django, newforms is a fairly young codebase. Continuous development efforts add and remove features all the time. So it should come as no surprise that one feature/quirk that you relied upon in your old code will no longer work with new versions of the framework. At least the Django team has the good sense to let everyone know what will break in new releases (unlike some other web framework people out there). Documentation is almost complete for newforms, so you should refer to the latest development snapshot of the docs when working with newforms.

Although they are an improvement over the old Manipulators system, newforms are not without their quirks though. They have a few undocumented quirks that can take one by surprise and leave one tearing bits of their hair out in frustration. For example:

# myform.py

from django import newforms as forms

class MyForm(forms.Form):
    name = forms.CharField(label=”Your Name”, max_length=64)
    age = forms.IntegerField(label=”Age”, min_value=0)

Suppose you wanted to access the fields in your code:

>>> import myform
>>> f = myform.MyForm()
>>> print f.name

You will get a surprise error message like the one below:

Traceback (most recent call last):
  File &#8220;<console>&#8220;, line 1, in <module>
AttributeError: &#8216;MyForm&#8217; object has no attribute &#8216;name&#8217;

What the heck just happened?!? Clearly the class definition has an attribute with the name ‘name’.

>>> dir(f)
['_BaseForm__errors', '__class__', '__delattr__', '__dict__', '__doc__', '__geta
ttribute__', '__getitem__', '__hash__', '__init__', '__iter__', '__metaclass__',
 '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr_
_', '__str__', '__unicode__', '__weakref__', '_errors', '_html_output', 'add_pre
fix', 'as_p', 'as_table', 'as_ul', 'auto_id', 'base_fields', 'clean', 'data', 'e
rrors', 'fields', 'full_clean', 'initial', 'is_bound', 'is_valid', 'non_field_er
rors', 'prefix']

As you can see, the fields you declared in the class are no longer there. What happened to them? Where did they go? What if you need to access field objects directly in your code? Using self.clean_data¹ is no good since all that would give you is the cleaned value of the field.

If you poke around the Django source code for newforms, you will see that the Form class is declared as follows:

class Form(BaseForm):
    “A collection of Fields, plus their associated data.”
    # This is a separate class from BaseForm in order to abstract the way
    # self.fields is specified. This class (Form) is the one that does the
    # fancy metaclass stuff purely for the semantic sugar — it allows one
    # to define a form using declarative syntax.
    # BaseForm itself has no way of designating self.fields.
    __metaclass__ = DeclarativeFieldsMetaclass

What this means is that class creation is customized using the DeclarativeFieldsMetaclass. This is one demonstration of Python’s metaclass programming. Read more about metaclass programming in Python here. For those in a hurry, in simple terms Django customizes the way Form objects are instantiated. The DeclarativeFieldsMetaclass.new() method scans the incoming class’s attributes for any instances of Field and its subclasses. These are then stuffed into a SortedDictFromList object which is then attached to the outgoing class’s fields attribute. The result is that, the outgoing class is “stripped” of the attributes that are instances of Field and its subclasses. The other fields that were declared in the incoming class are not touched and remain accessible in the outgoing class.

# myform2.py

from django import newforms as forms

class MyForm2(forms.Form):
    name = forms.CharField(label=”Your Name”, max_length=64)
    age = forms.IntegerField(label=”Age”, min_value=0)
    regular_attr = “This is a regular attribute, one that is not an instance of Field or its subclasses.”

And then…

>>> import myform2
>>> f2 = myform2.MyForm2()
>>> print f2.regular_attr
This is a regular attribute, one that is not an instance of Field or its subclasses.

But,

>>> print f2.name

Traceback (most recent call last):
  File &#8220;<console>&#8220;, line 1, in <module>
AttributeError: &#8216;MyForm2&#8242; object has no attribute &#8216;name&#8217;

So how does one get to the original fields? For instance, you would like to modify one of the field’s properties based on arguments given through a constructor. In one app, I needed to modify some field’s required attribute based on whether or not the form is used for editing or adding a new object. You will need to do:

def __init__(self, editing=False, *args, **kwargs):
    super(MyForm, self).__init__(*args, **kwargs)
    if editing:
        self.fields['field_name'].required = True

At one point I needed a Forms class that would automatically apply the CSS class “required” for all required fields in the form.

class StyledForm(forms.Form):
    def __init__(self, *args, **kwargs):
        super(StyledForm, self).__init__(*args, **kwargs)
        for field_name, obj in self.fields:
            if obj.required:
                css_classes = obj.widget.attrs.get(’class’)
                if not css_classes: obj.widget.attrs['class'] = ‘required’
                elif not ‘required’ in css_classes: obj.widget.attrs['class'] += ‘ required’

These are just a couple of things that I needed to do where I required access to the form object’s fields. I certainly hope that eventually it will be possible to get to the fields in a newforms form without the need to jump through hoops like this.

¹ clean_data is also known as cleaned_data in the latest SVN versions.

a

Not satisfied with the PHP5 frameworks I found I decided to write my own (again). I looked at one particular candidate — [Picora ™: PHP Micro Framework][picora]. It’s small and focused on doing one thing: provide an MVC framework for PHP. While it has both qualities that I was looking for in an MVC framework there are a couple of issues that I found:

• it currently lacks an ActiveRecord implementation.

Note that the model() calls use a PHP ActiveRecord implementation which is not part of Picora, substitute with your own database logic for the time being.

• it makes the same (IMHO) mistake as CakePHP and Rails in that actions are mapped directly into methods in your Controller class without giving you the option of delegating it to a separate class. While there’s nothing wrong with this, I found that having all your actions in one big file becomes unwieldy when your project grows. Which was why I came up with [a modified action mapper for CakePHP][cakephp-sucks01].

So, for now Picora ™ doesn’t quite cut it for me yet. I began writing a new framework, based on some ideas I had from a year ago as well as taking a lot of inspiration from Django. I am writing the model class hierarchy based on Django’s excellent ORM. This will also be my first large project written using PHP5. Needless to say it has been pretty frustrating to find that PHP5 lacks namespace support. I was left wondering, “With PHP5, they practically went out and copied Java and yet they forgot to implement namespaces?!?”.

I did a search on Google for “php namespace support” and what I got was a depressing state of affairs for this important feature. It turns out that at the time, PHP5 namespace support is not a simple thing to implement and one of the major issues is the namespace separator. Maybe this is a testament to the fact that the PHP language is horribly broken an a multitude of places? Or maybe it’s because “design by committee” does not work? I found [one particularly entertaining post about the issue][shalosh] from 2005 and [a more recent posting][phpnamespaces] citing a [patch that implements namespaces for PHP6][phpnamespace-patch]. It looks like [namespace support will be in PHP6][zend-weekly]. I can only hope that this will be backported into a point release for PHP5.

Among other things I would like to see in PHP6:

• Closures or real anonymous functions at the very least. No, create_function() does not count!
• Inner classes.
• Inner functions.
• Make built-ins like include(), include_once(), require(), and require_once() throw exceptions in addition to their original behavior. But it would really help to do away with the old *_error_handler() way of doing things.

These are just some of the features that I wish was in PHP, ever since PHP4.

[picora]: http://livepipe.net/projects/picora/ “Picora ™: PHP Micro Framework” [cakephp-sucks01]: http://abing.gotdns.com/posts/2006/cakephp-delegating-actions-to-separate-classes/ “CakePHP: Delegating Actions to Separate Classes” [shalosh]: http://blog.phpdoc.info/archives/27-+1-for-Shalosh-Nekudotayim.html “Shalosh Nekudotayim” [phpnamespaces]: http://www.gravitonic.com/blog/archives/000418.html “PHP Namespace Support” [phpnamespace-patch]: http://news.php.net/php.zend-engine.cvs/5894 “PHP6 Namespace Support Patch” [zend-weekly]: http://devzone.zend.com/article/2336-Zend-Weekly-Summaries-Issue-348 “Zend Weekly Summaries”

a

Of course, hosting for these patches continues as well as my ThemeableView class which some CakePHP users out there found useful. If anyone else is interested in taking the reins for this class, please feel free to do so. No need to contact me or anything, but a trackback to this post would be nice so that I will know who is currently maintaining this class. This will allow me to direct them to you for any updates that you may post.

On another note, the project that has been taking much of my time lately is a new website for posting free online classified ads for cars, trucks, and bikes. This website is currently US-centric. We have plans to extend worldwide if all goes well with the current setup. So, if you’re in the US and you wish to sell your vehicle, or if you’re looking to buy a preowned car, truck, or bike check out PreOwnedCar.com. If you know someone with the same needs, please tell them about this site.

We are also currently in the final design phases for PhilWeavers Version 4. I am currently divided between doing this new version using Django as a base or simply going back to the drawing board and creating a new PHP5-based framework. We’ll see how this pans out in the later stages of development.

a

It makes me wish CakePHP offered more when it comes to object-relational mapping. How I wish it was as simple as defining your models and running php cake/scripts/bake.php syncdb. CakePHP does it the other way around: you create the database first and then use cake/scripts/bake.php to create your models and views.

I found that Django’s way of doing ORM is significantly faster. You specify your models as Python classes using Django’s modeling API. You run python manage.py syncdb and Django takes care of generating the correct SQL for your database backend. It takes model relationships into account and also generates the correct foreign key constraints. Beautiful!

A similar system should also be possible to implement in CakePHP. I’ve been having a few ideas of how to go about it and hopefully, I can find the time to do them as an add-on to CakePHP 1.2.x (and hopefully, someone else is already working on the same thing).

Here is what I imagine it would look like:

class User extends MetaModel {
    var $username;
    var $password;
    var $lastname;
    var $firstname;
    var $is_active;
    var $is_staff;
    var $is_admin;
    var $created;
    var $updated;

    function __construct() {
        $this->username = FieldFactory::CharField(’Username’, 32, IS_PK, IS_REQUIRED, NOT_NULL);
        $this->password = FieldFactory::CharField(’Password’, 64, NO_INDEX, IS_REQUIRED, NOT_NULL);
        $this->lastname = FieldFactory::CharField(’Last Name’, 64, INDEX, IS_OPTIONAL, NOT_NULL);
        $this->firstname = FieldFactory::CharField(’First Name’, 64, INDEX, IS_OPTIONAL, NOT_NULL);
        $this->is_active = FieldFactory::BooleanField(’Active’, IS_REQUIRED, NOT_NULL, false);
        $this->is_staff = FieldFactory::BooleanField(’Staff’, IS_REQUIRED, NOT_NULL, false);
        $this->is_admin = FieldFactory::BooleanField(’Admin’, IS_REQUIRED, NOT_NULL, false);
        $this->created = FieldFactory::DateTimeField(’Created’, NO_INDEX, IS_REQUIRED, NOT_NULL);
        $this->updated = FieldFactory::DateTimeField(’Updated’, NO_INDEX, IS_REQUIRED, NOT_NULL);
    }
}

FieldFactory would have static methods for instantiating the correct field objects for each type. It would also have methods for generating constraints for implementing many-to-one and many-to-many relationships. I think a new model class is required to be able to do the additional magic without changing the current API.

I haven’t thought about how to implement the analogs of Admin and Meta inner-classes yet. I think those can be done as fields in the model, maybe $this->__meta and $this->__admin. As for QuerySets, Q objects and other classes in the Django DB API those are pretty much doable under PHP.

For SQL generation using the models, I imagine the backend will first load all the model classes found. Then models extending the MetaModel base class are instantiated and introspected to generate the correct DDL statements in database-specific SQL.

This is not a trivial undertaking and will take a lot of planning and thought. Model introspection and ORM should use as much of the existing CakePHP API as possible.

One last thing, if you are working on a similar system for CakePHP leave comments below. Remember: DRY.

a