In this part of our per-paragraph commenting app, we will build the AJAX API using Django-Rest-Framework, rather than reinventing a major part of the wheel with Django's inbuilt support for serializing and deserializing data.
So, having developed a very simple Ajax request handler in our previous tutorial, we now delve into Django REST framework and use it to develop our app's REST API before we finally hook it up to the front end. Long way, huh? Sorry.
Here is what we'll do... or before that, I think it would benefit if you read the tutorials on DRF here. For now, I'm implicitly using their terminology pretending that you and I fully understand what it means, just like while making cake we talk about bread, even when bread is made of what, yeast, and grain?
So, we'll first create serializers for the models we wish to expose. These will be:
- Users
- Annotations
- Blog Content (on which annotations were made)
Then, we'll write/modify our views to support the queries and test them. Serializers are like Forms in Django and more or less the same kind of easy or difficult depending on how you find forms.
Okay, lets go:
We first define very straightforward serializers using model Serializers. These create serializers like ModelForms created forms for our Data Models implicitly, but again, if you don't get it, please read their documentation first. Christie has done a fine job in documenting most relevant things in the form of a neat tutorial:
from rest_framework import serializers
from annotations.models import Annotation
from blogging.models import BlogContent
from django.contrib.auth.models import User
class BlogContentSerializer(serializers.ModelSerializer):
class Meta:
model = BlogContent
fields =('id', 'title', 'create_date', 'data', 'url_path',
'author_id', 'published_flag', 'section', 'content_type',
'tags',)
class AnnotationSerializer(serializers.ModelSerializer):
class Meta:
model = Annotation
fields = ('content_type', 'object_id', 'id', 'date_created', 'date_modified','content_object',
'body', 'paragraph',
'author', 'shared_with',
'privacy', 'privacy_override', )
class UserSerializer(serializers.ModelSerializer):
class Meta:
model = User
fields = ('id', 'username', 'first_name', 'last_name')
Now, let us try to get an instance of annotations and parse it. The good thing about Unit Tests is that I can do the python shell stuff there, in a unit test while I am tinkering with code, and make it grow as a test, without having to type the same thing in the shell again and again. So, that saves much time.
Also, I'd like to fix up those fixtures in every test so, I've separated out the setUp() and tearDown into a base class which all other test classes in this case can inherit.
class TestAnnotations(TestCase):
fixtures = ['fixtures.json',]
def setUp(self):
self.user = User.objects.get(username="craft")
def tearDown(self):
pass
The test is as:
from annotations.serializers import AnnotationSerializer
from rest_framework.renderers import JSONRenderer
class TestSerializers(TestAnnotations):
def test_create_serializer_class(self):
annotation = Annotation()
annotation.content_type=ContentType.objects.get(model='blogcontent', app_label="blogging")
annotation.object_id= str(1)
annotation.body="This is a test annotation"
annotation.paragraph="1"
annotation.author= User.objects.get(id=1)
annotation.save()
obj = AnnotationSerializer(annotation)
print(obj.data)
json = JSONRenderer().render(obj.data)
print json
Running the tests gives:
TypeError: <BlogContent: I have a dream by Martin Luther King> is not JSON serializable
Hurrah! You know, I like segmentation faults in C while development, and Exceptions in python for they tell exactly where things went wrong. So, it says that the content_object is not serializable. That is true. For that, we need to tell the BlogContent serializer of the relationship from annotations:
Adding to BlogContentSerializer and rerunning the tests:
annotation = serializers.RelatedField(many=True) AssertionError: Relational field must provide a `queryset` argument, or set read_only=`True`.
No, no, nom nom nom! Err, is that what we want? Remove it for now. What we want is a JSON representation of BlogContent Field when we are fetching annotations. So, lets stick to that. The BlogContent object is the 'content_object' field. So, whatever has to be done has to be done IN the annotations serializer.
I was never baptised into OOPs. Consequently, I never understood it natively. I understand the concept more or less from a C-Oriented perspective but some things have always confused me. For example, why does 'over-riding' a function work (vtables?)? So, I tried to imagine what might be happening behind the scenes in that scenario. Why do we need that? Because, we need to modify the behaviour of how a serialized value can be passed into content_object field, when it is passing an instance?
So, consider this:
A class may be visualized as a cluster of related attributes and methods (which in turn are attributes, except that they can do something). So, ultimately, all methods as well as attributes are represented as a single place in memory. The function just take larger space and the system/compiler implicitly does not know what or how to do something with them. What it does know is that they are composite data-types, made up of semantics, operations and interpretations that a compiler does understand. So, if we say 'int a', the compiler knows how to interpret it. It knows that it must read 4 bytes at once (in most cases). If we say a = 2+3, the compiler knows what to do with that '+'. But if we said 'a= addition(1,2)', compiler would not implicitly know. All it knows is to plough through to find out. It would start at the memory position where addition(c,d) resides and move forward, trying to see if it understands something. It would encounter 'sum=c+d' and it would know what to do with it. When it would see 'return sum', it would know what 'return' means. In this way, for a compiler/interpreter, function is also a big variable, or rather, chunk of interpreted simpler instructions. Now, say I wrote:
a= addition(1,2)
and then, somewhere down the line I wrote
a=subtraction(3,5)
It would just be valid. Now, suppose that I made the a=addition(1,2) implicit, by putting it in a class, and then while inheriting the class, I wrote a=subtraction(3,5) in the descendant class, it would mean just the same thing. So THAT as it appeals to me, is what happens when we 'inherit' a class and override its method or an attribute's value. There are ofcourse many other things going on, or else, how would super() work, but it is just for understanding how it works in a rudimentary fashion.
Thus, while we inherited the ModelSerializer, which gave the 'content_object' a RelatedField() class, we can, in our class, override how 'content_object' behaves. So, what we want to achieve is, to get a serialized value here (to rid ourselves of the error). So, we define a new class, which inherits the RelatedField class, and override the default method:
from rest_framework.fields import ReadOnlyField
class SerializeReadOnlyField(ReadOnlyField):
def to_representation(self, value):
if isinstance(value, BlogContent):
return BlogContentSerializer(value).data
Here, we check that if the passed object is an instance of BlogContent, then return its serialized value. If annotation can be plugged on other types, their cases would have to be added in here.
And then, we alter the AnnotationSerializer class:
class AnnotationSerializer(serializers.ModelSerializer):
content_object = SerializeReadOnlyField()
class Meta:
model = Annotation
fields = ('content_type', 'object_id', 'id', 'date_created', 'date_modified','content_object',
'body', 'paragraph',
'author', 'shared_with',
'privacy', 'privacy_override', )
Running the tests again gives:
TypeError: <taggit.managers._TaggableManager object at 0x7fc9c506d4d0> is not JSON serializable
That's progress! Also, we got the full Content object serialized. But if we go on like this, we don't know where it would end. (Or we could just not fetch Tags). But even then, it sounds stupid that for a small annotation, complete content object is returned. As a relation, we would need at the maximum, its Primary Key, or a hyperlink to it.
So, we could do the hyperlink part, by using the get_absolute_url() method of BlogContent passes our tests.
class SerializeReadOnlyField(ReadOnlyField):
def to_representation(self, value):
if isinstance(value, BlogContent):
return value.get_absolute_url()
But then, we can do better, let REST handle it within REST itself. We may switch back later (this new method will be very useful when complete projects are based on DRF rather than a mixture of each). So, we will use HTML Serializers instead of Model Serializers. But for that, we'll need a view, and we haven't written any, yet.
Now, because I said that I'll assume that you have read the tutorial on Django Rest Framework, I am going to play that assumption here and cut to the chase here in the views. We'll be using ViewSets.
We'd want to use some ready made functions for User objects, and BlogContent objects because we don't want to use them yet in a way to create them, but as read-only or relation fields. Now, to do that, Django Rest Framework needs to have a view for each (for URLs).
class UserViewSet(viewsets.ReadOnlyModelViewSet):
"""
This viewset automatically provides `list` and `detail` actions.
"""
queryset = User.objects.all()
serializer_class = UserSerializer
class BlogContentViewSet(viewsets.ModelViewSet):
"""
This viewset automatically provides `list`, `create`, `retrieve`,
`update` and `destroy` actions.
"""
queryset = BlogContent.objects.all()
serializer_class = BlogContentSerializer
permission_classes = (permissions.IsAuthenticatedOrReadOnly,
IsOwnerOrReadOnly,)
def pre_save(self, obj):
obj.author_id = self.request.user
class AnnotationViewSet(viewsets.ModelViewSet):
queryset = Annotation.objects.all()
serializer_class = AnnotationSerializer
permission_classes = (permissions.IsAuthenticatedOrReadOnly, AnnotationIsOwnerOrReadOnly,)
def pre_save(self, obj):
obj.user = self.request.user
And we would like to use the browsable API for REST to see things on the page. For that, we'd be using the URL router, and a REST API Homepage. In views, add:
@api_view(('GET',))
#If not set, the API root will assert for not having appropriate permissions.
@permission_classes((permissions.IsAuthenticatedOrReadOnly, ))
def api_root(request, format=None):
return Response({
'blogcontent': reverse('annotations:blogcontent-list', request=request, format=format),
'user': reverse('annotations:user-list', request=request, format=format),
'annotations': reverse('annotations:annotation-list', request=request, format=format),
'currentUser': reverse('annotations:current-user', request=request, format=format),
})
Here, we are providing URL links for 4 types of requests:
- For browsing blogcontent
- For browsing users
- For seeing the current user
- For manipulating/browsing annotations
For now, our REST API Root resides in the annotations App itself. In a bigger ecosystem, I'd break out the REST methods to their respective apps.
In annotations/urls.py add:
from annotations.views import (
BlogContentViewSet, UserViewSet, AnnotationViewSet,
BlogContentCommentView, CurrentUserView,
api_root)
blogcontent_list = BlogContentViewSet.as_view({
'get': 'list'
})
blogcontent_detail = BlogContentViewSet.as_view({
'get': 'retrieve',
})
user_list = UserViewSet.as_view({
'get': 'list'
})
user_detail = UserViewSet.as_view({
'get': 'retrieve'
})
annotation_list = AnnotationViewSet.as_view({
'get': 'list',
'post': 'create'
})
annotation_detail = AnnotationViewSet.as_view({
'get': 'retrieve',
'put': 'update',
'patch': 'partial_update',
'delete': 'destroy'
})
Here, we take advantage of auto-creating views from classes for readonly purposes in BlogContent and in Users, but add other REST methods for annotations App.
In the URL patterns, add:
url(r'^blogcontent/$', blogcontent_list, name='blogcontent-list'),
url(r'^blogcontent/(?P<pk>[0-9]+)/$', blogcontent_detail, name='blogcontent-detail'),
url(r'^blogcontent/(?P<pk>[0-9]+)/comments/$', BlogContentCommentView.as_view(), name='blogcontent-comments'),
url(r'^users/$', user_list, name='user-list'),
url(r'^users/current/$', CurrentUserView.as_view(), name='current-user'),
url(r'^users/(?P<pk>[0-9]+)/$', user_detail, name='user-detail'),
url(r'^annotations/$', annotation_list, name='annotation-list'),
url(r'^annotations/(?P<pk>[0-9]+)/$', annotation_detail, name='annotation-detail'),
url(r'^rest/$', api_root),
And, don't forget to add the 'rest_framework' app into your INSTALLED_APPS. If you have not synced your DB so far, do it once by:
python manage.py syncdb
and start the dev server by:
python manage.py runserver
Visit 127.0.0.1:8000/annotations/rest to view your REST homepage. If everything went fine, it should show you 4 links, to the ones we had written in the api_root view.
To add login support, put this URL pattern in the main project URL file:
url(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework')),
If you do login, you will see that an input form appears only for annotations, and not for any other hyperlink. So far so good.
Now, let us write some tests about this. As a general rule, we are not supposed to test other module. So, we will be testing our module's functionality rather than REST framework's. So far, we haven't written much of our own functionality, and have been piggybacking REST framework's functionality. So, what do we test about it? All we can do is test our paths, the expected outputs and failures. So, the teeny weeny 'tests' that we've been writing for REST framework are not really tests (if you look at them), but just a convenience while exploring the framework.
Erm, this tags is still going to be a nuisance, lets remove it from the serializer fields for now, we are not making BlogContent REST API right now. So, lets stick with the minimal functionality rule.
However, you will observe that the shared_with field is not being displayed in the Web API interface. That is because it is a ManyToMany Field and by default, it is set to ReadOnly in DRF. However, we don't want it such.
The serializer attribute names for foreign keys MUST be the same as their related names or Django will keep throwing an error saying that the field is not a part of the model.
Also note, that for Foreign Keys, when we are specifying 'related_name' attribute, the serializer attribute too must be named the same, or else, it won't work! I spent many hours confused about why was it happening until I just tried out using the same name in the serializer as was given in the model as its related_name.
The Annotation Serializer needs some rework, for it uses two patterns that are not 'regular'. First, it uses a manyToMany field relationship, and second, it is a generic foreign key, which again is a custom manager. So, we cannot use a ViewSet for it:
class AnnotationSerializer(serializers.ModelSerializer):
#I want the content object is a hyperlink to the ContentObject
content_object = SerializeReadOnlyField()
shared_with = serializers.PrimaryKeyRelatedField(many=True, read_only=False, queryset=User.objects.all())
author = serializers.ReadOnlyField(source='author.username')
class Meta:
model = Annotation
fields = ('content_type', 'object_id', 'id', 'date_created', 'date_modified','content_object',
'body', 'paragraph',
'author', 'shared_with',
'privacy', 'privacy_override', )
def create(self, validated_data):
print "In create"
print validated_data
annotation = Annotation()
annotation.author = validated_data.get('author')
annotation.body = validated_data.get('body')
annotation.content_type = validated_data.get('content_type')
annotation.object_id = validated_data.get('object_id')
annotation.paragraph = validated_data.get('paragraph')
annotation.privacy = validated_data.get('privacy')
annotation.privacy_override = validated_data.get('privacy_override', False)
#Get row from contentType which has content_type
content_object = ContentType.objects.get_for_id(annotation.content_type.id)
annotation.content_object = content_object.model_class().objects.get(id=annotation.object_id)
print annotation.content_object
annotation.save()
print validated_data.get('shared_with')
for user in validated_data.get('shared_with'):
sharing = AnnotationShareMap(annotation=annotation,
user=user)
sharing.save()
return annotation
def update(self, instance, validated_data):
print "In update"
annotation = instance
annotation.author = validated_data.get('author', annotation.author)
annotation.body = validated_data.get('body', annotation.body)
annotation.content_type = validated_data.get('content_type',annotation.content_type)
annotation.object_id = validated_data.get('object_id',annotation.object_id)
annotation.paragraph = validated_data.get('paragraph',annotation.paragraph)
annotation.privacy = validated_data.get('privacy',annotation.privacy)
annotation.privacy_override = validated_data.get('privacy_override',annotation.privacy_override)
#Get row from contentType which has content_type
content_object = ContentType.objects.get_for_id(annotation.content_type.id)
annotation.content_object = content_object.model_class().objects.get(id=annotation.object_id)
print annotation.content_object
annotation.save()
print validated_data.get('shared_with')
for user in validated_data.get('shared_with'):
sharing = AnnotationShareMap(annotation=annotation,
user=user)
sharing.save()
return annotation
All is well. We are doing here what we had done in the view in case of regular POST and Ajax requests. We fetch the instance, decode its generic foreign key and return the appropriate values, or save them.
Add up tests to POST, PUT and DELETE annotations and we're done here
def _create_annotation(self, content=None):
#use test client to POST a request
self._require_login()
print(self.user.is_authenticated()) # returns True
string_data = {
'content_type': content['content_type'],
'object_id':content['object_id'],
'paragraph':content['paragraph'],
'body': content['body'],
'author':content['author'],
'privacy':content['privacy'],
'privacy_override': content['privacy_override'],
'shared_with': content['shared_with'],
}
json_data = json.dumps(string_data)
return self.client.post(
'/annotations/annotations/',
content_type='application/json',
data = json_data,
)
def test_POST_annotation(self):
response = self._create_annotation(content={
'content_type': '9',
'object_id':'1',
'paragraph':'1',
'body':'Dreaming is good, day dreaming, not so good.',
'author':str(self.user.id),
'privacy':'3',
'privacy_override': '0',
'shared_with': ['1'],
})
#Expect a JSON object in response
#Try to get all the annotations. Count should be 1, and it must be ours.
#print "Response"
#print response.content.decode()
self.assertEqual(Annotation.objects.all().count(), 1)
annotation = Annotation.objects.all()[0]
#print 'Annotation content_object'
#print annotation.content_object
self.assertEqual(annotation.body, 'Dreaming is good, day dreaming, not so good.')
self.assertEqual(annotation.paragraph, 1)
map = AnnotationShareMap.objects.all()
for share in map:
print str(share)
def test_PUT_annotation(self):
self._create_annotation(content={
'content_type': '9',
'object_id':'1',
'paragraph':'1',
'body':'Dreaming is good, day dreaming, not so good.',
'author':str(self.user.id),
'privacy':'3',
'privacy_override': '0',
'shared_with': ['1'],
})
#Update the annotation
annotation = Annotation.objects.all()[0]
string_data = {
'content_type': '9',
'object_id':'1',
'paragraph':'1',
'body':'This is the updated annotation',
'author':str(self.user.id),
'privacy':'3',
'privacy_override': '0',
'shared_with': ['1'],
}
json_data = json.dumps(string_data)
url = '/annotations/annotations/'+ str(annotation.id)+'/'
response = self.client.put(
url,
content_type='application/json',
data = json_data,
)
#print "Response"
#print response.content.decode()
self.assertEqual(Annotation.objects.all().count(), 1)
annotation = Annotation.objects.all()[0]
#print 'Annotation content_object'
#print annotation.content_object
self.assertEqual(annotation.body, 'This is the updated annotation')
self.assertEqual(annotation.paragraph, 1)
def test_DELETE_annotation(self):
self._create_annotation(content={
'content_type': '9',
'object_id':'1',
'paragraph':'1',
'body':'Dreaming is good, day dreaming, not so good.',
'author':str(self.user.id),
'privacy':'3',
'privacy_override': '0',
'shared_with': ['1'],
})
annotation = Annotation.objects.all()[0]
url = '/annotations/annotations/'+ str(annotation.id)+'/'
response = self.client.delete(url, content_type='application/json', data={})
print "Response"
print response.content.decode()
self.assertEqual(Annotation.objects.all().count(), 0)
Running the tests says all tests passed. I heaved a sigh of relief when these last tests passed. Do you know what it indicates? It indicates that we are done developing our backend code for this app for a while now. Now we can actually hook it up with the front end and put the complete app into perspective. That, we will do in the next and final tutorial of this series.