Author Archives: Alex Holmes

RxJS: An object lesson in terrible good software

We recently used RxJS on a large, complex asynchronous project integrated with a bit third-party distributed system. We now know more about it than, frankly, anyone would ever want to.

While we loved the approach, we hated the software itself. The reasons for this are a great lesson in how not to do software.

Our biggest issue by far with RxJS is that there are two actively developed, apparently stable versions at two different URLs. RxJS 4 is the top result from google for RxJS and lives at https://github.com/Reactive-Extensions/RxJS, and briefly mentions that there is an unstable version 5 at a different address. RxJS 5 lives at https://github.com/ReactiveX/rxjs and has a completely different API to version 4, completely different, far worse (“WIP”) documentation, doesn’t allude to its level of stability, and is written in typescript, so users will need to learn some typescript before they can understand the codebase.

Which version should new adopters use? I have absolutely no idea. Either way, when you google for advice and documentation, you can be fairly certain that the results you get will be for a version you’re not using.

RxJS goes to great lengths to swallow your errors. We’re pretty united here in thinking that it definitely should not. If an observable fires its “error” callback, it’s reasonable that the emitted error should be picked up by the nearest catch operator. Sadly though RxJS also wraps all of the functions that you pass to it with a try/catch block and any exception raised by those functions will also be shunted to the nearest try/catch block. Promises do this too, and many have complained bitterly about it already.

What this means in practice is that finding the source of an error is extremely difficult. RxJS tries to capture the original stack trace and make it available in the catch block but often fails, resulting in a failed observable and an “undefined” error. When my code breaks I’d like it to break where it broke, not in a completely different place. If I expect an error to occur I can catch it as I would anywhere else in the codebase and emit an observable error of the form that I’d expect in my catch block so that my catch blocks don’t all have to accommodate expected failure modes and any arbitrary exception. Days and days of development were lost to bisecting a long pile of dot-chained functions in order to isolate the one that raised the (usually stupidly trivial) error.

At the very least, it’d be nice to have the choice to use an unsafe observable instead. For this reason alone we are unlikely to use RxJS again.

We picked RxJS 5 as it’s been around for a long time now and seems to be being maintained by Netflix, which is reassuring.

The documentation could be way better. It is incomplete, with some methods not documented at all, partially documented as a mystery-meat web application that can’t be searched like any normal technical documentation. Code examples rarely use real-world use-cases so it’s tough to see the utility of many of the Observable methods. Most of the gotchas that caught out all of our developers weren’t alluded to at any point in any of the documentation (in the end, a youtube talk by the lead developer saved the day, containing the first decent explanation of the error handling mechanism that I’d seen or read). Worst of all, the only documentation that deals with solving actual problems with RxJS (higher-order observables) is in the form of videos on the paywalled egghead.io. I can’t imagine a more effective way to put off new adopters than requiring them to pay $200 just to appreciate how the library is commonly used (though, to be clear, I am a fan of egghead).

Summed up best by this thread, RxJS refuses to accept its heritage and admit that it’s a functional library. Within the javascript community there exists a huge functional programming subcommunity that has managed to put together a widely-adopted specification for writing functional javascript libraries that can interoperate with the rest of the available javascript functional libraries. RxJS chooses not to work to this specification  and a number of design decisions such as introspecting certain contained values and swallowing them drastically reduces the ease with which RxJS can be used in functional javascript codebases.

RxJS makes the same mistake lodash did a number of years ago, regularly opting for variadic arguments to its methods rather than taking arrays (the worst example is merge). Lodash did eventually learn its lesson, I hope RxJS does too.

Taming the Async Beast with FRP and RxJS

The Problem

We’ve recently been working on an in-browser vision mixer for the BBC (previous blog posts here, here, here, and here). Live vision mixing involves keeping track of a large number of BBC R&D logointerdependent data streams. Our application receives timing data for video tapes and live video streams via webrtc data channels and websocket connections and we’re sending video and audio authoring decisions over other websockets to the live rendering backend.

Many of the data streams we’re handling are interdependent; we don’t want to send an authoring decision to the renderer to cut to a video tape until the video tape is loaded and ready to play, so we need to wait until the video tape is ready to play before we send an authoring decision; if the authoring websocket has closed we’ll need to reconnect to it then retry sending that authoring decision.

Orchestrating interdependent asynchronous data streams is a fundamentally complex problem.

Promises are one popular solution for composing asynchronous operations and safely transforming the results, however they have a number of limitations. The primary issue is that they cannot be cancelled, so we need to handle teardown separately somehow. We could use the excellent fluture or Task Future libraries instead, both of which support cancellation (and are lazy and chainable and fantasy-land compliant), but futures and promises handle one single future value (or error value), not a stream of many values (or error value). The team working this project are fans of futures (less so of promises) and were aiming to write the majority of the codebase in a functional style using folktale and ramda (and react-redux) so wanted a functional, composable way to handle ongoing streams of data that could sit comfortably within the rest of the codebase.

A Solution

After some debate, we decided to use FRP (functional reactive programming) powered by the observable pattern. Having used RxJS (with redux-observable) for smaller projects in the past, we were confident that it could be an elegant solution to our problem. You can find out more about RxJS here and here but, in short, it’s a library that allows subscribers to listen to and transform the output of a data stream as per the observer pattern, and allows the observable (the thing subscribed to) to “complete” its stream when it runs out of data (or whatever), similar to an iterator from the iterator pattern. Observables also allow their subscribers to terminate them at any point, and typically observables will encapsulate teardown logic related to their data source – a websocket, long-poll, webrtc data channel, or similar.

RxJS implements the observer pattern in a functional way that allows developers to compose together observables, just as they’d compose functions or types. RxJS has its roots in functional reactive programming and leverages the power of monadic composition to chain together streams while also ensuring that teardown logic is preserved and handled as you’d expect.

Why FRP and Observables?

The elegance and power of observables is much more easily demonstrated than explained in a wordy paragraph. I’ll run through the basics and let your imagination think through the potential of it all.

A simple RxJS observable looks like this:

Observable.of(1, 2, 3)

It can be subscribed to as follows:

Observable.of(1, 2, 3).subscribe({
  next: val => console.log(`Next: ${val}`),
  error: err => console.error(err),
  complete: () => console.log('Completed!')
});

Which would emit the following to the console:

Next: 1
Next: 2
Next: 3
Completed!

We can also transform the data just as we’d transform values in an array:

Observable.of(1, 2, 3).map(x => x * 2).filter(x => x !== 4).subscribe(...)
2
6
Completed!

Observables can also be asynchronous:

Observable.interval(1000).subscribe(...)
0 [a second passes]
1 [a second passes]
2 [a second passes]
...

Observables can represent event streams:

Observable.fromEvent(window, 'mousemove').subscribe(...)
[Event Object]
[Event Object]
[Event Object]

Which can also be transformed:

Observable.fromEvent(window, 'mousemove')
  .map(ev => [ev.clientX, ev.clientY])
  .subscribe(...)
[211, 120]
[214, 128]
[218, 139]
...

We can cancel the subscriptions which will clean up the event listener:

const subscription = Observable.fromEvent(window, 'mousemove')
  .map(ev => [ev.clientX, ev.clientY])
  .subscribe(...)

subscription.unsubscribe();

Or we can unsubscribe in a dot-chained functional way:

Observable.of(1, 2, 3)
  .take(2)  // After receiving two values, complete the observable early
  .subscribe(...)
1
2
Completed!
Observable.fromEvent(window, 'mousemove')
  .map(ev => [ev.clientX, ev.clientY])
   // Stop emitting when the user clicks
  .takeUntil(Observable.fromEvent(window, 'click'))
  .subscribe(...)

Note that those last examples left no variables lying around. They are entirely self-contained bits of functionality that clean up after themselves.

Many common asynchronous stream use-cases are catered for natively, in such a way that the “operators” (the observable methods e.g. “throttle”, “map”, “delay”, “filter”) take care of all of the awkward state required to track emitted values over time.

Observable.fromEvent(window, 'mousemove')
  .map(...)
  .throttle(1000) // only allow one event through per second
  .subscribe(...);

… and that’s barely scratching the surface.

The Benefits

Many of the benefits of RxJS are the benefits of functional programming. The avoidance of state, the readability and testability of short, pure functions. By encapsulating the side-effects associated with your application in a generic, composable way, developers can maximise the reusability of the asynchronous logic in their codebase.

By seeing the application as a series of data transformations between the external application interfaces, we can describe those transformations by composing short, pure functions and lazily applying data to them as it is emitted in real-time.

Messy, temporary, imperative variables are replaced by functional closure to give observables access to previously emitted variables in a localised way that limits the amount of the application logic and state a developer must hold in their head at any given time.

Did It Work?

Sort of.  We spent a lot of our time in a state of low-level fury at RxJS, so much so that we’ve written up a long list of complaints, in another post.

There are some good bits though:

FRP and the observable pattern are both transformative approaches to writing complex asynchronous javascript code, producing fewer bugs and drastically improving the reusability of our codebase.

RxJS operators can encapsulate extremely complex asynchronous operations and elegantly describe dependencies in a terse, declarative way that leaves no state lying around.

In multiple standups throughout the project we’ve enthusiastically raved about how these operators have turned a fundamentally complex part of our implementation into a two line solution. Sure those two lines usually took a long time to craft and get right, but once working, it’s difficult to write many bugs in just two lines of code (when compared to the hundreds of lines of imperative code we’d otherwise need to write if we rolled our own).

That said, RxJS is a functional approach to writing code so developers should expect to incur a penalty if they’re new to the paradigm as they go from an imperative, object-oriented approach to system design to a functional, data-flow-driven approach instead. There is also a very steep learning curve required to feel the benefits of RxJS as developers familiarise themselves with the toolbox and the idiosyncrasies.

Would We Use It Again?

Despite the truly epic list of shortcomings, I would still recommend an FRP approach to complex async javascript projects. In future we’ll be trying out most.js to see if it solves the myriad of problems we found with RxJS. If it doesn’t, I’d consider implementing an improved Observable that keeps its hands off my errors.

It’s also worth mentioning that we used RxJS with react-redux to handle all redux side-effects. We used redux-observable to achieve this and it was terrific. We’ll undoubtedly be using redux-observable again.

 

A Quick Introduction to Backbone

Who Uses Backbone?

everyone

airbnb, newsblur, disqus, hulu, basecamp, stripe, irccloud, trello, …

Why?

http://backbonejs.org/#FAQ-why-backbone

It is not a framework

Backbone is an MVP (Model View Presenter) javascript library that, unlike Django, is extremely light in its use of conventions. Frameworks are commonly seen as fully-working applications that run your code, as opposed to libraries, where you import their code and run it yourself. Backbone falls solidly into the latter category and it’s only through the use of the Router class that it starts to take some control back. Also included are View, Model and Collection (of Models), and Events, all of which can be used as completely standalone components and often are used this way alongside other frameworks. This means that if you use backbone, you will have much more flexibility for creating something unusual and being master of your project’s destiny, but on the other hand you’ll be faced with writing a lot of the glue code yourself, as well as forming many of the conventions.

Dependencies

Backbone is built upon jQuery and underscore. While these two libraries have some overlap, they mostly perform separate functions; jQuery is a DOM manipulation tool that handles the abstractions of various browser incompatibilities (and even in the evergreen browser age offers a lot of benefits there), and underscore is primarily a functional programming tool, offering cross-browser support for map, reduce, and the like. Most data manipulation you do can be significantly streamlined using underscore and, in the process, you’ll likely produce more readable code. If you’re on a project that isn’t transpiling from ES6 with many of the functional tools built in, I enthusiastically recommend using underscore.

Underscore has one other superb feature: templates.

// Using raw text
var titleTemplate = _.template('<h1>Welcome, <%- fullName %></h1>');
// or if you have a <script type="text/template">
var titleTemplate = _.template($('#titleTemplate'));
// or if you're using requirejs
var titleTemplate = require('tpl!templates/title');

var renderedHtml = titleTemplate({title: 'Martin Sheen'});
$('#main').html(renderedHtml);

Regardless of how you feel about the syntax (which can be changed to mustache-style), having lightweight templates available that support escaping is a huge win and, on its own, enough reason to use underscore.

Backbone.Events

Probably the most reusable class in Backbone’s toolbox is Events. This can be mixed in to any existing class as follows:

// Define the constructor
var MyClass = function(){};

// Add some functionality to the constructor's prototype
_.extend(MyClass.prototype, {
  someMethod: function() {
    var somethingUseful = doSomeThing();
    // trigger an event named 'someEventName'
    this.trigger('someEventName', somethingUseful);
  }
});

// Mix-in the events functionality
_.extend(MyClass.prototype, Backbone.Events);

And suddenly, your class has grown an event bus.

var thing = new MyClass();
thing.on('someEventName', function(somethingUseful) {
  alert('IT IS DONE' + somethingUseful);
});
thing.someMethod();

Things we don’t know about yet can now listen to this class for events and run callbacks where required.

By default, the Model, Collection, Router, and View classes all have the Events functionality mixed in. This means that in a view (in initialize or render) you can do:

this.listenTo(this.model, 'change', this.render);

There’s a list of all events triggered by these components in the docs. When the listener also has Events mixed in, it can use .listenTo which, unlike .on, sets the value of this in the callback to the object that is listening rather than the object that fired the event.

Backbone.View

Backbone.View is probably the most useful, reusable class in the Backbone toolbox. It is almost all convention and does nothing beyond what most people would come up with after pulling together something of their own.

Fundamentally, every view binds to a DOM element and listens to events from that DOM element and any of its descendants. This means that functionality relating to your application’s UI can be associated with a particular part of the DOM.

Views have a nice declarative format, as follows (extend is an short, optional shim for simplifying inheritance):

var ProfileView = Backbone.View.extend({

  // The first element matched by the selector becomes view.el
  // view.$el is a shorthand for $(view.el)
  // view.$('.selector') is shorthand for $(view.el).find('.selector');
  el: '.profile',

  // Pure convention, not required
  template: profileTemplate,

  // When events on left occur, methods on right are called
  events: {
    'click .edit': 'editSection',
    'click .profile': 'zoomProfile'
  },

  // Custom initialize, doesn't need to call super
  initialize: function(options) {
    this.user = options.user;
  },

  // Your custom methods
  showInputForSection: function(section) {
    ...
  },

  editSection: function(ev) {
    // because this is bound to the view, jQuery's this is made
    // available as 'currentTarget'
    var section = $(ev.currentTarget).attr('data-section');
    this.showInputForSection(section);
  },

  zoomProfile: function() {
    $(ev.currentTarget).toggleClass('zoomed');
  },

  // Every view has a render method that should return the view
  render: function() {
    var rendered = this.template({user: this.user});
    this.$el.html(rendered);
    return this;
  }

});

Finally, to use this view:

// You can also pass model, collection, el, id, className, tagName, attributes and events to override the declarative defaults
var view = new ProfileView({ user: someUserObject });
view.render();  // Stuff appears in .profile !

// Once you've finished with the view
view.undelegateEvents();
view.remove();
// NB. Why doesn't remove call undelegateEvents? NFI m8. Hate that it doesn't.

The next step is to nest views. You could have a view that renders a list but for each of the list items, it instantiates a new view to render the list item and listen to events for that item.

render: {
  // Build a basic template "<ul></ul>"
  this.$el.html(this.template());

  _.each(this.collection.models, function(model) {
    // Instantiate a new "li" element as a view for each model
    var itemView = new ModelView({ tagName: 'li' });
    // Render the view
    itemView.render();
    // The jquery-wrapped li element is now available at itemView.$el
    this.$('ul')  // find the ul tag in the parent view
      .append(itemView.$el);  // append to it this li tag
  });
}

How you choose to break up your page is completely up to you and your application.

Backbone.Model and Backbone.Collection

Backbone’s Model and Collection classes are designed for very standard REST endpoints. It can be painful to coerce them into supporting anything else, though it is achievable.

Assuming you have an HTTP endpoint /items, which can:

  • have a list of items GET’d from it
  • have a single item GET’d from /items/ID

And if you’re going to be persisting any changes back to the database:

  • have new items POSTed to it
  • have existing items, at /items/ID PUT, PATCHed, and DELETEd

Then you’re all set to use all the functionality in Backbone.Model and Backbone.Collection.

var Item = Backbone.Model.extend({
  someMethod: function() {
    // perform some calculation on the data
  }
});

var Items = Backbone.Collection.extend({
  model: Item,
});

Nothing is required to define a Model subclass, although you can specify the id attribute name and various other configurables, as well as configuring how the data is pre-processed when it is fetched. Collection subclasses must have a model attribute specifying which Model class is used to instantiate new models from the fetched data.

The huge win with Model and Collection is in their shared sync functionality – persisting their state to the server and letting you know what has changed. It’s also nice being able to attach methods to the model/collection for performing calculations on the data.

Let’s instantiate a collection and fetch it.

var items = new Items();
// Fetch returns a jQuery promise
items.fetch().done(function() {
  items.models // a list of Item objects
  items.get(4) // get an Item object by its ID
});

Easy. Now let’s create a new model instance and persist it to the database:

var newItem = new Item({some: 'data'});
items.add(newItem);
newItem
  .save()
  .done(function() {
    messages.success('Item saved');
  })
  .fail(function() {
    messages.danger('Item could not be saved lol');
  });

Or to fetch models independently:

var item = new Item({id: 5});
item
  .fetch()
  .done(function() {
    someView.render();
  });

// or use Events!
this.listenTo(item, 'change', this.render);
item.fetch();

And to get/set attributes on a model:

var attr = item.get('attributeName');
item.set('attributeName', newValue);  // triggers 'change', and 'change:attributeName' events

Backbone.Router and Backbone.History

You’ll have realised already that all the above would work perfectly fine in a typical, SEO-friendly, no-node-required, django views ftw multi-page post-back application. But what about when you want a single page application? For that, backbone offers Router and History.

Router classes map urls to callbacks that typically instantiate View and/or Model classes.

History does little more than detect changes to the page URL (whether a change to the #!page/fragment/ or via HTML 5 pushState) and, upon receiving that event, orchestrates your application’s router classes accordingly based on the urls it detects. It is Backbone.History that takes backbone from the library category to the framework category.

History and Router are typically used together, so here’s some example usage:

var ApplicationRouter = Router.extend({

  routes: {
    'profile': 'profile',
    'items': 'items',
    'item/:id': 'item',
  },

  profile: function() {
    var profile = new ProfileView();
    profile.render();
  },

  items: function() {
    var items = new Items();
    var view = new ItemsView({items: items});
    items.fetch().done(function() {
      items.render();
    });
  },

  item: function(id) {
    var item = new Item({id: id})
    var view = new ItemView({item: item});
    item.fetch().done(function() {
      view.render();
    });
  }

});

Note that above, I’m running the fetch from the router. You could instead have your view render a loading screen and fetch the collection/model internally. Or, in the initialize of the view, it could do this.listenTo(item, 'sync', this.render), in which case your routers need only instantiate the model, instantiate the view and pass the model, then fetch the model. Backbone leaves it all to you!

Finally, let’s use Backbone.History to bring the whole thing to life:

$(function(){
  // Routers register themselves
  new ApplicationRouter();
  if (someCustomSwitch) {
    new CustomSiteRouter();
  }

  // History is already instantiated at Backbone.history
  Backbone.history.start({pushState: true});
});

Now the most common way to use a router is to listen for a click event on a link, intercept the event, and rather than letting the browser load the new page, preventDefault and instead run Backbone.history.navigate('/url/from/link', {trigger: true}); This will run the method associated with the passed route, then update the url using pushState. This is the key: each router method should be idempotent, building as much of the page as required from nothing. Sometimes this method will be called with a different page built, sometimes not. Calling history.navigate will also create a new history entry in the browser’s history (though you can avoid this happening by passing {trigger: true, replace: true}.

If a user clicks back/forward in the browser, the url will change and Backbone.history will again look up the new url and execute the method associated with that url. If none can be found the event is propagated to the browser and the browser performs a typical page change. In this case, you should be sure in your router method to call .remove and .undelegateEvents on any instantiated views that you no longer need or else callbacks for these could still fire. YES, this is incredibly involved.

Finally, you’ll sometimes be in the position where you’ve updated one small part of a page, some sub-view perhaps but you want this change to be reflected in the URL. You don’t necessarily want to trigger a router method because all the work has been done but you do have a router method that could reconstruct the new state of the page were a user to load that page from a full refresh. In this case, you can call Backbone.history.navigate('/new/path'); and it’ll add a new history entry without triggering the related method.

Conclusion

Backbone is unopinionated. It provides an enormous amount of glue code in a way that is very useful and immediately puts you in a much better position than if you were just using jquery. That said, loads more glue code must be written for every application so it could do much, much more. On one hand this means you gain a tremendous amount of flexibility which is extremely useful given the esoteric nature of the applications we build, and it also gives you the power to make your applications incredibly fast, since you aren’t running lots of the general-purpose glue code from much more involved frameworks. It also gives you the power to inadvertently hang yourself.

If you’re looking for something easy to pick up and significantly more powerful than jQuery, but less of a painful risk than the horrible messy world of proper javascript frameworks (see also: the hundreds of angular “regret” blog posts), Backbone is a great place to start.

About usIsotoma is a bespoke software development company based in York and London specialising in web apps, mobile apps and product design. If you’d like to know more you can review our work or get in touch.

Reviewing Django REST Framework

Recently, we used Django REST Framework to build the backend for an API-first web application. Here I’ll attempt to explain why we chose REST Framework and how successfully it helped us build our software.

Why Use Django REST Framework?

RFC-compliant HTTP Response Codes

Clients (javascript and rich desktop/mobile/tablet applications) will more than likely expect your REST service endpoint to return status codes as specified in the HTTP/1.1 spec. Returning a 200 response containing {‘status’: ‘error’} goes against the principles of HTTP and you’ll find that HTTP-compliant javascript libraries will get their knickers in a twist. In our backend code, we ideally want to raise native exceptions and return native objects; status codes and content should be inferred and serialised as required.

If authentication fails, REST Framework serves a 401 response. Raise a PermissionDenied and you automatically get a 403 response. Raise a ValidationError when examining the submitted data and you get a 400 response. POST successfully and get a 201, PATCH and get a 200. And so on.

Methods

You could PATCH an existing user profile with just the field that was changed in your UI, DELETE a comment, PUT a new shopping basket, and so on. HTTP methods exist so that you don’t have to encode the nature of your request within the body of your request. REST Framework has support for these methods natively in its base ViewSet class which is used to build each of your endpoints; verbs are mapped to methods on your view class which, by default, are implemented to do everything you’d expect (create, update, delete).

Accepts

The base ViewSet class looks for the Accepts header and encodes the response accordingly. You need only specify which formats you wish to support in your settings.py.

Serializers are not Forms

Django Forms do not provide a sufficient abstraction to handle object PATCHing (only PUT) and cannot encode more complex, nested data structures. The latter limitation lies with HTTP, not with Django Forms; HTTP forms cannot natively encode nested data structures (both application/x-www-form-urlencoded and multipart/form-data rely on flat key-value formats). Therefore, if you want to declaratively define a schema for the data submitted by your users, you’ll find life a lot easier if you discard Django Forms and use REST Framework’s Serializer class instead.

If the consumers of your API wish to use PATCH rather than PUT, and chances are they will, you’ll need to account for that in your validation. The REST Framework ModelSerializer class adds fields that map automatically to Model Field types, in much the same way that Django’s ModelForm does. Serializers also allow nesting of other Serializers for representing fields from related resources, providing an alternative to referencing them with a unique identifier or hyperlink.

More OPTIONS

Should you choose to go beyond an AJAX-enabled site and implement a fully-documented, public API then best practice and an RFC or two suggest that you make your API discoverable by allowing OPTIONS requests. REST Framework allows an OPTIONS request to be made on every endpoint, for which it examines request.user and returns the HTTP methods available to that user, and the schema required for making requests with each one.

OAuth2

Support for OAuth 1 and 2 is available out of the box and OAuth permissions, should you choose to use them, can be configured as a permissions backend.

Browsable

REST framework provides a browsable HTTP interface that presents your API as a series of forms that you can submit to. We found it incredibly useful for development but found it a bit too rough around the edges to offer as an aid for third parties wishing to explore the API. We therefore used the following snippet in our settings.py file to make the browsable API available only when DEBUG is set to True:

if DEBUG:
    REST_FRAMEWORK['DEFAULT_RENDERER_CLASSES'].append(
        'rest_framework.renderers.BrowsableAPIRenderer'
    )

Testing

REST Framework gives you an APITestCase class which comes with a modified test client. You give this client a dictionary and encoding and it will serialise the request and deserialise the response. You only ever deal in python dictionaries and your tests will never need to contain a single instance of json.loads.

Documentation

The documentation is of a high quality. By copying the Django project’s three-pronged approach to documentation – tutorial, topics, and API structure, Django buffs will find it familiar and easy to parse. The tutorial quickly gives readers the feeling of accomplishment, the high-level topic-driven core of the documentation allows readers to quickly get a solid understanding of how the framework should be used, and method-by-method API documentation is very detailed, frequently offering examples of how to override existing functionality.

Project Status

At the time of writing the project remains under active development. The roadmap is fairly clear and the chap in charge has a solid grasp of the state of affairs. Test coverage is good. There’s promising evidence in the issue history that creators of useful but non-essential components are encouraged to publish their work as new, separate projects, which are then linked to from the REST Framework documentation.

Criticisms

Permissions

We found that writing permissions was messy and we had to work hard to avoid breaking DRY. An example is required. Let’s define a ViewSet representing both a resource collection and any document from that collection:

views.py:

class JobViewSet(ViewSet):
    """
    Handles both URLS:
    /jobs
    /jobs/(?P<id>d+)/$
    """
    serializer_class = JobSerializer
    permission_classes = (IsAuthenticated, JobPermission)

    def get_queryset(self):
        if self.request.user.is_superuser:
            return Job.objects.all()

        return Job.objects.filter(
            Q(applications__user=request.user) |
            Q(reviewers__user=request.user)
        )

If the Job collection is requested, the queryset from get_queryset() will be run through the serializer_class and returned as an HTTPResponse with the requested encoding.

If a Job item is requested and it is in the queryset from get_queryset(), it is run through the serializer_class and served. If a Job item is requested and is not in the queryset, the view returns a 404 status code. But we want a 403.

So if we define that JobPermission class, we can fail the object permission test, resulting in a 403 status code:

permissions.py:

class JobPermission(Permission):
    def get_object_permission(self, request, view, obj):
    if obj in Job.objects.filter(
        Q(applications__user=request.user) |
        Q(reviewers__user=request.user)):
        return True
    return False

Not only have we duplicated the logic from the view method get_queryset (we could admittedly reuse view.get_queryset() but the method and underlying query would still be executed twice), if we don’t then the client is sent a completely misleading response code.

The neatest way to solve this issue seems to be to use the DjangoObjectPermissionsFilter together with the django-guardian package. Not only will this allow you to define object permissions independently of your views, it’ll also allow you filter querysets using the same logic. Disclaimer: I’ve not tried this solution, so it might be a terrible thing to do.

Nested Resources

REST Framework is not built to support nested resources of the form /baskets/15/items. It requires that you keep your API flat, of the form /baskets/15 and /items/?basket=15.

We did eventually choose to implement some parts of our API using nested URLs however it was hard work and we had to alter public method signatures and the data types of public attributes within our subclasses. We required entirely highly modified Router, Serializer, and ViewSet classes. It is worth noting that REST Framework deserves praise for making each of these components so pluggable.

Very specifically, the biggest issue preventing us pushing our nested resources components upstream was REST Framework’s decision to make lookup_field on the HyperlinkedIdentityField and HyperlinkedRelatedField a single string value (e.g. “baskets”). To support any number of parent collections, we had to create a NestedHyperlinkedIdentityField with a new lookup_fields list attribute, e.g. ["baskets", "items"].

Conclusions

REST Framework is great. It has flaws but continues to mature as an increasingly popular open source project. I’d whole-heartedly recommend that you use it for creating full, public APIs, and also for creating a handful of endpoints for the bits of your site that need to be AJAX-enabled. It’s as lightweight as you need it to be and most of what it does, it does extremely well.

About us: Isotoma is a bespoke software development company based in York and London specialising in web apps, mobile apps and product design. If you’d like to know more you can review our work or get in touch.

API First

Recently, we were faced with the task of writing an API-first web application in order to support future mobile platform development. Here’s a summary of the project from the point of view of one of the developers.

Agile API

For the first couple of iterations, we had problems demonstrating the project progress to the customer at the end of iteration meetings. The customer on this project was extremely understanding and reasonably tech-savvy but despite that, he remained uninterested in the progress of the API and became quite concerned by the lack of UI progress. Although we were busy writing and testing the API code sitting just beneath the surface, letting the customer watch our test suite run would have achieved nothing. It was frustrating to find that, when there was nothing for the customer to click around on, we couldn’t get the level of engagement and collaboration we would typically achieve. In the end, we had to rely on the wireframes from the design process which the customer had signed off on to inform our technical decisions and, to allay the customer’s fears, we ended up throwing together some user interfaces which lacked any functionality purely to give the illusion of progress.

On the plus side, once we had written enough of our API to know that it was fit for purpose, development on the front-end began and progressed very rapidly; most of the back-end validation was already in place, end-points were well defined, and the comprehensive integration tests we’d written served as a decent how-to-use manual for our API.

Extra Work

Developing the application API-first took more work and more lines of code than it would have required if implemented as a typical post-back website.

Each interface had to be judged by its general usefulness rather than by its suitability for one particular bit of functionality alluded to by our wireframes or specification. Any view that called upon a complex or esoteric query had to instead be implemented using querystring filters or a peculiar non-generic endpoint.

In a typical postback project with private, application-specific endpoints, we’d be able to pick and choose the HTTP verbs relevant to the template we’re implementing however our generic API required considerably more thought. For each resource and collection, we had to carefully think about the permissions structure for each HTTP method, and the various circumstances in which the endpoint might be used.

We wrote around 4000 lines of integration test code just to pin down the huge combination of HTTP methods and user permissions however I sincerely doubt that all of those combinations are required by the web application. Had we not put in the extra effort however, we’d have risked making our API too restrictive to future potential consumers.

In terms of future maintainability, I’d say that each new generic endpoint will require a comparable amount of otherwise-unnecessary consideration and testing of permissions and HTTP methods.

Decoupling

Having such an explicitly documented split between the front and back end was actually very beneficial. The front end and back-end were developed and tested based on the API we’d designed and documented. For over a month, I worked solely on the back-end and my colleague worked solely on the front and we found this division of labour was an incredibly efficient way to work. By adhering to the HTTP 1.1 specification, using the full range of available HTTP verbs and response codes, and to our endpoint specification, we required far less interpersonal coordination than would typically be the case.

Beyond CRUD

The two major issues we found with generic CRUD endpoints were (1) when we needed to perform a complex data query, and (2) update multiple resources in a single transaction.

To a certain extent we managed to solve the first problem using querystrings, with keys representing fields on the resource. For all other cases, and also to solve the second problem, we used an underused yet still perfectly valid REST resource archetype: the controller, used to model a procedural concept.

We used controller endpoints on a number of occasions to accommodate things like /invitations/3/accept (“accept” represents the controller) which would update the invitation instance and other related user instances, as well as sending email notifications.

Where we needed to support searching, we added procedures to collections, of the form /applicants/search, to which we returned members of the collection (in this example “applicants”) which passed a case-insensitive containment test based on the given key.

Conclusion

API-first required extra implementation effort and a carefully-considered design. We found it was far easier and more efficient to implement as a generic, decoupled back-end component than in the typical creation process (model -> unit test -> url -> view -> template -> integration test), with the front-end being created completely independently.

In the end, we wrote more lines of code and far more lines of integration tests. The need to stringently adhere to the HTTP specification for our public API really drove home the benefits to using methods and status codes.

In case you’re curious, we used Marionette to build the front-end, and Django REST Framework to build the back end.

About us: Isotoma is a bespoke software development company based in York and London specialising in web apps, mobile apps and product design. If you’d like to know more you can review our work or get in touch.