SOMA in use during the 2017 Edinburgh Festival

Live video mixing with the BBC: Lessons learned

In this post I am going to reflect on some of the more interesting aspects of this project and the lessons they might provide for other projects.BBC R&D logo

This post is one of a series talking about our work on the SOMA video mixing application for the BBC. The previous posts in the series are:

  1. Building a live television video mixing application for the browser
  2. The challenges of mixing live video streams over IP networks
  3. Rapid user research on an Agile project
  4. Compositing and mixing video in the browser
  5. Taming the Async Beast with FRP and RxJS
  6. RxJS: An object lesson in terrible good software
  7. Video: The future of TV broadcasting
  8. Integrating UX with agile development

In my view there are three broad areas where this project has some interesting lessons.

Novel domains

First is the novel domain.

This isn’t unfamiliar – we often work in novel domains that we have little to no knowledge of. It is the nature of technical agency in fact – while we have some domains that we’ve worked in for many years such as healthcare and education there are always novel businesses with entirely new subjects to wrap our heads around.  (To give you some idea, a few recent examples include store-and-forward television broadcasting, horse racing odds, medical curricula, epilepsy diagnosis, clustering automation and datacentre hardware provisioning.)

Over the years this has been the thing that I have most enjoyed out of every aspect of our work. Plunging into an entirely new subject with a short amount of time to understand it and make a useful contribution is exhilarating.

Although it might sound a bit insane to throw a team who know nothing about a domain at a problem, what we’re very good at is designing and building products. As long as our customers can provide the domain expertise, we can bring the product build. It is easier for us to learn the problem domain than it is for a domain expert to learn how to build great products.

The greatest challenge with a new domain is the assumptions. We all have these in our work – the things we think are so well understood that we don’t even mention them. These are a terrible trap for software developers, because we can spend weeks building completely the wrong thing with no idea that we’re doing so.

We were very lucky in this respect to be working with a technical organisation within the BBC: Research & Development. They were aware of this risk and did a very good job of arranging our briefing, which included a visit to a vision mixing gallery. This is the kind of exercise that delivers a huge amount in tacit understanding, and allows us to ask the really stupid questions in the right setting.

I think of the core problem as a “Rumsfeld“. Although he got a lot of criticism for these comments I think they’re bizarrely insightful. There really are unknown unknowns, and what the hell do you do about them? You can often sense that they exist, but how do you turn them into known unknowns?

For many of these issues the challenge is not the answer, which is obvious once it has been found, but facilitating the conversation to produce the answer. It can be a long and frustrating process, but critical to success.

I’d encourage everyone to try and get the software team into the existing environment of the target stakeholder groups to try and understand at a fundamental level what they need.

The Iron Triangle

The timescale for this project was extraordinarily difficult – nine weeks from a standing start. In addition much of the scope was quite fixed – we were largely building core functionality that, if missing, would have rendered the application useless. In addition we wanted to achieve the level of finish for the UX that we generally deliver.

This was extremely ambitious, and in retrospect we bit off more than we could reasonably chew.

Time is the greatest enemy of software projects because of the challenges in estimation. For reasons covered in a different blog post, estimation for software projects is somewhere between an ineffable art reserved only for the angels, and completely impossible.

Triangle with sides labeled Quality, Scope and TimeWhen estimates are impossible, time becomes an even greater challenge. One of the truisms of our industry is the “Iron Triangle” of time, scope and quality. Like a good chinese buffet, you can only choose two. If you want a fixed time and scope, it is quality that will suffer.

Building good software takes thought and planning. Also, the first version of a component is rarely the best – it takes time to assemble, then consider it, and then perhaps shape it into something near its final form.

Quality is, itself, an aggregate quality. Haste lowers the standards for each part and so, by a process of multiplication, lowers far more the overall quality of a product. The only way to achieve a very high quality for the finished product is for every single part to be of similarly high quality. This is generally our goal.

However. Whisper it. It is possible to “manage” quality, if you understand your process and know the goal. Different kinds of testing can provide different levels of certainty of code quality. Manual testing, when done exhaustively, can substitute in some cases for perfection in code.

We therefore managed our quality, and I think actually did well here.

Asynchronous integration components had to be of absolute perfection because any bugs would result in general lack of stability which would be impossible to trace. The only way to build these is carefully, with a design and the only way to test these is exhaustively with unit and integration tests.

On the other hand, there were a lot of aspects of the UI where it was crucial that they performed and looked excellent, but the code could be rougher around the edges, and could just be hacked out. This was my area of the application, and my goal was to deliver features as fast as possible with just acceptable quality. Some of the code was quite embarrassing but we got the project over the line in the time, with the scope, and it all worked. This was sufficient for those areas.

Experimental technologies

I often talk about our approach using the concept of an innovation curve, and our position on it (I think I stole the idea from Ian Jindal – thanks Ian!).

If you can imagine a curve like this one, where the X axis is “how innovative your technologies are”, the Y axis is “pain”.

In practical terms this can be translated into “how likely I am to find the answer to my problems on Stack Overflow“.

At the very left, everything has been seen and done before, so there is no challenge from novelty – but you are almost certainly not making the most of available technologies.

At the far right, you are hand crafting your software from individual photons and you have to conduct high-energy physics experiments to debug your code. You are able to mould the entire universe to your whim – but it takes forever and costs a fortune.

There is no correct place to sit on this curve – where you sit is a strategic (and emotional) decision that depends on the forces at play in your particular situation.

Isotoma endeavours to be somewhere on the shoulder of the curve. The software we build generally needs to last 5+ years, so we can’t pick flash-in-the-pan technologies that will be gone in 18 months. But similarly we need to be relatively recent so it doesn’t become obsolete. This is sometimes called “leading edge”. Almost bleeding edge, but not so close you get cut. With careful choice of tools it is possible to maintain a position like this successfully.

This BBC project was off to the right of this curve, far closer to the bleeding edge than we’d normally choose, and we definitely suffered.

Some of the technologies we had to use had some serious issues:

  1. To use IPStudio, a properly cutting edge product developed internally by BBC R&D, we routinely had to read the C++ source code of the product to find answers to integration questions.
  2. We needed dozens of coordinated asynchronous streams running, for which we used RxJS. This was interesting enough to justify two posts on this blog on its own.
  3. WebRTC, which was the required delivery mechanism for the video, is absolutely not ready for this use case. The specification is unclear, browser implementation is incomplete and it is fundamentally unsuited at this time to synchronised video delivery.
  4. The video compositing technologies in browsers actually works quite well, but was entirely new to us and it took considerable time to gain sufficient expertise to do a good job. Also browser implementations still have surprising sharp edges (only 16 WebGL contexts are allowed! Why 16? I dunno.)

Any of these one issues could have sunk our project, so I am very proud we shipped good software, with all four issues.

Lessons learned? Task allocation is the key to this one I think.

One person, Alex, devoted his time to the IPStudio and WebRTC work for pretty much the entire project, and Ricey concentrated on video mixing.

Rather than try and skill up several people, concentrate the learning in a single brain. Although this is generally a terrible idea (because then you have a hard dependency on a single individual for a particular part of the codebase), in this case it was the only way through, and it worked.

Also, don’t believe any documentation, or in fact anything written in any human languages. When working on the bleeding edge you must “Use The Source, Luke”. Go to the source code and get your head around it. Everything else lies.

Summary

I am proud, justifiably I think, that we delivered this project successfully. It was used at the Edinburgh festival and actual real live television was mixed using our product, given all the constraints above.

The lessons?

  1. Spend the time and effort to make sure your entire team understand the tacit requirements of the problem domain and the stakeholders.
  2. Have an approach to managing appropriate quality that delivers the scope and timescale, if these are heavily constrained.
  3. Understand your position on the innovation curve and choose a strategic approach to managing this.

The banner image at the top of the article, taken by Chris Northwood, shows SOMA in use during the 2017 Edinburgh Festival.

Integrating UX with agile development

Incorporating user centred design practices within Agile product development can be a major challenge. Most of us in the user experience field are more familiar with the waterfall “big design up front” methodology. Project managers and developers are also likely to be more comfortable with a discreet UX design phase that is completed before development commences. But this approach tends to be inefficient, slower and more expensive. How does the role of the UX designer change within Agile product development, with its focus on transparency and rapid iteration?

BBC R&D logoWhile at Isotoma we’ve always followed our own flavour of Agile product development, UX is still mostly front-loaded in a “discovery” phase, as at most agencies. Our recent vision mixer project for BBC Research & Development, however, required a more integrated approach. The project had a very tight timeframe, requiring overlapping UX and development, with weekly show & tells.

From a UX perspective, it was a positive experience and I’m happy with the result. This post lists some of the techniques and approaches that I think helped integrate UX with Agile. Of course, every project and organisation is different, so there is definitely no one-size-fits-all approach, but hopefully there is something here you can use in your work. Continue reading

Timesheets: some observations on observation

Just as a throwaway in my post on understanding your team’s progress I said something like “everyone hates timesheets”. And it’s true, they do. They’re onerous, boring and they’re usually seen as invasive, “big brother”-esque, make-work. But, as I also said in that post, good quality time recording is vital to understanding what’s going on within your teams.

Feeling the need

We first started looking at timesheet systems nine or ten years ago when it was becoming abundantly clear that we weren’t making the progress we were expecting on certain projects, but we didn’t know why.

The teams were skilled in the tools they were using, they were diligent, they’d done similar work before, but they just weren’t hitting the velocities that we had come to expect. On top of that, the teams themselves thought they were making good progress. And every which way we approached the problem we were missing the information needed to get to the bottom of the mismatch between expectation and reality.

At that point in the company’s life timesheets were anathema to us; we felt very strongly they indicated a lack of trust, and in a company built entirely on the principles behind the Agile Manifesto… Well… You can see our problem.

Build projects around motivated individuals.
Give them the environment and support they need,
and trust them to get the job done.

But however we cut it we really needed to understand what people were actually doing with their day. We trusted that if people thought they were making good progress then they were, but we definitely knew that we weren’t making the same kind of progress that we had been a year ago on the same types of project. And back then we were often on fixed price projects and billing by the day, so when projects started to overrun our financial performance started to dip and the quality of our code went the same way (for all the reasons I outlined in that previous post).

So we hit on Harvest (at the time one of the poster children of the burgeoning Rails SaaS community) and asked everyone to fill in their sheets for a couple of months so we could generate some data.

We had an all hands meeting, we explained exactly why we were doing it, and we asked, cajoled and bullied people into using it so that at least we had something to work on and perhaps uncover the problems we were hitting.

And of course we found it quickly enough, because accurate timesheets filled in honestly expose exactly what’s going on. By our nature we are both helpful and curious – that’s how we ended up doing what we’re doing. But helpful and curious is easily distracted; a colleague asking for help, an old customer with a quick question, a project manager from another project with an urgent request, the account management team asking “can you just…” And all of this added up. In the worst cases some people were only spending four hours a day on the project they were allocated to; the rest of their time spent helping colleagues and old customers… However, how you cope with these things is probably the subject of another post.
My point here is that once we had that data we realised how valuable it was and knew that we couldn’t go without it again. Our key takeaway was that timesheets are a key part of a company’s introspection and without good data you don’t know the problem you’re actually trying to solve. And so we had to make timesheets part of our everyday processes.

Loving the alien

Like I said; people hate timesheets. They’re invasive. They’re time consuming. They feel like you’re being watched, judged. They imply no trust. They’re alien to an agile environment. And the data they produce is a key part of someone else’s reporting, too. So how do you make sure they’re filled in accurately and honestly? And not just in month one, when you first introduce them, but in month fifty seven when your business relies on them and you may not be watching quite so closely.

We’ve found the following works for us:

  • Make it crystal clear what they’re for, and what they’re not
  • Make it explicit that timesheets are for tracking the performance of estimates and ensuring that progress can be reported accurately
  • It’s not about how much you do, but how much got done
  • Tie them together with things like iDoneThis, so that people can give context to their timesheets in an informal unstructured manner
  • Make sure that everyone who uses the data throughout the management chain is incentivised to treat it honestly – this means your project managers mustn’t feel the need to manipulate it or worse manipulate how it’s entered (we’ve seen this more than once in other organisations)

And Dan, one of our project managers, sends round a gentle chivvying email each evening (filled with the day’s fun facts, of course) to make sure that people actually fill them in.

[Photo by Sabri Tuzcu on Unsplash]

External agencies vs. in-house teams

As you’ll already know because you’re windswept and interesting; we record a semi regular podcast where we look into an aspect of life in a technical agency that we think will interest the outside world. We’ve just finished recording the latest episode about internal versus external teams and honestly I think it’s one of the most interesting chats we’ve had.

Joining us on the podcast are Andy Rogers from Rokker and Dan Graetzer from Carousel Group. Both Andy and Dan have tons of experience both commissioning work from internal teams and navigating the selection of external agencies. They were able to speak with clarity about the challenges that each task can bring.

One of the interesting things for me was getting a glimpse ‘over the fence’ into some of the thought processes and pressures that lead people to keep work internal – something that I’ve really only been able to guess at in the past.

Here’s a quick summary of things we speak about.

Agencies developing symbiotic/parasitic relationships with larger clients.

This tendency of larger agencies to act almost as though they are internal teams is becoming more and more common. There are upsides and downsides to this, obviously, in that while bodies like Deloitte et al can mobilise 200-strong dev teams, they also make it more and more likely that their customers will have to keep going back to them in future. (We discuss this subject mostly in terms of how Isotoma are not a larger agency!)

Good agencies are expensive but not as expensive as bad recruitment

The cost of hiring an agency for a given software project is likely to cost around the same as the annual salary of a developer and/or development team. Given this, it can seem galling for potential customers that they’re spending the right amount of money in the wrong place. We discuss how a good agency can help mitigate both the opportunity cost and assume all the tricky recruitment risk in the relationship. (Aren’t we nice?)

Continuous delivery shouldn’t necessarily mean continuous agency billing

One of the goals of any software project should be to build and develop the skills to support it in-house. If you’ve had a key piece of software in production for 18 months and you’re still relying on a third party to administer, fix or deploy it then you might have a problem.

Asking an agency to do something is the easy bit

Commissioning work with third party agencies is one step in a multi-step journey. This journey needs to include understanding how you’re defining your requirements, how you plan to receive it when it’s done and how you’re going to give the project good governance when it’s in flight.

Also there is a good deal of talk about werewolves

We’re not mega sure why.

Hopefully you’ll find it as interesting as we did. You can listen to the podcast and subscribe!

DotYork Event Announcement

Isotoma working in partnership with DotYork

We’re excited to announce that DotYork has joined forces with Isotoma, the York based software development agency. Isotoma have been long term supporters of the event, attending and sponsoring every conference so far and even speaking at DotYork 2016, so when they offered their help it felt like a natural fit.

We’ve always thought DotYork was a great way to highlight our beautiful city and York’s rapidly growing digital community, and during conversations with Rick we realised that we had loads of ideas for future events and could offer Dot York the support it needed. We’re really excited about DotYork 2017 and are looking forward to future events in 2018 and beyond.
Andy Theyers, director and founder, Isotoma

Working with Isotoma on DotYork 2017 has already sparked some new ideas and with a bigger team we’re able to look at expanding the event, including workshops and an evening event this year, with who knows what to come in 2018
Rick Chadwick, DotYork

Reposted from the original DotYork blog

 

The future of TV broadcasting

Earlier this year we started working on an exciting project with BBC Research & Development on their long term programme developing IP Studio – the next generation IP-network based broadcast television platform. The BBC are developing new industry-wide standards working with manufacturers which will hopefully be adopted worldwide.

This new technology dramatically changes the equipment needed for live TV broadcasting. Instead of large vans stocked with pricey equipment and a team of people; shows can be recorded, edited and streamed live by a single person with a 4k camera, a laptop and internet access.

The freedom of being able to stream and edit live TV within the browser will open up endless possibilities for the entertainment and media sector.

You can see more Isotoma videos on Vimeo.

A blog post about estimating

First of all, a provocative but sweeping statement about the subject to kick us off: If your agency won’t talk to you about how they estimate projects then they’re either liars or fools.

You’ll have heard of Zeno’s Paradox. The one where a journey can theoretically never be completed because in order to travel the full distance you must first go halfway. And then once you’re halfway, you must then go half the remaining distance and so on.

The paradox is that in order to do something as simple as walking across a room, one must complete an infinitely regressing set of tasks. And yet, without wishing to boast, I’ve crossed the room twice already today and I managed it just fine.

Software estimation is a bit like that. If you analyse it closely you’ll see the tasks you have to complete multiply infinitely until even the simplest thing looks impossible and the budget is smashed to smithereens. And yet, as a company, we’ve got a track record of delivering on time and to budget that goes back years.

The various methods that we use are described in the episode of our podcast that this post supports (Why not go and check it out?) and we won’t go into detail here suffice to say that the process is always time-consuming and rarely problem-free.

So it’s hard. And prone to error. And time consuming to even do badly. So why do it?

The obvious answer – so you know how much to charge – is not actually all that applicable. More and more of the work we do on agile projects is charged on a time and materials basis. Additionally, there are a hundred good reasons why an agency might want to charge a price that wasn’t just literally [amount of time estimated] multiplied by [hourly rate].

No, the real reason that we put so much effort into estimation is that estimation is a great disinfectant. Everyone who works in this industry has a story about a project that went from perfectly fine to completely awful in a matter of seconds. Estimation helps us expose and resolve the factors that cause this horror: hidden complexity, differences of assumption, Just Plain Goofs etc.

It’s important to note though that even a carefully produced estimate can still be wrong and so the other key tools an agency needs are mature processes and procedures. You need to be able to effectively communicate how the estimate failed, assess what the impact of the failure will be to the broader project and, vitally, put all this information in a place where it can’t be forgotten or ignored.

This last step is effectively giving the organisation an institutional memory that lasts longer than 10 working days and it’s the vital step that ensures that by the end of the project the stakeholders can remember that there was a problem, see that it was resolved and how it affected timelines overall. Mistakes are always going to be made but the key thing is to ensure you’re always making exciting new ones rather than repeating the old ones.

All of the above is discussed to some extent in our Estimating podcast. Myself, Andy Theyers and Richard Newton spend around half an hour discussing the subject and, honestly, it’s quite interesting. I urge you to check it out.

Video: Serverless – the real sharing economy

Serverless is a new application design paradigm, typified by services like AWS Lambda, Azure Cloud Functions and IBM OpenWhisk. It is particularly well suited to mobile software and Single-Page Application frameworks such as React.

In this video, Doug Winter talks at Digital North in Manchester about what Serverless is, where it comes from, why you would want to use it, how the economics function and how you can get started.

You can see more Isotoma videos on Vimeo.

RxJS: An object lesson in terrible good software

We recently used RxJS on a large, complex asynchronous project integrated with a big third-party distributed system. We now know more about it than, frankly, anyone would ever want to.

While we loved the approach, we hated the software itself. The reasons for this are a great lesson in how not to do software.

Our biggest issue by far with RxJS is that there are two actively developed, apparently stable versions at two different URLs. RxJS 4 is the top result from google for RxJS and lives at https://github.com/Reactive-Extensions/RxJS, and briefly mentions that there is an unstable version 5 at a different address. RxJS 5 lives at https://github.com/ReactiveX/rxjs and has a completely different API to version 4, completely different, far worse (“WIP”) documentation, doesn’t allude to its level of stability, and is written in typescript, so users will need to learn some typescript before they can understand the codebase.

Which version should new adopters use? I have absolutely no idea. Either way, when you google for advice and documentation, you can be fairly certain that the results you get will be for a version you’re not using.

RxJS goes to great lengths to swallow your errors. We’re pretty united here in thinking that it definitely should not. If an observable fires its “error” callback, it’s reasonable that the emitted error should be picked up by the nearest catch operator. Sadly though RxJS also wraps all of the functions that you pass to it with a try/catch block and any exception raised by those functions will also be shunted to the nearest try/catch block. Promises do this too, and many have complained bitterly about it already.

What this means in practice is that finding the source of an error is extremely difficult. RxJS tries to capture the original stack trace and make it available in the catch block but often fails, resulting in a failed observable and an “undefined” error. When my code breaks I’d like it to break where it broke, not in a completely different place. If I expect an error to occur I can catch it as I would anywhere else in the codebase and emit an observable error of the form that I’d expect in my catch block so that my catch blocks don’t all have to accommodate expected failure modes and any arbitrary exception. Days and days of development were lost to bisecting a long pile of dot-chained functions in order to isolate the one that raised the (usually stupidly trivial) error.

At the very least, it’d be nice to have the choice to use an unsafe observable instead. For this reason alone we are unlikely to use RxJS again.

We picked RxJS 5 as it’s been around for a long time now and seems to be being maintained by Netflix, which is reassuring.

The documentation could be way better. It is incomplete, with some methods not documented at all, partially documented as a mystery-meat web application that can’t be searched like any normal technical documentation. Code examples rarely use real-world use-cases so it’s tough to see the utility of many of the Observable methods. Most of the gotchas that caught out all of our developers weren’t alluded to at any point in any of the documentation (in the end, a youtube talk by the lead developer saved the day, containing the first decent explanation of the error handling mechanism that I’d seen or read). Worst of all, the only documentation that deals with solving actual problems with RxJS (higher-order observables) is in the form of videos on the paywalled egghead.io. I can’t imagine a more effective way to put off new adopters than requiring them to pay $200 just to appreciate how the library is commonly used (though, to be clear, I am a fan of egghead).

Summed up best by this thread, RxJS refuses to accept its heritage and admit that it’s a functional library. Within the javascript community there exists a huge functional programming subcommunity that has managed to put together a widely-adopted specification for writing functional javascript libraries that can interoperate with the rest of the available javascript functional libraries. RxJS chooses not to work to this specification  and a number of design decisions such as introspecting certain contained values and swallowing them drastically reduces the ease with which RxJS can be used in functional javascript codebases.

RxJS makes the same mistake lodash did a number of years ago, regularly opting for variadic arguments to its methods rather than taking arrays (the worst example is merge). Lodash did eventually learn its lesson, I hope RxJS does too.

Taming the Async Beast with FRP and RxJS

The Problem

We’ve recently been working on an in-browser vision mixer for the BBC (previous blog posts here, here, here, and here). Live vision mixing involves keeping track of a large number of BBC R&D logointerdependent data streams. Our application receives timing data for video tapes and live video streams via webrtc data channels and websocket connections and we’re sending video and audio authoring decisions over other websockets to the live rendering backend.

Many of the data streams we’re handling are interdependent; we don’t want to send an authoring decision to the renderer to cut to a video tape until the video tape is loaded and ready to play, so we need to wait until the video tape is ready to play before we send an authoring decision; if the authoring websocket has closed we’ll need to reconnect to it then retry sending that authoring decision.

Orchestrating interdependent asynchronous data streams is a fundamentally complex problem.

Promises are one popular solution for composing asynchronous operations and safely transforming the results, however they have a number of limitations. The primary issue is that they cannot be cancelled, so we need to handle teardown separately somehow. We could use the excellent fluture or Task Future libraries instead, both of which support cancellation (and are lazy and chainable and fantasy-land compliant), but futures and promises handle one single future value (or error value), not a stream of many values (or error value). The team working this project are fans of futures (less so of promises) and were aiming to write the majority of the codebase in a functional style using folktale and ramda (and react-redux) so wanted a functional, composable way to handle ongoing streams of data that could sit comfortably within the rest of the codebase.

A Solution

After some debate, we decided to use FRP (functional reactive programming) powered by the observable pattern. Having used RxJS (with redux-observable) for smaller projects in the past, we were confident that it could be an elegant solution to our problem. You can find out more about RxJS here and here but, in short, it’s a library that allows subscribers to listen to and transform the output of a data stream as per the observer pattern, and allows the observable (the thing subscribed to) to “complete” its stream when it runs out of data (or whatever), similar to an iterator from the iterator pattern. Observables also allow their subscribers to terminate them at any point, and typically observables will encapsulate teardown logic related to their data source – a websocket, long-poll, webrtc data channel, or similar.

RxJS implements the observer pattern in a functional way that allows developers to compose together observables, just as they’d compose functions or types. RxJS has its roots in functional reactive programming and leverages the power of monadic composition to chain together streams while also ensuring that teardown logic is preserved and handled as you’d expect.

Why FRP and Observables?

The elegance and power of observables is much more easily demonstrated than explained in a wordy paragraph. I’ll run through the basics and let your imagination think through the potential of it all.

A simple RxJS observable looks like this:

Observable.of(1, 2, 3)

It can be subscribed to as follows:

Observable.of(1, 2, 3).subscribe({
  next: val => console.log(`Next: ${val}`),
  error: err => console.error(err),
  complete: () => console.log('Completed!')
});

Which would emit the following to the console:

Next: 1
Next: 2
Next: 3
Completed!

We can also transform the data just as we’d transform values in an array:

Observable.of(1, 2, 3).map(x => x * 2).filter(x => x !== 4).subscribe(...)
2
6
Completed!

Observables can also be asynchronous:

Observable.interval(1000).subscribe(...)
0 [a second passes]
1 [a second passes]
2 [a second passes]
...

Observables can represent event streams:

Observable.fromEvent(window, 'mousemove').subscribe(...)
[Event Object]
[Event Object]
[Event Object]

Which can also be transformed:

Observable.fromEvent(window, 'mousemove')
  .map(ev => [ev.clientX, ev.clientY])
  .subscribe(...)
[211, 120]
[214, 128]
[218, 139]
...

We can cancel the subscriptions which will clean up the event listener:

const subscription = Observable.fromEvent(window, 'mousemove')
  .map(ev => [ev.clientX, ev.clientY])
  .subscribe(...)

subscription.unsubscribe();

Or we can unsubscribe in a dot-chained functional way:

Observable.of(1, 2, 3)
  .take(2)  // After receiving two values, complete the observable early
  .subscribe(...)
1
2
Completed!
Observable.fromEvent(window, 'mousemove')
  .map(ev => [ev.clientX, ev.clientY])
   // Stop emitting when the user clicks
  .takeUntil(Observable.fromEvent(window, 'click'))
  .subscribe(...)

Note that those last examples left no variables lying around. They are entirely self-contained bits of functionality that clean up after themselves.

Many common asynchronous stream use-cases are catered for natively, in such a way that the “operators” (the observable methods e.g. “throttle”, “map”, “delay”, “filter”) take care of all of the awkward state required to track emitted values over time.

Observable.fromEvent(window, 'mousemove')
  .map(...)
  .throttle(1000) // only allow one event through per second
  .subscribe(...);

… and that’s barely scratching the surface.

The Benefits

Many of the benefits of RxJS are the benefits of functional programming. The avoidance of state, the readability and testability of short, pure functions. By encapsulating the side-effects associated with your application in a generic, composable way, developers can maximise the reusability of the asynchronous logic in their codebase.

By seeing the application as a series of data transformations between the external application interfaces, we can describe those transformations by composing short, pure functions and lazily applying data to them as it is emitted in real-time.

Messy, temporary, imperative variables are replaced by functional closure to give observables access to previously emitted variables in a localised way that limits the amount of the application logic and state a developer must hold in their head at any given time.

Did It Work?

Sort of.  We spent a lot of our time in a state of low-level fury at RxJS, so much so that we’ve written up a long list of complaints, in another post.

There are some good bits though:

FRP and the observable pattern are both transformative approaches to writing complex asynchronous javascript code, producing fewer bugs and drastically improving the reusability of our codebase.

RxJS operators can encapsulate extremely complex asynchronous operations and elegantly describe dependencies in a terse, declarative way that leaves no state lying around.

In multiple standups throughout the project we’ve enthusiastically raved about how these operators have turned a fundamentally complex part of our implementation into a two line solution. Sure those two lines usually took a long time to craft and get right, but once working, it’s difficult to write many bugs in just two lines of code (when compared to the hundreds of lines of imperative code we’d otherwise need to write if we rolled our own).

That said, RxJS is a functional approach to writing code so developers should expect to incur a penalty if they’re new to the paradigm as they go from an imperative, object-oriented approach to system design to a functional, data-flow-driven approach instead. There is also a very steep learning curve required to feel the benefits of RxJS as developers familiarise themselves with the toolbox and the idiosyncrasies.

Would We Use It Again?

Despite the truly epic list of shortcomings, I would still recommend an FRP approach to complex async javascript projects. In future we’ll be trying out most.js to see if it solves the myriad of problems we found with RxJS. If it doesn’t, I’d consider implementing an improved Observable that keeps its hands off my errors.

It’s also worth mentioning that we used RxJS with react-redux to handle all redux side-effects. We used redux-observable to achieve this and it was terrific. We’ll undoubtedly be using redux-observable again.