Category Archives: Uncategorized

Our Plants Need Watering Part II

This is the second post in a series doing a deep dive into Internet of Things implementation.  If you didn’t read the first post, Our Plants Need Watering Part I, then you should read that first.

This post talks about one of the most important decisions you’ll make in an IoT project: which microcontroller to use. There are lots of factors and some of them are quite fractal – but that said I think I can make some concrete recommendations based on what I’ve learned so far on this project, that might help you in your next IoT project.

This post gets really technical I am afraid – there’s no way of comparing microprocessors without getting into the weeds.

There are thousands of different microcontrollers on the market, and they are all different.  How you choose the one you want depends on a whole range of factors –  there is no one-size-fits all answer.

Inside a microcontroller

A microcontroller is a single chip that provides all the parts you require to connect software and hardware together. You can think of it as a tiny, complete, computer with CPU, RAM, storage and IO. That is where the resemblance ends though, because each of these parts is quite different from the computers you might be used to.

 

CPU

The Central Processing Unit (CPU) takes software instructions and executes them. This is the bit that controls the rest of the microcontroller, and runs your software.

Microcontroller CPUs come in all shapes and sizes all of which governs the performance and capabilities of the complete package. Mostly the impact of your CPU choice is smaller than you might think – toolchains and libraries protect you from most of the differences between CPU platforms.

Really it is price and performance that matter most, unless you need very specific capabilities. If you want to do floating point calculations or do high-speed video or image processing then you’re going to select a platform with those specific capabilities.

Flash Memory

The kind of computers we are used to dealing with have hard disks to hold permanent storage. Microcontrollers generally do not have access to hard disks. Instead they have what is called “flash” memory. This is permanent – it persists even if power is disconnected. The name “flash” comes from the way the memory is erased “like a camera flash”. It’s colloquially known as just “flash”.

You need enough flash to store your code. The amount of flash available varies tremendously. For example the Atmel ATtiny25 has a whole 2KB of flash whereas the Atmel ATSAM4SD32 has 2MB.

Determining how big your code will be is an important consideration, and often depends on the libraries you need to use. Some quotidian things we take for granted in the macro world, like C’s venerable printf function are too big to fit onto many microcontrollers in its normal form.

Static RAM (SRAM)

Flash is not appropriate for storing data that changes. This means your working data needs somewhere else to go. This is generally SRAM. You will need enough SRAM to hold your all changeable data.  

The amount of SRAM available varies widely. The ATtiny25 has a whole 128 bytes (far less than the first computer I ever programmed, the ZX81, and that was 35 years ago!). At the other end of the scale the ATSAM4SD32 has 160K, and can support separate RAM chips if you need them.

I/O Pins

Microcontrollers need to talk to the outside world, and they do this via their I/O pins. You are going to need to count the pins you need, which will depend on the devices you plan to connect your microcontroller to.

Simple things like buttons, switches, LEDs and so forth can use I/O pins on an individual basis in software, and this is a common use case. Rarely do you build anything that doesn’t use a switch, button or an LED.

If you are going to talk digital protocols however you might well want hardware support for those protocols. This means you might consider things like I²C, RS232 or ISP.

A good example of this is plain old serial. Serial is a super-simple protocol that dates back to the dark ages of computing. One bit at a time is sent over a single pin, and these are assembled together into characters. Serial support needs a bit of buffering, some timing control and possibly some flow control, but that’s it.

The ATtiny range of microprocessors have no hardware support for serial, so if you want to even print text out to your computer’s serial port you will need to do that in software on the microprocessor. This is slow, unreliable and takes up valuable flash. It does work though, at slow speeds – timing gets unreliable pretty quickly when doing things in software.

At the other end you have things like the SAM3X8E based on the ARM Cortex M3 which have a UART and 3 USARTs – hardware support for high speed (well 115200 baud) connections to several devices simultaneously and reliably.

Packaging

There are loads of different packaging formats for integrated circuits. Just check out the list on Wikipedia. Note that when you are developing your product you are likely to use a “development board”, where the microcontroller is already mounted on something that makes it easy to work with.

Here is a dev board for the STM32 ARM microprocessor:

(screwdriver shown for scale).

You can see the actual microprocessor here on the board:

Everything else on the board is to make it easier to work with that CPU – for example adding protection (so you don’t accidentally fry it), making the pins easier to connect, adding debug headers and also a USB interface with a programmer unit, so it is easy to program the chip from a PC.

For small-scale production use, “through hole” packages like DIP can be worked with easily on a breadboard, or soldered by hand. For example, here is a complete microcontroller, the LPC1114FN28:

Some, others, like “chip carriers” can fit into mounts that you can use relatively easily, and finally there are “flat packages”, which you would struggle to solder by hand:

Development support

It is all very well choosing a microcontroller that will work in production – but you need to get your software written first. This means you want a “dev board” that comes with the microcontroller helpfully wired up so you can use it easily.

There are dev boards available for every major platform, and mostly they are really quite cheap.

Here are some examples I’ve collected over the last few years:

The board at the bottom there is an Arduino Due, which I’ve found really useful.  The white box connected to it is an ATMEL debug device, which gives you complete IDE control of the code running on the CPU, including features like breakpoints, watchpoints, stepping and so forth.

Personally I think you should find a dev board that fits your needs first, then you need to choose a microcontroller that is sufficiently similar. A workable development environment is absolutely your number one goal!

Frameworks, toolchains and libraries

This is another important consideration – you want it to be as easy as possible to write your code, whilst getting full access to the capabilities of the equipment you’ve chosen.

Arduino

Arduino deserves a special mention here, as a spectacularly accessible way into programming microprocessors. There is a huge range of Arduino, and Arduino compatible, devices starting at only a few pounds and going right up to some pretty high powered equipment.

Most Arduino boards have a standard layout allowing “shields” to be easily attached to them, giving easy standardised access to additional equipment.

The great advantage of Arduino is that you can get started very easily. The disadvantage is that you aren’t using equipment you could go into production with directly. It is very much a hobbyist solution (although I would love to hear of production devices using Arduino code).

Other platforms

Other vendors have their own IDEs and toolchains – many of which are quite expensive.  Of the ones I have tried Atmel Studio is the best by far.  First it is free – which is pretty important.  Second it uses the gcc toolchain, which makes debugging a lot easier for the general programmer.  Finally the IDE itself is really quite good.

Next time I’ll walk through building some simple projects on a couple of platforms and talk about using the Wifi module in earnest.

 

One Pound in Three

Can we talk about this:

Big opportunities for small firms: government set to spend £1 in every £3 with small businesses

When its predecessor (of £1 in 4) was announced in 2010 many of us were sceptical, so it was fantastic news in 2014 when the National Audit Office announced that this target had not only been met, but exceeded. I don’t think anyone doubts that the new £1 in 3 target will be achieved by 2020; a real measure of confidence in the commitment to these plans.

It’s fair to say that it’s genuinely been a great move forward. It’s taken some time – as you might expect – both for this to trickle all the way down to the smaller end of the SME sector and for departments and other bodies to get their procurement processes aligned, but in the last couple of years we’ve have seen many positive and concrete changes to the way the public sector procures services.

We’ve been involved in quite a few of these SME tendering processes in the last year or so and have seen a full range of tenders from the very good through to the very bad. What’s clear is that things are continuing to improve as buyers and their procurement departments learn to navigate the new types of relationships that the public sector has with these smaller suppliers.
So a lot’s changed, but what could still improve?

Procurement workshops and briefing days

Soon after the 2010 announcement and in the midst of a fashion for “hackathons” and the open web these were all the rage; you could hardly go a week without one body or another running an event of this type.

You know the ones. Every Government department and even most local councils run them; non-government public bodies like the BBC, Channel 4 and JISC love them too. The intention is absolutely sound – you want to get us excited about working with you, outline the projects that we might be working on, help shape our proposals, and ultimately make sure we understand that you’re worth the effort of us pitching to.

There’s no doubt that these are great events to attend. But. They’re often marketed as “great opportunities” and there’s frequently a sense that we must attend to ensure that we don’t miss out. But time out of the office costs money, as does getting half way across the country because the “North” briefing is in London (I kid you not, that’s happened to me more than once). On top of that the audience and content of the talks at these events can be scarily similar regardless of location or presenting organisation. There’s nothing more disheartening than arriving at another one of these events to a feeling that only the venue and speakers have changed.

It’s obviously vitally important that you get these messages across, but please try and make sure that the events themselves don’t feel compulsory. SMEs are time poor (particularly the good ones); if it’s clear that I’m not going to miss out if I don’t attend and that all the information I need will be online then I may well choose not to come. It doesn’t mean I’m not engaged, just that new channels like this are things I often need to explore outside the usual working day.
There’s often a sense of “if we make it really explicit what we’re after at the workshop” that you’ll cut down on the number of inappropriate responses to your tenders. Sadly the opposite is often true – once someone has spent a lot of time and money in attending one of the briefing days they will pitch for absolutely everything, because they now feel invested, and they’ve met you. Sunk cost thinking affects us all.

Luckily the number of these apparently mandatory briefing days is reducing, with some organisations doing away with them entirely, replacing them with live web conferences, pre-recorded video presentations and detailed (and high quality) documentation. I’d love to see them done away with entirely, though.

Keeping contracts aligned

It’s a fair assumption that during the briefing days every single speaker will have made at least one reference to Agile. And it’s likely that Agile was the main topic of at least one talk. Because Agile is good. You get that. We get that. Agile makes absolute sense for many of the kinds of projects that the public sector is currently undertaking. Digital Transformation is certainly not easy, it’s definitely not cheap and it’s absolutely not going to be helped by a waterfall, BDUF approach.

But if you’re honestly committed to Agile please please ensure that your contracts reflect that. We’ve recently had to pull out of two tenders where we’d got down to the last round because the contract simply couldn’t accommodate a genuine Agile delivery. We know Agile contracts are hard, but if you’ve spent the entire procurement process actively encouraging people to pitch you an Agile approach you need to present an Agile contract at the end of it. Companies as old and grizzled as Isotoma may feel forced – and be willing – to back away, but for many agencies it’s a trap they unwittingly fall into which ultimately does nothing for either party.

It’s also worth remembering that it’s unlikely any SME you deal with has internal legal advice, so contract reviews are an expensive luxury. If you present a mandatory contract at the start of the tender process most of us will glance over it before ploughing ahead. We certainly aren’t going to pay for a full scale review because we know it’ll cost a fortune and the lawyer is only going to tell us it’s too risky and we shouldn’t pitch anyway. One contract we were presented with by a government department was described by our lawyer as a “witch’s curse”. We still pitched. Didn’t win it. Probably for the best.

Timelines

They say it’s the hope that kills you.

Small businesses are, by definition, small. The kind of procurements I’m talking about here are for services, not products, which means that people – our people, our limited number of people – are going to be required for the delivery. If the timeline on the procurement says “award contract on 17th February 2017, go live by end June 2017” we’re going to start trying to plan for what winning might look like. This might well involve subtly changing the shape of other projects that we’ve got in flight. If we’re really confident it might even mean turning away other work.

When we get to the 17th February and there’s no news from you what are we supposed to do? Do we hold the people we’d pencilled in for this work back and live with the fact that they’re unbilled?. And then when 24th February comes and there’s another round of clarification questions, but you commit to giving us an answer by the following week what do we do then? And so on. And so on.

The larger the business you’re dealing with the easier they find absorbing these kind of changes to timelines, but that’s one of the reasons they’re more expensive. SMEs are small, they’re nimble, but they also rely on keeping their utilisation high and their pipeline flowing. Unrealistic procurement timelines combined with fixed delivery dates can make pitching for large tenders very uncomfortable indeed.

To summarise

As I said at the start things have made huge leaps forward over the past couple of years. The commitment to pay 80% of all undisputed invoices within 5 days is a great example of how the public sector is starting to really understand the needs of SMEs, as is removing the PQQ process for smaller contracts, the commitment to dividing contracts into lots and explicitly supporting consortia and subcontracting.

In 2016 we’ve been to sadly uninformative developer days for an organisation that has offered wonderfully equitable Agile contracts and extremely clear and accurate timelines. We’ve pitched for work that was beautifully explained online with no developer day, but that presented a bear trap of a contract, and we’ve pitched for work that was perfect except for the wildly optimistic timelines and that finally awarded the contract 3 months after the date in the tender.

Things are definitely getting better, but a few more little tweaks could make them perfect.
Here’s to £1 in 3, and the continuing good work that everyone is doing across the sector.

Internet Security Threats – When DDoS Attacks

On Friday evening an unknown entity launched one of the largest Distributed Denial of Service (DDoS) attacks yet recorded, against Dyn, a DNS provider. Dyn provide service for some of the Internet’s most popular services, and they duly suffered problems. Twitter, Github and others were unavailable for hours, particularly in the US.

DDoS attacks happen a lot, and are generally uninteresting. What is interesting about this one is:

  1. the devices used to mount the attack
  2. the similarity with the “Krebs attack” last month
  3. the motive
  4. the potential identity of the attacker

Together these signal that we are entering a new phase in development of the Internet, one with some worrying ramifications.

The devices

Unlike most other kinds of “cyber” attack, DDoS attacks are brute force – they rely on sending more traffic than the recipient can handle. Moving packets around the Internet costs money so this is ultimately an economic contest – whoever spends more money wins. The way you do this cost-effectively, of course, is to steal the resources you use to mount the attack. A network of compromised devices like this is called a “botnet“.

Most computers these days are relatively well-protected – basic techniques like default-on firewalls and automated patching have hugely improved their security. There is a new class of device though, generally called the Internet of Things (IoT) which have none of these protections.

IoT devices demonstrate a “perfect storm” of security problems:

  1. Everything on them is written in the low-level ‘C’ programming language. ‘C’ is fast and small (important for these little computers) but it requires a lot of skill to write securely. Skill that is not always available
  2. Even if the vendors fix a security problem, how does the fix get onto the deployed devices in the wild? These devices rarely have the capability to patch themselves, so the vendors need to ship updates to householders, and provide a mechanism for upgrades – and the customer support this entails
  3. Nobody wants to patch these devices themselves anyway. Who wants to go round their house manually patching their fridge, toaster and smoke alarm?
  4. Because of their minimal user interfaces (making them difficult to operate if something goes wrong), they often have default-on [awful] debug software running. Telnet to a high port and you can get straight in to adminster them
  5. They rarely have any kind of built in security software
  6. They have crap default passwords, that nobody ever changes

To see how shockingly bad these things are, follow Matthew Garrett on Twitter. He takes IoT devices to pieces to see how easy they are to compromise. Mostly he can get into them within a few minutes. Remarkably one of the most secure IoT device he’s found so far was a Barbie doll.

That most of these devices are far worse than a Barbie doll should give everyone pause for thought. Then imagine the dozens of them so many of us have scattered around our house.  Multiply that by the millions of people with connected devices and it should be clear this is a serious problem.

Matthew has written on this himself, and he’s identified this as an economic problem of incentives. There is nobody who has an incentive to make these devices secure, or to fix them afterwards. I think that is fair, as far as it goes, but I would note that ten years ago we had exactly the same problem with millions of unprotected Windows computers on the Internet that, it seemed, nobody cared about.

The Krebs attack

A few weeks ago, someone launched a remarkably similar attack on a security researcher Brian Krebs. Again the attackers are unknown and they launched the attack using a global network of IoT devices.

Given the similarities in the attack on Krebs and the attack on Dyn, it is probable that both of these attacks were undertaken by the same party. This doesn’t, by itself, tell us very much.

It is common for botnets to be owned by criminal organisations that hire them out by the hour. They often have online payment gateways, telephone customer support and operate basically like normal businesses.

So, if this botnet is available for hire then the parties who hired it might be different. However, there is one other similarity which makes this a lot spookier – the lack of an obvious commercial motive.

The motive

Mostly DDoS attacks are either (a) political or (b) extortion. In both cases the identity of the attackers is generally known, in some sense. For political DDOS attacks (“hacktivism”) the targets have often recently been in the news, and are generally quite aware of why they’re attacked.

Extortion using DDoS attacks is extremely common – anyone who makes money on the Internet will have received threats, and have been attacked and many will have paid out to prevent or stop a DDoS.  Banks, online gaming, DNS providers, VPN providers and ecommerce sites are all common targets – many of them so common that they have experienced operations teams in place who know how to handle these things.

To my knowledge no threats were made to Dyn or Krebs before the attacks and nobody tried to get money out of them to stop them.

What they have in common is their state-of-the-art protection. Brian Krebs was hosted by Akamai, a very well-respected content delivery company who have huge resources – and for whom protecting against DDOS is a line of business. Dyn host the DNS for some of the world’s largest Internet firms, and similarly are able to deploy huge resources to combat DDOS.

This looks an awful lot like someone testing out their botnet on some very well protected targets, before using it in earnest.

The identity of the attacker

It looks likely therefore that there are two possibilities for the attacker. Either it is (a) a criminal organisation looking to hire out their new botnet or (b) a state actor.

If it is a criminal organisation then right now they have the best botnet in the world. Nobody is able to combat this effectively.  Anyone who owns this can hire it out to the highest bidder, who can threaten to take entire countries off the Internet – or entire financial institutions.

A state actor is potentially as disturbing. Given the targets were in the US it is unlikely to be a western government that controls this botnet – but it could be one of dozens from North Korea to Israel, China, Russia, India, Pakistan or others.

As with many weapons a botnet is most effective if used as a threat, and we many never know if it is used as a threat – or who the victims might be.

What should you do?

As an individual, DDoS attacks aren’t the only risk from a compromised device. Anyone who can compromise one of these devices can get into your home network, which should give everyone pause – think about the private information you casually keep on your home computers.

So, take some care in the IoT devices you buy, and buy from reputable vendors who are likely to be taking care over their products. Unfortunately the devices most likely to be secure are also likely to be the most expensive.

One of the greatest things about the IoT is how cheap these devices are, and the capability they can provide at this low price. Many classes of device don’t necessarily even have reliable vendors working in that space. Being expensive and well made is no long-term protection – devices routinely go out of support after a few years and become liabilities.

Anything beyond this is going to require concerted effort on a number of fronts. Home router vendors need to build in capabilities for detecting compromised devices and disconnecting them. ISPs need to take more responsibility for the traffic coming from their networks. Until being compromised causes devices to malfunction for their owner there will be no incentives to improve them.

It is likely that the ultimate fix for this will be Moore’s Law – the safety net our entire industry has relied on for decades. Many of the reasons for IoT vulnerabilities are to do with their small amounts of memory and low computing power. When these devices can run more capable software they can also have the management interfaces and automated patching we’ve become used to on home computers.

 

The economics of innovation

One of the services we provide is innovation support. We help companies of all sizes when they need help with the concrete parts of developing new digital products or services for their business, or making significant changes to their existing products.

A few weeks ago the Royal Swedish Academy of Sciences awarded the Nobel Prize for Economics to Oliver Hart and Bengt Holmström for their work in contract theory. This prompted me to look at some of his previous work (for my sins I find economics fascinating), and I came across his 1998 paper Agency Costs and Innovation. This is so relevant to some of my recent experiences I wanted to share it.

Imagine you have a firm or a business unit and you have decided that you need to innovate.

This is a pretty common situation – you know strategically that your existing product is starting to lose traction. Maybe you can see commoditisation approaching in your sector. Or perhaps, as is often the case, you can see the Internet juggernaut bearing down on your traditional business and you know you need to change things up to survive.

What do you do about it?  If you’ve been in this situation the following will probably resonate:

agency2

This describes the principal-agent problem, which is a classic in economics. This describes how a principal (who wants something) can incentivise an agent to do what they want. The agent and “contracting” being discussed here could be any kind of contracting including full time staff.

A good example of the principal-agent problem is how you pay a surgeon. You want to reward their work, but you can’t observe everything they do. The outcome of surgery depends on team effort, not just an individual. They have other things they need to do other than just surgery – developing standards, mentoring junior staff and so forth. Finally the activity itself is very high risk inherently – which means surgeons will make mistakes, no matter how competent. This means their salary would be at risk, which means you need to pay huge bonuses to encourage them to undertake the work at all.

In fact commonly firms will try and innovate using their existing teams, who are delivering the existing product. These teams understand their market. They know the capabilities and constraints of existing systems. They have domain expertise and would seem to be the ideal place to go.

However, these teams have a whole range of tasks available to them (just as with our surgeon above), and choices in how they allocate their time. This is the “multitasking effect”. This is particularly problematic for innovative tasks.

My personal experience of this is that, when people have choices between R&D type work and “normal work”, they will choose to do the normal work (all the while complaining that their work isn’t interesting enough, of course):

variance

This leads large firms to have separate R&D divisions – this allows R&D investment decisions to take place between options that have some homogeneity of risk, which means incentives are more balanced.

However, large firms have a problem with bureaucratisation. This is a particular problem when you wish to innovate:

monitoring

Together this leads to a problem we’ve come across a number of times, where large firms have strong market incentives to spend on innovation – but find their own internal incentive systems make this extremely challenging.

If you are experiencing these sorts of problems please do give us a call and see how we can help.

I am indebted to Kevin Bryan’s excellent A Fine Theorem blog for introducing me to Holmström’s work.

 

Transforming a business platform

The-Key-Transform-CMS

As you may have seen in our previous blog post – Evolving The Key: insights from user research – after a year in design and development we recently helped The Key Support relaunch The Key for School Leaders and School Governors. This post looks at the technology selections for the refresh of The Key’s content management platform and why certain elements were chosen.

In the nine years since launching The Key has grown to support almost half of schools in England, with an amazing 75,000 registered school leaders and 17,000 registered governors having access to 5,000 original resources every month. It is now one of the most trusted sources of information in the education sector.

Selecting a platform

The sustained growth of The Key in both size and breadth meant there was a real need to TheKey-Screen1
update the underlying platform.  The new content management system (CMS) needed to be efficient at managing user subscriptions, making the right content available to the right users (The Key has 7 different classes of user), as well as being ready for any future expansion plans.

The platform for the past 9 years has been Plone, an open source enterprise content management system first released 15 years ago. In 2007 – when we built the first version of The Key – Plone was the ideal choice, but as the business requirements have expanded and been refined over the years we felt it was a good time to revisit that selection when we were presented with the opportunity to completely refresh both sites.

As The Key has grown in size so has the variety of content they are displaying on the site. As the breadth and types of this content has developed The Key have struggled with the restrictions created by the traditional template-driven nature of Plone. This prompted us to consider more flexible CMS options.

The solution? A shift from Plone to Wagtail.

The-Key-Wagtail-CMS

We were already pretty impressed with Wagtail, having already used it on a couple of smaller projects. Like Plone it’s an open source CMS written in Python, but Wagtail is built on Django, our preferred web framework, giving us all the advantages that Django brings. We wanted to make sure that the new platform would stand the test of time as well as the previous Plone solution had, so we ran a careful evaluation process between a group of Django based solutions – including Wagtail, Mezzanine, Django CMS and a bespoke pure Django approach – to see which would best meet The Key’s requirements. We’re pleased to say that Wagtail came out the clear winner!

There are a few reasons we were particularly impressed with Wagtail for an application of this size and scale…

  • It is highly extensible, meaning that we could push the user model very hard and accommodate the intricacies of The Key’s user base
  • There’s an extremely high quality ‘out of the box’ admin system, meaning that we could hit the goal of improving the editor experience without huge amounts of bespoke development
  • Wagtail supports the notion of page centric content management (through its StreamFields) which allowed us to build much richer pages than a traditional template driven CMS
  • There are powerful versioning tools built into the framework which would give The Key the level of control they need when managing changes to sensitive content

These features of Wagtail aligned beautifully with The Key’s requirements, allowing us to focus on delivering the features that they really needed.

Wagtail is a new and exciting open source platform which is constantly growing with new features and contributions. We were really looking forward to being involved and contributing some elements of our own.

Making the move…

One of the first tasks to complete as part of the move was to export the data out of Plone and into Wagtail. This involved the careful migration of over 30,000 pages across two websites, complete with full page history, allowing us to preserve all of The Key’s valuable content and metadata.

The goals of this project were manyfold for The Key:

  • Improve the member experience, making it easier to manage a school’s membership
  • Improve members’ ability to self-serve, improving their experience and reducing the workload of the team as the business grows
  • Improve the quality and measurability of online marketing activities
  • Improve the quality and robustness of reporting tools.

Making the move from Plone to Wagtail held so many benefits for The Key we couldn’t write about them all, but have summarised our favourites:

  • Improved user acquisition journey
  • Improved signposting of the huge variety of content on the site
  • It’s a long term solution, Wagtail can expand and grow alongside The Key
  • Flexible modular home page

Another important task was to ensure that any user behaviour tracking was successfully migrated over to Wagtail. The Key harness their large database of users to track and record vital information which is then translated into leading insights, ensuring The Key remain at the forefront of trends and industry changes.

Through our longstanding relationship with The Key we understand how valuable this data is, so we used a custom API to integrate a data warehousing service called Keen.io. This service intelligently stores the data allowing The Key to collate, store and build their own queries and analysis of user behaviour, allowing them to constantly refine and improve their content to better serve their members.

Monitoring performance

To ensure the stability of the complex infrastructure that supports a project of this scale we installed New Relic – a real-time software analytics program. New Relic provides deep performance analysis for every part of TheKey-Screen2The Key’s platform, enabling us to make faster decisions, monitor interactions, quickly pinpoint errors and achieve better business results for The Key.

What we’ve found working with Wagtail is that it’s so flexible, customisable, scalable and user friendly. It’s working wonders for some of our other clients too. If you’re interested to know what moving to Wagtail could do for the performance of your site then get in touch, we won’t try and sell you something you don’t want or need!

Stay tuned

The next blog installment: How has The Key benefited this update a month after deployment?

In our next blog post about The Key we’ll be revisiting the site a month after deployment to find out how their staff members got on with the CMS change and what impact it has had on the business.

If you found this article interesting and are looking for an agency to help you with an upcoming project, please do contact us and find out how we can help you. Alternatively you can read about some more of our work and see how we have helped other companies achieve their goals.

Beating Cancer To The Punch: Isotoma Project Officer Goes Three Rounds For Charity

Fighting back against cancer

Our Project Officer Daniel Merriman took part in his first charity Ultra White Collar Boxing (UWCB) match last weekend to raise money for Cancer Research UK. In the interview below, I ask Daniel what drove him to get into the ring…

1. So, what made you decide to take part in a charity boxing match?
“It was a spur of the moment thing, really! I did some boxing at university over a decade ago, but never had the chance to fight in front of a big crowd. Then earlier this year I saw an article in the local paper about these charity boxing events, and I speculatively sent them a message asking what was involved. Three days later I was in the gym with forty other novice boxers.”

2. How long did you train for, did you discover any difficult areas in training?
“Training was two sessions a week for eight weeks at Chokdee Academy, under the watchful eye of former world thai boxing champion Rich Cadden. It brought back a lot of memories of when I trained at university, but also emphasised just how out of condition I’d become. I’d been doing some running as training for an upcoming 10km race, but the type of fitness involved is on a totally different level. There were plenty of evenings when I came home with bruises and aching joints, but it was also immensely satisfying to see (and feel) the improvement as the weeks passed. In fact, the most frustrating aspect for me was having to miss some sessions because I wasn’t able to get a lift to the gym!”

3. Tell us about the fight night…
“Fight night for me actually started around 1pm with a medical, checking my blood pressure, shining a torch in my eyes and signing the all-important medical consent form. There were then talks from the organiser and the referee, explaining the format for the evening and the rules for the event. Unfortunately it turned out that my bout was slot 21 out of 22, so I
had a long time toDan-Close-Up wait! I spent the next few hours out in the audience with my guests at our VIP table, watching the other bouts and trying to conserve my energy.

Eventually I went backstage to warm up and mentally prepare myself. I don’t remember feeling nervous especially, but by that point in the evening I was just desperate to get in the ring. When the announcer called my name and my music started playing, though, the adrenaline started pumping and I knew it was time to (attempt to…) put into practice everything I’d learned in the previous eight weeks.

The fight itself seemed to fly by. I had sparred against my opponent in training before so I knew he was tough, and so he proved. The first round was relatively even, but he got me with a good shot in the second round, and by the end of the third round I was gasping for air. It’s hard to overestimate just how much the adrenaline of being in the ring saps your stamina, but I had some great supporters cheering me on and they helped me dig deep and get through to the final bell.”

4. Were there any unexpected benefits to the experience?
“Well I’ve dropped two sizes in jeans, so that’s something! I guess the other thing is that I wasn’t sure how I’d feel fighting in front of such a large crowd of people – there were close to 1,000 guests in attendance, and I didn’t know how that would affect me. I’m confident enough with public speaking, having done talks in front of a hundred or so students in my past life as a lecturer, but this was a different world altogether. Once I was in the ring, though, I was able to focus entirely on the man in front of me.”

5. Have you got any plans to continue the sport?
“I’ll probably do another bout towards the end of the year, but for now I’ve got a 10km race to prepare for. A team of us from work have signed up to do the Leeds 10k next month, which will be my first race at that distance, so I need to get the miles in. A few of my friends have signed up for the next boxing event though, so I’m sure I’ll be sparring a few rounds with them over the next couple of months.”

6. Most importantly, how much money have you raised for Cancer Research UK?
“Thanks to the generosity of my supporters, and particularly to Isotoma who sponsored me and matched the amount raised by fight night, I’ve raised £890 so far for Cancer Research UK.”

Dan-Close-Up2

7. What advice would you give anyone who is interested in taking up boxing? 
“One of the great benefits of taking part in this event was the top class training I received at Chokdee Academy. If you have any interest in competing, definitely take your time in finding a good gym. Check if they have fighters there who currently compete at an amateur or professional level, talk to the coaches and see what the atmosphere is like at the club. Ask yourself: do I feel comfortable here? Anyone can run a boxercise class, but it takes knowledge and experience to teach a skill like boxing.

Also, don’t be afraid of being “too unfit”! There were plenty of people who fought at the event who, at the beginning of the eight week training cycle, were struggling to do a single push up. The training you’ll receive in a boxing gym, along with advice and guidance regarding diet, will get you fitter than you ever thought possible.”

Dan undertook 8 weeks of training with the professionals at Chokdee Academy to prepare for the fight. If you’d like more information about taking up the sport then please do get in touch with the team at Chokdee, or your local training centre.

Hacking together a standing desk

We’ve had a few people in the office transition to standing desks recently and the process by which everyone achieved their goal has been quite interesting. Or at least interesting to me.

doug's deskDoug was the first to try it and ended up, as a lot of people do, taking the old ‘big pile of books’ approach. It’s cheap, quick and,so long as you’re using a solid and non-tottering pile of books, probably pretty safe. Luckily the Isotoma office has a pretty extensive library of books–which in Doug’s case have mostly been memorised.

(I wouldn’t let him tidy his desk before I took this photo. He wanted me to tell you.)

A different approach that David introduced (as far as I know) and which I’ve adopted is the music stand approach. It’s a wee bit more expensive and depends on you having the kind of job that doesn’t really involve paper or double monitors – but I love it. The bonus of this approach is you get to feel a little bit like a member of an nineties synth-rock band. Always a bonus.

But the canonical Isotoma standing desk was “Ikea-hacked” together by Tom. He went to town on it to such an extent that we made him do an intranet post about it:

I’ve had a couple of enquiries about the parts for my standing desk, so rather than send sporadic emails, I thought I’d just put it here.

8717637419_6b5bfa4fa8_o8717637485_65832cd578_o

I built my standing desk based on this article, which is great, and has assembly instructions and everything (although it’s pretty trivial to put together). The guy bought all the bits from IKEA for $22. But obviously we don’t live in America, and also I needed to double up on some of the parts due to my dual monitor setup, so here is the full parts list, with links to the UK IKEA website:

Also, you will need to find 4 screws to screw the brackets into the table legs because annoyingly, the brackets have holes drilled, but no screws to go in said holes.

Grand total: £29 (including £10 postage)

Oh and don’t forget, you’ll probably want some kind of chair to give your legs a rest every now and then (yes it’s hilarious that Tom still has a chair etc, but it does come in handy, even if it’s just for 2×30 minute spells during the day). I got a bar stool with a back from Barnitts 🙂

This concludes my blog post about standing desks. Please add photos of your sweet, customised desks below in the comments.

Just kidding.

About usIsotoma is a bespoke software development company based in York and London specialising in web apps, mobile apps and product design. If you’d like to know more you can review our work or get in touch.

 

What Heartbleed means for you

On the 8th April a group of security researchers published information about a newly discovered exploit in a popular encryption library. With some marketing panache, they called this exploit “Heartbleed”.

A huge number of Internet services were vulnerable to this exploit, and although many of them have now been patched many remain. In particular, this was an Open Source library and so many of the very largest and most popular sites were directly affected.

Attention on the exploit has so far focused on the possible use of the exploit to obtain “key material” from affected sites, but there are some more immediate ramifications and you need to act to protect yourself.

Unfortunately the attack will also reveal other random bits of webserver’s memory, which can include usernames, passwords and cookies. Obtaining this information will allow attackers to log into these services as you, and then conduct more usual fraud and identity theft.

Once the dust has settled (so later today on the 9th, or tomorrow on the 10th) you should go and change every single one of your passwords. Start with the passwords you’ve used recently and high value services.

It’s probably a good idea to clear all your cookies too once you’ve done this, to force you to re-login to every service with your new password.

You should also log out of every single service on your phone, and then re-login in, to get new session cookies. If you are particularly paranoid, wipe your phone and reinstall. Mobile app session cookies are likely to be a very popular vector for this attack.

This is an enormous amount of work, but you can use it as an opportunity to set some decent random passwords for every service and adopt a tool like LastPass, 1Password or KeePass while you are at it.

Most people are hugely vulnerable to password disclosure because they share passwords between accounts, and the entire world of black-hats are out there right now slurping passwords off every webserver they can get them from. There is going to be a huge spike in fraud and identity theft soon, and you want to make sure you are not a victim to it.

The Man-In-The-Middle Attack

In simple terms this would allow an attacker to impersonate the site’s SSL certificate, so they can show the padlock icon that demonstrates a secure connection, even though they control your connection.

They can only do this if they also manage to somehow make your browser connect to their computers for the request. This can normally only be done by either controlling part of your connection directly (hacking your router maybe), or by “poisoning” your access to the Domain Name Service with which you find out how to reach a site (there are many ways to do this, but none of them are trivial).

You can expect Internet security types to be fretting about this one for a long time to come, and there are likely to be some horrific exploits against some high-profile sites executed by some of the world’s most skilled hackers. If they do it well enough, we may never hear of it.

The impact of this exploit is going to have huge ramifications for server operators and system designers, but there is very little in practical terms that most people can mitigate this risk for their own browsing.

About us: Isotoma is a bespoke software development company based in York and London specialising in web apps, mobile apps and product design. If you’d like to know more you can review our work or get in touch.

Reviewing Django REST Framework

Recently, we used Django REST Framework to build the backend for an API-first web application. Here I’ll attempt to explain why we chose REST Framework and how successfully it helped us build our software.

Why Use Django REST Framework?

RFC-compliant HTTP Response Codes

Clients (javascript and rich desktop/mobile/tablet applications) will more than likely expect your REST service endpoint to return status codes as specified in the HTTP/1.1 spec. Returning a 200 response containing {‘status’: ‘error’} goes against the principles of HTTP and you’ll find that HTTP-compliant javascript libraries will get their knickers in a twist. In our backend code, we ideally want to raise native exceptions and return native objects; status codes and content should be inferred and serialised as required.

If authentication fails, REST Framework serves a 401 response. Raise a PermissionDenied and you automatically get a 403 response. Raise a ValidationError when examining the submitted data and you get a 400 response. POST successfully and get a 201, PATCH and get a 200. And so on.

Methods

You could PATCH an existing user profile with just the field that was changed in your UI, DELETE a comment, PUT a new shopping basket, and so on. HTTP methods exist so that you don’t have to encode the nature of your request within the body of your request. REST Framework has support for these methods natively in its base ViewSet class which is used to build each of your endpoints; verbs are mapped to methods on your view class which, by default, are implemented to do everything you’d expect (create, update, delete).

Accepts

The base ViewSet class looks for the Accepts header and encodes the response accordingly. You need only specify which formats you wish to support in your settings.py.

Serializers are not Forms

Django Forms do not provide a sufficient abstraction to handle object PATCHing (only PUT) and cannot encode more complex, nested data structures. The latter limitation lies with HTTP, not with Django Forms; HTTP forms cannot natively encode nested data structures (both application/x-www-form-urlencoded and multipart/form-data rely on flat key-value formats). Therefore, if you want to declaratively define a schema for the data submitted by your users, you’ll find life a lot easier if you discard Django Forms and use REST Framework’s Serializer class instead.

If the consumers of your API wish to use PATCH rather than PUT, and chances are they will, you’ll need to account for that in your validation. The REST Framework ModelSerializer class adds fields that map automatically to Model Field types, in much the same way that Django’s ModelForm does. Serializers also allow nesting of other Serializers for representing fields from related resources, providing an alternative to referencing them with a unique identifier or hyperlink.

More OPTIONS

Should you choose to go beyond an AJAX-enabled site and implement a fully-documented, public API then best practice and an RFC or two suggest that you make your API discoverable by allowing OPTIONS requests. REST Framework allows an OPTIONS request to be made on every endpoint, for which it examines request.user and returns the HTTP methods available to that user, and the schema required for making requests with each one.

OAuth2

Support for OAuth 1 and 2 is available out of the box and OAuth permissions, should you choose to use them, can be configured as a permissions backend.

Browsable

REST framework provides a browsable HTTP interface that presents your API as a series of forms that you can submit to. We found it incredibly useful for development but found it a bit too rough around the edges to offer as an aid for third parties wishing to explore the API. We therefore used the following snippet in our settings.py file to make the browsable API available only when DEBUG is set to True:

if DEBUG:
    REST_FRAMEWORK['DEFAULT_RENDERER_CLASSES'].append(
        'rest_framework.renderers.BrowsableAPIRenderer'
    )

Testing

REST Framework gives you an APITestCase class which comes with a modified test client. You give this client a dictionary and encoding and it will serialise the request and deserialise the response. You only ever deal in python dictionaries and your tests will never need to contain a single instance of json.loads.

Documentation

The documentation is of a high quality. By copying the Django project’s three-pronged approach to documentation – tutorial, topics, and API structure, Django buffs will find it familiar and easy to parse. The tutorial quickly gives readers the feeling of accomplishment, the high-level topic-driven core of the documentation allows readers to quickly get a solid understanding of how the framework should be used, and method-by-method API documentation is very detailed, frequently offering examples of how to override existing functionality.

Project Status

At the time of writing the project remains under active development. The roadmap is fairly clear and the chap in charge has a solid grasp of the state of affairs. Test coverage is good. There’s promising evidence in the issue history that creators of useful but non-essential components are encouraged to publish their work as new, separate projects, which are then linked to from the REST Framework documentation.

Criticisms

Permissions

We found that writing permissions was messy and we had to work hard to avoid breaking DRY. An example is required. Let’s define a ViewSet representing both a resource collection and any document from that collection:

views.py:

class JobViewSet(ViewSet):
    """
    Handles both URLS:
    /jobs
    /jobs/(?P<id>d+)/$
    """
    serializer_class = JobSerializer
    permission_classes = (IsAuthenticated, JobPermission)

    def get_queryset(self):
        if self.request.user.is_superuser:
            return Job.objects.all()

        return Job.objects.filter(
            Q(applications__user=request.user) |
            Q(reviewers__user=request.user)
        )

If the Job collection is requested, the queryset from get_queryset() will be run through the serializer_class and returned as an HTTPResponse with the requested encoding.

If a Job item is requested and it is in the queryset from get_queryset(), it is run through the serializer_class and served. If a Job item is requested and is not in the queryset, the view returns a 404 status code. But we want a 403.

So if we define that JobPermission class, we can fail the object permission test, resulting in a 403 status code:

permissions.py:

class JobPermission(Permission):
    def get_object_permission(self, request, view, obj):
    if obj in Job.objects.filter(
        Q(applications__user=request.user) |
        Q(reviewers__user=request.user)):
        return True
    return False

Not only have we duplicated the logic from the view method get_queryset (we could admittedly reuse view.get_queryset() but the method and underlying query would still be executed twice), if we don’t then the client is sent a completely misleading response code.

The neatest way to solve this issue seems to be to use the DjangoObjectPermissionsFilter together with the django-guardian package. Not only will this allow you to define object permissions independently of your views, it’ll also allow you filter querysets using the same logic. Disclaimer: I’ve not tried this solution, so it might be a terrible thing to do.

Nested Resources

REST Framework is not built to support nested resources of the form /baskets/15/items. It requires that you keep your API flat, of the form /baskets/15 and /items/?basket=15.

We did eventually choose to implement some parts of our API using nested URLs however it was hard work and we had to alter public method signatures and the data types of public attributes within our subclasses. We required entirely highly modified Router, Serializer, and ViewSet classes. It is worth noting that REST Framework deserves praise for making each of these components so pluggable.

Very specifically, the biggest issue preventing us pushing our nested resources components upstream was REST Framework’s decision to make lookup_field on the HyperlinkedIdentityField and HyperlinkedRelatedField a single string value (e.g. “baskets”). To support any number of parent collections, we had to create a NestedHyperlinkedIdentityField with a new lookup_fields list attribute, e.g. ["baskets", "items"].

Conclusions

REST Framework is great. It has flaws but continues to mature as an increasingly popular open source project. I’d whole-heartedly recommend that you use it for creating full, public APIs, and also for creating a handful of endpoints for the bits of your site that need to be AJAX-enabled. It’s as lightweight as you need it to be and most of what it does, it does extremely well.

About us: Isotoma is a bespoke software development company based in York and London specialising in web apps, mobile apps and product design. If you’d like to know more you can review our work or get in touch.

PloneConf2010 – and there’s more

I’m here at PloneConf2010 too, and I’ve been to mostly different talks to Mitch, so here’s my write up too.  It’s taken a bit of time to find the time to write this up properly!

Calvin Hendryx-Parker: Enterprise Search in Plone using Solr

We’ve been using Solr for a few years here at Isotoma, but so far we’ve not integrated it with Plone.  Plone’s built-in Catalog is actually pretty good, however one thing it doesn’t do fantastically well is full-text search.  It is passable in English, but has very limited stemming support – which makes it terrible in other languages.

Calvin presented their experience of using Solr with Plone. They developed their own software to integrate the Plone catalog with Solr, instead of using collective.solr, which up till then was the canonical way of connecting them. Their new product alm.solrindex sounds significantly better than collective.solr.  Based on what I’ve heard here, you should definitely use alm.solrindex.

To summarise how this all hangs together, you need an instance of Solr installed somewhere that you can use.  You can deploy a solr specifically for each site, in which case you can deploy it through buildout.  Solr is Java, and runs inside various Java application servers.

You can also run a single Solr server for multiple Plone sites – in which case you partition the Solr database.

You then configure Solr, telling it how to index and parse the fields in your content. No configuration of this is required within Plone.  In particular you configure the indexes in Solr not in Plone.

Then install alm.solrindex in your plone site and delete all the indexes that you wish to use with Solr. alm.solrindex will create new indexes by inspecting Solr.

Then reindex your site, and you’re done!  It supports a lot of more complex use cases, but in this basic case you get top-end full text indexing at quite low cost.

Dylan Jay, PretaWeb: FunnelWeb

Funnelweb sounds invaluable if you want to convert an existing non-Plone site into a Plone site, with the minimum effort.

Funnelweb is a tool based on transmogrifier. Transmogrifier provides a “pipeline” concept for transforming content. Pipeline stages can be inserted into a pipeline, and these stages then have the ability to change the content in various ways.

Dylan wrote funnelweb to use transmogrifier and provide a harness for running it in a managed way over existing websites.  The goal is to create a new Plone site, using the content from existing websites.

Funnelweb uploads remotely to Plone over XML-RPC, which means none of transmogrifier needs to be installed in a Plone site, which is a significant advantage.  It is designed to be deployed using buildout, so a script will be provided in your build that executes the import.

A bunch of pipeline steps are provided to simplify the process of importing entire sites.  In particular funnelweb has a clustering algorithm that attempts to identify which parts of pages are content and which are templates.  This can be configured by providing xpath expressions to identify page sections, and then extract content from them for specific content fields.

It supports the concept of ordering and sorts, so that Ordered Folder types are created correctly.  It supports transmogrify.siteanalyser.attach to put attachments closer to pages and transmogrify.siteanalyser.defaultpage to detect index pages in collections and to make them folder indexes in the created sites.

Finally it supports relinking, so that pages get sane urls and all links to those pages are correctly referenced.

Richard Newbury: The State of Plone Caching

The existing caching solution for Plone 3 is CacheFu, which is now pretty long in the tooth.  I can remember being introduced to CacheFu by Geoff Davis at the Archipelago Sprint in 2006, where it was a huge improvement on the (virtually non-existent) support for HTTP caching in Plone.

It’s now looking pretty long in the tooth, and contains a bunch of design decisions that have proved problematic over time, particularly the heavy use of monkeypatching.

This talk was about the new canonical caching package for Plone, plone.app.caching. It was built by Richard Newbury, based on an architecture from the inimitable Martin Aspeli.

This package is already being used on high-volume sites with good results, and from what I saw here the architecture looks excellent.  It should be easy to configure for the general cases and allows sane extension of software to provide special-purpose caching configuration (which is quite a common requirement).

It provides a basic knob to control caching, where you can select strong, moderate or weak caching.

It can provide support for the two biggest issues in cache engineering: composite views (where a page contains content from multiple sources with different potential caching strategies) and split views (where one page can be seen by varying user groups who cannot be identified entirely from a tuple of URL and headers listed in Vary).

It provides support for nginx, apache, squid and varnish.  Richard recommends you do not use buildout recipes for Varnish, but I think our recipe isotoma.recipe.varnish would be OK, because it is sufficiently configuration.  We have yet to review the default config with plone.app.caching though.

Richard recommended some tools as well:

  • funkload for load testing
  • browsermob for real browsers
  • HttpFox instead of LiveHttpHeaders
  • Firebug, natch
  • remember that hitting refresh and shift-refresh force caches to refresh.  Do not use them while testing!

Jens Klein: Plone is so semantic, isn’t it?

Jens introduced a project he’s been working on called Interactive Knowledge Stack (IKS), funded by the EU.  This project is to provide an open source Java component for Content Management Systems in Europe to help the adoption of Semantic concepts online.  The tool they have produced is called FISE. The name is pronounced like an aussie would say “phase” 😉

FISE provides a RESTful interface to allow a CMS to associate semantic statements with content.  This allows us to say, for example that item X is in Paris, and in addition we can state that Paris is in France.  We can now query for “content in France” and it will know that this content is in France.

The provide a generic Python interface to FISE which is usable from within Plone.  In addition it provides a special index type that integrates with the Plone Catalog to allow for updating the FISE triple store with the information found in the content.  It can provide triples based on hierarchical relationships found in the plone database (page X is-a-child-of folder Y).

Jens would like someone to integrate the Aloha editor into Plone, which would allow much easier control by editors of semantic statements made about the content they are editing.