When we announced raindrop, we had a CouchDB based data-model in place but no API for accessing the data – the front-end Javascript application was forced to issue CouchDB API requests (ie, query views and fetch documents by ID) directly.  This presented us with a few problems:

• The front-end code had to perform a fair bit of ‘data munging’ – for example, given CouchDB has no concept of a ‘join’, multiple requests are often necessary to obtain the required data.
• The front-end code became complicated – instead of focusing on the user-experience, the Javascript developer was forced to deal with low-level couch API calls and how documents and schemas relate to each other.
• The front-end code was exposed to the storage details; if we wanted to change how documents are related or managed, alot of front-end code needed modification.
• The front-end code was processor-intensive; as we move towards Raindrop on mobile platforms, the platform may not have the memory nor processing capabilities to be usable.

We have made a move towards solving these issues by creating a Raindrop REST API layer.  This API layer can expose the data used by Raindrop in a format targeted at the specific requirements of the front-end application, while consuming the resources of a ‘server’ instead of the client.  A few key features of the API implementation are outlined below:

Extensible API

As the API needs to meet the specific requirements of a front-end application, and as Raindrop is intended to be a platform instead of a single application, we have allowed for the application itself to simply define new API end-points.  APIs are implemented using the same basic mechanism as all extensions; a single couchdb document can be introduced, and the API end-points it implements are automatically made available.  This same technique can be used by both the application and its extensions – for example, the ‘inflow’ application will provide a number of entry-points suitable for the core inflow application – but extension authors are free to install both back-end extensions, and new API end-points which target their extension.  For example, a complex raindrop extension could provide a back-end extension to capture additional meta-data about each message, a front-end extension to render things based on this extra meta-data, plus a new API end-point targeted specifically at the front-end extension’s requirements so that extension can push some of its work back to the server.

Leveraging CouchDB

CouchDB provides a powerful facility called ‘externals’.  These externals are defined in terms of operating-system processes (ie, a command-line couchdb executes), and when a request is made to such an external, CouchDB itself manages the spawning of this process and exchanging data with it.  A key benefit of this approach is that raindrop need not implement a “network facing” REST server to expose the API; the normal CouchDB front-end still handles all requests, so the external can focus on the job at hand rather than directly managing outside connections.

Currently CouchDB has a slight scaling issue with these externals (specifically, only one request at a time can be handled; subsequent requests are “queued”), it is expected CouchDB will grow to support “process pooling”, and ideally may even support such pooling across machines to offer serious scalability.

Data suited to the application

The current raindrop implementation defines a series of ‘inflow’ related end-points, and these end-points are all focused on the requirements of this application.  For example, one API end-point can be accessed via http://127.0.0.1:5984/raindrop/_api/inflow/conversations/personal?limit=30.  This end-point is defined as returning the 30 most recent conversations which include messages sent directly to you.  Each conversation object has a list of messages, pre-sorted by time, and with all the schemas which make up each message unified into a single “container” object.  While the format of the result is quite different than the format of data in the database, it is very suitable for the inflow application’s requirements.

In a later post, we will explore the API in more detail, including a look at the request parameters and the objects returned.

Tags: raindrop