A major impediment to working without your password has been accessing your email.  While twitter supports Open Authentication (OAuth), it was impossible to access your email without a password, so support for a password-less raindrop was put on hold pending some answer to this problem.  Fortunately, Google and Yahoo have recently been working on a spec for OAuth-based IMAP authentication, and with GMail providing a working implementation, raindrop could make that leap.

The raindrop front-end now has support for configuring gmail and twitter accounts using OAuth.  During the account configuration process you are directed to Google or Twitter to authorize raindrop to access your account.  You may need to provide your password to Google or Twitter, and after authorizing raindrop, you are directed back to raindrop to complete the account setup.  On this return step, Google or Twitter provides raindrop with OAuth tokens which raindrop stores on the file-system with the account configuration information.  These tokens can then be used in lieu of the username and password when communicating with Twitter or the Gmail IMAP server.

The key reason this matters is that raindrop never needs to know your password – let alone store it – and the user remains in complete control over what operations can be performed by raindrop.  Thus, if raindrop managed to be compromised we might leak enough information for our existing tokens to used temporarily, but at any time the user can visit Twitter or GMail and revoke the tokens making that leaked information useless.  It will be impossible for raindrop to leak your password as raindrop doesn’t know what it is.

The OAuth support for twitter was fairly easy to implement by following the examples found on the twitter API documentation site.  Raindrop uses the Python twitter library, and although this library doesn’t currently support OAuth directly, we are in the process of rectifying this by working with the project on the changes necessary for such support.

The OAuth support for GMail was similarly easy given the nice examples provides by Google.  While OAuth support for IMAP is currently in the early stages of becoming a standard, we hope that the appeal of password-less IMAP login will be such that other major IMAP servers will start to support this in the future – until then, Raindrop will still need to store your password for IMAP servers which do not implement this new functionality.

While raindrop still has work to do in ensuring your private information remains secure, we see OAuth support as a nice step in the right direction.

I don’t want to repeat anything that avid readers have already seen so here are some quick highlights and some news for the future.

Design

If you’ve been reading the Raindrop Design Blog you’d have already seen the latest mockups from Mobile to Attachments.

Be sure to take a look at the latest design blog entry, Raindrop Design Status

Here’s a quick video demo of some of the new interactions guided by our Graphic Designer Andy :

Vimeo

For all of our latest videos check out Mozilla Messaging on Vimeo. Here are a few more of the latest videos:

• File Attachments
• Really Cool Video

Twitter

On Twitter you can now follow @MozillaRaindrop for all the latest screenshots and news about Raindrop design and development. To catch you up, here are some of the tweets you’ve missed.

• #raindrop now pulls out GPS information from EXIF data on images so you can see where that photo was taken $
• Flickr link expanding landed in raindrop, http://i.imgur.com/OY0JL.png How does that look? $
• Vimeo links are now data mined inside raindrop too, check it out: http://i.imgur.com/f4fRg.png (@_andychung) $
• NYT API (@timesopen) getting link info for NYT links in Raindrop, http://i.imgur.com/uWv3w.png maybe @donohoe will get it running now $
• http://vimeo.com/8771552 – Quick video of me clicking through some attached links in #Raindrop $
• http://vimeo.com/8772031 – one more quick video showing file attachments being clicked through. @jrburke is pumping good stuff out today! $
• http://i.imgur.com/ekReU.png – having some fun with our metrics prompt $
• Cool! First link to Raindrop from the nytimes (a blog, but still!) http://bit.ly/5w4CiB $
• Kittens inspired by kittens in #raindrop: http://i.imgur.com/RMqJ9.png$
• Now with super paper clip action! http://i.imgur.com/bJt9u.png (@clarkbw) $

Flickr

Our Raindrop Flickr Design Group continues to be an excellent place for comments and ideas on new mockups posted for fun and for blogging.

Development Discussions

To continue to keep pace with Raindrop Development you’ll want to subscribe to the Raindrop Development mailing list on Google Groups. It’s necessary to at least watch the group if you’re developing along so you know when new code has landed and it’s time to reprocess your development raindrop.

If you’re not interested in the development discussion or latest code to index links from the new york times then you’ll at least want to watch for Jame’s weekly design and developer emails to keep pace with what’s happening without following the source code.

For more in depth development discussion, check out Mark’s latest entry Dynamic dependencies in the raindrop data-model

2010

We have several goals for the coming year, I’ll list them briefly, check back here on on the Raindrop wiki if you’re looking for more details and specific milestones of when these will come out.

1. Polish, clean up our existing experiences to get ready for more
2. Clarity, better conversation layout and attachment support
3. Find, a simple, new, and innovative way to provide search that actually works
4. Actions, quick and easy ways to process and handle your conversations
5. Relationships, real and unique communication relationship assistance

Can we say “One more thing…” ?

A hosted preview version of Raindrop targeting availability for alpha testers by the end of Q2.

More details to follow as we build out a hosted version and of course invitations quickly there after. Stay tuned.

And if you’re wondering how we keep up with all these different channels, we use Raindrop

To support this requirement, we have added new meta-data to each raindrop schema – a field called rd_deps.  This item is an array of tuples, which each tuple holding an rd_key and a schema ID.  In our example above, the rd_deps on the message object would list the rd_key of the sender, and the schema ID which holds the attribute being examined.  There can be any number of rd_deps entries for an item.  As raindrop processes the work queue, it checks to see if the document being processed is listed in the rd_deps for any items and if so, all these items are re-processed as if they themselves have changed, thereby picking up the new values.

We currently use this dependency model in two places:

Moving a message to “bulk”:

Raindrop now supports the ability to nominate a particular sender of a message (eg, an email address) as being a “bulk sender”.  When a message from a bulk sender is received, it is always marked as a “bulk” message, regardless of how the message itself is addressed.  This means that if a particular sender is marked as being a bulk sender after messages have been classified, all messages from that sender must be re-processed so the message is correctly classified with the new attribute.  This is done by marking the document with this classification as depending on the identity sending the message.  Thus, whenever the attribute on an email address is changed, all messages with that dependency are re-run.  Note that this works even though the “bulk” attribute on the email address may not yet exist; it may be created in the future.

Conversation summaries:

A conversation summary is a schema which summarizes the messages in a conversation to apply a number of classifications to the conversation as a whole.  For example, while each message has an attribute to indicate whether the message is a “personal” message, we also need to classify the conversation itself as being personal or non-personal based on the attributes of the individual messages.  Therefore, the conversation summary schema lists as dependencies the schema holding the “personal” attribute of each message.  In other words, there will be as many dependencies on the conversation summary schema as there are messages in the conversation.  If the ‘personal’ attribute on any message in the conversation changes, the conversation summary extension is re-run.

You may notice some potential “chaining” between these 2 examples; if the sender of an email is marked as a bulk sender, the classification for all messages sent by that user will be re-calculated.  This message re-calculation will then independently cause the conversation summary extension to be re-executed as one of its dependencies (the attribute on the message) changed.

Note that these dependencies must be explicitly nominated by the extension author – only they know what was used in the determination of the values.  Thus, the extension which classifies the message must explicitly list the key for the sender, while the conversation summary extension must explicitly list the keys of the messages in the conversation.  See the rd.ext.core.convo-to-summary and rd.ext.core.msg-email-to-recip-target extensions for examples.

Since launching, we started a server-side API, and converted the Inflow experiment to a third design iteration. The basic design is really firming up now. We have set the basic structure, and now we are crafting each component as part of that structure.

Here is how Inflow looks today. This is a picture of actual working code from the latest Raindrop source. It is a picture of a mailing list view:

Note how the Summary area at the top of the second column is used to show information about the list and actions you can take related to the list. If the Unsubscribe button is pressed, then Raindrop will do all the email work to unsubscribe you, and if you keep this view open, the status will change to “unsubscribe pending” then eventually to “unsubscribed”.

This view highlights where we want to take the Summary area — showing specific information about the list of conversations you are viewing, and certain actions you can take related to the list of conversations or the context around them. Andy did a person-focused conversation view mock-up that also demonstrates this concept.

Since launching, we have focused on designing an “emotional sense of relief”. We have been using the code as a way to try out those designs.

As a result, making Raindrop easy to install or run has not been a priority. The  foundations were still being established, and we wanted an experience that was more useful than the 0.1 launch version before tackling ease of installation.

In 2009 we made great progress on the foundations and design. Expect to see an even more useful Raindrop in 2010, along with an easier way to try it out. Our Milestone Planning page lists out the three major goals we are pursuing in 2010:

• Emotional feeling of better organization: Make sure the experience gives the user a sense of relief via easy setup and automatic filtering/grouping done for the user.
• Dog Food: The inflow app should be usable by all the team members on a day-to-day basis.
• Cloud enabled: Raindrop is runnable in the cloud, with clear instructions on how to set it up to make it secure.

The Milestone Planning page lists out our milestones and current work, so check it out periodically to track our progress. If you see something on that list that really interests you, get involved in the community and contribute!

• The front-end code had to perform a fair bit of ‘data munging’ – for example, given CouchDB has no concept of a ‘join’, multiple requests are often necessary to obtain the required data.
• The front-end code became complicated – instead of focusing on the user-experience, the Javascript developer was forced to deal with low-level couch API calls and how documents and schemas relate to each other.
• The front-end code was exposed to the storage details; if we wanted to change how documents are related or managed, alot of front-end code needed modification.
• The front-end code was processor-intensive; as we move towards Raindrop on mobile platforms, the platform may not have the memory nor processing capabilities to be usable.

We have made a move towards solving these issues by creating a Raindrop REST API layer.  This API layer can expose the data used by Raindrop in a format targeted at the specific requirements of the front-end application, while consuming the resources of a ‘server’ instead of the client.  A few key features of the API implementation are outlined below:

Extensible API

As the API needs to meet the specific requirements of a front-end application, and as Raindrop is intended to be a platform instead of a single application, we have allowed for the application itself to simply define new API end-points.  APIs are implemented using the same basic mechanism as all extensions; a single couchdb document can be introduced, and the API end-points it implements are automatically made available.  This same technique can be used by both the application and its extensions – for example, the ‘inflow’ application will provide a number of entry-points suitable for the core inflow application – but extension authors are free to install both back-end extensions, and new API end-points which target their extension.  For example, a complex raindrop extension could provide a back-end extension to capture additional meta-data about each message, a front-end extension to render things based on this extra meta-data, plus a new API end-point targeted specifically at the front-end extension’s requirements so that extension can push some of its work back to the server.

Leveraging CouchDB

CouchDB provides a powerful facility called ‘externals’.  These externals are defined in terms of operating-system processes (ie, a command-line couchdb executes), and when a request is made to such an external, CouchDB itself manages the spawning of this process and exchanging data with it.  A key benefit of this approach is that raindrop need not implement a “network facing” REST server to expose the API; the normal CouchDB front-end still handles all requests, so the external can focus on the job at hand rather than directly managing outside connections.

Currently CouchDB has a slight scaling issue with these externals (specifically, only one request at a time can be handled; subsequent requests are “queued”), it is expected CouchDB will grow to support “process pooling”, and ideally may even support such pooling across machines to offer serious scalability.

Data suited to the application

The current raindrop implementation defines a series of ‘inflow’ related end-points, and these end-points are all focused on the requirements of this application.  For example, one API end-point can be accessed via http://127.0.0.1:5984/raindrop/_api/inflow/conversations/personal?limit=30.  This end-point is defined as returning the 30 most recent conversations which include messages sent directly to you.  Each conversation object has a list of messages, pre-sorted by time, and with all the schemas which make up each message unified into a single “container” object.  While the format of the result is quite different than the format of data in the database, it is very suitable for the inflow application’s requirements.

In a later post, we will explore the API in more detail, including a look at the request parameters and the objects returned.

Read the original blog post for all the details.

A couple days ago I began creating the first iterations for the Raindrop mobile web app. I’ve posted an article on these first directions up on the Raindrop design page, as well as all of the images on the Raindrop flickr group. Check them out and let us know what you think!

We hope to lead and spur the development of extensible applications that help users easily and enjoyably manage their conversations, notifications, and messages across a variety of online services. A central principle behind Raindrop is that messaging should be personal — we want Raindrop to be people-centric both in how we process messages, and in how we can help give people control over their personal data and experiences.

When a friend’s link from YouTube or flickr arrives, your messaging client should be able to show the video or photos near or as part of the message, rather than rudely kicking you over to a separate browser tab. Notifications from computers and mailing lists should be organized for you, not clutter your Inbox or require tedious manual filter setup. It should be easy to smoothly integrate new web services into your conversation viewer entirely using open web technologies.

Raindrop 0.1: A Prototype

To this end, we’ve started Raindrop, which we hope will become both a customizable product and a platform for a variety of innovative messaging applications on the Web.  We’ve also drafted a set of principles that we believe will help ensure that it is as useful, usable, and compelling as possible. Today, we’re releasing version 0.1, which is very much an iterating prototype, not yet ready for everyday use. Join us and help drive it there!

Design

What does a conversation on today’s web look like? Email used to house the bulk of the conversations that took place on the internet, but that’s no longer the case today. In today’s world people use a combination of Twitter, IM, Skype, Facebook, Google Docs, Email, etc. to communicate. For many of us this means that we have to keep an eye on an ever-growing number of places we might get new messages. As a result, we never know that we’ve actually processed all the important messages, because our email has been overwhelmed by noise which obscures the real messages from real people.

Raindrop is an effort that starts by trying to understand today’s web of conversations, and aims to design an interface that helps people get a handle on their digital world. At the same time, it creates a programming interface (API) that helps designers and developers extend our work and create new systems on top of that data. We aren’t trying to invent new protocols or build new messaging systems, rather focusing on building a product that lets users get a handle on the systems we already use.

The Raindrop team has already gone through 2 initial iterations which demonstrate the power of the platform we already have and the possibilities that lay ahead.

The following video discusses some of the fundamental ideas in today’s Raindrop.

First iteration of the Raindrop user interface

Second iteration of the Raindrop design

Development Platform

We intend to support front-end applications of various kinds (including mobile), but our flagship applications will be built entirely for any modern web browser that supports Open Web technologies.  Version 0.1 of our prototype embeds Bespin to support a fast, iterative development style.  It also provides front-end widgets and back-end code that supports important high-level concepts such as people, conversations, and mailing lists, with more to come.  CouchDB and Python are key parts of our prototype architecture as well.

What’s Next

We’re just getting started and expect Raindrop to evolve quickly.

One of our first priorities is to make downloadable installers or setup a hosted installation that anyone can use to try things out — making it easier for non-developers to check it out.

Because Raindrop is very much iterative and design-driven, another priority is to implement a default configuration that asks (but does not require!) each user to contribute anonymized usage data back to the Mozilla design community in a way that respects your privacy.

We’re tracking other priorities in our evolving roadmap.

Get Involved

Mozilla Labs is a virtual lab where people come together online to create, experiment and play with Web innovations for the public benefit. The Raindrop exploration is still in its infancy and just getting started.

All of the source code underlying the Raindrop exploration & prototype is being released as open source software under the the MPL.

There are many ways to join the team and get involved:

• Ask questions and contribute ideas at our GetSatisfaction community.
• Upload a mock-up or design to the Raindrop Design group on Flickr.
• Pull the source code and start hacking.
• Join us in our chat room.

We’ll also be using the #raindrop hashtag on twitter.  Our community page has more details about these and other ways to connect.

– Andy Chung, Bryan Clark, Dan Mosedale, David Ascher, Mark Hammond, and James Burke on behalf of the Raindrop development team