Last July, the Mozilla Labs team launched Test Pilot into Firefox 4 beta via the Feedback channel. Over the past eight month, with your support, we’ve rolled out several valuable studies and gained many insights that have helped improve Firefox. Meanwhile, our Test Pilot community is growing quickly – 3 million active users milestone! We want to continue delivering the best experience to the millions who help us improve Firefox, so over the past few weeks we’ve been working hard to update the extension with some important new features.

What’s New?

1.Targeted Study Deployment
Often times, we are interested in questions that apply to just a subset of our Test Pilot population; for example, the subset of Windows 7 users, or users with a particular non-default setting. With Targeted Study Deployment, we can now run studies on just the relevant subsets of users. Technically, these studies will be delivered to all Test Pilots, but will only run on profiles that match the pre-defined criteria, for example, computer operating system, Firefox version or preference settings.

2.Random Sample Deployment
Similar to Targeted Study Deployment, Random Sample Deployment gives us the ability to run studies on a random sample of our users. With over 3 million users, our studies can achieve statistical significance without covering the entire population. On the user side, Test Pilot will less frequently interrupt browsing with notifications, since users will no longer receive every study. (We are also working on a solution for those that would like it participate in all Test Pilot Studies.)

3. Stop Switch
This function offers the ability to turn off a study immediately in case there are any critical issues. Fortunately, we’ve yet to encounter any security issues, and we will continue to thoroughly review each study before its release. This Stop Switch feature provides an extra layer of protection on top of our review process.

4. Error Reporting
In the past we’ve relied on the Test Pilot discussion group to discover study or extension bugs. Now when you submit your study data to our server, it will also include a log of any errors that occurred. This will help us to debug and improve the study for the future.

5. UI Bug Fix
We now have also fixed a couple Test Pilot notification window bugs, including the transparent background bug that was discovered by several people. Thanks to Archaeopteryx [:aryx], Eitan Adler, Tim (fmdeveloper), Ryan Dunlop and Alexandr Lookoshkoff for filing the bug and sending us the screenshot!

What’s the next

1. Test Pilot for Mobile
Yes, we will work with Mozilla Mobile team to understand how Firefox on Mobile is used in the near future. It would also be fun to understand how Firefox are used differently between desktop and mobile devices. Try Firefox on your Android phone or Nokia N900 now and we are open to your feedback!

2. 2. Localization underway
Localizing each study for different languages is a goal for this year. We plan to solicit help from the localization and web development teams that localized our Mozilla web sites – if you want to join the effort, please let us know!

Test Pilot is a platform for studying Mozilla internal products, but more importantly it is an opportunity for the web community as a whole to evaluate ideas. It allows non-Mozilla projects to develop their own studies that aim to benefit the community and improve the Web.

Recently, we (the Web Security group at Carnegie Mellon University) released the first third-party Test Pilot study – Evaluation of Proposed Security Standard – in hopes of evaluating the compatibility of Entry Isolation, a newly-proposed browser security policy. The result of our study could help browser vendors and website developers decide whether this new policy is suitable for adoption.

Our study

The study we released on Test Pilot is a part of our ongoing research on securing websites and applications from malicious web entities. The security policy we are proposing allows web applications to specify a list of URLs that third-party sites can link to. When a third-party website requests a sensitive URL that is not on the list, the request will be blocked.

This whitelist-based security policy relies on participating websites having distinctive landing pages, which makes it impractical for socially integrated sites (e.g., Facebook). As such, we want to find out exactly what type of websites may benefit from this security policy, and its practicality.

In our Test Pilot study, we simulated a prototype of our security policy on ten websites. For each site, we whitelisted only the homepage, and recorded every time we ran into an incompatibility issue. A more detailed description can be found here.

This study ran for three days, and gathered data from more than 230,000 unique users. We were extremely grateful to the many users who participated, and Test Pilot for enabling us to collect such a large data set.  The sheer amount of data would have been practically impossible to collect with any other test platform.

Our Results

Initially, we believed security-sensitive sites such as Gmail or online banking sites would have an easy time adopting our Entry Isolation security policy. We did not expect many third-party sites to link deeply into these sensitive sites. Contrary to our beliefs, we found that security-sensitive sites get linked almost as frequently as other sites (e.g. CNN and New York Times).  Most of these third-party links differed from our selected landing pages, triggering violations in our policy.

We believe that for security-sensitive sites, most links are simply to different landing pages, rather than to content deep in the website like for other websites. As such, we plan to release a new version of our study soon, adding alternate landing pages into the policy in hopes of mitigating these compatibility issues.

Reflections

Since our study was the first third-party study released by Test Pilot, we have been working very closely with the Test Pilot team to make sure we provide a pleasant experience for all users who join our studies. Unfortunately, due to our lack of experience with the platform, we may have missed a few bugs and caused inconvenience for some users. For this, we offer our sincere apology. Test Pilot is maturing steadily, and we plan to improve our study release process to ensure the final version will pass all necessary tests and reviews. We are open to suggestions if anyone wishes to help.

We would like to highlight again that both our Web Security group and the Test Pilot team will always abide by the following principles:

• Respect user privacy: no personally identifiable information will be collected.
• Give users full control of their study: they have the freedom to cancel the study at any time, or choose not to submit the data.

Get involved

Here are some ways that you can get involved with our study or Test Pilot in general:

1. Help review the studies before they get released.
2. Learn more about Test Pilot
3. Develop your own studies.

Back in November, Mozilla Labs and the Metrics Team together launched the first Mozilla Open Data Visualization Competition. While we set out to discover creative visual answers to the open question, “How do people use Firefox,” we really didn’t know what level of participation to expect from the Mozilla and data analysis communities. In fact, we were overwhelmed by both the number and quality of submissions – so much so that we had to give ourselves an extra few days to thoroughly review them all!

In all, we received 32 high-caliber submissions. The “visualizations’ took a number of forms, from tools to easily query the data to interactive web applications. They also covered a broad range of important topics, from plugin memory consumption to user web activities. You can find all 32 submissions here; entrants, if you haven’t already, be sure to check out the page as our panel of judges has left feedback on each and every submission.

Needless to say, we want to thank all the participants – your work has made our initial open data competition an overwhelming success and many of your insights will directly help the Firefox team develop a better web browser. In thanks, we’ll be sending this awesome Firefox t-shirt to each entrant:

We also want to thank our 3 partner judges: David Smith, Revolution Analytics; Andrew Vande Moere, Information Aesthetics; and Brian Suda, author of A Practical Guide to Designing with Data. The success of the competition was largely due to your help in publicizing the event and thoroughly evaluating the entries.

And now…lets get to the winners!

Grand Prize

Survey Participants vs. All Users – Contributed by: James Fiedler

While deciding amongst the 32 entries was difficult, the focus on a single, very relevant and important question distinguished this entry. James focused on contrasting survey participants with all users (critical as we often use survey data for segmentation), then set up a simple and helpful environment for the user to explore and discover interesting conclusions of their own. This submission is exactly the type of work we were hoping for: an elegant visualization that presents data around an important and complex question in a clear and easy-to-understand way. James will receive a $300 Amazon gift card for his excellent work.

Finalists

Test Pilot Explorer – Contributed by: Lon Riesberg

One of the more creative entries, Lon created a custom “explorer” that essentially “plays back” time-ordered events as animated plots and includes filters to customize what data is shown. This explorer really shows how you can “see” user behavior on a mass scale, and while we had some quibbles about some of the details of the visualization itself, we found it to be a powerful and enjoyable data exploration tool. Lon will receive a set of all 4 Edward Tufte books for his work.

Firefox Usage by Age – Contributed by: Tom Haynes (University of Michigan)

Tom’s entry also focused on one particular element of the data. His execution sets this submission apart, as his visualization doesn’t try to encompass everything, but tells a clear, specific story around how Firefox usage times vary across age groups. Tom will receive a set of all 4 Edward Tufte books for his work.

Honorable Mention

Given the number of worthy submissions, we decided to hand out 5 Honorable Mention Awards in addition to the original 3 prizes. For varying reasons, we thought these entries were particularly valuable and each team will receive Tufte’s latest book, Beautiful Evidence, in recognition of their great work. Good Job!

Firefox browser – Event Sequences – Contributed by: Benoît Pointet

Firefox 4 beta UI Component Use vs. User Expertise – Contributed by: Nicolas Garcia Belmonte, Maria Luz Caballero

Browser Usage Over the Course of a Day Contributed by: Christian Kreibich

Bookmark a Lot, Browse a Lot – Contributed by: Eugene Tjoa

Firefox Plugin Memory Consumption- Contributed by: Diederik van Liere and David Eaves

Again, thanks to all the participants, judges, and everyone else who helped make this first open data competition such a success! Participants should receive an email within the week with details on how to receive the prizes and t-shirts.

And keep refining those data hacking skills – there will be more open data competitions in the near future!

We want to give a short update on the announcement of the winners of our first Mozilla Open Data Visualization Competition. We originally planned to announce these winners tomorrow, but will now make the announcement on Tuesday, January 11th.

Dont worry, we do have a decent reason for this postponement: the number and quality of submissions far exceeded our expectations. The depth of many of the analyses also caught us off guard, and we really wanted our judges to have ample time to not only evaluate, but also give feedback on each and every entry.

Needless to say, we are extremely excited about the level of participation in our first open data competition, and are even more excited to share the results with the community on Tuesday!

We are also happy to announce a third, special partner judge, Brian Suda. Brian is an informatician currently residing in Reykjavík, Iceland and author of the book, A Practical Guide to Designing with Data. Brian and our other distinguished partners, David Smith of Revolution Analytics and Andrew Vande Moere of Information Aesthetics, all look forward to seeing your submissions!

Good Luck!

Hello Data Hackers!

We just wanted to remind everyone that the submission deadline for the first Mozilla Open Data Visualization Competition is just 10 days away! Submit your entries by December 17th for a chance at a $300 Amazon gift card and a set of all 4 Edward Tufte books!

We’ve already received some great entries, and our panel of expert judges (Kevin Fox and Jinghua Zhang, Mozilla Labs; Hamilton Ulmer, Chris Jung and Blake Cutler, Mozilla Metrics) along with our partner judges (David Smith, Revolution Analytics; Andrew Vande Moere, Information Aesthetics) look forward to seeing the rest of the submissions!

Remember to visit the Official Competition Page for all the information you need, including how to download the data and enter the competition.

Good Luck!

This course was conducted by Martin Tomitsch from the Design Lab at the Faculty of Architecture Design and Planning in collaboration with Mozilla Labs. Students enrolled in the Design Computing program and with varied interests and backgrounds, including media & art, interaction design and computer science joined the course.

Course Outline

The course was designed to be shared with the Open Source community at the end. In the 2-month course period, students planed their own user research, explored users goals and current problems, created designs based on these findings and tested their designs using prototypes. It contained three phases:

Phase one: Discovering (unarticulated) user needs
- Conduct field studies with at least 3 users
- Mozilla Guest Lecture: The value of early stage user research in real work
- Mozilla Labs provides the user interview consent form

Phase two: Product design based on identified user needs
- Brainstorm ideas on user needs, create scenarios or storyboards, subsequently develop a meaningful idea into a paper prototype, conduct user testing, revise the concept and create an interactive prototype (using HTML, CSS and JavaScript), continue testing product
- Mozilla Guest Lecture: Applying leading technology in your prototype
- Mozilla mentor on user testing techniques and interactive prototyping

Phase three: Documentation
– Contents will be shared to the public under a Creative Commons license

Designing for Social

While the idea behind the course are to encourage open design and research, the topic for the first case also needed to be open enough for the wider community to be inspired and useful. By talking to several Open Source projects, the wider concept of “social” came up again and again. Luckily, students are familiar with this topic too.

Exploring User Needs

After 12 weeks of hard work, several insightful findings on social behaviors and aspirations were identified by students. Here are some social user persons* that are created by students:

• A popular teen Talisha who likes to be kept ‘in the loop’ on social networking sites and gets frustrated if someone in her group knows something she doesn’t.
• A 17 year old high school girl who grows up in the digital world, does most of her studies and communicates with teachers online.
• A boy called Andy who wants to act differently in different social groups, and sets clear boundaries to separate each group.
• A school girl Gillian who started to procrastinate after spending too much time on social media sites and feels pressured by social invites.
• A college student Marko Knight who uses various social channels to keep up with all kinds of friends and feels the pain of managing too many platforms.
• A 22-year-old girl Jess who uses social media as a main way to share her experience of horse riding and learn from other horse lovers.
• Julia, a 26 year old Web designer who uses social media as a main way to promote her work.
• A 27 year old teacher Rose Tyler who uses social media as a major information resource to prepare her class contents on commerce, geography and economics.
• A 40-year-old mother of two, Catherine Wakeman, who learns how to use social media sites to connect with her kids.
• Henry, a professional architect, who uses social media sites to maintain relationship with clients but feels uncomfortable to know too much about others, including his 19 year old son.
• A Chinese girl Cheng who studies abroad and misses the real human emotional connection that cannot be fully conveyed via online media.

*A user persona is a representation of the goals and behavior of a real group of users. In most cases, personas are synthesized from data collected from interviews with users. They are captured in 1–2 page descriptions that include behavior patterns, goals, skills, attitudes, and environment, with a few fictional personal details to make the persona a realistic character.

Brainstorm Ideas

Students then went on to brainstorm solutions for the identified use cases through storyboards, just to highlight a few:

Final Concepts

At the end, students developed the final concept for their solution. Here are videos demonstrations of how their concept work:

• SN Meister by Mark Fonacier and Renato Gaylican
• CAN toolbar by Alan Tran and Ben Alcantara
• ShowBar 1,2 by Benjamin Cha and Jin Zi Ho
• GroupShare by Adriano DiPalma and Morgan Carter
• FaceGROUPS by Betinho (BJ) Soares and Nick Dsouza
• Bounce by Hanley Weng and Nadine Denten
• Smart Wheel by Danielle Yu and Rory McPherson
• Team H by Dale Hurwitz and Tobias Reid
• SharePad by Connie Yan and Kathy Qiu
• Sweets by Catherine Park and Eva Hung
• The Procrastinator by Adrian Yoong and Arthur Jing
• ClickSwitch by Laura Minchella and Roxanne Phan
• Social Cloud by James Dumesny and Selhan Haksoz
• Lexicon by Harry Mann and Stella Kim
• Keep me in Touch by Hyun Jin Celia Kim and Shayon Jiang

A full list of all the amazing work from the participating students, including user research findings, ideas, user testing and final working prototypes is available at the class page.

Taking it forward

Supporting this course is a part of Mozilla Labs’ Open Design and Research initiative. We pilot this initiative by providing guest lectures, introducing the latest Web technologies to interested students and providing interesting projects to the classroom, as well as bringing professional UX practitioners to class with the aim to to expose, highlight and mentor students and their artifacts.

Currently local UX groups, including IXDA Australia, UPA Australia and individual professionals assist this project by giving exposures to students work, providing feedback or organizing related UX events.

• Steve Baty, the Vice President of IxDA and the principal at Meld Studios
• Lisa Herrod, President at Usability Professionals Association, Sydney
• Penny Hagen, Design Strategist at smallfire
• Grant Young, Innovation Strategist at Zumio
• Michelle Gilmore, Co-Director at Neoteny Service Design

As the course is completed, Mozilla Labs is also helping contact Open Source teams that are interested in this domain, share these valuable artifacts with the community and trigger more discussions on user needs and creative solutions.

Since not only the product concepts created in the course are open for any Open Source project to take forward, but also the user research findings are shared with the public, we encourage everyone to join forces creating further discussion and ideas, as well as help transfer any insightful and interesting concepts further into real products.

Get Involved

- Get inspired by students work on the class page of the faculty or follow along with the lecture notes when they become available.
- Contact Martin Tomitsch or Jinghua Zhang to learn more about this course.
- Contact Jinghua Zhang (Jinghua [at] mozilla [dot] com) if you are interested to collaborate on similar student projects with Mozilla Labs.

All Test Pilot studies, resulting insights and aggregated data are shared under Creative Commons licenses for the benefit of the Open Source community. The results have triggered some interesting discussions within the wider community. For example, people did their own analysis of the tab data from our first study as well as use that data to assist other research topics.

When Firefox 4 Beta was released In July 2010, a version of Test Pilot called “Feedback” was included in order to better understand how the browser is used by beta testers. Firefox 4 Beta testers don’t have to download the Test Pilot add-on and still have the choice to submit their data or opt out from a particular study as well as the whole program. Being included in Firefox 4 Beta helped Test Pilot recently cross the 1 million user milestone.

In the last year, we released 12 studies including log studies and surveys. The most recent study is a Firefox 4 Beta UI study, where we learned how people use interface elements through an interactive heatmap; this heatmap also allows us to compare usage patterns with a previously created Firefox 3.6 heatmap.

We couldn’t gain all this knowledge without the help of our Test Pilot participants. They not only contribute their usage data via focused studies, and also share unique use cases to help us design studies better, provide new study ideas, give corresponding UI suggestions, report bugs, and even fix programming syntax errors. We really appreciate all your help!

Moving forward

We accumulate tremendous data sets with Test Pilot that are valuable to the entire Open Source community. In order to share this knowledge with the community, we released the first Mozilla Open Data Analysis Competition. The goal is to encourage people to make sense of data. Please let us know (via email: jinghua[at]mozilla[dot]com) if you are interested to collaborate with us on it.

Get Involved

Besides calling participants for the Data Analysis Competition, we are also looking for people to help us design and implement user studies. If you have an interesting study idea or want to try other research methods on the Test Pilot platform, please submit your proposal or let us know in the discussion forum.

• Tell your friends about Test Pilot!
• Download Firefox 4 Beta to enroll in the Test Pilot crew.
• Check previous and upcoming Test Pilot studies and data analysis.
• Let us know your general feedback on Test Pilot in the discussion forum.
• Join the Mozilla Open Data Analysis Competition!

Two weeks ago Mozilla Labs, together with Mozilla Metrics team and the growing Mozilla Research Initiative, announced our first Open Data Visualization Competition. Today, we are excited to release the data sets for this competition!

These data sets come from Mozilla’s own open data program, Test Pilot. Test Pilot is a user research platform that collects structured user data through Firefox. Currently, over 1 million Firefox users from all over the world participate in Test Pilot studies, which aim to explore how people use their web browser and the Internet in general.

For this challenge, data samples from two recent Test Pilot studies have been made available – check out the data sample pages below for a thorough description:

• A Week in the Life of a Browser – Version 2
• Firefox 4 Beta Interface – Version 2

In addition, please note that the submission deadline has been extended from Dec. 5 to Dec. 17th to encourage more participation. Be sure to visit the official Competition page for more general info on dates, judges, and prizes.

Good luck and start hacking!

Mozilla Labs and the Mozilla Metrics Team, together with the growing Mozilla Research initiative, are hosting a Open Data Visualization Competition based on Test Pilot data.

For this competition we’d like to explore creative visual answers to the question: “How do people use Firefox?” For example, visualizations that investigate general usage patterns, reveal interesting user behavior, or explore browser performance. We are looking for compelling visualizations that tell detailed, meaningful and yet easy-to-interpret stories about interesting user activities.

The Data

This competition is based on Mozilla’s own open data program, Test Pilot. Test Pliot is a user research platform that collecting structured user data through Firefox. All data is gathered through pre-defined Test Pilot studies which aim to explore how people use their web browser and the Internet.

Currently, over 1 million Firefox users from all over the world participate in Test Pilot studies. The goal for this platform is to encourage everyone from all skill levels to improve the Web experience by conducting and participating in these studies. Test Pilot study results are made available under open licenses, with the data being anonymized before release. (For more information about the Test Pilot data policy, please check Privacy Policy.)

For this challenge, we will use data from two recent Test Pilot studies:
* A Week in the Life of a Browser – v2
* Firefox 4 Beta Interface – Round 2

Partners and Judges

We are honored to have David Smith from Revolution Analytics partnering with us and serving as a special judge. Members from Mozilla Labs and Mozilla Metrics will form the rest of the judging panel.

Prizes

To recognize the awesome work from participants, some great prizes are at stake:
* Grand prize will be a $300 Amazon gift card
* Two “Best in Class” teams will receive a set of Edward Tufte’s books

We’ll also present all submissions on the Test Pilot website and Mozilla Metrics blog in a special post to highlight your work.

Join the competition!

You can choose any tools you like for your analysis and visualization, including but not limited to: R, Matlab, Protovis, Processing or IBM many eyes. You can participate solo or team up with other people. You are also welcome to enter as many times as you like. If you are interested to join the competition, here are the important dates:

• Nov. 17th: Check the official Competition page on Nov. 17th to download the data
• Dec.5th : Go here to submit your results and enter the competition before Dec.5th.
• Dec. 14th: Winning visualizations will be announced on Dec. 14th.

To facilitate the free exchange of ideas, all visualizations and other contributions you make to this challenge must be contributed under the Creative Commons Attribution-ShareAlike 3.0 license.

Please read the official Competition page for more details. Also follow @moztestpilot on Twitter for news and updates.