Test Pilot is an idea for a new user testing program for Mozilla Labs that aims to build a 1% representative sample of the Firefox user base for soliciting wide participation and structured feedback for Labs experiments.
Presently, whenever Mozilla publishes a new experiment we rely upon the blogosphere and word-of-mouth to attract the attention of people interested in trying it out. We then hope that some number of those early adopters will come back and find the appropriate place to provide feedback.
The problem is that we are systemically biasing feedback towards only those who happen to hear about a particular experiment (on a given day) and in those cases skew to the technically-savvy early adopter. It is therefore very hard to derive conclusions representative of our much larger and increasingly mainstream user base.
We also have no structured way in which to request feedback, and in particular, positive feedback. People are much more likely to take the time to dig in and figure out how to provide feedback if they have a negative reaction. It’s also likely true that the less technical adopters are confused when faced with Mozilla’s panoply of feedback options.
And finally, at the end of this “testing process,” we effectively throw away the interested people who have taken the time to volunteer as testers and start over again from scratch the next time around.
Overview of Test Pilot
We can provide a much more satisfying experience all around by putting in place some basic infrastructure. Here’s the idea:
- We develop and promote a formal Test Pilot program with a Firefox add-on at its core.
- The first time the Test Pilot add-on is run, it asks a few simple non-personally-identifiable questions in order to put the user into a demographic bucket, e.g. technical level, locale, etc., and to let them opt in to additional anonymous instrumentation.
- Test Pilot will then notify its users when a new experiment is available for testing. If the user opts in, it will download the required software (if any) and load any information required to get started with the new experiment, e.g. overview, use cases, etc.
- After either a specified amount of time or upon completion of a specific action, Test Pilot will prompt the user for feedback. The feedback form will only ask a few questions selected from a much larger set. A link will be provided to provided more comprehensive unstructured feedback or bug reports.
- The set of questions posed for feedback will be randomly distributed within each demographic bucket to ensure statistical significance of the results.
- Anonymized aggregate results and analysis will then be posted automatically to the Test Pilot site.
- All participants will receive a “flight badge” displayed in their Test Pilot profile and available to embed on blogs, social networks, etc.
The idea is that by reducing the amount of required feedback to only a few clicks we can increase overall rates of participation.
While this will not take the place of our traditional testing and feedback mechanisms, Test Pilot should provide a new set of tools and capabilities to increase the efficiency and effectiveness of our user testing. It should also allow us to grow our community and mindshare with people who want to help but to whom we haven’t yet given the opportunity to participate.
Roadmap
We’re pulling together a basic framework now with plans to launch the first version within the next few weeks. Specific features and roadmap are being developed in the Test Pilot discussion forum on the Mozilla Labs site.
Getting Involved
- Discuss, debate, and add to the design in the Labs discussion forum
- Jump into #labs on irc.mozilla.org
Tags: testpilot
Introducing Test Pilot…
Test Pilot is an idea for a new user testing program for Mozilla Labs that aims to build a 1% representative sample of the Firefox user base for soliciting wide participation and structured feedback for Labs experiments….
The image of the Firefox Test Pilot, on the top of the post, is available in a bigger size or in vectorial version ? It’s Free, Open Source, Creative Commons or closed ?
Thanks!
Luca
Love the idea,
Would defenitely participate when the testpilot plugin is available.
Keep up the good work!
[...] (reference) (more) I want to be a Mozilla Test Pilot! http://labs.mozilla.com/2008/03/introducing-test-p…; Filed under Uncategorized [...]
[...] (reference) (more) I want to be a Mozilla Test Pilot! http://labs.mozilla.com/2008/03/introducing-test-p…; Filed under Uncategorized [...]
Introducing? Well where is the download link?
[...] Blog Archive » Introducing Test Pilot (reference) (more) I want to be a Mozilla Test Pilot! labs.mozilla.com/2008/03/introducing-test-pilot/ Filed under Uncategorized [...]
[...] Blog Archive » Introducing Test Pilot (reference) (more) I want to be a Mozilla Test Pilot! labs.mozilla.com/2008/03/introducing-test-pilot/ Filed under Uncategorized [...]
[...] Blog Archive » Introducing Test Pilot (reference) (more) I want to be a Mozilla Test Pilot! labs.mozilla.com/2008/03/introducing-test-pilot/ Filed under Uncategorized [...]
[...] Blog Archive » Introducing Test Pilot (reference) (more) I want to be a Mozilla Test Pilot! labs.mozilla.com/2008/03/introducing-test-pilot/ Filed under Uncategorized [...]
[...] site of the National Civil Rights Museum. Todahttp://www.centredaily.com/business/story/496981.htmlMozilla wants unbiased testing reports, creates Test Pilot"Test Pilot is an idea for a new user testing program for Mozilla Labs that aims to build a 1% [...]
Sounds like a great idea!
But surely, if you don’t somehow include it randomly in the Firefox installs (like talkback used to be included) then you will still only be getting geeks involved?
[...] gente de Mozilla Labs está perfilando una idea, bajo el nombre de Test Pilot, que tiene como fin obtener retroalimentación -selectiva- de una muestra representativa del 1% de [...]
I would love to participate. I have plenty of time to play on ‘this new fangled machine’. Currently running FF 3.0 B5 with Weave 0.1.xx on MS Vista Home Prem.
Let me know ASAP it is ready to dl.
I’ll participate in this when it’s ready!
I have Ubuntu 7.10 (will install 8.04 when out of beta) and MS Vista Home Premium and are using both FF2 and FF3b5.
[...] Test Pilot (pilote d’essai, en français) est une extension destinée à permettre aux utilisateurs de Firefox 3 de donner leur avis sur différentes fonctionnalités du logiciel. [...]
[...] used to improve the user interface of Firefox, and another is a project in Mozilla Labs called Test Pilot. They’re both early and very limited in [...]
[...] helfen soll, das User-Interface des Browsers zu verbessern. Das andere hat den Namen Test Pilot und soll eine repräsentative Gruppe von Firefox-Nutzern zusammenbringen, die den Entwicklern [...]
[...] In a blog post, he says “data is one of the most important pieces to faciliate understanding (and innovation), and is also one of the most under-explored areas of the modern web.” He also says that Mozilla has two early projects that touch on the idea – Spectator and Test Pilot. [...]
[...] In a blog post, he says “data is one of the most important pieces to faciliate understanding (and innovation), and is also one of the most under-explored areas of the modern web.” He also says that Mozilla has two early projects that touch on the idea – Spectator and Test Pilot. [...]
[...] data collection, with the simple logging tool Spectator returning to compatibility recently and the Mozilla Labs participant pool (aka Test Pilot) [...]
The image of the Firefox Test Pilot, on the top of the post, is available in a bigger size or in vectorial version ? It’s Free, Open Source, Creative Commons or closed ?
Thanks!
ready to serve my browser! going to dl as soon as download link is available… xD…
I would like to participate as long as we can see some of the results I often run into issues and end up trying to find out if it is Firefox, windows, or some combination thereof. Is this also going to be something we can install into fennic?
I’d love to participate, even if i’m one of your horrible early adopters ;D
you’ll be getting at least 2 test cases out of me, one mac and one XP with portablefirefox
Sounds like a nice project.
Great idea, can’t wait to take part in it!
You know what another would another great idea? If you removed all pings from the comments and moved them to a separate (collapsed) location on the page, like most good blogs do.
That would be nice to see in action.. Good luck guys.. I’l be in the first ten person to dl this
Cool !
Sign me in. keep me posted when DL is available
Please sign me up.
How do I sign up or where do we download? Keep up the great work!
I’d love to be a part of the testing process. I’m running Firefox on Windows and OS X, mostly the 3.1 beta 2-version. Please feel free to e-mail me regarding this if you please. Thanks!
Please let me know how to subscribe.
Eu quero muito compor este time de contribuidores para o Firefox pois tenho uma grande conhecimento de Internet e trabalho com a área de suporte técnico e também sou uma defendedor e um protetor do Firefox, sendo ele o meu preferido para o uso.
Este projeto para mim é de suma importância e de grande avalia para os seus usários. Mais uma vez eu tenho de parabelizalos por mais esta iniciativa e contem comigo para a melhoria deste software.
hello world. testing presently extensions and script to improve the browsing confort & ergonomy, i’d like to be able to give you some feedback on the general use of firefox. how can i subscribe to the “test pilot” program ?
no entiendo lo que quiere decir Website ssique no lo puedo rellenar solo pido que de Mocilla Firefox no tenga que sali por nada del mundo me fio de este sistema tampoco quiero perder mi correo que es portiz47@gmail.com muchas gracias por dejarme escrivir lo que pienso
Sign me up too, would love to help if I can.
Happy to help. Let me know when you want folks to volunteer! Cat
I Can Haz Test Pilot for Firefox?
Seriously, would definitely like to know more on how to sign up.
I’d like to get in on this, if possible. Let me know.
This is a GREAT Idea. I hope it will help for develop a powerful Firefox!
I want to be a test pilot!
Sorry for the english!
I’m interested I’m a novice can’t type well or fast, I’d still like to help
Count me in. I also can do Turkish localization. Let me know.
Sign me up too, I would love to help.
PLZ sign me up. I would really like to be a part of this.
I would also like to be a part of this. Please sign me up as well.
Would love to test also. Sign me up too.