User testing results

This week I had the opportunity to conduct some user tests on Matthew’s wireframes for the Teach site.*

The goal of the user test was to validate the information architecture. We tested two nav variations:

Variation B

Variation A

Variation B

Variation B

Nav B performed slightly better, but both variations had some challenges. We shouldn’t draw any solid conclusions from the tests, partially because it wasn’t a large enough sample (we had five user testers, which meets the best practice standard, but since we tested two variations, we should have had more), and partially because most of the participants were highly familiar with our work already, making them a somewhat biased sample.

However, the user testers were fantastic, and we were able to generate some valuable insights that will be useful moving forward. Here are some of the key learnings:

  • Mentors want to connect with other mentors. Regardless of the task we gave them (“find something to do for your after-school program,” “find a way to improve your own skills,” “find people to connect with in your community”), nearly all of the user testers wanted to accomplish those tasks by seeing profiles of other mentors, learning about what those mentors had done, and finding ways to connect directly to them.
  • Mentors make distinctions between independent, bite-sized activities, and full-blown curricula. There’s a need for both. The word “activities” seems to suggest shorter, one-off icebreakers and games, whereas some of the mentors expected to find full curriculum somewhere else (e.g. under “Clubs”).
  • Mentors tend to know what they’re looking for. The wireframes introduced the idea of curated content—featured mentors, featured activities, and featured discussion topics. However, user testers consistently had a specific idea in mind, and gravitated towards the search field (even though it was nearly below the fold). We may want to promote searching over browsing, though we’ll likely want to pair it with the browsing framework provided by the web literacy map
  • “Badges” is not a destination. Several testers were confused by the idea of a “featured badge.” In the users’ minds, badges are tied to skills and accomplishments, or perhaps pieces of curriculum, rather than being an end in themselves.
  • People need pathways for becoming a mentor. For those who want to contribute as a mentor, but who don’t have regular access to learners (i.e. those who aren’t professional educators), we need to provide clear pathways and opportunities for mentorship.

That last point hints at a critical discussion we need to have about the various audiences we’re serving. Mentors who have regular access to learners have different needs from the mentors who don’t have that access. There are some overlapping needs, of course—namely, access to high-quality activities and professional development—but how we package those offerings will be different.

We’ll be exploring some options through the ongoing design process over the next few heartbeats.

Thanks to the very rad user testers who volunteered their time to help with this!

*I’m tempted to call this The Site That Shall Not Be Named, because the name is a little bit up in the air. It’s unlikely that we’ll go with “Teach.webmaker.org,” at least in part because our resources for teachers are not entirely tied to the Webmaker tools. In fact, our curriculum and resources are pretty tool-agnostic. “Mozilla Learning” might be a better encapsulation of our product offering.

Building Better Badge Tooling

Over the past few weeks, I’ve had lots of in-depth conversations with many people about the badges project. The conversations have been helpful in terms of clarifying the purpose and value of badges, as well as defining a product roadmap for the near term future.

I’d like to share a summary of where we are now.

PART 1: VALUE

First, a reminder of the value of badges to learners and mentors:

  • Structure and self-tracking—badges provide clear pathways for learning and contribution
  • Professional development, reputation, and career advancement—badges provide clear pathways for increasing skills and knowledge, are perceived as evidence of mastery and expertise, and are connected to specific projects of which the recipient is proud
  • Recognition—badges provide a concrete way to recognize people’s contributions and accomplishments
  • Motivation—badges can keep people engaged and desiring to progress through a pathway. Badges can  also be used to engage new/bashful learners
  • Relevance—badges are connected to things people care about

And the reasons badges are important to the Webmaker program:

  • Pathways—badges provide primary navigation for people entering the fold
  • Proof of learning / metrics — badges provide a concrete way for us to measure learning in our on-the-ground programming and online trainings
  • Partner strategy — when paired with a rigorous curriculum, badges provide a framework for certifying a learner’s digital literacy skills and competencies. This has significant appeal to many of our partners.
  • Growing user base — badges provide a significant incentive for learners connected to a ground game effort (e.g. Maker Party, Club, or Hive) to join the broader community (and be made aware of the full range of our offerings)
  • Recognizing local leaders – the “Mentor,” “Super Mentors,” “Club Captain,” and “Hive Community Member” badges convey social status and legitimacy

PART 2: BADGE OFFERINGS

As a team, the next thing to do is get on the same proverbial page about our canonical  badge offerings, as well as consider changes to our issuing scheme to allow us to scale. Based on input from many people, I’d like to  propose the following for our suite of Webmaker Contributor badges for 2015:

(Please note: this list is not finalized)

Contributor badges issued by staff:

  • Event Host
  • Teaching Kit Remixer
  • Club Captain (New!)
  • Hive Community Member
  • Mentor
  • Super Mentor

(At least a couple of the above might eventually be automatically awarded by the system based on a user’s activity on the site)

Contributor badges issued by Mentors, Super Mentors, and Club Captains:

  • Mentor (issued by Super Mentors and staff only)
  • Skill Sharer
  • Participation badges (this is a collection of badges Club Captains may want to issue to their Club members to recognize participation and effort. We’ll be examining these requirements during the rest of Q1.)

We also need to finalize a canonical list of Web Literacy Badges. The current list needs refining, based on upcoming changes to the Web Literacy Map. More to come on this soon!

PART 3: PRODUCT ROADMAP

Thanks to these rich conversations with stakeholders, I’ve become familiar with the tooling we have around badges now, as well as some of the desired changes to the user experience and functionality.

A high-level version of the product roadmap for badges is as follows, though it’s unlikely we’ll build in the exact order presented below:

a) Clearly present our badge offerings and highlight learning pathways

Once we finalize the badge offerings, we’ll build a landing page that showcases and clearly differentiates the several types of badges we offer. The landing page will illustrate the progressive nature of our badge offerings, and serve as navigation through our program offerings.

b) Improve user experience for badge earners

We’ll make a few key improvements to the badge application workflow, including allowing applicants to view the status of their pending applications, and edit their submissions while they’re pending.

We’ll create a Badge Manager page for mentors and learners. From the Badge Manager page, people will be able to:

  • see the badges they’ve earned, including the evidence that was provided
  • share their earned badges, and invite or nominate others to also earn them
  • see their pending applications, and respond to any questions from reviewers
  • see recommended badges (based on previously earned badges and our pathways)

c) Improve user experience for badge issuers

We’ll also make some improvements to the badge issuing dashboard:

  • Allow badge issuers option to receive notifications or emails about new badge applications
  • Show counts of outstanding applications, and provide multiple views of outstanding applications (e.g. by submission date, by badge type)
  • Provide search tool for issued badges, with criteria such as issue date, badge type, or reviewer (the search results will be exportable, enabling us to contact everyone who’s received a certain badge, for example)

We’ll improve the design of the application reviewer pane by including (in addition to the applicant-provided evidence):

  • badge criteria
  • information about the applicant, including other badges received/applied for

d) Let Club Captains issue badges to their Club members

With the launch of the Clubs program, we’ll have a need to grant badge-issuing privileges to Club Captains. The permissions may be limited in two ways:

  • They’ll be able to grant badges only to members of their Club (this will require creating a connection between individuals and Clubs on the backend)
  • They may only be able to grant Web Literacy and participation badges

All of the above is now reflected on the Badges wiki page. Would welcome feedback in the comments.

A new online home for those who #teachtheweb

We’ve recently begun work on a new website that will serve the mentors in our Webmaker community—a gathering place for anyone who is teaching the Web. They’ll find activity kits, trainings, badges, the Web Literacy Map, and more. It will also be an online clubhouse for Webmaker Clubs, and will showcase the work of Hives to the broader network.

Our vision for the site is that it will provide pathways for sustained involvement in teaching the Web. Imagine a scenario where, after hosting a Maker Party, a college student in Pune wants to build on the momentum, but doesn’t know how. Or imagine a librarian in Seattle who is looking for activities for her weekly teen drop-in hours. Or a teacher in Buenos Aires who is looking to level up his own digital literacy skills. In each of these scenarios, we hope the person will look to this new site to find what they need.

We’re in the very early stages of building out the site. One of our first challenges is to figure out the best way to organize all of the content.

Fortunately, we were able to find 14 members of the community who were willing to participate in a “virtual card-sorting” activity. We gave each of the volunteers a list of 22 content areas (e.g. “Find a Teaching Kit,” “Join a Webmaker Club,” “Participate in a community discussion”), and asked them to organize the items into groups that made sense to them.

The results were fascinating. Some grouped the content by specific programs, concepts, or offerings. Others grouped by function (e.g “Participate,” “Learn,” “Lead”). Others organized by identity (e.g. “Learner” or “Mentor”). Still others grouped by level of expertise needed.

We owe a debt of gratitude to those who participated in the research. We were able to better understand the variety of mental models, and we’re currently using those insights to build out some wireframes to test in the next heartbeat.

Once we firm up the information architecture, we’ll build and launch v1 of the site (our goal is to launch it by the end of Q1). From there, we’ll continue to iterate, adding more functionality and resources to meet the needs of our mentor community.

Future iterations will likely include:

  • Improving the way we share and discover curriculum modules
  • Enhancing our online training platform
  • Providing tools for groups to self-organize
  • Making improvements to our badging platform
  • Incorporating the next version of the Web Literacy Map

Stay tuned for more updates and opportunities to provide feedback throughout the process. We’ve also started a Discourse thread for continuing discussion of the platform.

Teach.webmaker.org: Initial Card Sorting Results

This past week I conducted a small user research project to help inform the IA of the new teach.webmaker.org site.

I chose a card sorting activity, which is a common research method for IA projects. In a card sorting activity, you give members of your target audience a stack of cards, each of which has one of the site content areas printed on it. You ask the participants to group items together and explain their thought process. In this way, you gain an understanding of the participants mental models. This is helpful for avoiding a common pitfall in site design, which is organizing content in a way that make sense to you but not your users.

Big Giant Caveat

This study was flawed in a couple ways. First, Jacob Nielsen (who is generally considered to be a real smartypants when it comes to usability and user research) recommends that you do card sorting with 15 users. I’ve only been able to get 11 to do the activity so far, though I think a few more are pending.

Another flaw is that I deviated from a common best practice of running these activities in person. A lot of the insights are gained by listening to the person think aloud. There are some tools for running an online card sorting activity, but they’re largely for what’s called “closed” card sorts, where you pre-determine the categories and the person’s task is to sort cards within those categories. Since one of my goals with this activity was to generate a better understanding of what terminology to use, I wanted to do an “open” sort, where the participants name their groupings themselves.

All that’s to say that we shouldn’t take these results or my analysis as gospel. I do think the participant responses will be useful as we move forward with designing some wireframes to user test in the next heartbeat.

Participant Demographics and Background Information

There were a range of ages and locations represented in the study.

Four participants are between 18 and 24 years old, three are between 25 and 34, two between 35 and 44, one between 45 and 54, and one between 55 and 64.

Four participants are from the United States, three from India, and one each from Colombia, Bangladesh, Canada, and the United Kingdom.

Participants were asked to rate their level of familiarity with the Webmaker Mentors program on a scale of 1 to 5, with 5 being the most familiar. Again, there was a range. Four participants rated themselves a 5, two a 4 or 4.5, two a 3, one a 2, and two a 1.

Initial Findings

The participants in the study had a range of different mental models they used to organize the content. Those models were:

  1. Grouping by program offering—that is, organizing by specific programs, concepts, or offerings, typically expressed as nouns (e.g. Web Literacy, Teaching Kits, Webmaker Clubs, Trainings, Activities, Resources, Social, Learning, Philosophy, Mentoring, Research, Events, Supportive Team)Five participants used a model like this as their primary model. The average familiarity level with Webmaker Mentoring for these participants matches the average for the entire sample (3.7 on a five-point scale).
  2. Grouping by functional area—that is, actions that a user might take, typically expressed as verbs (e.g. participate, learn, market/promote, meet others, do, lead, get involved, collaborate, organize, develop yourself, teach, experiment, host, attend).Four participants used a model like this as their primary model. Notably, all of the participants are from the United States, Canada, or the United Kingdom, and their average familiarity with Webmaker Mentoring is below the average of the entire sample (2.75 as compared to 3.67).
  3. Grouping by role or identity—some study participants organized the content by the type of user who would be interested in it (e.g. Learner, Mentor).One participant  used this as their primary model. Another made a distinction between Learning and Teaching, but it was framed more like the functional areas described above. One more used “Learning Geeks” as a topic area.
  4. Level of expertise—in this model, there is a pathway through the content based on level of expertise (e.g. intermediate, advanced).One participant used this as their primary model.

Other patterns, themes, and notable terminology:

  • Seven participants grouped together content related to hosting or attending events, and three participants made references to face-to-face communication. Of the seven who grouped content into the “Events” topic, five of them included the one item that referenced “Maker Party” (including two participants who rated their level of familiarity with the program at a 1), indicating a strong understanding of “Maker Party” as a type of event.
  • Five participants made references to the broader community. Three of them are from the United States, one from Canada, and one from India. (The specific terminology used were “Meet others,” “Social,” “Webmaker Community,” “Collaborate,” and “Supportive team”).
  • Four participants used the word “Webmaker” in their groupings, which gives us some insight into how they understand the brand. In each case, participants seem to connect the term to either teaching and teaching kits, or to the community of interested people.
  • Three participants used the term “Leading.”
  • One participant referenced a particular context (“Webmaker for Schools”).
  • One participant distinguished Mozilla-produced content (as “Mozilla Outputs”).
  • We included the term “Peer Learning Networks” in the content list to represent Hives (we assumed the meaning of “Hive” would be difficult to intuit for those unfamiliar). While we can’t draw any conclusions based on this data, it’s notable that this term was grouped into a wide variety of topics, including community (“Meet others,” “Social,” and “Collaborate”), “Get Involved,” “Intermediate,” “Mozilla Outputs,” and “Learning Geeks.” Three participants felt it didn’t fit under any category.
  • We tested both “Professional Development” and “Trainings” to see if we could understand how people interpret those terms. The results are fairly ambiguous. Both terms were associated with “Activities for teachers & mentors”, “Leading,” “Get Involved,” and “Research (things you learn on your own).” “Professional Development” was also associated with “Learning,” “Develop Yourself,” and “Learning Geeks”. “Trainings” was associated with “Intermediate,” “Mentoring,” “Organize in person events,” and “Supportive team.” For both terms, three participants could not categorize this term.

Let me know if you’re interested in seeing the raw data.

I need to reprint my business cards

Okay, so I don’t actually have business cards, but on this morning’s All Mofo call, it was announced that I’m leaving the wonderful Engagement team to serve as a product manager for the equally wonderful Learning Networks team. So, if I had business cards, I’d need new ones.

This is, for obvious reasons, bittersweet. I’ve LOVED working with the engaging folks on the Engagement team, and it provided a fantastic vantage point for learning the ins and outs of Mofo. I’m sending a big heartfelt thank you to Geoffrey and Co. for being so dang awsm to me ever since I joined.

Fortunately for me, I’m not going far. I’ve been admiring the work of both the Mentor Networks and the Product teams from a distance, so I’m thrilled with my new spot right smack in the middle of them.

Wait….Hannah, what do you know about product management? And Learning Networks?

It might seem strange at first blush, since I’ve been talking about scrum mastering and engagement-y stuff on this blog so far. But, lest you think I’m totally unqualified, let me share a few relevant experiences I haven’t shared here before:

  • I was a Product Owner at my last job. “Product Owner” is a title specific to Scrum shops, but it’s got a whole lot in common with Product Manager. Working with devs? Check! UI and UX designers? Yup, them too. End users and stakeholders? Love ‘em. Caring about product adoption rates, product marketing, and customer service? For sure, uh huh, no doubt.
  • I have experience shepherding a product to serve a local groups-based organizing model, much like the vision for Webmaker Clubs. I hope to bring useful knowledge from that project, drawing on both successes and failures (because, hey, “what not to do” lists are useful, too!)
  • While I won’t be contributing in this way, I do have a bit of experience as a trainer—I’ve developed and delivered service-learning and social justice careers curricula to kids, college students, and adults. I’m nowhere near as savvy as our #TeachTheWeb team, but I can promise to keep the needs of the mentors at the forefront of my brain.

OK, so what are you going to be working on?

The Big Picture answer is: developing products that serve the needs of our constituents in our ground game programs (Hive, Webmaker Clubs, Maker Parties). These products will be separate from, but complementary of, the Learning Products, which serve independent learners who aren’t (yet!) affiliated with our ground game programs.

In the short term, my top priorities are:

  • Building a new home for all of our teaching resources and a launching point for all of our ground game programs (teach.webmaker.org). v1 will be a re-org of our existing content, so we’ve launched a small user research study (you may have seen Lucy’s recent email to the Webmaker listserv asking for volunteers). The participants are doing a virtual version of what’s called a “card sorting” activity to help us understand their mental models around all of the content we currently have. The results will inform the information architecture for Teach.w.o v1.
  • Launching a platform for local groups (i.e. Clubs and potentially Hives). Q1 is about two flavors of research: 1) Developing a deep, nuanced understanding of our own business needs and the needs of our users—in this case, the Club Leaders in the Q1 Pilot. 2) Investigating off-the-shelf options for the platform.
  • Iterating on our credentials platform. Again, this starts with developing a deep, shared understanding of business needs. Stakeholder kick-off meeting coming soon!

I’m so very excited to be working on these things. Like, I’m seriously being a nerd about it all.

At the same time, I’m already missing my Engagement team buddies (though that’s tempered by the fact that I still get to work with nearly all of them :)).

Questions? Want to discuss the Learning Networks products? Hit me up.

We are very engaging

Yesterday someone asked me what the engagement team is up to, and it made me sad because I realized I need to do a waaaaay better job of broadcasting my team’s work. This team is dope and you need to know about it.

As a refresher, our work encompasses these areas:

  • Grantwriting
  • Institutional partnerships
  • Marketing and communications
  • Small dollar fundraising
  • Production work (i.e. Studio Mofo)

In short, we aim to support the Webmaker product and programs and our leadership pipelines any time we need to engage individuals or institutions.

What’s currently on our plate:

Pro-tip: You can always see what we’re up to by checking out the Engagement Team Workbench.

These days we’re spending our time on the following:

  • End of Year Fundraising: With the help of a slew of kick-ass engineers, Andrea and Kelli are getting to $2M. (view the Workbench).
  • Mozilla Gear launch: Andrea and Geoffrey are obsessed with branded hoodies. To complement our fundraising efforts, they just opened a brand new site for people to purchase Mozilla Gear (view the project management spreadsheet).
  • Fall Campaign: Remember the 10K contributor goal? We do! An-Me and Paul have been working with Claw, Amira, Michelle, and Lainie, among others, to close the gap through a partner-based strategy (view the Workbench).
  • Mobile Opportunity: Ben is helping to envision and build partnerships around this work, and Paul and Studio Mofo are providing marketing, comms, and production support (the Mobile Opportunity Workbench is here, the engagement-specific work will be detailed soon).
  • Building a Webmaker Marketing Plan for 2015: The site and programs aren’t going to market themselves! Paul is drafting a comprehensive marketing calendar for 2015 that complements the product and program strategies. (plan coming soon)
  • 2015 Grants Pipeline: Ben and An-Me are always on the lookout for opportunities, and Lynn is responsible for writing grants and reports to fund our various programs and initiatives.
  • Additional Studio Mofo projects: Erika, Mavis, and Sabrina are always working on something. In addition to their work supporting most of the above, you can see a full list of projects here.
  • Salesforce for grants and partnerships: We’ve completed a custom Salesforce installation and Ben has begun the process of training staff to use it. Much more to come to make it a meaningful part of our workflow (Workbench coming soon).
  • Open Web Fellows recruitment: We’re supporting our newest fellowship with marketing support (view the Hype Plan)

How to Mofo

OpenMatt and I have been talking about the various ways of working at Mofo, and we compiled this list of what we think works best. What do y’all Mofos think?

When starting a new project:

  • Clearly state the problem or goal. Don’t jump ahead to the solution. Ameliorating the problem is what you’ll measure success against, not your ability to implement an arbitrary solution.
  • Explicitly state assumptions. And, whenever possible, test those assumptions before you build anything. You may have assumptions about the nature of the problem you’re trying to solve, who’s experiencing it, or your proposed solution.
  • Have clear success metrics. How will you know if you’re winning? Do you have the instruments you need to measure success?
  • Determine what resources you need. Think about design, development, content, engagement, evaluation, and ongoing maintenance. We’re working on improving the ways we allocate resources throughout the organization, but to start, be clear about what resources your project will need.
  • Produce a project brief. Detail all of the above in a single document. (Example templates here and here.) Use the project brief when you…
  • …Have a project kick-off meeting. Invite *all* the stakeholders to get involved early.

Communication:

  • Have a check-in plan. Will you have daily check-ins? Weekly email updates? How are you checking in and holding each other accountable?
  • Build a workbench and keep it updated.  We recommend a wiki page that will serve as a one-stop shop for anyone needing information about the project. Things to include: links to project briefs and notes, logistics for meetings, a timeline, a list of who’s involved, and, of course, bugs! Examples here, here, and here.
  • Put your notes in one spot.  A single canonical pad for notes and agendas. We don’t need to create a  new pad every time you have a meeting or a thought! That makes them very hard to track and find later. Examples here and here.

Doing the do:

  • Plan in two-week heartbeats. This helps us stay on track and makes it clear what the priorities are. Speaking of priorities…
  • Learn the Fine Art of Prioritizing. Hint: Not everything can be P1. The product owner or project manager should rank tasks in order of value added. Remember: prioritization is part of managing workflow. It may be true that all or most of the tasks are required for a successful launch, but that doesn’t help a developer or designer who’s trying to decide what to work on next.
  • Work with your friendly neighborhood Tactical Priorities Syndicate. The name sounds scary, but they’re here to serve you. They meet weekly to get your priorities into each two-week heartbeat process. https://wiki.mozilla.org/Webmaker/TPS

Update: To hack on the next version of this, please visit http://workopen.org/mofo (thanks to Doug for the suggestion!)