User Testing the Teach Site

We are soooooo close to releasing the new Teach site.

People seem to dig the bright colors and quirky illustrations throughout the site.

People seem to dig the bright colors and quirky illustrations throughout the site.

In advance of the release, I wanted to conduct some user tests to make sure we’re still on the right track. This week I conducted two user tests with members of the community (yay!). As is always the case with user testing, I learned a lot from observing users interact with the site.

You can see detailed notes here and read my recommendations below.

These recommendations are based on formal user tests with two users as well as feedback from people who’ve been involved or observing the process throughout.  Also, please note that I wasn’t able to test the primary functionality on the site (adding a Club to the map), so these recommendations are more about IA and other content issues.

Findings and Recommendations

Screen Shot 2015-04-10 at 2.48.07 PM



  • People want to see more activities and resources.
  • People expect to be able to sort and filter.
  • Our internal distinction between the Clubs Curriculum (the official curriculum for Clubs; with a strong recommendation for following the prescribed path) and Teaching Activities (more “grab and go”-style) is not intuitive to users.
  • The Teach Like Mozilla content needs to be more integrated into common user flows.


  • Continue with current plan for developing and publishing more approved curriculum and activities.
  • Continue brainstorming work around scalable presentation of curriculum begun in this heartbeat. The ideas discussed so far address sorting and filtering, and make good use of the Web Literacy Map as an organizing tool.
  • As part of that design work, we should also allow users to access all teaching materials from the same page, and provide specific views for “official Clubs curriculum.” I recommend we keep the Teaching Activities page, and remove the Clubs Curriculum sub-page. This content is one of our primary offerings so it belongs at the top level. /cc @iamjessklein
  • We need to offer a solution for sharing resources—e.g. maker tools, other curricula, programs. (Hello, Web Lit Mapper!)
  • We need to design a stronger connection between teaching activities and the Teach Like Mozilla content. A short-term solution might be to link to the TLM page from every individual activity page, but we should also be working towards a better longer-term solution. /cc @laurahilliger

Screen Shot 2015-04-10 at 2.50.46 PM



  • The Clubs Toolkit is not findable, and needs to be supplemented with content targeted towards helping people “get started.”
  • We are not providing enough information for the use case of a person who is deciding in the moment whether to start a Club.


  • Make the Clubs Toolkit more visible on the page.
  • Consider renaming the Clubs Toolkit something like “Getting Started Guide” or “A Club’s First Month” – and editing content to match. /cc @thornet
  • Based on my understanding of the expected pathways to starting Clubs, I do not think we need to make any significant changes to the Clubs page to address the use case of someone coming to the site and deciding in the moment whether or not to start a Club. As I understand it, our plan for growing Clubs makes use of the following scenarios:
    • 1) Someone is “groomed” by staff member, Regional Coordinator, or other community member. By the time they arrive at the site, they have the specific intent of adding their Club.
    • 2) Someone finds out about us through Maker Party, and through a series of communications learns about Clubs and decides to start one. They are coming to the site with the specific intent to add their Club.
    • 3) Someone with an existing program or group wants to be listed in the database. Again, they are coming to the site with the intent to add their Club.

In short, I don’t think we’ve yet seen a reason to have the site serve a “selling” or persuasive function. I *do* think the Clubs page is a natural first stop for someone who is looking to understand how to start a Club. I think the changes recommended in the bullet points above address that.

Screen Shot 2015-04-10 at 2.51.21 PM



  • The copy describing the Events page in the main navigation is misleading, since the content on the Events page is about Maker Party.
  • People may understand throwing a Maker Party as a “first step” to starting a Club, rather than a lower-bar option for people who do not want to start a Club (and perhaps never will).


  • I think we should re-brand what is currently the Events landing page as “Maker Party.” We’ve already sort of done this in that, while the page is called “Events” in the nav, the h1 copy in the hero image is “Host a Maker Party.” I suggest we change the copy in the nav to “MAKER PARTY” and the teaser copy to “Our annual global campaign”. /cc @amirad

Screen Shot 2015-04-10 at 2.52.06 PM



  • Users tend to ignore, not see, or misinterpret the CTAs at the bottom of every page
  • Users do not notice links to the sub-pages in the main navigation


  • We need to design better, more intuitive pathways for viewing secondary pages


I’m going to keep banging this drum: We need to clarify our audience! I think we’ve made good progress in terms of clarifying that our “first line” audience includes educators and activists. But I think we have to take it a step further and clarify who those educators and activists are working with. There are at least two axes that I think are important to be clear about: first, the global nature of our work, and second, the specific age groups of what I’m calling the “end learners,” for lack of a better term.

I think we do a pretty decent job of conveying the global nature of the program through copy and imagery, though obviously implementing our l10n strategy is absolutely fundamental to this.

I think we are less clear when it comes to the age groups we’re targeting with our programs and materials.   For example, I think we ought to specify the appropriate age level for each activity. (And the images, activity titles, and copy should reflect the target audience.)

Questions, comments, disagreements wholeheartedly welcomed!

An update on the Teach site + some thoughts on iterative development

It’s been a minute! We’ve just started a new heartbeat, and now we have a giant team working on the Teach site. Our stand-up today looked like The Brady Bunch.

We’re now entering a stage where we have three strands of work running in parallel:

  1. Engineers are setting up the technical framework for the new site, and are beginning to build out the pages that have been designed
  2. Designers are polishing up the work they’ve done so far (accommodating some last minute requests from me!), and starting on a few new pages and workflows
  3. Members of the Learning Networks team are finalizing copy and content for the various pages

If every strand can keep moving, we’ll be in good shape for the release at the end of the next heartbeat. Ideally, we’ll get into a rhythm where copy is finalized in the heartbeat *before* the design work, which is done in the heartbeat *before* development, but right now things are overlapping a bit.

Iterative Dev {It’s been so long since I’ve done an anagram! Here we go: Tea Revived It}

I’ve been thinking about how to build out this site in a way that reflects the evolving nature of the Learning Networks team work, and specifically the Clubs program. The answer is (always) to be agile.

It’s often difficult to agree on what should be included in a v1. I think the term “MVP” is one of the most abused terms in software. It is sometimes used to mean “do as little work as possible,” but the definition I like best is, “do as much as is needed to prove or disprove a hypothesis.”

This definition provides a great framework for determining what to include in a v1. If you can agree on a hypothesis, then every decision can be made by answering the question, “Do we need this to prove the hypothesis?”

For the Teach site, the hypothesis we’re testing is: “People will use the site to find teaching activities and add their Clubs to the map.”

So the v1 will seek to prove that. The priority items are a couple static pages that point to our high-quality curriculum modules, a Clubs page with an “Add Your Club” workflow, and some additional static content to flesh out the site.

Beyond Version One {Anagram: Sobered Onion Envy}

While  v1 is being created, we also need to start gearing up for the next iteration, which will be focused on building out more advanced tooling for the curriculum (so that we can add to those pages without developer involvement, and incorporate feedback and remix ideas from the community). This is essentially adding a fourth strand to the list above—a kind of “requirements gathering and brainstorming” strand.

(Side note: I like that the “build, measure, learn” cycle we’ll use to iterate on the site echoes the process we’re using to develop the curriculum with our community. Agile methods all around!)

Of course, even while using iterative development practices, we still need to keep the big picture in mind. To see the current thinking about future iterations, you can always check out the roadmap. I’m updating it regularly as our needs change, or as they become clearer to me.

User testing results

This week I had the opportunity to conduct some user tests on Matthew’s wireframes for the Teach site.*

The goal of the user test was to validate the information architecture. We tested two nav variations:

Variation B

Variation A

Variation B

Variation B

Nav B performed slightly better, but both variations had some challenges. We shouldn’t draw any solid conclusions from the tests, partially because it wasn’t a large enough sample (we had five user testers, which meets the best practice standard, but since we tested two variations, we should have had more), and partially because most of the participants were highly familiar with our work already, making them a somewhat biased sample.

However, the user testers were fantastic, and we were able to generate some valuable insights that will be useful moving forward. Here are some of the key learnings:

  • Mentors want to connect with other mentors. Regardless of the task we gave them (“find something to do for your after-school program,” “find a way to improve your own skills,” “find people to connect with in your community”), nearly all of the user testers wanted to accomplish those tasks by seeing profiles of other mentors, learning about what those mentors had done, and finding ways to connect directly to them.
  • Mentors make distinctions between independent, bite-sized activities, and full-blown curricula. There’s a need for both. The word “activities” seems to suggest shorter, one-off icebreakers and games, whereas some of the mentors expected to find full curriculum somewhere else (e.g. under “Clubs”).
  • Mentors tend to know what they’re looking for. The wireframes introduced the idea of curated content—featured mentors, featured activities, and featured discussion topics. However, user testers consistently had a specific idea in mind, and gravitated towards the search field (even though it was nearly below the fold). We may want to promote searching over browsing, though we’ll likely want to pair it with the browsing framework provided by the web literacy map
  • “Badges” is not a destination. Several testers were confused by the idea of a “featured badge.” In the users’ minds, badges are tied to skills and accomplishments, or perhaps pieces of curriculum, rather than being an end in themselves.
  • People need pathways for becoming a mentor. For those who want to contribute as a mentor, but who don’t have regular access to learners (i.e. those who aren’t professional educators), we need to provide clear pathways and opportunities for mentorship.

That last point hints at a critical discussion we need to have about the various audiences we’re serving. Mentors who have regular access to learners have different needs from the mentors who don’t have that access. There are some overlapping needs, of course—namely, access to high-quality activities and professional development—but how we package those offerings will be different.

We’ll be exploring some options through the ongoing design process over the next few heartbeats.

Thanks to the very rad user testers who volunteered their time to help with this!

*I’m tempted to call this The Site That Shall Not Be Named, because the name is a little bit up in the air. It’s unlikely that we’ll go with “,” at least in part because our resources for teachers are not entirely tied to the Webmaker tools. In fact, our curriculum and resources are pretty tool-agnostic. “Mozilla Learning” might be a better encapsulation of our product offering.

Building Better Badge Tooling

Over the past few weeks, I’ve had lots of in-depth conversations with many people about the badges project. The conversations have been helpful in terms of clarifying the purpose and value of badges, as well as defining a product roadmap for the near term future.

I’d like to share a summary of where we are now.


First, a reminder of the value of badges to learners and mentors:

  • Structure and self-tracking—badges provide clear pathways for learning and contribution
  • Professional development, reputation, and career advancement—badges provide clear pathways for increasing skills and knowledge, are perceived as evidence of mastery and expertise, and are connected to specific projects of which the recipient is proud
  • Recognition—badges provide a concrete way to recognize people’s contributions and accomplishments
  • Motivation—badges can keep people engaged and desiring to progress through a pathway. Badges can  also be used to engage new/bashful learners
  • Relevance—badges are connected to things people care about

And the reasons badges are important to the Webmaker program:

  • Pathways—badges provide primary navigation for people entering the fold
  • Proof of learning / metrics — badges provide a concrete way for us to measure learning in our on-the-ground programming and online trainings
  • Partner strategy — when paired with a rigorous curriculum, badges provide a framework for certifying a learner’s digital literacy skills and competencies. This has significant appeal to many of our partners.
  • Growing user base — badges provide a significant incentive for learners connected to a ground game effort (e.g. Maker Party, Club, or Hive) to join the broader community (and be made aware of the full range of our offerings)
  • Recognizing local leaders — the “Mentor,” “Super Mentors,” “Club Captain,” and “Hive Community Member” badges convey social status and legitimacy


As a team, the next thing to do is get on the same proverbial page about our canonical  badge offerings, as well as consider changes to our issuing scheme to allow us to scale. Based on input from many people, I’d like to  propose the following for our suite of Webmaker Contributor badges for 2015:

(Please note: this list is not finalized)

Contributor badges issued by staff:

  • Event Host
  • Teaching Kit Remixer
  • Club Captain (New!)
  • Hive Community Member
  • Mentor
  • Super Mentor

(At least a couple of the above might eventually be automatically awarded by the system based on a user’s activity on the site)

Contributor badges issued by Mentors, Super Mentors, and Club Captains:

  • Mentor (issued by Super Mentors and staff only)
  • Skill Sharer
  • Participation badges (this is a collection of badges Club Captains may want to issue to their Club members to recognize participation and effort. We’ll be examining these requirements during the rest of Q1.)

We also need to finalize a canonical list of Web Literacy Badges. The current list needs refining, based on upcoming changes to the Web Literacy Map. More to come on this soon!


Thanks to these rich conversations with stakeholders, I’ve become familiar with the tooling we have around badges now, as well as some of the desired changes to the user experience and functionality.

A high-level version of the product roadmap for badges is as follows, though it’s unlikely we’ll build in the exact order presented below:

a) Clearly present our badge offerings and highlight learning pathways

Once we finalize the badge offerings, we’ll build a landing page that showcases and clearly differentiates the several types of badges we offer. The landing page will illustrate the progressive nature of our badge offerings, and serve as navigation through our program offerings.

b) Improve user experience for badge earners

We’ll make a few key improvements to the badge application workflow, including allowing applicants to view the status of their pending applications, and edit their submissions while they’re pending.

We’ll create a Badge Manager page for mentors and learners. From the Badge Manager page, people will be able to:

  • see the badges they’ve earned, including the evidence that was provided
  • share their earned badges, and invite or nominate others to also earn them
  • see their pending applications, and respond to any questions from reviewers
  • see recommended badges (based on previously earned badges and our pathways)

c) Improve user experience for badge issuers

We’ll also make some improvements to the badge issuing dashboard:

  • Allow badge issuers option to receive notifications or emails about new badge applications
  • Show counts of outstanding applications, and provide multiple views of outstanding applications (e.g. by submission date, by badge type)
  • Provide search tool for issued badges, with criteria such as issue date, badge type, or reviewer (the search results will be exportable, enabling us to contact everyone who’s received a certain badge, for example)

We’ll improve the design of the application reviewer pane by including (in addition to the applicant-provided evidence):

  • badge criteria
  • information about the applicant, including other badges received/applied for

d) Let Club Captains issue badges to their Club members

With the launch of the Clubs program, we’ll have a need to grant badge-issuing privileges to Club Captains. The permissions may be limited in two ways:

  • They’ll be able to grant badges only to members of their Club (this will require creating a connection between individuals and Clubs on the backend)
  • They may only be able to grant Web Literacy and participation badges

All of the above is now reflected on the Badges wiki page. Would welcome feedback in the comments.

A new online home for those who #teachtheweb

We’ve recently begun work on a new website that will serve the mentors in our Webmaker community—a gathering place for anyone who is teaching the Web. They’ll find activity kits, trainings, badges, the Web Literacy Map, and more. It will also be an online clubhouse for Webmaker Clubs, and will showcase the work of Hives to the broader network.

Our vision for the site is that it will provide pathways for sustained involvement in teaching the Web. Imagine a scenario where, after hosting a Maker Party, a college student in Pune wants to build on the momentum, but doesn’t know how. Or imagine a librarian in Seattle who is looking for activities for her weekly teen drop-in hours. Or a teacher in Buenos Aires who is looking to level up his own digital literacy skills. In each of these scenarios, we hope the person will look to this new site to find what they need.

We’re in the very early stages of building out the site. One of our first challenges is to figure out the best way to organize all of the content.

Fortunately, we were able to find 14 members of the community who were willing to participate in a “virtual card-sorting” activity. We gave each of the volunteers a list of 22 content areas (e.g. “Find a Teaching Kit,” “Join a Webmaker Club,” “Participate in a community discussion”), and asked them to organize the items into groups that made sense to them.

The results were fascinating. Some grouped the content by specific programs, concepts, or offerings. Others grouped by function (e.g “Participate,” “Learn,” “Lead”). Others organized by identity (e.g. “Learner” or “Mentor”). Still others grouped by level of expertise needed.

We owe a debt of gratitude to those who participated in the research. We were able to better understand the variety of mental models, and we’re currently using those insights to build out some wireframes to test in the next heartbeat.

Once we firm up the information architecture, we’ll build and launch v1 of the site (our goal is to launch it by the end of Q1). From there, we’ll continue to iterate, adding more functionality and resources to meet the needs of our mentor community.

Future iterations will likely include:

  • Improving the way we share and discover curriculum modules
  • Enhancing our online training platform
  • Providing tools for groups to self-organize
  • Making improvements to our badging platform
  • Incorporating the next version of the Web Literacy Map

Stay tuned for more updates and opportunities to provide feedback throughout the process. We’ve also started a Discourse thread for continuing discussion of the platform. Initial Card Sorting Results

This past week I conducted a small user research project to help inform the IA of the new site.

I chose a card sorting activity, which is a common research method for IA projects. In a card sorting activity, you give members of your target audience a stack of cards, each of which has one of the site content areas printed on it. You ask the participants to group items together and explain their thought process. In this way, you gain an understanding of the participants mental models. This is helpful for avoiding a common pitfall in site design, which is organizing content in a way that make sense to you but not your users.

Big Giant Caveat

This study was flawed in a couple ways. First, Jacob Nielsen (who is generally considered to be a real smartypants when it comes to usability and user research) recommends that you do card sorting with 15 users. I’ve only been able to get 11 to do the activity so far, though I think a few more are pending.

Another flaw is that I deviated from a common best practice of running these activities in person. A lot of the insights are gained by listening to the person think aloud. There are some tools for running an online card sorting activity, but they’re largely for what’s called “closed” card sorts, where you pre-determine the categories and the person’s task is to sort cards within those categories. Since one of my goals with this activity was to generate a better understanding of what terminology to use, I wanted to do an “open” sort, where the participants name their groupings themselves.

All that’s to say that we shouldn’t take these results or my analysis as gospel. I do think the participant responses will be useful as we move forward with designing some wireframes to user test in the next heartbeat.

Participant Demographics and Background Information

There were a range of ages and locations represented in the study.

Four participants are between 18 and 24 years old, three are between 25 and 34, two between 35 and 44, one between 45 and 54, and one between 55 and 64.

Four participants are from the United States, three from India, and one each from Colombia, Bangladesh, Canada, and the United Kingdom.

Participants were asked to rate their level of familiarity with the Webmaker Mentors program on a scale of 1 to 5, with 5 being the most familiar. Again, there was a range. Four participants rated themselves a 5, two a 4 or 4.5, two a 3, one a 2, and two a 1.

Initial Findings

The participants in the study had a range of different mental models they used to organize the content. Those models were:

  1. Grouping by program offering—that is, organizing by specific programs, concepts, or offerings, typically expressed as nouns (e.g. Web Literacy, Teaching Kits, Webmaker Clubs, Trainings, Activities, Resources, Social, Learning, Philosophy, Mentoring, Research, Events, Supportive Team)Five participants used a model like this as their primary model. The average familiarity level with Webmaker Mentoring for these participants matches the average for the entire sample (3.7 on a five-point scale).
  2. Grouping by functional area—that is, actions that a user might take, typically expressed as verbs (e.g. participate, learn, market/promote, meet others, do, lead, get involved, collaborate, organize, develop yourself, teach, experiment, host, attend).Four participants used a model like this as their primary model. Notably, all of the participants are from the United States, Canada, or the United Kingdom, and their average familiarity with Webmaker Mentoring is below the average of the entire sample (2.75 as compared to 3.67).
  3. Grouping by role or identity—some study participants organized the content by the type of user who would be interested in it (e.g. Learner, Mentor).One participant  used this as their primary model. Another made a distinction between Learning and Teaching, but it was framed more like the functional areas described above. One more used “Learning Geeks” as a topic area.
  4. Level of expertise—in this model, there is a pathway through the content based on level of expertise (e.g. intermediate, advanced).One participant used this as their primary model.

Other patterns, themes, and notable terminology:

  • Seven participants grouped together content related to hosting or attending events, and three participants made references to face-to-face communication. Of the seven who grouped content into the “Events” topic, five of them included the one item that referenced “Maker Party” (including two participants who rated their level of familiarity with the program at a 1), indicating a strong understanding of “Maker Party” as a type of event.
  • Five participants made references to the broader community. Three of them are from the United States, one from Canada, and one from India. (The specific terminology used were “Meet others,” “Social,” “Webmaker Community,” “Collaborate,” and “Supportive team”).
  • Four participants used the word “Webmaker” in their groupings, which gives us some insight into how they understand the brand. In each case, participants seem to connect the term to either teaching and teaching kits, or to the community of interested people.
  • Three participants used the term “Leading.”
  • One participant referenced a particular context (“Webmaker for Schools”).
  • One participant distinguished Mozilla-produced content (as “Mozilla Outputs”).
  • We included the term “Peer Learning Networks” in the content list to represent Hives (we assumed the meaning of “Hive” would be difficult to intuit for those unfamiliar). While we can’t draw any conclusions based on this data, it’s notable that this term was grouped into a wide variety of topics, including community (“Meet others,” “Social,” and “Collaborate”), “Get Involved,” “Intermediate,” “Mozilla Outputs,” and “Learning Geeks.” Three participants felt it didn’t fit under any category.
  • We tested both “Professional Development” and “Trainings” to see if we could understand how people interpret those terms. The results are fairly ambiguous. Both terms were associated with “Activities for teachers & mentors”, “Leading,” “Get Involved,” and “Research (things you learn on your own).” “Professional Development” was also associated with “Learning,” “Develop Yourself,” and “Learning Geeks”. “Trainings” was associated with “Intermediate,” “Mentoring,” “Organize in person events,” and “Supportive team.” For both terms, three participants could not categorize this term.

Let me know if you’re interested in seeing the raw data.

I need to reprint my business cards

Okay, so I don’t actually have business cards, but on this morning’s All Mofo call, it was announced that I’m leaving the wonderful Engagement team to serve as a product manager for the equally wonderful Learning Networks team. So, if I had business cards, I’d need new ones.

This is, for obvious reasons, bittersweet. I’ve LOVED working with the engaging folks on the Engagement team, and it provided a fantastic vantage point for learning the ins and outs of Mofo. I’m sending a big heartfelt thank you to Geoffrey and Co. for being so dang awsm to me ever since I joined.

Fortunately for me, I’m not going far. I’ve been admiring the work of both the Mentor Networks and the Product teams from a distance, so I’m thrilled with my new spot right smack in the middle of them.

Wait….Hannah, what do you know about product management? And Learning Networks?

It might seem strange at first blush, since I’ve been talking about scrum mastering and engagement-y stuff on this blog so far. But, lest you think I’m totally unqualified, let me share a few relevant experiences I haven’t shared here before:

  • I was a Product Owner at my last job. “Product Owner” is a title specific to Scrum shops, but it’s got a whole lot in common with Product Manager. Working with devs? Check! UI and UX designers? Yup, them too. End users and stakeholders? Love ‘em. Caring about product adoption rates, product marketing, and customer service? For sure, uh huh, no doubt.
  • I have experience shepherding a product to serve a local groups-based organizing model, much like the vision for Webmaker Clubs. I hope to bring useful knowledge from that project, drawing on both successes and failures (because, hey, “what not to do” lists are useful, too!)
  • While I won’t be contributing in this way, I do have a bit of experience as a trainer—I’ve developed and delivered service-learning and social justice careers curricula to kids, college students, and adults. I’m nowhere near as savvy as our #TeachTheWeb team, but I can promise to keep the needs of the mentors at the forefront of my brain.

OK, so what are you going to be working on?

The Big Picture answer is: developing products that serve the needs of our constituents in our ground game programs (Hive, Webmaker Clubs, Maker Parties). These products will be separate from, but complementary of, the Learning Products, which serve independent learners who aren’t (yet!) affiliated with our ground game programs.

In the short term, my top priorities are:

  • Building a new home for all of our teaching resources and a launching point for all of our ground game programs ( v1 will be a re-org of our existing content, so we’ve launched a small user research study (you may have seen Lucy’s recent email to the Webmaker listserv asking for volunteers). The participants are doing a virtual version of what’s called a “card sorting” activity to help us understand their mental models around all of the content we currently have. The results will inform the information architecture for Teach.w.o v1.
  • Launching a platform for local groups (i.e. Clubs and potentially Hives). Q1 is about two flavors of research: 1) Developing a deep, nuanced understanding of our own business needs and the needs of our users—in this case, the Club Leaders in the Q1 Pilot. 2) Investigating off-the-shelf options for the platform.
  • Iterating on our credentials platform. Again, this starts with developing a deep, shared understanding of business needs. Stakeholder kick-off meeting coming soon!

I’m so very excited to be working on these things. Like, I’m seriously being a nerd about it all.

At the same time, I’m already missing my Engagement team buddies (though that’s tempered by the fact that I still get to work with nearly all of them :)).

Questions? Want to discuss the Learning Networks products? Hit me up.