User Testing the Teach Site

We are soooooo close to releasing the new Teach site.

People seem to dig the bright colors and quirky illustrations throughout the site.

People seem to dig the bright colors and quirky illustrations throughout the site.

In advance of the release, I wanted to conduct some user tests to make sure we’re still on the right track. This week I conducted two user tests with members of the community (yay!). As is always the case with user testing, I learned a lot from observing users interact with the site.

You can see detailed notes here and read my recommendations below.

These recommendations are based on formal user tests with two users as well as feedback from people who’ve been involved or observing the process throughout.  Also, please note that I wasn’t able to test the primary functionality on the site (adding a Club to the map), so these recommendations are more about IA and other content issues.

Findings and Recommendations

Screen Shot 2015-04-10 at 2.48.07 PM

ACTIVITIES / CURRICULUM / RESOURCES

Findings:

  • People want to see more activities and resources.
  • People expect to be able to sort and filter.
  • Our internal distinction between the Clubs Curriculum (the official curriculum for Clubs; with a strong recommendation for following the prescribed path) and Teaching Activities (more “grab and go”-style) is not intuitive to users.
  • The Teach Like Mozilla content needs to be more integrated into common user flows.

Recommendations:

  • Continue with current plan for developing and publishing more approved curriculum and activities.
  • Continue brainstorming work around scalable presentation of curriculum begun in this heartbeat. The ideas discussed so far address sorting and filtering, and make good use of the Web Literacy Map as an organizing tool.
  • As part of that design work, we should also allow users to access all teaching materials from the same page, and provide specific views for “official Clubs curriculum.” I recommend we keep the Teaching Activities page, and remove the Clubs Curriculum sub-page. This content is one of our primary offerings so it belongs at the top level. /cc @iamjessklein
  • We need to offer a solution for sharing resources—e.g. maker tools, other curricula, programs. (Hello, Web Lit Mapper!)
  • We need to design a stronger connection between teaching activities and the Teach Like Mozilla content. A short-term solution might be to link to the TLM page from every individual activity page, but we should also be working towards a better longer-term solution. /cc @laurahilliger

Screen Shot 2015-04-10 at 2.50.46 PM

CLUBS

Findings:

  • The Clubs Toolkit is not findable, and needs to be supplemented with content targeted towards helping people “get started.”
  • We are not providing enough information for the use case of a person who is deciding in the moment whether to start a Club.

Recommendations:

  • Make the Clubs Toolkit more visible on the page.
  • Consider renaming the Clubs Toolkit something like “Getting Started Guide” or “A Club’s First Month” – and editing content to match. /cc @thornet
  • Based on my understanding of the expected pathways to starting Clubs, I do not think we need to make any significant changes to the Clubs page to address the use case of someone coming to the site and deciding in the moment whether or not to start a Club. As I understand it, our plan for growing Clubs makes use of the following scenarios:
    • 1) Someone is “groomed” by staff member, Regional Coordinator, or other community member. By the time they arrive at the site, they have the specific intent of adding their Club.
    • 2) Someone finds out about us through Maker Party, and through a series of communications learns about Clubs and decides to start one. They are coming to the site with the specific intent to add their Club.
    • 3) Someone with an existing program or group wants to be listed in the database. Again, they are coming to the site with the intent to add their Club.

In short, I don’t think we’ve yet seen a reason to have the site serve a “selling” or persuasive function. I *do* think the Clubs page is a natural first stop for someone who is looking to understand how to start a Club. I think the changes recommended in the bullet points above address that.

Screen Shot 2015-04-10 at 2.51.21 PM

EVENTS/MAKER PARTY

Findings:

  • The copy describing the Events page in the main navigation is misleading, since the content on the Events page is about Maker Party.
  • People may understand throwing a Maker Party as a “first step” to starting a Club, rather than a lower-bar option for people who do not want to start a Club (and perhaps never will).

Recommendations:

  • I think we should re-brand what is currently the Events landing page as “Maker Party.” We’ve already sort of done this in that, while the page is called “Events” in the nav, the h1 copy in the hero image is “Host a Maker Party.” I suggest we change the copy in the nav to “MAKER PARTY” and the teaser copy to “Our annual global campaign”. /cc @amirad

Screen Shot 2015-04-10 at 2.52.06 PM

INFORMATION ARCHITECTURE

Findings:

  • Users tend to ignore, not see, or misinterpret the CTAs at the bottom of every page
  • Users do not notice links to the sub-pages in the main navigation

Recommendation:

  • We need to design better, more intuitive pathways for viewing secondary pages

ADDITIONAL RECOMMENDATIONS

I’m going to keep banging this drum: We need to clarify our audience! I think we’ve made good progress in terms of clarifying that our “first line” audience includes educators and activists. But I think we have to take it a step further and clarify who those educators and activists are working with. There are at least two axes that I think are important to be clear about: first, the global nature of our work, and second, the specific age groups of what I’m calling the “end learners,” for lack of a better term.

I think we do a pretty decent job of conveying the global nature of the program through copy and imagery, though obviously implementing our l10n strategy is absolutely fundamental to this.

I think we are less clear when it comes to the age groups we’re targeting with our programs and materials.   For example, I think we ought to specify the appropriate age level for each activity. (And the images, activity titles, and copy should reflect the target audience.)

Questions, comments, disagreements wholeheartedly welcomed!

Teach.webmaker.org: Initial Card Sorting Results

This past week I conducted a small user research project to help inform the IA of the new teach.webmaker.org site.

I chose a card sorting activity, which is a common research method for IA projects. In a card sorting activity, you give members of your target audience a stack of cards, each of which has one of the site content areas printed on it. You ask the participants to group items together and explain their thought process. In this way, you gain an understanding of the participants mental models. This is helpful for avoiding a common pitfall in site design, which is organizing content in a way that make sense to you but not your users.

Big Giant Caveat

This study was flawed in a couple ways. First, Jacob Nielsen (who is generally considered to be a real smartypants when it comes to usability and user research) recommends that you do card sorting with 15 users. I’ve only been able to get 11 to do the activity so far, though I think a few more are pending.

Another flaw is that I deviated from a common best practice of running these activities in person. A lot of the insights are gained by listening to the person think aloud. There are some tools for running an online card sorting activity, but they’re largely for what’s called “closed” card sorts, where you pre-determine the categories and the person’s task is to sort cards within those categories. Since one of my goals with this activity was to generate a better understanding of what terminology to use, I wanted to do an “open” sort, where the participants name their groupings themselves.

All that’s to say that we shouldn’t take these results or my analysis as gospel. I do think the participant responses will be useful as we move forward with designing some wireframes to user test in the next heartbeat.

Participant Demographics and Background Information

There were a range of ages and locations represented in the study.

Four participants are between 18 and 24 years old, three are between 25 and 34, two between 35 and 44, one between 45 and 54, and one between 55 and 64.

Four participants are from the United States, three from India, and one each from Colombia, Bangladesh, Canada, and the United Kingdom.

Participants were asked to rate their level of familiarity with the Webmaker Mentors program on a scale of 1 to 5, with 5 being the most familiar. Again, there was a range. Four participants rated themselves a 5, two a 4 or 4.5, two a 3, one a 2, and two a 1.

Initial Findings

The participants in the study had a range of different mental models they used to organize the content. Those models were:

  1. Grouping by program offering—that is, organizing by specific programs, concepts, or offerings, typically expressed as nouns (e.g. Web Literacy, Teaching Kits, Webmaker Clubs, Trainings, Activities, Resources, Social, Learning, Philosophy, Mentoring, Research, Events, Supportive Team)Five participants used a model like this as their primary model. The average familiarity level with Webmaker Mentoring for these participants matches the average for the entire sample (3.7 on a five-point scale).
  2. Grouping by functional area—that is, actions that a user might take, typically expressed as verbs (e.g. participate, learn, market/promote, meet others, do, lead, get involved, collaborate, organize, develop yourself, teach, experiment, host, attend).Four participants used a model like this as their primary model. Notably, all of the participants are from the United States, Canada, or the United Kingdom, and their average familiarity with Webmaker Mentoring is below the average of the entire sample (2.75 as compared to 3.67).
  3. Grouping by role or identity—some study participants organized the content by the type of user who would be interested in it (e.g. Learner, Mentor).One participant  used this as their primary model. Another made a distinction between Learning and Teaching, but it was framed more like the functional areas described above. One more used “Learning Geeks” as a topic area.
  4. Level of expertise—in this model, there is a pathway through the content based on level of expertise (e.g. intermediate, advanced).One participant used this as their primary model.

Other patterns, themes, and notable terminology:

  • Seven participants grouped together content related to hosting or attending events, and three participants made references to face-to-face communication. Of the seven who grouped content into the “Events” topic, five of them included the one item that referenced “Maker Party” (including two participants who rated their level of familiarity with the program at a 1), indicating a strong understanding of “Maker Party” as a type of event.
  • Five participants made references to the broader community. Three of them are from the United States, one from Canada, and one from India. (The specific terminology used were “Meet others,” “Social,” “Webmaker Community,” “Collaborate,” and “Supportive team”).
  • Four participants used the word “Webmaker” in their groupings, which gives us some insight into how they understand the brand. In each case, participants seem to connect the term to either teaching and teaching kits, or to the community of interested people.
  • Three participants used the term “Leading.”
  • One participant referenced a particular context (“Webmaker for Schools”).
  • One participant distinguished Mozilla-produced content (as “Mozilla Outputs”).
  • We included the term “Peer Learning Networks” in the content list to represent Hives (we assumed the meaning of “Hive” would be difficult to intuit for those unfamiliar). While we can’t draw any conclusions based on this data, it’s notable that this term was grouped into a wide variety of topics, including community (“Meet others,” “Social,” and “Collaborate”), “Get Involved,” “Intermediate,” “Mozilla Outputs,” and “Learning Geeks.” Three participants felt it didn’t fit under any category.
  • We tested both “Professional Development” and “Trainings” to see if we could understand how people interpret those terms. The results are fairly ambiguous. Both terms were associated with “Activities for teachers & mentors”, “Leading,” “Get Involved,” and “Research (things you learn on your own).” “Professional Development” was also associated with “Learning,” “Develop Yourself,” and “Learning Geeks”. “Trainings” was associated with “Intermediate,” “Mentoring,” “Organize in person events,” and “Supportive team.” For both terms, three participants could not categorize this term.

Let me know if you’re interested in seeing the raw data.

I need to reprint my business cards

Okay, so I don’t actually have business cards, but on this morning’s All Mofo call, it was announced that I’m leaving the wonderful Engagement team to serve as a product manager for the equally wonderful Learning Networks team. So, if I had business cards, I’d need new ones.

This is, for obvious reasons, bittersweet. I’ve LOVED working with the engaging folks on the Engagement team, and it provided a fantastic vantage point for learning the ins and outs of Mofo. I’m sending a big heartfelt thank you to Geoffrey and Co. for being so dang awsm to me ever since I joined.

Fortunately for me, I’m not going far. I’ve been admiring the work of both the Mentor Networks and the Product teams from a distance, so I’m thrilled with my new spot right smack in the middle of them.

Wait….Hannah, what do you know about product management? And Learning Networks?

It might seem strange at first blush, since I’ve been talking about scrum mastering and engagement-y stuff on this blog so far. But, lest you think I’m totally unqualified, let me share a few relevant experiences I haven’t shared here before:

  • I was a Product Owner at my last job. “Product Owner” is a title specific to Scrum shops, but it’s got a whole lot in common with Product Manager. Working with devs? Check! UI and UX designers? Yup, them too. End users and stakeholders? Love ‘em. Caring about product adoption rates, product marketing, and customer service? For sure, uh huh, no doubt.
  • I have experience shepherding a product to serve a local groups-based organizing model, much like the vision for Webmaker Clubs. I hope to bring useful knowledge from that project, drawing on both successes and failures (because, hey, “what not to do” lists are useful, too!)
  • While I won’t be contributing in this way, I do have a bit of experience as a trainer—I’ve developed and delivered service-learning and social justice careers curricula to kids, college students, and adults. I’m nowhere near as savvy as our #TeachTheWeb team, but I can promise to keep the needs of the mentors at the forefront of my brain.

OK, so what are you going to be working on?

The Big Picture answer is: developing products that serve the needs of our constituents in our ground game programs (Hive, Webmaker Clubs, Maker Parties). These products will be separate from, but complementary of, the Learning Products, which serve independent learners who aren’t (yet!) affiliated with our ground game programs.

In the short term, my top priorities are:

  • Building a new home for all of our teaching resources and a launching point for all of our ground game programs (teach.webmaker.org). v1 will be a re-org of our existing content, so we’ve launched a small user research study (you may have seen Lucy’s recent email to the Webmaker listserv asking for volunteers). The participants are doing a virtual version of what’s called a “card sorting” activity to help us understand their mental models around all of the content we currently have. The results will inform the information architecture for Teach.w.o v1.
  • Launching a platform for local groups (i.e. Clubs and potentially Hives). Q1 is about two flavors of research: 1) Developing a deep, nuanced understanding of our own business needs and the needs of our users—in this case, the Club Leaders in the Q1 Pilot. 2) Investigating off-the-shelf options for the platform.
  • Iterating on our credentials platform. Again, this starts with developing a deep, shared understanding of business needs. Stakeholder kick-off meeting coming soon!

I’m so very excited to be working on these things. Like, I’m seriously being a nerd about it all.

At the same time, I’m already missing my Engagement team buddies (though that’s tempered by the fact that I still get to work with nearly all of them :)).

Questions? Want to discuss the Learning Networks products? Hit me up.

We are very engaging

Yesterday someone asked me what the engagement team is up to, and it made me sad because I realized I need to do a waaaaay better job of broadcasting my team’s work. This team is dope and you need to know about it.

As a refresher, our work encompasses these areas:

  • Grantwriting
  • Institutional partnerships
  • Marketing and communications
  • Small dollar fundraising
  • Production work (i.e. Studio Mofo)

In short, we aim to support the Webmaker product and programs and our leadership pipelines any time we need to engage individuals or institutions.

What’s currently on our plate:

Pro-tip: You can always see what we’re up to by checking out the Engagement Team Workbench.

These days we’re spending our time on the following:

  • End of Year Fundraising: With the help of a slew of kick-ass engineers, Andrea and Kelli are getting to $2M. (view the Workbench).
  • Mozilla Gear launch: Andrea and Geoffrey are obsessed with branded hoodies. To complement our fundraising efforts, they just opened a brand new site for people to purchase Mozilla Gear (view the project management spreadsheet).
  • Fall Campaign: Remember the 10K contributor goal? We do! An-Me and Paul have been working with Claw, Amira, Michelle, and Lainie, among others, to close the gap through a partner-based strategy (view the Workbench).
  • Mobile Opportunity: Ben is helping to envision and build partnerships around this work, and Paul and Studio Mofo are providing marketing, comms, and production support (the Mobile Opportunity Workbench is here, the engagement-specific work will be detailed soon).
  • Building a Webmaker Marketing Plan for 2015: The site and programs aren’t going to market themselves! Paul is drafting a comprehensive marketing calendar for 2015 that complements the product and program strategies. (plan coming soon)
  • 2015 Grants Pipeline: Ben and An-Me are always on the lookout for opportunities, and Lynn is responsible for writing grants and reports to fund our various programs and initiatives.
  • Additional Studio Mofo projects: Erika, Mavis, and Sabrina are always working on something. In addition to their work supporting most of the above, you can see a full list of projects here.
  • Salesforce for grants and partnerships: We’ve completed a custom Salesforce installation and Ben has begun the process of training staff to use it. Much more to come to make it a meaningful part of our workflow (Workbench coming soon).
  • Open Web Fellows recruitment: We’re supporting our newest fellowship with marketing support (view the Hype Plan)

How to Mofo

OpenMatt and I have been talking about the various ways of working at Mofo, and we compiled this list of what we think works best. What do y’all Mofos think?

When starting a new project:

  • Clearly state the problem or goal. Don’t jump ahead to the solution. Ameliorating the problem is what you’ll measure success against, not your ability to implement an arbitrary solution.
  • Explicitly state assumptions. And, whenever possible, test those assumptions before you build anything. You may have assumptions about the nature of the problem you’re trying to solve, who’s experiencing it, or your proposed solution.
  • Have clear success metrics. How will you know if you’re winning? Do you have the instruments you need to measure success?
  • Determine what resources you need. Think about design, development, content, engagement, evaluation, and ongoing maintenance. We’re working on improving the ways we allocate resources throughout the organization, but to start, be clear about what resources your project will need.
  • Produce a project brief. Detail all of the above in a single document. (Example templates here and here.) Use the project brief when you…
  • …Have a project kick-off meeting. Invite *all* the stakeholders to get involved early.

Communication:

  • Have a check-in plan. Will you have daily check-ins? Weekly email updates? How are you checking in and holding each other accountable?
  • Build a workbench and keep it updated.  We recommend a wiki page that will serve as a one-stop shop for anyone needing information about the project. Things to include: links to project briefs and notes, logistics for meetings, a timeline, a list of who’s involved, and, of course, bugs! Examples here, here, and here.
  • Put your notes in one spot.  A single canonical pad for notes and agendas. We don’t need to create a  new pad every time you have a meeting or a thought! That makes them very hard to track and find later. Examples here and here.

Doing the do:

  • Plan in two-week heartbeats. This helps us stay on track and makes it clear what the priorities are. Speaking of priorities…
  • Learn the Fine Art of Prioritizing. Hint: Not everything can be P1. The product owner or project manager should rank tasks in order of value added. Remember: prioritization is part of managing workflow. It may be true that all or most of the tasks are required for a successful launch, but that doesn’t help a developer or designer who’s trying to decide what to work on next.
  • Work with your friendly neighborhood Tactical Priorities Syndicate. The name sounds scary, but they’re here to serve you. They meet weekly to get your priorities into each two-week heartbeat process. https://wiki.mozilla.org/Webmaker/TPS

Update: To hack on the next version of this, please visit http://workopen.org/mofo (thanks to Doug for the suggestion!)

Talking to people

At the beginning of October, I went to Austin for the Digital PM Summit, which is an amazingly useful gathering of digital project managers, now in its second year. I was invited to speak about Retrospectives—my favorite topic! I enjoyed putting together a presentation and talking with some really talented PMs about how to create a culture of experimentation and continuous improvement.

Then, earlier this week, I had the opportunity to talk with the very smart and fun YNPN Launchpad Fellows about how to apply Agile methodologies to non-technical projects in nonprofit organizations. I’m becoming a little obsessed with non-technical applications of Agile (see ScrumYourWedding, coming soon!)

I <3 talking to people!

Kayaks, Nachos, Pipelines, and Funnels, or: The Mofo Engagement Fall Work Week

Last week, the Mofo Engagement Team and Friends (terrible band name) met in Toronto for some some serious boating, eating, and hacking.

Prelude: With some much appreciated assistance from Erika, I got over my fear of boats and managed to canoe up and down a bit of the Humber river during our pre-work week Super Engagement Team Fun Day. We may not have been fast, but we had style (assuming your definition of style includes crashing into the riverbank). Later, I tricked several of my co-workers into ordering giant platters of nachos at Sneaky Dee’s. It was a cheesy, oversized start to the work week.

On to the work-y part!

The agenda was ambitious. We had four concurrent tracks, each with their own projected outcomes:

  • The set-up for the End of Year fundraising campaign
  • The creative brief, RACI, and roadmap for the Fall Webmaker Sales Campaign
  • The 2015 Grants Pipeline
  • And the partnership strategy, systems, and sales team for growing Webmaker

Did we achieve what we wanted to achieve?

All that, and more.

The End of Year Fundraising team was on fire. The project had been well-prepped in advance, so the team was able to use the work week as a sprint and deliver a slew of prototypes and designs. They tackled the snippet, optimized donation forms including a brand new sequential flow, localization, the fundraising.mozilla.org website redesign, overarching branding, and even an awesome community marketing idea.

The Partnerships team had several rich conversations where they identified possible partners, clarified roles, and simulated the entire sales flow using human props, sticky notes, and impressive improv skills. (I left that session with a post-it note on my laptop that read, “Why would clown mentors come back to the site?” A question for the ages.)

A centerpiece of the Fall Webmaker Campaign is the post-sales funnel which includes custom partner landing pages and a choose-your-own-adventure style event wizard to help people get started with one of three easy/fun Maker Party events. The entire funnel got spec’ed and wireframed and the Event Wizard got some design love during the work week.

(Note: The Partnerships track and the Fall Webmaker Campaign track were largely merged into a single Fall Webmaker Campaign with elements including sales, marketing, design, and product dev. The campaign is focused on reaching our 10K contributor goal, and will leverage key partners with large networks. In addition to honing our value proposition to partners, we’ll also use the opportunity to refine our marketing funnel. Later, we’ll go out and introduce a broader addressable audience to the top of the funnel. At that point, there will be a clear distinction between sales and marketing, but for the Fall campaign, we’re working in tandem.)

The 2015 Grants Pipeline team got to spend some quality time with our brand new Salesforce installation. They make web-to-lead forms, trigger events, and dashboards seem downright glamorous!

Phew.

Does this seem like a lot of stuff? That’s because it is a lot of stuff! But never fear, it’s all summarized on the Engagement Team Workbench. (You can also see a complete list of what we delivered during the work week here.)

I was amazed at how much was accomplished during the work week. My co-workers are some of the raddest, most capable and action-oriented people I’ve had the privilege of knowing. Lucky me to be a part of it!