Teach.webmaker.org: Initial Card Sorting Results

This past week I conducted a small user research project to help inform the IA of the new teach.webmaker.org site.

I chose a card sorting activity, which is a common research method for IA projects. In a card sorting activity, you give members of your target audience a stack of cards, each of which has one of the site content areas printed on it. You ask the participants to group items together and explain their thought process. In this way, you gain an understanding of the participants mental models. This is helpful for avoiding a common pitfall in site design, which is organizing content in a way that make sense to you but not your users.

Big Giant Caveat

This study was flawed in a couple ways. First, Jacob Nielsen (who is generally considered to be a real smartypants when it comes to usability and user research) recommends that you do card sorting with 15 users. I’ve only been able to get 11 to do the activity so far, though I think a few more are pending.

Another flaw is that I deviated from a common best practice of running these activities in person. A lot of the insights are gained by listening to the person think aloud. There are some tools for running an online card sorting activity, but they’re largely for what’s called “closed” card sorts, where you pre-determine the categories and the person’s task is to sort cards within those categories. Since one of my goals with this activity was to generate a better understanding of what terminology to use, I wanted to do an “open” sort, where the participants name their groupings themselves.

All that’s to say that we shouldn’t take these results or my analysis as gospel. I do think the participant responses will be useful as we move forward with designing some wireframes to user test in the next heartbeat.

Participant Demographics and Background Information

There were a range of ages and locations represented in the study.

Four participants are between 18 and 24 years old, three are between 25 and 34, two between 35 and 44, one between 45 and 54, and one between 55 and 64.

Four participants are from the United States, three from India, and one each from Colombia, Bangladesh, Canada, and the United Kingdom.

Participants were asked to rate their level of familiarity with the Webmaker Mentors program on a scale of 1 to 5, with 5 being the most familiar. Again, there was a range. Four participants rated themselves a 5, two a 4 or 4.5, two a 3, one a 2, and two a 1.

Initial Findings

The participants in the study had a range of different mental models they used to organize the content. Those models were:

  1. Grouping by program offering—that is, organizing by specific programs, concepts, or offerings, typically expressed as nouns (e.g. Web Literacy, Teaching Kits, Webmaker Clubs, Trainings, Activities, Resources, Social, Learning, Philosophy, Mentoring, Research, Events, Supportive Team)Five participants used a model like this as their primary model. The average familiarity level with Webmaker Mentoring for these participants matches the average for the entire sample (3.7 on a five-point scale).
  2. Grouping by functional area—that is, actions that a user might take, typically expressed as verbs (e.g. participate, learn, market/promote, meet others, do, lead, get involved, collaborate, organize, develop yourself, teach, experiment, host, attend).Four participants used a model like this as their primary model. Notably, all of the participants are from the United States, Canada, or the United Kingdom, and their average familiarity with Webmaker Mentoring is below the average of the entire sample (2.75 as compared to 3.67).
  3. Grouping by role or identity—some study participants organized the content by the type of user who would be interested in it (e.g. Learner, Mentor).One participant  used this as their primary model. Another made a distinction between Learning and Teaching, but it was framed more like the functional areas described above. One more used “Learning Geeks” as a topic area.
  4. Level of expertise—in this model, there is a pathway through the content based on level of expertise (e.g. intermediate, advanced).One participant used this as their primary model.

Other patterns, themes, and notable terminology:

  • Seven participants grouped together content related to hosting or attending events, and three participants made references to face-to-face communication. Of the seven who grouped content into the “Events” topic, five of them included the one item that referenced “Maker Party” (including two participants who rated their level of familiarity with the program at a 1), indicating a strong understanding of “Maker Party” as a type of event.
  • Five participants made references to the broader community. Three of them are from the United States, one from Canada, and one from India. (The specific terminology used were “Meet others,” “Social,” “Webmaker Community,” “Collaborate,” and “Supportive team”).
  • Four participants used the word “Webmaker” in their groupings, which gives us some insight into how they understand the brand. In each case, participants seem to connect the term to either teaching and teaching kits, or to the community of interested people.
  • Three participants used the term “Leading.”
  • One participant referenced a particular context (“Webmaker for Schools”).
  • One participant distinguished Mozilla-produced content (as “Mozilla Outputs”).
  • We included the term “Peer Learning Networks” in the content list to represent Hives (we assumed the meaning of “Hive” would be difficult to intuit for those unfamiliar). While we can’t draw any conclusions based on this data, it’s notable that this term was grouped into a wide variety of topics, including community (“Meet others,” “Social,” and “Collaborate”), “Get Involved,” “Intermediate,” “Mozilla Outputs,” and “Learning Geeks.” Three participants felt it didn’t fit under any category.
  • We tested both “Professional Development” and “Trainings” to see if we could understand how people interpret those terms. The results are fairly ambiguous. Both terms were associated with “Activities for teachers & mentors”, “Leading,” “Get Involved,” and “Research (things you learn on your own).” “Professional Development” was also associated with “Learning,” “Develop Yourself,” and “Learning Geeks”. “Trainings” was associated with “Intermediate,” “Mentoring,” “Organize in person events,” and “Supportive team.” For both terms, three participants could not categorize this term.

Let me know if you’re interested in seeing the raw data.

I need to reprint my business cards

Okay, so I don’t actually have business cards, but on this morning’s All Mofo call, it was announced that I’m leaving the wonderful Engagement team to serve as a product manager for the equally wonderful Learning Networks team. So, if I had business cards, I’d need new ones.

This is, for obvious reasons, bittersweet. I’ve LOVED working with the engaging folks on the Engagement team, and it provided a fantastic vantage point for learning the ins and outs of Mofo. I’m sending a big heartfelt thank you to Geoffrey and Co. for being so dang awsm to me ever since I joined.

Fortunately for me, I’m not going far. I’ve been admiring the work of both the Mentor Networks and the Product teams from a distance, so I’m thrilled with my new spot right smack in the middle of them.

Wait….Hannah, what do you know about product management? And Learning Networks?

It might seem strange at first blush, since I’ve been talking about scrum mastering and engagement-y stuff on this blog so far. But, lest you think I’m totally unqualified, let me share a few relevant experiences I haven’t shared here before:

  • I was a Product Owner at my last job. “Product Owner” is a title specific to Scrum shops, but it’s got a whole lot in common with Product Manager. Working with devs? Check! UI and UX designers? Yup, them too. End users and stakeholders? Love ‘em. Caring about product adoption rates, product marketing, and customer service? For sure, uh huh, no doubt.
  • I have experience shepherding a product to serve a local groups-based organizing model, much like the vision for Webmaker Clubs. I hope to bring useful knowledge from that project, drawing on both successes and failures (because, hey, “what not to do” lists are useful, too!)
  • While I won’t be contributing in this way, I do have a bit of experience as a trainer—I’ve developed and delivered service-learning and social justice careers curricula to kids, college students, and adults. I’m nowhere near as savvy as our #TeachTheWeb team, but I can promise to keep the needs of the mentors at the forefront of my brain.

OK, so what are you going to be working on?

The Big Picture answer is: developing products that serve the needs of our constituents in our ground game programs (Hive, Webmaker Clubs, Maker Parties). These products will be separate from, but complementary of, the Learning Products, which serve independent learners who aren’t (yet!) affiliated with our ground game programs.

In the short term, my top priorities are:

  • Building a new home for all of our teaching resources and a launching point for all of our ground game programs (teach.webmaker.org). v1 will be a re-org of our existing content, so we’ve launched a small user research study (you may have seen Lucy’s recent email to the Webmaker listserv asking for volunteers). The participants are doing a virtual version of what’s called a “card sorting” activity to help us understand their mental models around all of the content we currently have. The results will inform the information architecture for Teach.w.o v1.
  • Launching a platform for local groups (i.e. Clubs and potentially Hives). Q1 is about two flavors of research: 1) Developing a deep, nuanced understanding of our own business needs and the needs of our users—in this case, the Club Leaders in the Q1 Pilot. 2) Investigating off-the-shelf options for the platform.
  • Iterating on our credentials platform. Again, this starts with developing a deep, shared understanding of business needs. Stakeholder kick-off meeting coming soon!

I’m so very excited to be working on these things. Like, I’m seriously being a nerd about it all.

At the same time, I’m already missing my Engagement team buddies (though that’s tempered by the fact that I still get to work with nearly all of them :)).

Questions? Want to discuss the Learning Networks products? Hit me up.

We are very engaging

Yesterday someone asked me what the engagement team is up to, and it made me sad because I realized I need to do a waaaaay better job of broadcasting my team’s work. This team is dope and you need to know about it.

As a refresher, our work encompasses these areas:

  • Grantwriting
  • Institutional partnerships
  • Marketing and communications
  • Small dollar fundraising
  • Production work (i.e. Studio Mofo)

In short, we aim to support the Webmaker product and programs and our leadership pipelines any time we need to engage individuals or institutions.

What’s currently on our plate:

Pro-tip: You can always see what we’re up to by checking out the Engagement Team Workbench.

These days we’re spending our time on the following:

  • End of Year Fundraising: With the help of a slew of kick-ass engineers, Andrea and Kelli are getting to $2M. (view the Workbench).
  • Mozilla Gear launch: Andrea and Geoffrey are obsessed with branded hoodies. To complement our fundraising efforts, they just opened a brand new site for people to purchase Mozilla Gear (view the project management spreadsheet).
  • Fall Campaign: Remember the 10K contributor goal? We do! An-Me and Paul have been working with Claw, Amira, Michelle, and Lainie, among others, to close the gap through a partner-based strategy (view the Workbench).
  • Mobile Opportunity: Ben is helping to envision and build partnerships around this work, and Paul and Studio Mofo are providing marketing, comms, and production support (the Mobile Opportunity Workbench is here, the engagement-specific work will be detailed soon).
  • Building a Webmaker Marketing Plan for 2015: The site and programs aren’t going to market themselves! Paul is drafting a comprehensive marketing calendar for 2015 that complements the product and program strategies. (plan coming soon)
  • 2015 Grants Pipeline: Ben and An-Me are always on the lookout for opportunities, and Lynn is responsible for writing grants and reports to fund our various programs and initiatives.
  • Additional Studio Mofo projects: Erika, Mavis, and Sabrina are always working on something. In addition to their work supporting most of the above, you can see a full list of projects here.
  • Salesforce for grants and partnerships: We’ve completed a custom Salesforce installation and Ben has begun the process of training staff to use it. Much more to come to make it a meaningful part of our workflow (Workbench coming soon).
  • Open Web Fellows recruitment: We’re supporting our newest fellowship with marketing support (view the Hype Plan)

How to Mofo

OpenMatt and I have been talking about the various ways of working at Mofo, and we compiled this list of what we think works best. What do y’all Mofos think?

When starting a new project:

  • Clearly state the problem or goal. Don’t jump ahead to the solution. Ameliorating the problem is what you’ll measure success against, not your ability to implement an arbitrary solution.
  • Explicitly state assumptions. And, whenever possible, test those assumptions before you build anything. You may have assumptions about the nature of the problem you’re trying to solve, who’s experiencing it, or your proposed solution.
  • Have clear success metrics. How will you know if you’re winning? Do you have the instruments you need to measure success?
  • Determine what resources you need. Think about design, development, content, engagement, evaluation, and ongoing maintenance. We’re working on improving the ways we allocate resources throughout the organization, but to start, be clear about what resources your project will need.
  • Produce a project brief. Detail all of the above in a single document. (Example templates here and here.) Use the project brief when you…
  • …Have a project kick-off meeting. Invite *all* the stakeholders to get involved early.

Communication:

  • Have a check-in plan. Will you have daily check-ins? Weekly email updates? How are you checking in and holding each other accountable?
  • Build a workbench and keep it updated.  We recommend a wiki page that will serve as a one-stop shop for anyone needing information about the project. Things to include: links to project briefs and notes, logistics for meetings, a timeline, a list of who’s involved, and, of course, bugs! Examples here, here, and here.
  • Put your notes in one spot.  A single canonical pad for notes and agendas. We don’t need to create a  new pad every time you have a meeting or a thought! That makes them very hard to track and find later. Examples here and here.

Doing the do:

  • Plan in two-week heartbeats. This helps us stay on track and makes it clear what the priorities are. Speaking of priorities…
  • Learn the Fine Art of Prioritizing. Hint: Not everything can be P1. The product owner or project manager should rank tasks in order of value added. Remember: prioritization is part of managing workflow. It may be true that all or most of the tasks are required for a successful launch, but that doesn’t help a developer or designer who’s trying to decide what to work on next.
  • Work with your friendly neighborhood Tactical Priorities Syndicate. The name sounds scary, but they’re here to serve you. They meet weekly to get your priorities into each two-week heartbeat process. https://wiki.mozilla.org/Webmaker/TPS

Update: To hack on the next version of this, please visit http://workopen.org/mofo (thanks to Doug for the suggestion!)

Talking to people

At the beginning of October, I went to Austin for the Digital PM Summit, which is an amazingly useful gathering of digital project managers, now in its second year. I was invited to speak about Retrospectives—my favorite topic! I enjoyed putting together a presentation and talking with some really talented PMs about how to create a culture of experimentation and continuous improvement.

Then, earlier this week, I had the opportunity to talk with the very smart and fun YNPN Launchpad Fellows about how to apply Agile methodologies to non-technical projects in nonprofit organizations. I’m becoming a little obsessed with non-technical applications of Agile (see ScrumYourWedding, coming soon!)

I <3 talking to people!

Kayaks, Nachos, Pipelines, and Funnels, or: The Mofo Engagement Fall Work Week

Last week, the Mofo Engagement Team and Friends (terrible band name) met in Toronto for some some serious boating, eating, and hacking.

Prelude: With some much appreciated assistance from Erika, I got over my fear of boats and managed to canoe up and down a bit of the Humber river during our pre-work week Super Engagement Team Fun Day. We may not have been fast, but we had style (assuming your definition of style includes crashing into the riverbank). Later, I tricked several of my co-workers into ordering giant platters of nachos at Sneaky Dee’s. It was a cheesy, oversized start to the work week.

On to the work-y part!

The agenda was ambitious. We had four concurrent tracks, each with their own projected outcomes:

  • The set-up for the End of Year fundraising campaign
  • The creative brief, RACI, and roadmap for the Fall Webmaker Sales Campaign
  • The 2015 Grants Pipeline
  • And the partnership strategy, systems, and sales team for growing Webmaker

Did we achieve what we wanted to achieve?

All that, and more.

The End of Year Fundraising team was on fire. The project had been well-prepped in advance, so the team was able to use the work week as a sprint and deliver a slew of prototypes and designs. They tackled the snippet, optimized donation forms including a brand new sequential flow, localization, the fundraising.mozilla.org website redesign, overarching branding, and even an awesome community marketing idea.

The Partnerships team had several rich conversations where they identified possible partners, clarified roles, and simulated the entire sales flow using human props, sticky notes, and impressive improv skills. (I left that session with a post-it note on my laptop that read, “Why would clown mentors come back to the site?” A question for the ages.)

A centerpiece of the Fall Webmaker Campaign is the post-sales funnel which includes custom partner landing pages and a choose-your-own-adventure style event wizard to help people get started with one of three easy/fun Maker Party events. The entire funnel got spec’ed and wireframed and the Event Wizard got some design love during the work week.

(Note: The Partnerships track and the Fall Webmaker Campaign track were largely merged into a single Fall Webmaker Campaign with elements including sales, marketing, design, and product dev. The campaign is focused on reaching our 10K contributor goal, and will leverage key partners with large networks. In addition to honing our value proposition to partners, we’ll also use the opportunity to refine our marketing funnel. Later, we’ll go out and introduce a broader addressable audience to the top of the funnel. At that point, there will be a clear distinction between sales and marketing, but for the Fall campaign, we’re working in tandem.)

The 2015 Grants Pipeline team got to spend some quality time with our brand new Salesforce installation. They make web-to-lead forms, trigger events, and dashboards seem downright glamorous!

Phew.

Does this seem like a lot of stuff? That’s because it is a lot of stuff! But never fear, it’s all summarized on the Engagement Team Workbench. (You can also see a complete list of what we delivered during the work week here.)

I was amazed at how much was accomplished during the work week. My co-workers are some of the raddest, most capable and action-oriented people I’ve had the privilege of knowing. Lucky me to be a part of it!

Maker Party Engagement Week 5

Week 5!

tl;dr highlights of the week:

  • Though we saw significant jumps in Wm accounts and events, our Contributor numbers did not increase accordingly
  • We’re identifying many opportunities from the partner calls
  • Hack the Snippet is coming soon, along with the next iteration of the snippet funnel
  • The TweetChat created a temporary increase in Twitter engagement, but took attention away from press

Overall stats:

  • Contributors: 5552 (2% increase from last week’s 5441)
  • Webmaker accounts: 124K (17% increase from last week’s 106.3K)
  • Events: 1799 (crazy 50% jump from last week’s 1199)
  • Hosts: 493 (10% increase from last week’s 450)
  • Expected attendees: 76,200  (23% increase from 61,910)
  • Cities: 362 (40% increase from 260 – what caused this jump?)
  • Traffic: here’s the last three weeks. We continue to see the major boost from the snippet.
 

traffic

  • And the Webmaker user account conversion rate increased a bit further:
 

conversion

——————————————————————–

Engagement Strategy #1: PARTNER OUTREACH

We are learning a lot from the partner calls. Here are some of the most salient takeaways (borrowing from Amira and Melissa’s notes during Friday’s call):

Partner trends
  • Partners see value in badging their event mentors, speakers and volunteers as a form of appreciation. But there is a potential for those who receive the badges to have no idea who is badging them or what it means (lack of connection to MP). Opportunity: We need to better explain to people why they’ve received a badge and why they might want to create a Webmaker account.
  • Partners are doing things but we just haven’t captured them.  Opportunity: We need to offer real value to users in order to increase the amount of sharing/broadcasting/badging that happens through the site. 
  • Some people need way more training — Opportunity: this is where the event wizard might play a role; there also might be an opportunity to run TTT within certain orgs and spaces.
  • We need to clarify our value statement for partners. It may not be in  adding numbers to their events or traction to their programs/site, or getting press for non-Hive partners. Instead it may be in providing resources and curriculum. We can better segment partners into affinity groups (e.g. afterschool programs) and provide content, trainings, resources, CTAs specifically for them.  We can also localize those offerings to reduce hand-holding.
  • People don’t understand how broad our definition of Maker Party is: everyday events, small events, stands/booths/tables within other events — have to push them to realize that and include all of these on the events platform (note from HK: I would argue we have to offer them a reason to)
  • Opportunity: There’s the summer wave and back to school waves. We need to have strategies and actions towards both.
  • Challenges:
    • Age and time continue to be a blocker for new Wm accounts.
    • Mass emails to order swag, upload events, share information just didn’t work. They need 1-to-1.
    • We lost interest by a lot of people along the way. There’s a good 20-30% we will not be able to bring back in.
    • Parties sound like fun kid-like things (making toys etc.)
    • Getting the Maker Party logo/brand included in event promotion in a meaningful way is not happening, and the meaning behind the brand seems to cause confusion in some cases.

PROMOTIONAL PARTNERS: We continue to see only a tiny amount of referrals from promotional partner urls with RIDs.

——————————————————————–

Engagement Strategy #2: ACTIVE MOZILLIANS

Haven’t heard anything this week, but Amira and I are meeting with the FSA Community Manager on Monday of this week.

——————————————————————–

Engagement Strategy #3: OWNED MEDIA

Snippet Funnel:

The snippet funnel continues to perform well in terms of driving traffic. We’re aiming to beat a baseline 1.8% conversion rate.

We were a bit blocked by technical issues this week and weren’t able to release the new tailored account signup pages, but we continue to work on that.

The “hack the snippet” test was delayed, but will be live soon. We have a comms strategy around it (for after it’s been tested).

——————————————————————–

Engagement Strategy #4: EARNED MEDIA

Press this week:

Aside from a cross-post of last week’s Washington Post Magazine story (http://www.tampabay.com/news/business/workinglife/want-a-tech-job-what-to-study-in-a-fast-moving-field/2193050), we didn’t see press this week. We were focused on our Net Neutrality tweetchat instead.

SOCIAL (not one of our key strategies):

As expected, the Tweetchat temporarily increased our Twitter engagement for a two-day period—we saw double the usual amount of favorites, retweets, and replies. You can view the Storify here: https://storify.com/mozilla/net-neutrality-tweet-chat-from-mozilla-s-teaminter

The #MakerParty trendline for this week is back up to where it had been two weeks ago: 

trend

 

See #MakerParty tweets here: https://twitter.com/search?q=%23makerparty