Quick update: engagement on the MLN Site

Pledge to Teach

In my last post, I mentioned that we had recently launched the Pledge to Teach the Web. Since we launched it three weeks ago, 240 people have taken the pledge.

Screen Shot 2015-07-24 at 3.25.11 PMScreen Shot 2015-07-24 at 3.26.32 PMScreen Shot 2015-07-24 at 3.27.10 PMOf those who’ve taken the pledge, about a quarter of them have also completed a survey that we sent as a follow-up. The survey is helping us gain a better understanding of our audience, their contexts for teaching, and their needs. We’ll share an analysis of the survey results next month.

Site Traffic

Since we launched teach.mozilla.org back in April, we haven’t been particularly focused on driving traffic to the site. That changed recently, as we began our Maker Party promotion efforts in earnest. We started promoting Maker Party on both beta.webmaker.org and on mozilla.org. Those two referrals, along with our email campaign, led to our most highly trafficked week on the site since launch, during the lead-up to Maker Party. Our highest day was July 13th, when we had over 11K sessions. Since the initial bump, traffic has dropped back down again to between 1200 and 2500 sessions per day.

Unsurprisingly, the Maker Party page is the most popular content, after the homepage. The Activities page is the next most popular.


What we’re doing next with regard to user engagement

  • Adding analytics tracking to several things so we can better measure conversion rates.
  • Experimenting on pledge flow to increase conversion rate. One possibility is to make the pledge the only CTA on the homepage.
  • Determining our post-Maker Party strategy for people who take the pledge. We’re discussing ideas here.
  • Experimenting with “Community” link to increase Discourse activity. This is a larger-than-the-site effort , though. We can be promoting Discourse across all of our work.

Whistler Wrap-up

What an amazing week!

Last week members of the Mozilla community met in beautiful Whistler, BC to celebrate, reflect, brainstorm, and plan (and eat snacks). While much of the week was spent in functional teams (that is, designers with designers and engineers with engineers), the Mozilla Learning Network website (known informally as “Teach”) team was able to convene for two meetings—one focused on our process, and the other on our roadmap and plans for the future.

Breakthroughs

From my perspective, the week inspired a few significant breakthroughs:

  1. The Mozilla Learning Network is one, unified team with several offerings. Those offerings can be summarized in this image: Networks, Groups, and Convenings.MLN ProgramsThe breakthrough was realizing that it’s urgent that the site reflects the full spectrum of offerings as soon as possible. We’ve adjusted our roadmap accordingly. First up: incorporate the Hive content in a way that makes sense to our audience, and provides clear pathways for engagement.
  2. Our Clubs pipeline is a bit off-balance. We have more interested Club Captains than our current (amazing) Regional Coordinators can support. This inspired an important conversation about changes to our strategy to better test out our model. We’ll be talking about how these changes are reflected on the site soon.
  3. The most important content to localize is our curriculum content. To be fair, we knew this before the work week, but it was definitely crystallized in Whistler. This gives useful shape to our localization plan.
  4. We also identified a few areas where we can begin the process of telling the full “Mozilla Learning” story. By that I mean the work that goes beyond what we call the Mozilla Learning Network—for example, we can highlight our Fellowship programs, curriculum from other teams (starting with Mozilla Science Lab!), and additional peer learning opportunities.
  5. Finally, we identified a few useful, targeted performance indicators that will help us gauge our success: 1) the # of curriculum hits, and 2) the % of site visitors who take the pledge to teach.

Site Updates

I also want to share a few site updates that have happened since I wrote last:

    • The flow for Clubs has been adjusted to reflect the “apply, connect, approve” model described in an earlier post.
    • We’ve added a Protect Your Data curriculum module with six great activities.
    • We added the “Pledge to Teach” action on the homepage. Visitors to the site can choose to take the pledge, and are then notified about an optional survey they can take. We’ll follow up with tailored offerings based on their survey responses.pledge

Questions? Ideas? Share ’em in the comments!

Medium-term roadmap

Earlier this week, I wrote about the short-term roadmap for teach.mozilla.org. Now I’d like to share a few details about what we envision a little farther out (Q3 and into Q4).

Badges

The bulk of the work here will be improving the user experience for both badge applicants and badge reviewers. We’ll also be rolling out some new badges that are aligned with our programmatic plans, and will recognize the key volunteer roles we’ve identified (i.e. Regional Coordinators and Club Captains).

Directory

I’m really excited about this project because it will transform the site from simply being a place to find resources to a community, and because we’ll be able to offer more customized experiences for users once we know more about them. The Mozilla Learning Network Directory will include rich mentor profiles and group pages (where “groups” include Clubs, Hives, and organizations), as well as the ability to search and browse. The initial build will also include a full integration of Discourse. (We’re drawing heavily on the Hive Directory for inspiration.)

Curriculum functionality

It’s been a long time coming, but soon we’ll begin designing a more permanent solution for making our curriculum content dynamic. This will include adding basic user interactions (“Likes,” ratings, comments), as well as dynamically facilitating the creation and display of remixes and translations. We’ll likely also have a tool for users to create and share their own playlists, and to submit curriculum for consideration.

Ongoing iteration on the engagement flow

We’ll continue to learn what works in terms of connecting people quickly to what they need, and we’ll likely continue to make changes as a result of those learnings. Our engagement strategy will get some serious power behind it as we move forward with the email tooling project that’s happening in parallel.

Thimble!

Finally, our team at CDOT are actively working on making improvements to Thimble, our open source code editor for teachers and learners. We wrote about those improvements a few weeks ago.

Short-term roadmap for teach.mozilla.org

Teach.mozilla.org was released into the wild at the end of April, and we spent part of the last month doing a round of iterations based on early feedback.

The most notable iteration is the homepage. Our goals were to clarify the three main actions a user can take, incorporate the blog, and use more plain language across the page. Do you think we succeeded?

homeI want to share an updated short-term roadmap, so you can know what’s coming up in the immediate future:

  • The next thing we’ll release is a revised workflow for new Clubs. The workflow reflects our evolving thinking about the organizing model. It will work like this: people will submit their Clubs on the site, they’ll then be connected with a Regional Coordinator who will work with the Club Captain to refine the description if necessary. Once approved, the Club will be visible to everyone who visits the site. For more on what a Regional Coordinator is, check out Michelle’s blog post.
  • Next up: Since the three main actions require a pretty heavy lift for users, we want to provide a lower-effort action. We’re going to test out a “Pledge to Teach” action that simply allows the user to, well, pledge to teach the web in their community.
  • Maker Party is right around the corner. We’ll be promoting it on the homepage, and making some changes to the Maker Party page itself, including adding case studies of successful parties, and linking to the recommended Maker Party activities for this year.
  • The Clubs page will get a makeover soon, too. The goal is to provide better scaffolding for people who are thinking about starting a Club, as well as to provide a more useful experience for people who are running Clubs.
  • We’ll keep on adding more curriculum as it’s tested and finalized!
  • We’re starting the localization effort! Check out this globalization ticket for an overview of all the different pieces of this project.

I’ll blog about the medium-term roadmap soon!

User Testing the Teach Site

We are soooooo close to releasing the new Teach site.

People seem to dig the bright colors and quirky illustrations throughout the site.

People seem to dig the bright colors and quirky illustrations throughout the site.

In advance of the release, I wanted to conduct some user tests to make sure we’re still on the right track. This week I conducted two user tests with members of the community (yay!). As is always the case with user testing, I learned a lot from observing users interact with the site.

You can see detailed notes here and read my recommendations below.

These recommendations are based on formal user tests with two users as well as feedback from people who’ve been involved or observing the process throughout.  Also, please note that I wasn’t able to test the primary functionality on the site (adding a Club to the map), so these recommendations are more about IA and other content issues.

Findings and Recommendations

Screen Shot 2015-04-10 at 2.48.07 PM

ACTIVITIES / CURRICULUM / RESOURCES

Findings:

  • People want to see more activities and resources.
  • People expect to be able to sort and filter.
  • Our internal distinction between the Clubs Curriculum (the official curriculum for Clubs; with a strong recommendation for following the prescribed path) and Teaching Activities (more “grab and go”-style) is not intuitive to users.
  • The Teach Like Mozilla content needs to be more integrated into common user flows.

Recommendations:

  • Continue with current plan for developing and publishing more approved curriculum and activities.
  • Continue brainstorming work around scalable presentation of curriculum begun in this heartbeat. The ideas discussed so far address sorting and filtering, and make good use of the Web Literacy Map as an organizing tool.
  • As part of that design work, we should also allow users to access all teaching materials from the same page, and provide specific views for “official Clubs curriculum.” I recommend we keep the Teaching Activities page, and remove the Clubs Curriculum sub-page. This content is one of our primary offerings so it belongs at the top level. /cc @iamjessklein
  • We need to offer a solution for sharing resources—e.g. maker tools, other curricula, programs. (Hello, Web Lit Mapper!)
  • We need to design a stronger connection between teaching activities and the Teach Like Mozilla content. A short-term solution might be to link to the TLM page from every individual activity page, but we should also be working towards a better longer-term solution. /cc @laurahilliger

Screen Shot 2015-04-10 at 2.50.46 PM

CLUBS

Findings:

  • The Clubs Toolkit is not findable, and needs to be supplemented with content targeted towards helping people “get started.”
  • We are not providing enough information for the use case of a person who is deciding in the moment whether to start a Club.

Recommendations:

  • Make the Clubs Toolkit more visible on the page.
  • Consider renaming the Clubs Toolkit something like “Getting Started Guide” or “A Club’s First Month” – and editing content to match. /cc @thornet
  • Based on my understanding of the expected pathways to starting Clubs, I do not think we need to make any significant changes to the Clubs page to address the use case of someone coming to the site and deciding in the moment whether or not to start a Club. As I understand it, our plan for growing Clubs makes use of the following scenarios:
    • 1) Someone is “groomed” by staff member, Regional Coordinator, or other community member. By the time they arrive at the site, they have the specific intent of adding their Club.
    • 2) Someone finds out about us through Maker Party, and through a series of communications learns about Clubs and decides to start one. They are coming to the site with the specific intent to add their Club.
    • 3) Someone with an existing program or group wants to be listed in the database. Again, they are coming to the site with the intent to add their Club.

In short, I don’t think we’ve yet seen a reason to have the site serve a “selling” or persuasive function. I *do* think the Clubs page is a natural first stop for someone who is looking to understand how to start a Club. I think the changes recommended in the bullet points above address that.

Screen Shot 2015-04-10 at 2.51.21 PM

EVENTS/MAKER PARTY

Findings:

  • The copy describing the Events page in the main navigation is misleading, since the content on the Events page is about Maker Party.
  • People may understand throwing a Maker Party as a “first step” to starting a Club, rather than a lower-bar option for people who do not want to start a Club (and perhaps never will).

Recommendations:

  • I think we should re-brand what is currently the Events landing page as “Maker Party.” We’ve already sort of done this in that, while the page is called “Events” in the nav, the h1 copy in the hero image is “Host a Maker Party.” I suggest we change the copy in the nav to “MAKER PARTY” and the teaser copy to “Our annual global campaign”. /cc @amirad

Screen Shot 2015-04-10 at 2.52.06 PM

INFORMATION ARCHITECTURE

Findings:

  • Users tend to ignore, not see, or misinterpret the CTAs at the bottom of every page
  • Users do not notice links to the sub-pages in the main navigation

Recommendation:

  • We need to design better, more intuitive pathways for viewing secondary pages

ADDITIONAL RECOMMENDATIONS

I’m going to keep banging this drum: We need to clarify our audience! I think we’ve made good progress in terms of clarifying that our “first line” audience includes educators and activists. But I think we have to take it a step further and clarify who those educators and activists are working with. There are at least two axes that I think are important to be clear about: first, the global nature of our work, and second, the specific age groups of what I’m calling the “end learners,” for lack of a better term.

I think we do a pretty decent job of conveying the global nature of the program through copy and imagery, though obviously implementing our l10n strategy is absolutely fundamental to this.

I think we are less clear when it comes to the age groups we’re targeting with our programs and materials.   For example, I think we ought to specify the appropriate age level for each activity. (And the images, activity titles, and copy should reflect the target audience.)

Questions, comments, disagreements wholeheartedly welcomed!

An update on the Teach site + some thoughts on iterative development

It’s been a minute! We’ve just started a new heartbeat, and now we have a giant team working on the Teach site. Our stand-up today looked like The Brady Bunch.

We’re now entering a stage where we have three strands of work running in parallel:

  1. Engineers are setting up the technical framework for the new site, and are beginning to build out the pages that have been designed
  2. Designers are polishing up the work they’ve done so far (accommodating some last minute requests from me!), and starting on a few new pages and workflows
  3. Members of the Learning Networks team are finalizing copy and content for the various pages

If every strand can keep moving, we’ll be in good shape for the release at the end of the next heartbeat. Ideally, we’ll get into a rhythm where copy is finalized in the heartbeat *before* the design work, which is done in the heartbeat *before* development, but right now things are overlapping a bit.

Iterative Dev {It’s been so long since I’ve done an anagram! Here we go: Tea Revived It}

I’ve been thinking about how to build out this site in a way that reflects the evolving nature of the Learning Networks team work, and specifically the Clubs program. The answer is (always) to be agile.

It’s often difficult to agree on what should be included in a v1. I think the term “MVP” is one of the most abused terms in software. It is sometimes used to mean “do as little work as possible,” but the definition I like best is, “do as much as is needed to prove or disprove a hypothesis.”

This definition provides a great framework for determining what to include in a v1. If you can agree on a hypothesis, then every decision can be made by answering the question, “Do we need this to prove the hypothesis?”

For the Teach site, the hypothesis we’re testing is: “People will use the site to find teaching activities and add their Clubs to the map.”

So the v1 will seek to prove that. The priority items are a couple static pages that point to our high-quality curriculum modules, a Clubs page with an “Add Your Club” workflow, and some additional static content to flesh out the site.

Beyond Version One {Anagram: Sobered Onion Envy}

While  v1 is being created, we also need to start gearing up for the next iteration, which will be focused on building out more advanced tooling for the curriculum (so that we can add to those pages without developer involvement, and incorporate feedback and remix ideas from the community). This is essentially adding a fourth strand to the list above—a kind of “requirements gathering and brainstorming” strand.

(Side note: I like that the “build, measure, learn” cycle we’ll use to iterate on the site echoes the process we’re using to develop the curriculum with our community. Agile methods all around!)

Of course, even while using iterative development practices, we still need to keep the big picture in mind. To see the current thinking about future iterations, you can always check out the roadmap. I’m updating it regularly as our needs change, or as they become clearer to me.

User testing results

This week I had the opportunity to conduct some user tests on Matthew’s wireframes for the Teach site.*

The goal of the user test was to validate the information architecture. We tested two nav variations:

Variation B

Variation A

Variation B

Variation B

Nav B performed slightly better, but both variations had some challenges. We shouldn’t draw any solid conclusions from the tests, partially because it wasn’t a large enough sample (we had five user testers, which meets the best practice standard, but since we tested two variations, we should have had more), and partially because most of the participants were highly familiar with our work already, making them a somewhat biased sample.

However, the user testers were fantastic, and we were able to generate some valuable insights that will be useful moving forward. Here are some of the key learnings:

  • Mentors want to connect with other mentors. Regardless of the task we gave them (“find something to do for your after-school program,” “find a way to improve your own skills,” “find people to connect with in your community”), nearly all of the user testers wanted to accomplish those tasks by seeing profiles of other mentors, learning about what those mentors had done, and finding ways to connect directly to them.
  • Mentors make distinctions between independent, bite-sized activities, and full-blown curricula. There’s a need for both. The word “activities” seems to suggest shorter, one-off icebreakers and games, whereas some of the mentors expected to find full curriculum somewhere else (e.g. under “Clubs”).
  • Mentors tend to know what they’re looking for. The wireframes introduced the idea of curated content—featured mentors, featured activities, and featured discussion topics. However, user testers consistently had a specific idea in mind, and gravitated towards the search field (even though it was nearly below the fold). We may want to promote searching over browsing, though we’ll likely want to pair it with the browsing framework provided by the web literacy map
  • “Badges” is not a destination. Several testers were confused by the idea of a “featured badge.” In the users’ minds, badges are tied to skills and accomplishments, or perhaps pieces of curriculum, rather than being an end in themselves.
  • People need pathways for becoming a mentor. For those who want to contribute as a mentor, but who don’t have regular access to learners (i.e. those who aren’t professional educators), we need to provide clear pathways and opportunities for mentorship.

That last point hints at a critical discussion we need to have about the various audiences we’re serving. Mentors who have regular access to learners have different needs from the mentors who don’t have that access. There are some overlapping needs, of course—namely, access to high-quality activities and professional development—but how we package those offerings will be different.

We’ll be exploring some options through the ongoing design process over the next few heartbeats.

Thanks to the very rad user testers who volunteered their time to help with this!

*I’m tempted to call this The Site That Shall Not Be Named, because the name is a little bit up in the air. It’s unlikely that we’ll go with “Teach.webmaker.org,” at least in part because our resources for teachers are not entirely tied to the Webmaker tools. In fact, our curriculum and resources are pretty tool-agnostic. “Mozilla Learning” might be a better encapsulation of our product offering.