A/B Testing: Lessons Learned with the National Council of Teachers of Mathematics

A/B Testing: Lessons Learned with the National Council of Teachers of Mathematics
Silhouette of head with brain filled with gears

How do I Increase Audience Engagement?

When we talk marketing and communications, associations ask a lot of questions and we’ve noticed there is a commonality between them all. Oftentimes it boils down to one all-encompassing question: how do we best engage our audiences—especially those that aren’t currently engaged at all?

While you might think of your role as being a thought leader, an event planner or even a publisher, everything that happens within your org—from growing your member base to driving success for your products and beyond—relies on members being engaged with what you’re doing. This engagement comes from being able to effectively connect with your audience.

The Solutions team at McKinley Advisors (McKinley) has an ongoing relationship with the National Council of Teachers of Mathematics (NCTM), serving as an extension of their internal marketing team. We collaborate on both the marketing strategy and tactical implementation for a variety of NCTM’s initiatives, including marketing of their conferences and events.

Over the course of several successful event and membership marketing campaigns, which ran through NCTM’s email marketing automation platform, we began to develop hypotheses on what types of messaging resonated best with NCTM’s members and nonmembers. In order to develop a complete understanding of what messaging was most effective, McKinley and NCTM decided to collaborate on a series of several A/B tests looking into different messaging factors.

McKinley and NCTM worked together and developed this approach for engaging their audience. While the details might be different, the process can be easily applied to your organization.

Understand Your Goals

The first step to driving audience engagement is identifying the desired, quantifiable results you wish to achieve. Are you trying to grow your member base? Increase event registrations? Improve the performance of your products? By clearly identifying your goal, you’ll begin to understand where to focus to improve your marketing efforts.

With NCTM, for example, we knew our goal was to increase engagement with email communications to drive overall event registrations—email communications being a major component of their marketing effort. If we could gain additional insights into what influenced email open rates, we could then make any necessary changes to improve marketing communications. It was the perfect opportunity to implement A/B testing.

Identify Your Opportunity

Once you’ve identified both your goal and the method that will best help you achieve it, it’s time to think through your opportunity. How will you use best practices to develop a new strategy? How can you make the tools at your disposal work best for you?

For NCTM, the opportunity was to increase engagement by optimizing the audience’s connection to the message. Leveraging A/B testing helps by testing a variable (E.g. from address, subject line, call-to-action, etc.) to a small portion of your entire audience. After the initial test segment receives the A and B message, there is a “winning” send. That winning variable is then sent to the entire audience.

A/B testing for NCTM’s events was an opportunity to gain crucial insight into the audience—insight that could be immediately applied to the ongoing campaign. Because we had access to and expertise in NCTM’s marketing automation tool, the process would be straightforward, relatively clean and easy to learn from.

As you’re thinking through your possible opportunities for A/B testing, try to identify what objective data you’re missing. Do you have insight into whether your audience prefers emails to come from a single sender (e.g. Association XYZ <xyz@xyz.org>) or if it should be from a specific department or subject area (e.g. education@xyz.org, science@xyz.org)? What about certain keywords in subject lines? Building an understanding of high-level factors that influence your audience’s engagement with your messaging will increase the effectiveness of your email communications organization-wide.

Build a Testing Strategy and Iterate for Results

With your goals and resources understood and your opportunity identified, it’s time to get moving. Within the confines of your scalable, data-driven approach, let your expertise guide your testing strategy. Determine where you can most effectively introduce test variables, explore your hypotheses regarding effective messaging and delivery and, most importantly, move fast with your results.

We knew that we had to start by testing the “from” field for event email communications. NCTM and McKinley had asked the question, “do you think it matters who the email is addressed from? Will a recipient be more likely to open the message depending on who it’s from?” After gaining a clearer understanding on which “from” address drove more engagement, we continued posing questions and testing other variables including subject lines. (Note, it’s very important to test only one variable at a time to get accurate results. A sophisticated email service provider and automation platform will help ensure only one variable is at play per A/B test.)

The result? A 10% increase in event email engagement compared to the previous year. By using the proven ‘winning’ variable for the overall audience after each test, we maximized the success of our efforts to engage.

And it’s moments like this—where a single effort creates indisputable gains—that you should seize on building momentum. Testing even one small component in an email creates learning that can be scaled and tested in different parts of your strategy, allowing the data-driven discoveries you make to begin permeating (and updating) your entire marketing effort.

In Summary

While there isn’t a single, simple solution for boosting member engagement for your association, there is a clear process to begin doing so. By understanding your goals and resources, identifying your opportunity and completing multiple tests, you’ll discover more than just an increase in your numbers—you’ll begin to understand what your audience cares about. By doing so, you’re not just creating successful marketing strategies for event registration, but building marketing communications for the future.

For a closer look at how McKinley Advisors implemented A/B testing for NCTM’s Fall events, read our latest case study.