INTegrated insights report

Email Analytics | Litmus | 01-2021 to 02-2022

Product Design, UI, UX, Product Strategy

Lead designer

Overview

I was the lead designer supporting a core strategic pillar to develop Email Analytics solutions for Litmus’ Enterprise customers. My team included a product lead, UX researcher and a front-end and back-end infrastructure development team. Over the course of a year, we launched several iterations of the Integrated Insights report, from the evolution of filtering to a new feature ‘Email Performance Indicators’ which auto-flags high and low performing emails. We also launched other EA solutions in 2021 such as the Apple Audience report and a new Analyze Overview page. The combined offerings resulted in an increase in EA accounts usage and overall Enterprise EA revenue for the company.

RESEARCH AND INSIGHTS

To kickoff the new Email Analytics initiative, I conducted exploratory interviews with Litmus customers to understand the problem space. Throughout the year, we did concept validation interviews, moderated/unmoderated usability testing, and participatory design . Some insights we learned from the first round of research:

  • Multiple analytics tools - Many email marketers compile reporting from multiple sources, including ESP data and BI tools for customer data

  • Engagement data is unique - Litmus users relied heavily on their engagement data, such as read time and email clients

  • Email send data is in ESP - Core email metrics like Open and Click rates are only captured in the Email Service Provider

THE GOAL

The goal of my team was to increase Enterprise accounts using Email Analytics by 25% in 2021.

We focused on the Expand stage of the customer lifecycle, turning Regulars into Champions. Since our goal was specific to Enterprise, we know that our users are mature in the lifecycle and are seeking new ways to obtain value from our product. They are interested in how Litmus can continuously grow to address their evolving needs.

The PROBLEM

Based on our insights, we know that email marketers spend hours in multiple analytics tools and often piece together a report from various sources. The A-ha moment for Litmus users today is “when Litmus previews catches an error I would have missed”. How can we create the same value where Litmus will catch analytics insights that users would have missed? We know that automation is important to these users, so our north star was to help email marketers save manual time analyzing their email performance.

How Might We…design a new EA solution that will empower marketing leaders and teams to identify actionable insights that improve email performance?

THE CHALLENGES

  • ESP integration has friction - there are multiple steps to set up an integration with an Email Service Provider

  • Quickly iterate from learnings - we launch and learn and continue to improve the product throughout the year

THE FIRST ITERATION

The first version we launched of the Integrated Insights report provides a hybrid view of email send data and engagement data using an ESP integration with Litmus.

  • Includes a top section highlighting engagement metrics and audience breakdown

  • Read rate column in the table deep links to the Litmus EA report for that email

  • Improved the connect ESP flow to reduce setup friction by offering a Help Scout side panel to minimize toggling between browser windows

  • Supplemental weekly digest email with report highlights and traffic drivers from Analyze and Home ribbon modules

V2 Filtering

The second version we launched of the Integrated Insights report included more advanced filters. Real value for the report relies on categorization of email types, which differs by ESP and email program. We addressed this by offering the ability to filter out Automated emails to make for better comparisons. We did this on the backend by comparing the Last sent date, and if it appeared to be sent multiple times we labeled it as Automated.

 

APPLE MAIL PRIVACY

As adoption of Integrated Insights steadily increased, we were blindsided by Apple’s mail privacy announcement in October of 2021 - which greatly impacted the accuracy of Open rates. In the meantime, we launched an Apple Audience report, but continuously innovated towards a more automated Integrated Insights report. Without Open rates, I led an ideation workshop around other ways to evaluate email performance.

 

CONCEPT VALIDATION

We conducted concept validation interviews with customers using a data set and a prototype for Email Performance Indicators as well as a prototype for Email Quality Score.

In testing, we learned that the Email Quality Score still required mental computation to compare all emails, and low scores created concern for managers. Auto-flagging outliers was well received since it removed the manual analysis and had no negative connotations.

 

EMAIL PERFORMANCE INDICATORS

The final feature Email Performance Indicators highlight high and low performing emails with a badge. Green badges are high outliers while orange badges are low outliers. Clicking an orange badge offers tips for improvement and links to Litmus’ abundant help articles. Instead of toggling between Help Scout and Litmus, we introduced an in-app Help panel to get started and understand all the possible badges as well as linked resources for improving email performance. A new feature callout points to the filters dropdown where EPI lives.

 

ReSULTS and IMPACT

  • Since launching in March 2021, the Integrated Insights report has been enabled for over 100 Enterprise accounts by end of year

  • Exceeded our original goal of increasing ENT accounts using EA by 27% compared to our goal of 25%

  • Generated $1,316,393 in ENT EA Revenue by the end of 2021, a 318% increase in revenue from 2020

“The Litmus Integrated Insights Report already saves us so much time by providing powerful metrics from our ESP in Litmus without having to switch between tools. The addition of Email Performance Indicators makes it even easier to quickly and easily see which emails might need attention and take action with the suggested next steps. The UI is beautiful and seamless and the help documentation provides so much rich content to help us improve our strategy.”

- Aubrey Badger, Associate Director, Performance Marketing at Arizona State University