Challenging Long Held Design Assumptions with Concept Testing
What happens when research results challenge design assumptions? For years, Big Health developed large scale multi-channel enrollment campaigns featuring bespoke illustration. But when product enrollment fell and troubling user feedback emerged, I went looking for answers. This case study reveals how concept testing can lead to insights that enable designers to approach old challenges in new ways.
Company
Big Health is a digital therapeutics company that provides safe and effective non-drug alternatives for the most common mental health conditions including insomnia and anxiety.
Product
Sleepio is digital therapeutic for insomnia that addresses unique sleep needs with a personalized program, featuring sessions with step-by-step guidance through evidence-based techniques.
Team
Robyn Anderson, Research & Product Marketing
Tephany-Lou Padregada, Visual Design
My Expertise​ ​
-
Concept testing
-
Survey design
-
Process design
-
Metrics formation
-
Project management
-
Cross-functional collaboration
My Role
During my time on Big Health's product team I was responsible for boosting product enrollment by researching, testing, and launching multi-channel product marketing campaigns. For this project, I was tasked with establishing a concept testing process and using that process to develop a new Sleepio campaign concept. I conducted the entirety of the research and process design work presented in this case study.
The Challenge
Big Health sells it's digital therapeutic for insomnia, Sleepio, as a health benefit to large employers and health plans. Big Health then enrolls health plan members into Sleepio by working with clients to develop and launch multi-channel marketing campaigns. Clients expect to be consulted in the campaign design process and arrived at kickoff meetings with specific visions that often conflicted with Big Health's strategy. For years Big Health developed highly customized campaigns for clients, big and small, regardless of how well these campaigns performed.
Developing dozens of bespoke campaigns each year is a massive undertaking for a small company without a dedicated marketing function. Members of the Customer Success team struggled to create complex, multi-channel campaigns while juggling their many other pressing client responsibilities. These campaigns were launched to populations ranging from 5K to 800K employees without prior testing. When Sleepio enrollment fell below target it was impossible to identify why because the custom nature of each campaign made cross-client comparison nearly impossible.
​
Meanwhile, troubling feedback about Sleepio ad campaigns was emerging from interviews with users. Research findings suggested that the illustrations favored by the team's designers was hurting Sleepio's credibility with target audiences. This feedback was as shock because the Sleepio illustration style were universally beloved, not just by clients but by Big Health employees too. The suggestion that key segments of Big Health's audience might not feel the same way was unwelcome news to say the least.
​
1. Bespoke campaigns were expensive to develop & difficult to validate
2. Campaigns launched to large audiences without prior testing
3. Research suggested ad illustrations hurt Sleepio's credibility with target users
My Objectives
As a founding member of the Product Marketing team I was responsible for boosting Sleepio enrollment by researching, testing, and launching multi-channel product marketing campaigns. For my first project, I was tasked with developing a standardized Sleepio campaign that could be leveraged by any client at any time of the year (a.k.a. an "evergreen" campaign).
However, because this campaign would be standardized instead of bespoke, we'd need to work harder to sell it to our clients. To make matters more complicated, product leadership insisted that I use this campaign to address the negative feedback we'd received from users about our ad illustrations, which I suspected might not go over well the design team.
In short, I needed a way to validate this new "evergreen" campaign concept in order to help convince both our designers and our clients that it was worth investing in. The only problem? Big Health didn't have a formalized concept testing process in place, so I'd need to create one.
​
​
Goal
Develop an "evergreen" Sleepio campaign concept that meets the following criteria:
-
Educates the recipient on the importance sleep to overall health
​
-
Is relevant year round, meaning not tied to a specific seasonal theme or topic
​
-
Reinforces the Sleepio’s efficacy claims
Objectives
-
Design a concept testing process that can be leveraged cross-functionally to validate campaign concepts prior to development and launch
​
-
Use this concept testing process to select a winning Sleepio "evergreen" campaign concept
Concept Testing
Why concept testing matters
Concept testing involves asking consumers questions about a product or service to help evaluate consumer receptiveness before investing resources to bring the concept to market.
Ad concept testing takes this basic principle and applies it to the testing and development of marketing campaigns. Why bother testing ads? Companies on average spend roughly 11% of their annual budget on advertising. That's a lotta cash to throw at something without first testing to see if it meets your goals.
​
​
The importance of cross-functional collaboration
Prior to this project, campaign design decisions were made largely by clients in an ad hoc fashion. Clients came to the table with a specific vision in mind and we'd have difficulty advocating for the changes we felt were necessary for the campaign to be a success. I wanted this new concept testing process to empower product marketing and design to defend our design decisions to clients. For this to work well, I felt it was important for both product marketing and design to leverage this new process so that decisions were made collaboratively.
​
I knew next to nothing about concept testing when I was asked to develop this process. But product leadership was confident I could figure it out, so I rolled up my sleeves and took myself through a crash course in concept testing a survey design. I built the concept testing process in Confluence as was the standard practice at Big Health. Unfortunately, I did not fully document how it looked with screenshots when I left my job, so you'll just have to trust me when I say it was well designed and easy to use! The survey design guide below is just one part of the overall concept testing platform I built in Confluence and the version linked here was simplified for download from the Confluence system.
​
Concept Testing - Survey Design Guide
-
Survey components & introduction
-
Screening questions
-
Stimuli Exposure
-
Metrics & the Likert Scale
-
Metrics question examples
-
Forced-choice question examples
-
Demographic question examples
Survey Design
Designing a concept test survey
Concept testing is typically conducted using a survey. There are two main types of survey methods: Monadic and Sequential. I chose the Sequential method in which respondents are shown all concepts in random order. I selected this method because the turnaround time is quick and the smaller sample size means it's more affordable. I referred to SurveyMoneky's handy methods guide to help make my decision. The document to the right is a draft of the survey I created in SurveyMonkey.
​
Defining the right metrics
Stimuli may be the basic building blocks of a concept test, but the questions you ask about them are just as important. In concept testing, questions should be driven by metrics—the criteria respondents will use to judge your stimuli. I devised the metrics below so that testers would have a list of metrics to choose form that aligned with our overarching campaign strategy. Given the feedback we'd received from Sleepio users, I felt it was important to focus on Comprehension, Credibility, Benefit, and Action for this test.
Testing only the essential variables
Deciding which variables to include in a concept test is tricky because it can be tempting to test everything. However, the more variables you test the larger your sample needs to be, so for this test I limited my variables to only those essential to achieving my objectives (e.g., slogan and creative) and kept the product efficacy claims and call to action consistent across all ads.
A quick note on survey stimuli
The creative used in the test stimuli was not intended to be a finished product. I asked my visual designer to design a simple flyer for testing purposes. We intended to refine the designs once a winning concept was selected. For example, we used stock photography for the sake of time and intended to take our own photographs if the winning concept required it.
Survey 1 Stimuli
Concept 1A
Vs.
Concept 1B
Survey 2 Stimuli
Concept 2A
Vs.
Concept 2B
The Results
Survey 1
Survey 2
Box Scoring
Key Takeaways
All four ads shared some similarities in the eyes of our respondents...
-
All four ads performed well at communicating the intended message (i.e. comprehension) with box scores over 50% and above, but the stand out was concept 2A
-
All four ad-s underperformed in the benefit category, likely because respondents did not have all the information they needed to make an informed decision about whether Sleepio would help
Concepts 2 were the stroger performers in all four metric categories...
-
Concepts 2 outperformed concepts 1 by roughly 10% when it comes to credibility. Qualitiatvie feedback suggests the use of photography helped to lend credibility to the ad
-
Concepts 2A and 2B were the best performers in nearly all the metric categories, suggesting that “better health starts in bed” theme is perhaps slightly stronger
​
When asked to pick the strongest concepts, concepts 2 were the ultimate winners...
-
When asked which concept was stronger of the two, respondents selected concept b (1B & 2B) as the concept that gave them the most confidence that Sleepio would “deliver on its promise” and was “right for them”
Survey 1
1A
1B
Forced-choice rankings
Which of these two advertisements gives you the most confidence that Sleepio will deliver on it's promise to help you fall asleep faster, stay asleep longer, and feel more energized throughout the day?
A closer look at the winning ad: 1B
Q: Please summarize the main message of the advertisement in your own words (try not to repeat the slogan).
Positive interpretations
-
"Sleepio will help accomplish your dreams"
-
"Goals are achieved easier after a good nights sleep"
-
"Sleep better for better health"
-
"Sleepio offers a product or technique to asssit with improving sleep."
Negative interpretations
-
"Makes a cartoon of problems"
-
"My brain is scrambled while sleeping"
-
"Take our meds and sleep; bs"
Q: Which specific elements of the advertisement do you like and why?
How clearly the ad communicates the idea that sleep can help achieve goals / dreams
-
“The message is clear, sleep makes it possible to do want you want”
-
“The idea of accomplishing goals and dreams”
-
“The visualization of their dreams”
-
“It's obvious that these people are sleeping, but dreaming about the things they'd like to accomplish”
The photographs communicate a feeling of restfulness
-
"I prefer that this set has pictures of people"
-
"The visuals of people at rest communicates the right mood/feeling for me"
-
"I like the inclusion of people in a place of restful sleep. It gives the idea that sleepio works"
Q: Which specific elements of the advertisement do you dislike and why?
Illustrations distract from the positive message the ad is attempting to communicate
-
“The drawings behind the people feel both busy and extraneous”
-
“Background images are distracting”
-
“The belittling cartoons again trivializing the problem”
-
"The artwork/drawings, seem VERY childish, & don't explain HOW the program can help"
Uncertainty about what Sleepio is or how it works
-
"Are they pills, diet, a mattress, vapors??"
-
"No evidence of what the program does or any evidence that it might work."
The ad is difficult to read​
-
"The written part is too wordy"
-
"The text is too small; the people images too large; and, once again, the whole ad is too visually busy. I understand that it portrays the mind's potential distractions but there are clearer/cleaner ways to so the same."
Survey 2
2A
2B
Forced-choice rankings
A closer look at the winning ad: 2B
Q: Please summarize the main message of the advertisement in your own words (try not to repeat the slogan).
-
"Sleeping well is the key to being healthy"
-
"A good night sleep is the beginning of a healthier life"
-
"Learning to sleep well and how Sleepio can make a difference in your life by helping you get rest and feel energized"
-
"Real people benefit from Sleepio"
-
"You can be healthier if you sleep better, by the way, we have something that can help with that"
-
"Finally a way to sleep better using this product"
Q: Which specific elements of the advertisement do you like and why?
Photographs of “real people” read as more credible
-
"I like seeing real people “sleeping” and the third picture is best"
-
"Pictures of real people. Better than cartoon drawings"
-
"The images are peaceful and not gimmicky"
-
"Better pictures and seems more adult"
-
"The pictures give it a better quality feel then the cartoon pictures"
-
"I like that it's showing real people sleeping comfortably. It let's me know that Sleepio is a legitimate sleep program"
-
"I like the images and the colors. And I like that men, women, and POC are used in the ads"
Soothing color scheme
-
"The blue color schemes are nice on the eyes"
-
"The soothing colors"
-
"I like the advertisement photos and the relaxing colors"
-
"Calming colors, the people look calm and health"
The intended theme/message of the ad is easy to understand
-
"I like the simplicity of the message compared to the other ad"
-
"Text and image make it clear what the point of the ad is. Simple and easy to understand"
-
"Easy to understand that good sleep is necessary"
-
"The photos of people sleeping are clearly conveying the message about Sleepio. The moons and stars are eye-catching without being distracting"
​
Q: Which specific elements of the advertisement do you dislike and why?
Not interesting enough / does not inspire action
-
"Not exciting"
-
"Again, it doesn’t drive me to take any action"
-
"It seems kinda plain and all the same"
-
"It seems too simple, it doesn’t really pull me in"
-
"The text can have more interesting design to it"
Difficult to read
-
"The dark blue makes the ad difficult to read"
-
"The colors and placement of the text. It's hard to read, especially the fact that the "starts in bed" bleeds into the images"
Photographs are not for everyone
-
"Pictures-not sure why"
-
"I don’t like the photos and I prefer illustrations"
-
"I don't care to see people in bed asleep. It's akin to watching restaurant commercials on TV where you see people forking food into their mouths, it's too intimate for my taste"
Unclear what Sleepio is or how it works
-
"They still do not suggest any ideas as to how one can get better sleep"
-
"I don’t know why or how they fell asleep"
-
"Not enough information"
Impact
Socializing my concept testing process
Introducing a new process is one thing, getting people to actually use it is another challenge entirely. The concept testing process I created in Confluence was intended to be leveraged cross-functionally by the Product Marketing and Design teams to validate campaign concepts prior to launch. Members of my team were on board with the process from the start, however, I'd rolled out enough internal programs and processes to know that getting people to adopt a new process isn't easy. After sharing the results of the concept test, I invited the Design and Product Marketing teams to a training workshop where I walked them through creating a draft concept test using the new process. The draft we created in that workshop ended up laying the groundwork for subsequent tests. I also set up several optional office hours in case anyone had remaining questions or just wanted to chat about the process.
Challenging design assumptions
The winning campaign concept featured photography, not illustration, as the primary design element. This concept gave survey respondents the most confidence that Sleepio would “deliver on its promise” and was “right for them”. And while these results reinforced similar feedback we'd received in interviews with Sleepio users, I was still surprised at the outcome. It's difficult to overstate just how much the Sleepio illustration style was beloved at Big Health. To suggest our visual design might be hurting our credibility with Sleepio users felt a bit like blasphemy.
​
I shared the results of the concept test with the design team during our bi-weekly meeting. I knew the results were likely to be controversial, so I began by walking the team through a careful explanation the rationale behind my test design and methodology choices. Next, I presented the box score results followed by a selection of quotes from the open response questions. I held my breath, fully prepared for push back, but none came. To my delight, the team was more intrigued than upset by the results. They were excited at the prospect of running more concept tests and a lively discussion on the pros and cons of photography followed.
​
1. Design saw the concept test results as a welcome creative challenge
2. Concept testing became standard practice for all new large-scale campaings
3. Photography was successfully incorporated into some Sleepio campaigns
Judging long term impacts
I left Big Health for graduate school shortly after completing this project so I was unable to see first hand how the impacts of the Sleepio concept test and my new process unfolded over time. However, I recently caught up with a former colleagues and through those conversations learned that the team had battled some of the usual challenges when it came to rolling out the new Sleepio campaign to clients. While clients loved seeing testing results, not all clients were convinced by the data to adopt photography in place of illustration. Some clients felt photography was too specific and therefore less inclusive. Others clients preferred illustration to photography because it paired better with their own internal Benefits branding. In short, some clients have their preferences and sometimes no amount of data will convince them otherwise.
​
Including photography in campaign assets came with a few internal challenges as well. Producing original photography in house was more expensive than creating illustrations. On the other hand, using stock photography wasn't ideal either because it limited the designer's options and constrained creativity. From my conversations with former colleagues I gathered that more research might be needed to judge for which audiences and in which contexts it was worth investing the resources needed to produce original photographic content.
1. Some clients did not welcome the inclusion of photography in ad materials
2. Original photography proved more expensive to work with than illustration
3. More research is needed to help Design leverage photography effectively
Photography is now featured on Big Health's landing page. I have no idea if that decision was partly influenced by the results of my concept test, or if it had always been a planned part of the re-design, but I still found it interesting none the less!
Big Health homepage banner before redesign
Big Health homepage banner after redesign