Doing SaaS Cancellation Interviews (the Jobs-to-be-Done Way)

The other day I saw this great chart by Tomasz Tunguz on churn:

Churn Table

The numbers above are exactly what I’ve seen and heard over the years. Since Bidsketch customers are usually small businesses, we’re constantly working to improve churn.

A while back I improved our cancel feedback by changing the cancellation process.

While this was better than a free-form text field, we still had a big problem: over 50% of our churn was in the Not Using it Enough category.

When we followed up, we found that there were multiple reasons for this, but I felt like we weren’t getting the whole story.

This is where Jobs-to-be-Done comes in.

If you’re not familiar with Jobs-to-be-Done, it’s a framework created by Clayton Christensen which helps you figure out what “jobs” your customers hire your product for.

I’ll admit, when I first heard about Jobs-to-be-Done I thought it was obvious and kinda dumb.

What jobs do my customers hire my products for?

Easy:

  • Quickly creating sales proposals
  • Closing sales
  • Getting new clients
  • Eliminating mistakes in proposals
  • etc.

But there’s actually a lot more to it than that. What I listed above is all surface stuff without something important: context.

Context is what will give you the real insights.

To get a better understanding of what Jobs-to-be-Done is, check out the milkshake story by Clayton Christensen (5 min video):

Also, when you get a chance, I highly recommend checking out “the mattress interview” with Jason Fried of Basecamp (47 min audio).

My favorite way to implement this framework is by doing “switch interviews.”

I learned how to do these interviews by attending a workshop by Bob Moesta and Chris Spiek. (They help Clayton Christensen implement Jobs-to-be-Done, and do a great job teaching the framework.)

Switch Interviews

What’s so special about switch interviews?

Unlike other types of interviews, you don’t ask people about how they usually do things. You focus on a single story. The story of how they stopped using one product and started using another.

So if you’re trying to find out why people bought, you wouldn’t ask them this question:

“What convinced you to buy today?”

The problem with this question is that you’ll just get surface level stuff. We’ve asked these types of questions and customers usually give us fluffy answers:

“We wanted to streamline our proposal process.”

“Save time on proposals.”

“Spending two hours on proposals, so trying to cut that down.”

While this stuff sounds reasonable, there aren’t any real insights here.

Doing a switch interview you’ll hear stuff like (this is part of a real story uncovered using switch interviews):

“I was traveling and had a big proposal due the next day. My connecting flight was delayed, and by the time I got to my hotel, I was exhausted. There was no way I’d be able to write a detailed proposal from scratch. I did a Google search for starter content and found a blog post that recommended you guys. When I saw your website promising to create proposals in minutes, I just had to try it.”

This is a condensed snippet of the actual story, but even in this small piece we get some key pieces of information besides this customer just wanting to “save time”:

  • Next day deadline (does our 14-day trial really matter here?)
  • Needed sample content fast
  • Guest posting on high ranking sites is a good thing

And this is just from a single interview. Once you have around 10 interviews you’ll start seeing some obvious patterns.

We’ve done over 30 of these types of interviews for customers that switched to Bidsketch. But for some reason, we never did them for people that switched away from Bidsketch.

It wasn’t until I had a conversation with the always helpful Hiten Shah, that I knew I had to do them for cancellations.

We were talking about Jobs-to-be-Done interviews for churn, when he just smiled and said, “Trust me, you want to do those. You’ll learn different things than the normal switch interviews.”

I’m glad I listened. He was 100% right.

We discovered a lot of useful stuff from those interviews, and we’re using what we learned to make big changes in our product and positioning.

What follows is the process we used to recruit interviewees, the questions we asked, and an example of insights learned from one of the actual interviews.

Recruiting People

Switch interviews typically last 45 minutes, so recruiting people can be a little tricky. It’s even harder to recruit someone that just canceled.

If you’ve ever tried to email people that cancel, you know it’s almost impossible to get them to even reply.

So instead we used an incentive and recruited inside the app.

After someone canceled we showed them this page:

Recruit Page

You don’t have to use an incentive but I’ve found that it really speeds things up if you do, so I often use them.

The Timeline

Switch interviews focus on the actual story of someone getting fed up with a product enough to switch to something else.

You have to keep in mind that the decision to cancel didn’t take place the day they canceled — it happened weeks, months, or even years before.

So you can’t just get the story of the day they canceled, you need to build the entire timeline of key events that led to the cancellation.

You want to build a timeline that shows the following:

  • First Thought – This is when they first had a thought that maybe the product wasn’t working for them. An early struggling moment.
  • Event One (Passive Looking) – Something happens here that gets them to start considering alternatives, though probably not actively looking.
  • Event Two (Active Looking) – Usually a time-sensitive event that triggers active looking.
  • Buying/Cancel Moment – When they evaluate options and actually buy/cancel. Pay attention to tradeoffs.
  • Consumption – When and how they consume (product usage).
  • Satisfaction – How do they like it? Could it be better?

Seems like a lot of stuff for someone that simply cancels a $29/month app. Does everyone really go through all of this just to cancel a web app?

In my experience, surprisingly, most people actually do go through these stages.

Cheat Sheet

To help guide our interviews we find that there are some questions that we often ask (learned in the switch workshop).

Below is our cheat sheet for these interviews…

Setting the Stage

  • Thank them for taking the time and explain that we’re doing these interviews to learn how we can improve Bidsketch. Mention: “We love blunt feedback, so don’t worry about hurting our feelings, it’s not going to happen.”
  • Tell them: “We’re going to focus on the story of how you started using Bidsketch, first started struggling or not finding it helpful, and then eventually canceled. So we’re going to ask detailed questions about where you were and what was happening, to just help paint the picture.”

Key Questions

  • “Why did you initially sign up for Bidsketch? Did you evaluate other tools?” (This is a fluffy question designed to ask them what they expect so they can focus on what we want to know after.)
  • “When was the first time you had the first thought that maybe Bidsketch wasn’t going to work? Or realized that you weren’t using it?”
  • “What happened the last time that you used Bidsketch for a proposal?”
  • “What happened the first time you had a proposal that you didn’t use Bidsketch for?”
  • “Why did you cancel the day that you canceled? Why that day, and not the day before or after?”
  • “What are you using now? How did you start to prepare to use the new product (export data, etc.)?”
  • What happened the day you started to move to the new product/solution? Why did you start that day and not after?”

Keep in mind that most questions follow the story and ask details about that, so the above questions are mainly used to keep things on track and dig further into things.

A Real Cancel Switch Interview

Let’s make this concrete by digging into one the cancellation switch interviews we did for Bidsketch.

We’ll call this customer Tim.

Here was the feedback we received from Tim when he canceled his account:

Category: Other

Comments: “Not using anymore”

Yikes. Not much to go on here. Why wasn’t he using our product?

Early in the interview, Tim told us why he canceled:

“I don’t do many proposals since they’re mostly for new clients, and I work with a lot of repeat clients. Also, I couldn’t figure out how to get the design to match my brand.”

Seems straightforward, right?

At this point, it’s easy to just go along with what they tell you, but if you want real insight, you’ll have to dig.

We interviewed Tim for 35 minutes and ended up with this final timeline of events:

First Thought
  • He didn’t need new clients for several months and got his monthly invoice email from us.
Event #1 (Passive Looking)
  • Saw a well-designed proposal from another designer and realized he needed to update his design.
  • Started a rebranding project to update the look of his website, invoices, contracts, and proposals.
Event #2 (Active Looking)
  • Finished up rebranding project and now it was time to implement the new design into everything they had.
  • Struggled to implement specific elements of new design in Bidsketch.
  • Searched and found another app but also couldn’t implement what he needed. Two months passed and paid for two accounts.
  • Downgraded to a cheaper plan during this time to save money while finding an alternative.
  • Spent a lot of time implementing new design in MS Word.
Cancel Moment
  • Was reviewing quarterly expenses and saw the $29 charge and remembered to cancel.

 

Some of our takeaways from this:

  • Custom template designs should be shown during onboarding (this would have let him implement his design).
  • Frequency wasn’t the issue, making him feel better about his proposal design (compared to other designers) was the actual “job.”
  • We should reach out to customers that downgrade, since it may mean they’re getting ready to cancel.
  • He churned months before the actual cancellation happened. There were warning signs and ways we could have saved the customer.

Recording and Reviewing Everything

When I do these switch interviews, I record the audio and write/review at a later time.

Each interview gets a document outlining the main stories, takeaways, and pressures at work:

jtbd-worksheet

 

You can get that Google Doc here.

I also write up a more detailed version of the switch interview, which is basically a summarized transcript of the interview.

The last thing I do is go through each of the interviews and try to put everything in a spreadsheet to help identify patterns:

Spreadsheet Summary

End Result

The real goal isn’t just to collect interesting stories about your product, but to get a deep understanding of the jobs customers truly hire (or fire) your product for.

From this, you’ll often come up with some obvious big wins, but other times you might discover that some of the fundamentals of your product need to change.

What was the end result for us? We were able to cut down churn by over 30%.

Reducing churn is great, but that’s just the quick win. Longer term, we’ve got huge changes and opportunities that we’re even more excited about 🙂

29 thoughts on “Doing SaaS Cancellation Interviews (the Jobs-to-be-Done Way)

  1. Fantastic post, Ruben! A few other tricks that I’ve used when doing these:

    – On the recruiting side, I’ve added “We promise we’re not trying to ‘get you back’ if you chose to talk to us. This is all about learning how to make our produce better,” to the message that you show to canceling users when you mention the $100 Amazon Gift Card.

    – To switch things up, sometimes I start the interviews with “What are you using now?” and get the story of them switching to the new (current) solution. As much as this is an interview about them cancelling our service, it’s really an interview about uncovering their “new definition of progress” and understanding their switch to the new solution. Did they go up-market? Down-market? Are we interested in keeping this client (we should change things about our product to satisfy others like them), or are we okay that people like this leave us?

    I’ll post a link to this on Jobstobedone.org so everyone can take advantage of the step-by-step guide you’ve put together here. Awesome work!

    1. Thanks! Great tips for recruiting and starting off the interviews. I’ll make sure to update our recruiting message (we’re doing another batch focused on a specific customer segment now). Interestingly enough 99% of people’s definition of new progress when they cancel is going back to what they originally were using (Word, Google Docs, Pages, etc.).

      1. Oh man that brings so many great questions to mind.

        – Did we promise something that we didn’t deliver on? They thought, “when I buy this, my life will be better because …” and it didn’t play out the way they expected it to.
        – Are we making it “too easy to get in?” In other words, are we attracting people who don’t have real *push* and because it’s so easy to get in (free trial, etc), we get a bunch of people who try, fail, and leave.
        – How can we see these people in our data so that our forecasts get better? This gets back to Des’ concept of Zombie Revenue (we know these people will leave in two months because we can see how they’re behaving).
        – Is this something we need to fix, or are we okay with it?

        1. Nice. Yeah, you’re 100% right, Chris. We dug into this and there’s definitely something to the products in this space only partly meeting expectations. The other part, is that we’re sometimes not dealing with the right customer (also expectations, but there’s more to this one than I can talk about in a comment). While we’ve improved that as much as we can with our current approach, we’re at a stage where we’ll have to make fundamental changes to the product/positioning.

          I also like your last question of whether it’s something we need to fix. I feel like we’ve spent too much time trying to fix something with a specific customer segment (certain types of freelancers) that isn’t worth fixing, but now we’ve got clarity on a good way forward.

  2. Great writeup, and good follow-up point, Chris, especially about asking what they do now. I’ve done some cancellation interviews as well as find them to be so much more valuable than just collecting reasons through surveys.

    1. Thanks! Yeah, we’ve done a lot of surveys and different types of interviews, switch interviews have been the most helpful.

  3. This post and the convo is blowing my mind! So glad I found this (via Growth Hackers).

    Ruben, just a quick question if you don’t mind. How many interviews do you think it takes to unlock definite patterns? Sorta like the 80/20 rule I guess — have you found that a certain amount of chats yield the majority of what you’ll learn from all your calls?

    Cheers!

    1. Happy to hear it’s helpful, Roy! One thing I didn’t mention in the post that I should’ve is that it’s important to focus on a single customer segment. Kind of obvious but we didn’t do this when we first started doing switch interviews and saw patterns that changed once we went deep on customer segments. So if you’re focusing on a specific customer segment I like the number that Chris and Bob give in the workshops: 10 interviews. We typically end up doing around 12 or 13 because we’ll usually get a dud (or an extreme outlier) in the bunch.

      1. Hi Ruben, great article! Is the 10 interview number suggested by Chris and Bob a per month number? Are you still doing interviews? If so, how often do you do them?

        1. Thanks, Jerry! Yep, we still do these interviews from time to time (in batches). The suggested 10 interview number is not a hard rule, just something that Chris and Bob say they’ve noticed once you’ve interviewed that many people you’ll pick up on most of the patterns. Sometimes you might need a few more (or less) but I agree with that number as a good rule of thumb.

    1. Sorry, I’ve got a couple of competitors that are watching (and copying) everything we do, so can’t go into the details just yet.

      1. Sure, I can (kinda) understand that. 😉

        It is always interested to hear how people have applied JTBD and it would be great to hear more, if you feel comfortable.

    1. Thanks! Sorry, I don’t remember off the top of my head but it only took us a few days to get everyone we needed for this (since it was running in the background whenever someone cancelled).

  4. This is awesome, thank you for posting! Is it correct to say that you recruited people who canceled after payment? I’m curious if you have also done this with people who canceled after a trial (and what the differences might be in the stories you hear). Thank you!

    1. Thanks, Jill! Correct, we only recruited people that had been paying and then eventually canceled. Generally, people that cancel after a trial are because of onboarding and activation reasons (there’s some overlap with people that pay and cancel).

  5. Hey Ruben, thanks for this post. I really enjoyed it and found the examples insightful. Did you also do those interviews with higher ARPA customers too that tend to churn less? If so, did you learn something different?
    Thanks!

    1. Glad to hear it! Yep, we’ve done many segments, including our highest value ones. I think this campaign was specific to our highest ARPU segment actually. There were several differences compared to other segments, so it was worth splitting up in our case.

Leave a Reply

Your email address will not be published. Required fields are marked *