Measuring value: beyond retention and cohorts. How to measure what matters most

image

I landed on a website yesterday and in the navbar it advertised: 76,799 people received my free newsletter last week.

And so what I wondered? Yes received is a better metric than just sent, but how many opened it? How many actually cared?

Vanity metrics

You might have heard the story about vanity metrics before, how bad and useless they are, but if you haven't let me break it out for you.

The best definition of a vanity metric I've come across is:

a metric that can only go up

The number of signups since you launched is a good example. That total can only grow and means pretty much nothing because 3/4 of those people might have not stayed on the platform for longer than a week.

But when you're starting out it's probably the biggest legit number in your hands and it undoubtedly feels good to tell people about it. Not to mention that it's unfortunately what will look good in a slide deck for investors.

This is one of the many conflicts that raising money will cause you: you're pushed to focus on growing something that looks good so that you can get the money to work on the actual thing that creates value… sort of backwards if you ask me.

Other examples of vanity metrics are:

  • number of downloads
  • total revenue
  • number of subscribers
  • pageviews
  • minutes using the product (more on this specific metric below since I use it on my homepage!)

Good metrics AARRR

Ok great, so now that we're past vanity metrics the question is, what's a good metric?

One of the most well known frameworks for measuring the pulse of a startup is the AARRR pirate metrics by Dave McClure. It's really awesome system as it's both simple and powerful in capturing all the important stages a users goes through in his interaction with a product or service.

AARRR stands for:

  • Acquisition, users come to the site
  • Activation, users do something key to the platform
  • Retention, users come back and keep doing that key activity
  • Referral, users are so happy that they refer others
  • Revenue, users pay for the value they are getting

One thing to clarify is key activity: the point is for the user to engage with whatever you're offering; if you have a SaaS product signing up for the mailing list might be good, but it's not key to the success of your business - using the service is.

If you want more in depth information about the framework you can check out the original slides here.

One important thing to note about those metrics is that they are powerful at describing how your startup is doing. In other words they change your behavior, which is what a good metric should do.

Cohorts

But even a good metric, if taken broadly, can be misleading.

For example in the case of retention you might have made changes to the user experience of your product so that it's easier to use. If you look at the aggregate across all users on the platform it might look like a month after you made the changes the situation hasn't changed as you hoped and you roughly lose the same amount of people.

But what would it happen if you looked at it grouping users depending on when they signed up? isolating those that have seen the new experience when they were brand new from those that were already familiar with the platform when you introduced it for example

Welcome to cohorts.

A cohort is just a fancy word to indicate a group of people that has a set of common variables, the most common one being time, more specifically signup date.

When you look at it that way it might turn out that being the fifth month since a cohort joined a large number of users was dropping out while the one month old cohort has a far smaller drop than the other group did at the end of their first month. Bingo! You're making progress.

You can do better than good

You'd think by now you would be done, after all you've identified what a good metric is and have a solid path to measure the evolution of your startup as visitors go through their life-cycle as customers.

But you can actually do better. Much better in fact. How? by looking at metrics that directly represent value delivered to the user.

The problem with all those good metrics is that they are about you and your company, but your company only exists to serve a need so wouldn't it make more sense to measure how that need is satisfied?

If you're clear about your value proposition, which you should have summarized in a sentence (you can do that using this template), then you're equipped to find the most important metrics of your startup.

For example if your product is an analytics tool your value proposition has likely to do with helping your customers making informed decisions that help their business move forward. In the specific case of an e-commerce site the tool might help your clients understand where their customers get lost in the buying process so that they can help them find what they wanted leading to increased sales.

Just imagine this: a customer who's paying for the analytics tool, but getting little insight, versus one that's increasing revenue thanks to it. Both are paying customers so retention and revenue metrics would be looking equally good, but if your mission is to empower customers to grow their business only one screams success.

An example with Spikelab.org

Here's what it would all look like if I applied the above to my own site.

Value proposition

My goal is to help entrepreneurs because they are the single best chance we have to change the status quo by coming up with awesome stuff that helps people.

I offer help through blogging, free office hours that anybody can book and a mentoring program limited to a few particularly promising entrepreneurs.

Vanity metrics

On my home section I actually share two numbers:

  • total calls done
  • total minutes spent talking to entrepreneurs

Ironically those are at all effects vanity metrics. What the heck, didn't you just say how bad and evil they are?!. I hear you, read on.

The keys is that I'm not using them to assess my business. Rather, I'm publishing them alongside reviews and recommendations from other people I helped as a form of social proof. In that sense they have the same function of the VC-pitch use case.

AARRR

If I mapped the AARRR framework to this site and the mentoring calls this is what it'd look like:

  • Acquisition, people visiting this website
  • Activation, people read a blog post or book a call
  • Retention, people who read a post or booked a call do it again
  • Referral, people share my blog posts or refer people to book a call with me
  • Revenue, NIL - this year I'm experimenting with a year of volunteering so everything I do is free - crazy I know, but you gotta try everything.

That's a pretty good picture of what I need to watch out for and optimize to make sure I'm succeeding, however it leaves out the most important thing…

Measuring value

What do my visitors care about? their startup! More specifically to make their startup into a success.

They don't care about me or my site, not yet at least, so why am I exclusively measuring stuff about that?

In some cases there's a straightforward answer: you can't measure the real thing.

For example since I provide value through blogging, or so I try at least, the thing I'd really want to measure is the impact of actions my visitors take after reading what I wrote.

So for example once you've finished reading this article you may decide to change what you measure and as a result create and capture more value. That would be the most awesome thing, however what are the chances I'll ever figure if it happened? (you could always tell me tho, but statistically speaking that has a 0% probability to happen).

And so I use a proxy, two to be precise:

  • time spent reading
  • returning visits

If you take the time to read through a post thoroughly when most people just skim headlines that's a good sign it's adding value. Likewise if you keep coming back for more there's also a good chance that I'm doing something right.

The same happens with the calls, especially if people only call once. Every call the feedback I hear is positive, people leave great reviews and when we part ways they say to have a much clearer perspective on the problems we discussed and that they'll certainly follow through. However I have no way to prove that I've been helpful, i.e. that our conversation will result in them overcoming the problem we talked about and moving their business forward.

Again I use three things to asses value delivered by proxy:

  • the review
  • booking another call
  • referring somebody

If you take my mentoring program however the situation is pretty different.

In that scenario we're working 1:1, we fully trust each other, talk often and in depth. Because of that I have a deep insight into your startup which means I'm not only able to help more effectively, but I can also see directly if my work is having an impact or not.

This is why I prefer ongoing calls and started the mentoring program: I get a real shot at helping entrepreneurs.

If you can design your product or service so that you have direct access to the value created you are in the best position to help and delight your customers.

Are you measuring what matters? Let's talk

I realize that a blog post only goes so far and your situation is unique and as such it might be tricky to put the above into practice. That's exactly why I run free office hours.

Regardless if you're at idea stage or deciding how to design your MVP, I'd love to jump on a call and talk about what you're doing and what metrics you're picking.

Best case scenario you leave with a better set of metrics. Worst case you have a stronger confirmation to be on the right track.

It's as easy as picking a time that works for you from the list below and it won't cost you a penny.



Spike is awesome! He got to the heart of our challenge quickly and provided our team with wonderful insight that helped put us on a clear, straight path to getting our product to market - in just one phone call! Even better, Spike is a generous man on an authentic mission to help people be successful in life and business.

― Michelle May, Shenami

Spike was just great. He gave me the helicopter view of the path we are on, what we could choose to do now, and what lies ahead. He was generous with suggesting practical steps for right now, and also things to work towards. I hope to talk with him again.

― Carrie N Ballard, Utrecht Area, Netherlands

As soon as Spike started to speak - in a thoughtful, careful and considered way, I knew I was going to really benefit from listening to him. Spike had some good answers to my questions around pricing and ways to look at it not only in terms of finding a sweet spot, but also using it as a way to assert our values. And great follow up advice by email - thanks!

― Bridget Harris, YouCanBook.Me

I approached Spike unsure of what my next steps should be. He helped us to figure out just that and identify the key actions to build an experience that will wow our customers. Spike also pointed out very interesting strategies for testing monetization and getting that engine of growth going. Very useful. Thanks!

― Anders Hasselstrøm, StartupTravels, Denmark

Spike is incredibly insightful. He has a way of asking the right questions that gets you to see problems from a new perspective and with greater clarity. Every conversation with Spike, I walk away with an 'Aha!' moment that helps me move forward.

― Steven Kim, San Francisco

I am incredibly inspired by Spike's advice + expertise in regards to applying lean start-up to my business. Spike is very knowledgeable, sharp and supportive. He's just the mentor we need and I am excited to be part of his mentoring program!

― Nicole Kasal, Cleanse Culture, Chicago

Spike has a sharp mind and a well articulated analytic sense that cuts to the core of any question. The help he provided on Startup Advice allowed me to clarify my goals and refine who I think my audience is. I highly recommend talking to Spike!

― Paul V. Weinstein, Deal maker and fundraiser, SF

Spike asked hard questions and within the first few minutes of our conversation homed in on the core of most of my confusion. I now have an actual plan of attack that will get me the information I need to make the right decisions to move forward.

― Katrina Owen, exercism.io

TIMES SHOWN IN YOUR TIMEZONE:
comments powered by Disqus