Is your startup going in the right direction? Introduction to metrics for startups

Why should you care about metrics?

During all my years as a mobile developer and now as a product owner, I was always fascinated by the particular breed of people I loved working with – founders. No matter the domain – weather app, challenger bank or new femtech device – the audacious vision they usually had and the I-don’t-know-but-I’ll-figure-it-out attitude is something that made working for them extremely rewarding. The most accurate description of this “founders phenomenon” and the impact it has on workers can be found in Steve Jobs’ biography written by Walter Isaacson. In it he states that Jobs possessed an extraordinary ability to convince himself and others around him, to believe almost anything with a mix of charm, charisma, bravado, hyperbole, marketing, appeasement and persistence. It was said to distort his co-workers’ sense of proportion and scales of difficulties and to make them believe that whatever impossible task he had at hand was possible. Jobs was creating a reality distortion field around himself, and so are you Dear Founder. That’s why it’s so pleasurable to work for you. But is it optimal in the long run?

A good proportion between people blindly following the vision and the people asking tough questions is a must for a business to succeed. That’s why I used to ask questions. A lot of them. Trying to keep management as close to reality as possible:

• Are you sure that we found product-market fit only because 1000 people downloaded our app last weekend?
• What is our unique value proposition? Why would users even choose us among all the competitors?
• It’s nice that we have 100 monthly active users, but are they in our target audience? How much time are they spending on the app? Can we monetize them in a way which will exceed the cost of user acquisition?
• How do you know we should pivot now?

All these questions are meant to start meaningful conversations and help everyone in a team to align and be on the same page. But probably the most important thing about them is that they could all be answered with data – correct metrics set up in correct places and visualized in a correct way. This is exactly why you should care about metrics in your startup. They aid in preserving the balance between your reality distortion field and the harsh market reality, which leads to better decision making, which, in the end, can lead to your company becoming a unicorn.

In this blog post, I’ll give you a solid introduction to the world of analytics and metrics for startups. I’ll start by showing you the tool that Pragmatic Coders very often uses with founders during our workshop sessions and explain why it’s important with regard to the topic we’re discussing. Later, I’ll describe what a good metric is and what are the most common types of metrics, including the dreaded vanity metrics. Next, you’ll read some of my thoughts about setting targets in your analytics. I’ll close the article with an introduction to the most popular framework used with metrics in startups – the Lean Analytics framework.

The Lean Canvas

In the previous paragraph, I stressed the importance of finding a good balance between a reality distortion field and a market reality. I have also mentioned how valuable a candid discussion is for business health. What’s more, you’ve probably already started thinking about what to track in your startup using some common metrics, but maybe you wonder if you missed anything? From my experience, there is one tool that can address all of the issues above. It’s called the Lean Canvas.

You can think of it as a one-page definition of all crucial areas in your startup. Plan a meeting with your team to fill it just before you decide what metrics to choose, and you’ll be amazed by the level of clarity the Lean Canvas can provide. During workshops with our clients, we always use sticky notes to complete each block in the canvas since they can be easily discarded, changed or moved, promoting more flexibility, even long after the meeting is over. Take a look below for a short description of the most important sections from the Lean Canvas.

Problem

You should definitely try to fill this block first. Don’t start with a solution. Try defining a problem you’re solving. For example, Uber solved the problem of unreliable taxi rides, which were often too costly and sometimes took forever to arrive without the client knowing how much longer he should wait.

Customer segments

Now go to this section. Do you know for whom you’re solving the problem? Is it only one group of customers, or are there multiple ones? At the very beginning, you need to be focused. For example, Facebook, at first, wasn’t targeted at a mass audience – you had to be a university student to get in. Founded in 2004, it opened its doors for all people over the age of 13 only in 2006.

Unique value proposition

Your app or website will do a lot of things, but is at least one of them significantly different from the ones the competitors already provide? Try to be concise, clear and honest. For example, all of my friends didn’t create a Revolut account because they liked the UI or wanted a new place to store their money. They loved the fact that Revolut provides the easiest and cheapest way to exchange currency and then use it abroad.

Solution

You’ve already defined common problems for given customer segments. How are you going to solve them? For example, the problem of unreliable taxi rides was solved by Uber with a mobile app containing an interactive map. On it, users could see where their taxi was. In addition, before they got in, they could see an estimate of the fare they would have to pay.

Channels

Connecting with customers and potential customers is crucial not only during the launch of a product but at every step of the startup building process. Do you know how to reach them? Where can you find them? For example, if I were to create an online course about programming for beginners, I might check active Facebook groups with people trying to learn a new programming language, go to student job fairs or stand outside the technology university in my hometown, waiting for computer science freshmen to come out after classes. A carefully prepared LinkedIn post wouldn’t help me much or could even attract an incorrect type of audience.

Revenue streams

How are you planning to make money? Subscription? Freemium? Ads? One-time payment like e-commerce? For example, Airbnb takes the majority of its revenue from booking service fees charged to both guests and hosts.

Cost structure

What are some of the most important costs you’ll have to deal with? For example, many Pragmatic Coders clients, during their first stages of product building, spend the majority of the money on development costs, including UX services.

Unfair advantage

The things you’ll put here can’t be easily copied or bought, making your startup stand out even more. For example, patents, existing community, domain experts in the team, established personal brand with a large following, years of industry experience, and insider information.

Key metrics

Filling all of the previous blocks should have given you the clarity to come up with metrics that will indicate if your startup is going in the right direction. But don’t dive into this block just yet! In the next paragraph, I’ll show you what are the characteristics of effective metrics for your startup, along with a few common pitfalls you might come across.

What makes a good metric for a startup?

Below you’ll find a list of the most important characteristics of an effective metric. There is no need to follow each bullet point religiously. Every business is different, but based on my experience, having a checkmark next to all of them makes you, as a founder, less susceptible to being uninformed about the real condition of the product you worked so hard to put on the market.

Simple

This one is crucial. Imagine that a metric you’re trying to come up with is put on a large screen in your office, next to a team trying hard to make the product succeed. Would they understand it at first glance? Would they know how it’s being calculated? Would it require any explanation whatsoever for a new employee? The harder it is for team members to grasp, the harder it will be for them to act or even notice when the metric will go in the wrong direction. Complexity is your enemy here, not only when it comes to quality but also quantity. That’s why Alistair Croll and Benjamin Yoskovitz, in their book “Lean Analytics: Use Data to Build a Better Startup Faster”, popularized the term OMTM – One Metric That Matters. The idea behind it is that at every stage of your business there is one number that you should focus on. It doesn’t mean that you ignore everything else, but, to promote focus, you should actually work on improving only this number until you move to the next stage of your startup (you can read a bit more about these stages in the last paragraph). I highly recommend this approach.

Comparable

When defining a metric, ask yourself if it’ll be easy to compare. You might want to be able to compare it to some standard points of reference. For example, here you’ll find email marketing benchmarks prepared by Mailchimp, based on the vast amount of data they have. Find your industry and check if there is room for improvement in metrics such as average open rate, average click rate or unsubscribe rate. Maybe you’re developing an iOS app which generates the majority of revenue from targeted ads? Here’s an interesting link to a blog post from one of the largest mobile analytics providers – Flurry. Is your opt-in rate below or above the average?

A good metric can also be compared to competitors. Say you’re developing a product that will directly compete with Brand24 – a media monitoring tool. Their CEO is quite vocal on LinkedIn about the milestones they achieve. Use it to set realistic targets for your metrics.

To make your metric comparable to other time periods, user segments and cohorts stick to a ratio rather than a plain number. Ratios are comparable by nature, make trends more visible and help to define whether the startup is really growing. For example, the number of new users the app gained this month does not tell us much until we divide it by a one-year average of the number of new users creating an account every month.

Actionable

A good metric translates to a change of behavior in your startup and informs business decisions. It should be constructed in a way that empowers employees and management to act when the metric goes down or suddenly skyrockets. This characteristic is especially visible when performing product experiments, during which an experimenter starts by setting a clear hypothesis, for example, “Our vacation rental business will generate 30% more revenue in the next 3 months if we use professional photos of the apartments in our marketplace”. What should founders do if the hypothesis is correct? Is a metric change leading to action? Is revenue the correct number to track in this case, or can founders be informed a bit earlier than they’re onto something? Whereas the answer to the last question remains debatable (maybe an increase in the number of bookings on a property compared to a previous month might be a better option), the answer to the first two can be observed on the Airbnb website.

Accurate

There are three most common factors that might mess with the accuracy of metrics in a startup: technical, data related and psychological. Technical are often the simplest to fix. For example, data from the analytics tool might not be reaching the server due to a bug in implementation or misconfiguration. That’s why it’s so crucial to thoroughly test every user path before deploying the code to production. When you’re receiving product statistics, but they are really polluted, e. g. they have missing values that might be interpreted as zeros, you are dealing with data-related factors and probably need a small help from a data scientist. Inaccuracy connected with human psychology tends to be a bit harder to tackle, but it’s crucial for founders to be aware of this phenomenon. It can be summarized by a famous quote from the TV series “Dr House”:

For example, which of the below metrics would you consider more accurate to evaluate your new app idea?

• Per cent of your family members and friends who answered “Yes” to the question “Do you like my new app idea?”.
• Per cent of people from a simple targeted Facebook ad, who clicked the link taking them to the landing page of your new app, which you haven’t even started developing yet.

The description of ways to alleviate this problem might require another full-scale blog post, so for now, I’ll just leave you with increased awareness of its existence and move on to the most common types of metrics you’ll come across.

Vanity metrics

A few months ago, I was looking for a reliable Facebook group about investing. I was still new to this and wanted to see what others thought about some of the decisions I’ve made. After a short research, I found two:

• 15 000 users and, on average, over five posts per day.
• 700 users and, on average, one post per day.

I chose the first one and posted a question. After 3 days, my post had 0 reactions and 3 comments from what appeared to be bots asking me to make some shady investments in crypto. Questions from other users seemed to be following the same pattern. I didn’t give up. I joined the second Facebook group with 700 users and immediately got an absolutely fabulous set of responses from people who knew a ton about investing. Now imagine the owners of these groups having a small party with their friends. Who will impress more? “I have a small but very active Facebook group with 700 members” or “That’s nice, but I already crossed the 15k milestone and have 700 members join me every week”?

Vanity metrics are metrics that make founders look and feel good but have very little to do with the actual performance and health of the startup. They make for a great headline in the press release or story during a party but don’t provide enough context to make informed business decisions. In other words, they tick all the boxes from the “What makes a good metric for a startup?” paragraph except for the most important one – “Actionable”.

Vanity metrics can sometimes be tricky to spot, so below are a few common examples. Since every metric has the potential to become a vanity metric, be careful with the suggestions I’ve put in the brackets and embed them in your startup’s business context first.

• Page views (probably better metrics to focus on: time spent on a page, number of new subscribers per 1000 page views)
• App installs (probably better metrics to focus on: time spent in-app, monthly active users)
• Number of users on the email list (probably better metrics to focus on: open rate, click rate)
• Social media followers (probably better metrics to focus on: average post engagement)
• Total revenue (probably better metrics to focus on: customer lifetime value, cost of user acquisition)

Other categories of metrics for startups

I’ve already described the characteristics of practical metrics: simple, comparable, actionable and accurate. They are opposite to dreaded vanity metrics, which don’t have that much to do with the real condition of a startup. There are, however, other contrasting categories that metrics can fall into. Use them to further refine and discuss with your team the reason for tracking a given number.

Qualitative vs quantitative metrics

I see a tendency among founders to quantify everything, and, trust me, I’ve been there myself. After all, it’s easier and less chaotic to create a simple poll asking users to rate a sign-up experience of the app from 0 to 10 and then plot the data using an Excel spreadsheet. Many people also think that this way of collecting information is statistically significant since they get thousands of responses in their surveys. It has to be, right? Well, not quite. The quantitative way, based on numbers, plots and order, is often not the best one. Personally, I’ve learnt more about flawed sign-up experience in my app after watching only 6 users go through it than by looking at the numbers in the analytics tool. Both qualitative (output from an in-depth user interview) and quantitative (rating of sign-up experience from a user survey) metrics have their places when looking for product-market fit, but, from my experience, talking to users directly and listening to them usually gives founders the best results at the beginning of the journey with their startup.

Leading vs lagging metrics

Imagine that the churn rate in your SaaS company was 2% last month, which means that, compared to the beginning of the last month, 2% of subscribers discontinued using the software and stopped paying for it. Churn rate is a solid metric to track. It’s definitely simple to understand, can be compared to the competition, provokes action (e. g., when after removing an old feature, your churn rate increases, you should probably put it back) and is usually very accurate since it affects the bottom line. It is however a lagging metric because the damage is already done – the user deleted the account or cancelled a subscription, and, most of the time, there’s very little you can do about it. On the other hand, the leading metric of the event of losing a subscriber could have given you an indication that something was about to happen way before it actually did. For example, a number of software crashes or a number of complaints in a given month may lead to users being very frustrated with your service and, in effect, discontinuing to use it a few weeks later. In order to have a full picture of the business reality, startups need to track both leading and lagging metrics.

Correlated vs causal metrics

Correlation does not imply causation. Some time ago, I found a nice explanation of this old statistics rule reading The Guardian: “Just because people in the UK tend to spend more in the shops when it’s cold and less when it’s hot doesn’t mean cold weather causes frenzied high-street spending. A more plausible explanation would be that cold weather tends to coincide with Christmas and the new year sales”. Finding the causation, especially for such valuable metrics as monthly recurring revenue, is the Holy Grail for many startup founders. It could possibly give the opportunity to control the factor that’s causing the metric to move and eventually swing it in the company’s favour. Looking for causation starts with finding a correlation and then performing an experiment by controlling every other factor. It is very difficult to achieve, especially in a dynamic startup environment, but can give a hint about what’s working and what’s not.

Setting targets for startup metrics

Once you carefully craft a practical set of metrics, you will eventually reach the point of asking “Am I doing ok?”. It’s nice to know that your churn rate is 2%, but should it be better? Maybe it would be more beneficial to stick to 2% and improve on other numbers in the analytics? Setting targets for startup metrics is always tricky. There are myriad ways of going about it, so let me just focus on a very subjective approach stemming from my experience building apps used by millions of users.

I believe that it’s enough to set ANY target for a metric as long as it loosely fulfils 3 requirements. First of all, it must not be too outrageous. Yes, you are a visionary, but not everybody in your team is. The goal should challenge them, not make them cry or laugh at you when you’re not watching. Setting a correct magnitude of the metric’s target can increase morale and productivity among you and your colleagues. Secondly, it must not be too low. The business has to make money from the users engaging with the product. Now is a good time to take an educated guess if, e. g. gaining five new users per day for the next month is enough to eventually make your startup sustainable. Use common sense, industry knowledge, available competition reports and benchmarks to avoid under or overestimation. Lastly, the target has to be time-bound. In other words, the success criteria for a metric require a specific value and a date before which this value should be reached. The self-imposed time constraint will force you to frequently ask the most important question you can probably ask yourself as a founder of a startup: “Why did it fail, and what can I do to avoid it next time?”. Maybe the metric target was too ambitious? Maybe there’s a huge leak in the conversion funnel? Or maybe it’s finally time to pivot?

I want to emphasize my point about using common sense, industry knowledge, available competition reports and benchmarks to set targets for metrics. The order here is not random – common sense is the most important thing in this process. Take, for example, a metric often referred to as time spent in an app. If its value exceeds 30 minutes per user, you might be happy, right? Maybe now it’s time to even increase the target to an hour? Think again. What if users are spending all this time writing complaints via in-app support chat? Or what if your app is a productivity one, created to save users’ time? It’s always a good idea to be careful and never allow yourself to compare metrics to industry benchmarks or competition reports without further diving into the business context of what’s exactly being tracked.

The Lean Analytics framework

After spending all this time learning about the motivation behind tracking metrics, the Lean Canvas, characteristics of practical metrics, different categories of metrics and setting targets, you might be a little overwhelmed. That’s why I want to conclude this article with a short introduction to the Lean Analytics framework, originally proposed by Alistair Croll and Benjamin Yoskovitz. It consists of five consecutive stages. Your business can only be at one stage at a time, and you can move on to the next one only if one of the metrics (often defined as OMTM – One Metric That Matters, see paragraph: “What makes a good metric for a startup?”) shows a clear indication of reaching the “gate”. The Lean Analytics framework promotes focus and can help you come to order with various metrics you want to track.

The stages of the Lean Analytics framework:

• Empathy (“gate” needed to move forward: I’ve found a real poorly met need a reachable market faces).
• Stickiness (“gate” needed to move forward: I’ve figured out how to solve the problem in a way they will accept and pay for).
• Virality (“gate” needed to move forward: I’ve built the right product/features/functionality that keeps users around).
• Revenue (“gate” needed to move forward: The users and features fuel growth organically and artificially).
• Scale (“gate” needed to move forward: I’ve found a sustainable, scalable business with the right margins in a healthy ecosystem).

You can read more about this and other analytics frameworks in Chapter 5 of “Lean Analytics”. And since a picture is worth a thousand words, below I’m attaching an illustration comparing most of them.

Summary

In this article, we took a closer look at the most important aspects any founder needs to take into account when measuring the success (or failure) of a startup. We talked about the motivation behind tracking metrics, the Lean Canvas, characteristics of a good metric, vanity metrics and other metrics categories, and targets, and finally, we wrapped up with an introduction to the Lean Analytics framework proposed by Alistair Croll and Benjamin Yoskovitz. Suppose you’re ready to dive into the world of entrepreneurship and turn your vision into reality. In that case, it’s crucial to not only understand the importance of metrics but also to have a clear roadmap for building your business. For those considering this exciting journey, you might find it helpful to learn about the key steps to start a fintech company to ensure you’re laying a strong foundation for success. I do hope that this knowledge will help founders remain visionary, but also bring them closer to reality, which sometimes is harsh even for the best products we’re trying to bring into the market.