Follow Reforge on Feedspot

Continue with Google
Continue with Facebook

Reforge by Brian Balfour - 7M ago

This is the third post in a four post series co-written by Brian Balfour, Casey Winters, Kevin Kwok, and Andrew Chen. Subscribe here to get the rest of the series. To go deeper on this concepts, join us for one of our upcoming fall programs - Growth Series, Deep Dive: Retention + Engagement, or Deep Dive: Growth Loops + Models.

Brian Balfour
Founder/CEO @ Reforge. Former VP Growth @ HubSpot.

Casey Winters
Growth Advisor, Former Pinterest, Grubhub. 

Kevin Kwok
Former Greylock Ventures. 

Andrew Chen   Partner @ Andreesen Horowitz

The AARRR funnel framework has been the dominant guiding framework to metrics, goal setting, and strategic growth conversations. Funnels were a good starting point but do not accurately represent how the fastest growing products grow. It is time to move past the funnel framework and focus on Growth Loops.

This is the third post in a four post series. In the previous posts in the series we went through three important points. Growth wins, the game has changed, and to adapt we need a system of product, process, and team.

This system is just the beginning. We need new frameworks and tools to think about how products grow that incorporate these changes to growth and the lessons we've learned.

The Most Important Question Your Team Should Be Able To Answer

One thing we ask participants in Reforge Programs is to go around their company and ask five different people to whiteboard the answer to a seemingly simple question: How does your product grow?

This seems like a simple question. But what everyone inevitably finds is one or more things:

  1. Everyone has a different answer
  2. The answer represents only one piece of the puzzle
  3. The answer talks about the output of $$$, but not the inputs of usage

This is a BIG problem. If everyone has a different or incomplete picture of how the product grows, then you can't have apples to apples discussions about priorities, metrics, goals, or strategy. This leads to a few things:

  • People focused on different things
  • The teams moving in opposite directions
  • People not on the same page with CEO/exec team/others as to what is most important

“How does your product grow?” is simply the most important question to be able to answer. Growing is the entire reason why products and companies exist (especially in venture backed startups). Companies that continually grow also provide the largest positive outcomes. More importantly, personally in your career if you drive growth at your company, you are rewarded vs others who do not drive growth.

So, what is the best way to answer this question?

Funnels Are Not The Answer

One of the common answers to “How does your product grow?” is a picture of a funnel. The funnel AARRR framework was originally created by Dave McClure. It was a great starting point. It helped me and millions of others level up their game. But the framework is now > 11 years old and since then we've learned a lot about how the fastest software products grow.

The biggest thing we've learned is that the funnel framework is too micro of a view in order to answer “How does your product grow?” It helps explain a specific step within a Growth Loop, but misses the larger picture of the loop itself. When the funnel is applied at the company level and used to explain how a product grows, it leads to a few common issues:

Funnels Create Strategic Silos

When building a new product, the most common approach we see is to “build a great product” and then test a lot of different channels to see what works. This is exactly the wrong way to approach it. This treats product strategy and acquisition strategy in silos. In larger more developed products, you see this silo'd strategic planning as well. Typically the product team goes off and plans their product strategy and then marketing goes off and creates the acquisition strategy.

This silo'd strategic thinking is the cause for most distribution failures. Product Channel Fit tells us why. We commonly forget that we do not control the rules of the channels. The channels control the rules. As a result, we have to mold our product to fit the channels, not the other way around.

To make it worse, we also tend to treat our monetization strategy in a third silo. But, we know due to Channel Model Fit, our monetization model enables or disables certain channels.

Product, channels, and monetization need to be thought about together. They are interlinked. But the funnel framework leads a lot of teams to treat these as silo'd layers.

Funnels Create Functional Silos

It is common for companies to structure teams by layers of the funnel. Marketing owns acquisition. Product owns retention. Sales (if B2B) owns revenue. Then each one of those teams is given a metric that corresponds to that layer of the funnel.

The problem is that the teams then optimize at the expense of each other in order to reach their silo'd goal. Marketing brings in low quality users/leads at top of funnel to hit their goal, but that tanks retention or further down funnel metrics. All sorts of checks and balances get put in place over time to try and fix this which ends up complicating the understanding and goal setting of the metrics.

Funnels Operate In One Direction

Funnels operate in one direction. Put more in at the top, get more out at the bottom. There is no concept of how to reinvest what comes out at the bottom to get more at the top to continue to feed growth over time. In other words, no compounding effect. This means we have to keep putting more into the top to get more at the bottom. More money, more people, more tactics, more channels, more, more, more. This is unsustainable. Understanding the connection of how you reinvest to get more growth changes the way you think about where to focus and what to invest in (more on that below).

What is a framework that represents how the fastest companies grow? One that combines product, channels, and monetization into one system? One that looks for compounding growth vs linear growth?

Growth Loops
"Compound interest is man's greatest invention." - Einstein

The fastest growing products are better represented as a system of loops, not funnels. Loops are closed systems where the inputs through some process generates more of an output that can be reinvested in the input. There are growth loops that serve different value creation including new users, returning users, defensibility, or efficiency.

Here are a couple examples:


The driving force behind Pinterest's growth is the following loop:

  1. User signs up (or returns)
  2. They activate you on the product with specific/relevant content
  3. You save new content or repin existing content which gives Pinterest quality signals
  4. Pinterest distributes the quality content to search engines
  5. A user finds the content via search engines and either signs up/returns (see step 1)

Credit: Casey Winters - https://caseyaccidental.com/what-are-growth-teams-for-and-what-do-they-work-on/


One of the driving forces of SurveyMonkey's growth is the following loop:

  1. New user signs up
  2. % creates a survey
  3. % send that survey to others
  4. As people finish the survey they see a Survey Monkey landing page
  5. % of those sign up over time repeating step 1

These are two of over 20 growth loops we've identified in our research for the Growth Models Deep Dive program that drive acquisition, retention, defensibility, efficiency, or a combination. Those that understand them and organize their product/teams around them will be the ones who create the most value. There are two primary reason why Growth Loops are the key to the fastest growing products.

Loops Provide Sustainable Compounding Growth

Loops force you to answer “How does one cohort of users lead to another another cohort of users?” You focus on how you reinvest the output of one cycle of the loop into the next cycle of the loop to get more output. This creates a compounding effect that is more sustainable.

Not all loops are created equally. You'll be tempted to draw a ton of loops for your product, but what that typically means is that you just have a ton of low powered loops that aren't sustainable. The fastest growing products are typically powered by 1- 2 major loops that transition over time. Measuring and understanding the power/health of your loops is critical to understanding where to focus.

Loops Are More Defensible

Loops combine how your product, channel, and monetization model work together in a single system rather than treating them as silos. As a result they end up being more specific to your product and company making them harder for others to replicate.

On the other hand, strategies and tactics that aren't specific to your product/user/model by definition can be replicated with ease by others. As they get copied, effectiveness decreases and always trends to zero requiring you to constantly invent new strategies and tactics. This is not sustainable over the long term.

Loops Change Everything

Once you start looking at things through the loop framework, you start to make very different set of decisions.

You Approach Growth From A Different Perspective

Once you start viewing things through loops, you stop approaching acquisition, product, and monetization in silos. It forces you to think about how the three work together in a system. You stop thinking about the never ending cycles of more tactics, more channels, more of everything just to keep filling the top of the funnel, and you start thinking about how what you are producing can be reinvested.

You Make Investment Decisions Differently

If you had two options, which one would you choose?

  • Initiative A: Output of the initiative gives you 500 new engaged users this week, but nothing after.
  • Initiative B: Output of the initiative gives you 20 new users in week one, 22 in week 2, etc (growing 10% WoW) for every week going forward.

We really hope you choose Initiative B. This highlights how you make investment decisions differently. Rather than looking for the short term bumps and sugar rushes, loops help you start looking for the things that will compound over time producing much better results over the long term.

You Organize and Goal Teams Differently

In the second post in the series, we talked about the larger need for Cross Functional teams as product, data, engineering, and design play a larger role in outcomes like acquisition, retention, and monetization. Loops as you can see above traverse typical functional lines. To enable and improve them you typically need every function represented working towards the same goal, the output of the loop. This helps the teams align and organize around the loop rather than by function and reduces the teams optimizing at the expense of each other as it will be reflected in the output of the loop.

Putting Loops Into Action

Understanding loops, how to measure them, and how to map them to your product is just the first step. It is a phenomenal qualitative tool to change the way you think about growing a product. But it is hard to represent all the individual levers and their effect on your metrics. You need to translate your loops into a quantitative growth model to help communicate, prioritize, make strategic bets, set goals, and drive your metrics roadmap. We'll talk about this in the next (and final) post of the series. Subscribe here to make sure you don't miss it.

Go deep on Growth Loops + Models in our upcoming fall program along with other experienced practitioners from Google, Facebook, Spotify, Adobe, HubSpot, and many more. We'll go through the properties that make a loop, detailed examples of 20+ growth loops, how to measure/analyze your loops, and how to build quantitative models.


Subscribe here to receive the rest of the posts in the series. If you or your team are practitioners with 3+ years of experience, check out one of our upcoming Fall Programs:

  • Growth Series - A comprehensive program on how to construct and operate a repeatable, predictable, and sustainable growth machine.  Created and hosted by Brian Balfour (former HubSpot) and Andrew Chen (Andreesen Horowitz and former Uber).
  • Retention + Engagement Deep Dive - A deep dive on how to measure, analyze, and improve retention and engagement.  Created and hosted by Casey Winters (former Pinterest/Grubhub), Shaun Clowes (Metromile and former Atlassian), Brian Balfour (former HubSpot) and Andrew Chen (Andreesen Horowitz and former Uber).
  • Growth Loops + Models Deep Dive - One of the most impactful frontier topics, this program dives deep on establishing compounding, defensible growth loops and how to model them quantitatively.  Created and hosted by Casey Winters (former Pinterest/Grubhub), Brian Balfour (former HubSpot) and Kevin Kwok (former Greylock Ventures). 
Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

This is the second post in a four post series co-written by Brian Balfour, Casey Winters, and Kevin Kwok. Subscribe here to get the rest of the series. To go deeper on this concepts, join us for one of our upcoming fall programs - Growth Series, Deep Dive: Retention + Engagement, or Deep Dive: Growth Loops + Models.

Brian Balfour
Founder/CEO @ Reforge. Former VP Growth @ HubSpot.

Casey Winters
Growth Advisor, Former Pinterest, Grubhub. 

Kevin Kwok
Former Greylock Ventures. 

Let's travel back in time to 2007.  Facebook is at 50 million monthly active users, growing at a rate to hit about 300 million monthly active users by 2012.

Chamath Palihapitiya (now Founder at Social Capital) recalls in a Recode Decode podcast a conversation with then-new COO Sheryl Sandberg.

Sheryl: “What do you want to focus on?”
Chamath: “I'm going to change the product, do some SEO and SEM, apply some algorithms, etc.”
Sheryl: “What do you call that?”
Chamath: “I don't know, I call that Growth. You know, we're going to try and grow. I'll be the head of growing stuff.”

There you have it, the first Growth Team was born. Facebook's MAU growth changed trajectory. Rather than finishing 2012 with 300 million MAU's, they reached 1 billion MAU's.

Was that team 100% responsible for that growth? Of course not. But they played a big part, and as a result almost every high-growth company in tech established their own growth teams including LinkedIn, Pinterest, Uber, SurveyMonkey, Airbnb, Slack, and more.

Since then, the term “Growth” has been slapped onto everything. Growth marketing, growth hacking, growth engineering, growth product, blah blah blah. It has created a ton of confusion and negative biases. Every week we get questions like, “What is the difference between product and growth team? Growth marketing vs. marketing? Isn't growth just a bunch of local optimization? WTF is growth?”

The term has taken on so many different meanings that it ultimately creates more friction and obscures what we should really be focusing on.

Whatever your opinion is towards the term, throw it away for a second. The term does not matter. The important thing is to realize that all those high-growth companies embraced what we talked about in the first post in this series - Growth wins. The game has changed.

To review how the game has changed, we explained four things:

  1. Distribution has become more competitive and expensive.
  2. The lifecycle of channels/tactics has accelerated.
  3. The accessibility of data has increased and costs have decreased.
  4. The lines between product/engineering/marketing/sales have blurred.

So, what do we do about it? The first step is that we need to approach our growth strategy with the same discipline, effort, and investment as we do in our product strategy. That means building a system towards growth that embraces these changes and is:

  • Systematic
  • Deterministic
  • Repeatable
  • Sustainable

This system involves three high-level areas, and as the game has changed, so have these areas, which we will break down in more detail:

  1. “Growth Product”
  2. “Growth Process”
  3. “Growth Team”
Growth Product: The Forgotten Areas Of Product

In Growth Wins, we talked about how the lines between product/marketing/eng/sales have blurred:

Data, technology, and product play a much larger role in outcomes like acquisition, retention, and new sales. The lines have blurred, but most of orgs are still in silos.

Areas like new user experience, retention, optimizing monetization flows, tech infrastructure behind acquisition and communication channels (i.e. email/push/paid), and more have a massive and direct impact on overall growth of a product. All of these areas require product/eng skills to move the needle in a meaningful way.

We call these the forgotten areas of product because these areas previously have been ignored, abandoned, avoided or at best received a fraction of the resources because they didn't fit into what most product teams see as their primary purpose and output - new core product features.

Casey Winters (former Growth Lead at Pinterest/Grubhub) separates these areas in his definition of Growth:

Most product teams are built to create or improve the core value provided to customers. Growth is connecting more people to the existing value.

There are times in a product's life that new feature development is the biggest growth lever, but too often this becomes the default thinking and teams never shift out of this. Building a discipline around the areas of product, retention, engagement, and monetization that incorporates the right mixture of skill sets is the first part of the system.

Growth Process: Hypothesis-Driven Experimentation

Every team has a process to solve the problems they are tasked with. Design Thinking, Agile, Waterfall, Campaign Planning, etc are all different tools to solve different problems.

In Growth Wins, we talked about how the accessibility of data has massively increased:

With the slew of data tools, data has become cheaper and more accessible. More people in the org have data at their fingertips especially those working on growth initiatives. We are at the tip of the iceberg on new methods of personalization, machine learning, and deeper insights via data to drive growth.

This is important, because it has made a new process much more available. One that is ideally suited to solve growth-related problems.

Steven Dupree (former VP @ SoFi and LogMeIn) has the simplest explanation of this process. He calls it the “Scientific Method applied to KPI's.”

The basic steps of this process are:

  1. Build a growth model that helps you identify the most impactful variables/levers around acquisition, retention, engagement and monetization.
  2. Understand the psychology of users behind those variables/levers.
  3. Develop hypothesis-driven experiments informed by your growth model and user psychology.
  4. Apply the learnings back to your growth model and user psychology to get better over time.

Some have been doing this for years and will scream, “This isn't new!” It isn't. But the process has become available to everyone with the massive increase in accessibility to data and infrastructure required to run this process effectively.

Steven's description is great, because just like in Science, it isn't all data or all intuition/creativity. You have to combine both to have the biggest breakthroughs. The important thing is that the process is designed to drive towards the truth of what actually works, not what we think works.

The truth is vitally important. The more we know about our product/channels/users, the easier it is to create things that drive growth.

While this has been labeled “growth process” this approach shouldn't be constrained to a single functional area. It can, and should, be learned by all functional areas of marketing, product, eng, design, sales. The key is matching this process/tool to the right set of problems.

Growth Team: Cross Functional Teams

We have seen a lot of companies try to establish growth teams. The starting point for that is almost always, “What should the structure for the growth team be? Where should it live in the org?” This is approaching the problem backwards.

The first step is defining the problems/areas that have the biggest impact on Growth, and working your way backwards to the team needed to execute effectively. When you do that and combine it with the Growth Process, you quickly realize that to execute effectively you need a cross functional team with a mix of eng, product, data, design, marketing, and sales skills. The mix will depend on the problem area.

When we say cross functional teams, these are NOT teams that have weekly stand up meetings. These are teams that:

  1. Sit together.
  2. Share the same metric/goal.
  3. Are running the same process.
  4. Are rewarded by the same things.

Unfortunately, growth problems are typically thought about from a function first perspective. When you think about it from a function first perspective, the question changes from “What has the biggest impact on growth?” to “What is the most impactful thing I can do given the skill sets available to me in my function?” The most impactful areas either never emerge or are constrained on what can be solved because only one function is represented.

If you are establishing a growth team, we have five tips:

  1. Find One Problem - Find a single problem that could help drive growth. Typically this is a neglected area at the company (see above). Don't try to own all of growth. It is too wide and broad and will spread the team thin.
  2. Evolution, Not A Revolution - Respect the culture and principles that made the company successful so far. Evolve from those principles, don't re-write them.
  3. Expect Failure - Seek quick wins, but expect the team will fail early and often. Give them enough time to work through it.
  4. Communicate The Wins - Use the experiment wins as a carrot to display how you are approaching the problems. Change and resources gravitate towards those that have wins.
  5. Don't Call It A Growth Team - Some have the perception that they don't need to worry about growth since there is a team that owns that. Some feel they aren't getting the credit they deserve because “everyone contributes to growth.” Or one of the many negative biases around the term creep in. What to do instead? Don't call them growth. Name the teams by the problems/missions they are solving for - New User Experience, Lifecycle Team, etc.
What Does The Future Hold?

It isn't clear. Parts of us wish we could kill the term “growth” because it creates more friction and confusion to solving for these important changes. Killing the term is unlikely, but what hopefully happens is:

  • “Growth Product” just becomes a part of “good product.”
  • Understanding/using “Growth Process” is just being a good product/marketer/eng/etc.
  • Cross functional teams are the norm, not silos

Developing a systematic, deterministic, repeatable, and sustainable system towards growth is one step in embracing how the game has changed. But there are some other key pieces:

  • In the third post in this series we'll talk about how the funnel framework was a good starting point, but we need to move beyond it. Funnels do not represent how software companies actually grow. We'll introduce a new framework that better describes how the top companies grow.
  • In the fourth post in this series we'll talk about the new tool/skill every practitioner needs to properly do things like set goals, prioritize, and make investments... the Growth Model.

Subscribe here to receive the rest of the posts in the series. If you or your team are practitioners with 3+ years of experience, check out one of our upcoming Fall Programs:

  • Growth Series - A comprehensive program on how to construct and operate a repeatable, predictable, and sustainable growth machine.  Created and hosted by Brian Balfour (former HubSpot) and Andrew Chen (Andreesen Horowitz and former Uber).
  • Retention + Engagement Deep Dive - A deep dive on how to measure, analyze, and improve retention and engagement.  Created and hosted by Casey Winters (former Pinterest/Grubhub), Shaun Clowes (Metromile and former Atlassian), Brian Balfour (former HubSpot) and Andrew Chen (Andreesen Horowitz and former Uber).
  • Growth Loops + Models Deep Dive - One of the most impactful frontier topics, this program dives deep on establishing compounding, defensible growth loops and how to model them quantitatively.  Created and hosted by Casey Winters (former Pinterest/Grubhub), Brian Balfour (former HubSpot) and Kevin Kwok (former Greylock Ventures). 
Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 
Reforge by Brian Balfour - 7M ago

This is the first post one in a four post series co-written by Brian Balfour, Casey Winters, and Kevin Kwok. Subscribe here to get the rest of the series. To go deeper on this concepts, join us for one of our upcoming fall programs - Growth Series, Deep Dive: Retention + Engagement, or Deep Dive: Growth Loops + Models.

Brian Balfour
Founder/CEO @ Reforge. Former VP Growth @ HubSpot.

Casey Winters
Growth Advisor, Former Pinterest, Grubhub. 

Kevin Kwok
Former Greylock Ventures. 

Growth wins.

Not growth at all costs, bad unit economics, dark patterns, spam your users, shark fin growth. But authentic growth built on a solid foundation of retention and engagement.

This advice isn't new:

“Poor distribution - not product - is the number one cause of failure.” - Peter Theil
“The best product doesn't always win. The one everyone uses wins” - Andrew Bosworth
“Most value creation takes place not at the startup phase, when new companies are formed but at the “scale-up” phase, when a select number of these companies grow at dizzying pace.” - Reid Hoffman

We've gone through three phases as a tech ecosystem. The early phase in the 90's was all about “Can it be built?” The main risk was technology risk. Mid 2000's we transitioned to the second phase, where the main question was “Can you build a great product?” Then the tools and technology got better and cheaper. Tech and product risk decreased, and distribution risk increased.

Let's make this clear. We aren't transitioning into the the third phase of distribution risk. We are in it!

Yet, it still seems we are stuck one phase behind. We've seen countless presentations, blog posts, Q&A's where the answer and guidance to every growth question seems to be “Build great product.”

The ecosystem is littered with deaths of great products because they never got distribution. One of many reasons is Product Channel Fit. These get overlooked though. Andrew Bosworth, VP @ Facebook has probably the best explanation for this:

“Later we all tell ourselves a story about why and how that product was really the best all along in an impressive display of survivorship bias.”

But let's think about it from the reverse direction. There are plenty of products that people consider “terrible products” that are $1 Billion companies. How many users of Salesforce have you met that proclaim they think Salesforce is a “great product?”

We aren't saying you shouldn't build a great product. Our point is that “build a great product” receives far more weight, discipline, and thinking then “build a great growth strategy.”

Why Growth Wins

It is worth digging into why growth wins. There are a lot of reasons why, but here are four:

Distribution = Defensibility = More Distribution

More distribution leads to more defensibility which leads to more distribution. There are two reasons for this.

From the company perspective, the more distribution you have, the more you can leverage it into other products and features to gain more distribution. You muscle competitors out of the market. Google has been doing this for years, but the more recent visceral example is Facebook copying Snapchat's features.

This is also true for B2B. HubSpot over recent years has used their stronghold of mid market customers in the marketing automation space and massive content presence to expand into the crowded but lucrative CRM, Sales Automation, and most recently Customer Support verticals.

We can also look at this from the user perspective.

“Habit and user expectation remains a stronger moat than people appreciate." - Ben Thompson

Habits are extremely hard to break. That goes for diets, a work out routine, and our use of various products. The switching costs of when a habit has been built are consistently underestimated.

One of the most defensible things you can have is a large distribution base that habitually uses your product.

Growth = Resources = More Growth

Between the three of us, we've sat in hundreds (maybe thousands) of company pitches across multiple VC firms. If there is one thing that trumps everything, it is growth. Growth very simply attracts capital and when applied well, creates more growth.

The same dynamic occurs with attracting top talent. The market for top talent has grown increasingly competitive with no end in sight. With more and more software startups to choose from, compensation aside, the best talent typically wants a few things:

  1. Validation that “it's working.”
  2. Opportunity for personal professional growth.
  3. To work with other talented people.

All of these stem from a product and company that is growing. Similar to capital, when applied directly more talent leads to more distribution and the cycle fuels itself.

This isn't just at a company level, it applies internally as well. Projects that are seen as growing or having a direct impact on growth have capital, attention, and people flow to them. If you want to position yourself well internally, find a way to to work on a project that is growing.

Growth = Learnings = More Growth

Teams that learn more about their users, product, and channel and apply those learnings win over the long run. The more users and engagement you have, the more experiments you can run, and the quicker you can get feedback to learn.

Growth = More Growth

The best companies are built on a system of compounding loops. The returns you get from these compounding loops are typically a function of the existing user base. Just like in finance, where you earn more interest dollars the larger the base principle is. We'll talk more about this concept in a future post, and have the Deep Dive: Growth Loops + Models program dedicated to it.

The Game Has ChangedDistribution Has Become More Competitive and Expensive

Distribution across every major channel has become more expensive.

On top of all these, we've seen salaries across all roles in technology companies including sales and marketing roles increase over the past few years which directly flows down to your fully loaded CAC. So what is going on here?

  1. The number of new major channels has decreased as we've seen consolidation.
  2. Those channels have tightened control in an effort to monetize.
  3. The number of companies that are competing in these channels has increased.

The three combined equal more competition and higher costs.

The Lifecycle Of Channels/Tactics Has Accelerated

In the Growth Series we touch on how every channel goes through a lifecycle of new, to golden age, to saturation and how to predict and plan your strategy around it. This lifecycle is evident at the macro level (channels) and the micro level (tactics):

Lifecycle of Channels.  Credit: James Currier

Lifecycle of Tactics

But the bigger point is that due to an increase and competition, the age of a lifecycle has decreased. As a result, every team needs to:

  1. Understand where they are in the lifecycle for their strategy.
  2. Expect more change and adapt more quickly.
  3. Move away from a collection of tactics, to thinking about growth in a more defensible way.
Increase In Accessibility Of Data

Let's rewind 10 years ago. If you wanted to implement a robust data and analytics solution you basically had three options:

  1. Google Analytics
  2. Double Click (and some other big enterprise solutions)
  3. Build Your Own

No matter the solution you were spending tons of money and/or ending up with an ineffective solution. Mixpanel, Segment, Amplitude, Heap, Looker, Chartbeat and many more hadn't been founded yet. Let's fast forward to today:

With the slew of data tools, data has become cheaper and more accessible. More people in the org have data at their fingertips especially those working on growth initiatives. We are at the tip of the iceberg on new methods of personalization, machine learning, and deeper insights via data to drive growth. Yet, most companies are still stuck in the Data Wheel of Death.

The Blurring Lines Of Product/Engineering/Marketing/Sales

We use to have very clean lines between marketing, sales, and product. Customers experienced the product in that segmented way. But this is not how software and technology products spread from the customer's point of view.

Consider what a user might experience in a bottoms-up SaaS product like Slack or Dropbox, or some of the new players like Front, Airtable, Loom and Pipefy.

  1. A user is likely to first experience the product through some invite via collaboration feature.
  2. That user then signs on and experiences the product.
  3. They then go through an activation flow to create something themselves.
  4. They then hit an upsell in the product, schedule a call with a salesperson, and buy some seats.
  5. Then they roll it out to some people within the company.

It isn't just bottom-ups SaaS. It's all software companies. Data, technology, and product play a much larger role in outcomes like acquisition, retention, and new sales. The lines have blurred, but most of orgs are still in silos.

We see this in the Retention Series. A company needs to improve retention, they assign a marketer or two to own the metric, the marketer can only change some emails to drive the metric, so they get frustrated because they can't get product and engineering resources. This is a lose-lose situation for that employee and for the company.

Marketing/sales begging for product/eng resources, product/eng saying marketing/sales don't understand product, divides between product and growth teams. Whatever it is, it stems from the fact that all these lines are blurring and we need a different approach.

Growth Wins + The Game Has Changed = ????

We know that growth wins and we've seen major foundational changes. The combo of these two might feel like gloom and doom. That isn't the message at all. It just means we need to embrace the change and develop new frameworks and tools to approach the discipline. In the next few posts we are going to walk through:

  1. What a system towards growth that embraces these changes looks like. The term “Growth” has been slapped on everything causing mass confusion. We'll wipe the slate clean and talk about a foundational approach.
  2. Funnels are dead. The funnel framework was a good starting point, but we need to move beyond it. Funnels do not represent how software companies actually grow. We'll introduce a new framework.
  3. The new tool/skill every practitioner needs to know to properly do things like set goals, prioritize, and make investments... the Growth Model.

Sign up here to receive the rest of the posts in the series. If you or your team are practitioners with 3+ years of experience, check out one of our upcoming Fall Programs:

  • Growth Series - A comprehensive program on how to construct and operate a repeatable, predictable, and sustainable growth machine.  Created and hosted by Brian Balfour (former HubSpot) and Andrew Chen (Andreesen Horowitz and former Uber).
  • Retention + Engagement Deep Dive - A deep dive on how to measure, analyze, and improve retention and engagement.  Created and hosted by Casey Winters (former Pinterest/Grubhub), Shaun Clowes (Metromile and former Atlassian), Brian Balfour (former HubSpot) and Andrew Chen (Andreesen Horowitz and former Uber).
  • Growth Loops + Models Deep Dive - One of the most impactful frontier topics, this program dives deep on establishing compounding, defensible growth loops and how to model them quantitatively.  Created and hosted by Casey Winters (former Pinterest/Grubhub), Brian Balfour (former HubSpot) and Kevin Kwok (former Greylock Ventures). 
Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Disclaimer: This is not legal advice for your company to use in complying with the GDPR. It simply provides our interpretations of how it will affect people working in growth. This information is not the same as legal advice. We insist you consult an attorney for advice on applying the law to your specific circumstances. You may not rely on this post as legal advice, nor as a recommendation of any particular legal understanding.

With the General Data Protection Regulation (GDPR) coming into effect in just 6 weeks, are the days of growth at any cost coming to an end?

The law is designed to safeguard data privacy for EU citizens, and applies to any business with EU users or customers, regardless of whether the business is based in the European Union or not. The penalty for non-compliance is “up to €20 million, or 4% of the worldwide annual revenue of the prior financial year, whichever is higher.” 

After doing a lot of research and interviewing leaders managing GDPR compliance at top companies, we’ve come to this conclusion:

If you’re in growth, and not preparing for GDPR, you should be.

Currently, many growth teams are violating the General Data Protection Regulation in multiple ways. And though it was approved by the European Parliament in April 2016, they’re unprepared to comply when it comes into effect on May 25, 2018.

Does this apply to your team?

It does if you’re:

  • Tracking user behavior on your website or in your app for marketing and personalization
  • Collecting email addresses and other personally identifiable information (PII) for email marketing
  • Testing strategies to resurrect churned users

If you’re doing any of these things without explicit consent (defined below) from users, then it’s time to make some changes to your growth practices. 

In this post, we’ll talk about what you need to know to comply with GDPR across the 3 common growth activities mentioned above. 

Table of Contents
  1. Two Concepts Every Growth Person Must Understand about GDPR: "consent" and "legitimate interests"
  2. Three Key Growth Activities to Evaluate for GDPR: (1) tracking and analytics (2) email marketing, and (3) resurrecting churned users
  3. Next Steps for Growth Practitioners
  4. Glossary of GDPR Terms Growth People Need to Know
  5. GDPR Reading List

But first, let’s walk through a few of the foundational principles from GDPR that are most important for growth.

Two Concepts Every Growth Person Must Understand about GDPR

For growth and marketing activities, two of the most important principles of GDPR (which replaces the EU’s 1995 Data Protection Directive 95/46/EC) to understand when you’re collecting, processing, and storing people’s personal data are: 

  1. Consent
  2. Legitimate interests

Consent, as described in Article 4.11 of the GDPR is:

  • A “clear affirmative action” taken by the data subject (user)
  • Freely given by the data subject
  • Specific, informed, and unambiguous
  • Documented in detail by the data controller (the company that determines how the data will be processed)
  • Easily withdrawn

The “data subject” is your free or paid user, and their “personal data” is any information that can identify them in any way, such as:

  • Name
  • Email address
  • An identification number
  • Location data
  • IP address
  • An online identifier, such as a cookie

So, what does all of this mean?

It means that to collect, store, and process personal data from your EU users for marketing, in many cases, you need to clearly communicate how you plan to use their data and give them an explicit choice to opt in or opt out. For a user to opt into your marketing, they have to take a “clear affirmative action.” 

The days of relying on pre-ticked checkboxes, silence, or inactivity as implicit consent to track activities and send marketing communications are over. 

Not only that, but you’ll also need to offer users an easy and obvious way to withdraw previously given consent, and maintain documentation of the details of where, how and when you collected consent from your users. If GDPR governing bodies in EU member states or users ever bring a case against your company, you want to ensure the details of consent are documented, available, and defensible.

But consent isn’t the only option for marketers, growth people, and service providers. “Legitimate interests” is another option that may provide an alternative to consent in certain cases. 

Legitimate Interests

According to Article 6.1 of the GDPR, legitimate interests can be used as grounds for collecting and processing users’ personal data when:

  • Data processing is necessary for the “legitimate interests” of the company
  • But not when the fundamental rights of the user override the legitimate interests of the company

For growth and marketing, legitimate interests are one of the most ambiguous concepts in the GDPR, and therefore, very open to interpretation. But understanding legitimate interests comes down to having a clear and defensible rubric for defining three key components:

  1. What “necessary” means when it comes to processing personal data
  2. The “legitimate interests” of the controller (your company) or third party (a data partner)
  3. The interests, fundamental rights, and freedoms of the data subject (your user)

To evaluate legitimate interests as a grounds for personal data collection, define these three components on a case by case basis. Ensure you’re balancing your company’s interests with those of its users, as legitimate interests cannot be used to override users’ interests and fundamental rights.

Growth Lessons from Leaders at the Fastest Growing Companies

Get our newsletter with insights from the field

Email Address *

Thank you for subscribing!

To dig deeper into understanding legitimate interests and when to apply them, the Data Protection Network’s guidance on legitimate interests is a useful resource for service providers. But ultimately, this is something you’ll need to work with your legal and compliance team to interpret and apply for your company’s specific circumstances. 

Now that we’ve established a basic understanding of consent and legitimate interests, let’s walk through the role they’ll play in the three growth activities we mentioned earlier.

3 Key Growth Activities to Evaluate for the General Data Protection Regulation

As a reminder, those three growth activities are: 

  • Tracking user behavior for marketing and personalization
  • Collecting email and other personally identifiable information (PII) for email marketing
  • Resurrecting churned users

We’ll address each one individually below.

Will GDPR Impact Web Tracking and Analytics?

The short answer is, yes. 

But still, many companies have scripts on their sites tracking user behavior without consent. And most tracking tools contain personally identifiable information (PII).

It's no longer acceptable to bury tracking notices in the terms of service saying, ‘If you use our service, we will be tracking your behavior.’ When PII is being stored, you need to ask for explicit consent at the time you want to start collecting customer data.

According to Recital 30 of the GDPR, even online identifiers (including cookies) and location data such as IP address are considered personal data. And any kind of personal data requires consent to be collected, processed and stored.

To learn how other companies are approaching consent for tracking, I spoke with Andrew Michael, Experience Team Lead at Hotjar. Hotjar is an analytics and feedback service offering heatmaps, visitor recordings, and other feedback tools. Andrew currently manages GDPR compliance for marketing. 

He explained,

“Our team is building new consent features to be ready for GDPR. For example, we have tools such as Heatmaps and Recordings that allow us to gather data about user behaviour on a website—but we introduced features that allow our users to suppress any PII from them. This means that Heatmaps and Recordings can still be used to understand visitor behavior, but there is no need to request consent because individual visitors are not being tracked. 

The only time when we do need consent is when we want to link recordings to a feedback tool: and for this, we have created a clear consent request when a user provides feedback through one of the tools.”

Data analytics vendors (data processors), like Hotjar and many others, are leading the charge to become GDPR compliant, as the stakes are especially high for them.

This doesn’t mean that you’re off the hook, though. It’s still your responsibility as the data controller to:

  • Collect only necessary data
  • Offer a clearly communicated process that allows users to opt-in or opt-out and easily access and control their data at any time

What growth teams need to focus on now is not whether they will ask for cookie permissions and other forms of tracking consent, but how they will ask. 

If you’re collecting any personally identifiable data via cookies or scripts from analytics vendors like Hotjar, Mixpanel, Google Analytics and others, you will need to show a consent box when new users visit your site. 

No more soft opt-ins allowed, which means those cookie pop-ups that say something like “By using this site, you accept cookies,” or bundling cookie consents into your terms, will no longer cut it. 

A non-GDPR compliant cookie notice from Facebook

What GDPR Compliant Tracking Consent Looks Like

There weren’t many GDPR compliant examples out in the wild yet (I bet that will change after May 25th), but below are a few examples I was able to dig up.

Mock Ups of Tracking Consent Forms

Tracking consent form from PageFair

Consent form from PageFair to share browsing data with a 3rd party

A Cookie Consent Form in the Wild

A cookie consent form live on the Cookiebot website

An expanded view of Cookiebot’s cookie consent form

Consent requirements like these will also have an impact downstream on our ability to personalize experiences for our users.

So, What Does GDPR Mean for Personalization?

When GDPR comes into effect, personalizing content without consent will be a no-no.

According to Aurélie Pols, a former Data Governance and Privacy Engineer at Salesforce,

“It’s no longer an option to personalize content by default. Permission to do so should be retractable and based on consent terms in line with the GDPR. Opting in for content personalization will become part of the standard user options you would expect from a data controller.”

According to this interpretation, companies that do user profiling, as described in Article 4.4 of the GDPR, will be required to ask for consent. This will impact the “big fish” most acutely - the ones who’ve pioneered and scaled personalization for growth - think Netflix, Amazon, Spotify, and Pinterest. Whether they actually do ask for consent, and how the Supervisory Authorities enforce the new privacy laws, remains to be seen.

So, what do we do if consumers don’t opt in?

We’ll be forced to get more creative with finding insights in anonymized or pseudonymized data. (For a deep dive into the differences between anonymization and pseudonymization and how they relate to GDPR read this primer.)

Andrew Michael explains how Hotjar is applying anonymization:

“We have planned 3 levels of suppression to make sure customers can use Hotjar in a GDPR-compliant manner:

First, we have an automatic suppression layer, that suppresses any form field, plus any number on a page greater than 9 digits (which may resemble a credit card or phone number), on any site that has Hotjar’s script installed. The data is suppressed on the end user’s side, so it never hits our servers.

Second, we give customers the option to activate automatic on-page suppression of any number or email address found on any page of their website or app. 

Third, customers can tag specific elements they want to suppress, to make sure they aren’t sending personally identifiable information to our servers without consent. 

With this, it is impossible to link identifiable information back to an individual user’s session recording and heatmap data. Hotjar customers can still show surveys and polls to solicit feedback from visitors to their website or app, but they won’t be able to leverage the data for automation or other marketing activities—unless explicit and clear consent has been given through our feedback tools.

The key to growth is delivering consistent and constant value to your users. You can still get smart with the way you use data to personalize approaches and fuel your growth experiments backlog, but you don’t need to know who a specific individual is and what they are doing. To get actionable insights, it is enough to look at how people use your website or app as a whole.”

In addition to tracking, analytics and personalization, the new rules of the GDPR will change how we approach email acquisition for marketing.

Growth Lessons from Leaders at the Fastest Growing Companies

Get our newsletter with insights from the field

Email Address *

Thank you for subscribing!

Where Email Marketing Goes Wrong With GDPR

When it comes to collecting and using email for marketing, again, consent is going to be the key to GDPR compliance. But many email marketers aren’t asking users for consent prior to collecting their email and other personal identifiable information and sensitive data. 

Below is a list of common email marketing activities teams engage in that don’t work with GDPR:

  • Automatically subscribing new users who create an account to marketing communications
  • Acquiring leads with gated content and sending them nurture email drip campaigns
  • Using pre-ticked boxes to automatically subscribe people to marketing emails
  • Automatically subscribing referrals to a marketing email list
  • Bundling consent to email marketing in with other terms and conditions
  • Gaining consent for a specific type of emailing marketing and sending a different type

Do any of these sound a little too close to home? If so...

Here’s What To Do Instead

There are three key things email marketers can do to comply with GDPR come May:

  1. Incorporate “Privacy By Design” into your email acquisition program
  2. Audit your existing subscribers for consent
  3. Build an email preference center
  4. Revamp your referral program

Let’s walk through each individually.

1. Incorporate “Privacy By Design” into Your Email Acquisition Program

Taking a user centric approach to email marketing is the best way to build trust with users, enhance data security, and survive in a post GDPR world. To do this bring privacy and consent forward into the strategy and design phase of your email acquisition program.

Let’s walk through a few examples of different email capture forms to see what Privacy by Design is, and what it isn’t, in the world of email acquisition.

Example: Sky

This email marketing subscribe example from Sky is the antithesis of Privacy by Design.

A non-compliant email capture form from Sky

Note that the:

  • First tick box prompts the user to click to agree
  • Second tick box switches the logic, asking the user to..
Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Editor’s Note: We surveyed growth practitioners from top companies like Nest, SurveyMonkey, Affirm, NerdWallet and others to find out what they’re reading to stay current with the latest in growth and tech. You’ll find their list of favorite content sources at the bottom of the post. What do YOU like to read -- that we might be missing? Tweet @reforge to let us know!

While content is getting better, its reach is decreasing, CAC is increasing (faster than CAC for paid marketing), and competition is skyrocketing. ProfitWell recently published a study revealing some grim statistics that illustrate these points - compared to 5 years ago, 300% more content is being published per month, posts are almost 100% longer in word count, and shares per post have dropped 90%(!) in the last 2 years.

The textbook symptoms of channel saturation have set in. Here at Reforge, we realized if we didn’t adapt, we’d risk getting left behind.

But how do you adapt when channel saturation takes over?

We started answering this question (it’s a work in progress) by going back to basics and talking to our target audience to drive insights. Our goal was to understand how our audience chooses which content to consume, and which to tune out, and to make sure our content doesn’t end up in the “tune out” pile. In this post, I’ll share what I learned from our research.

We are a small team that prioritizes moving quickly so the research process was both simple and scrappy. I conducted in-depth user interviews with seven people from our target audience (growth practitioners, product managers, marketers, and data scientists), posted questions to our discussion forum for Reforge growth program participants and alums, and conducted a 2 question survey.

I’ve distilled the learnings that came out of this research effort down into 3 themes:

  1. The format is as important as the topic
  2. Barriers to entry and usability drive content engagement
  3. Dark social has replaced broadcast sharing

Let’s walk through each in a bit more detail below.

Growth Lessons from Leaders at the Fastest Growing Companies

Get our newsletter with insights from the field

Email Address *

Thank you for subscribing!

1. The format is as important as the topic

Scanning content for relevance stood out as one of the critical “coping mechanisms” people have developed to deal with the onslaught of high quality content.

This means the title, subheadings, and format of your piece are the gatekeepers to your audience’s attention. Use them to expose the value of the piece without giving away too much.

Your title and subheadings must invite curiosity so your readers want to dig in, and the format must make the piece easily scannable for the high level takeaways. Be thoughtful about how you use spacing, bullets, numbered lists, the bolding of key concepts, and images to construct a compelling narrative.

From my interviews, I learned that people ask themselves 3 questions as they scan to decide whether to dive into a piece of educational content:

  1. Is this highly relevant to the problem I’m trying to solve, or topic I want to learn about?

  2. Is it actionable?

Only once the content crosses the threshold of being relevant and actionable, do they ask:

  1. How long will it take me to consume this?

  2. Do I have time for it now? Or do I need to bookmark it for later?

Keep in mind that the process someone will go through to assess your content for relevance will likely differ if your content is created to entertain, rather than educate.

2. Barriers to Entry and Usability Drive Content Engagement

A curious pattern emerged during my research - podcasts and books were often the first thing people mentioned when I asked them to share their favorite content on any topic (not just growth). In a world trending towards instant gratification, I expected people to list content that delivers easily digestible summaries of complex topics - think theSkimm.

I was surprised to hear that people are actually hungry to dig in - as long as the topic is relevant. One person said, “I like books because they have enough space to dig into paradigms.”

So, what put books and podcasts at the top of the list for our particular audience?

There are two answers to this question:

  1. High barriers to entry drive content quality

  2. Usability isn’t just for product, it matters for content too

High Barriers to Entry Drive Content Quality

Anyone with an idea, access to a CMS, and social media can “pump out” and distribute a long form blog post in 10-20 hours. If people like it, it will likely even get some free distribution via sharing.

Conversely, book ideas must first be pitched to and picked up by a book publisher. Then, it takes 1-3 years and tens (or hundreds) of thousands of dollars to produce and distribute the book. Similarly, if you’ve ever tried to produce and distribute a quality podcast, you know it is no small investment of resources.

We all know that the ROI of content marketing is often harder to track than that of other forms of acquisition, like paid marketing. This is especially true for books and podcasts, which generally drive brand and awareness, rather than lead generation.

These time, cost, and ROI burdens force anyone thinking about publishing a book or podcast to be more discerning, and serve as built in quality enforcers.

Usability Isn’t Just for Product, It Matters for Content Too

Audio content - predominantly podcasts and audiobooks -  boast a key advantage that no other type of content can compete with - it can be easily consumed while doing other things, like commuting, housework, exercising and more. Because it indexes high on both usability and depth of quality, when people listen to learning focused audio content, they listen a lot - the members of our small sample who consume content via audio tend to listen 4-5 hours per week on average!

The hands free, screen free nature of this favored content format, coupled with the insight on scannability above, just go to show that UX can be make or break, not only in the product world, but in the content world as well.

3. Dark social has replaced broadcast sharing

During the interviews, I heard various versions of this statement from multiple people:

“I used to share content on social but I almost never do anymore. I don’t want to add to the noise. But I do still share. If I’ve found a great piece of content, I’ll share it, but I have to know it’s highly relevant to the people I’m sharing it with.”

This is where dark social comes into the picture. Dark social is targeted one-to-one content  sharing via various messaging apps - text, Slack, Whatsapp, and Messenger, for example. It is challenging to track and, therefore, ends up getting labeled as Direct traffic in Google Analytics.

Dark social is the antidote to the noise caused by broadcast sharing and accounts for 80+% of outbound sharing.

Image from The Atlantic, representing traffic sources to TheAtlantic.com

For insights on how to measure dark traffic and what it implies about virality, read Susan Su’s Thoughts on Growth newsletter.

Growth is Nothing Without Qualitative Data

Recently, when I was listening to a Q&A session with Brian Balfour (CEO @ Reforge) and Shaun Clowes (VP of Product @ Metromile and former Head of Growth @ Atlassian) at one of our Retention + Engagement Series events, Shaun said something that hit a nerve for me:

“Growth is nothing without qualitative data.”

Too often I’ve heard people talk the talk of doing user research, but when put under the pressure of aggressive OKRs and tight deadlines, they fail to walk the walk. They’re in such a rush to get to market or run those growth experiments, they they skip the foundational step of talking to their customers.

Don’t fall into that trap.

Each audience is different, each product is different, and each company is different, so don’t automatically assume that the insights pulled from our Reforge audience will apply to your audience. If you take nothing else away from this post, I hope it inspires you to gather the qualitative and quantitative data you need to adapt to the every shifting content marketing landscape.

To close, I’ll leave you with this little treat. Here’s one of the of the lynchpin questions I asked each person I interviewed - “What are your favorite sources for content?”

Below is the recommended reading list that resulted.

Growth Lessons from Leaders at the Fastest Growing Companies

Get our newsletter with insights from the field

Email Address *

Thank you for subscribing!

The Reading List:

What do YOU like to read -- that we might be missing? Tweet @reforge to let us know!

Growth and Tech



Email newsletters


 Other Topics:

Email newsletters




Growth Lessons from Leaders at the Fastest Growing Companies

Get our newsletter with insights from the field

Email Address *

Thank you for subscribing!

About the Author

Lauren Bass is a growth marketer at Reforge, a company that provides masterclasses in frontier skill sets for mid-career tech professionals. Previously, she worked on growth marketing at NerdWallet and prior to that she was the Founder and CEO of LolaBee's Harvest, an online farmers market acquired by Good Eggs in 2013

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Editor’s Note: This post in the first in a series where the VP of Growth at HubSpot, and other members of his team, publish the frameworks they used to launch and scale the company’s first freemium products. The series goes in depth on how they experimented their way into freemium growth, instilled a growth culture, and evolved the org structure to help HubSpot grow into a product-first company. Subscribe to get the rest of the series.

Kieran Flanagan is the VP of Growth at HubSpot, a marketing and sales software company. He is responsible for the growth of HubSpot's freemium products, monetization of their freemium funnels and optimization of their global web strategy.

The future of growth belongs to product driven companies. At HubSpot, we realized this a few years ago, which is why we disrupted our own business model before anyone else could. 

At the time, HubSpot was still growing 30%-40% per year on the shoulders of our original marketing and sales driven inbound marketing model. Despite the success, we consciously chose to upend what had been working by launching our first freemium products in 2014.

Market dynamics and consumer behavior have been changing - increasingly consumers expect to use software and extract value from it before buying. To stay relevant over the long term we needed to adapt, or risk “getting our lunch eaten.”

We entered the world of freemium in 2014 with Sidekick, a sales automation product, and HubSpot CRM. In 2016, we rebranded Sidekick as HubSpot sales and deepened our commitment to becoming a product-first company, launching Customer Hub, a freemium product for customer success, in 2017. 

Product is the Future of Growth

Over the past 24 months, I’ve been dedicated to building out product driven growth at HubSpot - acquiring users into our freemium products and working with product and engineering to upgrade them to paying users. 

What do I mean when I say product driven growth?

I’m talking about using in-product levers to grow, in place of or in conjunction with external marketing and sales channels. When people can try your product for free, they experience the value of your product before making the decision to pay. This turns more people into happy users, creating more opportunity for them to tell their friends, who in turn tell their friends. This can trigger virality and widen the top of your funnel.

In a new reality where Google and Facebook are the only two platforms that offer opportunities for user acquisition at scale, product driven growth allows you to decrease customer acquisition cost by reducing dependence on paid marketing, and sales for B2B products.

Because it’s scalable and cheaper, product driven growth is how the biggest products have grown so large so quickly. It’s also how new products will win in the future.

How HubSpot Experimented Its Way Into Freemium Growth

The first step in adding freemium to our go-to market strategy was setting the overarching vision of where we wanted to go. Then, our goal was to run experiments to iterate towards the vision or inform how we needed to evolve the vision. 

We set our sights on providing companies from big to small with the right tools to grow. We wanted customers to be able to get started with our marketing, sales, and customer success products for free, and upgrade to different packages as their needs grew. Navigating the associated shift to product driven growth (while still growing 30-40% a year!), hasn’t been easy. But it has brought valuable learnings, which I’ll share with you in this post. 

I’ll walk you through the process our growth team used to experiment our way into higher and higher impact growth opportunities for HubSpot’s freemium products. I’ll detail the initiatives that drove step-change improvements to our funnel, rather than just small percentage gains, and the principles we used to arrive at those initiatives. 

Here’s the high level process that worked for our growth team:

  • Get wins on the board to build trust with leadership and other teams, such as product and engineering.
  • Prioritize growth experiments you can execute quickly to demonstrate results. 
  • Once you start to see a high-level of test failures or non-results, move on to tackle more complex growth opportunities (take big swings).
  • Eventually, tell your CEO you want to test pricing ;-) (take even bigger swings)

If you already work in growth, this process of getting quick wins and laddering up should be familiar. Where I’ll add value, is through transparently sharing how the growth team at a public company like HubSpot actually executed this process, applying it to build a freemium businesses, and the learnings and results that came out of it all.

Growth Lessons from Leaders at the Fastest Growing Companies

Get our newsletter with insights from the field

Email Address *

Thank you for subscribing!

Building a Product Qualified Lead Playbook to Get Early Wins

Not all tests are created equal in terms of the time and resources needed for execution. To start, we categorized all our experiment ideas as easy, medium, or hard. Then, we prioritized and executed on the easier experiments to get ourselves our first quick wins. 

Once we added freemium as part of our go-to market, product qualified leads (PQLs) - leads from free users who show interest in paid features - became a key source for both touchless sales and leads for the sales team. 

Initially, we noticed two important things about our PQLs:

  1. Tests to optimize capturing them generally fell in the easy or medium category for resources and time required.
  2. They converted 3-4x higher than our marketing qualified leads.

This was great news - PQL optimization was a high impact, lower effort area for experimentation. So we doubled down here and built out a playbook for optimizing PQLs. 

Our first objective was to identify which PQLs offered the biggest growth opportunities. To help us figure this out, we segmented our PQLs into 3 buckets:

  • Hand Raise PQLs: We would show free users call to actions within the product for paid only features or the opportunity to get assistance with a particular task, those users would interact with the CTA to reach out to us. We used them sparingly.
  • Usage PQLs: We triggered a call to action based on product usage, for example using all of your free call minutes or email templates would trigger an option to upgrade or talk with sales.
  • Upgrade PQLs: These were features only available to paid users, they would send users to an upgrade page.

Then we built a dashboard to show the funnel for each PQL (thanks to Scott Tousley & Sam Awezec). We needed to get the correct data infrastructure in place to track each, otherwise we wouldn’t have known which growth experiments were driving results.

Snapshot of our PQL Dashboard (with dummy data)

The PQL dashboard showed us the performance of each PQL point within our freemium products, giving us how many times people interacted with them, how many upgrades they generated and the average selling price (ASP) for each one. 

We could use the dashboard to look at our PQL data in many different ways. For example, we could see by bucketing the PQL's into different types (hand raise, usage, upgrade), the usage PQLs were our best performing category. This aligned with our fundamental hypothesis for why product-driven companies do so well - when users can use your product and get value out of it before being asked to pay, they're more likely to convert when you do show a paywall. 

We also used the dashboard to find opportunities to improve the conversion rate of hand raise PQLs, or areas of the product where we could test adding new ones.

Given we scored all tests on both their potential and ease of implementation, we decided to focus our initial efforts on hand raises. They were easy for us to iterate on, had good potential and had the added benefit of showing us points within the product where free users were highly engaged.

Below is an example of the type of experiments we ran for hand raise PQLs. From our analysis, we knew that once people import their data into our CRM, the probability of them both upgrading and retaining is high, given it's easier for them to see how valuable our product is. But we could also see that people who indicated their previous CRM was a spreadsheet struggled with this step. Our hypothesis was by adding in some human assistance, we could increase both the activation and upgrade rate of this cohort of users, and also gather information on how we could make it easier for people to import their data within the product.

A hand raise PQL experiment for users who previously used a spreadsheet as their CRM

Not only did this PQL point become one of our most popular upgrade points, but it also gave us information on how we could improve our onboarding around key actions, and started to make us think on how we could provide more support for free users within the product.

Today our PQL model has evolved towards usage PQLs as they continued to convert at higher rates. You can start using most of our features for free, within certain limits (e.g., sending a certain number of email templates, or booking meetings on your calendar, or using a certain amount of call minutes). We have few gated features, and hand raises have transitioned to live chat, which I'll speak to later in the post.

After 12 months of building out and optimizing the PQL playbook, we started to see diminishing returns on our PQL optimization efforts. But, having experimented our way into so many measurable funnel improvements, our growth team had earned the trust and support of the leadership and product teams. This gave us the opportunity to take on some big bet experiments that produced substantial payouts.

What You Need to Know About Taking Big Bets

Since big bets require a lot more resources and take longer to execute, drawing on your political capital and credibility in the process, it’s important to make the right bet. But how?

We identified two key ways to hone in on the right big bets:

  1. Speak with other growth teams experimenting with the same types of things you’re considering so you don’t have to reinvent the wheel - gather learnings to help you produce results faster.
  2. Do the math to make sure that, if successful, the big bet will significantly increase a key metric.

Once we had picked the low hanging fruit by optimizing most of the PQL points, we moved onto the bigger fruit higher up the tree. That bigger fruit could be found in optimizing the PQL funnel, which ended up requiring more resources to test, but also bringing big long term improvements.

Knowing where to make your big bets isn’t easy, but one of the most important things our growth team did was talk to other companies that already had successful freemium models. One of those companies was Dropbox. From talking to their growth team, we learned that they were seeing good results from using live chat to support upgrade opportunities in the product.

We realized that a similar strategy could be very helpful during our onboarding. Because our products had such a broad set of use cases, it wasn’t easy to identify user intent. It also wasn’t easy to create the optimal onboarding experiences to help users solve the problem they came to solve. We thought live chat, though resource intensive, could help us identify intent. We also saw it as an opportunity to give users a nudge in the right direction so they would experience the value they came for as fast as possible. 

But before jumping in, we did the math to make sure the potential outcome justified the investment. We determined the baseline conversion rate, did some sensitivity analysis on the impact of potential conversion improvements from adding live chat, and then projected the impact on revenue. 

When you’re analyzing the potential impact of a big bet in your own product, don’t just ask what the upside is. Make sure the experiment will impact a meaningful metric, not a vanity metric. In this instance, our analysis indicated live chat could create a significant increase in revenue, so we went forward with testing in an iterative way.

Growth Lessons from Leaders at the Fastest Growing Companies

Get our newsletter with insights from the field

Email Address *

Thank you for subscribing!

Increasing Conversions by 50%

In HubSpot Sales, when a user triggered a PQL, they would receive a modal that invited them to upgrade or talk to sales. Clicking on “Talk to sales” brought them to a form where they could submit their details for someone to call them.

To start, we tested if our conversion rate would improve by giving the user more options to interact with us.

The resources to run this experiment were considerably higher than optimizing PQL points. Although we didn't code Live Chat into the app, we did take users to a web page where they could start chatting with a support agent about the product. That meant we needed someone from the services team to be our Live Chat resource for this test.

We also allowed the user to schedule a meeting instead of waiting for someone to call them. If they filled out their email address to schedule a meeting they would receive a kickback email from one of our reps, along with a link to the rep’s calendar (via our Meetings app), that allowed the user to book time.

In this experiment, 'Call Now,' just provided our phone number. If this was successful, we planned to integrate this CTA with our "Calling" feature so users could reach a sales rep immediately.

The experiment proved successful. People wanted options, and we increased our overall conversions by 50%. 

So we doubled down. We optimized the experience further by allowing users to book time with a rep within the app instead of getting a kickback email from a sales rep with a link to their calendar, increasing the conversion rate by a further 20%.

This resulted in significant gains when rolled out across all PQL points.

The Double Impact of Live Chat in Onboarding

After launching our our CRM and sales acceleration tools, we learned that a large percentage of the people signing up for both had no prior experience using similar software.

And because our products had a broad set of use cases, it was easy for users to become overwhelmed by choice during onboarding. Attempting to alleviate the onboarding confusion, we ran multiple experiments highlighting different use cases to find out what the majority of users wanted to do.

But given the variety of use cases available, deciding on an onboarding sequence to activate users into weekly active teams (one of our key metrics) was complicated.

From our earlier experiments, we had seen positive signals that users wanted to engage via live chat. So, at one of our monthly performance meetings I showed this chart:

The chart was a poorly drawn out version of how live chat would facilitate onboarding. Users who signed up for the product and encountered friction during onboarding could reach out to a user success coach via live chat. The coach could then help the user navigate the friction points to become activated, while also providing user feedback to the product team. My goal with this chart was to demonstrate how adding live chat to onboarding could have a double impact to get buy in from leadership and the product team that a larger experiment was worth the investment. 

We pulled data to identify which usage actions correlated strongly with active teams and upgrades. The data showed that the highest leverage usage actions for new users were sending the first email template, importing their data, or getting the first meeting booked on their calendar.

We would empower the success coaches with this information. This would allow us to focus the live chat experience on the specific usage actions that demonstrated the value of our products to users and drove growth for the business. At the same time, the success coaches would be able to collect data on common friction points and relay it back to the product team, creating a feedback loop for improving onboarding. 

Over time we would want our chart for onboarding look like the image below - the more feedback we get from coaches, the better our touchless onboarding gets and the less we rely on live chat.

Naturally, we wanted to run a minimum viable test (MVT) to make sure we were on track before jumping in, right? (MVT's are the lean startup way to run growth experiments.)

The Problem with MVTs

The problem with MVT's is that you don’t know just how 'minimal' the test can be and still produce accurate results. To get some data back fast, we hacked together what we thought reached the minimum bar for an MVT. It turns out it was a sub-par experience for the user and didn’t give us clear..

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Most CEOs reach the chief executive office an average of 24 years after starting their first job. But, there’s one sub-segment of CEOs who get there dramatically faster. They’re called “CEO Sprinters” according to the CEO Genome Project, a 10 year study that looked at over 17K chief executives. 

“We discovered a striking finding: Sprinters don’t accelerate to the top by acquiring the perfect pedigree. They do it by making bold career moves over the course of their career that catapult them to the top.

In the tech world, we read about these “CEO Sprinters” all the time. They choose the right high-growth startup at the right time, land the highest profile projects, get an impressive assortment of quick wins under their belt, and climb the career ladder at lightspeed, leapfrogging their peers along the way. Think Sheryl Sandberg and Marissa Mayer, to name a few.

We hear about what they’ve accomplished (at such a young age no less!) and naturally want to know - what sets them apart? What do they do differently from everyone else? Do they work harder? Smarter? Faster? 

And... what about me? Do I have what it takes? Could I have a shot at the C-Suite?

After reading about, studying, and talking with many of these career phenoms, I’ve found one dominant thread that sets them apart - each and every one operates by their own set of principles and values that guide their decision making. 

In my quest to uncover the unique principles of as many of these trailblazers as possible, I recently spoke with Nate Moch, Zillow’s VP of Product and Growth, and an active member of the Reforge Collective of growth leaders. 

From our conversations, I learned that his career started like many people in the startup world. Directly out of college, he took a job at Microsoft as a software engineer. After 2.5 foundational years at Microsoft, he plunged into the deep end of the startup pool, joining Zillow as an individual contributor and engineer working on site stress and performance. 

It was at Zillow that Nate really started to mobilize his career. He climbed his way up from engineer, to program manager, to launching and scaling the company’s industry-changing mortgage product, to VP of Product in just 10 years. And in the process, he helped shephard the company from pre-launch, to product-market fit, to growth, to IPO, and beyond, now leading growth at scale.

I was dying to know - How did he accomplish all of that it in just 10 years?!

To answer this question, we delved into his career story - and it ended up being a crash course in how to get ahead.

Growth lessons from today's practitioners

Get our newsletter with insights from the field

Email Address *

Thank you for subscribing!

3 Principles of Career Acceleration

Though Nate hasn’t reached CEO (yet), his choices and resulting career trajectory offer a compelling proof point supporting the findings on CEO Sprinters. They also offer inspiring insights for the rest of us strategizing every career move, rolling up our sleeves, and honing our respective crafts in the hopes of reaching our career aspirations sooner than later. 

Throughout my conversations with Nate, one overarching message resounded:

Don’t be afraid to jump into new and unfamiliar territory - a new function, a new vertical, or a new industry. Don’t wait until you feel 100% ready, make the move when the opportunity arises, (especially if it’s high impact and high profile!) and know you can level up on the job.

This all sounds great, but few people are like Nate and actually do these things, so I had some questions for him:

  • How do we overcome the fear of jumping before we’re “ready”? 
  • How do we find our confidence? 
  • How do we level up on the job? 

And Nate had some smart and actionable answers. 

Over the course of his career, Nate built his success on the back of 3 cornerstone principles that can increase anyone’s probability of pulling off bold career moves:

  1. Assemble a network of experts inside and outside your company to help you level up where you lack experience and skill.
  2. Raise your hand for high impact opportunities, even if you aren’t 100% confident.
  3. Build strong teams by hiring and developing people who will enable you to scale your impact.

In this post, we’ll walk through each principle in detail, revealing how you can use them to blast open the doors of accelerated career growth. But first a story...

A Rough Start at Microsoft

A few days into his first job out of college, Nate had a painful realization - he was “woefully unprepared” for his new job as an engineer at Microsoft. 

He had graduated with a degree in computer science from a small liberal arts school in Indiana. As a liberal arts graduate, he had received significantly less hands on technical experience than his peers who graduated from highly technical universities. During college he had fewer technical CS projects and internships, which meant he wasn’t as prepared as the other recent grads in his starting class at Microsoft.

He explains,

“I felt completely underwater from a technical perspective. I talked to my manager about it and he confirmed: ‘Yes, you were below our normal technical bar, but we decided to take a risk and hire you based on potential.’”

Stop for a moment - imagine the emotional charge of this conversation. You already feel unprepared, and then your manager tells you straight up that you are even less qualified than all of your other new graduate peers. You know you have to overcome a skill deficit to prove yourself - but how?! 

This kind of pressure would derail many people, especially those just out of college - but it doesn’t have to. And as stressful as this experience sounds, it ended up becoming invaluable in helping Nate find his formula for rapid career advancement. It was this struggle with being inexperienced and underprepared that forced him to learn one of the most important lessons of his career - how to transition from novice to skilled practitioner in just 12 months.

Since his rough start at Microsoft, he’s leveraged this lesson countless times to ladder up from:

  • Struggling novice to confident engineer in his first 12 months at Microsoft
  • Real estate neophyte (he’d never bought a home!) to industry change maker
  • Site Performance Engineer to VP of Product and Growth at a public company in just 10 years
  • Individual contributor to leader of 10+ growth and data teams

Having to play “catch up” in his first job forced him to get really good at leveling up quickly. It also empowered him to to make bold, leapfrogging moves over and over throughout his career. 

Befriend the Experts

You might be wondering, “How did Nate go from struggling novice to confident engineer in just 12 months? That’s a fast turn around...”

And you’d be right - it is a fast turn around!

The secret of how he leveled up so quickly lies in the depth and breadth of the network of experts he built - and the questions he asked those experts. 

When he started at Microsoft and realized he was at the back of the pack, he proactively identified, reached out to, and built lasting relationships with people across the company who were experienced in the areas where he was weak. 

He did the same thing when he was making the transition from engineering to product management at Zillow, finding a handful of PM mentors across the company to study and learn from. He identified where they were strong, and tried to mimic the behaviors that led to their success. 

One of his PM mentors was amazing at driving consensus, running meetings, and pushing projects forward. To leverage their expertise, he observed how they ran meetings, reached out for ideas and feedback when he got stuck on a decision, or ran decks by them to make sure he was going to get the results he expected from a meeting. 

Another PM was amazing at technical details and writing specs. He would go to them for spec reviews and questions about technical features. 

Everyone has different strengths and weaknesses, even people in the same position. The key is to identify each person's strengths and what you might be able to learn from them.

He described his approach as follows:

  1. Identify the experts in the topics areas you care about.
  2. Build a cross section of these experts from different teams across the org.
  3. Figure out where each expert excels, so you know who to reach out to, depending on what you’re trying to learn or solve for.
  4. Tap into this network of experts by observing and asking questions - lots and lots of questions.
  5. Be a sponge - absorb as much info, knowledge, and insight from your network of experts as you can. 

He adds a caveat,

“You can ask as many questions as you want, but make sure you never ask anyone the same question twice.”

This approach allowed him to build depth into his network so he could get all of his questions answered without drowning one person.

The Smartest People Ask the Most Questions - Yes Even “Dumb” Ones Sometimes

This leads us to another one of Nate’s insights - building a network of experts isn’t enough.

You must also be fiercely inquisitive, and unafraid to ask “dumb” questions. Nate shared a story recounting one of his first meetings at Zillow. During that meeting, a coworker threw out a term Nate had never heard. He was hesitant to ask what it meant for fear of revealing his ignorance. 

But then, to his surprise, Zillow’s cofounder asked the very question he was too afraid to ask. 

In that moment, he realized something important - the smartest people in the room are the ones asking the lionshare of the questions. They’re too focused on learning to fear what others think of them. They know asking questions (especially of the experts) is the single fastest way to learn, level up, and become experts themselves. 

Nate explains,

“People who are successful seek out people who know more than they do. They find the experts and learn as much as possible from them. When I started the growth team at Zillow, I didn’t know that much about growth. So, I figured out who was doing growth really well at other fast growing companies, I connected with them, bought them coffee, and asked them question after question.”

How to Level Up Without the Safety Net of a Large Company

He used these skills first developed at Microsoft for building and tapping into a network of experts to level up, all over again when he joined Zillow. 

He explains, 

“At Microsoft I had been a software development engineer on a huge team surrounded by experts in similar roles who could help me. When I joined Zillow, my manager asked me to build a stress and performance team, and I thought... ‘Sure, I can figure that out.’ 

This time around, leveling up a had a different twist though. At Zillow, I was hired to be the “expert,” even though I’d never worked on anything related to stress and performance. I had to figure out how to do this job I’d never done before, but without the safety net of a huge company. That’s the beauty of joining a startup, you HAVE to learn a ton of new things on your own.”

He used 3 key tactics to teach himself the ropes of a totally new category of work without the support network of a huge company like Microsoft, and suggests the following for those trying to learn something new at a startup:

  1. Build a patchwork of relationships across the company with people who’ve previously done what you’re trying to learn.
  2. Find thought leaders and experts outside the company, reach out, and build relationships.
  3. Turn to self learning with resources like books, articles, Google, and YouTube to teach yourself.

Build a Patchwork of Relationships Across the Company

Right off the bat, Nate started investing in relationships with Zillow’s Director of Operations, a few network and operations engineers, one of the architect engineers, the Database Administrators, the CTO, and the VP of Development. None of these team members were actively goaled on performance, or responsible for helping Nate, but they took the time to help him nonetheless. Each helped him level up in their own way since they had some experience in various pieces of performance tuning and testing for high scale web services. 

Nate worked most closely with the Director of Operations, even though he was in a different org, because he had deep technical knowledge and was very interested in performance. But he also relied on other “teachers” across the company, working closely with one of the engineering architects to test his system and to help him understand the topology of the entire system. It was Nate’s focus on identifying and patching together the right network of mentors with the experience he needed and the willingness to advise, that helped Nate learn so quickly.

Find Thought Leaders and Experts Outside the Company

He didn’t just rely on his teachers within the company to help him learn, he also looked outside for experts to help him get up to speed. One of those experts was Steve Souders, a performance authority and thought leader whose blog he followed. 

Nate didn’t just passively consume Steve’s work though. He also emailed him to tell him he loved his new book, share how he was using his advice and tactics to improve performance, and ask him to speak on performance at Zillow. Steve even used a quote from Nate in his book! 

The myriad of ways in which Nate invested in building his relationship with Steve and learning from him, is just one example of how Nate sought out the experts outside his immediate sphere and invested in building mutually beneficial relationships with them.

Teach Yourself

Despite all of his efforts at building internal and external networks of experts, there were still gaps in his knowledge that Nate needed to fill through rigorous research, reading, and self learning. Nate told a story to illustrate this point,

“Not long after I started, our CTO came to my desk and said, "Your stress lab is here." He took me into the hallway and pointed to my new “stress lab” - which was actually just a row of equipment in cardboard boxes. Since Zillow was small at the time, I had to figure out how to setup and configure the test lab on my own. Then, not long after I started, the build and deployment engineer left. So I ended up learning and running the build and deployment system as well, until I hired someone else to own it. 

I figured out how to do all of these new things by googling, reading and trial and error. I spent a lot of time doing research to figure out which frameworks to use and teaching himself new programming languages to help me take on new challenges like these.”

While Nate’s first experience at Microsoft showed him that he could move from novice to skilled practitioner very quickly within the support system of a big company, his first experience at Zillow showed him that he could rely on himself to learn new things just as quickly at a small startup, without the same depth of resources. 

It Wasn’t All Inspiring Stories and Quick Career Wins Though

In his early days at Zillow, as much as he invested in teaching himself the ins and outs of site stress and performance management, he still ran into some big “failures” (and a lot of luck) along the way - one of which he describes in the anecdote below.

“My job was to make sure the site was performant and could handle our traffic. The day we launched we got a front page article in the Wall Street Journal. We ended up with one million users. The site was up for a few minutes, and then, to my horror, it crashed. 

I thought, ‘I had one job to do, and I completely blew it. I’m definitely going to be fired!’

It turns out the next day we got a front page article in the Seattle Times titled, “Zillow Swamped, Crashes,” which ended up being incredible PR. These days I joke that that was my first foray into Growth since it ended up driving even more PR and user traffic. 

It may have turned out well in hindsight, but at the time, I was sure I was going to lose my job.”

This story showcases the inherent risk associated with taking on roles you aren’t quite prepared for. But Nate argues those risks are worth taking when properly evaluated for career accelerating potential. Each progressive experience of leveling up empowered Nate to make bolder and bolder career moves that catapulted his career forward - which leads us to our next principle.

Growth lessons from today's practitioners

Get our newsletter with insights from the field

Email Address *

Thank you for subscribing!

Finding Your Confidence

In one of his early reviews at Microsoft, Nate’s manager gave him some game-changing feedback. He said, “Nate, you’re doing great, but you need to find your confidence.” 

With that feedback and a bit of reflection, he was able to see that his lack of confidence was slowing him down. Multiple times, he found himself thinking, “I don’t know how to do that” while hanging back from tackling new opportunities. Classic imposter syndrome! 

One year into his job at Microsoft, his manager suggested he blog about the product he was working on. After publishing his first few posts and getting feedback from developers reading his code examples, he started to realize he was an expert (in some people’s eyes). The same thing happened when he went to developer conferences. People asked him technical questions, and he was able to answer them. 

He explains, 

“I had become an expert on something I didn't know anything about a year earlier. That was a big “a-ha” moment for me. I realized I’m actually good at figuring out new things, and instead of hanging back, I started raising my hand for the high-impact projects, even when I didn’t feel 100% confident. You don’t need to feel completely confident to step up.”

Over the course of his career Nate identified two strategies for landing these types of career accelerating opportunities:

  1. Leverage your strengths to open doors to opportunities that at first glance look out of reach.
  2. Take on side projects to figure out what you’re good at, what lights you up, and to build experience and credibility in the direction you want to move.

When you're trying to expand into new territory and make career shifts, Nate suggests, 

“Ask yourself, ‘how can I have the biggest impact?’ The answer usually lies in leveraging your strengths. Use them to get your foot in the door for new opportunities, and to give yourself a leg up.”

That’s exactly how Nate made the transition from engineering to product management. 

While he was still in his first role at Zillow as an engineer, he had the opportunity to take on some product management responsibilities. He helped drive release, process across the org, and communication between engineering, test, and operations. In taking on this work outside his core role, he learned he was naturally very good at program management, and that it was where he could add the most value at the company. 

He also realized that product management was what got him most excited to come into work every day. Though he hadn’t had any experience with product management previously, the fact that he had started doing it outside of his role gave him the opportunity to make a career transition within the company. 

He told Zillow’s leadership he wanted to switch into product management, and they gave him two options:

  1. Take a promotion in his current test management role OR 
  2. Switch to program manager and start over at the bottom level to prove himself, forgoing the promotion. 

This was a hard choice. Take the direct path upwards, or take a few steps “backwards” and bet he could level up quickly and make a huge leap forward down the road. He gave up the promotion and started over at the bottom. In doing so, he set in motion a critical turning point in his career that put him on the path to VP of Product and Growth. Talk about bold moves.

By the time..

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 
Reforge by Andrew Chen & Susan Su - 1y ago

By now, you've probably read Mike Moritz' article on the work culture of China's big tech companies, and what that means for Silicon Valley.

Cyriac Roedig (Shopkick's founder) also wrote that Chinese startups just operate on a faster timetable:

"Big startups are built in three to five years versus five to eight in the U.S. Accordingly, entrepreneurs who try to jump on the bandwagon of a successful idea scramble to outcompete each other as fast as they can."

Chinese tech giants get there faster, but only partly because of 9-9-6 as Moritz observed.

It's also due to a handful of other, maybe more important factors:

  • a dense, interconnected domestic market they're selling to
  • hyper-competition driven by population pressure
  • a culture that lives by the 80-20 rule, preferring to throw features against the wall instead of waiting for months to make them perfect
  • being regulatory bedfellows with the Chinese government

Let's dig into each of these factors one by one.


Andrew has a great story about this, firsthand:

“My own experience with this came from competing with China's Didi when I first started at Uber.

When I first joined Uber back in 2015, we were right in the middle of the showdown with Didi in Beijing, China. Didi, like most Chinese tech companies, had a company wide philosophy of 9-9-6, which stood for “9 am to 9 pm, 6 days a week.”

This was incredibly intimidating because they had thousands of people following 9-9-6. We had a team of a few hundred folks focused on China (along with other, non-China Uber responsibilities).”

This isn't to say that we all need to be working 9-9-6 in order to build a successful company, but more hours x more talented people does tend to lead to bigger outcomes.

Dense, interconnected markets

There are 850 million people under the age of 40 in China (compared to 160 million under 40 in the US). WeChat has almost 1 billion monthly active users, and a platform ecosystem of over 200K developers.

One of the most interesting things about WeChat's growth is where their user engagement is deepening.

Existing users' new contacts are now skewing towards work-related connections, not just friends and family. People are increasingly relying on mobile to handle one of the most critical things we do in work and person — pay other people.

Mobile payment transaction volume is estimated between $5.5 and $8.5 trillion for 2016, and likely a lot higher for 2017. Over 40% of in-store purchases (in physical stores!) are transacted through mobile payments.

More WeChat users with more friends plus more developers working 9-9-6 to make more apps, means that more and more of daily work and personal life is consolidating onto mobile platforms. This creates a dense and efficient network for launching a feature or an entirely new product.

Growth lessons from today's practitioners

Get our newsletter with insights from the field

Email Address *

Thank you for subscribing!

Greater population (more players) leads to more intense competition.

A few years before Meituan's 2016 merger with Dianping, there were up to 5,000 “Groupon clones” all scrambling for a piece of the China e-commerce / daily deal market.

China's bikesharing market is another example. Ofo and Mobike launched within a year and a half of each other. Less than a year later, dozens of different colored competitors had jumped in, including Bluegogo, Youbike, Qibei, Yibu, Xiaoming, CCbike, Hellobike, and lots of others.

It's happening in China's coworking market, too. WeWork launched a well funded expansion there in late 2017, but by then Urwork, Newspace, Naked Hub and other Chinese competitors had been going after the Chinese coworking market for a few years.

P2P online lending platforms are yet another example. At its peak, there were over 2,100 companies competing in that space.

People talk about Silicon Valley's winner-take-all environment. This is magnified in China, where success and consolidation scale in proportion to its worker/innovator population, and in tandem with the velocity of iteration. Meituan-Dianping's merger made it the 5th most valuable unicorn in the world, and created a super-app that's much more than the “Groupon of China.”

There are more details around the Darwinian pressure of China's tech innovation, but one thing to remember is that its massive and densely interconnected population also means that sample sizes are bigger, and success is (or has to be) much bigger. A few million MAU is nothing in a total addressable market of a billion.

In her post on mindsets for thinking about innovation and China, Connie Chan shares some perspective on this:

“To put it bluntly, a company that has reached 1 million registered users in China hasn’t really “cracked” China; for instance, if Tencent had a product with just five million monthly active users in China, it might consider shutting the app down!...

In mobile, China Mobile apparently has (as of the end of last year) nearly as many 4G mobile subscribers as the entire U.S. population. Not to mention nearly three times as many total customers as there are people in the U.S.

China is thus an ideal place for startups to find all sorts of insights on user behaviors at scale.”

It's worth noting that China Mobile is just one of three major carriers in China.

China lives by the 80/20 rule

When James Palmer wrote about China's “chabuduo” culture, it was to point out the problems with 'good enough:'

“The prevailing attitude is chabuduo, or ‘close enough’. It’s a phrase you’ll hear with grating regularity, one that speaks to a job 70 per cent done, a plan sketched out but never completed, a gauge unchecked or a socket put in the wrong size.

Chabuduo implies that to put any more time or effort into a piece of work would be the act of a fool.”

But the flip side of “chabuduo” is how it enables people to get things done faster, even if it's (potentially) at the expense of quality.

A nose to the grindstone work ethic, lots of hands, and a culture of throwing it against the wall is what enables Chinese companies to build a train station from scratch in 9 hours, or to copy, build and sell a structurally simple piece of consumer hardware faster than its original creator can launch a Kickstarter campaign, and it applies to more than just simple hardware products or manual labor construction projects.

Across a three year research project, MIT researchers documented China's accelerated innovation, partly attributable to the lower cost and bigger supply of Chinese engineers, but in even larger part due to an “industrialization of innovation” that divides up the discrete steps of the innovation process and throws a team of (inexpensive) talent at each step, kind of like an assembly line.

“When Tencent launched the first version of the QQ reminder application, it was geared toward appointments, birthdays and anniversaries. Users quickly pointed out that the product had a missing feature: reminders for when their favorite sporting events were about to begin. More surprising to Tencent’s developers, however, was the flood of input they got from gaming enthusiasts who wanted reminders about the schedules of computer-game tournaments.

Within weeks, the Tencent team released a new version that incorporated both functions. This rapid cycle of launch-test-improve has now become core to Tencent’s innovation process.”

Chinese companies are doing this in non-Internet sectors, too.

“Mindray Medical International Ltd., a company based in Shenzhen that is China’s largest maker of medical equipment, for example, released the initial version of its BeneHeart R3 electrocardiograph machine into the market following 18 months of product development.

Mindray routinely launches new products every six months, in stark contrast to the typical two-year launch cycles of some of its foreign competitors.”

Regulatory bedfellows

In the US, regulations and legacy systems that used to serve us so well (ie, the banking system) can end up holding us back when it comes to disruptive innovation, and there's traditionally been a relatively strict divide between government and the private sector.

Google only hired its first lobbyist in 2006, years after it was already public and big enough to be a heavy hitter (Twitter's first lobbyist came in 2012, and Facebook's was some time around 2011). 'Consulting' government so late in the game contributed to a relatively adversarial relationship, which in turn has an impact on the access, and velocity, that companies are able to get.

By contrast, Chinese entrepreneurs were tight with the government from day one, which has led to a more collaborative environment (one that goes both ways though -- the government also forces large Chinese tech companies, like Alibaba and Baidu, to share user data and work on projects in their interest). This means the Chinese government is often actively clearing regulatory obstacles that would slow down innovation because it views tech as a national policy interest.


To understand China's innovation, we have to look at all the factors — socio-cultural, technical, IP, political. It's not about faster copycats, or just about 9-9-6, or a mobile-first ecosystem. China's companies won't be confined to their massive domestic market forever, either. Even if you aren't actively going after the China market, Chinese companies could be coming after yours.

China is and will continue to be exporting a lot more than manufactured goods. As consumers of Chinese-created tech, and as entrepreneurs, investors and market watchers, where that goes will be relevant for all of us.

Further Reading on This Topic

Mindsets for Thinking about Innovation In — and Competition from — China by Connie Chan
How a Chinese Food Unicorn Nails Acquisition, Monetization, and Defensibility by Xianhang Zhang
Fast and furious: Chinese unicorns to overtake American counterparts by Linda Lew
Chabuduo — What Chinese Corner Cutting Reveals about Modernity by James Palmer

Andrew Chen works on growth at Uber, and is a growth advisor to companies including  AngelList, Dropbox, Product Hunt, and Tinder.  


More Posts from Reforge

Susan Su is head of marketing at Reforge, a company that provides masterclasses in frontier skill sets for mid-career tech professionals. 

 More Essays on Growth from ReforgeGrowth lessons from today's practitioners

Get our newsletter with insights from the field

Email Address *

Thank you for subscribing!

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Bela Stepanova is currently a Director of Product Management at Box and a Reforge Growth Series alum. She founded Box's Growth team, and before that Box’s Product Operations team. Prior to joining Box, she ran product and engineering teams building large scale financial platforms for Accenture clients. She also spent a few years helping global non-profits create and execute their CRM & e-commerce technology strategies. As a product leader, her biggest passion is people - from bringing a user experience first mindset to B2B products, to building diverse teams and investing in the next generation of leaders.

Growth is no longer just for B2C and consumerized B2B companies - if you’re building an enterprise product and you’re not thinking about growth, you should be. 

Most people think “growth” isn’t relevant for enterprise. As a result, companies end up building complex products to satisfy a list of requirements from IT buyers that leave end users frustrated and looking for a “better mousetrap.” As large enterprises transition to SaaS products, this outdated approach leaves the doors for product disruption wide open. 

While product disruption in this space has been slower than in consumer and high volume SMB due to built-in defensive moats around contract structure, technical integrations, and brand, it’s now starting to happen faster. Think Salesforce, Box, and now Slack as it expands upstream towards enterprise. 

To unpack the dynamics behind this shifting enterprise landscape, I’ll walk you through the following in this post:

  • How user experience and growth is changing the enterprise game
  • How to customize the user funnel for enterprise 
  • The 4 pillars of enterprise growth and how to align them with consumer growth strategies

As this wave of change accelerates, some enterprise companies will hunt down the opportunities for growth embedded within the shifting landscape, while others will bury their heads in the sand until it’s too late. Which will you be - the lion or the ostrich?

How User Experience and Growth is Changing the Enterprise Game

Enterprises are undergoing a digital transformation due to three significant trends in the market. They have been switching from:

  1. On-premise hardware to cloud
  2. Purchasing and installing software to SaaS
  3. Desktop to mobile

It’s these changes that are facilitating the consumerization of IT and revolutionizing how decisions about software get made.

Traditionally, when large enterprises implemented new technology systems and onboarded employees, the costs of switching became prohibitive. These switching costs created huge moats for legacy enterprise software companies, making it nearly impossible for competitors to gain traction. But now, due to the forces of digital transformation, this conventional enterprise paradigm is collapsing and creating new opportunities for technology companies that prioritize user experience and growth.

These days, if users aren’t using a product, the real switching cost is logistical. Imagine this - a company installs Salesforce, but the team doesn’t really adopt it - no one’s inputting contacts or tracking communications. When IT identifies the issue, no internal development work is required and no big migration is needed to switch CRMs - IT can just plug and play another SaaS product.  

User adoption and engagement are becoming important measurement tools for enterprises as they evaluate switching technologies. IT buyers now measure post purchase success using product engagement metrics, and many now even measure end user productivity as a key input to core business metrics. 

These changes mean user experience is no longer a checkbox on Requests for Proposals (RFPs) - it’s a key measurable metric. While user experience and user growth might be a limiting growth factor for most older enterprise products, I see it as an opportunity for enterprise software companies to build a competitive advantage. 

This is why at Box, we take a proactive approach to user-focused growth. This may be unique in enterprise now, but we think it’s where things are going.

Growth lessons from today's top practitioners

Get our newsletter with insights from the field

Email Address *

Thank you for subscribing!

An Enterprise Flavor for the User Funnel

Like any good growth team, we have a well defined user funnel and crisp north star metrics. But because Box is an enterprise product, we customize the traditional funnel to account for both the user journey and the needs of the buyer and IT administrator. For Box, this means adding additional steps at the top of the funnel for “seats sold” and “seats deployed”.

With Box, and many other enterprise products, after seats are purchased, enterprise administrators often need to create new users in the system before the end users themselves ever touch the product. This step is critical to enterprise grade deployments. Companies often need to set up hundreds or thousands of users simultaneously, while configuring the product to fit specific business requirements, such as compliance and security policies, notification settings, and other specifications.

Without seats sold and deployed reflected in our user funnel, we wouldn't be able to correctly model growth for the product. Adding these additional steps allows us to get a full picture of the funnel so that we can accurately identify the highest impact growth opportunities and effectively apply consumer growth strategies. 

Like many other growth teams, we continuously tackle user on-boarding and improve retention by helping users get started with the product. However, during our initial activation experiments, we hit some limitations of common consumer activation strategies. Specifically, we saw that account level activity from buyers or IT admins could hinder the effectiveness of user level activation tactics. 

For example, if new users weren’t added into the right groups by the administrator, when they tried to access work documents on Box for the first time, their initial experience of the product was disorienting, no matter how on-point our onboarding was. This showed us that working with and onboarding administrators was also an integral piece of the user growth puzzle for enterprise products. 

From these insights about the differences between buyer and admin needs and user level needs, we developed a framework that includes both consumer and enterprise approaches to growth to guide our efforts. 

The Four Pillars of Enterprise Growth

In contrast to the common perspective in enterprise, at Box we're finding that consumer growth models and strategies can be applied to accelerate the growth of enterprise products in a repeatable way. Through our experiments we’ve produced retention gains by adding nurture steps to in-product retention levers, improved activation via onboarding flows, and increased engagement metrics through changes as simple as testing copy on key actions in the product. 

But when it comes to applying growth to enterprise, there's one caveat - enterprise products have a unique set of growth levers that consumer products do not. When blocked, these levers limit an enterprise product's growth potential and diminish the effectiveness of "traditional” consumer growth tactics. When unblocked they serve as boosters that catalyze user growth and adoption.

We’ve bucketed these growth levers into four key pillars that underpin growth for enterprise companies:

  • Business processes
  • Migrations and integrations
  • Employee events
  • Compliance and policies

These four pillars are not new - for years they have been leveraged in one form or another by enterprise practitioners to drive sales for enterprise products. At Box, they have been instrumental in our fast paced growth over the years. 

What is new however, is how we’re aligning them to consumer growth strategies to drive user activity. Let’s walk through each of the four pillars in detail below, unpacking how each interplays with consumer growth levers to either limit or supercharge enterprise growth.

Business process

Business process is the heart of operational efficiency, as it determines how work gets done in large, global organizations. It is comprised of a predefined set of steps companies put in place to help employees complete their day to day work. Some examples are - fiscal year planning, new hire offer approval, security protocols, and many others. 

As our growth team has been running experiments, we’ve confirmed that our ability to influence user level activity can’t be isolated from these deeply ingrained processes. We saw this interdependent relationship in action when we first tested in-product user onboarding at Box. We created a simple 4-step onboarding to walk users through the how-tos of the product, and saw our activation metrics improve for freemium and SMB users. But they didn’t budge for enterprise users. 

Why could that be? Working with our user research team, we ran a simple survey with our enterprise users to understand what they found difficult or easy in their first experience with the product. The survey revealed process as the culprit. 

Users wanted to understand why and when they should use Box. They needed to know which of their company’s existing processes it would help them navigate, before jumping into the “how” of using the product. 

Growth lessons from today's top practitioners

Get our newsletter with insights from the field

Email Address *

Thank you for subscribing!

Complex Migrations and Integrations

No product can live on its own island, especially in enterprise. The first time I realized how convoluted most IT landscapes are was over 10 years ago, when I created a roadmap for a new claims product at the insurance company where I was consulting. Working through how it would integrate with the company’s internal data technologies was eye opening. 

Large enterprises run on many varied and inflexible technologies, and what most of us consider legacy product can’t be easily retired. Helping customers retire an existing enterprise content management (ECM) system or a network shared drive, and replace it with Box (instead of layering Box on top of current solutions), allows us to boost both sales and user activity. 

Due to the complexity of migrations, Box formed a consulting team that engages with customers to guide them through the migration process. Once customers complete the migration of content from other systems to Box, user activity increases as users can get more of their work done within the product.

Similar to migrations, integrations with other software solutions also drive user activity. They improve user experience and smooth business processes across teams, acting as a lever to drive engagement and growth. For instance, Box has built out integrations with products like Salesforce and Office365 to make it easier for users to manage their work in one place, and ensure that content is centralized across the company.

Employee and company events

Mergers and acquisitions, policy and regulation changes, and new hire onboarding are all examples of events that can affect if, how, and when an enterprise customer and their employees engage with your product. All of these can trigger a sudden influx of new users or an unexpected exodus of existing users.

To manage these types of events effectively, enterprise growth teams must see them as a lever to drive growth or inhibit churn, depending on the nature of the event. They can prepare for these events by coordinating the user journey with the buyer and admin journey so that when these events do happen, they’re positioned to take advantage of any growth opportunities or mitigate any potential churn risks. 

For instance, enterprise teams need to make sure administrators have access to the right tools and understand the methodologies necessary to navigate any employee events as smoothly as possible. At Box, our growth team works closely with our customer success team to ensure we hear about, respond to, and leverage such opportunities. Similarly, we collaborate with the product team that focuses on the administrator experience to ensure we can take advantage of user growth opportunities facilitated by the admin. To reinforce these cross functional collaborations representatives from both teams are directly embedded into our core growth team.

Compliance and Policies

To build solutions to problems in highly regulated industries, such as finance, healthcare, and transportation, companies have to build compliance into their processes from the ground up. If they don’t, they can run into governmental blocks, regulations, and potential fines as they grow. Companies that operate in highly regulated industries are usually ones of scale, and for them to integrate external software it must meet the necessary regulatory restrictions. 

At Box, we’ve found that sometimes customers have older policies that restrict certain subsets of employees from using certain products. This often requires working with the customers to redefine these older policies or adapt our product, which can expose new segments for user acquisition. For example, many companies have policies dictating how work documents can be shared externally. We’ve found engaging with the customer, especially the IT administrator, to identify, understand, and modify the policies or our product, allows us to find solutions that open the door to user growth and engagement. 

It’s Just the Beginning for Growth in Enterprise

Most people in growth overlook enterprise, and most people in enterprise overlook growth. They think growth and enterprise are incompatible, but at Box we’re turning that mindset on its head. 

By bringing growth principles inside our organization and having a cross functional team dedicated to figuring out how to adapt the best that growth has to offer to the enterprise landscape, we’re shaping the future of how enterprise products grow. 

Growth lessons from today's top practitioners

Get our newsletter with insights from the field

Email Address *

Thank you for subscribing!

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Prior to running growth at startups, Joey DeBruin was a neuroscience researcher at two of the most productive labs in the world - Peter Calabresi’s multiple sclerosis lab at The Johns Hopkins School of Medicine and Scott Zamvil’s neuroimmunology lab at UCSF. During this time, he witnessed how these labs optimized the scientific method to produce successful experiments. Now he leverages those fundamentals to run rapid experimentation as the Head of Growth at Feastly, an online marketplace connecting adventurous diners with talented chefs outside of the traditional restaurant. Feastly has grown 10X since he joined two years ago.

Recently, a colleague was telling me about a “successful” experiment they were excited about. They hypothesized that the best time to encourage users to upgrade from a free trial was while they were using a certain common feature.

To test this they put a big button on that page and saw a measurable lift in conversion compared to users who didn't see the button. Based on this result, they were attempting to drive more free users to that feature.

I asked if they had experimented with putting the same button on other commonly used features, and they hadn’t. They hadn't controlled for the risk that the lift was simply a result of putting a prominent CTA in front of more users.

This was especially concerning given that they were funneling resources into more projects based on the assertion that this one feature is important for activation. It was apparent they just didn't understand how to design proper controls to prove their hypothesis.

I’ve seen examples like this countless times when talking with other growth professionals at companies from seed to IPO. Many experiments they describe are under-optimized or invalid due to flawed design - and they don’t even realize it.

It's great that companies are using the scientific method for growth. What used to be “billboards and a prayer,” has evolved into repeatable and measurable systems for growth.

But, many teams don’t get as much value out of the scientific method as they could. Even worse, they often apply it incorrectly which produces misleading results.

By now, most practitioners responsible for growth know they should be using the scientific method to guide their efforts, but until recently, many hadn’t actively used it since 6th grade science class. (Luckily, there are some useful resources out there for ramping up on building processes for growth.)

But if being effective in your job is dependent on using the scientific method, you need to study how the best teams in the world use it. And the best funded and most experienced teams that use the scientific method aren't in tech, they're in science. Unlike the growth community, which is still in its first decade of existence, the science community has been refining the scientific method for thousands of years.

Growth lessons from today's top practitioners

Get our newsletter with insights from the field

Email Address *

Thank you for subscribing!

The Scientific Method From the POV of This Scientist Turned Growth Practitioner

Let’s start with a quick recap of the steps of the scientific method:

I break the method into five steps, categorizing steps 1-3 as the qualitative inputs to the method, and steps 4-5 as the quantitative outputs.

What I’ve learned from working in neuroscience labs, is that the qualitative inputs - question, hypothesis, and experiment design - are the most important steps of the process. If you get these steps right, you increase your chances of running successful experiments that produce valid results and actionable insights.

Any competent data analyst on a growth team (or lab tech in a research org) knows which stats tests and tools to use to collect and analyze results.

So, what separates the good growth teams (and research labs) from the great ones?

It’s the quality of the questions they ask, the hypotheses they come up with, and how they construct their experiments.

In this post, I'll walk you through 3 ways you can leverage learnings from the science world to tackle the qualitative inputs to the scientific method more effectively:

  1. Question - Ask “outside” questions to reduce bias in your process
  2. Hypothesis - Relentlessly seek the “minimum” in your MVTs to find a strong hypothesis faster
  3. Experiment - Control and blind your experiments to produce accurate results and reliable insights

If you evolve how your growth team approaches these three areas, you’ll drive breakthroughs in your growth processes so you can launch more successful experiments on a faster timeline.

Ask “Outside” Questions

The quality of the questions we ask drives the quality of insights our growth process produces.

This is why it’s critical to build questioning into our growth processes, rather than leaving it to chance.

Yet, many of us still aren’t that good at identifying the questions that will drive our businesses forward. Sometimes we rely on gut, whatever’s top of mind, or a directive from leadership to define the questions that will steer our growth strategy.

It's hard to see our own work objectively, which makes it even harder to ask unbiased questions that pressure test our assumptions. Established opinions and preferences limit our scope for effective questioning, especially when we’re trying to move quickly.

To neutralize the negative impact of their biases, every lab I’ve worked with integrated regular “outside” questioning into their system. They actively solicited external perspectives, which allowed them to:

  • Shorten their feedback loops
  • Mitigate the risks of going off track
  • Increase the probability of running successful experiments

The most powerful way they’ve built questioning into their process is with the weekly “pizza review.”

What is The Weekly Pizza Review?

The pizza review is a weekly meeting scheduled with other labs. The meetings are usually an hour, with one or two presentations from each lab. It’s a tit-for-tat exchange in which both labs ask each other unbiased but well-informed questions about their ongoing projects. (They’re called “pizza reviews” because thousands of years of research has determined that the #1 way to get underpaid scientists into a room is by offering free lunch.)

We’ve all heard the adage a million and one times - “your network is everything.” So, we go to the occasional networking event or conference, maybe make a new connection or two, and listen to growth leaders from Slack, Facebook, and Uber share their latest learnings. Networking... check!

Not so fast - this type of networking is superficial. It doesn’t give you the opportunity to go deep on your work with outside professionals who can pressure test your process and provide feedback.

Though the pizza review sounds informal, it’s not. It’s foundational to how the best labs operate and is baked deeply into the culture of experimental inquiry in the science world. It isn’t about meeting up with a handful of fellow practitioners once in a while, eating a few slices of pizza, and trading high level tactics.

It’s about regular, scheduled, in-depth, unbiased, and targeted inquiry to improve the questions we ask, the hypotheses we come up with and the experiments we design. All of the best labs do it.

When I made the transition to growth, I was surprised to learn that, as a community, we don’t have a similar practice. It’s a big hole in our processes as practitioners of the scientific method. When I went through the Reforge Growth Series, I often heard participants say they were hoping to meet other growth professionals and learn how they run their teams. In science, I didn't have to apply to a program to do this. When I joined my first lab, I had a meeting with outside researchers my first week.

Let’s walk through a few learnings on how to make it easier and more actionable to create our own versions of the pizza review.

Find People Who Are Familiar With Your Field But Not Direct Competitors

While I was in the multiple sclerosis lab at Hopkins, our most fruitful pizza review was with an ALS group. Both diseases have autoimmune components, so we spoke the same language. But MS and ALS are also fundamentally different diseases, which meant we didn’t have to worry about stealing each other’s spots in the top journals.

Similarly, while at Feastly, I have set up regular lunch meetings with growth or product people working at food companies or marketplace companies - but never both. The goal is to solicit questions that are well-informed, but still lend fresh perspectives.

There are lots of ways to find these people:

  • Tap into existing networks of growth practitioners via companies like Reforge, Growthhackers, or Tradecraft.
  • Search for growth practitioners on LinkedIn at companies you admire. I have yet to be turned down on a cold reachout. People don’t like being sold to, but they love being asked for advice.
  • Build relationships with product, marketing, sales, or other teams within your company. In addition to getting fresh perspectives on your ideas, it will help bring about a growth oriented culture.
  • Launch a small growth mastermind group with practitioners from various companies - meet regularly and allow members to present projects for feedback and questioning.
If You’re Really Concerned About Keeping Your Data Private, Use Case Studies or Literature at Your Pizza Reviews

We did this occasionally in the lab if we were concerned about a major result leaking too early. In these cases, we’d pick analogous papers that had been published recently by other labs and present those in place of our own work. We’d write down all of the questions we received about the paper (people would really let loose in these cases!) and take them back to private meetings, where we’d identify overlap with our own projects.

Learn More Growth, One Email Per Post. Email Address *

Thank you for subscribing!

Relentlessly Seek the “Minimum” in Your MVTs

Whether in research or growth, great teams seek out ways to maximize the percentage of their projects that produce positive results. Once they’ve incorporated unbiased questioning into their process, the next step is to run better minimum viable tests (MVTs) so they can identify where they do and don’t want to invest.

This is an especially dire concern in the science world because projects in medical research require enormous investments - all of the projects I led at Hopkins or UCSF cost tens of millions of dollars. The ability to find “validating data,” is a skill researchers (and their bosses!) value above nearly anything else.

There are a handful of learnings I’ve brought from my time in the lab on how to collect validating data to inform better hypotheses. Let’s walk through them below.

Start “In Vitro”

In vitro literally means “in the glass” and is used to describe experiments that are conducted entirely in test tubes. Eventually, all medicines must go through in vivo experimentation to be tested in live animals, before going on to save lives. Yet, virtually every major in vivo project starts in a tube or petri dish because in vitro experiments give you complete control over the testing environment, and are far less expensive.

Right now you might be saying, “I know, I know, build an MVP before going all-in. This is old news!” But do you consistently follow that advice?

Under the pressure of aggressive growth targets from leadership, teams often get carried away without even realizing it, forgetting the “minimum” part of their MVT or MVP. Oftentimes, they end up with expensive failed experiments, in terms of resources and time, that could have produced the same results in days or weeks, instead of months.

For example, a recent company I spoke with thought introducing a subscription model would be a huge growth opportunity. Instead of kicking off their inquiry with an “in vitro” test, they jumped directly to “in vivo.” They changed the product and payment system, onboarded existing and new users, and after months of investment from product, design, engineering and marketing, they realized their customers just didn’t want a subscription.

A one-day or one-week in vitro test using a landing page builder to mockup a subscription pricing page, driving traffic to it via ads, and measuring conversion rates against current pricing may have quickly revealed that a subscription pricing model wasn’t appealing to their customer base.

However you do it, the point is to use an in vitro test to get validation that an in vivo test is worth the additional resources. Tools like Unbounce, Optimizely, and paid advertising make these kinds of in vitro tests easy and fast. And since you aren’t dealing with existing users who were acquired through different channels, had varying CACs, and have different lifetimes, the control is far higher.

One of my favorite mentors, Anantha Katragadda, told me when I was starting out in growth that if I couldn’t think of a way to fake a product experience in order to get validating data, I simply wasn’t trying hard enough.

Curate Your Own Journal of Failed Projects

One of the major challenges in science is that there is no journal of failed projects because only successful projects are published.

Let’s say a highly influential paper comes out with an obvious next step. Over the course of the next 10 years, 50 different scientists may say “Aha! I can’t believe nobody has published that no-brainer follow-up study yet. I will devote my own time and resources to it.” Unfortunately there may have been a flaw in the initial paper or some other reason that the no-brainer follow-up just won’t work. Fifty labs have now wasted valuable resources - an unfortunate result for the scientific community.

The best research labs approach this problem by seeking out negative results. They spend as much time asking for and sharing failed results as they do successful ones.

I’ve been in the unfortunate position where my team ran a test that another team in the organization had already run unsuccessfully. Ever since, I’ve made sure that every major project is shared even if the results aren’t favorable.

Sharing and collecting negative results isn’t fun, so it’s only going to happen if you:

  1. Bake it into your process
  2. Support open sharing of failed experiments within your company culture

If you don’t do both of these things, failed projects will be swept under the rug by whomever is working on them, only to resurface later as wasted time and resources when an unsuspecting team member runs a similar experiment.

Here’s how I keep track of failed projects with the help of my growth team at Feastly:

  • Build a project pipeline that requires analysis and sharing before any project can move to “complete.”
  • Whenever we meet with outside colleagues, we ask for one thing that’s worked recently, as well as one thing that hasn’t.
  • We build acceptance of and learning from failed experiments into our culture. Growth teams are optimized for learning, so finding negative results isn’t a bad thing because it facilitates learning. We celebrate the learnings that result from failed experiments constantly. For us, an automated celebration message is sent to Slack anytime a project finds positive results OR negative ones, and we’re diligent about measuring individual performance independent of the ratio between positive and negative results.
Control and Blind Your Experiments

Growth teams optimize for speed, which often makes running clean projects harder, while the culture of “move fast and break things” is often used to justify taking liberties on controlled design, proper stats, or both.

Given how projects become intertwined, a misinterpreted or poorly designed project can unravel months of work or set teams up to chase losing opportunities. Uncontrolled results can become building blocks for resource intensive follow-ups, with the flawed data only becoming apparent as the follow-ups miss their mark.

To publish a paper, every result must disprove the null hypothesis. This means that you prove, with statistical significance, that the result you are seeing is not due to random chance or biased observation. Both of these premises are easy to miss, so here’s are a few straightforward tips to help you design sound experiments.

Understand How to Properly Control Your Experiments

Placebo controls are not just for medical research.

Let’s say you launch an ad or email campaign retargeting users who purchased and later churned with copy around “missing out on what everyone is talking about.”

Your hypothesis is that FOMO is an effective strategy to increase resurrection rates.

Having a control group that doesn’t get the ads or emails is not enough to validate your hypothesis, even if the results are outstanding. You need a “placebo” group that controls for both independent variables so that your groups get the same frequency of ads or emails, but with different copy, to isolate the impact of the copy.

For every experiment, make sure you control for every independent variable in your hypothesis.

Understand How to Properly Blind Your Experiments

Any project that relies on qualitative observation should be blinded.

Let’s say you’re watching user sessions to compare how high LTV users interact with a feature compared to low LTV users. At the end of the session, you give the user a grade from 0-10, 0 indicating they didn’t understand how to use the feature and 10 indicating they easily understood it.

Your hypothesis is that low LTV users don’t understand how to use the feature.

If you view 10 sessions from high LTV users, and then view 10 from low LTV users, your results will be invalid. Since you’re coming in with a hypothesis, you’re far more likely to perceive things that support your ideas.

The correct way to conduct this test would be to have a colleague assign a number to all 20 videos and then send them to you. You record the results and then send them back. They then “unblind” the data by noting which videos were of high LTV users and which weren’t.

When you’re designing every experiment, verify that human bias won’t skew the observation of the data.

Be Diligent About Statistics

This should go without saying, but in growth at early stage companies I estimate that only 20% of results are passed through the appropriate statistical tests. I’ve found that people often think that stats are used by nerds who are too cautious and don’t want to move quickly, but the opposite is true.

Stats tell you how much risk you’re taking on by accepting the results as true, which is a powerful data point if you’re moving at hyperspeed and iterating rapidly.

If you want to move more quickly you can simply increase the minimum P value on your tests.

Medical journals require a P value of .05, which means that the results should accurately disprove the..

Read Full Article

Read for later

Articles marked as Favorite are saved for later viewing.
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Separate tags by commas
To access this feature, please upgrade your account.
Start your free month
Free Preview