5

I am hoping to reach someone who knows how to interpret data, if not, someone with better logic than me would still help :)

I had around 9000 users paying for monthly subscriptions for a service on site in last 3 years.

  1. 1554 active (with status "not cancelled" and with existent payment made in last 30 days)
  2. 1538 unpaid (with status "NOT cancelled" but failed payments in last 30 days or more)
  3. 6093 cancelled (with status "cancelled", cannot be activated or charged anymore).

Each day in last ~3 years subscribers signup, cancel, make payments, some cancel all their subscriptions and create new ones, etc.

Each subscriber (client email) can have multiple subscriptions, each in various states (active, cancelled, etc).

Around a year ago I started emailing subscribers, telling them "You had subscription X on my site, are you also interested in Y ?"

The problem: My total active subscribers are dropping in recent months.

The question: With the below available data, how can I see if my emails are generating new signups, OR the opposite (new cancellations), maybe users forgot about the subscription and cancel it when my email reminds them of it.

The data I have:

  • subscription_id
  • user_id (one user can have more subscriptions)
  • user_first_contacted (I also have each email date)
  • user_last_contacted
  • subscription_signup_date
  • subscription_last_payment_date
  • subscription_cancellation_date

Considering users cancel and signup any way for various reasons, can this even be tracked ?! (If the emails I sent cause more signups or cancellations). OR can it only be properly tracked if I split each new user in 2 equal A/B groups, then I only send emails to one of the segments (e.g. A) and later check if A has more cancellations than B.

Is this A/B test a must ? Because I started splitting the users months ago but this is too recent to interpret this data.

What I was able to track now by dates looks like this:

First I tried loading emails, then check if that receiver cancelled or signed up after the email:

total_emails_sent: 9914
total_cancellations_after_email: 122
total_signups_after_email: 75

total_users_with_cancellations_after_email: 98
total_users_without_cancellations_after_email: 9816


total_users_with_signups_after_email: 60
total_users_without_signups_after_email: 9854

Here I tried starting from subscription cancellations and go backwards 7 days to see if that user was contacted recently:

total_cancellations_with_email_before: 188 (37.23%)
total_cancellations_without_email_before: 317 (62.77%)

total_signups_with_email_before: 434 (36.66%)
total_signups_without_email_before: 750 (63.34%)

If I look at these percents in last lines, it looks like email notifications do not make a difference (signup percent vs cancellation percent), but I am not sure that is the right way to interpret this.

I have this CSV if it helps in any way: https://docs.google.com/spreadsheets/d/1q7ED6wjZPfLiJ5cXS5nrNPn9X80_oswTiQl7k9OKoNg/edit?usp=sharing

nwaldo
  • 500
  • 3
  • 13
adrianTNT
  • 151
  • 4

1 Answers1

5

Is an A/B test a must?

If you want definitive causal proof, then yes. Observational analyses can be heavily confounded. The experiment approach is the only certain way to isolate the effect of your emails from everything else happening in your business.

If you have a clear date on which you started emailing, which it sounds like you do, you can attempt the following:

  • Look at overall churn or signup rate by week/month for 6+ months before you started emailing.

  • Look at overall churn or signup rate by week/month after you started emailing.

  • Compare the "change" before vs. after in a difference-in-differences manner, ideally controlling for any known seasonal patterns.

Again, this is not perfect (other product or market changes may coincide with your emailing campaign), but it can give hints if you see a big shift around the time you started emailing.

Lastly, You found:

  • Cancellations with email in last 7 days: 188 (37.23%)
  • Cancellations without email in last 7 days: 317 (62.77%)
  • Signups with email in last 7 days: 434 (36.66%)
  • Signups without email in last 7 days: 750 (63.34%)

Notice that the breakdown is roughly 37% vs. 63% in both cases. As you point out, you might be tempted to say, "See, it’s basically identical--emails have no impact."

But:

  • Are these raw counts? Does "with email" vs. "without email" represent the same total population size?

  • Maybe you simply emailed a smaller fraction of your users at a time when fewer of them were likely to churn or sign up.

Hence, these raw numbers are not enough to form a definitive conclusion.

Anton Rasmussen
  • 51
  • 2