Randomized Controlled Trials

To understand causality—our ultimate goal—we must consider randomized controlled trials (RCTs). The crucial distinction here is that correlation does not imply causation. We aim to achieve real improvements in our users' lives.

Causality can be defined as follows: Event A "causes" Event B if, upon removing Event A, Event B would not occur. Essentially, a cause is the initial trigger for a series of events. If we eliminate the cause, the subsequent effects vanish as well.

To investigate causality, we can exclude a suspected causal factor and observe whether the effects persist, controlling for other variables. This is the essence of a controlled trial.

We frequently conduct informal controlled trials in daily life. For instance, if a lamp fails to illuminate, we hypothesize about potential issues—perhaps the lightbulb is burnt out, or the outlet is faulty.

Let’s formalize this:

Hypothesis: The outlet is defective, preventing power from reaching the lightbulb.

To test this, we have one trial with Outlet A (our Treatment trial). Next, we perform a Control trial by substituting Outlet A with Outlet B, which we know works. We maintain all other conditions constant.

Before proceeding with this experiment, we frame it to determine the Average Treatment Effect (ATE) of Outlet A on the lamp's ability to illuminate.

Treatment Effect = Probability of darkness with Outlet A - Probability of darkness with Outlet B.

If we find that P(Darkness|Outlet A) = 1 and P(Darkness|Outlet B) = 1, then ATE = 0, indicating Outlet A had no effect. Conversely, if the lamp lights up with Outlet B, P(Darkness|Outlet B) = 0, yielding ATE = 1, meaning Outlet A was entirely responsible for the failure.

Now let’s connect the lamp scenario to our users. Unlike appliances, human behavior is inherently unpredictable, complicating controlled trials. Users may interact with our app in vastly different ways: some might engage daily, while others may forget it exists. Some users may benefit from our product, while others might find it unhelpful or even detrimental.

Navigating User Variability

Conducting controlled trials with such unpredictable elements requires a different approach. We can't rely on a single Control and Treatment trial. Instead, we need numerous trials across diverse participants, ensuring random assignment to either group.

By randomizing assignments, we create comparable groups that, on average, share similar characteristics (e.g., gender, income level, etc.), allowing the effects of extraneous factors to balance out.

At this stage, we've established a randomized controlled trial (RCT)—the gold standard for determining causality. However, challenges remain. How can we ensure participants comply with their assigned group?

Addressing Compliance Challenges

Let's summarize our RCT framework:

1. Participants are randomly assigned to Treatment or Control.
2. Compliance may vary based on unobservable factors.
3. Responses depend on whether participants engage with the product.

We can illustrate these dynamics using a Bayesian Network, which outlines the conditional relationships between variables involved.

The ultimate objective is to calculate the Average Treatment Effect, which compares the likelihood of positive outcomes for participants who received treatment versus those who did not. However, without enforcing compliance, drawing causal conclusions becomes problematic.

In practical terms, we observe that various latent factors influence why participants either engage with the product or not. For example, some members might be more financially secure and thus more inclined to use Even regularly.

To dissect this complexity, we categorize compliance and response behaviors into archetypes, allowing us to simplify our understanding of participant interactions.

Compliance and Response Archetypes

1. Compliance Behaviors:
   • Always Takers: Access the product regardless of assignment.
   • Compliers: Follow their assigned group.
   • Deniers: Do the opposite of their assignment.
   • Never Takers: Do not use the product, regardless of assignment.
2. Response Behaviors:
   • Always Better: Achieve good outcomes regardless of product use.
   • Helped: Benefit from using the product.
   • Hurt: Experience negative outcomes when using the product but would have been fine without it.
   • Never Better: Do not achieve positive outcomes, regardless of product use.

This classification allows us to analyze participants based on their unique compliance and response types, ultimately leading to 16 distinct archetypes.

While we cannot pinpoint the exact distribution of these archetypes, understanding them helps us better formulate the Average Treatment Effect.

Exploring the Counterfactual

While we may never know the precise distribution of compliance and response behaviors, we understand that P(X, Y | Z) varies with U. In simpler terms, knowing the probabilities of outcomes based on treatment allows us to infer insights about underlying factors.

Now, let's pivot to our first video for further illustration.

The first video, "What is Your Product?", delves into understanding product effectiveness and how to assess if it meets user needs.

Although we lack the means to determine U's distribution, we can still derive valuable insights from the observable data.

The Role of Intention-to-Treat Analysis

Standard practice in experimental research often involves Intention-to-Treat (ITT) analysis, which assesses the effect of treatment assignment rather than actual compliance. While convenient, this method can obscure significant information and lead to misleading conclusions.

Consider a simulated experiment in which we gather extensive data on compliance and outcomes, only to see ITT reduce this rich dataset into two simplistic measures. This approach risks discarding crucial insights.

To illustrate the importance of a more nuanced analysis, let’s watch another video.

In "Want to test a product? | Does It Really Work?", we explore the implications of analyzing treatment effects more rigorously.

By employing a causal model, we can uncover the true impact of treatment by considering the factors influencing compliance and response. Such an approach allows for a more accurate evaluation of product efficacy, moving beyond surface-level analysis.

Conclusion: The Power of Causal Thinking

Adopting a causal framework rather than relying solely on observed data empowers us to make informed decisions about product effectiveness. Although many statistical methods may yield similar results, recognizing the potential for divergence highlights the importance of understanding the underlying processes that shape our data.

In summary, I hope this discussion equips you with new tools for evaluating product impact and a fresh perspective on problem-solving.