# Essential Guidelines for Effective eCommerce AB Testing
Written on
Introduction to eCommerce AB Testing
Understanding the intricacies of AB testing is fundamental for any eCommerce Product Manager. With numerous potential pitfalls, it’s essential to navigate these challenges effectively. A few weeks ago, I discussed the novelty effect, a risk that can arise from not monitoring the duration of your tests or the distribution of customer segments.
In addition to the novelty effect, here are three critical considerations to keep in mind for successful AB testing:
Isolating Change Variables
The waterfall method often leads to large-scale projects where extensive plans are created, timelines are established, and numerous changes are implemented simultaneously. Altering multiple variables at once can complicate your ability to attribute shifts in customer behavior or business metrics to specific changes.
By adjusting a single variable at a time and conducting rapid iterations, you can observe customer responses more clearly and link them directly to the modifications you’re making through statistically significant AB tests.
Maintaining a Flexible Backlog
Many Product Managers, often under organizational pressure, fall into the trap of crafting extensive backlogs that span an entire year. While this may seem efficient, it can stifle your ability to adapt based on test outcomes.
By fostering a flexible backlog and setting realistic expectations with stakeholders, you empower your team to succeed. The essence of AB testing lies in gaining timely insights into user reactions to new features or changes. If you can't leverage these insights to inform future work or adjust the sequencing of upcoming features, your efforts will be futile.
During my tenure at Nordstrom, our product page team ran multiple tests weekly. Regardless of your confidence in a feature's success, customer reactions can often surprise you—sometimes positively, leading to substantial conversion boosts, and other times showing no statistically significant improvements despite high expectations.
Embracing Change: Don’t Hesitate to Discard
In the world of product management, it’s essential to acknowledge that you won’t win every test. Holding onto features merely for the sake of retention can lead to unnecessary clutter. Just because a feature was meticulously designed doesn’t mean it deserves a place in your product.
Every additional element you introduce to your user experience adds complexity, either requiring more attention from the customer or creating additional maintenance for your team. Therefore, it’s vital to critically assess whether each AB test meets your success criteria and genuinely enhances value for your customers.
Conclusion
AB testing is a crucial strategy for any Product Manager. Always strive to isolate change variables, ensure your backlog remains adaptable, and don’t shy away from discarding features that don't add value.
About the Author
Ben Staples has over seven years of experience in Product Management and Marketing within the eCommerce sector. Currently, he serves as a Senior Product Manager at Nordstrom, overseeing product pages on Nordstrom.com. Previously, he held a similar role at Trunk Club, focusing on their mobile applications. Ben began his career at Vistaprint, managing cart and checkout experiences, and founded the Vistaprint Product Management guild, which now has over 40 members. For more information, visit www.Ben-Staples.com.
If you're interested in Product Management consulting, would like to receive notifications about new articles, or need book recommendations, feel free to reach out!