A/B Tests

How Can Marketers Prioritize A/B Tests? An Overview of PIE/PXL Methodologies

Not too long ago, we heard from my colleague, Lindsay Pugh, about the importance of A/B tests every change you make to your site. Hats off to her for laying out the risks of relying on gut instincts when determining how potential customers will interact with your site.

But if yours is a typical ecommerce site you have a long list of hypotheses worth testing, which means you’ll have time to do little else other than A/B test. How do you prioritize?

If you’ve been hard pressed to answer that question you’re not alone. Most marketers don’t have a methodology to prioritize A/B tests, or more concerning, a way to gauge success.

That’s where PIE and PXL come in. These methodologies are widely used frameworks for prioritizing A/B test, as well as determining whether or not your hypothesis is a success.

PIE (Potential, Importance, Ease)

PIE is an acronym for the three dimensions — potential, importance, ease — that are used to score each hypothesis. Each dimension is given a weight, and by adding them up together you arrive at a score. The A/B tests with the highest scores are the ones you should tackle first.

Here’s how Something Digital applies PIE to ecommerce:

  • Potential – What is the potential this change will have to improve your pages? If you anticipate just a slight improvement, then feel free to give it a lower priority.
  • Importance – How important is this change to your page? Will the change affect pages with the highest traffic, or pages that are seen by visitors who arrive on your site through costly pay-per-click ads? Is the element you plan to test essential for visitors to complete a transaction?
  • Ease – How complicated is it to test a particular change? Quick edits to product description copy are a breeze to test, whereas a complete redesign of your product page is inherently complex. The easier the fix, the higher the score.


Scoring the PIE Dimensions & Your Hypotheses

The first step of the framework is to score the dimensions. The easiest way to do that is to apply a scale, say 1 to 5 or 1 to 10 (at Something Digital we use a factor of 1 to 5). The scores are a bit subjective — what you think warrants a 3 in terms of ease your colleague (or competitor) may rank a 4.

Once you score all of your hypotheses and plot them in a matrix it will be simple to identify which are your low hanging fruit. But it also means that complex, yet important, hypotheses will be low on the your list. I’m not suggesting that you ignore them; rather, you’ll need to take a different approach to prioritizing them, one that is slightly less subjective. That’s where PXL comes in.

PXL Framework by ConversionXL

PXL is a framework that’s useful for prioritizing complex A/B tests. It’s a modification of the PIE methodology, developed by the editors of the ConversionXL blog. This framework, according the ConversionXL offers three benefits:

  • It makes any “potential” or “impact” rating more objective
  • It helps to foster a data-informed culture
  • It makes “ease of implementation” rating more objective


PXL helps you determine the potential by asking certain questions rather than applying a subjective score. For instance: Is this test for an element that’s above the fold? Is it noticeable within 5 seconds of landing on a page? Does it add or remove an element? Is it designed to increase user motivation? Does it run on high traffic pages?

PXL is super useful when doing web page testing, but in ecommerce, almost everything we do is above the fold, which means the efficacy of the methodology declines. At SD, we use PIE every day, and we use PXL more sporadically.

When Many Hypotheses Are Equally Important

Over time you’ll find that a great many of your hypotheses will have equal priority. This is to be expected: once you’ve identified the low-hanging fruit, the balance will sit in the middle. How do you prioritize them?

Clearly you’ll need a new set of criteria for establishing priority. Typically, your low-hanging fruit focus on improving actual conversions or sales. But there are plenty of micro-conversions that are critical to building your sales pipeline, such as signing up for your email newsletter, following your brand on social media, or even visiting your website (clicking product pages should be a micro goal because that behavior will likely put them into a retargeting funnel). Prompting visitors to interact with specific site content, such as a store locator or product configurations, are also worthy micro-goals.

By broadening the behaviors you can develop new scores for the hypothesis and apply the PIE methodology.

How to Handle Hard Stuff

Clients often approach us with help testing complex hypotheses, such as a product page redesign. By definition, such projects will never be easy to test. So how can you make it easier?

There are two ways to make complex testing easier. The first approach involves testing the hypothesis on a similar, but less risky, page. For instance, does your site have a page that looks similar to your product page that you can use to test your new design? Can you launch a static page with the proposed design so it can be tested in isolation? This approach boosts the score for ease as the web designer can focus on a single page, rather than coding the new design for all of pages and coming up with a way to split traffic.

The second approach involves breaking a big project into smaller hypotheses that you test individually. For instance, let’s say you want to redesign your product page. Rather than rolling out a new design all at once, you can begin by testing various elements: whether a larger Buy Now button brings more attention, rethinking where it sits on the page, or assessing if the size of the product title affects sales. All of these smaller outcomes will provide critical insight into the larger goal of revamping the product page.

One Last Thing

If you find that your hypotheses often prove false, don’t despair. False hypotheses means more data, and data lets you make better informed decisions. If we find that our hypotheses always prove true, I immediately question if we’re testing things that have value. We should be proven wrong sometimes because every client is different, as are their customers and websites. Broad assumptions are almost always wrong.

That’s not to say that nothing is universally true. We know that for consumers to purchase a product, they need to know what they’re buying (i.e. product photos and descriptions), how much it costs, and they need a buy button they can click. But outside of that, it’s all up for grabs. By testing all of your hypotheses you will drive continued traffic, micro conversions and ultimately sales for your business.

Phillip Jackson

A multi-instrumentalist, Phillip is an avid collector of vintage guitars, keyboards and amplifiers and has a home studio located in West Palm Beach.