Making changes to your eCommerce site without measuring the impact is like getting dressed in the dark. The first step in getting smarter about these enhancements and understanding how moving needles relies on a commitment to measuring their effect. To get the most accurate measurement, we recommend testing every change you make side by side with the original. In the event that doing so isn’t possible, a before and after report is your next best choice.
The two most widely used types of before and after reports are “year-over-year” (YOY) and “period-over-period” (POP) techniques. While both give you a glimpse of improvement, they both leave room for error. Year-over-year doesn’t take into account trends you’ve already been seeing this year. You may have already been doing well or poorly. Meanwhile, period-over-period doesn’t take seasonality into account.
There is a third type of report. It’s not as simple as the two I just mentioned, and therefore not as popular. That said, it can give you a more accurate reading.
The Third Option
It’s called the trending delta, a statistical analysis that allows you to compare the trending data with observable data. Trending data is the result of several calculations, which take into account seasonality and the overall trend of your KPI. By comparing it to the observable data, we can calculate our improvement with increased confidence.
For example, the chart below displays the revenue of an eCommerce site over the past year. We’ll assume that a change took place between B and C.
We can see that comparing A to C (a YOY comparison) makes it look like the change made an improvement. The same goes for comparing B to C (a POP comparison). But when we compare D to C (a trending delta comparison), it’s pretty obvious that our change didn’t make an improvement at all. In fact, our change hurt revenue.
In the graphs above, trending data is shown as a very simple, linear progression but the truth is never so straight and simple. A trending analysis should take prior seasonality into account. For most retailers, this will primarily be expressed as high spikes in traffic and revenue at the end of the year. The more years you take into account, the more accurate your analysis becomes.
Another important consideration when looking at trending data is how many changes you’ve made over the past year. The more changes you’ve made, the more erratic and less reliable trending data becomes. It’s like any experiment- introduce too many variables, and drawing a conclusion becomes fuzzy if not impossible.
Calculating Trending Data
There are a number of ways to calculate trending data. Your best bet is to export data from analytics into Excel and use a function to make your calculations. The first go will take some patience, but once you have it, the process shouldn’t take long at all.
Want to start calculating trending data? Check out this article from microsoft on how to use the FORECAST function in Excel to calculate trending data.
If you’re going to test, measure. If you’re going to measure, be accurate. And if you aren’t going to use trending data, make sure you understand how POP and YOY data may be skewed. Lastly, remember that it’s natural to favor information that confirms your beliefs or hypotheses, but doing so will lead to making bad decisions.