Various Heresies and Why Split Testing Is Bad

Various Heresies and Why Split Testing Is Bad

I thought that marketing was all about split testing, wasn't it? Myth busted! There is an improved method for answering 95% of the questions that could be tested using split testing. Here, we'll go step-by-step.

You need to settle on an end goal before you can begin. Will it be profitable? Rank in search engines? More people driving? How many more links are going out?

You need to figure out how to measure OTHER PEOPLE'S sites for your intended outcome once you've decided on it.

Yeah, I know that looking at other people's results is a lot quicker and more accurate than doing split tests, which is why I'm against them.

Statistical analysis is the name for it. The foundation of most scientific advancements is it! Does your doctor administer both antibiotics and physical therapy throughout your visits? Definitely not! He would be fired right away. Rather, he will prescribe the medication or treatment that was found to be safe and successful for a statistically significant number of other people, based on their experiences in studies where the targeted outcome was the same as yours (doing away with that fever, cough, etc.).

A large number of other people have success with it; does that mean it will always be effective for you? No, absolutely not. However, in most cases, it will be effective. Split testing actually has the same issues. Choosing version "a" over version "b" after 20 actions in split testing increases the likelihood that you made the incorrect choice. Raising the number of possible actions improves your odds of making the correct choice. The converse is also true when comparing your outcomes to those of other people.

It's fair to ask how we can quantify these factors. Ranking in Search Engines is a Breeze. Sites ranked #1 with sites ranked #100, for example, or sites ranked #10 with sites ranked 101–110 might be compared. Of course.

An increase in visitors? A large percentage of websites have their logs publicly available. Alexa or Jupiter ratings could be a less than ideal indicator. To deal with less precise metrics like that, you may either compare more distant extremes (Alexa ranking in the 99,900-100,000 traffic ranks vs. Alexa ranking in the top 100) or expand the sample size.

Add more links pointing to your site. Open logs and referral entries are available to you once again... or you may put your faith in MSN's link: command... , or something less trustworthy, like Google's link: command (remember to either compare more distant extremes in the number of incoming links or expand your sample size).

What about making a profit? I just adore this. For the most part, we've come for this. Every statistical study of copywriting that I have published on my blog or included in the Statistical Copywriting Online Home Study Course has made use of this metric, as has my Glyphius program.

Where can I see the numbers regarding other websites' profitability? Do they intend to simply hand them over? Publicly traded firms actually do that very thing. Consider making advantage of publicly available information. Indirect profitability metrics are provided by a number of affiliate networks; for example, the Clickbank marketplace and the CJ EPC rating are not always indicative of true profitability.

You might also do what marketers have been doing covertly for decades, long before the advent of online shopping and the Internet, as an alternative. Statistical Copywriting and Glyphius were born out of this process. Actually, it's pretty easy. Here are two questions to consider:

1. Is it reasonable to keep paying for ads on a profitable site if I were the advertiser?

2. Would I keep forking over money to an unprofitable website for their advertising over and over again?

Most people would say "yes" to the first question. Most people would say "no" when asked to answer #2. Exceptions do exist. The profitability of advertisements is not even tracked by several major corporations... hence, if we employ this metric, they will incorporate some erroneous data.

This is a common occurrence in the scientific community. Participants in medical research often inflate or fabricate their findings. No one can say for sure what a person will do. On sometimes, they act in a way that goes against their own self-interest.

Is this the answer? In the same vein as previous flawed measurement methods... broaden the scope of the comparison and/or collect more data. We wouldn't require a huge sample size to examine these websites if people were completely predictable.

Alright, let's look at an example to understand it better. Imagine for a second that some naive "guru" marketer has assured you that adding the number "7" to your prices will boost your conversion rate and your bottom line. Ah, alright. Here are three options for you:

First, trust that person. I am pleased to inform you that you have recently become a member of the exclusive club of ignorant "guru" marketers. Your decision to apply your religious beliefs to a readily researchable topic is an example of why science should be prioritized over religion. There are 98% of the world's population who can relate; that should make you feel better about yourself. The sad truth is that nearly everyone will retire broke by the time they're 65 years old.

2. Do a split test. Oh no! Do you realize the reason you haven't done this before? Obtaining data that are statistically significant takes an eternity. The next thing you know, you'll have hundreds, if not thousands, of things to split test.

3. Examine the pricing strategies and profitability of other individuals using statistical methods.

Listen, we need to split test a few things. Having said that, wouldn't it be great if you could get a good idea of the final conclusion before running the split tests? When it's plain to see that most profitable sites utilize white backgrounds, is it necessary to split test a vibrant purple backdrop on your site? Definitely not! Perform statistical analyses on other people's website, but focus on testing the elements that aren't readily apparent.

Then, how can we arrive at the solution given above?

We start by compiling a list of sites that generate revenue and those that do not. Surely we are already familiar with the process? No? Alright, let's dissect it.

We begin by gathering a large number of statistically significant keywords and entering them into Google or another paid advertising search engine (or by perusing periodicals that have advertisements for websites). alternatively, we could look through some classified ads that contain URLs in a penny ad sheet... It's irrelevant... just visit a place where advertisements with links are shown.

Next, we compile a list that includes the specific ad along with its URL. The time has come to wait. After what amount of time will we have to wait? We patiently wait for those advertisers who aren't making money to either adjust or delete their adverts. Allow six months if you made use of periodicals. You may only be able to place ads for three months in some periodicals. Unprofitable advertising don't disappear for almost three months. You might be able to get away with waiting as little as two weeks on Google. There will be fewer places to compare as time goes on (the classic adage about comparing increasingly distant extremes).

I see you're at it again. Gather all the ads that are still paying for advertising and write down their URLs.

Evaluate the two lists now. Finding an ad that remains unchanged months after publishing increases the likelihood that it will generate a profit. A "profitable" list should include that URL. If any of the URLs from your first list aren't on your second list, move them to the "unprofitable" list.

Okay, so you've compiled a list of sites that make money and those that don't. These sites now allow you to do several types of fast statistical analysis. Considering that reviewing 100 sites is significantly quicker than getting 100 results from a split test... Right now, you're a huge lead.

Finding out what proportion of your "proftiable" pricing contain a "7" is as easy as going through your list. Finding the percentage of prices that contain a "7" requires you to go through your list of unprofitable items. Review the figures side by side. How near are they to one another? Using a "7" generally won't have much of an impact on your pricing strategy's profitability. To what extent are they apart? Were you surprised to see that only 21% of lucrative sites utilized a "7" in their price, whereas 83% of unprofitable sites did? A big woo! Instead of waiting days for a split test, you have found your answer in just one hour.

Knowing that you tested a sufficiently large number of sites is crucial. That would need at least two years of formal statistics education to decipher, wow. If you're a single man thinking about majoring in statistics in college, I really suggest it, but if you're not, I have a practical method to find out that will save you a lot of trouble. It's interesting how statistics majors tend to have a more attractive appearance compared to other majors.

The ability to anticipate. Swap up your research and conduct a separate set of tests. For what proportion of cases did the split test's findings corroborate the statistical analysis's predictions? You can put greater faith in the dataset you created of successful and unsuccessful websites as you approach 100%. Naturally, zero percent isn't the absolute lowest... Fifty percent is. If you are just slightly over 50%, it is crucial to examine the study's assumptions... regarding the number of participants in the study... With a success rate of 75% or more, however, you can be confident in your dataset for use in further research.

You may be asking... hold on a second! Even yet, a split test was still necessary. Worse yet, it took me as much time as a split test to compile that list of successful and unsuccessful websites. so now what?

You need to compile a list of sites that generate revenue and those that do not, and then validate it once. After that, you're free to use the same data for as many research as you like!

Any kind of measurement follows the same pattern. With a 91% confidence level, my ranking data uses a single set of data on high- and low-ranked sites to ask thousands of questions! Using my single dataset of successful and unsuccessful websites, I have confidently answered hundreds of copywriting queries (96% confidence level in that example).

A lot of effort goes into answering the "7" question, while asking "what about the digit '3'?" requires hardly any effort at all. On the other hand, "Are blue headlines more lucrative than red headlines?" or "Does long copy actually generate more profit?"

Instead of spending days (or months, if you're a beginner still struggling to get traffic) doing split tests to find out which sites are profitable and which aren't, you can just spend an hour tabulating the results from your lists of profitable and unprofitable sites! After that, you'll have a response that's reliable 96% of the time.

Hey, guess what? In a split test with 20 actions (20 sales for "a" and "b"), your confidence in the results is only 95%. What is the expected time required to achieve a minimum of 20 sales for both the "a" and "b" versions in a split test?

I hope that was clear. My name is heretic. When I can just look at other people's sites to find the answers I need, I despise doing split tests on my own. Having already disagreed with 98% of marketing "gurus," I doubt this will be my last such outspoken disagreement. 

Location:

Share :