I run a Japanese candy subscription service called Candy Japan, and periodically try to improve the landing page. It's dark magic to me, as I don't really have a great feel of what convinces someone to subscribe. The best I can do is to try to imagine what might improve sales, then compare a change against the original to see if it really is resulting in more sales.
It would be most satisfying to do this one small tweak at a time, as then I could know exactly what change drove the improvement, but with limited amounts of data this is rather hard to do. For example if you want to test a tweak that results in 5% more conversions, you need about 3000 sales to detect it! For Candy Japan this would mean waiting for about 10 years for the test to complete.
Even if you never reach statistical significance, all things being equal it is still better to follow the result of the test. If you had to play one of two slot machines and someone told you that they played each one 30 times and machine A paid out 10% more than machine B, would it not still be the best choice to choose machine A even if you can't be very confident about the result?
Why test entire landing pages at a time?
My mental model of the landing page is that every little element (and its position) on the page will have its own impact on sales. Most changes to these elements make very little difference, but some will have much bigger impacts. Perhaps there are dozens of elements on the page that each have a small ±1% effect, then the occasional ±10% one. If I picked an element at random and tested it, it would likely be one of those small effects.
But what if I just tweak everything and test that? The small effects would likely end up partially canceling each other out, not resulting in a big effect. But I would likely also end up flipping those larger elements as well, and with a bit of luck not in completely opposite directions. It's possible I could introduce one change that does a +10% and at the same time another one that does -10%, roughly canceling each other out, but it seems it's still a better test than picking just one element at random.
If I pick an element at random I only have a small chance of hitting a significant change, but mixing everything up I'm bound to change some big things as well. Besides that the results would also be more exciting, as I mess both up the result would be disastrous, but if I happened to improve both they would multiply each other, resulting in more than twice as big an impact.
With this justification, I put in about 2 weeks of time to test a completely new landing page design. To get something completely different, I bought a landing page template off of ThemeForest and customized it. That was a lot more work than I thought it would be, as I discovered that trying to understand someone else's HTML and make it do what you want can be as much work as just writing it from scratch.
Besides changing the code, it also required new photo assets and copy that I did not have, so I had to sidetrack a bit to create those as well. But finally I managed to customize it to be mostly as I wanted it to appear. To see the difference, you can see the old landing page here, and the new one here.
Also setting things up such that you can even serve another template randomly and still have everything work properly took some time. It would be confusing if you got a random landing page design every time you refreshed the page, so when a visitor came to the site, I would pin them to a randomly assigned landing page design, such that on subsequent visits they would be served that same design again.
I personally thought the new design looked better, although I was not very confident about that. I hoped it would win, because that would mean an increase in sales. On another hand I wouldn't be too disappointed if it lost, since I preferred working with my old familiar HTML rather than the newly adopted one.
I let the test run for a bit over three months. During this time there were a total of 67 new subscriptions. Of these 58% (39) came from the new design and 42% (28) came from the old design. Looks like the new one is a clear winner. This is a bigger effect than I would have expected, and a quick check with an A/B testing calculator even says that this result has significance (~90% likely).
I'm happy with the results, conversions increased noticeably by testing a whole new landing page design.
It seems that for a developer like me, customizing a ready-made template is a better way to go than trying to design everything from scratch by yourself. It could be that the difference was just in the assets I had to create in order to be able to use the new template, but in any case it pushed me to create a better landing page.
Thanks for reading!