If you have a shop on Etsy, you’ve probably noticed that they do a lot of testing and experiments. A LOT. I often see sellers asking when the experiments and testing will stop, and I know that they won’t like the answer.
Table of Contents
- How often does Etsy run tests?
- Etsy tests are usually run on an account level
- Does Etsy stop testing during the holidays?
- Can we see what tests are being run?
How often does Etsy run tests?
According to the Etsy developer blog, Etsy has hundreds of tests running every day in order to determine if changes will increase conversion rate and revenue on the site. The pace of testing increased tremendously in 2017, according to Etsy CEO Josh Silverman in an investor presentation at the time. This increased pace coincided with a concerted effort on Etsy’s part to increase overall site revenue and to make the buying process easier for customers.
Etsy tries to run tests for the shortest amount of time possible in order to create fewer disruptions for sellers.
Regardless of how short a test is, though, it can be difficult to deal with if you’re included in multiple tests that affect the experience of being on Etsy.
Many sellers think that the tests are going to stop at some point, but unfortunately, that isn’t the reality of how Etsy’s experiments work.
Another point to make is that the search algorithm isn’t really a part of “tests” in the sense of people seeing a different version of shops and the Etsy interface.
When I’m talking about testing it’s more about the way that the site looks and how customers interact with it. It can be as simple as what color the “buy now” button is.
But changes in the search algorithm are a different process, and that’s not what Etsy is talking about when they mention tests and experiments.
The Etsy handbook says “What do we mean we say experiments?
Experiments are one way we gather data about how Etsy buyers and sellers interact with a new feature and measure how changes to that page or tool affect an end result.”
When they say “interact with new features,” they’re talking about site design changes, and “tools” refer to things like the listings manager. This isn’t about the search engine at all.
So, how do Etsy tests work? Why is it that sometimes one person will see something that other people don’t see? And if your shop is in a test, will you be able to figure out what is being tested on it?
Etsy tests are usually run on an account level
Etsy runs tests on an account level, which means that individual members will see the things that are being tested, but members who are not included in the test will not see the same thing. This means that if two customers come to your shop, and one is included in an experiment and one isn’t, they will see two different versions of the shop. A small percentage of all customers are included in tests, so shops won’t look the same to everyone.
If you’re an Etsy seller, this means that your shop isn’t included in a test in the sense that everyone who comes to it will see the thing that’s being tested.
If a customer whose account is included in a test comes to your shop they’ll see the format differently, but the majority of customers probably won’t.
This also means that your shop is protected from bad experiments to a certain extent.
If Etsy is testing something that ends up reducing revenue instead of increasing it, the risk for any given shop is decreased by having tests seen by a small percentage of customers.
So if you see a test that you don’t like, don’t worry about it too much.
If other people don’t like it either, and it makes them leave the site, Etsy will discontinue that test.
Etsy has said that they take a very ROI-positive approach to advertising and testing, and that if they don’t get a good return they discontinue whatever it was they were trying out as soon as possible.
I’ve seen tests come and go in a matter of days, so they do try to work very quickly to determine if something is increasing revenue or not. If it isn’t, they end the test.
Does Etsy stop testing during the holidays?
Etsy doesn’t stop running tests and experiments during the holiday shopping season, but they do slow the pace of testing down. This has been the case for the history of the site, and the developers refer to the slowdown period as “slush” due to the fact that it isn’t a total freeze on testing. The slush period lasts approximately from mid-October to the end of December. These dates can shift depending on what experiment priorities Etsy has in any given year, but the tradition is to slow down a few weeks before Cyber Monday.
A 2016 article on Etsy’s developer blog details the history of slush and how the concept used to be handled. At the time it was started they slowed down the pace from 5 experiments deployed per day to 1.
Those numbers have clearly increased, so even though the pace of testing might slow down during the holidays, it’s a pretty easy assumption to make that tests will still be happening frequently during the holidays.
It’s also fair to assume that the mentions of the flexibility of the dates of slush imply that the window of slowdowns is most likely shrinking as time goes on. I wouldn’t be surprised to hear that the slush period starts later than it used to.
What Etsy does tend to do during the slush period is to stop testing things that affect the customer interface, so that people don’t have to learn how to navigate shops in a new way during the holiday shopping season.
They put those types of tests on hold until the end of December so that customers and sellers don’t have to deal with changes that could confuse buyers and slow down the selling process.
The holiday shopping season is too important to start irritating people more than they’re already irritated, so as sellers we probably don’t have to worry about those types of changes in the slush period.
Can we see what tests are being run?
Etsy sellers can’t see what tests are being run on the site. Etsy used to disclose this information, but in order to preserve the integrity of the testing process they stopped revealing what experiments were being run. However, it’s possible to figure out what tests are happening based on observation of changes on the site and checking in with other sellers to see if they also see those changes. If the changes aren’t visible to everyone, it probably means that your account has been included in a group that’s seeing a new test.
In 2017 Etsy announced that they were going to let sellers know what tests were being run in the interest of being “transparent.”
They published a page in the Etsy Help section that detailed all of the current tests that sellers would want to know about.
However, what happened was that sellers would find out about a test, and immediately go change things in their shop to either play along with the test or to try to thwart it.
This ended up manipulating the test results enough that Etsy decided to stop telling sellers what tests were happening, and the page in the Help Section disappeared.
There was also an incident where someone at Etsy forgot to end a process, or they put the wrong piece of code in, or whatever it was, and suddenly it looked like you could get into the code of other people’s shops.
It turned out that it wasn’t actually code that you could alter, but it was code that showed exactly what tests were being run in that shop at the time. It made for some very interesting reading, and it basically showed which tests your shop was being shown for.
So while tests are run on an account level, the tests at that point were being turned on and off in the individual shops, then the customer who was included in that test saw it when they found a “match” in your shop.
Since Etsy doesn’t tell us how they decide to show tests, we can’t assume that that’s how it’s still done. However, it makes sense that it would be. Regardless, it doesn’t really matter, since we can’t affect the process!
Testing on Etsy is a constant process. It will never stop, because Etsy (and every other platform) is constantly trying to increase revenue through changing design elements and the way that customers interact with the site.
On top of that, there will constantly be changes in the search algorithm, which really isn’t part of “tests” per se. We can’t control the tests, so we need to focus on the things that we can control in our shops, and let Etsy handle the testing.
Recent Posts
When I fill orders on Etsy I try to get things in the mail as quickly as I can, but I've slowed down on that a little recently. Etsy wants us to ship things fast even, but is that the best...
Whether you're looking for a Christmas gift for new grandparents, a pregnancy announcement, or a gender reveal gift to let your parents know that they're going to be grandparents, our Artisan...