Common Flaws with Freelance Industry Surveys

It’s that time of year again. No doubt, if you haven’t already, you’ll start to see freelance industry surveys. They might summarize 2020, or they might offer trend predictions for 2021. Industry surveys can be fun to review, but be careful about using them to make serious business decisions.

Let’s look at why industry surveys can be interesting. Then we’ll go into some common flaws with freelance industry surveys and why you should be careful about taking action based on their reported data.

The Appeal of Freelance Industry Surveys

Industry surveys can be appealing for a number of reasons. For example, there can be a voyeuristic interest in simply seeing what others in your profession are up to from a distance.

When it comes to freelance industry surveys in particular though, it can also be about:

• inspiration;
• validation

For example, if you’re relatively new in your freelance writing career, looking at the high end of freelance pay from surveys can give you something to aspire to.

On the less-helpful end of the spectrum, these kinds of surveys can validate bad choices by making them seem more common than they might actually be. They can also overemphasize negative aspects of freelancing, based on how they choose their sample, which might make things feel hopeless.

Either way, it’s human nature to compare ourselves to others, and industry surveys let us do that on a large scale.

This is neither inherently good nor bad. But it does make sense to evaluate what you're hoping to get out of the data before you dive into the results.

5 Common Flaws with Freelance Industry Surveys

Coming from a PR background, I’ve worked with a lot of data reporting over the years. That’s why I understand the importance of getting it right. Sadly, not everyone conducting and reporting on this kind of data puts that effort in, potentially leaving you with misleading information that can hurt your career if you rely on it.

Here are five common flaws I’ve seen in freelance industry surveys. They don’t mean you should entirely dismiss them. Rather, know what to look for so you don’t let that misinformation influence your business decisions.

1. The sample size is too small to be useful.

A frequent issue with surveys run by bloggers in particular is that their sample size is often too small.

A sample size needs to be representative of the larger group the report makes claims about based on the data. For example, let’s say we’re talking about a general survey on the state of freelancing.

There were around 57 million freelancers in the U.S. alone in 2019. While you might ideally want to see a 10% representative sample in a survey, that would be tough with these numbers. On the other end of the spectrum though, if you saw a sample size of 100 respondents, you should immediately know the results aren’t reliable. Even 500-1000 respondents isn’t nearly enough in an example like this. And even several thousand results could never give you an accurate picture.

Obviously, the overall number of freelance writers would be smaller than that. But most freelance writing surveys I’ve come across still have unacceptably small sample sizes.

That said, bigger isn’t always better. You can sometimes find better, more actionable, information from a smaller well-targeted survey than a broader, larger one.

For example, while Freelance Writing Pros (All Freelance Writing's sister site for more experienced freelance writers) has a much smaller email list and reader base, it's much better-targeted than All Freelance Writing is. So a 100-200 response survey would tell you more about that target group (freelance writers with at least 5 years of experience and an interest in more advanced marketing and business tactics) than even 1000 responses would from a similar survey conducted here with a much broader base.

In this case, the results could show you what writers at a similar place in their career, from similar specialty backgrounds, interested in a similar approach covered on that specific site have to say. But I would publish those results along the lines of "Here's what FWP readers earned in 2020" rather than something overly-broad and unsupported like the "State of Freelance Writing in 2020."

2. The survey respondents don’t represent you.

Even if you find a freelance industry survey with thousands of respondents, and you feel reasonably confident in the numbers, you should dig deeper into the methodology:

• Who are those freelancers?
• Why were they chosen?
• How were they found for the survey?
• What kinds of specialties do they have?
• How much experience do they have?

Just because those individuals are freelancers, it doesn’t mean their experience is in any way relevant to your own.

When these kinds of freelance industry surveys are conducted by a single blogger or website, they’ll naturally get respondents from their existing networks. And those readers, customers, subscribers, or whomever are already inherently segmented.

For example, a large race-to-the-bottom bidding site that shall remain nameless sometimes releases these freelance industry surveys.

They might have thousands of respondents. But numbers tied to earnings will always skew low in their data because they’re surveying people from their user base – on a site where there’s an inherent business interest in driving freelance rates down.

Basing your own rates on anything that company publishes wouldn’t just be a mistake. It would be tragic.

Ultimately, you’ll want to look at the balance between sample size and the types of people providing the data. You want to be represented by that data, but you also want a large enough sample size that you’re basing your decisions on actual trends and not isolated examples.

3. The underlying agenda or bias isn’t made clear.

You’d be hard-pressed to find a freelance industry survey that doesn’t have some underlying agenda.

And that’s OK! “Agenda” isn’t always a dirty word.

The issue is that the agenda or biases should be clear up front and not hidden from those reading the results.

Bias For or Against Certain Gig Types or Sources

Let’s say a blogger, like me, were to run a freelance industry survey focusing on writers. Would I have an underlying bias? Of course.

For instance, I make it no secret that I despise content mills. I don’t consider exploitation to be representative of professional freelance markets, and I wouldn’t seek out individuals working for those sites. I would also likely include a question to let me filter out data from content mill writers entirely.

The question would be, “is that disclosed in the methodology?”

It should be. Data skews upward when you remove those non-professional markets that wouldn’t be relevant to the kind of research I would conduct or publish for a reader base of freelance professionals.

That would also be true if someone conducted a survey in support of those sites and they chose to weed out respondents who have never worked for them.

Why might they do that? It would skew rate data down, making it look like market norms are much lower than they are. This is why you’ll see clients question true professional rates. They’re used to seeing these non-professional markets promoted and think they represent something they do not.

The Issue of Sensationalizing Statistics

Sometimes the agenda has nothing to do with the specific sample, rates, or making a certain business type look better.

Sometimes the agenda is simply about making broader claims than the data supports for linkbait or publicity’s sake.

This is why so many sites that run surveys of narrow groups (freelancers using bidding sites, bloggers, etc.) title their reports in a general way.

What’s going to get more pick-up in media outlets or on blogs – a survey that claims to represent the state of the entirety of freelancing or a survey that claims to represent only freelancers using a certain site or platform? The former.

Big claims equal more backlinks, better search visibility, more social media shares, more traffic, and ultimately more money if the company relies on freelance interest for its revenue. They're also dishonest.

4. The survey contains leading questions.

A few years back, I read a freelance industry survey’s report and found major flaws. When a member of their staff contacted me asking me to promote that report here, I let them know precisely what I thought of it. Ultimately that company hired me the following year to help improve the annual survey, and I've gone on to do similar work for other clients.

One of the issues I came across in their initial survey was their use of several leading questions.

Leading questions prevent you from getting valid data by nudging respondents toward specific answers.

For example, this is a leading question:

“How much do you expect to raise rates next year?”

This question makes the assumption that rates will go up. A better example might be:

“What do you expect your freelance rates to be next year?”

You could follow that up asking how much of a change they expect (up or down) if the respondent said they expect their rates to change.

Another example of a leading question might be tied to the issue of bias we already looked at.

For instance:

“Would you ever look for gigs on exploitational bidding sites?”

By including a negative reference, you would lead more respondents to answer that question in the negative whether it’s true or not.

5. The reported results don’t accurately reflect the data.

Perhaps the most common problem I see with freelance industry surveys isn’t a problem with the surveys themselves. It’s an issue with the published interpretation of the data.

I’ve long lost count of how many times I’ve read industry reports and thought to myself:

“WTF? That’s not at all what your data says!”

This usually comes down to someone “spinning” the statistics to support a preconceived notion. But sometimes it’s a simple case of a publisher not understanding their own data, or assumptions being made where they shouldn’t.

For example, here’s a quote from an older report:

“Most freelancers use their income to pay for expenses typically covered by an employer, which may include health insurance, 401k, paid time off, sick days, etc.”

Where do I even begin with that?

First, nothing in their data even begins to support this statement. It’s an assumption and nothing more.

Second, how many freelancers do you know who actually cover those things? They should. But in all my years freelancing, I’ve yet to meet many who do. Again, there’s no data from this survey to support the claim either.

Third, their very next statement contradicted this one by pointing out that only a little over 50% of respondents even had health insurance, and only 25% of those paid for it themselves (as opposed to government programs and being covered through a spouse or non-freelance additional job).

Yikes.

In an even more blatant example from that same survey, the person writing up the report added percentages wrong, leading a claim that fewer writers fell into a certain pay bracket than they actually did.

This survey was also a good example of sampling issues, skewing toward beginners rather than seeking out more experienced professionals for balance (people who were less likely to use this company’s services to find work).

In Summary

These common problems with freelance industry surveys aren’t always malicious. The person or company releasing the data isn’t necessarily looking to mislead you, even if they do.

Sometimes there are honest mistakes made in the research design phase. And sometimes they don’t realize their own data is problematic.

At other times, it’s clear the issue was with the writer putting the report itself together. Sloppy work on that last front is all too common.

This is why it’s important for you to look at survey reports with a critical eye.

Understand the sample and methodology. Figure out what biases or agendas might pay a role in how results are reported. And learn to trust your gut. If something sounds fishy, there might be a reason. Look to verify data with third party sources when possible.

Freelance industry surveys can be fun. The information can be interesting, and sometimes surprising. There’s no harm in checking out the reports. But remember, they often provide insight into much narrower sets of freelancers than they claim when publishing those results.

Make sure you are represented by that data before you use it to make any changes in your own freelance writing career.