NEW: Sign up to get freelance writing jobs in your inbox. SUBSCRIBE

Common Flaws with Freelance Industry Surveys

Read Time: 8 min

Common flaws with freelance industry surveys - AllFreelanceWriting.com

It’s that time of year again. No doubt, if you haven’t already, you’ll start to see freelance industry surveys. They might summarize 2020, or they might offer trend predictions for 2021. Industry surveys can be fun to review, but be careful about using them to make serious business decisions.

Let’s look at why industry surveys can be interesting. Then we’ll go into some common flaws with freelance industry surveys and why you should be careful about taking action based on their reported data.

The Appeal of Freelance Industry Surveys

Industry surveys can be appealing for a number of reasons. For example, there can be a voyeuristic interest in simply seeing what others in your profession are up to from a distance.

When it comes to freelance industry surveys in particular though, it can also be about:

  • inspiration;
  • validation

For example, if you’re relatively new in your freelance writing career, looking at the high end of freelance pay from surveys can give you something to aspire to.

On the less-helpful end of the spectrum, these kinds of surveys can validate bad choices by making them seem more common than they might actually be. They can also overemphasize negative aspects of freelancing, based on how they choose their sample, which might make things feel hopeless.

Either way, it’s human nature to compare ourselves to others, and industry surveys let us do that on a large scale.

This is neither inherently good nor bad. But it does make sense to evaluate what you're hoping to get out of the data before you dive into the results.

5 Common Flaws with Freelance Industry Surveys

Coming from a PR background, I’ve worked with a lot of data reporting over the years. That’s why I understand the importance of getting it right. Sadly, not everyone conducting and reporting on this kind of data puts that effort in, potentially leaving you with misleading information that can hurt your career if you rely on it.

Here are five common flaws I’ve seen in freelance industry surveys. They don’t mean you should entirely dismiss them. Rather, know what to look for so you don’t let that misinformation influence your business decisions.

1. The sample size is too small to be useful.

A frequent issue with surveys run by bloggers in particular is that their sample size is often too small.

A sample size needs to be representative of the larger group the report makes claims about based on the data. For example, let’s say we’re talking about a general survey on the state of freelancing.

There were around 57 million freelancers in the U.S. alone in 2019. While you might ideally want to see a 10% representative sample in a survey, that would be tough with these numbers. On the other end of the spectrum though, if you saw a sample size of 100 respondents, you should immediately know the results aren’t reliable. Even 500-1000 respondents isn’t nearly enough in an example like this. And even several thousand results could never give you an accurate picture.

Obviously, the overall number of freelance writers would be smaller than that. But most freelance writing surveys I’ve come across still have unacceptably small sample sizes.

That said, bigger isn’t always better. You can sometimes find better, more actionable, information from a smaller well-targeted survey than a broader, larger one.

For example, while Freelance Writing Pros (All Freelance Writing's sister site for more experienced freelance writers) has a much smaller email list and reader base, it's much better-targeted than All Freelance Writing is. So a 100-200 response survey would tell you more about that target group (freelance writers with at least 5 years of experience and an interest in more advanced marketing and business tactics) than even 1000 responses would from a similar survey conducted here with a much broader base.

In this case, the results could show you what writers at a similar place in their career, from similar specialty backgrounds, interested in a similar approach covered on that specific site have to say. But I would publish those results along the lines of "Here's what FWP readers earned in 2020" rather than something overly-broad and unsupported like the "State of Freelance Writing in 2020."

2. The survey respondents don’t represent you.

Even if you find a freelance industry survey with thousands of respondents, and you feel reasonably confident in the numbers, you should dig deeper into the methodology:

  • Who are those freelancers?
  • Why were they chosen?
  • How were they found for the survey?
  • What kinds of specialties do they have?
  • How much experience do they have?

Just because those individuals are freelancers, it doesn’t mean their experience is in any way relevant to your own.

When these kinds of freelance industry surveys are conducted by a single blogger or website, they’ll naturally get respondents from their existing networks. And those readers, customers, subscribers, or whomever are already inherently segmented.

For example, a large race-to-the-bottom bidding site that shall remain nameless sometimes releases these freelance industry surveys.

They might have thousands of respondents. But numbers tied to earnings will always skew low in their data because they’re surveying people from their user base – on a site where there’s an inherent business interest in driving freelance rates down.

Basing your own rates on anything that company publishes wouldn’t just be a mistake. It would be tragic.

Ultimately, you’ll want to look at the balance between sample size and the types of people providing the data. You want to be represented by that data, but you also want a large enough sample size that you’re basing your decisions on actual trends and not isolated examples.

3. The underlying agenda or bias isn’t made clear.

You’d be hard-pressed to find a freelance industry survey that doesn’t have some underlying agenda.

And that’s OK! “Agenda” isn’t always a dirty word.

The issue is that the agenda or biases should be clear up front and not hidden from those reading the results.

Bias For or Against Certain Gig Types or Sources

Let’s say a blogger, like me, were to run a freelance industry survey focusing on writers. Would I have an underlying bias? Of course.

For instance, I make it no secret that I despise content mills. I don’t consider exploitation to be representative of professional freelance markets, and I wouldn’t seek out individuals working for those sites. I would also likely include a question to let me filter out data from content mill writers entirely.

The question would be, “is that disclosed in the methodology?”

It should be. Data skews upward when you remove those non-professional markets that wouldn’t be relevant to the kind of research I would conduct or publish for a reader base of freelance professionals.

That would also be true if someone conducted a survey in support of those sites and they chose to weed out respondents who have never worked for them.

Why might they do that? It would skew rate data down, making it look like market norms are much lower than they are. This is why you’ll see clients question true professional rates. They’re used to seeing these non-professional markets promoted and think they represent something they do not.

The Issue of Sensationalizing Statistics

Sometimes the agenda has nothing to do with the specific sample, rates, or making a certain business type look better.

Sometimes the agenda is simply about making broader claims than the data supports for linkbait or publicity’s sake.

This is why so many sites that run surveys of narrow groups (freelancers using bidding sites, bloggers, etc.) title their reports in a general way.

What’s going to get more pick-up in media outlets or on blogs – a survey that claims to represent the state of the entirety of freelancing or a survey that claims to represent only freelancers using a certain site or platform? The former.

Big claims equal more backlinks, better search visibility, more social media shares, more traffic, and ultimately more money if the company relies on freelance interest for its revenue. They're also dishonest.

4. The survey contains leading questions.

A few years back, I read a freelance industry survey’s report and found major flaws. When a member of their staff contacted me asking me to promote that report here, I let them know precisely what I thought of it. Ultimately that company hired me the following year to help improve the annual survey, and I've gone on to do similar work for other clients.

One of the issues I came across in their initial survey was their use of several leading questions.

Leading questions prevent you from getting valid data by nudging respondents toward specific answers.

For example, this is a leading question:

“How much do you expect to raise rates next year?”

This question makes the assumption that rates will go up. A better example might be:

“What do you expect your freelance rates to be next year?”

You could follow that up asking how much of a change they expect (up or down) if the respondent said they expect their rates to change.

Another example of a leading question might be tied to the issue of bias we already looked at.

For instance:

“Would you ever look for gigs on exploitational bidding sites?”

By including a negative reference, you would lead more respondents to answer that question in the negative whether it’s true or not.

5. The reported results don’t accurately reflect the data.

Perhaps the most common problem I see with freelance industry surveys isn’t a problem with the surveys themselves. It’s an issue with the published interpretation of the data.

I’ve long lost count of how many times I’ve read industry reports and thought to myself:

“WTF? That’s not at all what your data says!”

This usually comes down to someone “spinning” the statistics to support a preconceived notion. But sometimes it’s a simple case of a publisher not understanding their own data, or assumptions being made where they shouldn’t.

For example, here’s a quote from an older report:

“Most freelancers use their income to pay for expenses typically covered by an employer, which may include health insurance, 401k, paid time off, sick days, etc.”

Where do I even begin with that?

First, nothing in their data even begins to support this statement. It’s an assumption and nothing more.

Second, how many freelancers do you know who actually cover those things? They should. But in all my years freelancing, I’ve yet to meet many who do. Again, there’s no data from this survey to support the claim either.

Third, their very next statement contradicted this one by pointing out that only a little over 50% of respondents even had health insurance, and only 25% of those paid for it themselves (as opposed to government programs and being covered through a spouse or non-freelance additional job).

Yikes.

In an even more blatant example from that same survey, the person writing up the report added percentages wrong, leading a claim that fewer writers fell into a certain pay bracket than they actually did.

This survey was also a good example of sampling issues, skewing toward beginners rather than seeking out more experienced professionals for balance (people who were less likely to use this company’s services to find work).

In Summary

These common problems with freelance industry surveys aren’t always malicious. The person or company releasing the data isn’t necessarily looking to mislead you, even if they do.

Sometimes there are honest mistakes made in the research design phase. And sometimes they don’t realize their own data is problematic.

At other times, it’s clear the issue was with the writer putting the report itself together. Sloppy work on that last front is all too common.

This is why it’s important for you to look at survey reports with a critical eye.

Understand the sample and methodology. Figure out what biases or agendas might pay a role in how results are reported. And learn to trust your gut. If something sounds fishy, there might be a reason. Look to verify data with third party sources when possible.

Freelance industry surveys can be fun. The information can be interesting, and sometimes surprising. There’s no harm in checking out the reports. But remember, they often provide insight into much narrower sets of freelancers than they claim when publishing those results.

Make sure you are represented by that data before you use it to make any changes in your own freelance writing career.

Latest posts by Jennifer Mattern (see all)

2 thoughts on “Common Flaws with Freelance Industry Surveys”

  1. I find surveys pretty much useless unless conducted by people who know what they’re doing. And from what I’ve seen, there aren’t many out there who do.

    This reminds me a little of that annual Top Blog Award that basically had people begging friends for nominations. It was there to drive traffic to the blogger’s site, nothing more.

    I don’t like freelancer surveys for these very reasons. I loathe freelance rate charts even more, same reasons. There is no separation of new, mid-career, or veteran writers, so the median rates are ridiculous. It tells me that there are some new writers charging pennies or nothing at all. But it sure as hell doesn’t tell me what pro writers are getting for their work.

    Exactly why they’re useless, and precisely why writers need to be making their own rates and not asking everyone on the planet what they should charge.

    Reply
    • I’m all for companies conducting, and publishing, original research. But like you said, far too little know what they’re doing. It’s often them going in knowing what they want to say and finding ways to skew data to support it, even when the data really doesn’t. But what really bugs me are the lazy folks who share it on their own sites and via social media. They amplify misinformation because they believe what some popular blogger tells them rather than learning to think and process that information for themselves. When a survey like this keeps getting amplified in a community, that’s when it becomes a bigger problem with wider readerships trusting the claims and making business decisions based on bad data.

      And I’m with you on top blog lists. I’ve been railing against these, largely at NakedPR.com, since at least 2007. They started out trying to use real data — big companies like Ad Age & Edelman. I still took it to both of them for their misinformation though. They screwed up royally because even they didn’t understand some of the data points they were using (like Alexa rankings, before Amazon hijacked that branding).

      I’m OK with subjective lists like a blogger naming their own favorites. But they should be disclosed as such and not generically labeled the “top” or “best” anything. Just be honest with your readers ffs.

      Ones that allow nominations and voting are basically a joke. This is where you see the circle-jerk style of promotion where newbie bloggers nominate each other within a group of friends. It has nothing to do with being the best sites. Every list like this is just a linkbait tool for the site publishing it. It gets people talking about the site when they beg readers to nominate or vote for them, link to them when they’re making those asks, and then link to the final results when the site sends them a badge or something. While not everything that could be classed as “linkbait” is inherently bad, this kind of ego-bait is beyond gross, and it’s a true disservice to readers to subject them to it.

      If you want to get on top lists, get on those subjective ones that are honest. And do it by earning a place there with your quality. At least those are genuine recommendations. Print publications’ lists are better, but not always. This site spent years on Writer’s Digest best sites lists for example. But even there I stopped mentioning it when I found out they were increasingly adding sites where they had business relationships with the owners, and they weren’t disclosing those relationships. Recommendations like that without disclosure violate FTC guidelines, and it really tarnished their whole effort.

      I’m with you on rate charts too. Partly for the same reasons, and partly because they’re often behind the times. It’s like when print magazine writers criticize blogging rates on a per-article or per-word basis because they have no understanding of how the work can differ, or that those lower-priced blog posts can pay far more hourly in many cases. Or when online writers see print rates per-word and suddenly think that’s going to translate into a very different market, especially when they’re still inexperienced. Rate charts skew low in some areas and skew high in others. Either way, rates always come down to what you need to earn plus any premium your experience / credentials / etc. justify. Rate charts can’t tell you that, and they can lead writers to pursue markets that will never pay what they should actually be earning. It’s another case of being interesting to look at maybe, but not being all that helpful in making responsible business decisions.

      So much terrible information out there all around. It’s sad really.

      Reply

Leave a Comment