Thursday, 26 June 2014

Review Units Hurt the Audio Community & How We Can Help

Hi everyone, Lachlan here. As you hopefully know, I do product reviews. Many of the products I have reviewed on my channel have been provided by manufacturers as review units.  

Right now I want to make the following argument based on my experience so far as an audio reviewer, and talking to other audio reviewers:

  1. The current system whereby reviewers rely on manufacturers to provide review units produces bad incentives.
  2. These bad incentives unconsciously change the behaviour of reviewers and create an industry wide, systemic bias towards positive reviews.
  3. These biases hurt a community that needs open and transparent information to make decisions.
  4. This situation can only be resolved by a change in what the community is prepared to do for and accept from reviewers.

I am hoping this is a starting point for a community discussion about how we can improve things for everyone.



//////////////////////////////////////////////////////////////////////////////////////////////

HOW I CAME TO THINK THIS?

Now to start, I want to make something clear. In the past I have received review units. Some of these review units have been provided on a temporary loan basis, some of these units I have been provided to keep on a permanent basis. I have tried to be very transparent about review units. For some time I have maintained a public Google spreadsheet so that at any point my audience could look up any of my reviews and determine under what basis I acquired a product. I have declared whenever I have received a discount from manufacturers on products, or been invited to manufacturer events, or in other ways received some kind of benefit.

I have been proud of my ethics and my approach, and I think I have conducted things in the most professional manner possible considering that I am just a guy with a camera who started a Youtube channel from his bedroom.

But after doing many reviews, and speaking to many fellow reviewers, I have come to the opinion that the current model for doing reviews is flawed and it produces harmful biases. This is not because of any large degree of dishonesty among reviewers and manufacturers, but because the entire system currently produces a set of incentives that are not aligned with the interests of consumers.

Now when we start talking about bias, people can get very angry and defensive. And to be clear, the kind of bias I am talking about is not shady back alley deals where manufacturers are asking for positive reviews in exchange for free products. I think that kind of gross corruption tends to be very obvious, and I think most manufacturers and reviewers know that making such arrangements could potentially hurt them far more than it would benefit them. Every manufacturer that I have worked with has been open to honest feedback and criticism about their products, though I have personally encountered some instances where there has been shady behaviour behind closed doors.

The problems that I am going to detail are not due to any ill intent among reviewers or manufacturers. The problems that I am talking about arise because we are human beings. I will be the first to admit that I am a human being and as such I am influenced by biases and incentives, and any honest (or modest) reviewer will tell you the same thing - that is, after all, the entire point of seeking subjective opinion from a reviewer in the first place. And despite the availability of graphs and measurements, the value of the reviewer will always serve as a person who gives consistent and easy to understand reference points based on long experience. As the old adage goes, everyone hears things differently.

Problems arise when the information that the reviewer provides is influenced by factors unknown or unclear to the audience, and this influence is directly opposed to the audience’s interest in deciding a product’s merit.  Or simply put, problems arise when the interests of the reviewers are not aligned with the interests of the consumer. I think that there is a misalignment of interests in the industry and this is a result of poor incentives.

Human beings respond to incentives. Even Doctors who take the Hippocratic Oath to put their patients well being above all else have been found in studies to respond in subtle ways to external incentives besides optimal patient care. Just like any human being, they tend to become angry and defensive when this is suggested.

So in order to explain what incentives influence a reviewer, let’s break down how the review process works.

REQUESTING REVIEW UNITS

This is how I do my reviews. Individual reviewers may do things differently, but I imagine that this is pretty much the same for every reviewer in broad strokes:

  1. I am interested in a product. I am personally curious about it or my audience has expressed interest in a product.
  2. I send an email to a manufacturer requesting a review unit.
  3. The manufacturer decides to provide a review unit either on a loan basis or a permanent ‘to keep’ basis.
  4. I evaluate the product and ask the manufacturer any questions about the product that arise in the course of my evaluation
  5. I perform a review and return the unit or keep it.
  6. I keep in contact with the manufacturer to arrange future reviews. They may suggest other products to review and I may request other products.

Sometimes I receive unsolicited offers to send review units from manufacturers. This is rare, but because my Youtube channel is now visible enough, it happens every now and then.

Generally speaking, manufacturers do not tell reviewers what to say in reviews. They do not offer free products or payment in exchange for positive reviews. Sometimes they do not even ask that you review the product - they simply provide the review unit for your consideration and you are free to return it or keep it having done no review. A manufacturer that tells reviewers what to do and say is generally going to be a manufacturer that is not confident in their products, has inexperienced or unethical PR and most probably makes poor products anyway. I would not work with such manufacturers and I have not really encountered any like that.

But to say the manufacturer does not provide some kind of incentive to the reviewer with a review unit is false, because for a reviewer the review unit itself IS the incentive.

//////////////////////////////////////////////////////////////////////////////////////////////

REVIEWER’S INCENTIVES

Why is a review unit a big benefit for a reviewer? Because a review unit means the reviewer can make new content, and new content attracts eyeballs, and new eyeballs mean some kind of payoff for the reviewer. This payoff could be in the form of ad revenue monetisation, like my Youtube channel. Or  it could just be in the form of increased recognition from the community, which is why I make videos, since I like talking to people and I like to feel like I’m making a difference. This is something people should not forget, since emotional validation is a very strong payoff in and of itself.

This means that the more popular a product or a brand is, the more incentivised the reviewer is to get a review unit. There is an especially strong incentive to get a review unit early, since being the first on the scene means even more eyeballs.

Review units benefit reviewers because reviewers need review units to survive. This is a voracious market where the consumer always wants something new, something more, something now, even products from even a few years ago still sound perfectly good. The VSonic GR07 still sounds excellent no matter how many balanced armatures people try to cram into an earphone, as if they were running a razor blade company. While there are plenty of examples of products that stand the test of time, by and large people simply stop talking about old things because, you know, that’s sort of boring.

In this market reviewers are like sharks; they need to keep swimming or else they drown.

This also means that audio companies can be very selective about who they provide review units to. It’s different from very large and established low margin markets like computers or smartphones, where competition is extremely fierce and innovation is very fast, and where there are entrenched players like Engadget and The Verge have tremendous clout and editorial independence to get what they want. In audio reviews, particularly in high end, high margin audio, there are only a few big name reviewers. Personally, as a bit of a small fish, I’ve found trying to secure review units exhausting. Most of my emails don’t get a reply, and most that do are refusals.

Once you do secure a review unit from a manufacturer, another incentive kicks in: the incentive to preserve the relationship for future review units.

Again, let me repeat: manufacturers generally do not tell you to produce positive reviews, they do not (if they are a good manufacturer) criticise you for making negative reviews. But at any time they can always stop responding to emails and turn off the tap, or they can concentrate their efforts on other reviewers who they have better relationships with.

There is nothing sinister about this. Manufacturers are businesses, they have the right to pick and choose who they provide review units to, and they will make their decisions based on what benefits them. No one expects them to give products to a reviewer who they know is going to give them a negative review, because that would be silly and get someone fired.

So let me break down some of these incentives into a handy table.

Here are the incentives FOR and AGAINST a reviewer saying positive things about a product.

FOR saying positive things
AGAINST saying positive things
  • You think a product is good and merits recognition.
  • It feels good to say positive things.
  • People who own the product and like it will also like you. A large number of reviews are read by people who are not necessarily looking for information, but rather affirmation about their purchase. I have learnt over time that people (myself included) like reviews as a form of informational entertainment.
  • People in general like to hear positive things, so they will like content that is positive. This has an effect on the popularity of your reviews.
  • The manufacturers like you and will give you more review units.
  • The manufacturer may promote your review through their channels and give you more exposure.
  • Review units help the future success of your publication.
  • If the product is new, your positive review has a strong influence on community uptake. A snowball / flavour of the month effect occurs and as the reviewer you again stand to benefit.
  • If you do it too much or it’s unwarranted, people will think you are a shill and stop trusting you.

As you can see, there are MANY more incentives to say positive things about a product, but only ONE thing is something that the consumer actually cares about: "a product is good and merits recognition"

Now let’s look at the incentives FOR and AGAINST a reviewer saying negative things about a product.

FOR saying negative things
AGAINST saying negative things
  • You think a product is bad and this needs to be mentioned.
  • People will respect your honest opinion.
  • The manufacturer may use this feedback to improve the product.
  • It feels bad to say negative things.
  • If you do it too much or it’s unwarranted, people will think you are a troll and stop trusting you.
  • You are exposed to criticism from people who like the product and disagree.
  • You will make people disappointed or angry.
  • You risk jeopardising the relationship with the manufacturer.
  • The manufacturer may not promote your review.
  • You may stop receiving review units.
  • If the product is new, your negative review may kill any initial enthusiasm and doom the product.

Again, as you can see, there are MANY more incentives to NOT say negative things about a product, but most of these reasons do not particularly benefit a consumer that actually wants information about a product.

Of course, there are other incentives at play. I am not a psychologist or a behavioural economist. But because I run a Youtube channel I do have access to more data and information about how things work than your average punter, and this is what I have observed over time.

//////////////////////////////////////////////////////////////////////////////////////////////

SYSTEMIC BIAS

So, I’ve broken down some of the incentives that reviewers have. Reviewers have various incentives to produce negative reviews, and various incentives NOT to produce negative ones. But what kind of change in behaviours do these incentives produce for reviewers?

Based on my experience and my discussion with other reviewers, these incentives do not consciously influence reviewers to say positive things. Most reviewers would consider this unethical - but again, like Doctors, they are subject to unconscious bias.

Incentives DO produce certain subtle behaviours among reviewers.

  • Reviewers may moderate the negative things they may have to say about the product, or signpost the negativity by pointing out it is just a personal opinion.
  • Reviewers may avoid requesting review units for products they know or suspect they will not like.
  • Reviewers may avoid reviewing products that they have received and do not like.
  • Reviewers may avoid saying any other negative things about a company not related to the specific review unit.

What this means is that there is a subtle, industry wide bias towards saying positive things about a product and a brand. Reviewers who tend to say more positive things (within reason) are rewarded with increased exposure because they have access to more things to review. There is strong incentive to moderate negative opinions, but no real incentive to moderate positive ones.

Reviewers might not say this as self consciously as I have. Put in another way, it really isn’t interesting or fun to make negative reviews of products. It’s like being a big party pooper. But the net effect is that reviews tend to be more positive, and many bad products simply never get reviewed or heard of.

Looking at my list of product reviews, out of 62 headphone reviews I have given what I think are 28 positive reviews, 20 neutral / maybe reviews, and 14 negative reviews. Most of them were not review units, I’ve given negative reviews of review units, and I stand by my reviews (especially because I use such broad language, which is another bias in itself). But at the same time I have to question myself. Sometimes people make the joke about my channel that it should be called “Lachlan Likes Everything”. This isn’t true, and I always do point out that I have made negative reviews, but I have to ask if I am subject to a bias which modifies everything I do slightly?

I have the utmost respect for Tyll Hertsens of Innerfidelity as a fair and objective reviewer. I do not mean to single him out as an example of bad practice, but I wanted to point out that even a well respected figure is subject to human biases. This can be observed looking through the products that Tyll measures versus what he chooses to actually write about. He tends only to write about products that he likes, and avoids strongly negative language about any product which he doesn’t. This is of course natural - again there is very little incentive to write a negative review. It’s just not fun.

But Tyll has stated in a Reddit AMA:

"I try not to do negative reviews, I think it's much better to focus on finding new headphones that sound good."

tyll.jpg

I am not suggesting that Tyll is deliberately hiding anything or that he is dishonest. I am suggesting that he is responding to the same set of incentives all reviewers have when they are deciding what they will spend time and energy on to review. There is simply not enough time or resources to produce negative reviews when there are so many good reasons to produce positive ones.

Obviously this is problematic. If positive reviews are encouraged and negative reviews are suppressed, this distorts the perception of the market and allows manufacturers to escape scrutiny of bad reviews while enjoying the brand enhancement of good ones. Does it really serve the community when reviewers are not really incentivised to produce negative reviews?

//////////////////////////////////////////////////////////////////////////////////////////////

RETAIL VS REVIEW

Another issue with review units is that the experience of receiving a review unit is entirely different from the experience of buying something in a retail channel. When you buy something in retail, you are forced (if you are diligent) to do market research, to determine the street price, to select the best vendor, to buy the product and observe the pre-sale and post-sale process, and to evaluate the long term reliability and utility of a product. You are forced to compare among many other options, and learn from the tips and advice of other people who have also gone through the usual retail route.

All of this is useful information to the consumer, but the reviewer has to do none of these things when a review unit turns up at the doorstep, and especially not when the review unit must be returned within a typically 30 day loan period.

Again, this introduces a bias where companies have less incentive to produce good service, since reviewers are generally not even looking at that part of the equation.

//////////////////////////////////////////////////////////////////////////////////////////////

HURTING THE COMMUNITY

I would argue that it benefits the community when reviewers are transparent about their biases and they produce more information for the community, both positive and negative. I am a strong advocate for free markets, but free markets only work efficiently when people have access to more choices and good information. It takes effort and vigilance to provide that access. I feel that there is currently a bias in the audio industry towards producing positive reviews while disincentivising negative ones, and I think that in the end this hurts the community.

When most reviews tend to be positive, I think this also hurts the manufacturers. Companies do not receive negative feedback. Larger companies look better because they have a larger product portfolio, and thus tend to receive more positive reviews than negative ones. Smaller companies find it difficult to enter the market against the entrenched outfits that have the PR and the budget to give out review units to whomever they choose.

Many biases cannot be removed. They are human nature. As reviewers we have to sheepishly or proudly own up to our biases and let the audience decide whether or not we should be trusted.

In most ways, reviewers do not have any particular expertise or faculty beyond the average consumer. Actually, I would argue that in most ways, a single reviewer is less useful than looking at the aggregate ratings for a product from many “average consumer” reviews on Amazon.com.

What successful audio reviewers do have over the average consumer is the passion or inclination (or insanity) to try many many different types of gear, the ability to know what to look for based on this broad experience and the ability to articulate it to their audiences. The reviewers we tend to respect are the ones who have a broad experience with many different types of gear, and thus can act as a point of reference with which a consumer can triangulate their own likelihood of enjoying a given product.

That’s all there is to it. Good reviewers are good presenters who have managed to try a lot of stuff.

But the problem is, trying a lot of stuff is expensive.

So this brings us back to our original problem. Review units are great for reviewers, because they are FREE. Manufacturers benefit from sending out review units, because for them it is essentially free. The only people who do not benefit from this arrangement are consumers, who as I have said, are largely unaware or accepting of the incentives that are operating in the industry.

But we would not accept this model from journalists. We do not accept this model from politicians.

So like so many other problems, the issue here boils down to the basic problem of money. If review units produce bad incentives, and we want to remove those incentives, what is the alternative model?

//////////////////////////////////////////////////////////////////////////////////////////////

ALTERNATIVES TO REVIEW UNITS

There are many alternative models for providing reviewers with the resources they need to get the products they need to compare. Here are a few.

1. Be rich. A rich reviewer can afford to buy whatever they like in retail, without having to go asking around for review units. Of course, this produces it’s own set of biases when the entire community of audio reviewers has to be rich, so it’s obviously a silly solution.

2. Advertising. This is the monetisation model that has been in place for decades, whether in the form of advertisements in audio magazines or streaming ads on Youtube. This produces it’s own set of problems.

For one thing, advertising revenue is very low unless you are a very large entity. Even then most places still have to rely on review units, although places like The Verge can have very strict editorial guidelines about how review units are treated. This seems to work well for them because they can review a wide variety of products. It’s questionable how much advertising revenue is really available in the audio enthusiast market.

The second issue is that there are of obvious ethical issues when the advertisements you run on your publication are for companies whose products you are reviewing. Many reviewers therefore get sponsorship from companies that are more distanced from manufacturers, or in entirely different markets. On Youtube for instance, I have no control over what advertisements get run or when. Hosting Amazon affiliate links is another example of this kind of monetisation. There are still ethical implications for this, since you are still incentivised to make reviewers buy a product. It just might not be the product you are reviewing.

The third issue is that in this day and age, many people are using AdBlock anyway. They don’t even see the ads, even though they enjoy the content that is being made possible by the ad revenue. I think this is really sad given that I do put effort into my reviews, but then again people are bombarded with advertising almost every second of their lives. I don’t blame them for wanting to get away from it all.

3. Borrow stuff. This is actually a great solution that reflects good community spirit, when reviewers like ljokerl regularly receive things from other members to have a look at. This would be a great way of doing things, but sadly it isn’t a very consistent or sustainable way to conduct reviews. Another issue is that borrowing products to review again isolates the reviewer from the retail experience. It’s one of the better solutions, but it’s not practical.

//////////////////////////////////////////////////////////////////////////////////////////////

4. CROWDFUNDING

Crowdfunding is just fundraising. Fundraising has existed for ages. The only difference is that in recent years we have developed the online infrastructure necessary to send very small payments directly to other people in a secure way electronically. We can see the crowdfunding model starting to emerge in many areas, and most people recognise it as a fair and equitable way of directly supporting things that you love and want to see progress or continue.

This is the solution that I am proposing. I should state at this point that I am running a crowdfunding campaign to support my Youtube channel, so if you want you can discount this entire piece of writing as one giant pitch to get your money. In some ways it is!

But everything I have written here is the thoughts that have led me to launch my campaign, because I want to move the audio community to a point where they recognise that the most fair and objective way for reviews to be conducted is for the people who benefit from them to be the ones who support them.

In this piece of writing I am trying to convince the community that they should be not only willing, but proud to support their favourite reviewers so that they can buy the products in retail like any other consumer.

We are not talking large amounts. Most reviewers do not make much money at all from their reviews, and it is always possible for reviewers in a crowdsourced model to return or resell the units back to their sponsors, which means that beyond an initial investment only a modest ongoing outlay may be required.

We tip staff in restaurants for a night’s service - why should we not be willing to tip people who are providing a service to the community time and time again with their reviews?

I started my campaign on Patreon and I think that is the best model because it is a crowdfunding website which supports small ongoing contributions to any person who produces any sort of content on a regular basis. Consistency and stable revenue is important if you want to keep the shark swimming.  

But if ljokerl had a Kickstarter, I would pledge. If Tyll Hertsens had an Indiegogo, I would pledge. If ClieOS or Shigzeo or Tomscy2000 or any of the wonderful members of this community simply gave others a chance to support them, I would do it because I owe them for teaching me so much, and I want to see a situation where the people who spend so much time and effort giving to the community are paid not by external parties, but by the very people that enjoy their content.

At the same time I would like to see manufacturers being rewarded for making good products rather than organising good PR.

Crowdfunding is a better model for reviewers because it forces reviewers to be ultimately responsible to their audience, and not reliant on manufacturers for continued success. Crowdfunding means that instead of chasing after each new hit, reviewers can be rewarded for producing consistent, high quality reviews on an ongoing basis.

I want to close the loop in the community and replace the incentives with this:

FOR being a good reviewer
AGAINST being a good reviewer
  • People think you are good and merit recognition.
  • These people support your reviews and you want to give something back to them.

I am looking forward to your comments. I know some of you may think that what I am saying I am saying for selfish reasons. And of course, I do stand to benefit, like any reviewer, if crowdfunding is a successful model. But I think you should forgive this one particular bias.

For my part, I am going to finish reviewing the few review units I have received and then stop accepting or requesting new review units. I am going to rely on advertising revenue from my Youtube channel and hopefully one day rely solely on crowdfunding revenue from my audience.

I also know that some of you may think this is naive, since it is unproven whether or not the community is willing or able to support reviewers.

But even if my Patreon campaign fails to generate substantial pledges, I would like to see the community as a whole embrace crowdfunding as a way to keep all aspects of this community alive, from the reviewers to the forums to the meets.

I’m not saying it can happen overnight, but I do think the discussion needs to start somewhere.


Who’s with me?