This thread was... a read.
Warning: beginning of off-topic opinion piece.As an old geezer with far too much time on my hand who recently discovered that one of my favourite RPG settings had come out with a new edition, I can unequivocally say that the Drive-thru RPG review scores caused me, personally, to hold off on buying the 6th World core rulebook.
Not permanently, but the scores definitely merited some more research before investing time and money into a product. And so, I came here, looking for opinions from the fans. Spoiler alert: I had no idea what I was in for.
Some of the discussion in this thread has focused on the raw numbers, and then a whole lot of it was blatantly off-topic (pot, kettle, hello there!) with what, to a newcomer like me at least, seems like long-held grudges between a few individuals being aired in public.
While all of what follows, as well as my decision to not buy the book outright, is obviously anecdotal as far as the scientific method goes, I for one am here to learn more and hopefully play a game for fun, and not prove a scientific theory. I found it particularly interesting that several people jumped down the original posters' throat claiming that presenting numbers proved intent, instead of simply asking them what their intent was. Several of them were then, somewhat ironically, faced with the same treatment and responded about as well as one could expect when faced with what could easily be mistaken for personal attacks. Anyway, I digress...
Special mention to the few notable exceptions such as
this post by 0b, and
this post by dezmont, both of which were extremely on-topic and highly constructive to a discussion that drew my immediate interest (what do the numbers potentially mean!?).
Others... and there were unfortunately many such examples, not so much. I'm not going to go so far as to call out examples, but I think it's fair to say that most of the people in this thread could do with a bit of a break to re-evaluate what they spend their energy on, and how they come off to the new people who are just now joining the club.
The fact that people are declaring their intentions to leave the official forums, due to the way they've been treated, for a product I've loved since it first came out is something that I find... troubling. I had higher hopes when I first registered earlier today, I'll be honest. But, I'm not one to let first impressions deter me, and while this thread was a rough one I hope this is the exception rather than the rule,
I would like to thank AJCarrington in particular for
this response; a breath of fresh air, it was, and some much appreciated calm in the eye of the storm.
All right, end of off-topic rant. Disclaimer: all of the above are my personal opinions, as someone who is new to the current iteration(s) of the game and as someone who has not spent a lot of time in online communities, this one least of all. Feel free to take from the above what you want.On-topic, I do have some observations (also anecdotal, note that I'm not trying to prove a theorem, hypothesis, or theory here!) that I would like to add.
First of all, the review score numbers are just data. Anyone who has taken an introductory course in statistics knows that numbers without clear context can be used to represent pretty much whatever view you want. When presented with a rating system like the one used on DTRPG, I believe most people weight the total score as well as the number of reviews as an indication of user satisfaction. This is no different than when buying a product on Amazon; when presented with product options that have hundreds, if not thousands, of reviews and an average score of 4.5/5, I can safely say that I frequently just click "Buy" if the price is right on a product I'm looking for.
So, what do you do when presented with an option that seems less than satisfactory. Would you go to a restaurant with 3/5 stars on Yelp, or a garage with 2/5 stars on Google Maps, without first reading some of the reviews? I can't speak for you all, but I would want to know more before making a decision. And so, that is precisely what I did in this case.
As of the time of this writing, the Sixth World Core Rulebook has an average rating of 2.9 / 5 with 30 reviews. But only 13 of those reviews actually have comments. I find that it is much more common for people to rate something at either extremes of the scale if they don't have to leave comments, because these extremes are more often than not, at least in my personal experience, emotional responses.
If you only look at the scores of the 13 reviewers who also wrote comments, you're left with an average of 2.46 (5x 1*, 2x 2*, 1x 3*, and 4x 4*).
All right, that's not great, but these are still just numbers. Let's see if we can draw some common denominators from the various ratings. I pulled all 1181 words from the 1 and 2 star reviews into a word cloud generator, which created this:
https://i.imgur.com/Aqe9I3O.png. Here are the top 10 words, with number of occurrences preceding it:
20 edge
13 game
13 get
13 new
11 just
11 one
10 point
10 sam
9 smoke
9 can
Edge is definitely a hot topic, and it's not hard to imagine that some people associate "new" with "bad"; a lot of people fear change, after all. The rest are harder to draw much meaning from, and combined they highlight the problem with attempting to infer meaningful insight from a limited sample size. This is a topic that was brought up several times in this thread, and which I wholeheartedly agree with.
Now let's look at the 3 and 4 star reviews. With a total word count of 830, we can generate this map and again draw the top 10 words:
https://i.imgur.com/5M7hej6.png12 like
11 book
10 game
10 can
8 edition
8 rules
7 Shadowrun
7 system
7 much
7 lot
Like is an obvious emotional statement, and while we cannot infer that all instances of a word are positive (one could state "do not like" for example), I think it's fair to deduce that a more positive review score (i.e. above the average) combined with an expression of emotion is more likely to be favourable.
What does this all boil down to? For me, not much, to be honest. Finding objective truth in subjective material such as review scores is incredibly difficult, as there are too many factors at play. Take the above restaurant reviews I mentioned, and pick a local restaurant near you with a high review count. I can pretty much guarantee that you will find some low ratings because the person had to wait for a table or for the food, or because the reviewers impression of the servers was unfavourable. What does this tell me about the quality of the food at the restaurant? Nothing, really, because those reviews as a whole is more about the subjective "overall dining experience" than the objective "quality" of the food.
In other words, trusting review scores to tell us much of anything about a product is a gamble at best, especially when the total review count is small. There are just too many unknowns at play, and for this product these could be that the reviewer has previous experience with Catalyst products (it seems grammatical consistency is a big issue with many, for example), or that the reviewer is emotionally projecting ("I do not like this product, therefore it is bad!"). These issues are not invalid, per se, because they detail the reviewers personal experience with the product, but on the whole they are not particularly helpful either.
TL;DR
Using subjective source data to find objective truths is hard.I'd like to sign off by saying that I hope my rant hasn't come off as negative. I've been an avid fan of Shadowrun since 1st Edition, I played the game through most of 4th Edition, and then had to take a break for personal reasons. While I picked up many of the 5th Edition books for the fiction, I never did get around to playing it, so my familiarity with that system from a rules perspective is limited at best. My situation has improved, recently, and 6th Edition seemed like a good time to jump back in, but I think I will leave it alone for now and catch up on some novels while the errata team does their thing.
I look forward to engaging with you all in an earnest exchange of opinions that are, hopefully, free of some of the toxicity I've seen in this thread. I for one am a blank slate, and I hope that the release of 6th Edition brings many more like me back to the fold (or indeed, into the fold for perhaps the first time!). Ultimately, all I want for one of my favourite hobbies of all time is good things, and that is something I hope we can all agree on.