NEWS

An update on SR6e DriveThruRPG ratings

  • 188 Replies
  • 40211 Views

skalchemist

  • *
  • Omae
  • ***
  • Posts: 258
« Reply #120 on: <10-01-19/1153:54> »
Shadowrun 6th is not mediocre, it is polarizing.

I submit that it is both at the same time.

If you look, the 5 Star ratings are "anonymous" ratings, not reviews.  It is almost as if the raters knew there weren't words to justify the 5 Star rating.
Likewise, many of the 1 Star reviews probably should be 2 Stars - if the reviewer could set aside their feelings about the company / product.
The 4 Star reviews I read mentioned, and then quickly glossed over, the editing and production issues.  Someone might be able to sell me on the idea that those should be 3.5 Stars, but DriveThuRPG won't allow that - so the reviewer rounded to 4.  Maybe for some of them...  Some should probably be brought down to 3.
Just to be clear, by "mediocre" I meant "something that gets lots of 2 and 3 star ratings", as the average rating would seem to indicate.  I meant nothing more than that.

skalchemist

  • *
  • Omae
  • ***
  • Posts: 258
« Reply #121 on: <10-01-19/1216:32> »
If you look, the 5 Star ratings are "anonymous" ratings, not reviews.  It is almost as if the raters knew there weren't words to justify the 5 Star rating.
Likewise, many of the 1 Star reviews probably should be 2 Stars - if the reviewer could set aside their feelings about the company / product.
The 4 Star reviews I read mentioned, and then quickly glossed over, the editing and production issues.  Someone might be able to sell me on the idea that those should be 3.5 Stars, but DriveThuRPG won't allow that - so the reviewer rounded to 4.  Maybe for some of them...  Some should probably be brought down to 3.
I saw two anonymous reviews, while the anonymous ratings are 12 for 4/5 and 8 for 1/2 stars. But sure, it's only the people that like the game that are remaining anonymous. Wonder if it has something to do with not wanting their information out there.
Speculating on why people rate how they rate seems like a waste of time to me.  Ratings are always subjective, even meta-subjective, in that they combine multi-dimension subjective assessments (content, editing, system, coolness, whatever) into a single subjective summary.

But if we are going to speculate, I think it is worth mentioning it takes an extra step to actually write a review after you have rated something in the system (you first click the rating button, but then you have to click the "write a review" link to leave a review).   Also, you have to actually have the PDF in your library to rate it (although you don't have to be a "purchaser").  So I find it pretty unlikely that, for some reason, people leaving 5 star ratings somehow "not wanting their information out there". 

It seems more likely to me that people who are really steamed/angry/unhappy are motivated to actually write a review to vent about it, whereas if you read the book and you thought "seems great!" you just click the button and move on.

Lormyr

  • *
  • Catalyst Demo Team
  • Omae
  • ***
  • Posts: 820
« Reply #122 on: <10-01-19/1222:34> »
We'll also never know the who's or why's behind anonymous votes. They could potentially be either extreme of the spectrum: Catalyst employees or freelances 5 star rating their own product, or someone who doesn't like the company or system creating multiple accounts to 1 star it. There's no way to know. The best we can do is just take the numbers and ratings at face value.
"TL:DR 6e's reduction of meaningful choices is akin to forcing everyone to wear training wheels. Now it's just becomes a bunch of toddlers riding around on tricycles they can't fall off of." - Adzling

Stainless Steel Devil Rat

  • *
  • Errata Coordinator
  • Prime Runner
  • *****
  • Posts: 4572
« Reply #123 on: <10-01-19/1241:27> »
I don't know, but I would expect CGL employees and freelancers aren't getting their pdfs thru DTRPG.  I suppose it's possible that self-interested parties could pay for a copy thru DTRPG just to leave a good review, but if you're going to go to that bother wouldn't you actually leave a good review rather than just toggling the 5 star?  Besides, even if it's occurring, surely it's below statistical significance given the rather low number of people who qualify as CGL employees and freelancers...

Speaking of statistical significance, as of this post there have been 34 reviews given.  I don't see how ANY statistical significance can be achieved with a sample size of only 34.  We're inherently talking about a combination of pure conjecture and anecdote, with probable sprinklings of opinion... 
RPG mechanics exist to give structure and consistency to the game world, true, but at the end of the day, you’re fighting dragons with algebra and random number generators.

0B

  • *
  • Guest
« Reply #124 on: <10-01-19/1259:53> »
It's less likely in general for positive raters to leave a review. If you find something wrong, it's a lot easier to articulate than if you enjoy something. Take the 5th edition book, for example. Of the 5/5 ratings, 50 are just ratings and 20 are reviews (29% of 5-star raters left a review). Of the 1/5 ratings, 1 is just a rating and 8 are reviews (89% of 1-star raters left a review).

Lormyr

  • *
  • Catalyst Demo Team
  • Omae
  • ***
  • Posts: 820
« Reply #125 on: <10-01-19/1314:43> »
I don't know, but I would expect CGL employees and freelancers aren't getting their pdfs thru DTRPG.  I suppose it's possible that self-interested parties could pay for a copy thru DTRPG just to leave a good review, but if you're going to go to that bother wouldn't you actually leave a good review rather than just toggling the 5 star?

That was basically my point. Both ends of the extremes I listed are highly unlikely.
"TL:DR 6e's reduction of meaningful choices is akin to forcing everyone to wear training wheels. Now it's just becomes a bunch of toddlers riding around on tricycles they can't fall off of." - Adzling

Hephaestus

  • *
  • Omae
  • ***
  • Posts: 254
  • "Milk Run" is a mighty weird way to spell TPK
« Reply #126 on: <10-01-19/1331:41> »
Speaking of statistical significance, as of this post there have been 34 reviews given.  I don't see how ANY statistical significance can be achieved with a sample size of only 34.  We're inherently talking about a combination of pure conjecture and anecdote, with probable sprinklings of opinion...

Weren't there only 42 credited play testers?  ::)

Jokes aside, as much as people want to dissect the intentions of the reviewers, or the statistical variance, or any number of other factors, the bulk data shows that SR6 is not doing as well as it should/could be, and needs CGL to make some serious updates (hopefully before the physical CRB drops) if they want those ratings to get any better.
« Last Edit: <10-01-19/1336:24> by Hephaestus »

Iron Serpent Prince

  • *
  • Omae
  • ***
  • Posts: 384
« Reply #127 on: <10-01-19/1407:23> »
Shadowrun 6th is not mediocre, it is polarizing.

I submit that it is both at the same time.

If you look, the 5 Star ratings are "anonymous" ratings, not reviews.  It is almost as if the raters knew there weren't words to justify the 5 Star rating.
Likewise, many of the 1 Star reviews probably should be 2 Stars - if the reviewer could set aside their feelings about the company / product.
The 4 Star reviews I read mentioned, and then quickly glossed over, the editing and production issues.  Someone might be able to sell me on the idea that those should be 3.5 Stars, but DriveThuRPG won't allow that - so the reviewer rounded to 4.  Maybe for some of them...  Some should probably be brought down to 3.
Just to be clear, by "mediocre" I meant "something that gets lots of 2 and 3 star ratings", as the average rating would seem to indicate.  I meant nothing more than that.

Just to be clear, my point is that a lot of the ratings should be 2 and 3 Star ratings.

If the raters had fairly rated, rather than tried to push their agenda, I expect the curve would "scrunch" down to the vast majority falling on 2 and 3, with a significant amount on 4.

Lormyr

  • *
  • Catalyst Demo Team
  • Omae
  • ***
  • Posts: 820
« Reply #128 on: <10-01-19/1425:39> »
Just to be clear, my point is that a lot of the ratings should be 2 and 3 Star ratings.

While I personally happen to agree with that assessment (I'd give it 3), you have as much right to tell someone else how they should think or feel about something as they do to tell you.
"TL:DR 6e's reduction of meaningful choices is akin to forcing everyone to wear training wheels. Now it's just becomes a bunch of toddlers riding around on tricycles they can't fall off of." - Adzling

topcat

  • *
  • Newb
  • *
  • Posts: 93
« Reply #129 on: <10-01-19/1432:21> »
CPRed doesn't provide the same thing as Shadowrun- not just in terms of magic/setting, but also lethality. The ease in which characters die is a major turn-off for me for CPRed, since I prefer long-lived campaigns. You can eliminate a lot of that in SR with minimal fudging and competent players, but a lot of the lethality in CPRed seems to come down to luck- maybe that's just my impression from the QSR. It was definitely fun for a one-shot, but it does not meet my own needs as a GM. That's not to say other people can't enjoy that level of lethality- it's also possible that the game isn't as lethal as it seemed, just that the players were inexperienced (Since they were new). But as-is, it seemed like I would have to fudge or house-rule a lot of CPRed to get to the right level of lethality, and that feels like cheating. I'm not morally opposed to fudging dice, but I want to avoid it as much as possible, and I don't want it obvious to the players when I do so.

Interestingly, CPR is supposed to be much less lethal than old CP2020.  Haven't played CPR, but I did play 2020 back in the day and character permanence felt on par with Call of Cthulhu.  LOL!  CPR is just one option, though.

You can also play prior editions of SR, which hit everything most players of SR6 re looking for without the same issues.  This seems to be the most popular option.  I run an SR5 game and play in an SR4 game.  No real reason to update either to SR6 given the current state of the game.  That's coming from someone who gets twitchy just thinking about retrogaming (the SR4 game is difficult on me, but the company at the table is great).

Leaving the friendly confines of CGL, Blades in the Dark is an amazing basis for a Shadowrun game.  Fate can give you everything you want, too.  Android lacks magic, but it's a really good cyberpunk setting and game - FFG pays attention to details.

We don't really need to stop at cyberpunk, either.  As our interest in SR6 plummeted, my friends didn't fall back on more SR5.  I just started running a D&D game and I'm pretty excited about that.  We have Star Wars, Fate, and Shadows of Esteren games that are a bit irregular, but they're getting more attention.  I'm thinking about a Battletech game.

Put another way: why invest time in a dumpster fire?  The core rules for SR6 are so bad that we don't want to put time into fixing it, let alone playing it, and that dumpster fire is going to burn for the next five years.  Why not walk away and come back in 2024?

skalchemist

  • *
  • Omae
  • ***
  • Posts: 258
« Reply #130 on: <10-01-19/1437:32> »
Speaking of statistical significance, as of this post there have been 34 reviews given.  I don't see how ANY statistical significance can be achieved with a sample size of only 34.  We're inherently talking about a combination of pure conjecture and anecdote, with probable sprinklings of opinion...
Statistical significance is a red herring here for a number of reasons, I think.

1) There is no hypothesis being tested. 
2) Rating means are purely descriptive, so the only kind of extra statistics that would really apply would be variance/standard deviation.
3) Given the nature of ratings (a bounded discrete ordinal variable), its not even clear how one would calculate appropriately the variance anyway.

I think what you may mean by the phrase "statistical significance" is really "evidence".  As in "34 ratings doesn't constitute much evidence to draw any conclusions about how well Shadowrun has been received, and therefore no conclusions should be drawn."   I'm not sure I agree with that statement, but it could be argued.   34 ratings are more ratings than many games at the same tier of sales have, such that I think the fact that its pattern of ratings is so very different from other similar games is good evidence of something different about Shadowrun compared to other games. 

What that difference may be is open to interpretation, of course.  But the fact of the difference seems pretty clear.

Stainless Steel Devil Rat

  • *
  • Errata Coordinator
  • Prime Runner
  • *****
  • Posts: 4572
« Reply #131 on: <10-01-19/1448:49> »
To bring in contemporary political jargon, there's certainly a "There there".

I was saying that discussing what the "There" actually is can't be done objectively at this point, based on the data set of DTRPG's reviews/ratings.

Sure, I'd be completely unsurprised if I were to gain insider strategic info and learn that 6we was rushed to be available at Gen Con.  I think most everyone presumes it, actually.  Is that necessarily a bad thing? It's simple reality that there's a certain point where you have to publish what's available now rather than continuing to wait.  I certainly lack both the experience and information to judge whether 6we "should have" baked longer.  What I DO have the experience with RPGs to feel comfortable saying is that no matter how long they baked it, it'd STILL need some post-release updating.  Everything always does- perfection is never achieved no matter how long you try.

So, what I care about is what CGL is doing about the CRB. And from what I can see they've really learned, if only what NOT to do, since 5e.  Certainly remains to be seen if we'll get a proper errata treatment published, but we're well ahead of the curve compared to 5e which took 2 years for a partial treatment.
« Last Edit: <10-01-19/1451:01> by Stainless Steel Devil Rat »
RPG mechanics exist to give structure and consistency to the game world, true, but at the end of the day, you’re fighting dragons with algebra and random number generators.

skalchemist

  • *
  • Omae
  • ***
  • Posts: 258
« Reply #132 on: <10-01-19/1511:40> »
If the raters had fairly rated, rather than tried to push their agenda, I expect the curve would "scrunch" down to the vast majority falling on 2 and 3, with a significant amount on 4.
Ratings are no more or less than the subjective opinion of the person doing the rating.  It will always combine the persons assessment of the multiple dimensions of the game itself (e.g. rules, editing, layout, concepts, etc.) as well as their own excitement level to share their opinion online.  If I rate something a 5, its a 5 to me, and my reasons are completely my own. The only way one consider it "unfair" would be if DTRPG had some kind of "rating criteria" they published and asked their users to follow that set of criteria.  I could not find such a set of criteria and don't think it exists. 

But honestly, I"m not really interested in discussing whether or not the ratings are "wrong" in any fashion.  You seem to think they are, I disagree with you because I don't think "wrong" can even be applied in a meaningful way to such rating systems.  Fair enough.  I'm happy to leave it at that.

penllawen

  • *
  • Omae
  • ***
  • Posts: 804
  • Let's go. In and out. Twenty minute milk run.
« Reply #133 on: <10-01-19/1514:55> »
34 ratings are more ratings than many games at the same tier of sales have, such that I think the fact that its pattern of ratings is so very different from other similar games is good evidence of something different about Shadowrun compared to other games.
Yes. This is my thinking, exactly. Also: it's simply not a good look to be one of only one or two titles in the DTRPG top 50ish with a rating below 4 stars. (Which it is, I think; I got up to 40-something when I looked through earlier and they were almost all 4+ stars.) Spend ten minutes clicking through the catalog and you quickly realise that anything below 4 stars is pretty rare. It takes a strong negative reaction to make that happen.

Sure, I'd be completely unsurprised if I were to gain insider strategic info and learn that 6we was rushed to be available at Gen Con.  I think most everyone presumes it, actually.  Is that necessarily a bad thing?
You never get a chance to make a second impression. I, for one, have written 6e off at this point; unless the next errata is mind-bogglingly massive, I'm fairly likely to skip the entire cycle, perhaps just picking up a subset of splatbooks. I doubt I am alone.

Quote
So, what I care about is what CGL is doing about the CRB. And from what I can see they've really learned, if only what NOT to do, since 5e.  Certainly remains to be seen if we'll get a proper errata treatment published, but we're well ahead of the curve compared to 5e which took 2 years for a partial treatment.
Perhaps. But consider this perspective: the 5e CRB launched at some quality bar, as did 6e. Put numbers on that, mentally. It's true that 5e then stuck at that level for a long time, whereas 6e received one round of errata quickly. But is 6e + hotfix errata in a better place today than 5e was the day it shipped? Is the quality bar any higher? I wasn't around in 2013, but I have read the first printing of the 5e CRB, and I think 6e is much worse. Even after the hotfix errata. So I'm not mollified by this reasoning, to be honest.

Iron Serpent Prince

  • *
  • Omae
  • ***
  • Posts: 384
« Reply #134 on: <10-01-19/1722:35> »
Just to be clear, my point is that a lot of the ratings should be 2 and 3 Star ratings.

While I personally happen to agree with that assessment (I'd give it 3), you have as much right to tell someone else how they should think or feel about something as they do to tell you.

Then I guess it is a damn good thing that I am, and never have, told anyone how they should feel about Shadowrun Sixth World.

Isn't it?  Hmmm?