NEWS

An update on SR6e DriveThruRPG ratings

  • 188 Replies
  • 40210 Views

penllawen

  • *
  • Omae
  • ***
  • Posts: 804
  • Let's go. In and out. Twenty minute milk run.
« Reply #15 on: <09-12-19/1206:16> »
Actually yes I can, people who only buy from the CGL store may be more positively biased, as well as a difference between people who only buy online when compared to people who only buy thru FLGS. There also very significant bias  when you don't know the reasons for the reviews (good or bad) as well as a major deviation factor due to ratio of responses to sales. Not saying it wrong, but also says it does not represent the actual market either ... just missing too much data. The only thing this particular rating shows is that it is selling well on DTRPG.
You can make all these arguments about the other games on the list, most of which are available digitally from other stores and I think all of which are available physically via FLGS. Yet we don't see a large number of negative reviews for any of the others. Why is Shadowrun 6e different?

Edit -- and why didn't these factors affect SR5, which stands at 4.1/5?
« Last Edit: <09-12-19/1208:53> by penllawen »

penllawen

  • *
  • Omae
  • ***
  • Posts: 804
  • Let's go. In and out. Twenty minute milk run.
« Reply #16 on: <09-12-19/1247:52> »
Another comparison point; Cyberpunk V3 - which I think could be fairly described as "legendarily hated" - stands at 2.4/5.

FastJack

  • *
  • Administrator
  • Prime Runner
  • *****
  • Posts: 6367
  • Kids these days...
« Reply #17 on: <09-12-19/1325:33> »
Incorrect. The PDF version was always $15, the $30 comparative price shown on DTRPG is for the physical release. Citation: many places, but for example, IGN: "The digital edition, available as of August 1, 2019, will retail for $15.00 USD and the physical edition (available exclusively at GenCon 2019 initially then at your local hobby shop) will cost $29.99."
You're correct, the PDF price has always been $15.

dezmont

  • *
  • Chummer
  • **
  • Posts: 190
« Reply #18 on: <09-12-19/1330:16> »
I would like to note that 'everyone here liked it,' while not invalid, is not a great metric.

Reddit alone gets 10 times the uniques at least to Shadowrun Tabletop based on my brute force attempts to gauge its traffic using public information vs privileged information I have as a reddit mod. Some further math (backed by both my studies and one of my professors, as this was for a project) showed that it was likely far lower because many of Shadowruntabletop's uniques are likely people hitting the front page looking for Shadowrun information off a google search, as the site's bounce rate is exceptionally high.

That doesn't mean the opinion of people here is invalid, but it does seem to represent a rather minority opinion, backed up by the fact that shadowruntabletop has never been the first choice of the community to the point alternative competitive sites have been created and trend towards being more succesful than the official forums. One can attribute this in part to the fact people will often use facebook and reddit more casually and frequently than going to a specialist site, but based on some information I got from one of the people with access to the analytics for Runner's Union there are actually still more people actively checking their page than Tabletop.

This also makes sense because while the website has made massive improvements its still poorly optimized and laid out, which severely harms its growth and effectiveness as a community and marketing tool for CGL, and hurts their ability to cultivate the community they want under an umbrella they control.

Basically, while its valid to go 'well I like it' its critical to remember that the majority of fans seem to not just not use these forums, but actively seek alternative forums out. It is very likely these ratings are artificially lower than they should be due to a lack of consumer goodwill, especially when it comes to Streetpedia which likely just ate some hate for 6e, but that is... still kinda a major problem for a company that is making a product not just for the general RPG consumer base but for a specific niche fanbase that already has plenty of competition from its own past products. This problem is an ongoing thing CGL needs to figure out, rather than something to shrug at and go 'internet is toxic I guess.' Especially as its been shown that its peers can struggle as well, or thrive. Paizio despite having a loyal and supportive fanbase is struggling with Pathfinder 2e, but the myth of 'edition update hatered' is shown to not be a universal truth with lines like GURPS, D&D, and Hero System all managing to release new editions with minimal fuss or even active support. Sometimes this can be attributed to moving back from something generally disliked (D&D) but other times its just because edition updates are trusted and the company has good will with its fan base (Hero System/Champions, GURPS).

The internet isn't some magic bullet that makes edition changes always rough. The RPG industry is very young and the quality of their PR and community management varies EXTREMELY wildly and its going to be a long term problem. One of the most successful RPG companies, White Wolf, basically doesn't exist anymore as a direct consiquence of losing its fanbase's goodwill entirely. Compounding issues is a lot of people who are managing the brand's image aren't from a 'wageslave' background, which is generally good because I like my RPG companies run by RPG nerds and not businessmen and women, but as these brands become more 'brand-y' its showing more and more some fundementals of PR are being consistently broken due to inexperience or a lack of care.

I expect the problems with book ratings (That regardless of how good or bad they are that they reflect more community goodwill rather than actual quality, look at D&D 4e and how much of a meltdown that caused despite the book being a standard to good RPG in the end!) to get worse, not better, over time, as PR problems do not trend towards 'self correction.' And this is also an issue because if you evaluate SR's presence on online sales its swinging crazy above its 'weight class' vs paper sales, which indicates a massive amount of their business is actually being done online.

So like, even if you don't draw the conclusion 'the book is bad' from the ratings (And, despite not liking the book, I agree you shouldn't, the review rating is very divorced from that and you can get a better idea of that from in depth actual written reviews and seeing how many people hit the same notes) this is still a pretty important piece of data any canny company should take on board and incorporate into their future plans.

Lormyr

  • *
  • Catalyst Demo Team
  • Omae
  • ***
  • Posts: 820
« Reply #19 on: <09-12-19/1333:11> »
When it comes to needing folks to enjoy your product there is something to be said for either putting too much stock into their opinions or disregarding them. I have not seen many official reviews on 6e (not really the sort of thing I go out of my way to look for), but I have read through a ton of forum feedback from fans on numerous sites.

The substantial majority share similar praise and criticism points. That is always worth consideration.

"TL:DR 6e's reduction of meaningful choices is akin to forcing everyone to wear training wheels. Now it's just becomes a bunch of toddlers riding around on tricycles they can't fall off of." - Adzling

penllawen

  • *
  • Omae
  • ***
  • Posts: 804
  • Let's go. In and out. Twenty minute milk run.
« Reply #20 on: <09-12-19/1338:30> »
So like, even if you don't draw the conclusion 'the book is bad' from the ratings (And, despite not liking the book, I agree you shouldn't, the review rating is very divorced from that and you can get a better idea of that from in depth actual written reviews and seeing how many people hit the same notes) this is still a pretty important piece of data any canny company should take on board and incorporate into their future plans.
That is exactly what I think, including the important caveat about reading too much into this data point.

FastJack

  • *
  • Administrator
  • Prime Runner
  • *****
  • Posts: 6367
  • Kids these days...
« Reply #21 on: <09-12-19/1338:46> »
A few weeks ago, FastJack said:
Quote
Was my wording off, or were a lot of people surprised that sixth edition was more popular than they thought it would be? I suspect a lot of people are feeling shocked that Sixth World is the number one download on DriveThruRPG ever since it came out.

This made me curious so I compiled a list of the top 10 on DTRPG at the time, complete with the star rating that is displayed next to each item. This wasn't a good showing for SR6, with a rating at the time of 3.4/5 -- the next lowest in the top 10 was 4.8/5.

However, that was based on a scant four days of sales, so I made a note to revisit it when a few weeks had passed. I've now done this, and here it is. As you can see, SR6e has fallen from the #1 spot and is receiving a bit of a kicking from DTRPG reviewers.

   Position      Title      Score      Number of ratings      Release date   
   1      Cyberpunk Red Jumpstart Kit      4.8/5      42      August 01, 2019   
   2      Shadowrun, Sixth World Core Rulebook      2.8/5      19      August 26, 2019   
   3      Eclipse Phase Second Edition      5/5      13      August 09, 2019   
   4      Legend of the Five Rings: Courts of Stone      5/5      2      August 27, 2019   
   5      The Short Games Digest: Volume 2      5/5      5      September 09, 2019   
   6      Bayt al Azif #2: A magazine for Cthulhu Mythos roleplaying games      n/a      n/a      September 09, 2019   
   7      Stars Without Number: Revised Edition      4.9/5      154      December 29, 2017   
   8      Occult Philosophy      5/5      24      July 29, 2019   
   9      Rangers of Shadow Deep: Ghost Stone      4.3/5      3      August 02, 2019   
   10      Esper Genesis 5E Threats Database      5/5      6      August 15, 2019   
   11      Rangers of Shadow Deep: A Tabletop Adventure Game      4.8/5      62      November 02, 2018   
   12      Cyberpunk 2.0.2.0. The Second Edition, Version 2.01      4.5/5      79      March 09, 2014   
   13      Lords and Lands: a Witcher TRPG Expansion      3.3/5      12      August 01, 2019   
   14      Apocalypse World: Burned Over Hackbook      5/5      3      August 01, 2019   
   15      Star Trek Adventures: Alpha Quadrant Source Book      4.3/5      3      July 25, 2019   




Since the list was published, A Call to Arms: Star Fleet, Book Two, Early Bird Edition has jumped up to spot number 11, pushing the others further down the list.

Also, by the logic you're presenting here, Shadowrun: Anarchy with it's 4.5/5 rating is a better version of Shadowrun.

penllawen

  • *
  • Omae
  • ***
  • Posts: 804
  • Let's go. In and out. Twenty minute milk run.
« Reply #22 on: <09-12-19/1339:50> »
Also, by the logic you're presenting here, Shadowrun: Anarchy with it's 4.5/5 rating is a better version of Shadowrun.
I am not presenting any logic. I am presenting data.

FastJack

  • *
  • Administrator
  • Prime Runner
  • *****
  • Posts: 6367
  • Kids these days...
« Reply #23 on: <09-12-19/1350:47> »
Also, by the logic you're presenting here, Shadowrun: Anarchy with it's 4.5/5 rating is a better version of Shadowrun.
I am not presenting any logic. I am presenting data.
What is the purpose of sharing the data? When you release data, you are trying to prove a theory, correct? In this case, your theory is that Shadowrun 6E isn't any good and people should stop purchasing it so that Catalyst Game Labs stops production and sells off the IP to people that "know how to make the game". Correct?

There are rules in the ToS about starting Flame Wars. Rules that I usually only enforce when things spiral way out of control. We've been down the road of people not liking this edition ever since they announced they were making a new edition. Many have received warnings, some have received bans. All because people are "presenting their data" to back up their opinion. My goal here as moderator isn't to shut down discussion of the game, but to keep the discussion from turning into "IS SO"/"IS NOT" arguments. Which is where this is leading since the people that like the changes and the people that don't aren't changing any minds on either side.

If the purpose is to convince CGL to do something different, things are already in the works. You're now asking them to halt production on at least two announced books, probably a half-dozen unannounced, so they can go back to the drawing board and rewrite the core rulebook for release sometime next year? That's a lot of money that's already been paid out that they would have to write off.

Hephaestus

  • *
  • Omae
  • ***
  • Posts: 254
  • "Milk Run" is a mighty weird way to spell TPK
« Reply #24 on: <09-12-19/1401:02> »
If the purpose is to convince CGL to do something different, things are already in the works. You're now asking them to halt production on at least two announced books, probably a half-dozen unannounced, so they can go back to the drawing board and rewrite the core rulebook for release sometime next year? That's a lot of money that's already been paid out that they would have to write off.

To be fair, if the reception/sales for the core book are falling short of expectations, then taking time to revise/errata/fix the issues with it would probably do more to help future books. Both from a sales/marketing standpoint (righting the ship straight out of the harbor) and also from a game mechanics standpoint (so future books have more solid ground to stand on).

Its a matter of whether CGL thinks they will do better by continuing ahead with what they have, or making the changes they want/need to make now instead of later. In my experience with manufacturing, while the latter option seems like a giant pain in the ass, it almost always pans out better in the long run.

FastJack

  • *
  • Administrator
  • Prime Runner
  • *****
  • Posts: 6367
  • Kids these days...
« Reply #25 on: <09-12-19/1403:06> »
If the purpose is to convince CGL to do something different, things are already in the works. You're now asking them to halt production on at least two announced books, probably a half-dozen unannounced, so they can go back to the drawing board and rewrite the core rulebook for release sometime next year? That's a lot of money that's already been paid out that they would have to write off.

To be fair, if the reception/sales for the core book are falling short of expectations, then taking time to revise/errata/fix the issues with it would probably do more to help future books. Both from a sales/marketing standpoint (righting the ship straight out of the harbor) and also from a game mechanics standpoint (so future books have more solid ground to stand on).

Its a matter of whether CGL thinks they will do better by continuing ahead with what they have, or making the changes they want/need to make now instead of later. In my experience with manufacturing, while the latter option seems like a giant pain in the ass, it almost always pans out better in the long run.
I agree, we should be focusing on righting the ship and getting this corrected. I know the Errata team is hard at work on these issues right now. What I don't appreciate is comments telling CGL to stop production and sell off the IP.

penllawen

  • *
  • Omae
  • ***
  • Posts: 804
  • Let's go. In and out. Twenty minute milk run.
« Reply #26 on: <09-12-19/1407:57> »

There are rules in the ToS about starting Flame Wars. Rules that I usually only enforce when things spiral way out of control. We've been down the road of people not liking this edition ever since they announced they were making a new edition. Many have received warnings, some have received bans. All because people are "presenting their data" to back up their opinion. My goal here as moderator isn't to shut down discussion of the game, but to keep the discussion from turning into "IS SO"/"IS NOT" arguments. Which is where this is leading since the people that like the changes and the people that don't aren't changing any minds on either side.
What is the purpose of mentioning the ToS? When you bring up the ToS, you are trying to send a warning, correct? In this case, your theory is that I'm breaching the ToS and should stop posting. Correct?

Quote
your theory is that Shadowrun 6E isn't any good and people should stop purchasing it so that Catalyst Game Labs stops production and sells off the IP to people that "know how to make the game"
This is outright putting words into my mouth, FastJack. You've even used quote marks for something that I'm pretty sure I've never said. And you've completely misrepresented my position, then attacked it with barely-veiled threats that I'll shut up if I know what's good for me. Is there anything in the ToS about doing that?

No, I am not attempting to prove that 6e is no good. I am attempting to prove it's not going down well and that's not a good thing. I don't think that's a controversial opinion, but there's been a lot of pushback on here from people defending 6e to the hilt. My usual method of resolving differences of subjective opinion is to see if we can find an objective metric to guide us. This is one. It's imperfect, like all metrics. But it's something.

Hell, this started because you did the same thing. We were arguing about subjective metrics, and you introduced "well, it's selling well on DTRPG" as an objective metric of quality. All I did was take it to the next logical stage.

Banshee

  • *
  • Catalyst Demo Team
  • Ace Runner
  • ***
  • Posts: 1095
« Reply #27 on: <09-12-19/1410:04> »
Actually yes I can, people who only buy from the CGL store may be more positively biased, as well as a difference between people who only buy online when compared to people who only buy thru FLGS. There also very significant bias  when you don't know the reasons for the reviews (good or bad) as well as a major deviation factor due to ratio of responses to sales. Not saying it wrong, but also says it does not represent the actual market either ... just missing too much data. The only thing this particular rating shows is that it is selling well on DTRPG.
You can make all these arguments about the other games on the list, most of which are available digitally from other stores and I think all of which are available physically via FLGS. Yet we don't see a large number of negative reviews for any of the others. Why is Shadowrun 6e different?

Edit -- and why didn't these factors affect SR5, which stands at 4.1/5?

You misunderstood what I was saying I think ... I was not making any kind of argument about whether this reflected on how good 6E is, but rather the rating itself. I would not rely on any of these ratings to represent any useful information without much needed backup data to fill in the gaps. The sample size is too restricted, too small, and not detailed enough to make an informed judgement.

As a side note to other comments on this thread, I would say the same thing about all of the reviews and comments on the internet (primarily here and on reddit). When the vocal community (both positive and negative) only represent 10% (or less even) then the total product user population then the information is not useful. At best someone interested in said product (6E this time but can apply to all products when doing consumer research) can do is take such observations under advisement but test the product out for themselves.

EDIT: On sales, so far 6E has 0ut sold 5E based on the same product availability at this time in it's life span. I.E. when compared to when 5E was released in hard copy at Origins and Gencon only but PDF via the internet, which is where 6E currently stands ... 6E has sales numbers that far exceed 5E. Early sales does not reflect on quality, playabilty, or community reception.
« Last Edit: <09-12-19/1417:42> by Banshee »
Robert "Banshee" Volbrecht
Freelancer & FAQ Committee member
Former RPG Lead Agent
Catalyst Demo Team

penllawen

  • *
  • Omae
  • ***
  • Posts: 804
  • Let's go. In and out. Twenty minute milk run.
« Reply #28 on: <09-12-19/1412:32> »
What I don't appreciate is comments telling CGL to stop production and sell off the IP.
Again: did I say that anywhere? Did anyone say it in this thread?

dezmont

  • *
  • Chummer
  • **
  • Posts: 190
« Reply #29 on: <09-12-19/1420:33> »
To try to drag this back to a interesting and productive place more interesting than 'cata bad' the point of anarchy is an interesting one.

Anarchy had a lot of goodwill despite not being... great... Probably because it represented a good idea to split the line and came out around when cata had some warm feels.

I wonder how well 6e would have done as 'Advanced Anarchy.'