NEWS

Fun with Surveys

  • 104 Replies
  • 19173 Views

Reaver

  • *
  • Prime Runner
  • *****
  • Posts: 6422
  • 60% alcohol 40% asshole...
« Reply #15 on: <09-10-20/1423:33> »
I was refering to what I saw was your expectations of the community, not your data :P

The numbers are what they are, and reflect the answers by those who answered the poll.

Sadly, I will say thst I think the polling size is too small to get a feeling for anything other the the responders however.

We know the book (core book) has sold thousands of copies  a poll of less than a hundred responses isn't even 5% of the market.


On the flip side:
The fact that you only got less than 100 responses in a week after posting said poll to the '3 biggest platforms' in itself is a little telling...

Apathy towards the game (can't care enough to take a poll?)
Apathy rowards polls in general ("why bother? They are always wrong anyway!")
Where am I going? And why am I in a hand basket ???

Remember: You can't fix Stupid. But you can beat on it with a 2x4 until it smartens up! Or dies.

Stainless Steel Devil Rat

  • *
  • Errata Coordinator
  • Prime Runner
  • *****
  • Posts: 4572
« Reply #16 on: <09-10-20/1439:18> »
On the flip side:
The fact that you only got less than 100 responses in a week after posting said poll to the '3 biggest platforms' in itself is a little telling...

Apathy towards the game (can't care enough to take a poll?)
Apathy rowards polls in general ("why bother? They are always wrong anyway!")

Also it's an open question of how much of the Shadowrun fanbase is represented by the '3 biggest platforms'.  Shadowrunner's Union FB group, for example, has over 8 thousand members.  Even making the faulty assumption of presuming noone is a member of more than one of the three: I doubt dumpshock, r/Shadowrun, and this forum combined have even half that total.
« Last Edit: <09-10-20/1445:12> by Stainless Steel Devil Rat »
RPG mechanics exist to give structure and consistency to the game world, true, but at the end of the day, you’re fighting dragons with algebra and random number generators.

Marcus

  • *
  • Prime Runner
  • *****
  • Posts: 2802
  • Success always demands a greater effort.
« Reply #17 on: <09-10-20/1450:28> »
Well if the official site and Reddit are not the biggest platforms for SR, what are? I do think the IP had taken some fairly sizable hits particularly in the wake of the Xbox game. I still sometimes find people who think it died after that. Certainly the sample population is small. Less then 100 folks, but it accurately represents the opinions of folks in my area. I for one still hope someone will come to their senses fix AR/DR. Not betting on it mind you. But hoping, I mean miracles have happened.

Edit: thanks putting that together oB.
« Last Edit: <09-10-20/1452:15> by Marcus »
*Play-by-Post color guide*
Thinking
com
speaking

0B

  • *
  • Chummer
  • **
  • Posts: 173
  • Be seeing you
« Reply #18 on: <09-10-20/1451:45> »
Yeah, here's the thing about sample sizes and statistical significance: They are highly dependent on the size of a population. That is why I also included the confidence intervals as the margin of error.

If I did a survey of 10 respondents, that seems small, but what if the total population is 15? That might give me statistically significant results. (Assuming random sampling, use of confidence intervals, etc)

The other part is that this is NOT a survey of all customers. This is a survey of three sites. So we aren't using the population of the customer base, since nothing about this survey necessarily pertains to the customer base.

It doesn't really matter what we feel the results mean, I'm just doing it based on what the math says. When we say that 10.39% +/- 6.81% of the population likes 6e, what I am actually saying is best put like this:

With 95% confidence, between 3.58% and 17.20% of visitors of /r/shadowrun, dumpshock, and the official forums like shadowrun 6e.

It is completely twisting the facts to say that only 10% of shadowrun fans like 6e. It's also twisting the facts to say that this data is useless because it only measures a few forums, or because it only had 79 respondents. A larger sample size would make the margin of error smaller, a smaller sample size would make the margin of error bigger. These forums are significant, although I bet there's dozens to hundreds of discord servers out there where people talk about SR.

Either way, if you don't like how minimum sample size is calculated or don't think the intervals are large enough, there's not much I can do. I can send you some academic papers about the mathematics surrounding sample size determination, the law of large numbers, and other foundations of statistics, but I don't really want to and I bet you don't want to read them either.

I don't know if you've done a lot of polls, but honestly even a response rate this high isn't bad, and is not an indicator of apathy. Consider that many visitors of one of those sites might not have even seen the post.

0B

  • *
  • Chummer
  • **
  • Posts: 173
  • Be seeing you
« Reply #19 on: <09-10-20/1504:34> »
On the flip side:
The fact that you only got less than 100 responses in a week after posting said poll to the '3 biggest platforms' in itself is a little telling...

Apathy towards the game (can't care enough to take a poll?)
Apathy rowards polls in general ("why bother? They are always wrong anyway!")

Also it's an open question of how much of the Shadowrun fanbase is represented by the '3 biggest platforms'.  Shadowrunner's Union FB group, for example, has over 8 thousand members.  Even making the faulty assumption of presuming noone is a member of more than one of the three: I doubt dumpshock, r/Shadowrun, and this forum combined have even half that total.

That depends on how you define active members. Does the facebook group have 8 thousand members, or 8 thousand active members? If it's just 8 thousand total members, 2 out of 3 sites beat it out. r/shadowrun has 38,800 members. This site has 7,480 members. Dumpshock has 13,389 members.

If we're going off of active members, we can look at how the FB group's members react: measuring reactions is going to be better than counting comments, since of course people can comment twice on a single thing. The last 10 posts, according to my feed at this time, have these reactions:
* 39, 5 comments
* 19, 0 comments
* 0, 6 comments
* 5, 17 comments
* 19, 7 comments
* 10, 2 comments
* 39, 6 comments
* 1, 5 comments
* 12, 9 comments
* 14, 4 comments

The 5 announcements have these reactions:
* 113
* 72
* 218
* 249
* 124

Is that more or less active users than one of the other sites? It's hard to say. If you look for top posts on /r/Shadowrun from the past month, the top post has 439 upvotes (Which is possibly a higher number of interactions, since there could be some downvotes). If we look at the stickies, we see pen's post that has 129 upvotes and 115 comments. There are a few other 100+ posts in Hot as well.

Either way, I doubt all 8000 members are active on FB since I'm in the group and I only go on FB to talk to my mom

adzling

  • *
  • Prime Runner
  • *****
  • Posts: 3290
« Reply #20 on: <09-10-20/1524:46> »
Looking at this poll (thanks OB, really well documented and it's clear you know your stuff) together with sales numbers for 6e core compared to 5e core (and other products) it does look like 6e:

1). Is not liked by the player base.
2). Is not being purchased at any where near the levels 5e was.

This fits my anecdotal evidence from talking to people I play with and on the forums I frequent.

penllawen

  • *
  • Omae
  • ***
  • Posts: 804
  • Let's go. In and out. Twenty minute milk run.
« Reply #21 on: <09-10-20/1531:36> »
With 95% confidence, between 3.58% and 17.20% of visitors of /r/shadowrun, dumpshock, and the official forums like shadowrun 6e.
FWIW, I’m not a statistician but I sometimes have to impersonate one at my day job. 0B is completely correct here.

Banshee

  • *
  • Catalyst Demo Team
  • Ace Runner
  • ***
  • Posts: 1095
« Reply #22 on: <09-10-20/1559:34> »
Personally I find it interesting that roughly 70% of the responses that don't like 6E don't have the book. So how can they be giving it a fair assessment when they are obviously only able to judge on what others say rather than direct personal experience.

Robert "Banshee" Volbrecht
Freelancer & FAQ Committee member
Former RPG Lead Agent
Catalyst Demo Team

Michael Chandra

  • *
  • Catalyst Demo Team
  • Prime Runner
  • ***
  • Posts: 9920
  • Question-slicing ninja
« Reply #23 on: <09-10-20/1614:37> »
Personally I find it interesting that roughly 70% of the responses that don't like 6E don't have the book. So how can they be giving it a fair assessment when they are obviously only able to judge on what others say rather than direct personal experience.
And then there's the list of people who pirated the book and dislike it, and likely only pirated it because they already decided they hated it BEFORE it came out, pirated it the second it came out electronically (thanks to you-know-who), and we don't know how much of it they decided to actually read or give a fair shot.
How am I not part of the forum?? O_O I am both active and angry!

Lormyr

  • *
  • Catalyst Demo Team
  • Omae
  • ***
  • Posts: 820
« Reply #24 on: <09-10-20/1655:02> »
Personally I find it interesting that roughly 70% of the responses that don't like 6E don't have the book. So how can they be giving it a fair assessment when they are obviously only able to judge on what others say rather than direct personal experience.

Well you have to bear in mind it is trivially easy to read over a book without buying it these days. Between pirating, reading a friend's copy, reading it in a store, ect. Also, 72% of those that replied listed either buying or pirating the book, so do have it. I think you inverted the chart.

pirated it the second it came out electronically (thanks to you-know-who)

Fucking LOL Mikey, your hyperbole is truly legendary! I am totally sure there was only one, single, solitary bro who made copies available. That is totally believable!

LOL.



"TL:DR 6e's reduction of meaningful choices is akin to forcing everyone to wear training wheels. Now it's just becomes a bunch of toddlers riding around on tricycles they can't fall off of." - Adzling

0B

  • *
  • Chummer
  • **
  • Posts: 173
  • Be seeing you
« Reply #25 on: <09-10-20/1658:51> »
Personally I find it interesting that roughly 70% of the responses that don't like 6E don't have the book. So how can they be giving it a fair assessment when they are obviously only able to judge on what others say rather than direct personal experience.

That's one possibility. Another is that they played with the quickstart rules, didn't like them, so they didn't get the book. Or that their GM bought the book, they didn't enjoy playing it when they tested it out, so the players didn't buy a copy. We have no evidence to say one way or another. This might be a good follow-up question for a future survey- "Why didn't you buy the book?" or "Have you played 6e?"

I did post up above the result of the COUNTIFS if you only look at who buys the book- 42% of people who purchased the book disliked it, and another 39% have "complicated" feelings about it.

You are welcome to look at the raw data to see the stats for yourself, and filter away based on responses to questions.

Of the 36 people who said they purchased, 34 answered the "what do you dislike" question. I've bolded the ones where it differs from all respondents by more than 5%.

27 disliked Editing Quality. (75.0% of purchasers, vs 73.4% of all respondents)
23 disliked that the rules were not clear. (63.9% of purchasers, vs 57% of all respondents)
21 dislike edge. (58.3% of purchasers, vs 60.8% of all respondents)
14 dislike CGL. (38.9% of purchasers, vs 50.6% of all respondents)
14 dislike Style/Syntax. (38.9% of purchasers, vs 39.2% of all respondents)
14 dislike combat mechanics. (38.9% of purchasers, vs 43% of all respondents)
14 dislike mechanics in general. (38.9% of purchasers, vs 58.2% of all respondents)
12 dislike character creation. (33.3% of purchasers, vs 44.3% of all respondents)
11 dislike magic. (30.6% of purchasers, vs 36.7% of all respondents)
11 state that their group plays a different edition. (30.6% of purchasers, vs 39.2% of all respondents)
9 dislike that it wasn't easy to learn. (25.0% of purchasers, vs 21.5% of all respondents)
9 dislike the Matrix. (25.0% of purchasers, vs 36.7% of all respondents)
8 dislike the CRB fiction. (22.2% of purchasers, vs 20.3% of all respondents)
6 dislike skills. (16.7% of purchasers, vs 24.1% of all respondents)
6 selected "word-of-mouth" under this question (16.7% of purchasers, vs 19% of all respondents)
5 dislike the core dice pool mechanics. (13.9% of purchasers, vs 20.3% of all respondents)
4 dislike the setting. (11.1% of purchasers, vs 11.4% of all respondents)
4 dislike the price. (11.1% of purchasers, vs 13.9% of all respondents)
3 dislike the theme/genre. (8.3% of purchasers, vs 3.8% of all respondents)
1 didn't want to learn a new edition. (2.8% of purchasers, vs 7.6% of all respondents)
1 said Other: The mechanics make no sense, they do not reflect real world gun fights nor combat in any form. They are useless at everything they try do combat-wise.
1 said Other: Turning Seattle into Hong Kong because the HK people left is weaksauce. The whole Fiat-EMP plot is horrible and the missing SuperSoldiers from Hell are not better either. It's peak MagicRun.
1 said Other: Doubling Down on shit nobody liked in the previous Editions.

Respondents who purchased the book are slightly more likely to dislike the lack of rules clarity.

They feel about the same on editing quality, edge, style/syntax, combat mechanics, learning difficulty, CRB fiction, "word-of-mouth", setting, price, theme/genre, and desire to learn a new edition.

They are slightly less likely to dislike magic, dislike skills, say that their group plays a different edition, or dislike core dice pool mechanics.

They are moderately less likely to dislike CGL, dislike mechanics in general, dislike character creation, or dislike the matrix.

Personally I find it interesting that roughly 70% of the responses that don't like 6E don't have the book. So how can they be giving it a fair assessment when they are obviously only able to judge on what others say rather than direct personal experience.
And then there's the list of people who pirated the book and dislike it, and likely only pirated it because they already decided they hated it BEFORE it came out, pirated it the second it came out electronically (thanks to you-know-who), and we don't know how much of it they decided to actually read or give a fair shot.

You have nothing to go off of to say this is true of any portion of the population.

In both cases... why would someone buy a book if they didn't think they would like it? IE, they read a bad review, played a game of it and didn't like it, etc.

0B

  • *
  • Chummer
  • **
  • Posts: 173
  • Be seeing you
« Reply #26 on: <09-10-20/1705:39> »
Wasn't that leaked copy put up anonymously on a file sharing site? I know folks linked to it from reddit.

Or are you talking about the time one of the errata team talked about a rule, breaching NDA, but did not post the full text of the rule? (That's not piracy. Mechanics can't be copyrighted, only their text. US Copyright law is weird, but it does prevent Monopoly from having a monopoly on the use of six-sided dice)

Lormyr

  • *
  • Catalyst Demo Team
  • Omae
  • ***
  • Posts: 820
« Reply #27 on: <09-10-20/1714:36> »
Wasn't that leaked copy put up anonymously on a file sharing site? I know folks linked to it from reddit.

Or are you talking about the time one of the errata team talked about a rule, breaching NDA, but did not post the full text of the rule? (That's not piracy. Mechanics can't be copyrighted, only their text. US Copyright law is weird, but it does prevent Monopoly from having a monopoly on the use of six-sided dice)

oB, gotta say, I'm gaining some mad respect for you.
"TL:DR 6e's reduction of meaningful choices is akin to forcing everyone to wear training wheels. Now it's just becomes a bunch of toddlers riding around on tricycles they can't fall off of." - Adzling

0B

  • *
  • Chummer
  • **
  • Posts: 173
  • Be seeing you
« Reply #28 on: <09-10-20/1733:45> »
Wasn't that leaked copy put up anonymously on a file sharing site? I know folks linked to it from reddit.

Or are you talking about the time one of the errata team talked about a rule, breaching NDA, but did not post the full text of the rule? (That's not piracy. Mechanics can't be copyrighted, only their text. US Copyright law is weird, but it does prevent Monopoly from having a monopoly on the use of six-sided dice)

oB, gotta say, I'm gaining some mad respect for you.

TBH, we're working off of my memory since the reddit mods deleted the post (Piracy against the rules. Deletion was too quick for ceddit to grab the post). The file sharing site was also one where uploads only stay up for 24 hours, to evade DMCA, but I'm sure folks shared the file via discord or other private channels.

Also being fair here, LVN is trash and they did decide to hate it before it came out. That's one user, though, one who's been banned from SCN and various other SR networks for this kind of behavior, not really a representative of the community.

And don't get too mad at me about "well why do you know these things that only a pirate would know," since I've purchased every 6e book (At least the PDFs). Middle school me might not've bought the book, but middle school me also had a $5 weekly allowance that she either blew on snacks or saved for christmas presents. Her "internet browsing" habits did not affect the bottom line of WOTC one way or another (and getting into the game influenced parents to buy the books for her, eventually).

adzling

  • *
  • Prime Runner
  • *****
  • Posts: 3290
« Reply #29 on: <09-10-20/1757:05> »
You don't actually need to buy the 6e book yourself to understand it's trash.
You only need to ask someone who's opinion you value.
Like say your GM who bought it, or another player in your group who bought it.
Or read the reviews on the interweb that highlight it's many deficiencies.

Just like you don't need to buy a Delorean to know that you're gonna have a lot of problems down the road...literally.

This jumped out at me OB:

"1 didn't want to learn a new edition. (2.8% of purchasers, vs 7.6% of all respondents)"

Does this put the final nail in the coffin of the "dedicated/long-time shadowrun players don't like new stuff" meme that M.C. (and others) continually push?