Misreading the Player Feedback Loop

Hello, everyone! I present a case study on the dangers of misreading the player feedback loop.

Zilch: The name of the game and the amount of respect its developers have for the players

Recently a game made its way onto Kongregate called Zilch. Zilch is a simple dice rolling game similar to Yahtzee. Players take turns rolling 6 dice trying to find dice sets that score them points. They can then put these dice sets to the side and roll the remaining ones. As long as another scoring set can be found in those remaining dice, the player can score more points. They continue rolling less and less dice until one of two things happen. Either they score with all 6 dice, which lets them start the whole process over and rack up even more points, or they roll their remaining dice and are unable to find anything to score with. The latter case is called Zilch, and you lose all your points for the round. Strategy becomes similar to Pass the Pigs, where as you continue to roll on your turn and rack up points, it becomes more likely that you will roll a Zilch and lose all those points. The player must balance risk and reward and stop before he loses all his points.

Hopefully that was an ample description of the game. If it wasn’t, here’s a link to Zilch so that you can play yourself. This kind of game isn’t for me, because of how much luck is involved, but it may be your thing. In any case, this is all side talk. What I want to talk about today is the most common player feedback Zilch’s developer has received, and their inadequate response to it.

Dumber than a computer

Let’s me come right out and say what most players complain about in Zilch. The AI seems to be very lucky and very hard to beat. Zilch provides three AI opponents, called Reckless, Cautious, and Realist. Your first instinct will be to match these up to traditional game difficulties of easy, medium, and hard. This is a mistake. All three of these AI opponents have a sophisticated methodology to how they play Zilch, and all three of them have a better understanding of the rules and probabilities than a new player. Common player feedback is stuff like “The AI is rigged, he consistently gets over 1000 points on his turn”. Chances are this player was facing the Reckless AI, who tends to score very high a lot of the time and Zilch almost as frequently. He plays recklessly, after all.

New players tend to play flash games on easy until they get a feel for it. They expect an almost sandbox style of play, where they can try and grasp the game concept without having to suffer a humiliating defeat. It’s no surprise that people who play games really really like to win at them. Joining this notion is a Kongregate achievement centered around “winning 3 games on any difficulty”. For the achievement junkies that need to get those wins ASAP so they can move on to the next achievement, it makes sense to play the game on easy.

All of you are wrong because I’m right

The developer of Zilch, playr.co.uk, issued a response to all the players complaining. You can read that response here (response 1). In summary, they explain how the three AI’s work and that all the dice rolls in Zilch are fair from a random number generator. Clearly this wasn’t enough because later that day they made a more frantic response that you can read here (response 2). This time they used all caps and posted their Random code for players to audit. It is indeed random. You can almost imagine their inbox flooding with complaints. They’ve even made another response that you can read here (response 3), but it’s more of the same.

Finding the real problem

So what’s the problem here? Zilch’s developers are unable to get to the crux of the player complaints. They chose to instead defend their design choices and attempt to educate the player. Unfortunately they didn’t do this inside their game, but at their obscure company blog that maybe MAYBE 1% of players will actually read.

I will now try to lay out exactly what is happening. A player starts up Zilch for the first time. There’s no introduction or explanation, so when they pick a CPU opponent they choose the one that should be easy; mostly because they have no idea what they’re in for. “Reckless, well that’s never a good thing and it’s on the left… so it must be easy.” The game starts and players are educated while they take their turns. While this education gives a fairly good grasp of the game rules, it does not talk about strategy (and it shouldn’t). Unfortunately for the player, the Reckless AI has been employing a basic yet more effective strategy the entire time the player was in tutorial land. By the time the player is ready to take off auto-pilot, there’s a good chance the AI has the lead. As the game continues, players will notice a pattern in the way Reckless plays. He either scores large amounts of points on his turn, or he gets zilch. With no real understanding of the game and a more cautious playing style, the player will be taking smaller sums of points each turn and zilching more, simply because they haven’t yet figured out the game’s probabilities.

After reading what I would consider a typical first time experience with Zilch, is it hard to see why players send frustrated emails to the developer?

Actually making players happy

It turns out that the complaints about the random number generator are red herrings. Players don’t care about how the game works, all they care about is that their game expectations were incorrect and they were punished for it. What the Zilch developers need to do is stop trying to convince players that they were wrong. Not only will that not work, but the players are almost never wrong.

First, create a new difficulty and call it something recognizable like Beginner. Next, and this is the super important part, skew the dice rolls in favor of the PLAYER on the Beginner difficulty. Create an AI that purposely makes poor decisions. The goal here is to create an AI that approximates a new player who has absolutely no understanding of the rules let alone the strategy of Zilch. And then give that AI permanent bad luck. Now players will have a place to go to when one of the real AI’s hands it to them.

The next thing I would do, is create yet another difficulty, called Normal. This AI should contain the same lack of strategy that Beginner has, but the dice rolls are fair. Zilch has an interesting ‘problem’ in that the easiest AI to create turns out to be very competent. Typically, AI competency is directly proportional to its complexity, but in Zilch it will take more time to write an incompetent AI that is fair to new players.

Finally, take your three existing AI’s: Reckless, Cautious, and Realist, and put them all under a Hard difficulty heading. Depending on your strategically advanced opinions of Reckless and Cautious, you might want to revisit their AI’s to make them stronger. The idea is that these three AI’s are employing real Zilch strategies. Sure some are more simple-minded than others, but every single one of them is better than a new player. That’s your problem.

16 thoughts on “Misreading the Player Feedback Loop

  1. Pingback: flash games 6 | Digg hot tags

  2. Jim Bosun

    Andrew,

    I disagree with your philosophy. Fortunately for me, I understand probability and enjoy playing games of chance. I am familiar with outrageous turns of fortune and have the good grace to apologise to my opponent when lady luck smiles upon me. I also get outraged when things swing the other way but I note that it is not the fault of my opponent. In this case, I think the people in question who feel like they have been cheated have acted very immaturely. The programmer stands to gain nothing by stacking the dice against the player and his guarantee is enough for me.

    The mature thing to do would be to carry on playing and to learn when it is favourable to take chances. There is an excellent opportunity here to educate people about probability and how gambling should not be taken lightly. These people who expect the whole world to come in easy, normal, hard flavours will have difficulty surviving and will not make friends by moaning that the whole world is against them at every stage. I think the programmer should be commended for his unwillingness to compromise; perhaps he should put a warning on the game that it should not be played by those unwilling to lose at least half of the time.

    Anyway, those points aside, I accept that you of course have a point and that the game *could* be tailored to make it more accessible to the kind of people who just enjoy point and clicking on colourful boxes with only a superficial understanding of what is going on but there are enough of these games in the world already.

    Do you agree that it’s sometimes nice to play a game that involves something a bit deeper?

    Thanks for reading,

    Jim

    Reply
  3. Andrew Pellerano Post author

    Thanks for the reply, Jim. I don’t think there’s such a thing as a game that is only for people smart enough to “get it.” That seems to be a way to candy coat a failure of the game designer to explain the game and gently ramp players into it. The idea of Zilch itself is not hard for the lay man to grasp. You could imagine popping open its box and playing it at Thanksgiving with your family.

    You’ll note that my suggestions didn’t eliminate the existing AI, just pushed them off so that players knew what they were getting into. The complaints would stop if difficulties were created that are in line with what players are expecting. You could even illustrate the distinction between the dice rolls stacked in the player’s favor for some difficulties and how they are completely fair in others.

    Unfortunately for all of us, we don’t live in candy land. Sometimes we make something and it isn’t accepted the way we wanted it to be. The worst thing you can do in that situation is stick to your guns and try to convince the players that they are enjoying your game in the wrong way.

    Reply
  4. Thomas

    Wow! I couldn’t disagree with you more. Some guy makes a really cool game and he should be dissed because he didn’t dumb down the game instructions enough to mollycoddle the inanely (and probably irretrievably) stupid? Or because he didn’t make it so easy that people can win with no effort or learning curve? And did you actually suggest that he skew the dice rolls in favor of the player on the beginner setting?? Oh yeah, that’s just what I want to play, a game of chance that’s not really chance, just so I can win and feel good about myself! Maybe he should have coded the game to be a screen that just pops up graphics saying “You Win!” every few seconds.

    Basically your argument boils down to asking the developer to code for and coddle the lowest common denominator and in that context I think your subtitle to this post… “Zilch: The name of the game and the amount of respect its developers have for the players” is actually a better description of your position. You are saying that the computer should effectively throw games on the beginner level like a parent does for their 5 year old kid because otherwise the poor player might have a bad experience. That’s truly not respecting the players abilities or maturity, I mean what could be more pathetic than expecting the computer to throw a game of chance in order to make it fun for you to play. (because as every 5 year old kid knows it’s not fun to play if you don’t win, right?)

    I think the developers response is perfect… “The dice rolls are fair, here is the source code, If you don’t like games of chance then don’t play them.” And if you don’t understand probablity or the basics of cause and effect (a reckless player will sometimes win big but will zilch alot too) then use the internet and get some knowledge, but don’t whine like a 5 year old crybaby having a tantrum.

    “Oh Wwaahhh! the computer cheats!”

    Reply
  5. Andrew Pellerano Post author

    Thomas, there seems to be a fundamental difference in how you and I measure the success of a game. My success metrics are player-based: a game is more successful if it can cause more players to have more fun. Yours is design-based: a game is more successful if its design is perceived to be more pure or possibly more elegant.

    What I hinted at but didn’t directly mention in my article, is that a design-based view is selfish and ego-centric. We don’t make games simply to make ourselves happy. We make games to make others happy (which in turn makes us happy.) If you’re failing at making certain others happy, as is the case with Zilch, then your game is missing out on some of its success.

    I simply chalk up a mistake like that to inexperience. It doesn’t matter what kind of software you’re writing, game, application, or other. Sitting a new person down in front of your UI and watching them use it for the first time (without your help) is always a sobering experience. If you don’t do this with every single one of your products, you will always be missing out on low hanging fruit that improves the product’s quality.

    What’s interesting is that I didn’t propose that any features be removed from Zilch. I simply said that there should be more features put in to ease the player into the game. I’m at a loss why this suggestion would make anyone as angry as you appear to be.

    Reply
  6. Thomas

    I’m not angry, just annoyed. I thought the game was slick and well done. It had a tutorial that walked me through every aspect of the game and was more than sufficient to nail down virtually every aspect of play. At no time did I find it difficult to beat either the reckless or the cautious settings although not every time of course. It annoys me that someone would complain to the developer of a well done game because they are convinced that the computer cheats. Tell me, why would *any* developer code their game to cheat or cast crooked non-random rolls? What would be the motivation for that? It simply doesn’t make any sense and anyone who complains of something like that is either a paranoid delusional or someone who might want to try something more skill based. And that’s really what it comes down to… “Audience”. You say that a design based view is selfish and ego-centric but I think a game without a design focus is always going to be poor. The design is the game, otherwise, as I asked before, why not just pop up a “you win” screen every few minutes. Of course a game needs to be pleasing and fun, but a game needs a design and that design should be targeted to an audience. For Zilch that audience is people who want to play a game of chance with a little decision-making and strategy thrown in. What annoys me and makes me seem angry is the idea that this developer and the free game he provided are somehow lacking because he chose to create a fair chance based game for a competent audience rather than create a skewed rigged game (yes you suggested skewing the rolls in favor of a beginner) just so people could feel good about winning. The bottom line is you can’t please everyone all the time so I don’t think this developer deserves the dissing you’ve given him just because his game isn’t aimed at the run and jump super mario brothers crowd.

    Also, There will always be whiners but that doesn’t mean they are always right (especially when they are accusing the computer of cheating), and just because some people complain doesn’t mean that a developer has failed to please the large majority of his audience.

    Zilch has a 4 star rating on Kongregate by the way. I’d hardly call that failure, poor design, egocentric development or any of the other things you accused the poor guy of.

    Ultimately the real reason I may seem a little angry is that you’re dissing someone else’s hard work with inflammatory titles like “Zilch: The name of the game and the amount of respect its developers have for the players” simply because he got a small minority of vocal users complaining that he rigged the dice, and after explaining that he didn’t, and even providing the source code got a little short with the ones who continued to insist he was a cheater. I don’t think that he’s guilty of bad design, nor of disrespecting his users. He’s just a human being who has a limited tolerance for people insulting him for his efforts to create a *free* game. I would too! And I’m offended by the sanctimony of one developer preaching to another for the “poor design” and egocentricity of his free 4 star game because it didn’t please all of the people which is an impossibility anyway!

    I don’t know the guy, nor have I had any contact with him. But I found myself here and was inflamed by your inflammatory post. So here I am, posting on a blog that I’ve never been to before, about a game I’ve played for only an hour or so, in defense of a developer I don’t even know, on an issue that doesn’t affect me in the slightest. That’s kind of silly of me! But oh well, I’m not perfect either, just like the guy who wrote Zilch. 🙂

    Reply
  7. Andrew Pellerano Post author

    Thomas, I agree that the dice rolls are fair, I said so in my original article. I do not mean to disrespect the game itself. My issue is in how the developer responded to player feedback. My article is about misreading the player feedback loop, not the quality of the game. Although it is true that the quality suffers because of the misreading.

    Zilch is good enough to get a 4 star rating. What’s worth remembering is that Kongregate uses a 5 star system. This means that Zilch’s rating is actually missing an entire 20% of the total rating it could be getting. One way to look at this is not that your game is lacking in 1 out of 5 areas, but that 1 out of every 5 players does not like your game. Judging by the comments and the developer’s blog posts, my guess is that the “rigged dice rolls” is the biggest issue.

    With that in mind, we arrive at why I originally wrote my article. Zilch is a good game that could have been better. It made some mistakes, but we all make mistakes. The most important thing about a mistake is that you learn from it. Even better if you’re able to share a lesson with other peers.

    I’d like to cover some of your smaller points just so you’ll stop bringing them up. You asked why any developer would make crooked dice rolls. This happens all the time. Making an AI with an unfair statistical advantage is much easier than programming a competent AI, and time is money. People who play a lot of games know this, and will begin to blame cheating computers for why they are losing a lot. No one likes to lose and the computer can’t defend its own integrity, so it’s an easy target.

    Which brings me to the next point you keep making. You say that to cater to babies who just want to win all the time, developers should throw up a “you win” screen every few minutes. But this kind of screen is not a game by definition, because it never presents a challenge to the player. A fair fight has 50/50 odds. Pitting a new Zilch player against an AI with a strategy and understanding of the rules probably skews those odds in favor of the computer. I suggested creating a beginner mode and skewing the dice rolls in favor of the player to make things more even again. With a few wins and some confidence, allow them to tackle the fair AI and they’ll be more likely to stick around after losing.

    Having a good game design is unfortunately only a fraction of what it takes to have a good game. Equally important is your ability to communicate the game design to the player. In the world of small flash games, your design is almost all you have; flash games aren’t known for their vast content. With the design being such a large part of the game, and communication being equally as important as the design, you’re left with a very important communication task and not a lot of content to do it in. This is probably the most difficult part of making a flash game.

    Reply
  8. Thomas

    You said: “My issue is in how the developer responded to player feedback.” but my question would be how could he possibly have done a better job of that? His players complained about the dice being rigged. He assured them that they were not, they persisted in claiming they were, he explained his algorithms for both the dice rolls and for the AI strategies, they continued to insist the randonimity was flawed, so he released his source code. What more can you do? And I would argue that responsivity to the point of releasing your source code is above and beyond the call of duty.

    His players weren’t making specific complaints about losing, they were complaining because they said the rolls were rigged or flawed. You seem to be arguing that if he disregarded their actual complaint and assumed they actually meant that it was too hard and therefore made it easier for players to win in the beginning then that would have addressed the problem of people thinking the dice were unfair. I disagree, if anything making it easier in the beginning and having it get harder as people progressed would make them even more convinced the hard opponents were cheating.

    He already made it so you don’t get to play the ‘realist’ opponent until you’ve won 10 games. This is important because while it is true that ‘reckless’ and ‘cautious’ don’t perfectly match an ‘easy’ and ‘hard’ paradim they are close enough to it in practice because ‘reckless’ is *always* reckless and that’s a pretty easy trait to exploit. Realist is much harder because he actually has a good strategy. So the developer did include a ramp up in difficulty as you suggested.

    After I made my earlier posts here I read a blog post on his site about why ‘realist’ will take 1 dice instead of a pair when he has the chance (so he has a greater chance of getting a high combo with a greater number of remaining dice) and saw several posts after that claiming that they didn’t know they could do that. Putting aside the fact that Zilch is a dice game where you select which dice you want to keep and which ones to reroll so it shouldn’t be any mystery which needs to be explained that you can *select which dice you want to keep and which ones to reroll*… he took that feedback and made it easier to see what the computer was doing so people could extrapolate that yes, they too could *select which dice to keep and which to reroll*. In other words he took the feedback and modified the game to account for and improve it. How is that not properly responding to player feedback again?

    Yes, it’s not a 5 star game. But I haven’t seen many of those. Have you? 5 out of 7 of Kongregates top contest leading games for November have only 4 stars. The other three have such a small fractional percentage over 4 that you can barely see the mark in the point of the 5th star. So this game is ranked among the best of the best and that’s a bad thing? Kongregate’s ranking system is an average. Just a couple bozos can pull down your 5 stars to 4 by giving you a 1. So I think your claim that he’s lost out on 20% of his game’s potential is extremely simplistic. Not to mention the fact that its a *dice game* he made a 4 star dice game! How do you make dice exciting enough for 4 stars? Well I guess he licked that one because he did it! Not to mention the badges and accomplishments and even a Kongai card attached. This is not a game that deserves to be used as an example of poor programming, or of poor regard for it’s players!

    I guarantee you if that he would have followed your advice there would have been just as many people complaining that it was too easy. Or getting bored because it wasn’t enough of a challenge on the first or second play (there are over 9,000 other games to try all competing for attention), and they would have given him 1’s and he would still have ended up with only 4 stars.

    But then I guess you could write a blog post about how he failed to please the people who wanted a challenge or failed to create a stimulating and engaging experience.

    Reply
  9. Thomas

    I accidentally hit 7 instead of 8 in my last post. The sentence in question should have read: “5 out of 8 of Kongregates top contest leading games for November have only 4 stars. The other three have such a small fractional percentage over 4 that you can barely see the mark in the point of the 5th star.”

    Reply
  10. Andrew Pellerano Post author

    I don’t think anything will come from observing a specific month of Kongregate’s top games. 4 stars is a good game, but the top games are the ones that are able to eke out that extra fraction of a star between 4 and 5. Those games are the ones that not only have a good idea but also communicate it well.

    Reply
  11. Thomas

    I looked at your portfolio and checked out your games on Kongregate. I played Battalion Nemesis, It’s been weeks if not months and I even remember it. I liked it and think it was good, so I’m not dissing you, but when you hold up a game with 4 stars like Zilch (actually 3.95) to such criticism I think it’s relevant to have some context. Here are your games, and ratings…

    My Sweet 16: School Musical 2.13
    My Sweet 16: 2.24
    Critter Cannon: 2.64
    Tiki Balls – Curse of Tane: 3.0
    Zombieland: 3.07
    BattleMachy: 3.23
    Discarded – Online: 3.93
    Battalion Nemesis: 4.18

    Like I said, I liked Battalion Nemesis, and I’m not trying to diss you here. I think 4 stars is a great score. But you seem to be saying it’s not adequate and that if you get 4 stars you’ve left “low hanging fruit” on the vine. So can you explain how exactly 4 stars is such a poor score that the developer of Zilch should feel like he’s somehow failed to properly serve or respect his players when your top score is less than a quarter point above his and most of your games are far far below?

    Reply
  12. Andrew Pellerano Post author

    Sorry, but I never said that I’m perfect. Going back and looking at the scores of games I’ve worked on is kind of neat, it shows a steady growth in the strength of the games I’ve been involved in. And yes, every game is always going to have low hanging fruits for a better score. Despite Battalion: Nemesis having a 4.18 I believe it could have done better with a few key fixes.

    4 stars is a good score, good enough to put you in the monthly standings depending on the month. But all the greats (the ones with millions of plays), they are noticeably higher than 4 stars. Zilch has a good score because it’s a good game. With the right modifications it could have been more accessible which I believe would have lead to a higher score.

    It’s worth mentioned however, that I don’t believe a Kongregate score is the end-all indicator of how good your game is. It’s simply an indicator of how well your game is liked by people who go to Kongregate.

    Reply
  13. Thomas

    To me we aren’t debating whether Zilch is a good game or not. We are debating whether it is so bad that it deserves to have it alleged that the developer has no respect for it’s players as your title suggests. This game in no way deserves to be made an example of anything poor. Including “Misreading the Player Feedback Loop” which as I’ve noted is grossly unfair considering the guy modified the game in direct response to feedback, explained his algorithms and AI strategies, and even released his source code for review. The only things he could have done further would have been including an in game strategy guide that explains every facet of statistics and winning tactics, or rigging up an AI strategy that deliberately loses games to first time players. Neither of which I think is reasonable or necessary… and neither of which I think would have increased his Kongregate ranking or the overall experience of the average player.

    Reply
  14. Raitendo

    I couldn’t have agreed more to the points you made in your post. Thomas and Jim’s points are hard for me to grasp since what you’re asking for is in no way a dumbing down of the game, as you wouldn’t actually remove any of the existing modes. Which… is very obvious, and should go beyond saying.

    Reply
  15. Andrew Pellerano Post author

    There was an article at Gamasutra today about this very topic.

    http://www.gamasutra.com/view/feature/3947/intelligent_mistakes_how_to_.php

    From the article:
    “As I found with the “positional play” in snooker, random outcomes that happen to favor the computer are perceived as being intentional. If the ball ends up in a good place, or the poker AI makes a lucky call and wins on the river, it can be perceived as unfair or even cheating.”

    Reply
  16. Jesse

    I like all of Andrew’s points and suggestions.

    This is going to seem irrelevant but if you hear me out I think you will find that it isn’t. When going through the tutorial, you are encouraged to roll and then choose from the options that pop up. For example “Two 5s: 100 Points”. At first I thought you could only ever choose exactly one of the popped up options and were then required to roll again or bank. As I played more I found that I could choose multiple popped up options if I wanted to, as long as they didn’t “overlap” dice. For me it was an “Oh I’m just an idiot” moment.

    Anyway I think it’s natural not to watch the computer like a hawk, but I noticed that the computer did, in fact, seem to have overall better luck than me, despite my having good confidence in my strategy. Then I started watching very carefully and I noticed that the computer would frequently roll “Two 1s: 200” or “Two 5s: 100”, yet only one of the two dice would become “locked” in place.

    I flipped out. I started noticing it happening all the time and filed a bug report. It turns out that it is not a bug because the player can do so as well by individually clicking on the dice along the left. Something that was NOT addressed — let alone clearly explained — anywhere in the tutorial nor the rules.

    Furthermore it is, as I see it, a huge advantage to be aware of this option, as choosing one five instead of two fives can vastly improve your chances of avoiding a zilch.

    So if I did have constructive criticism to give, I would say that the tutorial and the instructions need to be clearer. Something that would be VERY easy to do and wouldn’t change the game at all.

    Without having the advantage of making the best choices (and probably never noticing that the computer players are exercising options that we aren’t), it would explain why the game seems unfair, or favoring the computer.

    My second point that I want to make is that I never received a response to my bug report, so I started looking around the sites more. I found the explanation about “Why the computer player sometimes only chooses one die.” This note by the game author explained everything in what I viewed to be a tone that implied we should have known all along, ending with “Please stop sending bug reports about it.”

    Obviously if so many reports are being sent, all of these complainers did not understand the rules of the game, so I think the appropriate response would be to explain the rules better in the game instead of making an otherwise unnoticeable link to an explanation somewhere at the bottom of the game description.

    The developer’s reaction left a bad enough taste in my mouth to write this long post about it. So, there you have it — perhaps the game just seems so unfair because we’re not informed enough to play with a good strategy!

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *