Misreading the Player Feedback Loop

Hello, everyone! I present a case study on the dangers of misreading the player feedback loop.

Zilch: The name of the game and the amount of respect its developers have for the players

Recently a game made its way onto Kongregate called Zilch. Zilch is a simple dice rolling game similar to Yahtzee. Players take turns rolling 6 dice trying to find dice sets that score them points. They can then put these dice sets to the side and roll the remaining ones. As long as another scoring set can be found in those remaining dice, the player can score more points. They continue rolling less and less dice until one of two things happen. Either they score with all 6 dice, which lets them start the whole process over and rack up even more points, or they roll their remaining dice and are unable to find anything to score with. The latter case is called Zilch, and you lose all your points for the round. Strategy becomes similar to Pass the Pigs, where as you continue to roll on your turn and rack up points, it becomes more likely that you will roll a Zilch and lose all those points. The player must balance risk and reward and stop before he loses all his points.

Hopefully that was an ample description of the game. If it wasn’t, here’s a link to Zilch so that you can play yourself. This kind of game isn’t for me, because of how much luck is involved, but it may be your thing. In any case, this is all side talk. What I want to talk about today is the most common player feedback Zilch’s developer has received, and their inadequate response to it.

Dumber than a computer

Let’s me come right out and say what most players complain about in Zilch. The AI seems to be very lucky and very hard to beat. Zilch provides three AI opponents, called Reckless, Cautious, and Realist. Your first instinct will be to match these up to traditional game difficulties of easy, medium, and hard. This is a mistake. All three of these AI opponents have a sophisticated methodology to how they play Zilch, and all three of them have a better understanding of the rules and probabilities than a new player. Common player feedback is stuff like “The AI is rigged, he consistently gets over 1000 points on his turn”. Chances are this player was facing the Reckless AI, who tends to score very high a lot of the time and Zilch almost as frequently. He plays recklessly, after all.

New players tend to play flash games on easy until they get a feel for it. They expect an almost sandbox style of play, where they can try and grasp the game concept without having to suffer a humiliating defeat. It’s no surprise that people who play games really really like to win at them. Joining this notion is a Kongregate achievement centered around “winning 3 games on any difficulty”. For the achievement junkies that need to get those wins ASAP so they can move on to the next achievement, it makes sense to play the game on easy.

All of you are wrong because I’m right

The developer of Zilch, playr.co.uk, issued a response to all the players complaining. You can read that response here (response 1). In summary, they explain how the three AI’s work and that all the dice rolls in Zilch are fair from a random number generator. Clearly this wasn’t enough because later that day they made a more frantic response that you can read here (response 2). This time they used all caps and posted their Random code for players to audit. It is indeed random. You can almost imagine their inbox flooding with complaints. They’ve even made another response that you can read here (response 3), but it’s more of the same.

Finding the real problem

So what’s the problem here? Zilch’s developers are unable to get to the crux of the player complaints. They chose to instead defend their design choices and attempt to educate the player. Unfortunately they didn’t do this inside their game, but at their obscure company blog that maybe MAYBE 1% of players will actually read.

I will now try to lay out exactly what is happening. A player starts up Zilch for the first time. There’s no introduction or explanation, so when they pick a CPU opponent they choose the one that should be easy; mostly because they have no idea what they’re in for. “Reckless, well that’s never a good thing and it’s on the left… so it must be easy.” The game starts and players are educated while they take their turns. While this education gives a fairly good grasp of the game rules, it does not talk about strategy (and it shouldn’t). Unfortunately for the player, the Reckless AI has been employing a basic yet more effective strategy the entire time the player was in tutorial land. By the time the player is ready to take off auto-pilot, there’s a good chance the AI has the lead. As the game continues, players will notice a pattern in the way Reckless plays. He either scores large amounts of points on his turn, or he gets zilch. With no real understanding of the game and a more cautious playing style, the player will be taking smaller sums of points each turn and zilching more, simply because they haven’t yet figured out the game’s probabilities.

After reading what I would consider a typical first time experience with Zilch, is it hard to see why players send frustrated emails to the developer?

Actually making players happy

It turns out that the complaints about the random number generator are red herrings. Players don’t care about how the game works, all they care about is that their game expectations were incorrect and they were punished for it. What the Zilch developers need to do is stop trying to convince players that they were wrong. Not only will that not work, but the players are almost never wrong.

First, create a new difficulty and call it something recognizable like Beginner. Next, and this is the super important part, skew the dice rolls in favor of the PLAYER on the Beginner difficulty. Create an AI that purposely makes poor decisions. The goal here is to create an AI that approximates a new player who has absolutely no understanding of the rules let alone the strategy of Zilch. And then give that AI permanent bad luck. Now players will have a place to go to when one of the real AI’s hands it to them.

The next thing I would do, is create yet another difficulty, called Normal. This AI should contain the same lack of strategy that Beginner has, but the dice rolls are fair. Zilch has an interesting ‘problem’ in that the easiest AI to create turns out to be very competent. Typically, AI competency is directly proportional to its complexity, but in Zilch it will take more time to write an incompetent AI that is fair to new players.

Finally, take your three existing AI’s: Reckless, Cautious, and Realist, and put them all under a Hard difficulty heading. Depending on your strategically advanced opinions of Reckless and Cautious, you might want to revisit their AI’s to make them stronger. The idea is that these three AI’s are employing real Zilch strategies. Sure some are more simple-minded than others, but every single one of them is better than a new player. That’s your problem.

16 thoughts on “Misreading the Player Feedback Loop

  1. Thomas

    I looked at your portfolio and checked out your games on Kongregate. I played Battalion Nemesis, It’s been weeks if not months and I even remember it. I liked it and think it was good, so I’m not dissing you, but when you hold up a game with 4 stars like Zilch (actually 3.95) to such criticism I think it’s relevant to have some context. Here are your games, and ratings…

    My Sweet 16: School Musical 2.13
    My Sweet 16: 2.24
    Critter Cannon: 2.64
    Tiki Balls – Curse of Tane: 3.0
    Zombieland: 3.07
    BattleMachy: 3.23
    Discarded – Online: 3.93
    Battalion Nemesis: 4.18

    Like I said, I liked Battalion Nemesis, and I’m not trying to diss you here. I think 4 stars is a great score. But you seem to be saying it’s not adequate and that if you get 4 stars you’ve left “low hanging fruit” on the vine. So can you explain how exactly 4 stars is such a poor score that the developer of Zilch should feel like he’s somehow failed to properly serve or respect his players when your top score is less than a quarter point above his and most of your games are far far below?

  2. Andrew Pellerano Post author

    Sorry, but I never said that I’m perfect. Going back and looking at the scores of games I’ve worked on is kind of neat, it shows a steady growth in the strength of the games I’ve been involved in. And yes, every game is always going to have low hanging fruits for a better score. Despite Battalion: Nemesis having a 4.18 I believe it could have done better with a few key fixes.

    4 stars is a good score, good enough to put you in the monthly standings depending on the month. But all the greats (the ones with millions of plays), they are noticeably higher than 4 stars. Zilch has a good score because it’s a good game. With the right modifications it could have been more accessible which I believe would have lead to a higher score.

    It’s worth mentioned however, that I don’t believe a Kongregate score is the end-all indicator of how good your game is. It’s simply an indicator of how well your game is liked by people who go to Kongregate.

  3. Thomas

    To me we aren’t debating whether Zilch is a good game or not. We are debating whether it is so bad that it deserves to have it alleged that the developer has no respect for it’s players as your title suggests. This game in no way deserves to be made an example of anything poor. Including “Misreading the Player Feedback Loop” which as I’ve noted is grossly unfair considering the guy modified the game in direct response to feedback, explained his algorithms and AI strategies, and even released his source code for review. The only things he could have done further would have been including an in game strategy guide that explains every facet of statistics and winning tactics, or rigging up an AI strategy that deliberately loses games to first time players. Neither of which I think is reasonable or necessary… and neither of which I think would have increased his Kongregate ranking or the overall experience of the average player.

  4. Raitendo

    I couldn’t have agreed more to the points you made in your post. Thomas and Jim’s points are hard for me to grasp since what you’re asking for is in no way a dumbing down of the game, as you wouldn’t actually remove any of the existing modes. Which… is very obvious, and should go beyond saying.

  5. Andrew Pellerano Post author

    There was an article at Gamasutra today about this very topic.

    http://www.gamasutra.com/view/feature/3947/intelligent_mistakes_how_to_.php

    From the article:
    “As I found with the “positional play” in snooker, random outcomes that happen to favor the computer are perceived as being intentional. If the ball ends up in a good place, or the poker AI makes a lucky call and wins on the river, it can be perceived as unfair or even cheating.”

  6. Jesse

    I like all of Andrew’s points and suggestions.

    This is going to seem irrelevant but if you hear me out I think you will find that it isn’t. When going through the tutorial, you are encouraged to roll and then choose from the options that pop up. For example “Two 5s: 100 Points”. At first I thought you could only ever choose exactly one of the popped up options and were then required to roll again or bank. As I played more I found that I could choose multiple popped up options if I wanted to, as long as they didn’t “overlap” dice. For me it was an “Oh I’m just an idiot” moment.

    Anyway I think it’s natural not to watch the computer like a hawk, but I noticed that the computer did, in fact, seem to have overall better luck than me, despite my having good confidence in my strategy. Then I started watching very carefully and I noticed that the computer would frequently roll “Two 1s: 200” or “Two 5s: 100”, yet only one of the two dice would become “locked” in place.

    I flipped out. I started noticing it happening all the time and filed a bug report. It turns out that it is not a bug because the player can do so as well by individually clicking on the dice along the left. Something that was NOT addressed — let alone clearly explained — anywhere in the tutorial nor the rules.

    Furthermore it is, as I see it, a huge advantage to be aware of this option, as choosing one five instead of two fives can vastly improve your chances of avoiding a zilch.

    So if I did have constructive criticism to give, I would say that the tutorial and the instructions need to be clearer. Something that would be VERY easy to do and wouldn’t change the game at all.

    Without having the advantage of making the best choices (and probably never noticing that the computer players are exercising options that we aren’t), it would explain why the game seems unfair, or favoring the computer.

    My second point that I want to make is that I never received a response to my bug report, so I started looking around the sites more. I found the explanation about “Why the computer player sometimes only chooses one die.” This note by the game author explained everything in what I viewed to be a tone that implied we should have known all along, ending with “Please stop sending bug reports about it.”

    Obviously if so many reports are being sent, all of these complainers did not understand the rules of the game, so I think the appropriate response would be to explain the rules better in the game instead of making an otherwise unnoticeable link to an explanation somewhere at the bottom of the game description.

    The developer’s reaction left a bad enough taste in my mouth to write this long post about it. So, there you have it — perhaps the game just seems so unfair because we’re not informed enough to play with a good strategy!

Comments are closed.