See Page XX: The Gentle Art of Playtest Feedback Wrangling

See Page XX

A Column About Roleplaying

by Robin D. Laws

A while back Cat asked me for guidance on an unheralded facet of tabletop RPG production, the gentle art of collating feedback from a group of playtesters into a single document of greatest use to the designer. After writing it up I figured that it might be more generally useful to budding line developers. They perform a tough job without access to as big a pool of advice to draw on. So I polished that memo up, inserting some strategic diplomacy, and here you have it.

As Cam Banks did a killer job assembling playtest feedback for Feng Shui 2, an alternate title for this piece could be “Cam Banksing Your Way To a More Efficient Playtest Feedback Document.”

Perennial playtesters could reverse-engineer this advice into guidelines for providing more effective feedback. However, if you fit that description, you’re worth your weight in gold already. Just keep on doing what you’re doing, and let the developer and designer worry about turning your reports into design and presentation changes.

The “you” found below refers to the line developer. The designer is either a singular “her” or a plural “us”, as context dictates.

The number one most useful thing you can do in assembling a feedback document is simply to group all of the comments in order, by chapter and major subject breaks within each chapter. Depending on how the designer laid out her manuscript, breaking it down to Header 1 categories ought to do the trick. (If your designer hasn’t formatted her document with headers, you need to have a talk with her about that.)

This ordering process alone saves the designer a ton of time. She now won’t have to jump around randomly in the manuscript as she addresses issues from various groups in sequence. Ordered collation allows her to consider possibly opposed views on particular issues at the same time.

The second most important task the developer can perform is to strip comments of any emotional petitions playtesters are making of the designer. Boil them down into tonally neutral observations of actual problems encountered during play.

A natural disjunction exists between the desires of playtesters and the needs of designers. People like to have opinions and to feel that they’re being heard. They want to feel their impact on the process when they read the final product. The designer, on the other hand, wants to drill past opinions into descriptions of experience. Here the line developer jumps in to reconcile those two divergent requirements.

In the able developer’s hand, “We hated hated hated the scuba diving rules. They were too complicated and disrespected the glorious field of oxygen tank repair” becomes “One group disliked the scuba diving rules, finding them too complicated.”

As developer, you will also find yourself encapsulating “It’s irresponsible in this day and age not to include a full character build system” as “one group wanted a full build system.”

By doing this you allow the designer to skip the cognitively costly step of processing the playtester’s unhappy emotions, moving straight on to fixing the problem, if indeed she finds that there is one.

Conversely, the stripping of pleas and demands from the original context prevents the designer from dismissing a valid concern because the respondent couched them in an off-putting way.

In the typical playtest, count on one group to vehemently reject the game’s entire premise and all of its attendant design goals. For a new iteration of an existing game, it is not unusual to get a group that asks for alterations to established elements of the core system that work perfectly well and are not up for grabs in the current playtest. You can safely drop these from your feedback report.

Sometimes the first class of objections, reframed in emotionally neutral terms, help the designer write the expectations management sidebars that explain why the game works as it does.

Now and then you’ll get feedback from groups expect the rules to serve their very idiosyncratic play styles, or to solve issues concerning their specific problem players.

“These rules don’t constrain Randy nearly enough. You know, Randy! He’s a jerk but he drives the rest of us to game.”

“When we heard of your English parlor mystery game we were really hoping for a rules set that fuses our favorite parts of Nobilis and Rolemaster. Your investigative game could still be that if you added fifty pages of combat results charts and dropped mystery solving for mythic interaction. And we’re going to keep emailing you about it until you see how important this is. Because everyone else must want that too.”

As developer, your job is to run interference, keeping your designer focused on meeting her design goals and undistracted by passionate campaigning to deviate from the premise. After all, it might be you who assigned her this remit in the first place.

When you do include a bit of feedback you find off-base, flag it as such. The designer can enjoy a chuckle and keep going.

Playtesters naturally find it way easier to spot problems than to call out the segments of the game that already work. When a group does do this, it is helpful to know, so the designer doesn’t drop a thing most groups have success with in order to satisfy a problem had by a few.

Whenever possible, indicate how widespread a particular issue is among groups. “One group found the procedural rules too complicated” calls for a very different response than “Two thirds of the playtesters found the procedural rules too complicated.”

You can safely omit another common strain of feedback: “We didn’t like the look of that rule so we made up our own and here’s what happened.”

If you sense, or are told, that comments are based only on a reading of the rules, throw them in the garbage. Do not waste your time sifting them for pearls. They’re guesses at what might happen at the table. Plausible-sounding guesses are the worst, as they can prove deeply misleading. The designer needs to hear what actually happens when rule X or Y hits the table.

Proposed solutions to design issues are almost always unhelpful. Nine tenths of the time they add additional complexity without taking the whole of the rules engine into account. Share them only if the designer asks.

As a designer, I’d rather just see the problems: combat was too slow, we had a TPK in the first scene, the players refused to get out of the starship once they realized there were Class-K entities on the planet, we found the procedural rules too hard to learn, the character with telekinesis outshone everyone else, the example of initiative doesn’t agree with the rules text, the arithmetic is wrong in the Fleeing example.

Specific notes on the balance of particular crunchy bits are quite helpful, even if I wind up disagreeing with some of them. Here theories by someone who has actually played the game but not seen a certain hosy combo come up can in fact be useful.

Comments rendered in the language of game theory or general philosophizing are of almost no use, except in expectations management.

Some groups really love composing detailed write-ups of what happened in their games. I’m always abashed at the work that goes into these, because I have to admit that I skim them at best.

One big exception: if play write-ups list what character types got played (in a game that has them), and you can collate those, that’s very useful. Here we might discover that every group has a hobo in it but none of them have professors, in which case we might want to buff up the professor because this is Trail of Cthulhu, dammit. We might also want to determine if people just really love hobos, or if we’ve accidentally assigned them twice the build points other starting PCs get.

Organizational complaints always arise in playtesting but are devilishly hard to evaluate. A raw manuscript without an index and page numbers is hard to learn from. Lacking the mnemonic qualities of art placement and layout, a work in progress is by definition a mess. Many observations stem from not being able to find a rule in a raw document. This one you simply have to expect and hope to address during production.

2 Responses to “See Page XX: The Gentle Art of Playtest Feedback Wrangling”

  1. Pete Darby says:

    “Proposed solutions to design issues are almost always unhelpful. Nine times of the time they add additional complexity without taking the whole of the rules engine into account. Share them only if the designer asks.” – Robin Laws

    “When a reader tells you something doesn’t work for them, they are always right. When they tell you what you should have done instead, they are nearly always wrong.” – Neil Gaiman on beta readers of early drafts.

  2. […] The Gentle Art of Playtest Feedback Wrangling – Robin D. Laws […]

Leave a Reply