Concept testing

For qualitative concept tests - that is, those where you will get rich data from players who speak aloud as they view your concepts and materials - you can get useful data from as few as five players. Use the same amount again for every extra target demographic you include. For primarily quantitative or survey-based studies, aim for larger numbers: at least 20 per group, but possibly more based on the specific questions you use.

Data from players who speak their responses and thoughts aloud provides rich insight into their perception and experience of your materials, but is not always helpful in terms of learning what they actually prefer. In these cases, consider supplementing your study with survey questions that provide direct comparisons between the different materials.

Need a quick round of player feedback on some preliminary materials, or a sense check on whether a new character design fits the game? Use a qualitative concept test with a small group of players, iterating and repeating as necessary. If you are looking for more representative feedback that is likely to replicate the responses of a wider player group, use a quantitative survey, and send it to a larger number of carefully-recruited players who are in line with your target demographic. If on the other hand you land somewhere in the middle, you may want to get rich feedback and insights into a pack of concept materials. Here, use a concept test that includes static and audiovisual materials, supported by a quantitative survey. Your analysis should naturally tie the spoken and survey responses together.