Educational Game Ratings
Educational Game Ratings
Educational Game Ratings
Spring 2009
Standardized Educational Games Ratings: Suggested Criteria
Video games have evolved greatly over the years, becoming a more complex and
effective interactive media and forming a highly popular and expansive industry; one that is
likely to continue to grow at a significant rate. In 2007 alone, game software sales grew over six
Capitalizing on this increasing popularity, educators have the unique opportunity to use
many components of game design and apply them towards a curriculum which utilizes game-
entertainment games. The idea of motivation, for example, can change the sometimes
lamented prospect of mastering a new subject into a player-initiated experience, where learning
occurs simultaneously with engaging game play. Harnessing the compelling and immersive
nature of games to teach subjects is no small feat, however, and the promotion of these games
is hindered by the lack of a consistent and reliable rating system. With no effective method of
distilling a game down to its educational content, parents and educators may find it hard to
In order to propose a new rating system for games with educational content, it is very
useful to look at ratings systems already in place. Entertainment games are currently rated
according to the Entertainment Software Ratings Board, or ESRB. Most of the current game
development studios submit their games to the ESRB for ratings (Entertainment Software Rating
Board, 2008), and some will even modify their in-game content to acquire a desired rating. This
rating can be found on the back of almost any commercially-available title. This rating system is
content-based, and categorizes the objectionable content of a game into age-specific ratings. A
game with a heavy violence factor may be given a 'M' rating, or over 18 years old
appropriateness, whereas a game with only comic-style mischief might be rated E for everyone
Games with educational content could very well be included within the ESRB's rating
structure, but the benefits of the current system would be diluted by the rating structure's lack
of attention to learning principles. In other words, games that define themselves as educational
would be graded solely on their amount of objectionable content, and not by educational
criteria. As few educational games have objectionable (as defined by the ESRB) content within
them, does an objectionable-content-based system make much sense in this context? We need
a rating system more tailored to the unique learning environment of educational games; one
which can simultaneously assess their educational potential and age relevance, allowing
The ESRB employs several subject-matter experts to help in their game rating process.
These experts play through parts of a game to determine whether the objectionable content
descriptors and examples (submitted by the game developers) matches their own proposed
rating for the game. Subject matter experts, in this case, are defined as, "adults [who] typically
have experience with children, whether through prior work experience, education or by being
While the ESRB's approach may be sufficient to point out objectionable content, any
ratings system designed for games with educational content should include subject-matter
experts with significant understanding of educational theory, game design, and possibly
psychology. It would help to have a mix of gaming experience levels across the subject matter
experts, as perspectives will be needed both from the gamers' standpoint and from the
Typical game-rating sites are primarily concerned with a game's technical presentation
and the level of enjoyment it provides. Although the staff at such rating sites are accomplished
gamers, they likely lack the expertise required to properly evaluate the effectiveness of
educational games. To provide meaningful and valid assessment of these games, education
Currently, there are no official standards or set of criteria that games must meet before
"educational" games can claim to be educational. In fact, one of the new markets opened by
Nintendo is the "brain training" line of software, which claims to enhance your brain's function
with continued play. There is quite a bit of controversy around this assertion: "A survey of ten-
that users can test and rejuvenate their grey cells" (Fernandez, 2009).
Since Nintendo is actively promoting their Brain Age games as educational, this brings
up a potential issue. If educational content were regulated in the same way as mature content,
then Nintendo would be required to submit supporting documentation about any educational
claims. A list of best educational outcomes, much like the ESRB mandatory objectionable-
content reports from the entertainment game developers, could validate their claims. An
educational games rating system could take this responsibility from the developer, allowing an
unbiased and specially trained group of raters to define any such content.
The current lack of oversight has allowed educational games to vary widely in
instructional value, usability and engagement. An objective set of standards would enable
schools and parents to determine the educational value of a game prior to purchase.
Standardized criteria would also allow consumers to compare several titles and choose the one
To facilitate the development of standards for the evaluation of games, a rating system
should be designed that addresses criteria relevant to education. These criteria include overall
The idea of a universal rating system for educational games is not a new one. One of the
most prominent educational game review groups is TEEM, located in the United Kingdom
(TEEM, 2008). TEEM accepts submissions from publishers and, after a fee is paid, reviews the
game by combining traditional text reviews with videos. These reviews are then uploaded to
their website for public perusal. TEEM's review staff contains a rotating group of subject matter
experts, most of which are school teachers. TEEM asserts that they are now, "the biggest
While the goal of groups such as TEEM is admirable, the implementation of their review
process, how they determine which games to review, and their end result of verbose,
multimedia presentations do not lend themselves well to either the desired objectivity or the
time constraints that a more universal system would have to adopt. A fee-based review
structure undermines the objectivity of any reviews produced, and is wholly incompatible with
the notion of a standardized, universal educational game rating system. The ESRB is a non-profit
As the trend of using educational games in instructional contexts continues, the need for
standardized ratings will only become greater. We suggest the following criteria as relevant to
Overall Value
The notion of value in entertainment games is centered around total length of possible
game play, replayability, online options and cost. However, value in educational games should
be measured differently to reflect more context-relevant concerns. In this case, the total length
of game play may not be as important as the length of a typical play session, the total of
available lessons or learning modules, and the length of the learning curve.
As teachers and parents strive to make the best educational choices for their children in
versus retail cost will be very effective. Value can affect educators looking at various programs
to implement in the classroom or computer lab, where the priorities might be low cost and
effective short-burst game play sessions. Value can speak to the home consumer, who will be
In this case, value will refer to educational content (subjects covered, and how
effectively), final retail price, and average session time. Value will be prominently featured in the
final rating system to help address some of these concerns.
Usability
The most compelling and enriching educational software in the world would have its
important category for reviewers to consider so that they can more accurately inform the
consumer about the actual age appropriateness of the software, and can point out any
imbalances inherent within. For example, a game traditionally rated E for everyone by the ESRB
might have little objectionable content but, due to bad user interface design or disconnected
and scattered learning modules, may be very inaccessible to the very students to which it is
marketed. Bad design could make a particular piece of software too complex for its intended
Usability ratings should be included in the proposed rating system to alleviate some of
the frustration encountered when the design of a game does not match with its audience, and
Accuracy
In order to effectively teach the player a subject or skill, the content of an educational
game must be accurate. Accuracy is determined by how successfully the game models the
reality of the scenario it attempts to represent (Feinstein & Cannon, 2002). Content or
mechanics that are inaccurate may lead to impaired comprehension, defeating the player's
learning goals and the purpose of the game. For instance, in the Civilization series, any culture
can become a nuclear power, even the Mayans and Romans. Conclusions drawn from game
Appropriateness is determined by how well the game's content and design are adapted
to the target user's needs. While the ESRB rating reflects the maturity level of a game's content,
it does not consider how well the presentation and mechanics of the game, from the menu
structure to the controls, consider the abilities and requirements of the user. Both elements are
important, in that one without the other could make the game unsuitable or unusable for the
player.
Although educational games aim to teach players a specific set of skills or knowledge,
additional non-content-based outcomes may be derived beyond the primary content. All games
teach players something; what varies is the usefulness of the knowledge outside of the game
context (Koster, 2005). Learning the names of all the regions of Azeroth, for example, may not
be very useful to a player outside of World of Warcraft; however, the game also allows the
player to practice transferable skills such as reading a map, recalling directions, and managing
resources. By engaging in a game, players may also learn to become participants or even experts
within a semiotic domain, a system with its own unique grammar, conventions, and ways of
communicating meaning (Gee, 2007). Learning how to speak the language of a domain is an
important skill that players will draw on continuously as they progress through different
Because of the variability among games and their players, it is difficult to quantify non-
content-based learning. A player with one style of gameplay may not derive the same outcomes
as another, and games themselves differ in terms of the skills that they teach. Therefore, a
special note will be made for each rated game, listing potential non-content based “side-
benefits” that lie outside of the main learning content. This list could be standardized to include
elements such as team work, logic, communication, social organization, and others. Novel
elements could also be added to the list when they are presented in a new game.
Relevance
Relevance is determined by how meaningful and applicable the game's content is to the
target user. To be relevant to players, the system must assist them in achieving their learning
goals (Driscoll, 2005). Injecting relevance into the game makes the content more relatable, and
therefore more meaningful and significant to the player. Whenever possible, an educational
game should ground the instructional material in a context that the player is likely to
understand; for example, games directed at children should present lessons in terms and
situations that are relevant to children, such as attending school or playing at a park.
learning context. With Civilization, for example, the game is appropriate for use in history
education, as long as the educational elements are emphasized over the fantasy elements.
Although this task would be the responsibility of the instructor or parent, the ratings system
Objectives
Objectives are also an important part of game design, and are especially critical for
educational games. Objectives define the player’s goals and the criteria that determine success
or failure. For entertainment games, objectives may include collecting fifty stars, saving the
princess, or eliminating all enemies from a level. In educational games, the objectives are tied to
learning outcomes. For example, players may have to correctly solve math problems in order to
earn fifty stars, or accurately diagnose and treat an illness in order to cure the patient.
Games that lack clearly-defined objectives will be frustrating for players, because they
will not know what is expected of them. Assessment of the player’s performance is also difficult
without objectives to compare it to. Educational games should have clear, appropriate, and
assessable objectives.
Feedback
To help the player understand how their performance compares to the game’s
objectives, feedback should be provided. The feedback may be as simple as a sound effect or
score indicating success or failure. Educational games often require more detailed feedback,
however. Players should not only know whether or not they achieved their objectives; they
should also know why or why not, as well as how to improve. Instructional feedback should help
the players learn from their mistakes by coaching them to the correct response (Driscoll, 2005).
Engagement
For an educational game to be successful, players must derive enjoyment from playing it
and be willing to progress through the game to pick up new knowledge and skills. These
requirements map to the factors of engagement and motivation (Virvou et al, 2005).
Engagement refers to the level immersion a player experiences, or how well the game
makes the player believe that they are actually experiencing the simulation (Paras & Bizzocchi,
2005). Highly engaging games encourage the player to keep playing, which in the case of
educational games, also means that the player will have a greater opportunity to continue
(Medina, 2005).
The features of games that make them engaging to players are under debate. Even game
developers have difficulty quantifying and applying the factors of engagement, as evidenced by
the number of unsuccessful games produced each year. Garris et al (2002) suggest that games
are appealing because they incorporate elements of fantasy and mystery, have clearly-defined
rules and goals, stimulate the players’ senses, provide an appropriate level of challenge, and
allow players to be in control. For educational games, these features mean that the player’s
experience is interesting in terms of content and presentation, and that the game is easy to
If a game is too difficult, players will become frustrated and be unmotivated to continue
playing. Games that are too easy or too quickly mastered, on the other hand, lead to boredom
and lack of engagement. Well-designed games provide just enough challenge to keep the
players interested. Instruction often follows a similar pattern, with students struggling to grasp a
concept, mastering it, and moving on to the next challenge. Ultimately, if the game and player
are successful, boredom is inevitable, as the player will have mastered all of the challenges
(Koster, 2005). Until mastery is achieved, however, game designers must find a balance between
boredom and frustration in order to maintain engagement. Because this aspect is so critical to
the instructional quality of the game, it will be addressed by the rating system.
Also critical to engagement is the factor of rapport. If non-player characters are present
in the game and interact frequently with the player, particularly if they fill the role of mentor or
guide, it is important that the character establishes a positive relationship with the player.
Gratch et al (2007) assert that virtual agents can be designed to build rapport through simulated
attentive body language, thereby facilitating communication and learning. This process is aided
by the tendency of players to empathize with agents as though they are real people, and feeling
While engaging characters encourage playing and learning, unappealing agents could
have the opposite effect. Players are unlikely to spend hours learning from characters who are
agents carefully and with consideration for their target player. In the case of exceptionally
engaging or repellent characters, a special note will be added under the Engagement rating.
Motivation
Motivation may be the most important characteristic of effective learning (Nadolski et al,
2008). Without motivation, the player is reluctant to play the game, and unlikely to benefit from
the instructional content and strategies. Effective educational games must therefore attempt to
provide motivation. Strategies for motivating the player may involve the setting of goals or
objectives, providing scores or feedback, and issuing challenges and rewards (Medina, 2005).
Engagement and motivation are significant factors influencing the player's willingness to
play an educational game. Educational games must employ strategies for engaging, motivating,
and establishing rapport with their players. Games that are enjoyable and motivating will score
Some publishers have attempted to expand the educational content of their games by
consulting with educators and formulating lesson plans to accompany all or part of a particular
step-by-step instruction for educators using Droid Works to teach the physics of pulleys in the
classroom. These instructions effectively illustrate and isolate the subjects or theories to be
learned, how to use the software within the classroom format, and how to blend that software
use with physical lessons and examples. In this case, should the games presented be
reevaluated within the context of the publisher's enhanced support, or should a separate rating
exist for publishers that continually go above and beyond to enrich their software?
rated, or should it only be noted that such additional material exists, and should be reviewed by
the end-user? To maintain our focus on the educational qualities of the reviewed games, we
feel that it is important to list additional materials, if available, but to restrict our site's
We propose that game publishers be rated in a separate section that details their level
of consistency in software quality, and their after-market support (if any). This separate rating
would allow parents and educators to tell, at-a-glance, if they can expect additional content in
Gaming Literacy
Not every person who plays a game can be expected to immediately grasp the
fundamentals of the controls, the flow of game play, and the expectations placed on the
educational supplements (as has been done in some classrooms with the Civilization series), the
learning curve can present a serious issue when trying to get students motivated. To someone
who does play games, but who has never played a simulation game, the basics of game play
may be completely foreign, which might force the instructor to devote more time to teaching
the fundamentals of the game itself, and not the desired educational content within.
A ratings section on video game and genre familiarity could point consumers to the
correct title for the correct audience, marking games as truly for the novice, or as requiring
For game ratings to be valuable, they must be objective. To maintain the objectivity and
integrity of the rating process, a number of policies should be enacted, influencing the funding
of the project, communication with the community, and the distribution of rating results.
As discussed in the Current Models of Review section, the TEEM model is dependent on
the willingness of developers to submit their games for evaluation; if too many negative reviews
are handed out, it is unlikely that the developers will continue supporting the initiative.
A similar issue has affected sites that allow developers to purchase advertising space. In
one particularly publicized instance, Gamespot editor Jeff Gerstmann allegedly lost his job after
issuing a low score for the game Kane & Lynch, whose publisher, Eidos, advertised heavily on the
Gamespot site (McWhertor, 2007). This incident underscores the importance of establishing a
ratings model that allows the reviewers to remain objective, and not dependent on the game
To avoid this issue, a serious games rating site could be independently funded through
educational technology grants. The US Department of Education and the MacArthur Foundation
for Digital Media and Learning are two potential sources for funding. Both organizations provide
millions of dollars a year for research into topics of educational technology and games. Although
maintaining a long-term project through grant funding is a tenuous proposition, it could be the
best way to ensure the objectivity and respectability of the serious games rating site.
depends on the ratings should know that the reviewers are competent, that user concerns and
suggestions are taken seriously, and that they have access to the full results of every review.
Providing the full report to users is a vital step in promoting transparency, eliciting feedback,
and increasing the number of rigorous evaluation studies available in the field of educational
games.
Accessibility
with Disability Act (ADA) requires public schools to provide every disabled child with an
means that schools must tailor instruction on an individual basis, depending on each student’s
abilities, ensuring that learning is maximized while hardship and frustration are minimized
Regrettably, accessibility is a low priority for most game developers, due perhaps to a
lack of awareness or to limited resources (Zahand, 2006). Developers also want to maximize the
appeal of their games. Accommodating the needs of a minority of players with a particular
disability may mean diminishing the game’s usability or engagement for a majority of players.
Although the major development studios have largely neglected accessibility, a handful
of independent games have been designed for players who have visual, auditory, or mobility
multimodal forms, such as subtitles for dialog and auditory cues to accompany images, to avoid
favoring one sense over another. Research into universally accessible games is ongoing.
At a time when few games consider accessibility, including it in the ratings system would
only lower the overall scores and harm otherwise very well-constructed educational games. For
this reason, accessibility should instead be highlighted in a separate category that addresses any
considerations a particular game has made, and to which players the game is best-suited.
The ESRB currently has a system for rating entertainment games, the majority of which is non-
applicable to educational games. The rating system in place uses a letter and possibly a symbol,
along with text keyword descriptors, to rate the objectionable content in games. The current
system uses EC for early childhood, E for everyone over 6, E+ for everyone over 10, T for teens,
M for mature, and AO for adults only (Gentile et al, 2005). The idea of shortening what can
easily be a verbose and in-depth analysis of a game to a letter and a symbol is appealing from a
consumer's standpoint (in that it makes a game easy to assess at a glance), but difficult and rife
with potential problems for the ratings board. The challenge here is to create a scale that is
easily understandable by a wide range of consumers without sacrificing too much flexibility,
not require a supplementary key to decipher. We propose that educational games be rated on
a scale that is already culturally recognizable, unmistakable, and already associated with
education. Individual criteria within the review should be rated on a A-F scale, with A being the
highest level of accomplishment. The average of these ratings will determine the Overall Value.
Using this scale helps our rating system to be recognizable within the context of educational
content, and in essence, we are "grading" the game according to its merit.
Example Rating
In Where in the World is Carmen Sandiego? (Broderbund, 1985), the players are put in
the role of a detective tasked with locating and arresting the notorious master thief, Carmen
Sandiego. Players must interview witnesses and collect clues to determine Carmen’s
whereabouts as she travels across the world, stealing treasured landmarks. The primary learning
content of the game is geography, which the players are exposed to as they track Carmen’s
Overall Value: A
The game overall is engaging, appropriate for a wide audience, and most importantly,
achieves its instructional goals. Players are provided with clearly-defined objectives and
The game’s interface and rules are easy to learn and simple to use. Special Note: all
conversations with characters are subtitled, so players with hearing impairments do not miss
Accuracy: B
When the game was first released in 1985, it accurately depicted the geography of the
time. Since then, countries have been renamed and the lines on the map redrawn, so the game
Appropriateness: A
Although this game is unrated by the ESRB, it would likely receive an E for Everyone
rating due to its mild content. The comedic situations, visual style, and challenge presented by
the game are appropriate for the pre-teen audience for which it was produced, but would likely
Sandiego, players not only learn geography; they may also practice time management and
Relevance: A
The knowledge taught by the game is relevant to the target audience. The knowledge
taught by the game is important for the pre-teen age group, which is required to learn
geography in school; also, because of the wide range of locations covered, a city near the
player’s hometown is likely to come up, making the game more personal and meaningful.
This game is appropriate to supplement geography lesson plans, or for at-home practice.
Objectives: A
The game clearly presents the player’s objectives at each stage of the game, listing
Feedback: B
Feedback in the game is limited. Players are encouraged to try again when an incorrect
decision is made, but are not given corrective feedback explaining why the choice was wrong.
Motivation: A
Though trial-and-error will allow players to determine Carmen’s last known location
without employing the clues that should have led them there; however, relying too much on
this tactic will allow time to elapse and Carmen to evade capture. Players are therefore
Engagement: A
The game employs eye-catching graphics and animation (for the time), comedic writing,
Conclusion
rating system is no small task. Though the ESRB does not address the concerns of educational
games in a substantial way, they have managed to raise their awareness among consumers,
force compliance from the developers, and, as a result, they are now the standard for
It is not this paper's intent to have a fully-functioning business model that simply needs
funding, but rather, to raise awareness and suggest some possibilities for the implementation of
a universal review method for educational games. At a time when the benefits of computer-
assisted educational programs are being researched, overall familiarity with games is growing,
and the ubiquity of computers within the classroom is solidifying, the need to educate the
consumers about the benefits and value present in many of today's educational games is
becoming all the greater. We feel that now is the time to assemble a regulatory body with the
purpose of reviewing and assessing the content of these games in a meaningful way, and we
hope that some of the ideas presented within are catalysts for the future of educational games
ratings.
References
Americans with Disabilities Act (2009). A guide to disability rights laws. Accessed 24 April,
2009 from http://www.ada.gov/cguide.htm
Bartimeus Accessibility Foundation (2009). Game accessibility. Accessed 20 April, 2009 from
http://www.accessibility.nl/games/index.php?pagefile=home
Driscoll, M. (2005). Psychology of learning for instruction. Boston: Pearson Education, Inc.
Entertainment Software Association. (2008). Industry facts. Accessed 10 April 2009, from
Entertainment Software Association: http://www.theesa.com/facts/index.asp
Entertainment Software Rating Board. (2008). ESRB home page. Accessed 13 April 2008, from
Entertainment Software Rating Board: http://www.esrb.org/index-js.jsp
Feinstein, A. & Cannon, H. (2002). Constructs of simulation evaluation. Simulation & Gaming.
Vol. 33(4).
Fernandez, A. (2009, January 27). Nintendo Brain Age / Training vs. Crossword Puzzles.
Accessed 20 February 2009 from SharpBrains:
http://www.sharpbrains.com/blog/2009/01/27/nintendo-brain-age-training-vs-
crossword-puzzles/
Garris, R., Ahlers, R. & Driskell, J. (2002). Games, motivation, and learning: A research and
practice model. Simulation and Gaming. Vol. 33(4).
Gee, J. (2007). What games have to teach us about learning and literacy. New York: Palgrave
MacMillan.
Gentile, D., Humphrey, J., & Walsh, D. (2005). Media Ratings for Movies, Music, Video Games,
and Television: A Review of the Research and Recommendations for Improvements.
Ames, Indiana.
Gratch, J., Wang, N., Okhmatovskaia, A, Lamothem F., Morales, M, van der Werf, R., &
Morency, L. (2007). Can virtual humans be more engaging than real ones? 12th
International Conference on Human-Computer Interaction.
Koster, Raph. (2005). A theory of fun for game design. Scottsdale: Paraglyph Press, inc.
Lucas Learning. (2009). Learning about pulleys with Star Wars Droid Works. Accessed 27
January, 2009 from Lucas Learning: http://www.lucaslearning.com/edu/lp_droid1.htm
McWhertor, M. (2007). Gamespot editor fired over Kane & Lynch review? Kotaku. 29
November. Accessed 25 April, 2009 from http://kotaku.com/gaming/rumor/gamespot-
editor-fired-over-kane--lynch-review-328244.php
Nadolski, R., Hummel. H., van den Brink, H., Hoefakker, A., Kurvers, H. & Storm, J. (2008).
EMERGO: A methodology and toolkit for developing serious games in higher education.
Simulation Gaming. Vol. 39(3).
TEEM. (2008). TEEM home page. Accessed 12 April 2009, from TEEM:
http://www.teem.org.uk/
Virvou, M., Katsionis, G. & Manos, K. (2005). Combining software games with education:
Evaluation of its educational effectiveness. Educational Technology & Society. Vol. 8(2). P.
54-65.
Zahand, B. (2006). Making video games accessible: Business justifications and design
considerations. Gamasutra. September 20. Accessed 12 February, 2009 from
http://gamasutra.com/features/20060920/zahand_01.shtml