How will you know if the users or participants in your project will be impacted in the way you hope?

Most of the time as teachers we know that our students have learned what we desire them to learn when we test them or set them a performance objective that they accomplish. However, with a game design project, where the objective may be a little more complex or less concrete, how do we know when they have been impacted in the way that we would hope?

I can only speak to my own game concept, which I've talked about before. I am interested in developing a role-playing game that follows along with the concepts we teach in a course here in BC called Planning 10. It's a career prep course combined with a bunch of other life skills, including financial planning, educational planning, family life -- it's basically a dumping ground for subjects the government thinks aren't being covered anywhere else.

Simply put, I believe that the majority of the curriculum of the course could be more effectively addressed with a role-playing game, where every student assumes a character and takes that character through the scenarios that mimic the decisions they will be faced with throughout the next few years of high school and post-secondary. An important factor to me is the ability for the students to reflect on what path their avatar's life has taken, the possibility for them to have a "do over" and choose different paths, explore the different results with less risk.

In a game such as this, reflective journalling would perhaps be the only real gauge of whether or not the participants have been impacted in the desired way; requiring students to keep a journal of their progress, choices they have made, the results of those choices, "do overs" that they have opted to explore and their end results, etc., would allow me as the instructor to track the students' development of mature decision-making skills, which are the ultimate objective of the process.

I could set an arbitrary goal of a certain income, but that negates life choices students might make that focus not on the accumulation of wealth, but on other issues that the students may deem more important. The same goes for academic accomplishment and post-secondary education, as these are choices not all students would make. The objective is rather to get them to focus on the importance of the decision-making process, allow them to explore the possibilities in front of them in a safe(r) environment, and possibly to take risks that they might otherwise not take.

Perhaps the best gauge of impact? My tendency would be to have the students write a brief summary of their life/career/education objectives at the beginning of the course; then at the end of the course, I would have them revisit the original objectives and assess if and how those objectives have changed as a result of their activities on the game. The comparison between the two pieces of writing should provide ample evidence of whether or not there has been an impact.