How to optimize early design methods with children?

How to optimize early design methods with children? R.J.W. Sluis-Thiescheffer User Centered Engineering group Eindhoven University of Technology r.j.w.sluis@tue.nl +31 40 2475228 ABSTRACT To operationalize the quality of a design session, it is important to develop a measurement instrument that allows for the comparison of different methods on effectiveness. McLeans [7] design rationale model of Options and Criteria served me as an observation scheme for the initial comparison of two early design methods with children. As the next step in my work I plan to explore the use of metrics for ideation effectiveness as proposed in the context of designing for adults by Shah and Vargas [12] in the context of designing. Early design methods with children are often evaluated on how well the children are involved, rather than on how effective the design session was. In my PhD I developed a framework on the basis of the theory of multiples by H. Gardner. The framework compares early design methods on the cognitive skills they require from the participants. Comparing the skills required by the methods with the skills acquired by children at different ages gives rise to interesting hypothesis about the effectiveness of a method with children. My first experiment showed that involving children in a rapid prototyping explores the design space better in terms of design options than a brainstorming session. In the doctoral consortium I will discuss the expected effectiveness of design methods with children in more detail and the setup of the next experiments. To compare methods I started from a cognitive approach. The theory of multiple intelligences by H. Gardner [3] proposes a model of human cognition by means of eight different intelligences. Although some psychologists think that this theory is difficult to verify on psychological Criteria, the theory is increasingly popular in the field of education in primary schools [3]. That is because the rationale behind the theory is considered successful in optimizing the educational curriculum for children. In my research I will explore whether we can successfully adopt this approach as a framework to compare various early design methods with children. Author Keywords Design methods, children, brainstorming, prototyping, Theory of Multiple Intelligences, Design Rationale, line of research ACM Classification Keywords H5.2 [Information Interfaces And Presentation (E.G., HCI)]: User Interfaces – Evaluation/Methodoly, Prototyping, Theory and Methods, User-Centered Design; A first study showed that with the proposed measurement instrument I can distinguish between the effectiveness of methods on the basis of multiple intelligences in terms of Options and Criteria. I would like to extend this research in two ways. The first approach is to explore whether the hypothesis based on the theory of multiple intelligences will also hold for other methods with children. The second approach is to find more evidence for the framework with respect to the intelligence profiles of children for different age groups. INTRODUCTION The quality of participatory design methods is often measured in terms of the role and involvement during the method, especially in the case of children [2, 10]. However, in the literature there is hardly any evidence that the role or involvement of children is somehow positively correlated with the effectiveness of designing with children. In this PhD I examine the quality of early design sessions with children. This PhD aims at finding empirical evidence for guidelines on choosing effective early design methods with children. In the paragraphs below I first explain the development of the measurement instrument. Next I describe the results obtained so far. Finally I propose two hypotheses I intend to pursue in the remainder of my PhD–project. I would like to discuss that proposal in the doctoral consortium. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. METRICS TO COMPARE EARLY DESIGN METHODS WITH CHILDREN Since early design methods are often used to generate ideas, one measure for optimal performance is how well a design space is expanded and explored. A basic measure for design space exploration and expansion is the number of ideas that are generated per session. The more ideas generated with a IDC '07, June 6-8, 2007 Aalborg, Denmark Copyright 2007 ACM 978-1-59593-747-6/07/0006... $5.00 1 method, the more likely it is that the design space is explored and expanded with known and unknown issues. Hence, the more ideas the generated, the more effective we consider a method to be. Criteria only give an indication of which method might be more effective in terms of Quantity. If these methods also score significantly different on Quality, Novelty and Variety, the comparison becomes more significant and provides designers with stronger arguments to choose one method over another when working with children. Idea generation can be defined as creating solutions for design problems. The best fitting solutions depends on the Criteria for successfully solving a problem. Olson et al. and MacLean et al. [8, 7] found that in the early stages of design, discussions between designers can meaningfully be analyzed in terms of Options (alternatives) and Criteria. Designers explore the design space by presenting solutions (Options) and informally evaluating those Options with questions and Criteria (further referred to as the QOCmodel or Design Rationale Theory). In the above paragraphs I have shown how I compare different methods, and how I can observe the differences between them. In the following paragraphs, I explain how I expect design methods to be different in effectiveness by comparing them according to the theory of multiple intelligences. COMPARING METHODS INTELLIGENCE THEORY In an unrecorded conversation, much of that information can be easily lost. By writing down the mentioned Options and Criteria, designers are set to create a tangible representation of the design space they are exploring. Therefore, if we can score Options and Criteria in a conversation with children, we can create the children’s representation of the design space. The larger the design space, the more effective we consider a method to be. Hence, by scoring Options and Criteria we can compare the outcome of different methods for idea generation used with adults as well as with children. AND THE MULTIPLE To predict which method would provide information about a larger part of a design space, I applied the theory of multiple intelligences [3]. This theory describes eight intelligences that are innate to any human. These intelligences are: linguistic, logic–mathematic, musical, visuo–spatial, bodily–kinesthetic, intrapersonal, interpersonal and naturalistic. I propose to use this theory as a framework to consider design methods. In using a method the intelligences are required in a three-fold manner. In the first way intelligences are required to instruct, explain and clarify the instructions provided to perform the method. In the second way the intelligences are required for the design domain. The design problem dealt with always touches on specific skills and experiences, for example the design of a musical application often requires ideas or experiences that are associated with musical skills. In the third way intelligences are required to perform the method itself, for example the method of using moodboards requires the visuo–spatial intelligence, whereas acting–out could benefit from bodily–kinesthetic skills. The focus of the framework is on the third way. The next problem however, is what classification should be used to compare different methods. Methods are rather different in technique and in output, for example the mood– board technique delivers concrete collages, whereas the output of roleplay is temporal. To be able to compare methods, we proposed a focus group setting, in which different design methods were embedded and evaluated. A typical session would contain an introduction into the domain problem and instructions on the method the participants were instructed to individually perform. For example, if moodboards were compared with a brainstorm, in one session each participant was asked to make a moodboard. Then the participants were asked to come back to the focus group setting and discuss their individual results. Hence, each participant takes part in both methods. The discussions after applying a particular method show the inspiration of the participants in terms of Options and Criteria, while they explain their creation. To avoid the bias of group differences in, e.g., talkativeness of the participants, the comparison should be done through a ‘within-subject’ result analysis. The intelligences are considered to be approaches or skills, with which a design solution can be created and proposed. The hypothesis is that the more intelligences a given method requires, the more ideas (Options and Criteria) the children are likely to generate. In the first study, in which we compared a brainstorm with a rapid prototyping session, we found some evidence confirming this hypothesis. Brainstorming is considered to require mainly one type of intelligence, namely the linguistic intelligence. Alternatively, prototyping requires a variety of intelligences; apart from the linguistic intelligence it also requires visuo–spatial and bodily–kinesthetic intelligence. The work presently done shows that children, indeed, generate more Options in a discussion after using a rapid prototyping method than in a discussion after a brainstorm. For Criteria, however, we found more criteria after a brainstorm than after a rapid prototyping session. In my first study [11] I have shown that both the metrics and the setup are useful when involving children of around ten. The next step is to further expand on the qualitative aspects of designing with children. Shah and Vargas [12] have proposed four metrics to determine ideation effectiveness: Quantity, Quality, Novelty and Variety. Again, first I intend to explore how these metrics perform with children. Second, I would like to understand what the quality of the effectiveness is in that context. Options and I would like to explore this hypothesis further, by examining how a set of intelligences would work for different methods that are considered to have a lower 2 number of intelligences involved. Hence I try to determine how fine–grained predictions can be made based on the theory of multiple intelligences. To determine the number of intelligences involved in a method I have carried out a preliminary study with designers. I introduced them to the theory of multiple intelligences and asked them to associate pre–defined methods with the skills and crafts associated with each type of intelligence. I have checked the inter–coder reliability by performing each any–two agreement on the session transcripts. Furthermore, I have checked whether the groups of children could be compared in terms of their intelligence profiles according the Multiple Intelligence theory. The results show that as expected all teams mentioned more Options after the prototyping activity than after the brainstorming activity. However, unexpectedly, nearly all teams mention fewer Criteria after the prototyping activity than after the brainstorming activity. I will have to analyse the quality of the data further to understand why this happened. A first glance on the results gives the impression that after the brainstorming session, the children were more likely to produce criteria that specified the context of use, rather than a requirement for the design. For the current results I conclude that the children were better able to provide concrete design suggestions (options) when they are involved in a prototyping session than in a brainstorming session. Developing children acquire and different skills and crafts at specific ages. Gardner has proposed a theory about which intelligence the children develop at which age . Naturally, children do not develop them one by one, but they do seem to show a specific profile at certain ages. For the study carried out in [11], we found that the forty participating ten–year–old children scored relatively high on intrapersonal and bodily–kinesthetic intelligence. That finding confirms the work of Piaget (as described in [1]) which describes that children at that age are in the middle of developing cooperative skills and actively look for opportunities to develop their motoric skills. Thus we intend to explore how the intelligences developed at different ages relate to different design activities. CONTRIBUTION For the field of research, the study showed that using Multiple Intelligence Theory as a framework for comparing different methods for idea generation is interesting for further exploration. Children tended to mention more Options after the prototyping activity that involves use of various types of inteligences comparing to the number of Options produced after the brainstorming activity that involves only few types of intelligences. In summary, the children participating in our study were better able to explore the design space in terms of Options after the prototyping activity. In future work, I want to distinguish between design methods used with children by finding empirical evidence for differences in the output of the children using different methods WORK DONE SO FAR In my first study I have compared the outcome of the idea generation process after applying two different methods: a brainstorm and a prototyping session in a focus group setting. A comparison of the discussions in the focus group sessions on the number of Options and Criteria generated after each method should give an answer to my hypothesis. Two design problems were provided by a telecommunication company for a device they were developing for children in school. The groups were composed of five children [4, 5]. A total of eight groups of ten–year–old children (fifth grade) from two Dutch primary schools participated in the study. All groups were of a mixed gender composition. The goal was to establish live communication between a child at home and the class in school whenever a child would not be fit enough to arrive at school. The children were asked to create a device that is to be located in the classroom and/or at home. The device was to be designed for two situations: one when children are supposed to work together in small groups and the other when the teacher is actively teaching in front of the class. For practitioners, this kind of research could uncover tradeoffs for various methods in terms of effectiveness and efficiency. A prototyping method for example, is more demanding to arrange than a brainstorming session. Thus, if the designers attempt to generate a large number of Options it may be worthwhile for them to set out a more demanding session, while once they only want to obtain a generic idea of the possible design space less demanding methods can appear more suitable. The results of my research could provide arguments to choose for a prototyping session because of the quality of the output, despite the higher demands. Finally we intend to create a comprehensive set of guidelines for how to involve children in early design based on our findings from the creative sessions we conduct throughout this project. Many other variables could influence the results, for example the combination of characters and the number of girls and boys. To avoid a strong influence of these variables on the results, I decided on a within-subject design. Every group of children went through both a brainstorm and a prototyping session. To avoid influence of the order and of the case, the sessions were performed in the four possible combinations of the conditions (brainstorm or prototyping) and the cases (group work setting or teaching setting). 3 Design Space Analysis. In J. M. Moran & T. P. Carroll (Eds.), Design Rationale: Concepts, Techniques and Use. Mahwah, NJ: Lawrence Erlbaum (1996) pp. 21-51 REFERENCES 1. Breeuwsma, G. De constructie van de levensloop, Amsterdam: Boom (1994) 8. Olson GM, Olson JS, Storrøsten M, Carter MR, Herbsleb J, Rueter H The structure of activity during design meetings. In: Moran TP, Carroll, JM (eds.) Design rationale: Concepts, techniques, and use. Lawrence Erlbaum Associates, Mahwah, NJ (1996) pp 217-239 2. Druin, A., Bederson, B., Boltman, A., Miura, A., Knotts-Callahan, D., & Platt, M. Children as our technology design partners. A. Druin (Ed.), The design of children's technology San Francisco, CA: Morgan Kaufmann (1999) pp. 51-72 3. Gardner, H. Frames of Mind: The Theory of Multiple Intelligences, New York: Basic, (1983). 9. Scaife, M., & Rogers, Y. Kids as Informants: Telling Us What We Didn't Know or Confirming What We Knew Already? In A. Druin (Ed.), The Design of Children's Technology San Francisco, CA: Morgan Kaufmann. (1999) pp.27-50 4. Heary, C. and Hennessy, E. The Use of Focus Group Interviews in Pediatric Health Care Research, In: Pediatric Health Care (2001) Special Issue on Methodology. 10. Shah, J. J., Smith, S. M., Vargas-Hernandez, N., , “Metrics for Measuring Ideation Effectiveness”, Design Studies, V24 (2), (2003) pp. 111-134. 5. Hennesy, E. and Heary, C., Exploring Children’s Views through Focus Groups, In: Green, S.M. and Hogan, D.M. (eds) Researching Children’s Experiences: Approaches and Methods, SAGA., chpt 13, (2004) pp236-252 11. Sluis-Thiescheffer, R.J.W., Bekker, M.M. and Eggen, J.H. Comparing Early Design Methods for Children In: Proceedings of Interaction Design and Children, ACM Press [in press] 6. Langford J and McDonagh D (eds.) Focus groups: supporting effective product development. London: Taylor and Francis, (2003). 12. Teele,. S. Teele inventory for multiple intelligences. Redlands, California, Sue Teele & Associates, (1992). 7. MacLean, A., Young, R. M., Bellotti, V. M. E., & Moran, T. Questions, Options, and Criteria: Elements of 4