Quantitative Methods for Studying Design Protocols, 2017
This chapter introduces a generic approach to carry out protocol analyses of designers using the ... more This chapter introduces a generic approach to carry out protocol analyses of designers using the Function-Behavior-Structure (FBS) ontology. It suggests codifying design protocols into FBS design issues and deriving FBS design processes using two models, a syntactic model and a semantic model. The syntactic model assumes that any design issues is cognitively related to its immediately preceding issue and as a consequence there is a design process; the concept of using Markov analysis is also presented as a tool to examine the syntactic model. Semantic design processes are derived from ontologically coded linkographs. The construction of the linkograph is further examined in this chapter, as it is the foundation upon which further concepts are built. The information captured in the linkograph is studied using statistics and clustering. The rationale of using information theory, entropy, to measure the linkograph is presented, as well as a concise explanation of information theory. 3.1 Design Ontology and Ontological-Based Coding This section explores the use of the FBS ontology (Gero 1990) to develop a general coding scheme. Its aim is to capture semantic information from design protocols. This semantic information can then be utilised: (1) to explore different aspects of designing according to the focus of interest; (2) to quantify the use cognitive resources; and (3) to locate different types of design transformation processes.
This paper presents the results of a quantitative analysis to measure the development of creative... more This paper presents the results of a quantitative analysis to measure the development of creative design ideas in design protocols. It presents a working definition of design creativity used in the analysis followed by the method of study. Five sets of video and audio data were collected from experiments that involved two designers designing similar tasks in three different settings: face-to-face, an Internet GroupBoard, and a 3D virtual world. The most creative pair was selected for qualitative and quantitative analysis. Our result shows that designers developed more design ideas in the face-to-face setting than in the 3D virtual world settings.
This paper explores using Shannon's entropy of information to measure linkographs of 12 design se... more This paper explores using Shannon's entropy of information to measure linkographs of 12 design sessions that involved six architects in two different experimental conditions. The aim is to find a quantitative tool to interpret the linkographs. This study examines if the differences in the design processes and the design outcomes can be reflected in the entropic interpretations. The results show that the overall entropy of one design condition is slightly higher than the other. Further, there are indications that the change of entropy might reflect design outcomes.
This paper presents a study investigating the differences between what designers and design manag... more This paper presents a study investigating the differences between what designers and design managers say about designing and what designers do. Five datasets are analysed, labelled Samsung, Philips, Design Centre Pharmaceutical, Design Seminar, (interview designers and managers when they talk about designing) and DTRS7 (design session about designers designing). Differences between describing designing and designing are identified using a combined method of protocol analysis, word frequency count and sentiment analysis. There is significantly more Function (F) and less Structure (S) issues mentioned during describing designing than when designing. Sentiment is more positive when describing designing than when designing. Results also show that protocol analysis methods and the FBS ontologybased coding scheme can be applied to design management and descriptions of the design process as well as to designing.
This chapter presents an application of the entropic measurement of linkographs. Linkographs from... more This chapter presents an application of the entropic measurement of linkographs. Linkographs from the empirical protocol studies of six architects are used to verify whether the design outcome correlates with some entropy measure of the linkograph.
Guidelines for a Decision Support Method Adapted to NPD Processes, 2007
In this paper we develop a method for measuring the design process and correlate it with the asse... more In this paper we develop a method for measuring the design process and correlate it with the assessment of the design outcome. We used linkography-a technique used in analyzing design protocols-which is reported to reveal the quality of design ideas and to capture the interconnectivity of ideas in the design process. Previous studies found that Shannon's entropy as a measure of a linkograph might indicate design outcomes. This paper reports on an empirical verification that design outcomes can be correlated to the entropic interpretations of a linkograph. Experimental data from protocol studies of six designers performing two similar design tasks in two different conditions were analysed. In one condition the designers were blindfolded during designing, and then given a relatively short time to draw their design at the end of the session. In the other condition the designers were allowed to sketch. We focus on the three highest scored and the three lowest scored design sessions. Their verbalization during the design session were segmented and coded to produce linkographs. The entropy and the change of entropy over time of these linkographs were studied in relation to the design outcomes. Our results demonstrate that entropy measurement of linkographs provides a way of to study design protocols and show that the design outcome is more related to the rate of change of entropy than to the entropy alone.
This paper reports a case that demonstrates the social aspect of creativity in Asian culture with... more This paper reports a case that demonstrates the social aspect of creativity in Asian culture within the architectural design education context. This inquiry arises when students started doing a structural modelling assignment within the requirement of the project brief but outside the boundary of the lecture’s perceived “learning outcome.” Creativity in the context of culture, problem solving, learning and social process are deliberated. The case study used action research methodology to reflect upon the prescribed outcomes and teaching objectives. The actual learning outcome, as derived from the reports produced by the students are given to form the basis of discussion regarding: process of creativity, creative problem solving and boundary of social norms. The author concludes, in this case, the creative problem solving is fostered by teaching creatively— expanding the boundaries of social norms in Asian’s framework.
This chapter presents a study on the impact of design scales on collaborations in 3D virtual envi... more This chapter presents a study on the impact of design scales on collaborations in 3D virtual environments. Different domains require designers to work on different scales; for instance, urban design and electronic circuit design operate at very different scales. However, the understanding of the effects of scales upon collaboration in virtual environment is limited. In this chapter, the authors propose to use protocol analysis method to examine the differences between two design collaboration projects in virtual environments: one large scale, and another small scale within a similar domain. It shows that the difference in scale impacted more on communication control and social communication.
This paper presents the development of a quantitative method to study team designing processes. I... more This paper presents the development of a quantitative method to study team designing processes. It revisits linkograhy-a graphical representation of the design moves-and depicts ways of interpreting the linkography using entropy from Shannon's information theory, to compare design processes. A case study with two different scenarios is presented followed by the results and conclusions.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this p... more The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.
This paper proposes methods to acquire information from linkography in protocol studies. The pape... more This paper proposes methods to acquire information from linkography in protocol studies. The paper commences with an introduction of linkography together with its use and then presents two cases of collaborative designing. A qualitative analysis of these cases and their linkographs is presented. Two different strategies to abstract information from the linkographs are described: one based on clustering and the other based on Shannon's entropy. The abstracted information is interpreted and mapped to the qualitative analysis of the protocols. The results are promising; cluster analysis is able to group the linkograph into meaningful clusters, while entropy measures the opportunities for idea development of a team and can also reflect the opportunistic contributions of individual participants.
For many, designing is an unknowable mystery. Science is founded on the axiom that things are kno... more For many, designing is an unknowable mystery. Science is founded on the axiom that things are knowable. Can designing be “known” through science? Science takes the approach that there are observables called phenomena that can be represented separately from the phenomena themselves and that these phenomena exhibit regularities. Further, science assumes that these phenomena can in some sense be measured. Hypotheses are conjectures about the regularities of these phenomena that can be tested against the data acquired through measurement. Tested hypotheses form the basis of the construction of models that can be used both to describe the regularities and to make predictions about the phenomena underlying those regularities. It has been argued that since the result of designing is a unique design, i.e., when you carry out the same design task again you produce a different design, where is the regularity that is required for science to apply to designing? The regularity in designing is not necessarily in the resultant design but in the process that produces that design—designerly behaviour. Using science to study designerly behaviour results in scientific models describing designerly behaviour based on empirical evidence rather than on personal experience. The remainder of this chapter will introduce a method for the capture of empirical data on designing. This is followed by the description of an ontology of designing that maps onto the phenomena of designing that are capturable. The rest of the chapter describes some of the scientific models that can be produced from this empirically grounded data.
Artificial intelligence for engineering design, analysis and manufacturing, Jan 30, 2017
This paper reports on a study characterizing design processes and the potential of design spaces ... more This paper reports on a study characterizing design processes and the potential of design spaces through measuring the information entropy of empirical data derived from protocol studies. The sequential segments in a protocol analysis can be related to each other by examining their semantic content producing a design session's linkograph, which defines the design space for a design session. From a linkograph, it is possible to compute the probabilities of the connectivity of each segment for its forelinks and its backlinks, together with the probabilities of distance among links. A linkograph's entropy is a measure of the information in the design session. It is claimed that the entropy of the linkograph measures the potential of the design space being generated as the design proceeds chronologically. We present an approach to the automated construction of linkographs by connecting segments using the lexical database WordNet and measure its entropy. A case study of two design sessions with different characteristics was conducted, one considered more productive and creative, the other more pragmatic. Those segments with high entropy and those associated with high rates of change of entropy are analyzed. The creative session has a higher linkograph entropy. This result indicates the potential of using entropy to characterize a design process.
Learning is difficult to study because of the paucity of data and the lack of adequate tools to m... more Learning is difficult to study because of the paucity of data and the lack of adequate tools to measure it. In this paper we measure the learning by observing the behavior changes of members of a design team as they work together. The paper presents results from protocol analysis techniques used to quantify behavioral changes. It utilizes an ontological coding scheme for design issues; linkography that examines the interaction and individuals' design processes; and entropy measurement of linkographs that investigates contributions of individuals. The results show some correlations with changes in behavior.
There has been considerable research in studying collaborative designing using protocol analysis.... more There has been considerable research in studying collaborative designing using protocol analysis. However, it is not possible to compare these studies because they use different models and have their own coding schemes. A coding scheme based on the Function-Behaviour-Structure (FBS) ontology has been proposed as the basis of a universal coding scheme that transcends the domain of application, the task being studied, and the number of designers being studied. This paper presents results from using the FBS coding scheme and sequential analysis to study the differences between face-to-face design collaboration and synchronized design collaboration using NetMeeting via the internet. In this case the formulation and reformulation processes are richer in the NetMeeting design collaboration. In the face-to-face session they are quicker in arriving at the structure of the design.
This paper reports on preliminary results of an explorative study of a protocol analysis of team ... more This paper reports on preliminary results of an explorative study of a protocol analysis of team learning while designing using in-situ data. Two measurement-based frameworks are proposed. One framework is based on a sequential analysis of communication utterances and the other on codes using the Function-Behavior-Structure ontology and linked using Linkography. Only the former is reported in this paper. Results indicate that team learning occurs and can be discerned from an analysis of the utterances only but only at a high level of granularity.
Quantitative Methods for Studying Design Protocols, 2017
This chapter introduces a generic approach to carry out protocol analyses of designers using the ... more This chapter introduces a generic approach to carry out protocol analyses of designers using the Function-Behavior-Structure (FBS) ontology. It suggests codifying design protocols into FBS design issues and deriving FBS design processes using two models, a syntactic model and a semantic model. The syntactic model assumes that any design issues is cognitively related to its immediately preceding issue and as a consequence there is a design process; the concept of using Markov analysis is also presented as a tool to examine the syntactic model. Semantic design processes are derived from ontologically coded linkographs. The construction of the linkograph is further examined in this chapter, as it is the foundation upon which further concepts are built. The information captured in the linkograph is studied using statistics and clustering. The rationale of using information theory, entropy, to measure the linkograph is presented, as well as a concise explanation of information theory. 3.1 Design Ontology and Ontological-Based Coding This section explores the use of the FBS ontology (Gero 1990) to develop a general coding scheme. Its aim is to capture semantic information from design protocols. This semantic information can then be utilised: (1) to explore different aspects of designing according to the focus of interest; (2) to quantify the use cognitive resources; and (3) to locate different types of design transformation processes.
This paper presents the results of a quantitative analysis to measure the development of creative... more This paper presents the results of a quantitative analysis to measure the development of creative design ideas in design protocols. It presents a working definition of design creativity used in the analysis followed by the method of study. Five sets of video and audio data were collected from experiments that involved two designers designing similar tasks in three different settings: face-to-face, an Internet GroupBoard, and a 3D virtual world. The most creative pair was selected for qualitative and quantitative analysis. Our result shows that designers developed more design ideas in the face-to-face setting than in the 3D virtual world settings.
This paper explores using Shannon's entropy of information to measure linkographs of 12 design se... more This paper explores using Shannon's entropy of information to measure linkographs of 12 design sessions that involved six architects in two different experimental conditions. The aim is to find a quantitative tool to interpret the linkographs. This study examines if the differences in the design processes and the design outcomes can be reflected in the entropic interpretations. The results show that the overall entropy of one design condition is slightly higher than the other. Further, there are indications that the change of entropy might reflect design outcomes.
This paper presents a study investigating the differences between what designers and design manag... more This paper presents a study investigating the differences between what designers and design managers say about designing and what designers do. Five datasets are analysed, labelled Samsung, Philips, Design Centre Pharmaceutical, Design Seminar, (interview designers and managers when they talk about designing) and DTRS7 (design session about designers designing). Differences between describing designing and designing are identified using a combined method of protocol analysis, word frequency count and sentiment analysis. There is significantly more Function (F) and less Structure (S) issues mentioned during describing designing than when designing. Sentiment is more positive when describing designing than when designing. Results also show that protocol analysis methods and the FBS ontologybased coding scheme can be applied to design management and descriptions of the design process as well as to designing.
This chapter presents an application of the entropic measurement of linkographs. Linkographs from... more This chapter presents an application of the entropic measurement of linkographs. Linkographs from the empirical protocol studies of six architects are used to verify whether the design outcome correlates with some entropy measure of the linkograph.
Guidelines for a Decision Support Method Adapted to NPD Processes, 2007
In this paper we develop a method for measuring the design process and correlate it with the asse... more In this paper we develop a method for measuring the design process and correlate it with the assessment of the design outcome. We used linkography-a technique used in analyzing design protocols-which is reported to reveal the quality of design ideas and to capture the interconnectivity of ideas in the design process. Previous studies found that Shannon's entropy as a measure of a linkograph might indicate design outcomes. This paper reports on an empirical verification that design outcomes can be correlated to the entropic interpretations of a linkograph. Experimental data from protocol studies of six designers performing two similar design tasks in two different conditions were analysed. In one condition the designers were blindfolded during designing, and then given a relatively short time to draw their design at the end of the session. In the other condition the designers were allowed to sketch. We focus on the three highest scored and the three lowest scored design sessions. Their verbalization during the design session were segmented and coded to produce linkographs. The entropy and the change of entropy over time of these linkographs were studied in relation to the design outcomes. Our results demonstrate that entropy measurement of linkographs provides a way of to study design protocols and show that the design outcome is more related to the rate of change of entropy than to the entropy alone.
This paper reports a case that demonstrates the social aspect of creativity in Asian culture with... more This paper reports a case that demonstrates the social aspect of creativity in Asian culture within the architectural design education context. This inquiry arises when students started doing a structural modelling assignment within the requirement of the project brief but outside the boundary of the lecture’s perceived “learning outcome.” Creativity in the context of culture, problem solving, learning and social process are deliberated. The case study used action research methodology to reflect upon the prescribed outcomes and teaching objectives. The actual learning outcome, as derived from the reports produced by the students are given to form the basis of discussion regarding: process of creativity, creative problem solving and boundary of social norms. The author concludes, in this case, the creative problem solving is fostered by teaching creatively— expanding the boundaries of social norms in Asian’s framework.
This chapter presents a study on the impact of design scales on collaborations in 3D virtual envi... more This chapter presents a study on the impact of design scales on collaborations in 3D virtual environments. Different domains require designers to work on different scales; for instance, urban design and electronic circuit design operate at very different scales. However, the understanding of the effects of scales upon collaboration in virtual environment is limited. In this chapter, the authors propose to use protocol analysis method to examine the differences between two design collaboration projects in virtual environments: one large scale, and another small scale within a similar domain. It shows that the difference in scale impacted more on communication control and social communication.
This paper presents the development of a quantitative method to study team designing processes. I... more This paper presents the development of a quantitative method to study team designing processes. It revisits linkograhy-a graphical representation of the design moves-and depicts ways of interpreting the linkography using entropy from Shannon's information theory, to compare design processes. A case study with two different scenarios is presented followed by the results and conclusions.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this p... more The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.
This paper proposes methods to acquire information from linkography in protocol studies. The pape... more This paper proposes methods to acquire information from linkography in protocol studies. The paper commences with an introduction of linkography together with its use and then presents two cases of collaborative designing. A qualitative analysis of these cases and their linkographs is presented. Two different strategies to abstract information from the linkographs are described: one based on clustering and the other based on Shannon's entropy. The abstracted information is interpreted and mapped to the qualitative analysis of the protocols. The results are promising; cluster analysis is able to group the linkograph into meaningful clusters, while entropy measures the opportunities for idea development of a team and can also reflect the opportunistic contributions of individual participants.
For many, designing is an unknowable mystery. Science is founded on the axiom that things are kno... more For many, designing is an unknowable mystery. Science is founded on the axiom that things are knowable. Can designing be “known” through science? Science takes the approach that there are observables called phenomena that can be represented separately from the phenomena themselves and that these phenomena exhibit regularities. Further, science assumes that these phenomena can in some sense be measured. Hypotheses are conjectures about the regularities of these phenomena that can be tested against the data acquired through measurement. Tested hypotheses form the basis of the construction of models that can be used both to describe the regularities and to make predictions about the phenomena underlying those regularities. It has been argued that since the result of designing is a unique design, i.e., when you carry out the same design task again you produce a different design, where is the regularity that is required for science to apply to designing? The regularity in designing is not necessarily in the resultant design but in the process that produces that design—designerly behaviour. Using science to study designerly behaviour results in scientific models describing designerly behaviour based on empirical evidence rather than on personal experience. The remainder of this chapter will introduce a method for the capture of empirical data on designing. This is followed by the description of an ontology of designing that maps onto the phenomena of designing that are capturable. The rest of the chapter describes some of the scientific models that can be produced from this empirically grounded data.
Artificial intelligence for engineering design, analysis and manufacturing, Jan 30, 2017
This paper reports on a study characterizing design processes and the potential of design spaces ... more This paper reports on a study characterizing design processes and the potential of design spaces through measuring the information entropy of empirical data derived from protocol studies. The sequential segments in a protocol analysis can be related to each other by examining their semantic content producing a design session's linkograph, which defines the design space for a design session. From a linkograph, it is possible to compute the probabilities of the connectivity of each segment for its forelinks and its backlinks, together with the probabilities of distance among links. A linkograph's entropy is a measure of the information in the design session. It is claimed that the entropy of the linkograph measures the potential of the design space being generated as the design proceeds chronologically. We present an approach to the automated construction of linkographs by connecting segments using the lexical database WordNet and measure its entropy. A case study of two design sessions with different characteristics was conducted, one considered more productive and creative, the other more pragmatic. Those segments with high entropy and those associated with high rates of change of entropy are analyzed. The creative session has a higher linkograph entropy. This result indicates the potential of using entropy to characterize a design process.
Learning is difficult to study because of the paucity of data and the lack of adequate tools to m... more Learning is difficult to study because of the paucity of data and the lack of adequate tools to measure it. In this paper we measure the learning by observing the behavior changes of members of a design team as they work together. The paper presents results from protocol analysis techniques used to quantify behavioral changes. It utilizes an ontological coding scheme for design issues; linkography that examines the interaction and individuals' design processes; and entropy measurement of linkographs that investigates contributions of individuals. The results show some correlations with changes in behavior.
There has been considerable research in studying collaborative designing using protocol analysis.... more There has been considerable research in studying collaborative designing using protocol analysis. However, it is not possible to compare these studies because they use different models and have their own coding schemes. A coding scheme based on the Function-Behaviour-Structure (FBS) ontology has been proposed as the basis of a universal coding scheme that transcends the domain of application, the task being studied, and the number of designers being studied. This paper presents results from using the FBS coding scheme and sequential analysis to study the differences between face-to-face design collaboration and synchronized design collaboration using NetMeeting via the internet. In this case the formulation and reformulation processes are richer in the NetMeeting design collaboration. In the face-to-face session they are quicker in arriving at the structure of the design.
This paper reports on preliminary results of an explorative study of a protocol analysis of team ... more This paper reports on preliminary results of an explorative study of a protocol analysis of team learning while designing using in-situ data. Two measurement-based frameworks are proposed. One framework is based on a sequential analysis of communication utterances and the other on codes using the Function-Behavior-Structure ontology and linked using Linkography. Only the former is reported in this paper. Results indicate that team learning occurs and can be discerned from an analysis of the utterances only but only at a high level of granularity.
Uploads
Papers by Jeff Kan