- Computer Science, New Media, Design, Software Engineering, Digital Humanities, Information Systems, and 96 moreE-learning, Ethnography, Innovation statistics, Programming Languages, Information Science, Learning and Teaching, Digital Libraries, Distance Education, Information Literacy, Interaction Design, Design Research, Adult Education, Computer-Mediated Communication, Open Source Software, Information Communication Technology, ICT in Education, Social Cognition, Information Visualization, Usability, Information Society, Open Access, Online Communities, Design Methods, Design education, Game Design, Computer Aided Design, Evaluation Research, Learning Technology, Computer Supported Cooperative Work (CSCW), Participatory Research, Adult Continuing and Professional Education, Participatory Design, Diffusion of Innovations, Social Study of Information & Communication Technology, Open Access Publishing, Requirements Engineering, Soft Computing, Design practice, Digital Media, Device, and Software Interface Design, Information Ethics, Tacit Knowledge, Non-formal Education, Technology in Teacher Education, Computer Ethics, Software Engineering education, Sociotechnical Systems, New Media Arts, Institutional Repositories, Agile Software Process Improvement, Software Studies, Models of Innovation Types and Processes, Software Process Improvement and Assessment, Software Security, Social Informatics, Technologies Applied to Cultural Heritage, Techno-Information Literacy, Design pedagogy, Open Journal, ICT in libraries, Teaching and Learning Programming, Learner Centered Approach, Information use, Theory of Relation Between Innovation and Creativity, Innovation Ecosystems and Their Dynamics, Peer Support (Learning and Teaching), Tacit Knowledge Transfer Within Virtual Teams, Diagram understanding, Collaborative Learning (Education), Relation Between Creation Practices and Innovation Practices, Embedding Degree of Innovations within Communities of Practice, Digital Litracy, Institutional Repository, Information work, Innovation Studied with Particular Research Methods or Data Source Types, Infovis, Infoviz, Fluid Self-Organizing Coalition versus "Champion" Models of Innovation, Ownerless Big Innovations--innovation without creators, inventors, heroic males, or other individualizations and Western culture myths, The Social Life of Information (Brown, Duguid)--social practices assumed in info systems and design decisions, Dialectic Innovation--opposings of innovations that spawn new innovations, The Dependence of Innovation on Huge Non-Innovation Practices and Dynamics, Institutional Repository of University Consortium, Open Systems, Copylefts, Capability Democratizations designers and Designing and Designs by Everyone, Users, Customers, Clients, Innovation Studied In Particular Domains art, Science, Politics, Porno, Cooking, Theology, Human Computer Interaction, and Etceteraedit
- Research interests: HCI; CSCW; informal social learning of technology; collaborative information retrieval; the usabi... moreResearch interests:
HCI; CSCW; informal social learning of technology; collaborative information retrieval; the usability of open source software; the adoption, appropriation, tailoring, combination and adaptation of collaborative technologies; rapid prototyping and evaluation; software informatics.edit
The availability of advanced computing technologies is leading to radical changes in business, education, and society in general. The aim of this project is to explore the potential of new technologies to enhance the experience of... more
The availability of advanced computing technologies is leading to radical changes in business, education, and society in general. The aim of this project is to explore the potential of new technologies to enhance the experience of visiting a museum. The focus of this proposal is not remote access to the virtual museum; rather, I wish to examine a relatively under-explored area: how to enhance the experience of an actual museum visit in real life, through the deployment of different information technologies.
Abstract Groupware is defined as software, systems, and services that help groups of workers do their jobs better [2]. Computer Supported Cooperative Work is defined by Greif and Cashman as the support of multiple individuals working... more
Abstract Groupware is defined as software, systems, and services that help groups of workers do their jobs better [2]. Computer Supported Cooperative Work is defined by Greif and Cashman as the support of multiple individuals working together with computer systems [1]. Groupware is currently a widespread tool in the professional community[3]. To educate and prepare students in these areas, many universities are teaching their students groupware and CSCW.
We describe the development of the Designers' NotePad; a tool to support the rapid creation and refinement of ideas. The tool supports collaborative dialogues with features that enable idea fragments to be rapidly entered and revised. The... more
We describe the development of the Designers' NotePad; a tool to support the rapid creation and refinement of ideas. The tool supports collaborative dialogues with features that enable idea fragments to be rapidly entered and revised. The core features are entities and links. Users are able to gradually add more structure while still keeping the freedom to rapidly refine and without being required to make premature commitments about classification. An iterative design approach was used involving frequent studies of the tool in use.
This white paper and attached pre-prints are part of the three-year interim project report for the IMLS Digital Collections & Content Project, summarizing major findings October 2002 through September 2005. Project is hosted at the... more
This white paper and attached pre-prints are part of the three-year interim project report for the IMLS Digital Collections & Content Project, summarizing major findings October 2002 through September 2005. Project is hosted at the University of Illinois at Urbana-Champaign. Project Director is Timothy W. Cole (t-cole@uiuc.edu). Full report is available at http://imlsdcc.grainger .uiuc.edu. The material in this report is based upon work supported by the Institute of Museum and Library Services under IMLS National Leadership Grant Award No.
Abstract: iSchools have a well-established reputation for rigorous analysis, applying a range of multidisciplinary methods. However, increasingly many of us are teaching design elements as well as doing design in our research.
ABSTRACT This chapter explores the concept of patchwork prototyping: the combining of open source software applications to rapidly create a rudimentary but fully functional prototype that can be used and hence evaluated in real-life... more
ABSTRACT This chapter explores the concept of patchwork prototyping: the combining of open source software applications to rapidly create a rudimentary but fully functional prototype that can be used and hence evaluated in real-life situations.
ABSTRACT This paper presents an analysis of written and electronic records that document the collaborative process of packing museum artifacts in preparation for a move. The majority of data recorded detailed the process of packing, while... more
ABSTRACT This paper presents an analysis of written and electronic records that document the collaborative process of packing museum artifacts in preparation for a move. The majority of data recorded detailed the process of packing, while only a small amount of the data concerned which artifacts were packed in which boxes. Museum staff members were able to use these process data to solve the numerous errors that occurred during packing.
ABSTRACT In this workshop, we explore emerging research themes in web mash-ups which are relevant to CSCW research. Web mash-ups are web sites which combine data or services from more than one source into an integrated whole. The mash-up... more
ABSTRACT In this workshop, we explore emerging research themes in web mash-ups which are relevant to CSCW research. Web mash-ups are web sites which combine data or services from more than one source into an integrated whole. The mash-up community resembles open-source communities and likewise could benefit from further study. We explore several themes in and related to mash-ups which connect to core research areas in CSCW.
Abstract The movement from the physical to the digital library offers both dangers and opportunities. Alongside the greater quantity of online material goes the problem of quality assurance: how can be the information searcher be sure of... more
Abstract The movement from the physical to the digital library offers both dangers and opportunities. Alongside the greater quantity of online material goes the problem of quality assurance: how can be the information searcher be sure of the status of a document. We suggest that this be addressed by supporting recommendations and that the key feature that links these recommendations together is that of usage.
Abstract We explore the application of a wiki-based technology and style of interaction to enabling the incremental translation of a collaborative application into a number of different languages, including variant English language... more
Abstract We explore the application of a wiki-based technology and style of interaction to enabling the incremental translation of a collaborative application into a number of different languages, including variant English language interfaces better suited to the needs of particular user communities.
This paper presents a qualitative and formative study of the uses of a starfield-based visualization interface for analysis of library collections. The evaluation process has produced feedback that suggests ways to significantly improve... more
This paper presents a qualitative and formative study of the uses of a starfield-based visualization interface for analysis of library collections. The evaluation process has produced feedback that suggests ways to significantly improve starfield interfaces and the interaction process to improve their learnability and usability. The study also gave us clear indication of additional potential uses of starfield visualizations that can be exploited by further functionality and interface development.
Abstract The paper discusses issues to be considered when evaluating an Intelligent Learning Environment. In particular it considers problems that may arise when using rigorous experimental methods and the usefulness of informal... more
Abstract The paper discusses issues to be considered when evaluating an Intelligent Learning Environment. In particular it considers problems that may arise when using rigorous experimental methods and the usefulness of informal techniques. It advocates the frequent use of informal in-depth studies on prototypes during the development of a system to reveal problems with the ILE in use and to raise general issues applicable across ILEs.
ABSTRACT This paper considers how studies of collaborative activities in the real world can be used to aid in the design of digital libraries. We describe two studies of collaboration in physical environments. The first study examines... more
ABSTRACT This paper considers how studies of collaborative activities in the real world can be used to aid in the design of digital libraries. We describe two studies of collaboration in physical environments. The first study examines behaviour at the enquiry desk in an academic library. The second study examines collaboration in an office environment-concentrating on'over the shoulder learning'.
Abstract Although searching for documents is the dominant paradigm in information retrieval it is often more effective to ask another person. A library can serve as a matchmaker that introduces users with common interests to each other.... more
Abstract Although searching for documents is the dominant paradigm in information retrieval it is often more effective to ask another person. A library can serve as a matchmaker that introduces users with common interests to each other. Several methods of matchmaking are described and their implications for privacy are discussed. An approach to prototyping matchmaking functionality is outlined.
Abstract This article reports on a study which used results from 119 scenario–based evaluations of 36 museum Web sites to develop a conceptual framework for analyzing the usability flaws of museum Web sites. It identifies 15 unique... more
Abstract This article reports on a study which used results from 119 scenario–based evaluations of 36 museum Web sites to develop a conceptual framework for analyzing the usability flaws of museum Web sites. It identifies 15 unique dimensions, grouped into five categories, that exemplify usability problems common to many museum Web sites. Each dimension is discussed in detail, and typical examples are provided, based on actual usability flaws observed during the evaluations.
Can open source software save the world? Recently Brian Behlendorf, who helped found and develop the Apache Web server open source project and now sits on the Mozilla Foundation board, asked this question. He discussed how open source... more
Can open source software save the world? Recently Brian Behlendorf, who helped found and develop the Apache Web server open source project and now sits on the Mozilla Foundation board, asked this question. He discussed how open source systems are being built to address some of the world's major problems: economic distress, natural-disaster responses, broken healthcare systems, education crises, and more.
ABSTRACT The paper examines the concept of over-the-shoulder learning: the informal collaborative learning of a computer application that occurs in brief episodes as part of regular work. Certain design implications are explored,... more
ABSTRACT The paper examines the concept of over-the-shoulder learning: the informal collaborative learning of a computer application that occurs in brief episodes as part of regular work. Certain design implications are explored, including the representation of the process of system use, and the consequences of regarding interfaces as media for human-human interaction.
ABSTRACT This chapter presents results from our empirical studies of metadata quality in large corpuses of metadata harvested under open archives initiative (OAI) protocols. Along with a discussion of why and how metadata quality is... more
ABSTRACT This chapter presents results from our empirical studies of metadata quality in large corpuses of metadata harvested under open archives initiative (OAI) protocols. Along with a discussion of why and how metadata quality is important, an approach to conceptualizing, and assessing metadata quality is presented. The approach is based on a more general model of information quality for many kinds of information beyond just metadata.
Abstract Over The Shoulder Learning is the informal, spontaneous workplace help-giving interaction that is often used by people to learn from their colleagues how to use part of a computer application. The concept is analysed in the light... more
Abstract Over The Shoulder Learning is the informal, spontaneous workplace help-giving interaction that is often used by people to learn from their colleagues how to use part of a computer application. The concept is analysed in the light of a study of office activity, revealing both the strengths and weaknesses of the method.
Abstract: With current systems, we are forced to make decisions either to keep or not keep; delete or not delete a file. Unfortunately our opinions about many information items do not easily fit into this binary worldview. As a part of an... more
Abstract: With current systems, we are forced to make decisions either to keep or not keep; delete or not delete a file. Unfortunately our opinions about many information items do not easily fit into this binary worldview. As a part of an exploratory study of looking at file organization on personal computers, this paper describes how people deal with this difficulty on their computers. It implies that people need a facility for information items that falls between the categories of keep and not keep.
Abstract We explore the application of a wiki-based technology and style of interaction to enabling the incremental translation of a collaborative application into a number of different languages, including variant English language... more
Abstract We explore the application of a wiki-based technology and style of interaction to enabling the incremental translation of a collaborative application into a number of different languages, including variant English language interfaces better suited to the needs of particular user communities.
We describe a Web-based metadata quality tool that provides statistical descriptions and visualisations of Dublin Core metadata harvested via the OAI protocol. The lightweight nature of development allows it to be used to gather... more
We describe a Web-based metadata quality tool that provides statistical descriptions and visualisations of Dublin Core metadata harvested via the OAI protocol. The lightweight nature of development allows it to be used to gather contextualized requirements and some initial user feedback is discussed.
Abstract The growth of network availability has led to a dramatic increase in the amount of on-line information users can access remotely. However, users often have difficulty locating the details they require due to the manner in which... more
Abstract The growth of network availability has led to a dramatic increase in the amount of on-line information users can access remotely. However, users often have difficulty locating the details they require due to the manner in which this information is provided. One reason for this difficulty is that limited facilities are provided which allow users to exploit their existing spatial skills in managing access to shared information. We consider Populated Information Terrains (PITS) as a way of addressing this issue.
UK PubMed Central (UKPMC) is an archive of life sciences journal literature.
Abstract A study was conducted across multiple open source software online technical help communities. This paper presents the types of discussions that occur, the types of questions asked and the type of responses that are given. The... more
Abstract A study was conducted across multiple open source software online technical help communities. This paper presents the types of discussions that occur, the types of questions asked and the type of responses that are given. The implications for socio-technical design are considered, exploring how help requests and discussions can be used to improve future help-giving, documentation and interface and functionality redesign.
Abstract: iSchools have a well-established reputation for rigorous analysis, applying a range of multidisciplinary methods. However, increasingly many of us are teaching design elements as well as doing design in our research.
Abstract: Many computer-assisted language learning systems specifically designed to be used in the curriculum and which exploit AI techniques have neither a learner model nor consequently any deep error analysis. Evidence from applied... more
Abstract: Many computer-assisted language learning systems specifically designed to be used in the curriculum and which exploit AI techniques have neither a learner model nor consequently any deep error analysis. Evidence from applied linguistics shows that learners have their own system of rules for the production of a foreign language. We believe the central issue is to determine the appropriate level of description of these rules and uncover the strategies used by the learners in particular situations.
We describe the development of the Designers' NotePad; a tool to support the rapid creation and refinement of ideas. The tool supports collaborative dialogues with features that enable idea fragments to be rapidly entered and revised. The... more
We describe the development of the Designers' NotePad; a tool to support the rapid creation and refinement of ideas. The tool supports collaborative dialogues with features that enable idea fragments to be rapidly entered and revised. The core features are entities and links. Users are able to gradually add more structure while still keeping the freedom to rapidly refine and without being required to make premature commitments about classification. An iterative design approach was used involving frequent studies of the tool in use.
Abstract This paper examines emerging interface paradigms, to a certain extent exemplified by Apple's iPhone interface for interacting with complex visual content. It provides an overview of 15 years of research devoted to the creation of... more
Abstract This paper examines emerging interface paradigms, to a certain extent exemplified by Apple's iPhone interface for interacting with complex visual content. It provides an overview of 15 years of research devoted to the creation of intuitive user-friendly interfaces, as well as conceptual analysis of design principles. Implications for the design of interfaces for cultural heritage information are also discussed.
Abstract The paper describes the benefits of providing an Interactive Learning Environment with an interface which allows explicit planning and instantiation. In addition to providing information about the student's beliefs to the student... more
Abstract The paper describes the benefits of providing an Interactive Learning Environment with an interface which allows explicit planning and instantiation. In addition to providing information about the student's beliefs to the student model, it facilitates the student's learning and metacognition by proving a supportive problem solving environment. Features from a system for teaching logic proofs have been extended and generalised by developing a system for supporting problem solving in the domain of accounting and finance.
ABSTRACT: We describe the development of a system to support cooperative software design. An iterative development approach has been used, based upon the observation of system use in authentic design sessions. This allows us to correct... more
ABSTRACT: We describe the development of a system to support cooperative software design. An iterative development approach has been used, based upon the observation of system use in authentic design sessions. This allows us to correct interface errors, and also to learn more about the nature of collaborative design. The observations of use and the resulting refinements of the system are described. In particular we note the variability in design activity both amongst designers and according to circumstances.
Abstract We consider the role of collaborative learning during information searching. We report on observations of situated collaboration in a physical library, which informed the development of our system, Ariadne. This was intended both... more
Abstract We consider the role of collaborative learning during information searching. We report on observations of situated collaboration in a physical library, which informed the development of our system, Ariadne. This was intended both to investigate and support the learning of search skills. An iterative development and testing methodology was applied. The system has a mechanism for recording an interaction history of the search process.
Abstract: Many computer-assisted language learning systems specifically designed to be used in the curriculum and which exploit AI techniques have neither a learner model nor consequently any deep error analysis. Evidence from applied... more
Abstract: Many computer-assisted language learning systems specifically designed to be used in the curriculum and which exploit AI techniques have neither a learner model nor consequently any deep error analysis. Evidence from applied linguistics shows that learners have their own system of rules for the production of a foreign language. We believe the central issue is to determine the appropriate level of description of these rules and uncover the strategies used by the learners in particular situations.
Abstract The purpose of this project is to investigate spontaneous (informal) collaboration and learning emerging from three contexts: in a classroom (Computer Supported Collaborative Learning), in game playing setting (Computer Supported... more
Abstract The purpose of this project is to investigate spontaneous (informal) collaboration and learning emerging from three contexts: in a classroom (Computer Supported Collaborative Learning), in game playing setting (Computer Supported Collaborative Play) and in a workplace (Computer Supported Collaborative Work).
The paper explores the nature of docents' work in museums. From this is derived the concept ofthe docent function to describe different aspects ofthat work, some of which may be performed by otherpeople and also by objects such as maps... more
The paper explores the nature of docents' work in museums. From this is derived the concept ofthe docent function to describe different aspects ofthat work, some of which may be performed by otherpeople and also by objects such as maps andguidebooks. This analysis leads to the idea of theCyberdocent–an extension of the docent function totake advantage of new possibilities afforded byadvanced technologies. The potential of theCyberdocent in both virtual and real museums isinvestigated.
Abstract Groupware is defined as software, systems, and services that help groups of workers do their jobs better [2]. Computer Supported Cooperative Work is defined by Greif and Cashman as the support of multiple individuals working... more
Abstract Groupware is defined as software, systems, and services that help groups of workers do their jobs better [2]. Computer Supported Cooperative Work is defined by Greif and Cashman as the support of multiple individuals working together with computer systems [1]. Groupware is currently a widespread tool in the professional community[3]. To educate and prepare students in these areas, many universities are teaching their students groupware and CSCW.
The movement from the physical to the digital library offers both dangers and opportunities. Alongside the greater quantity of online material goes the problem of quality assurance: how can be the information searcher be sure of the... more
The movement from the physical to the digital library offers both dangers and opportunities. Alongside the greater quantity of online material goes the problem of quality assurance: how can be the information searcher be sure of the status of a document. We suggest that this be addressed by supporting recommendations and that the key feature that links these recommendations together is that of usage. The main use of usage data within information science is currently that of a research tool in the form of transaction log analysis.
Abstract Many collaborative design tools may suffer from being too generic to address the specific complexities inherent in multidisciplinary collaboration. We provide accounts of several multidisciplinary HCI courses at our institution,... more
Abstract Many collaborative design tools may suffer from being too generic to address the specific complexities inherent in multidisciplinary collaboration. We provide accounts of several multidisciplinary HCI courses at our institution, elaborating on the challenges student teams face when integrating design practice from a wide variety of disciplines. Of particular interest are the distinct approaches that these multidisciplinary teams adopt that differ from more common forms of collaborative design.
ABSTRACT In this paper, we describe the conceptual idea of the Document Sandbox: an environment where users can make exploratory changes to a document safely and without concern of irreversibly altering the original, and which facilitates... more
ABSTRACT In this paper, we describe the conceptual idea of the Document Sandbox: an environment where users can make exploratory changes to a document safely and without concern of irreversibly altering the original, and which facilitates a user's search for the right feature for a task. The aim of this tool is to allow the user freedom and framework to systematically find the right feature for a situation by trial-and-error in a safe, easily and infinitely recoverable environment.
When personal computers first arrived, many predicted that by the end of the century our desktops would become free from clutter as we moved to the paperless office. However, if we look at desktops of current researchers, we can see that... more
When personal computers first arrived, many predicted that by the end of the century our desktops would become free from clutter as we moved to the paperless office. However, if we look at desktops of current researchers, we can see that this has definitely not happened (see Figure 1). But it is not only researchers' physical desktops that are cluttered; their computer desktops (and file systems) are just as cluttered. This should not come as a surprise.
Abstract The recent emergence of Web mash-ups and open source software is driving the development of new practices in software and systems development. In this paper we explore novel practices of user-driven innovation through an... more
Abstract The recent emergence of Web mash-ups and open source software is driving the development of new practices in software and systems development. In this paper we explore novel practices of user-driven innovation through an examination of several case studies which illustrate how users and developers are exploiting the proliferation of open APIs and open source systems. Developers can rapidly create proofs of concept that are robust enough for actual use by combining preexisting software components.
Abstract We draw on the music master class as an established face-to-face pedagogical model as one that supports reflective, collaborative learning practices. We then illustrate how this model is being reinterpreted and extended within... more
Abstract We draw on the music master class as an established face-to-face pedagogical model as one that supports reflective, collaborative learning practices. We then illustrate how this model is being reinterpreted and extended within the context of online learning communities exploring the skills of user interface design.
Abstract: The current folder system on personal computers is not only a storage place for information organization and retrieval; it also serves as a set of potential workplaces for projects or tasks. We performed a qualitative study on... more
Abstract: The current folder system on personal computers is not only a storage place for information organization and retrieval; it also serves as a set of potential workplaces for projects or tasks. We performed a qualitative study on two different participant groups in an academic institution. The study examined creation and use of folder content and structure.
Abstract As part of a federation project providing integrated access to over 170 digital collections, we are studying how collections can best be represented to meet the needs of service providers and diverse user communities. This paper... more
Abstract As part of a federation project providing integrated access to over 170 digital collections, we are studying how collections can best be represented to meet the needs of service providers and diverse user communities. This paper reports on recent results from that project on how digital resource developers conceive of and define their collections.
ABSTRACT A review of previous work in the field of learner modelling reveals an emphasis on the surface descriptions of errors of understanding with only partial consideration of the underlying misconceptions that might explain the cause... more
ABSTRACT A review of previous work in the field of learner modelling reveals an emphasis on the surface descriptions of errors of understanding with only partial consideration of the underlying misconceptions that might explain the cause of the errors. Within a general framework for learner modelling we propose a technique, involving the use of a structured interface, for the effective acquisition of a richer and deeper model of learners' errors.
Abstract: The paper explores the effectiveness of persona-based design methods in HCI education. We describe the experiences of our students in learning, practicing, and using personas in a number of design contexts. From these... more
Abstract: The paper explores the effectiveness of persona-based design methods in HCI education. We describe the experiences of our students in learning, practicing, and using personas in a number of design contexts. From these experiences, we identify common patterns of use and misuse and characterize challenges in incorporating and using personas in the classroom. We conclude with advice on how to effectively teach design using personas.
Abstract: The paper proposes the term'software informatics' to describe research on the individual, collaborative, and social aspects of software production and use, spanning multiple representations of software from design, to source... more
Abstract: The paper proposes the term'software informatics' to describe research on the individual, collaborative, and social aspects of software production and use, spanning multiple representations of software from design, to source code, to application. It does this with a particular focus on information processes around software development.
Abstract This paper describes how an area of computer science research, Computer Supported Cooperative Work, can be applied to the world of libraries. Collaborative activities can be described by whether they occur in the same time and in... more
Abstract This paper describes how an area of computer science research, Computer Supported Cooperative Work, can be applied to the world of libraries. Collaborative activities can be described by whether they occur in the same time and in the same place. These activities can be broadly arranged into three groups depending on the participants: staff-staff, user-staff and user-user interactions.
ABSTRACT We explore the potential of syndicating copy actions between group members, particularly as a way to provide lightweight activity awareness information. The advantages of the approach include its ease of integration with existing... more
ABSTRACT We explore the potential of syndicating copy actions between group members, particularly as a way to provide lightweight activity awareness information. The advantages of the approach include its ease of integration with existing activities and tools, its flexibility and its lightweight nature for both giver and receiver of information. Its disadvantages include the small amount of information captured limiting its interpretability and usefulness for full awareness, and regular concerns about privacy.
Abstract: In this position paper, we discuss the role of abstraction in designing for appropriation. We examine the ways in which varying the level of abstraction of tools affects the ability of users to appropriate them. We close with... more
Abstract: In this position paper, we discuss the role of abstraction in designing for appropriation. We examine the ways in which varying the level of abstraction of tools affects the ability of users to appropriate them. We close with some words about the difficulties of evaluating the appropriability of systems and how they might be addressed in an experimental framework.