See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/254040638

A case study of using HCI methods to improve tools for programmers

Article · June 2012

DOI: 10.1109/CHASE.2012.6223018

CITATIONS READS
19 859

4 authors, including:

Brad A. Myers
Carnegie Mellon University
516 PUBLICATIONS 24,291 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

API Usability View project

Obsidian View project

All content following this page was uploaded by Brad A. Myers on 26 May 2014.

The user has requested enhancement of the downloaded file.

 A Case Study of Using HCI Methods to Improve Tools for Programmers

Andrew Faulring, Brad A. Myers Yaad Oren, Keren Rotenberg

Human-Computer Interaction Institute Technology & Innovation Platform BU
Carnegie Mellon University SAP Labs Israel Ltd.
Pittsburgh, PA, USA Ra'anana, Israel
{faulring, bam}@cs.cmu.edu {yaad.oren, keren.rotenberg}@sap.com

Abstract—For more than five years, researchers at Carnegie so forth. SAP’s software enables developers to create and
Mellon University have been collaborating with several SAP customize new processes according to their company’s busi-
teams to improve the usability of SAP’s developer tools and ness needs. SAP provides the core software, sometimes with
programming APIs. Much research has shown that HCI tech- industry-specific functionality, and then developers, who are
niques can improve the tools that developers use to write soft- often consultants, customize the software for each customer,
ware. In a recent project, we applied HCI techniques to a SAP often by writing code. SAP provides a range of programming
developer tool for the SAP NetWeaver Gateway product. The and software management tools to support the customization
SAP team building this tool uses agile software development process. Developers use SAP’s ABAP programming lan-
processes, which allowed them to quickly improve the tool’s
guage to write programs that interoperate with Business
usability based upon the evaluations.
Suite systems.
Keywords-APIs; heuristic evaluation; cognitive walkthrough; Recognizing the growing market demand of exposing
agile software development; REST; OData; SAP NetWeaver SAP services to non-SAP UI environments, such as mobile
Gateway; natural programming devices, the web, and social media applications, SAP created
the NetWeaver Gateway 1 product, which provides a web
services interface for applications to consume data stored
I. INTRODUCTION
inside a Business Suite system. SAP recognized that many
The field of human-computer interaction (HCI) has de- mobile and web application developers were familiar with
veloped a wide variety of techniques for the design and eval- new web technologies such as REST [12], but did not have
uation of user interfaces (UIs). The Natural Programming experience with SAP’s ABAP language. So SAP developed
project [1] at Carnegie Mellon University (CMU) has repeat- the NetWeaver Gateway product to facilitate the consump-
edly demonstrated that many of the these techniques can be tion of SAP services by non-SAP developer communities.
adapted to improve the effectiveness of the tools and lan- The NetWeaver Gateway provides an OData-based API for
guages used by developers. For example, we have applied consuming services. OData 2 builds upon the HTTP REST
them to the user interfaces of integrated development envi- architecture by adding a model for the data exchanged be-
ronments (IDEs) [2, 3], debuggers [4], programming lan- tween client and server. Data is modeled as typed objects,
guages [5], and APIs (application programming interfaces) called entities, such as a sales order or a customer. Entity sets
[6–10]. Many HCI techniques are applicable, and we have are collections of entities of the same type, such as a set of
used contextual inquiry, heuristic evaluation, cognitive walk- sales orders. Links define relationships between entities,
through, ethnography, and controlled lab studies. Improving such as the customer of a sales order. OData uses the HTTP
the usability of developer tools can allow developers to work REST methods (GET, PUT, POST, DELETE) for exchang-
more efficiently and enable more people to become develop- ing the messages. There are OData libraries for common
ers. This in turn should improve the quality of the resulting mobile programming languages (Objective-C, Java, .NET),
programs since more iterations can be made and more people and also for the Microsoft ASP.NET framework when an
can be involved in the development process. HTML-based user interface is desired. These programming
For more than five years, our group at CMU has been languages enable applications to target the web and most
collaborating with several teams at SAP to improve the usa- mobile devices.
bility of their developer tools. Members of our group have As part of the NetWeaver Gateway product, SAP devel-
helped with the redesign of a business process rules engine oped the “SAP NetWeaver® Gateway developer tool for
called SAP BRFplus [11], and proposed improvements to Visual Studio®” (hereafter “Gateway VS tool”) that helps
SAP’s eSOA (enterprise Service Oriented Architectures) developers create “starter kit” applications for the ASP.NET
web services [6, 10]. framework. The Gateway VS tool provides a wizard-style
user interface in which developers design a basic version of
II. SAP NETWEAVER GATEWAY their application. The developer selects a Gateway service,
The SAP Business Suite provides customers with a set of
business applications and tools for managing their supply 1
chains, supplier relations, human resources, accounting, and http://www.sdn.sap.com/irj/sdn/gateway
2
http://www.odata.org/

978-1-4673-1824-2/11/$26.00 c 2011 IEEE 556 CHASE 2012, Zurich, Switzerland

chooses the types of entities and the entity properties to be the Gateway VS tool and what their expectations might be
displayed, and specifies the navigation between entities. The about how the user interface should work.
Gateway VS tool then generates code, which implements the The SAP NetWeaver Gateway Product Team reviewed
UI, and proxies, which connect to the SAP system. The de- the internal report and prioritized the issues that it raised into
veloper can inspect and modify the generated code, and the different severity levels. As is accustomed in agile software
code also serves as an example to help the developer learn development methodologies, there was a clear need to turn
how to consume Gateway services using the ASP.NET li- the evaluations into product specifications of tangible fea-
braries. Similar tools and plug-ins for IDEs have also been tures for the next release. Hence, the CMU findings that were
developed for Eclipse and Adobe Flash Builder. identified as “important” and “very important” were added to
the next Gateway VS tool release development plan.
III. USABILITY EVALUATIONS
B. Cognitive Walkthrough
The CMU team consulted for six months with the SAP
NetWeaver Gateway Product Team. The CMU and SAP In contrast to heuristic evaluation, which takes a more
teams were not collocated, so we communicated electroni- holistic view, the cognitive walkthrough technique evaluates
cally and held weekly teleconferences. As part of our consul- how well a user interface supports one or more specific tasks
tation, SAP provided to us an early version of the Gateway [14, 15]. The SAP team developed a common business use
VS tool, which had not yet been released to the market, and case for a typical application that would consume SAP
we performed two evaluations to find usability problems. In NetWeaver Gateway services. The CMU team then attempt-
the first evaluation, we performed a heuristic evaluation [13] ed to build such an application using the Gateway VS tool
of the Gateway VS tool. In the second evaluation, we per- and the documentation provided by SAP. As we worked to
formed a cognitive walkthrough [14] of the Gateway VS tool build the application we carefully documented the issues that
by building an application for a common business use case we encountered. For this evaluation, we used the updated
that SAP had identified. Gateway VS tool, which had already fixed several of the
important problems that we had reported based on our heu-
A. Heuristic Evaluation ristic analysis.
The heuristic evaluation technique [13] helps to identify This evaluation focused on modifying the “starter kit”
usability problems by having an expert examine the user application code that the Gateway VS tool generates. We
interface with respect to heuristics that describe high-level discovered several problems with the generated code. Since
behaviors that applications should have. Nielson developed the NetWeaver Gateway was still under development, this is
an initial set of heuristics such as “speak the user’s language: not unexpected. While we were able to work around the bugs
… the terminology in user interfaces should be based on the in the generated code, we encountered problems with the
users’ language and not on system-oriented terms” (p. 123 in server that prevented us from completing the common busi-
[13]) and provide good error messages that use clear lan- ness use case. We also received error messages that we did
guage and “constructively help the user solve the problem” not fully understand and found a few additional issues with
(pp. 142–3 in [13]). We examined the tool’s user interface to the user interface of the tool itself. We discussed these prob-
find where it might violate the heuristics. We identified usa- lems during the weekly teleconferences and summarized
bility gaps relating to the button labels, screen organization, them in a final report, which carefully documented the usa-
error messages, inconsistent workflow, lack of warning mes- bility problems and functional errors with the Gateway VS
sages, and so forth. We discussed these issues during our tool, the generated code, and the server.
weekly teleconferences and later summarized all the issues in Just as in the previous phase, SAP’s use of the agile
an internal report documenting the problems and suggesting software development method enabled them to fix the bugs
ways to improve the user interface. and provide us with updated versions of both the Gateway
The SAP team developing this software uses the agile VS tool and the Gateway server. We were then able to con-
software development process 3 . The quick iteration cycle tinue to work on the use case from the point where we had
allowed them to improve the software based upon our feed- previously become stuck. With the bugs fixed, we were able
back. We were often given new versions to evaluate that to complete the common business use case and write a sum-
incorporated changes in response to issues we had recently mary report. In this final report, we noted some of the diffi-
reported. After about three months from when we delivered culties we had modifying the code due to our lack of famili-
our report, SAP released an updated version of the Gateway arity with the ASP.NET libraries. Since SAP plans to target
VS tool to market, which incorporated several of our sug- this tool to developers who do have experience using that
gested changes. library, our findings highlight the need for the tool’s docu-
The heuristic evaluation that was conducted by the CMU mentation to clearly state what pre-requisite knowledge a
team provided valuable input to the SAP NetWeaver Gate- developer should have and to provide suggestions for obtain-
way Product team from the point of view of a non-SAP de- ing information about third-party tools when needed.
veloper who would like to interact with SAP services from
their own native environment. The CMU team’s internal IV. AGILE DEVELOPMENT AND HCI TECHNIQUES
report revealed how ASP.NET developers might approach As have others [16], we found the agile development and
HCI techniques to work well together. The agile process
3
http://www.agilealliance.org/ splits the development effort into a series of sprints, each

557
 lasting a pre-defined number of weeks. Each sprint begins REFERENCES
with a planning meeting in which the product owner explains [1] B. A. Myers, A. J. Ko, and M. M. Burnett, “Invited research
the product requirements to the team. The team then drills overview: end-user programming,” in Proc. ACM Conference
into each requirement to create tasks and provide effort esti- Extended Abstracts on Human Factors in Computing Systems
mates, and reports back to the product owner to commit to (CHI'06), Montreal, Canada, 2006, pp. 75–80.
the requirements that can fit into the sprint. The HCI evalua- [2] M. Mooty, A. Faulring, J. Stylos, and B. A. Myers, “Calcite:
tion techniques can be used to quickly evaluate the latest completing code completion for constructors using crowds,” in Proc.
IEEE Symp. on Visual Languages and Human-Centric Computing
version of the software, generating requirements to be ad- (VL/HCC'10), Leganés-Madrid, Spain, 2010, pp. 15–22.
dressed in upcoming sprints. The rapid, iterative nature of [3] T. D. LaToza and B. A. Myers, “Visualizing call graphs,” in Proc.
both these processes work well together. IEEE Symp. on Visual Languages and Human-Centric Computing
The sprint ends in a review meeting where the team (VL/HCC'11), Pittsburgh, PA, 2011, pp. 117–124.
shows the product owner the improvements made during the [4] A. J. Ko and B. A. Myers, “Debugging reinvented: asking and
sprint. In addition, the product owner can decide after the answering why and why not questions about program behavior,” in
review meeting to release the product to the market or only Proc. ACM/IEEE Int. Conference on Software Engineering
(ICSE'08), Leipzig Germany, 2008, pp. 301–310.
to small a group of people, such as to the CMU evaluators,
[5] J. F. Pane, B. A. Myers, and L. B. Miller, “Using HCI techniques to
according to the product’s stability, maturity, and so forth. design a more usable programming system,” in Proc. IEEE 2002
Symposia on Human Centric Computing Languages and
V. CONCLUSIONS Environments (HCC'02), Arlington, VA, 2002, pp. 198–206.
Our collaboration has demonstrated that HCI techniques [6] J. Beaton, S. Y. Jeong, Y. Xie, J. Stylos, and B. A. Myers, “Usability
can be applied to improve the usability of software develop- challenges for enterprise Service-Oriented Architecture APIs,” in
Proc. IEEE Symp. on Visual Languages and Human-Centric
ment tools. When combined with agile software develop- Computing (VL/HCC'08), Herrsching am Ammersee, Germany, 2008,
ment methodology, HCI techniques can yield quick im- pp. 193–196.
provements to such tools. [7] D. S. Eisenberg, J. Stylos, A. Faulring, and B. A. Myers, “Using
SAP NetWeaver Gateway Product team has been con- association metrics to help users navigate API documentation,” in
stantly engaging with different non-SAP development com- Proc. IEEE Symp. on Visual Languages and Human-Centric
munities in order to create a product that makes it easier to Computing (VL/HCC'10), Leganés-Madrid, Spain, 2010, pp. 23–30.
consume the SAP services from any UI environment. The [8] J. Stylos, A. Faulring, Z. Yang, and B. A. Myers, “Improving API
documentation using API usage information,” in Proc. IEEE Symp.
collaboration with CMU had corroborated the importance of on Visual Languages and Human-Centric Computing (VL/HCC'09),
evaluating how non-SAP developers would interact with Corvallis, OR, 2009, pp. 119–126.
SAP services and of identifying the gaps in expectations, [9] J. Stylos, B. A. Myers, and Z. Yang, “Jadeite: improving API
especially when compared to a more SAP-savvy developer. documentation using usage information,” in Proc. ACM Conference
The applicability of CMU findings extends beyond the Extended Abstracts on Human Factors in Computing Systems
Gateway VS tool. Based on the reports, several product spec- (CHI'09), Boston, MA, 2009, pp. 4429–4434.
ifications were also incorporated for the next release of SAP [10] B. A. Myers, S. Y. Jeong, Y. Xie, J. Beaton, J. Stylos, R. Ehret, J.
NetWeaver Gateway plug-in for Eclipse. In October 2011 Karstens, A. Efeoglu, and D. K. Busse, “Studying the documentation
of an API for enterprise Service-Oriented Architecture,” JOEUC: The
SAP NetWeaver Gateway, including the developer tools, Journal of Organizational and End User Computing, vol. 22, pp. 23–
was released for general availability in the market. 51, Jan–Mar 2010.
Looking into the future, the SAP NetWeaver Gateway [11] J. Stylos, D. K. Busse, B. Graf, C. Ziegler, R. Ehret, and J. Karstens,
Product team has been inspired to continue to utilize human- “A case study of API design for improved usability,” in Proc. IEEE
centered approaches when creating tools for software devel- Symp. on Visual Languages and Human-Centric Computing
opers, through close engagement with the different developer (VL/HCC'08), Herrsching am Ammersee, Germany, 2008, pp. 189–
192.
communities. This will enable those developers to consume
[12] R. T. Fielding, “Architectural styles and the design of network-based
services from their SAP systems in an easy-to-use, con- software architectures,” Ph.D. dissertation, Information and Computer
trolled, and effective way. Meanwhile, the CMU team is now Science, Univ. of California, Irvine. 2000.
collaborating with a different group at SAP, to further evalu- [13] J. Nielsen, Usability Engineering. Boston, MA: Academic Press,
ate and adapt HCI methods to improve many different forms 1993.
of APIs and programmer tools. [14] M. H. Blackmon, P. G. Polson, M. Kitajima, and C. Lewis,
“Cognitive walkthrough for the web,” in Proc. ACM Conference on
ACKNOWLEDGMENT Human Factors in Computing Systems (CHI'02), Minneapolis, MN,
2002, pp. 463–470.
We thank SAP Labs for funding this research, in particu-
[15] C. Lewis, P. G. Polson, C. Wharton, and J. Rieman, “Testing a
lar Ralf Ehret, Paul Hofmann and Ike Nassi. Opinions, find- walkthrough methodology for theory-based design of walk-up-and-
ings and conclusions or recommendations expressed in this use interfaces,” in Proc. ACM Conference on Human Factors in
material are those of the authors and do not necessarily re- Computing Systems (CHI'90), Seattle, WA, 1990, pp. 235–242.
flect those of SAP. [16] M. Budwig, S. Jeong, and K. Kelkar, “When user experience met
agile: a case study,” in Proc. ACM Conference Extended Abstracts on
Human Factors in Computing Systems (CHI'09), Boston, MA, 2009,
pp. 3075–3084.

558

View publication stats