Computing infrastructure for big data processing

With computing systems undergone a fundamental transformation from single-processor devices at the turn of the century to the ubiquitous and networked devices and the warehouse-scale computing via the cloud, the parallelism has become ubiquitous at many levels. At micro level, parallelisms are being explored from the underlying circuits, to pipelining and instruction level parallelism on multi-cores or many cores on a chip as well as in a machine. From macro level, parallelisms are being promoted from multiple machines on a rack, many racks in a data center, to the globally shared infrastructure of the Internet. With the push of big data, we are entering a new era of parallel computing driven by novel and ground breaking research innovation on elastic parallelism and scalability. In this paper, we will give an overview of computing infrastructure for big data processing, focusing on architectural, storage and networking challenges of supporting big data paper. We will briefly discuss emerging computing infrastructure and technologies that are promising for improving data parallelism, task parallelism and encouraging vertical and horizontal computation parallelism.

Authors and Affiliations

1. Distributed Data Intensive Systems Lab, School of Computer Science, Georgia Institute of Technology, Atlanta, 30332, USA

   Ling Liu

Corresponding author

Correspondence to Ling Liu.

Ling LIU is a full professor in the School of Computer Science at Georgia Institute of Technology. She directs the research programs in Distributed Data Intensive Systems Lab (DiSL). Prof. Liu’s researoh interests are in the areas of cloud computing, big data and big data analytics, distributed computing, and Internet services, with the focus on performance, availability, fault tolerance, security and privacy. She has published over 300 international journal and conference papers. She is a recipient of 2012 IEEE Computer Society Technical Achievement Award and an Outstanding Doctoral Thesis Advisor Award in 2012 from Georgia Institute of Technology. She is a co-Editor-in-Chief of the 5 volumes of Encyclopedia of Database Systems (Springer 2010), the Editorin-Chief of IEEE Transactions on Service Computing, and is on the editorial board of over a dozen international journals. Her research is primarily supported by grants from the U.S. National Science Foundation (NSF) and industrial companies such as IBM and Intel.