Abstract
Distributed file system is one of the key blocks of cloud computing systems. With the fast increase of user scale and data amount, metadata management has become a crucial point affecting the overall performance of a distributed file system. In this paper, we design and implement PPMS, a novel metadata management strategy in a peer to peer way. Different from existing metadata management methods, we adopt a two layer structure to achieve high scalability and low latency. The upper layer is metadata index server, which is used to store metadata of directories, while the lower layer consists of metadata servers to store the metadata of files. More importantly, the lower layer is organized in a peer to peer way to further improve scalability. We implement a prototype file system based on PPMS and evaluate its performance via experiments. The results show that our design can achieve high performance with in terms of time latency and system throughput.
Chapter PDF
Similar content being viewed by others
References
Roselli, D.S., Lorch, J.R., Anderson, T.E.: A Comparison of File System Workloads. In: USENIX Annual Technical Conference, General Track, pp. 41–54 (2000)
Brandt, S.A., Xue, L., Miller, E.L., et al.: Efficient metadata management in large distributed storage systems. In: 2012 IEEE 9th International Conference on Mobile Ad-Hoc and Sensor Systems (MASS 2012), p. 290. IEEE Computer Society (2012)
Corbett, P.F., Feitelson, D.G.: The Vesta parallel file system. ACM Transactions on Computer Systems (TOCS) 14(3), 225–264 (1996)
Weil, S.A., Pollack, K.T., Brandt, S.A., et al.: Dynamic metadata management for petabyte-scale file systems. In: Proceedings of the 2004 ACM/IEEE Conference on Supercomputing, p. 4 (2004)
Zhu, Y., Jiang, H., Wang, J.: Hierarchical bloom filter arrays (hba): a novel, scalable metadata management system for large cluster-based storage. In: 2004 IEEE International Conference on Cluster Computing, pp. 165–174. IEEE (2004)
Hua, Y., Zhu, Y., Jiang, H., et al.: Scalable and adaptive metadata management in ultra large-scale file systems. In: ICDCS, pp. 403–410 (2008)
Borthakur, D.: The hadoop distributed file system: Architecture and design. Hadoop Project Website 11, 21 (2007)
Ghemawat, S., Gobioff, H., Leung, S.T.: The Google file system. ACM SIGOPS Operating Systems Review 37(5), 29–43 (2003)
Leung, A.W., Shao, M., Bisson, T., et al.: Spyglass: Fast, Scalable Metadata Search for Large- Scale Storage Systems. FAST, pp.153-166 (2009)
Hua, Y., Jiang, H., Zhu, Y., et al.: SmartStore: A new metadata organization paradigm with semantic-awareness for next-generation file systems. In: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, pp. 1–12. IEEE (2009)
Broder, A., Mitzenmacher, M.: Network applications of bloom filters: A survey. Internet mathematics 1(4), 485–509 (2004)
Weil, S.A., Brandt, S.A., Miller, E.L., et al.: Ceph: A scalable, high-performance distributed file system. In: OSDI, pp. 307–320 (2006)
Wang, J., Feng, D., Wang, F., et al.: MHS: A distributed metadata management strategy. Journal of Systems and Software 82(12), 2004–2011 (2009)
Moosefs, http://www.moosefs.org/
Katcher, J.: Postmark: A new file system benchmark. Technical Report TR3022, Network Appliance (1997), http://www.netapp.com/tech_library/3022.html
Patil, S., Gibson, G.A.: Scale and Concurrency of GIGA+: File System Directories with Millions of Files. In: FAST 2011, p. 13 (2011)
Trace,tracehost.cs.berkeley.edu
Yu, J., Wu, W., Li, H.: DMooseFS: Design and implementation of distributed files system with distributed metadata server. APCloudCC, pp.42-47 (2012)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 IFIP International Federation for Information Processing
About this paper
Cite this paper
Yang, D., Wu, W., Li, Z., Yu, J., Li, Y. (2014). PPMS: A Peer to Peer Metadata Management Strategy for Distributed File Systems. In: Hsu, CH., Shi, X., Salapura, V. (eds) Network and Parallel Computing. NPC 2014. Lecture Notes in Computer Science, vol 8707. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-44917-2_36
Download citation
DOI: https://doi.org/10.1007/978-3-662-44917-2_36
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-662-44916-5
Online ISBN: 978-3-662-44917-2
eBook Packages: Computer ScienceComputer Science (R0)