Abstract
As storage systems scale from a few storage nodes to hundreds or thousands, data distribution and load balancing become increasingly important. We present a novel decentralized algorithm, RDIM (Replication Under Dynamic Interval Mapping), which maps replicated objects to a scalable collection of storage nodes. RDIM distributes objects to nodes evenly, redistributing as few objects as possible when new nodes are added or existing nodes are removed to preserve this balanced distribution. It supports weighted allocation and guarantees that replicas of a particular object are not placed on the same node. Its time complexity and storage requirements compare favorably with known methods.
Supported by the National Basic Research Program 973 of China (No.2003CB317008).
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Xin, Q., Miller, E.L., Long, D.D.E., Brandt, S.A., Schwarz, T., Litwin, W.: Reliability mechanisms for very large storage systems. In: Proceedings of the 20th IEEE / 11th NASA Goddard Conference on Mass Storage Systems and Technologies, April 2003, pp. 146–156 (2003)
Litwin, W., Neimat, M.-A., Schneider, D.A.: LH*—a scalable, distributed data structure. ACM Transactions on Database Systems 21(4), 480–525 (1996)
Devine, R.: Design and implementation of DDH: A distributed dynamic hashing algorithm. In: Proceedings of the 4th International Conference on Foundations of Data Organization and Algorithms, pp. 101–114 (1993)
Choy, D.M., Fagin, R., Stockmeyer, L.: Efficiently extendible mappings for balanced data distribution. Algorithmica 16, 215–232 (1996)
Brinkmann, A., Salzwedel, K., Scheideler, C.: Efficient, distributed data placement strategies for storage area networks. In: Proceedings of the 12th ACM Symposium on Parallel Algorithms and Architectures (SPAA), pp. 119–128. ACM Press, New York (2000) Extended Abstract
Brinkmann, A., Salzwedel, K., Scheideler, C.: Compact, adaptive placement schemes for non-uniform capacities. In: Proceedings of the 14th ACM Symposium on Parallel Algorithms and Architectures (SPAA), Winnipeg, Manitoba, Canada, August 2002, pp. 53–62 (2002)
Honicky, R.J., Miller, E.L.: A fast algorithm for online placement and reorganization of replicated data. In: Proceedings of the 17th International Parallel & Distributed Processing Symposium, Nice, France (April 2003)
Honicky, R.J., Miller, E.L.: Replication under scalable hashing: A family of algorithms for scalable decentralized data distribution. In: Proceedings of the 18th International Parallel & Distributed Processing Symposium (IPDPS 2004), Santa Fe, NM, April 2004. IEEE, Los Alamitos (2004)
Liu, Z., Zhou, X.-M.: An Adaptive Data Objects Placement Algorithm For Non-Uniform Capacities. In: Proceedings of the 3rd International Conference on Grid and Cooperative Computing, WuHan (October 2004)
Matsumoto, M., Nishimura, T.: Mersenne Twister: A 623-dimensionally equidistributed uniform pseudorandom number generator. ACM Trans. on Modeling and Computer Simulation 8(1), 3–30 (1998)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Liu, Z., Xiao, N., Zhou, XM. (2005). RDIM: A Self-adaptive and Balanced Distribution for Replicated Data in Scalable Storage Clusters. In: Pan, Y., Chen, D., Guo, M., Cao, J., Dongarra, J. (eds) Parallel and Distributed Processing and Applications. ISPA 2005. Lecture Notes in Computer Science, vol 3758. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11576235_6
Download citation
DOI: https://doi.org/10.1007/11576235_6
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-29769-7
Online ISBN: 978-3-540-32100-2
eBook Packages: Computer ScienceComputer Science (R0)