Document - Static and dynamic big data partitioning on apache spark

2016

Conference article Restricted

Static and dynamic big data partitioning on apache spark

Bertolucci M., Carlini E., Dazzi P., Lulli A., Ricci L.

Apache Spark BigData Data partitioning Graph algorithms

Many of today's large datasets are organized as a graph. Due to their size it is often infeasible to process these graphs using a single machine. Therefore, many software frameworks and tools have been proposed to process graph on top of distributed infrastructures. This software is often bundled with generic data decomposition strategies that are not optimised for specific algorithms. In this paper we study how a specific data partitioning strategy affects the performances of graph algorithms executing on Apache Spark. To this end, we implemented different graph algorithms and we compared their performances using a naive partitioning solution against more elaborate strategies, both static and dynamic.

Source: International Conference on Parallel Computing, pp. 489–498, Edinburgh, Scotland, 1-4 September 2015

Metrics

Back to previous page

Cite as

BibTeX entry

@inproceedings{oai:it.cnr:prodotti:366242,
	title = {Static and dynamic big data partitioning on apache spark},
	author = {Bertolucci M. and Carlini E. and Dazzi P. and Lulli A. and Ricci L.},
	doi = {10.3233/978-1-61499-621-7-489},
	booktitle = {International Conference on Parallel Computing, pp. 489–498, Edinburgh, Scotland, 1-4 September 2015},
	year = {2016}
}