Scalability issues for self similarity join in distributed systems

Gennaro C., Rabitti F.

Scalability  Similartiy self join  Metric Space  Information Search and Retrieval 

Efficient processing of similarity joins is important for a large class of data analysis and data-mining applications. This primitive finds all pairs of records within a predefined distance threshold of each other. However, most of the existing approaches have been based on spatial join techniques designed primarily for data in a vector space. Treating data collections as metric objects brings a great advantage in generality, because a single metric technique can be applied to many specific search problems quite different in nature. In this paper, we concentrate our attention on a special form of join, the Self Similarity Join, which retrieves pairs from the same dataset. In particular, we consider the case in which the dataset is split into subsets that are searched for self similarity join independently (e.g, in a distributed computing environment). To this end, we formalize the abstract concept of epsilon-Cover, prove its correctness, and demonstrate its effectiveness by applying it to two real implementations on a real-life large dataset.

Source: The 18th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP), pp. 309–316, Pisa, 17-19 February 2010

Metrics