VSEncoding: efficient coding and fast decoding of integer lists via dynamic programming

Silvestri F., Venturini R.

Information Storage  Data Storage Representations  Data Compression  Inverted lists 

Encoding lists of integers efficiently is important for many applications in different fields. Adjacency lists of large graphs are usually encoded to save space and to improve decoding speed. Inverted indexes of Information Retrieval systems keep the lists of postings compressed in order to exploit the memory hierarchy. Secondary indexes of DBMSs are stored similarly to inverted indexes in IR systems. In this paper we propose Vector of Splits Encoding (VSEncoding), a novel class of encoders that work by optimally partitioning a list of integers into blocks which are efficiently compressed by using simple encoders. In previous works heuristics were applied during the partitioning step. Instead, we find the optimal solution by using a dynamic programming approach. Experiments show that our class of encoders outperform all the existing methods in literature by more than 10% (with the exception of Binary Interpolative Coding with which they, roughly, tie) still retaining a very fast decompression algorithm.

Source: Conference on Information and Knowledge Management, pp. 1219–1228, Toronto, Canada, 27 ottobre 2010

Publisher: ACM Press, New York, USA