If you’re reading this blog because you’re interested in ANN search, this is a small update for you! The product quantization (PQ) support I’ve been working on in the SPTAG project has been open-sourced.

SPTAG is an ANN search algorithm that leverages a combination of a space partitioning tree (usually, a balanced k-means tree or k-d tree) and graph (usually, a relative neighborhood graph or KNN graph) to try and provide a good balance between high recall, reasonable index build times, and throughput. Product Quantization support offers a new trade-off between recall, memory usage, and throughput that may be a better fit for some scenarios. I discuss PQ in some more detail in another post on this blog.

There has been a lot of exciting open source development in SPTAG recently. Another feature (not my work) that was recently open-sourced is support for hybrid memory-disk indexes. This helps to serve very large indexes efficiently. You can read more about the hybrid index feature in this paper.

I plan on posting some benchmarks on a public dataset soon. Until then, if you’re interested in the feature, try it out!