Advanced Solr/Lucene topics: high-performance nested search for e-commerce applications
Solr/Lucene has emerged over the last few years as a leading open source search platform for large-scale e-commerce search engines. Systems based on Solr power major sites including Macy’s, Kohl’s, Walmart, Etsy, and many others. An increasing number of tier-1 digital retailers are building their next-generation search and catalog navigation platforms using the Solr technology stack, often replacing commercial engines such as Oracle Endeca, FAST or Mercado.
Grid Dynamics has been one of the early adopters of Solr for large-scale, complex e-commerce catalogs with millions of SKUs, providing a highly optimized, omni-channel, “Black Friday-ready” experience for some of the world’s largest retailers.
Our engineers have made numerous contributions to Solr and Lucene. They’ve spoken at technical conferences and published many technical blogs that help Solr developers. One specific area of extensive research and innovation where our team shines is dealing with nested document structures, which are very useful when modeling complex e-commerce catalogs. The Nested Search and Faceting approach solves many of the performance and scalability challenges encountered when building large scale, contextualized, omnichannel catalogs. It’s especially useful for effective and scalable processing of relationships (such as “chair is a part of furniture set”) and their attributes in a search system, optimized to deal with documents which are completely independent from one another.
- We were early adopters of, and actively advocated the Block Join (also known as index-time join) approach to implementing nested search. This approach is supported in Lucene with Block Join Query (BJQ) and in Solr with the Block Join Facet component, which we contributed to the Solr/Lucene codebase.
For a more complete introduction to this topic, please see the joint talk at Lucene Revolution 2015 given by Eugene Steinberg of Grid Dynamics and Peter Gazaryan of Macys.com. Both slides and a video are online.
In this series of blog posts on Nested Search for E-commerce, we are republishing a collection of revised and updated blog posts written over the last 2 years. These posts are specifically targeted at the developers of the search engines, so the information is presented in rather technical form, and assumes good familiarity with Solr/Lucene framework. Specifically, these blog posts will cover the following aspects of BJQ design:
Post 1: Introduction to Block Join Faceting
Post 2: High-Performance Join in Solr with BlockJoinQuery
Post 3: How to Implement Block Join Faceting in Solr/Lucene
Post 4: Using Block Join to Improve Search Efficiency with Nested Documents in Solr
Post 5: The Segmented Filter Cache and Block join Query Parser in Solr
Post 6: Searching Grandchildren and Siblings with Solr Block Join
Post 7: A Frustrating Personal Experience with Unfaceted Search
If you have questions about any of the topics covered in this series of posts or more generally related to the design and tuning of search engines, please drop us a line and one of our Search Architects will follow up promptly.
Eugene Steinberg, Victoria Livschitz