Game Changer! Building Search Into Your Applications

Listen up, people! You've got data and you've got users and your users need to access your data. So whether it's Google, whether it's Amazon, whether it's Stack Overflow, Salesforce says that 87% of e-commerce shoppers start their journey in the search bar. And Forrester says that 68% of those shoppers will give up their journey if you provide a bad user experience.

Now this search bar looks simple but on the other side of that search bar are your users and they are not so simple. They don't know what they want. They don't know how to express what they want. They don't know how to spell it. And that's getting them to get what they want is what I call the search game. And when you play the search game right you could take this search bar and you can turn it into this giant goal. A goal so big in fact that your users simply can't miss. They will net everything that they're looking for including the things that they didn't even know that they were looking for. When you play the search game right, that means you get more engagement, more clicks, more users, more likes, more sharing, and more revenue. So then your competitors and everybody who is not you is your competitor.

So I am going to coach you today on how to implement the right strategies and the right tools you'll need to get this. The first thing you need for your proper equipment is a document database. More users when they want your data they're probably searching through volumes and volumes of unstructured and semi-structured data. Now, relational databases are fantastic for tables. Anything that's in a column in a row is great. When you know the query pattern ahead of time, it's great for that. But for search, the performance goes down so you'll want a document data, database.

The next star player you have is Apache Lucene. Apache Lucene is your star player. All the winning teams play with Apache Lucene. Netflix, Walmart, Ebay, it is a battle-tested, open source. It's been around for 20 years. That's why they play it. And you can build your own thing but why would you farm a promising, upcoming player when Messi's already warmed up, ready and hungry to play for you. So Apache Lucene is your star player and its big power play is that it takes that document database and it runs that data through a process called analysis. Analysis is going to take that data, break it down into different tokens depending on the analyzer you use and those tokens are stored in an inverted index. So Lucene uses an inverted index.

So let's run through this process in a practice play, see how it feels. If I have these four documents with these soccer teams in it and their unique underscore ID field, if I were to look through those documents for Manchester United, it would lowercase everything, remove all the punctuation, I'm left with two different tokens, Manchester and United, using the standard analyzer in Lucene. Those are my two tokens.

So when I look through these documents for those things, my tokens or my terms will map two documents, one in two for Manchester and one in three for United. So my inverted index will hold my tokens or my terms, what documents and other helpful metadata, frequency, position, etc. Now having the right tokens or the right terms can make or break a good search experience for you. So it's important to use the right analyzer to get the right terms.

Now I'm going to show you what I mean in this example. So this is an app that I wrote called Atlas Search Soccer. It uses Atlas Search. In it, I used the FIFA Player Database so you can find lots of different search options to find your FIFA dream team and you can put your own players on there. It'll also show you the code on the queries on how to do that. Now I know it's called football everywhere else in the entire world except the United States, but I already bought the domain name so we're just going to stick with that. So in this one I am looking for players from Manchester United. I'm using the standard analyzer. As you remember, it's Manchester United. So I'm going to get 697 players when I look for Manchester United because it's giving me Manchester United and West Ham United and Manchester City and anything else with either Manchester United. If, however, I change to the keyword analyzer as I'm doing here, I find 33 matching players and they are all indeed from Manchester United because when I pass, this is using the keyword analyzer which takes everything, it keeps the punctuation, it keeps the capital letters, all the casing, and it gives me that one token. So keyword analyzers are fantastic if you're using check boxes.

So your tokens matter which means your analyzers matter. The next thing you need to consider are your query operators, whether it's regex, phrase, text, whether using facets or if you're using auto-complete. This is a way to let your users take their best shot. Every user is different. Every user has a different preference of how they're going to go through things so you want to, in your application, give them as many options as possible. And of course I can't talk about giving your users those best shot at finding your data without talking about scoring. Relevant scoring is so important in search. All search engines are going to grade all your documents based on how well they match the search query and that is called relevance. And it is going to return your documents to you with the score in descending order. So it's going to give you what it thinks the best matches are, the most relevant matches first.

In this example, for instance, I'm looking for Cristiano Ronaldo for my FIFA dream team and I just look for Ronaldo and I get this lovely gentleman first but that's not the Ronaldo I want because it's looking for relevance first. I want the overall FIFA score to be very high in that. So in this one, in the query, I'm going to take the overall FIFA score which is the field in every one of my documents. I'm going to factor it in to my relevance score and now I get Cristiano Ronaldo first. He's very difficult personality-wise but he's very good and I want him on my dream team. So with that, you want scoring matters, custom scoring because you want everything right, think about your data, think about your user interface, think about your tokens and I'm not clicking through, I had such a nice interactive, oh there it goes. Think about your tokens, goes in, pick your analyzer accordingly that goes in your index inside of your queries and all of that will be served up to your users so they have their best shot at finding your data before they find it at your competitors.