Dropdown search results: Video Suggestion or Keyword Suggestion?

Introduction:

Which one is better? Video suggestion or Keyword suggestion.

(Since this A/B test was conducted on a VOD product, you can rely on its impact only for this type of product.)

Here, we will examine two options of dropdown suggestions in Aparat's internal search box.Before dealing with this article, let's take a look at how Aparat's internal search works. Aparat's internal search uses a video suggestion drop-down system and when the user types a word, a drop-down offers 4 related videos. If the user doesn't find the desired result, he can click on the More result button or hit the “Enter” key to visit the search result page.

Research:

Since we received some reports from users that the search suggestion was not working correctly during interviews and user tests, we began analyzing the search data.We requested the data team to prepare all the data related to the search for us. Data such as the number of users using search, total engagement with drop-down results, video watching percentage after clicking on a search result, the number of times users hit "Enter" to see more results, etc.

According to our data, the total engagement with the drop-down results is 50%, and 12% of users who clicked on a video in the search result drop-down watched 40% of it.So, the First engagement after clicking on the search input is 50% and 68% of them are using the “Enter” key and the others clicked on items.Having checked the quantitative data of the search suggestion, we held meetings with the algorithm team and our product team to discuss how the suggested content could be better. Based on its data, we assumed Aparat's search results aren't very accurate, and the content algorithm isn't up-to-date, so it doesn't lead users to accurate information.Moreover, we found out that we don’t take advantage of search trends. When the user types something in the search bar, our algorithm searches the titles of all videos for that exact word without prioritizing them or sorting them by considering social trends or events.

Not a good idea to show an entertaining video about "Ukraine" in this position.

A/B Test:

As it became clear that the current search performance could be improved, we decided to change the content type of our search suggestions from Videos to Keywords and then conduct an A/B test to assess its performance.To design the Keyword suggestion version of the search result drop-down, I wanted my teammates to take into consideration the UX of Designing search suggestions, as explained by the NNgroup: https://www.youtube.com/watch?v=7XpQVWaTO2sHere's what we came up with:

This is all we wanted to test for a month in an AB test.

Result:

Once it had been developed, the Data and performance team prepared the following report after a month of gathering data:

In comparison to the first interaction, the new search suggestion is 24% more engaging for users, and 25% more effective in using the "Enter" key.(The first engagement in the old search was 50%, and68% used the “Enter” key, while the rest clicked on items; in the new search, the first engagement was62%, and 85% used the “Enter” key.)