Building a dynamic analytics dashboard with Apache solr and banana in 10 minutes

Steps

1. Install Solr

2. Start Solr and create a new collection

3. Download and modify the dataset

4. Use NLTK stop words

5. Index the dataset

6. Clone and install Banana

7. Configure Banana

8. Create the dashboard and panels

1. Installing Apache Solr

Download the latest version of Apache Solr from Apache mirrors (https://lucene.apache.org/solr/downloads.html) and extract it to a local directory or to a directory on a server if you plan to run this tutorial on a server. We will assume that Solr is extracted to $SOLR_HOME directory.

2. Updating Stopwords File

Out of the box, Solr comes with a stop words lists for several languages, however a more comprehensive stop words list, for English in this tutorial, is included with NLTK Natural Language Processing library. For this tutorial, download the stop words list from https://gist.github.com/sebleier/554280 and copy the file to _default/stopwords.txt under configsets directory:

$ cd $SOLR_HOME/server/solr/configsets/_default/conf

$ curl https://gist.githubusercontent.com/sebleier/554280/raw/7e0e4a1ce04c2bb7bd41089c9821dbcf6d0c786c/NLTK\'s%2520list%2520of%2520english%2520stopwords > nltk_stopwords.txt

$ cp nltk_stopwords.txt stopwords.txt

3. Starting Solr and Creating a New Collection

Start Solr in cloud mode:

$ cd $SOLR_HOME

$ bin/solr start -c

Next, create a new collection to be used for indexing the data:

$ bin/solr create -c greads

4. Indexing the Dataset

For this tutorial, Goodreads books dataset will be used. Download the dataset from Kaggle in CSV format and update column names to match dynamic fields schema settings in Solr. Dynamic fields is a feature that allows inference of fields data types based on a postfix naming convention. The most common / basic postfixes are:

_i and _l: integer and long respectively

_f and _d: float and double respectively

_s and _t: string and text respectively

Note: adding s to any of these postfixes creates a multi-value field of the same data type instead of a single-value field.

So, update column names in the CSV file as follows:

• bookID_i

• title_t

• authors_s

• average_rating_f

• isbn_s isbn13_s

• language_code_s

• num_pages_i

• ratings_count_i

• text_reviews_count_i

5. Indexing the Dataset Using Out-of-the-box Post Command Line

To index the dataset into the collection created in step 3, the post command can be used as follows:

$ bin/post -c greads /path/to/data/books.csv

Note: the dataset contains 5 records that have invalid average_rating_f values. Removing these records is necessary for the indexing to complete successfully.

6. Installing Banana

Now we have the data indexed into Solr, we are ready to build the dashboard. Clone Banana from GitHub repository and follow the installation instructions in the Readme file.

7. Configuring The Dashboard

Navigate to http://localhost:8983/solr/banana/index.html. If the installation is successful, the default dashboard should be displayed inside the browser window. Click the top right gear and select Solr tab then change collection name value to greads. Since the data set is not a timeseries, remove the timepicker panel from the second row and the histogram from the fourth row. The timepicker filter also needs to be removed as well from the third row.

8. Creating Panels

To get the longest book in the dataset, click __num_of_pages_i column to sort rows in a descending order with respect to the number of pages. The longest book turns out to be “The Complete Aubrey/Maturin Novels (5 Volumes)” by Patrick O’Brian consisting of 6,756 pages. To get the shortest book, click again on the same column. We find that there are many books with 0, 1, 2, … pages in the dataset. Most of these books could apparently either audio books, digital books or the number of pages was entered incorrectly.