Exploring Kibana 4 with real world data

With Kibana 4 generally available, we took it for a test drive with real world data from Alameda County, California. The Alameda County Data Sharing Initiative provides raw datasets in ten different categories such as education, public safety, health, transportation, business etc. The data is available through API (SODA, ODATA) as well as various downloadable formats such as JSON, CSV, XML etc. We used the API since we intend to continuously update our dashboard with the latest data and analyze the data in greater detail (just for our own interest).

What did we analyze with Kibana 4?

There is tons of interesting datasets in the Alameda County website, but for this Kibana 4 test drive, we analyzed the public safety data produced by Alameda County Sheriff Department with Kibana 4. The dataset consists of Sheriff Reports from 2011 till date. Kibana with Elasticsearch would be an ideal tool to analyze the crime data and description. Needless to say, all kinds of crime information ranging from DUI to Violent Crimes is available in this rich data set. We focused our analysis on three broad types of crimes:

  • Violent Crimes
  • Burglary (other than auto)
  • Auto burglary

Our objective was to quickly create a dashboard with heat map of these crimes in Alameda County, visualize the trend and identify cities/zip codes with most incidents of above crime types using data from last 3 years. Below are the Kibana 4 dashboards for Violent Crimes, Burglary, and Auto Burglary data from Alameda County.

Kibana-4-Alameda-County-Violent-Crimes

Alameda County Violent Crimes

Kibana-4-Alameda-County-Auto-Burglary

Alameda County Auto Burglary

Kibana-4-Alameda-County-Burglary

Alameda County Burglary

 

 

 

 

 

 

 

 

What we observed from the data

They say that a picture is worth a thousand words. The dashboards helped with the following observations:

  • The area close 880 and 238 intersection seems to be a hotspot for crime
  • San Leandro tops the list of cities with most violent crimes, Castro Valley tops in both burglary and auto burglary.
  • Zip code 94541 tops the list for violent crimes and burglary, while 94546 tops in the auto burglary.
  • Overall crime trend across violent crimes, burglary, and auto burglary is down, with a peak in late 2012. Data is missing in Oct and November 2014.
  • Domestic abuse (injury on spouse/co-inhabitant) is the top violent crime.

Our take on Kibana 4

In our opinion, Kibana 4 is an order of magnitude improvement over Kibana 3. Kudos to the Kibana team for a great effort. One of the main challenge with the Kibana 3 was the absence of a clear workflow and repository approach to building dashboards. Kibana 4 addresses this issue by separating the workflow into Discovery, Visualization, and Dashboard. You can create multiple searches and save them for reuse. You can create a repository of visualizations from scratch or using saved searches. Finally, you can create multiple dashboards using the combination of searches and visualizations.

Discover_Page Kibana-4-Visualize-PageKibana-4-Dashboard-Page Kibana-4-Settings-Page

There are still some areas where Kibana can be further improved.

  • Creating new visualizations and dashboards are simple in Kibana 4, but managing them is a bit confusing. Kibana 4 often creates duplicate visualizations and dashboards which you have to delete using the settings tab.
  • Kibana 4 is a great tool for developers and savvy data analysts who know what they are looking for and can write the Lucene syntax to search through the data. Business users who expect Tableau like simplicity might find Kibana 4 to be a bit difficult to use.
  • Ease of data discovery, reporting and scheduling capability, role based access are some other improvements which would make Kibana ready for the masses.

Kibana 4 with Alameda County data sandbox is available

We have an online Kibana 4 sandbox with the Alameda County data. We are working to make it available with some additional features that we have added to the kibana framework. If you would like to play with Kibana 4 using the Alameda County data, please contact us at info at guidanz dot com. We can offer you an AWS image with data or even our sandbox environment.

Share your thoughts/questions

What are your thoughts about Kibana 4? We look forward to your comments/questions. Shankar Radhakrishnan from Guidanz will be at the Elasticon ’15. If you are attending the conference, contact him to learn more about our experience with Kibana 4 and Elasticsearch.

Leave Comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

*

clear formSubmit