Organizations are constantly struggling for faster access to information that can help them make better decisions. The dynamic nature of many businesses has made it imperative for them to acquire solutions that can help move the needle on decision making. This is where a need for faster and flexible reporting becomes a necessity and that can really help provide data driven insights into meaningful KPIs thus empowering businesses to make quick and efficient decisions.
Many enterprises use relational databases for running their core business as it provides good transactional capabilities. However, when it comes to using a transactional relational database for reporting, the same becomes a bottleneck. Operations involving bulk retrieval of data from a traditional Database can really hurt the performance. This is where a NOSQL database like Elasticsearch comes to the rescue.
We will use one of our customers in the Insurance industry that creates new plans for Insurance companies as an example to demonstrate the value of using Elasticsearch. These plans are created for Insurance companies and then an approval from respective State Government needs to be obtained before being released into the market. All the plans are stored in a No SQL database and data retrieval using the reporting tool was slow because of performance bottlenecks with the No SQL database. The plan data was migrated to Elastic Search in order to enable faster access to data.
We have used Elastic Search in order to implement a reporting solution for our customer. The data source is a No SQL database from which we extract the plan details. The data is then transformed and loaded to the index. An index is like a ‘database’ in a relational database. The reports are then generated from these indexes and submitted to the US government for approvals.
Elasticsearch is an open source, distributed, RESTful, full-text search engine built on top of Apache Lucene. Elasticsearch is developed in JAVA. It uses schema free JSON documents and comes with extensive REST APIs for storing and searching the data. The primary way of interacting with Elasticsearch is using the REST API over HTTP. We can use any of the popular HTTP clients, such as Curl or Postman.
Curl is a command line-based client available on most operating systems while Postman is an UI-based HTTP client available for major operating systems. The figure below shows how the Elasticsearch query works.
Elasticsearch has its own Query Domain Specific Language, where you specify the query in JSON format. The documents in the index are distributed across multiple shards, that are further distributed across multiple nodes. This ensures there is redundancy and it increases the query capacity. Elasticsearch will automatically rebalance the entire cluster by adding or removing the shards based on the need.
The following APIs (GET, POST, PUT, DELETE) were used to query, extract and load plan details into Elasticsearch.
This API returns the results that matches the query. We can either specify in the request body or pass it as a string. In the below screenshot we are extracting the plan details from Marklogic.
Posts the data into the Index. In the below example, we are inserting the details for one of the plan. (http://localhost:9200/index/type/id/_update)
DELETE API: This API will delete the specific plan id (removes the JSON document from the index) from the index before loading the details. We must specify the index name and ID and if we want to drop the entire index, we can just specify the index name (http://localhost:9200/index/type/id).
Other applications of Elasticsearch include Structured Data Search, JSON Documents Storage, Metrics and Analytics, Text Search.
Elasticsearch has helped our customers achieve efficient reporting in a short time because of its high performance. In addition to this, it is based on a distributed architecture that allows for storage of thousands of gigabytes of data, thus providing scalability to the customer. It is document oriented, so the customer does not need to maintain any schema or table to store the data. Elasticsearch will help you resolve many search optimization problems in your existing applications. It is useful for giving your users a quality search experience and also letting them find what they are really looking for.
For more information on how Apps Associates can help you, visit us at www.appsassociates.com.
Garima Kukreja is working as a Senior Consultant with Apps Associates Hyderabad, India. She is an ETL specialist with 8 years of experience in Analytics and worked on different tools for both ETL and Reporting like Informatica, Clover, Knowi, Cognos. She has led and managed projects for many reputed clients.