Developing an AI-based data management solution to remove redundancy from data points

Daffodil Software, on analyzing the requirement, proposed the idea of building a self-learning, AI-based record linkage solution. A technical proposal was shared by the team at Daffodil that illustrated how the idea can be executed using the BERT model. The power and advantages of developing an AI-based solution were exhibited through a result comparison with a NodeJS application. 

Team Daffodil developed two different BERT models for merging organizations and meeting records with similar entities. Bidirectional Encoder Representations from Transformers (BERT) is a transformer-based machine learning technique for natural language processing (NLP) pre-training.

Building the BERT Model for Organizations

Daffodil started by analyzing the possibilities in which the organization data may exist. The organization names had text, numbers, and Unicode characters. Also, the database had entries made in short form. To remove unwanted entries from the database, text analysis, and data cleaning was performed on it. The data was then fed to different BERT models to analyze which BERT model gave the accurate output.

Manually, this task would have taken 60+ days to complete. With the NodeJS solution, the process would take 2-3 hours but with no idea of score accuracy (as the solution won’t grow with data and time). With the AI solution, identifying and merging similar entities took 4-6 hours with 99% accuracy. The best part about the BERT model was its accuracy would remain consistent, irrespective of the database’s growth.

Building the BERT Model for Meetings

For every meeting, the date and time would vary and these were the only fields that were available for all the records. The meeting database was fed to a variety of models – Convolutional Neural Network (CNN), Sequential Model, Random Forest, and Decision Trees. These models were tested with 30,000 records to figure out which one offers the accurate output. With the NodeJS solution, this task would have taken 10 hours to complete and with the AI technology, the same task took over 24 hours with an accuracy of 98.8%.

The record linkage solution built using Artificial Intelligence has proven to be time-efficient, has an incomparable accuracy level, and is self-learning. This ascertains that even if the size of the database increases in the coming days, the functionality of the solution won’t be affected, which wouldn’t have been the case with a static, rule-based algorithm built using NodeJS.