HIquality bug cycle

Usually our customers are satisfied with our product suite. At customer support I never receive information about the successful implementations. I got to know our software through the problems that occur, and in almost 15 years of acceptance testing and customer support, I’ve seen all kind of bugs passing by.
Software crashes and never ending loops are nasty. Worse are those bugs that are not that visible in the beginning, but keep on growing in the course of time.
Recently we caught such a bug in our longest existing product HIquality Identify.

HIquality Identify is often used in search applications. Just like police and justice use “descriptions” of a criminal, HIquality Identify uses descriptions of source data to detect the right records in a database. Source records are decomposed to core words and the phonological codes of the core words of streets, names and places are stored in the description table. This table is the base of the search application.

Nulls, nels and nols
Whenever a source record is changed, the descriptions have to be updated as well. A synchronize procedure is used to keep the description table up to date.
Due to a little mistake in this procedure we recently released a version of the Oracle Upgrade pack, that didn’t recognize null values in the database any more. Empty fields in the database resulted in core words with the value ‘null’, and the phonological codes ‘nel’ and ‘nol’.
As a result the scores of evaluations became less accurate, and end scores became too high. The phonological codes of the core words are used as indexes. These indexes are used to pre-calculate the maximum number of evaluations. Since more and more of these fields are changed to nul nel and nol, after several months, instead of search results time-outs occur, stating that not enough relevant search data was entered. 

Did you want to know this?
In a short time we figured out which customers received this special software release, and our consultants visited them for an upgrade. In the end all of them upgraded without knowing the actual reason or what kind of harm potentially could have been caused.
As a customer you want problems to be repaired and bugs to be fixed, without knowing every single detail. Is this nol-worm kind of virus something you would have wanted to know about? Probably not. In case you will have nightmares about nulls, nels and nols in your database, you can contact me any time at Human Inference’s Customer Support, where I cope with all those kind of things that can go wrong in software.

A major bank in Dongguan (China) refused a potential customer because his name is Li Jun. Apparently, there were already over 300 bank accounts assigned to the name Li Jun. Not that this particular Li Jun was responsible for opening all these accounts, there were just too many men with exactly the same name. The bank states that the refusal is nothing personal, since nobody with the name Li Jun will be accepted as customer in the near future….. In the meanttime, Li Jun is taking legal action against the bank.

When I read this news article this morning, my first thoughts were that it was perhaps a hoax. It turns out , however, that the news fact is true. From a data quality point of view this strikes me as really strange. How does this particular bank manage its customer data? Are there no additional identifiers (address, date of birth, etc.) to determine that you are actually dealing with the customer you think you are dealing with? Imagine that every John Smith would have a hard time to open a bank account, to apply for a job or to buy a product via the web. Or Jenny Jones? Bob Johnson? When is a name too “common”? It is common misbelief that the complexity of ideographic characacters such as Mandarin Chinese makes it harder to identify. At Human Inference we carried out some pretty serious dedups of Chinese files and-taking into account that Mandarin Chinese is a tonal language and other priciples of fault-tolearnce apply- the duplicate identification was rather accurate.

It is all a matter of using an intelligent matching method and knowing what kind of data one is working on. Every name can be identified; even “common” names.

The added value of an integrated customer view depends strongly on the quality of that integrated customer view. Every organization that is seriously planning to create a single customer view should ask itself the following question: “What determines the quality of my customer view and so the accompanying level of added value?”

Prior to answering this question we need to take one step back. Why does not every organization have a single view of the customer? The cause lies in the fact that many organizations have their customer data spread across multiple systems all facilitating separate business processes. Additionally customer data is often highly polluted, fragmented and incomplete.

So it appears that the data itself plays a crucial role in the lack of an integrated customer view. Or more accurately, the better the data – the better the customer view. And the better the matching of customer records across separate systems the better the integrated customer view.

So Data Quality and Matching (Identity Resolution) determine in large parts the quality of the integrated customer view and the added value that it delivers. Take a look at this demo showing a step-by-step approach how to build a single customer view and get a better idea of the role of Data Quality and Matching within this process.