A major bank in Dongguan (China) refused a potential customer because his name is Li Jun. Apparently, there were already over 300 bank accounts assigned to the name Li Jun. Not that this particular Li Jun was responsible for opening all these accounts, there were just too many men with exactly the same name. The bank states that the refusal is nothing personal, since nobody with the name Li Jun will be accepted as customer in the near future….. In the meanttime, Li Jun is taking legal action against the bank.

When I read this news article this morning, my first thoughts were that it was perhaps a hoax. It turns out , however, that the news fact is true. From a data quality point of view this strikes me as really strange. How does this particular bank manage its customer data? Are there no additional identifiers (address, date of birth, etc.) to determine that you are actually dealing with the customer you think you are dealing with? Imagine that every John Smith would have a hard time to open a bank account, to apply for a job or to buy a product via the web. Or Jenny Jones? Bob Johnson? When is a name too “common”? It is common misbelief that the complexity of ideographic characacters such as Mandarin Chinese makes it harder to identify. At Human Inference we carried out some pretty serious dedups of Chinese files and-taking into account that Mandarin Chinese is a tonal language and other priciples of fault-tolearnce apply- the duplicate identification was rather accurate.

It is all a matter of using an intelligent data matching method and knowing what kind of data one is working on. Every name can be identified; even “common” names.

The term Golden Record is closely related to Customer Data Integration or MDM for Customer data. It refers to the “single truth” which has been created or calculated from all those duplicate customer records from different systems. This post is not about finding or tagging all those duplicate records. There all kinds of ways to find them using advanced statistical methods, fuzzy matching etc.

But what do you once you have found the duplicates. How do you create the best possible customer data out of all gathered elements?

First of all we have to define what is meant by the Golden Record. We at Human Inference use the acronym ACCU, short for Actual, Correct, Complete and Unique. Ofbviously, we want one unique record. That’s why we use matching or identity resolution software. But Actual, Correct and Complete are less absolute, they can be interpreted in a subjective manner. You can have never-ending discussions about it, build the most complex business-rules ever etc. But I prefer to start with simply determining the superlative of Actual, Correct and Complete. In other words the most actual, the most correct and the most complete data-element or attribute “wins” and makes it to the Golden Record. Let’s take the following example, two almost identical records are gathered from two different systems (A & B).

The basic rule is that only Correct data will make it into the Golden Record. So, if you can validate data please do so. For instance you can check social security, bank account and credit card numbers using algorithms. You can validate email addresses. Using postal reference data, it is also possible to verify the correctness of addresses. The most difficult is to validate names. Extensive knowledge is needed to check whether names of persons and organizations are valid.

In my own experience and opinion you should always discard incorrect data, or let it be corrected by a data steward. In the end nobody should be in doubt whether a Golden Record has been established using doubtful data.

The next step is to examine attribute (field) by attribute. So using the example from above.

So using validation techniques to distinguish the correct data from incorrect data and determining the length of each attribute in the provided records will result in the following Golden Record:

Mister J.F. (John) Miller

26 Spring Gardens

Manchester, Lancashire, M2 1BB

United Kingdom

Even if you have a lot more of attributes in your Golden Record, this method still works. Determine the correct data and use only correct data. And using the function Length (LEN) to determine the “most complete” data. Most complete simply refers to consisting of the most characters. If the source systems also provide dates for “date entered” and “date last changed” you can use this to determine what the most recent data is. The most recent data is determined by formulas like MIN (“CurrentDate” minus “”Last Changed Date”).

I believe this method will lead to a very usable Golden Record in 90 to 95% of all cases. Only when you have to deal with complicated data, for instance father and son living on the same address and having the same initials it becomes much more complex. I am curious which rules-of-thumb and methods you use when calculating the Golden Record. Please put your ideas in the comments.

One of my current projects has been to take an intelligent approach to the removal of duplicates already on an existing system (SAP).

The client has already successfully used our software in their IT environment to effectively stop all new duplicates being entered into SAP. They now want to use the same technology to remove all existing duplicates. Their idea is so simple I am amazed that I have not heard of it being done elsewhere before.

Every evening the whole clients SAP database will be searched for duplicates in their Companies and Contacts (> 3 million records deduplicated in less than an hour!) The results are stored in a master result table that SAP has been given access to. Now depending on the likelihood of the match, the duplicates can fall into one of three categories: automatic merging, manual merging or no merge. If the score for the whole duplicate group is above the threshold for automatic merging then the automatic merging process is started.

This merge process has been created by an external SAP consultancy group that does a lot of clever stuff in giving each record a score depending on its’ financial relevance. E.g. open payments, current order status, payment reminders etc. (Hey, it’s SAP and in the world according to SAP only financial dealings have a value!) In the end the one record with the highest score is set to be the lead duplicate. All information from the other records in the duplicate group is placed onto the leading record to create a unique (‘Golden’) record. All duplicate records with the exception of the lead duplicate are then removed from the system, in the case of SAP, these records are given a ‘set for deletion’ flag and subsequently archived.

The ‘Non merges’, i.e. where the match score is below the accepted threshold level, are discarded and all remaining records are sent to a separate SAP mask for manual inspection for the following day. All that is required is to identify if the records shown belong in a duplicate group or not. After this decision has been made each duplicate group goes to the ‘merging’ process. Just the same as the automatic merge process.

At the end of the day the whole process starts again. Wash, rinse, repeat! Simple! The first thing to happen is that over a short period of time all the secure duplicates disappear as they are merged automatically. This is highly visible, no more multiple identical records that pop up whenever a new record has been entered. The impact on the quality on the surrounding systems is just as direct. No sending out bills or marketing mails x times to the same person (having worked in Marketing before, I know the problem and it always leaves such a professional impression with the customer!) So it’s already something easy to sell to your managers and so far you have not had to lift a finger. Great!

The brilliance of the SAP data quality solution though lies elsewhere. The simple fact is that it really does not matter whether the rest of the results are worked through in 1 day, 1 month or a year – as they are always captured, every day anew. The net result is that the total level of duplicates is constantly decreasing. Where the merge process has taken place, the duplicates will disappear. Only a change on the record will force it to be rechecked in the next round of deduplication. This means that apart for the costs of enhancement of the current system the client has an effective DQ firewall that now not only protects them from duplicate data being entered onto their IT systems, but will now over time cleanse the system from within. Even if it means putting an employee to sporadically make a decision on the manual matches. It is something that the company/department can concentrate on where they have time/resources available. (That should be easy after showing what success you have had with it already!)

How about if it the process could be easily and readily monitored? Say by using Excel or a similar product. Bar graphs and pie charts always tell way more than actual figures! Then the impact on what is happening is all the more visible and easy to sell (a good budget retainer!)

Good luck in dealing with your duplicates.

I came from a large logistics company, where DQ was preached heavily and seen as a way of reducing costs. The further though we went into what DQ could actually mean – the more vague and indirect the costs and effects seemed to be. The one thing we knew we really suffered from it was that we had a whole lot of duplicates in the system. This was always visible and the effects from it very tangible. They effectively helped screw up a perfectly good CRM tool. The solution was simple. Buy a deduplication tool and identify the duplicates!

Now there’s a good few deduplication tools out on the market. All of them will tell you how good they are at finding duplicates using mathematical-, probabilistic- and fuzzy matching. The list can go on and on. Where all providers though seem to stop short, is what they do with the duplicates found. Now all vendors have ways of identifying the duplicate pairs or even duplicate groups and most vendors will offer clever and fancy ways of bringing the duplicates together, but in almost all of these cases this happens OUTSIDE the clients’ current IT systems. Of course bulk loading old/new data back into the IT system is always so simple! So much fun! IT/ICS departments are always so understanding and helpful!!

This is what we had to find out for ourselves in a painful fashion. We quickly came to realise that finding the duplicates is barely even the tip of the iceberg. Actually trying to group the information together in such a way as to create a unique (golden) record was going to cost the company a lot of money. It would need an involvement of a Systems Integrator – because the problems are never just related to only one system, right? Multiple man years spent in project time (God! Consultants just prey for those types of projects.) Naturally, project costs rapidly rising into million of euros.

Where’s the saving then? I mean, the company has lived with the same problem for years right? So it can’t be that bad, can it?!? (Ever wondered exactly how the banks got into the current credit crisis?)

Unfortunately for the logistics company – the project never really got off the ground. Although the benefits of non duplicates were visible: reduced overhead – a more streamlined sales force, increased effectiveness, the risks about project timelines and overall cost killed it.

All is not lost. Good DQ providers will also offer software that effectively stops duplicates being allowed on to the clients system. Of course the safest way of ensuring a high level of DQ is to capture all the mistakes by their entry. Sounds wonderful, but like the Irish saying goes “If I was you – I wouldn’t start from here.” The DQ providers are only asked for a solution because the need has already been identified by the prospective client. The client has no other choice but to start from where they are. Just stopping the problem coming in through the front doors does not make the waste that’s already on the system magically disappear.

So data quality tools will of course offer solutions – but far too often they won’t go far enough! DQ can never really be about just installing a CD to solve the problem. Good expertise and guidance is necessary. It will always require a deeper understanding of the origin of the problem, a sharp focus on the companies’ pain points and a tight integration into the IT landscape. Suddenly a simple CD solution doesn’t fit anywhere; it quickly becomes an exponential cost, sucking up time and resources simply to make it work. At a time where companies are looking to become more efficient, cut costs and unfortunately overhead too, is that really the best way?