HIquality bug cycle

Usually our customers are satisfied with our product suite. At customer support I never receive information about the successful implementations. I got to know our software through the problems that occur, and in almost 15 years of acceptance testing and customer support, I’ve seen all kind of bugs passing by.
Software crashes and never ending loops are nasty. Worse are those bugs that are not that visible in the beginning, but keep on growing in the course of time.
Recently we caught such a bug in our longest existing product HIquality Identify.

HIquality Identify is often used in search applications. Just like police and justice use “descriptions” of a criminal, HIquality Identify uses descriptions of source data to detect the right records in a database. Source records are decomposed to core words and the phonological codes of the core words of streets, names and places are stored in the description table. This table is the base of the search application.

Nulls, nels and nols
Whenever a source record is changed, the descriptions have to be updated as well. A synchronize procedure is used to keep the description table up to date.
Due to a little mistake in this procedure we recently released a version of the Oracle Upgrade pack, that didn’t recognize null values in the database any more. Empty fields in the database resulted in core words with the value ‘null’, and the phonological codes ‘nel’ and ‘nol’.
As a result the scores of evaluations became less accurate, and end scores became too high. The phonological codes of the core words are used as indexes. These indexes are used to pre-calculate the maximum number of evaluations. Since more and more of these fields are changed to nul nel and nol, after several months, instead of search results time-outs occur, stating that not enough relevant search data was entered. 

Did you want to know this?
In a short time we figured out which customers received this special software release, and our consultants visited them for an upgrade. In the end all of them upgraded without knowing the actual reason or what kind of harm potentially could have been caused.
As a customer you want problems to be repaired and bugs to be fixed, without knowing every single detail. Is this nol-worm kind of virus something you would have wanted to know about? Probably not. In case you will have nightmares about nulls, nels and nols in your database, you can contact me any time at Human Inference’s Customer Support, where I cope with all those kind of things that can go wrong in software.

The proactive versus reactive debate rages, but in fact data quality must be both a proactive and a reactive process. The data will dictate which to use, or whether both are required.

The first time right proactive approach applies to all data. Some data, such as transactional data, must be collected correctly because corrections after the event are time-consuming and very costly. Note the wrong information when a product purchase is telephoned in, for example, and your customer, the call centre staff, the warehouse staff and so on will all need to be included when trying to correct the error after the wrong products are delivered, with the accompanying damage to your reputation and your brand image.

When transactional data is collected correctly first time, no further cleansing will ever be required – the record represents a historic event that will never change.

Other data may be cleansed later, but reactive cleansing never achieves the high level of data accuracy that right first time practices achieve. Postal addresses are a good example. Many addresses can be found and validated against postal data files after collection. But without the ability to hold a dialogue with the customer, reactive processing will only correct a percentage of the issues.

The situation with personal names is far more serious. People give their children’s names unusual spellings, casings and other twists to mark them out from the crowd. Without being collected correctly first time, no automated post-processing can correct these errors. In fact, many attempts to correct the errors will result in further inaccuracies being introduced. Organisations also often reactively gender code their customers based on given names. People with the same names can have different genders both within and between cultures, and as in our mobile world people move around, you can no longer assume that the American Joan is a female and not a male originating from Spain, for example. This sort of post-processing significantly reduces data accuracy.

So if first time right is always the best policy for data quality, where does reactive data processing come in? When the data refers to an entity, you need to remember that the entity’s situation may change over time. If it is a person they may marry, change their name, move house, change job etc. If it is a city with a postal code the city name may change, or the postal code, or the street name may change. Somebody living in Sudan on 8th July 2011 may find themselves living in South Sudan on 9th July. Somebody buying with Estonian Krooni on 31st December 2010 will be buying in Euros on 1st January 2011. It is remarkable how we attempt to track the changes in the lifecycles of our customers but tend to overlook the dynamic changes in their environments which affect them and you – they happen very much more often than we think.

So the lesson must be that first time right is always right and should never be replaced with reactive processing. Reactive processing is required when real world change affects our data.

I was reminded recently that this rule applies also to company names when reviewing a directory containing Croatian companies.  The directoryshowed a great variation in words that at first glance would seem ideal candidates for correction and standardization. For example, many companies contained strings like these:

Commerc, Comerce, Comerc, Kommerce, Kommerc, Komerce, Komerc

There are many other examples which had me scratching my head: Compani, Konsulting, Konzalting, Konsalting and so on.

Why the variance is spelling?  Are these companies with the English word commerce in their names where that word has been typed as heard by call centre workers with a limited knowledge of English? Are they typos of a valid Croatian word? Are they accurate representations of a valid Croatian word as rendered in different dialects? Is it a mixture of all these factors? 

I am assured that commerce does not have a similar Croatian equivalent. The best translation of commerce would be trgovina or obrt. Typos and mis-rendering aside, it would appear that in a significant number of cases these strings as written are actually part of an accurate company name – they are attempts to anglicise and internationalise the company name, either mis-spelling the English word or rendering it to sound like ‘commerce’ in the local Croatian dialect. 

If one needed to reactively cleanse company data that has not been collected correctly, one could choose to translate these words when found, or to standardise them to ‘commerce’, but either process would reduce the accuracy of the data, because, wrong or not, these words are included as parts of company names. They show the individuality of the company, and attempts to ‘correct’ data is as bad as correcting a personal name like SuZann because it is ‘written wrongly’.

 There are parts of company names that can be processed, most often the legal form of the company – PLC, Ltd, Inc. or, to continue the Croatian theme, d.o.o., but the rest of the company string needs to be left alone. 

Like personal names, company names are carefully chosen to show individuality, to set that company apart from its competitors. The names vary greatly in form and spelling. Unless one has an intimate knowledge of a company, viewing a name after data collection will not show where any errors in it exist, and no post processing will allow those errors to be corrected – in fact, post processing often reduces the accuracy of company names.  As with personal names, the rule for company names is to collect then correctly – right first time.

For over 25 years Human Inference has collected reference data, for instance on persons. Because of our reference set we immediately recognize that ‘Ask Me’ and ‘Any Body’ are fake names. People are using these either in test situations or to hide their actual names.

In the old days we only needed to test on ‘Test Test’, in more recent years we see great inventiveness on these fake names. A brief example can be seen in the following list.

In case you cannot rely on reference data and interpretation you need to provide a check list. Providing it is one thing, but since users tend to be really creative, maintaining it is essential.

In these 25 years we identified a move from ‘real fake names’ towards ‘real names used in a fake way’. In the USA, for example, we identified popular Hollywood names and names of politicians being used as fake names. Currently the usage of the name ‘George Bush’ is decreasing, whereas ‘Barack Obama’ is increasingly used. We recognize the false usage of these names because of the change in frequency figures of the given name and family name as well as the usage of the combination itself. Remarkable is that ‘Abraham Lincoln’ and ‘George Washington’ are quite steady.

Back to ‘Walter von Stolzing’. By now you might have guessed what is happening here. We recognized that in German speaking areas this name is also passing our threshold on validity. By googling the name you can see that Walter is actually a character in Wagner’s opera ‘Die Meistersinger von Nürnberg’ back from 1868!

Let’s see if in 100 years time people are still using ‘Darth Vader’, ‘Lord Rings’ or ‘Snoop Dogg’!

All the names used in this blog are ‘real’ names coming from a popular social media site. Please check our cleansing products in case you need cleansing solutions.

This week, the latest results have been presented in The Netherlands on Transparency in the Banking area. And although some institutions score really good, others really need to take it at least one mile further to get a good or even fair score.

We agree with the recommendations of the report that compliance regulations can help/force in being more transparent, e.g., the SEC in the USA is enforcing more detailed information than their Dutch peer, the AFM. And also for Basel II the financial institutions need to know who they are dealing with in the end. The phrase - in the end – makes it even more difficult for the CSR, because not only the ultimate legal entity is now needed, but additional details per region and per sector are required.

In our daily practice in implementing Customer Data Integration (CDI or MDM for Customer Data) projects, we face these challenges at our customers. They are absolutely willing to provide the right figures, however it’s far from a trivial task. There are many underlying systems that were never created to aggregate this kind of information in an easy way sufficient for reporting the CSR. There is a huge demand on bridging the gap between these systems in an non-intrusive way. To combine individual records in and across systems in so-called Golden Records, so on these can be used both for compliance and transparency on your social responsibility.

After the retreat of the French, the Dutch authorities wisely kept the system of family names. They too wanted their tax system to function properly (they were Dutch, after all). The registration of family names gradually became common practise and with the growing population in the Netherlands, the municipal registers kept growing as well. During the last decades the scattered registers of all the municipalities were investigated, linked and combined, and put in large computer systems.

And now it became apparent that in the old days, data quality might not have been a top priority after all: For example, in the Netherlands there are more than thirty spelling variations of the name Mathijsen.

Which leaves the true data quality fans wondering whether a family name is actually the right family name…..

For more information on family names, please check the Human Inference website.

More challenges appear if the typo has caused another existing word; now we need to make a decision on how equal the two entries are. In case you have some knowledge on the frequency of usage of words you can use that in the equation. How to get the frequency of usage for words is another ballgame – at least you can assume that a ‘wrong word’ is never used (bit of a paradox).

A large group of possible matches that are not found (i.e. missed matches) by fuzzy search methods are the ones that sound the same but are written rather differently. Often a callcenter agent types the name exactly like he hears it. An example would be the family name ‘Farren’ and ‘Pharan’. They have already so many differences that it becomes rather hard for a string comparison to treat both entries as equal. Phonetic search would definitely help here. Drawback on only phonetics is that you can now combine entries that are for sure no matches (i.e. mismatches), e.g.:

1. René Meierhofer and
2. Renée Mayrhofer

Two valid family names, but the given names show both a male and a female entry.

In a real life example, we would expect a complete name with titles and we’d still need to match in a correct way. Take, for example,

1. Dr. John J. Farren jr.
2. John J. Pharan jr. PhD

Pure string comparisons based searches won’t work in this case. The complete entry could be matched in combination with some smart academic synonyms and some n-gram or matrix comparison on the individual elements.

Introducing synonyms immediately generates new types of challenges. In address matching you will go a long way when you take into account the abbreviations for street types (Avenue for Av., Street for Str. etc). For company names it definitely helps to have a synonym table on legal forms (Limited for Ltd, Incorporated for Inc., etc). With the actual company name itself it becomes more challenging. A German example might look like:

1. Fahrrad-Handel Anna Cintula and
2. Zweirad-Shop Anna Cintula,

Two synonyms for bike shop. Quite often people think in such situations that by adding a synonym table the challenge is gone. They are absolutely right for part of the problem but still there is a large set of words that get their specific meaning based on the context of that word – and by that they refer to a particular synonym. If we take for example the following three entries, it seems evident that we cannot replace the word ‘art’ with one single synonym here

1. Art Gallery Garfunkel
2. ART Auto Rendition Technology
3. Paul Simon & Art Garfunkel

String comparison is fine as a start in matching problems. To really avoid a serious amount of mismatches or missed matches – preventing a serious amount of manual work – you need to know what you’re dealing with. You need to compare apples with apples, oranges with oranges. What would really help here, is a bit of natural language processing ;-)

Your parents probably gave a lot of thought to the name they once gave you, and as it turns out they were right to do so! Research tells us a name can do wonders for its owner, as well as a lot of damage for that matter. Let’s have a look at some remarkable results.

Peter for President!
Recent studies show that in the US a student called Fred is more likely to fail his exam than a student who just happened to be named Andrew: people tend to indentify with their name and, in general, have a positive feeling about letters that correspond with their initials. Consequently Fred is far more likely to settle for a meager F, while Andrew will have an extra motive to strive for an A. It also explains how in choosing a partner we show a slight preference for someone whose name resembles our own, or why Mary will prefer to live in Maryland, while Monica is more inclined to settle in Santa Monica. Most of these preferences only show themselves through our subliminal selves, so we are not actually aware of the motivation for some of our choises. Another US study endorses these findings: inspired by the results mentioned above, researchers decided they’d investigate on another letter. They came up with the letter K, which in baseball stands for strikeout. The study showed once again that there is a connection between a letter and its causer: batters whose names began with a K struck out more often than other batters.

Ominous names
A UK research tells us that as much as one in 5 parents regret how they named their child. The novelty might have worn off after a few years, but can there be any real objections to a certain name? Apparently, there are plenty! Ironically it’s not the parents who’ll have to carry this burden for the rest of their lives…

“Hi, I’m Antwan, but you can call me Antoine…”
It seems that even children’s language skills are influenced by their name. This has to do with the effect negative emotions can have on a child’s performance. If for example you decided to name your son ‘Gene’ but spell it ‘Jene’, he is very likely to get confronted with disbelief from his teachers. “Are you sure your name isn’t spelled with a ‘G’?” This can severely undermine Jene’s sense of confidence. That explains why children with an unusual name or a name that is unusually spelled generally are less adequate spellers and readers.

“But Sissi is a Royal name, dear!”
When a girl is called Frankie we think it’s a fun name, a cool and robust statement to fit a strong personality. Yet when a boy is called Mckenzie, (yes, some parents think it’s cute to give their boy a name that has a feminine touch to it ) we see a similar effect, but with a different outcome. This is something his parents obviously had not foreseen: their son will constantly be shaking off his girly image. The effect is striking: boys with a androgynous name misbehave more often than their unambiguously named peers, especially when they reach puberty. A boy called Mckenzie or Aubrey is even more likely to display bad behaviour when there is a girl with the same name among his peers. One more reason for parents to stick to conventions when choosing a name for their newborn.

Want to produce the new Einstein? Call her Kate!
A name can be a burden, but if you use this knowledge wisely, you might just turn it into an advantage. What happens to a girl when she has finished school and needs to choose what subject to study? Well, according to a US study, her choice depends on her name. As it turns out girls with a very feminine name like Julietta or Isabella are more likely to study humanities, while those whose name is less obviously feminine are more partial towards science. The question is: who’s aspiring to whom? Could it be that parents would treat Kate in a different way than Barbara? Or did the parents subconciously decide they wanted to raise a scientist when they decided to call their daughter Kate?

Would you rather hire Vanity or Grace?
Of course it’s not just letters or gender that determines how we feel about a name. In fact, how other people perceive us very much depends on the meaning of our name. For example: when looking for a new member on your marketing team, would you rather hire Vanity or Grace? In spite of what her name tells us, Grace might be a job jumper who doesn’t know how to work in unison with her colleagues. Vanity on the other hand could just be a daughter of a well-read mother who had just finished her latest Thackeray when she gave birth. Still, both women will either meet a lot of prejudice or feel the need to live up to a very high standard because of their name.

It all goes to show that a name defenitely posesses some self-fulfilling qualities. Given the fact that so many parents regret their choice of names afterwards makes me think that the owners of that name might share these sentiments. So what does that mean when looking at it from a data quality point of view? Unisex names for example are responsible for a lot of data quality issues. As the borders between male and female names are fading we’ll need to update our knowledge continually. The human in Human Inference will definitely take care of that. After all, we wouldn’t want to you to put off Mrs Clinton when sending her a petition to take pity on the Syrian citizens starting: “Dear Mr. Clinton…”.

Source: Livescience.com & Babynames.com

 Throughout the years Human Inference has carried out and supported research with regard to the importance, the impact and the perception of customer data quality in business environments. This reasearch shows that the phenomenon customer data quality is subject to a perception shift. In general, one could argue that, in the early years, data quality used to be perceived as “something that is being carried out by the IT-department”, whereas nowadays more and more companies and organizations are recognizing the importance of customer data and information quality. Issues and initiatives such as the value of a single customer view, data integration, fraud prevention, customer relationship management, operational risk management, compliance and anti-terrorism have become boardroom themes.

As a result, high quality customer data has become the prerequisite for successful business decisions. In order to reach the intended data quality level, a lot of money is being invested in solutions for input control, file merging, data enrichment and duplicate identification. But do these investments guarantee more effective marketing campaigns, accurate lead generation, a higher conversion rate and, eventually, better business results?

In my opinion, sound data quality management can be compared with good cooking: It is all about being in control. In cooking, putting the right ingredients together does not necessarily account for an excellent meal. You need to know how and when (timing!) the ingredients “interact” in order to achieve the taste and presentation you are aiming for.

In marketing campaigns, being in control is essential. The “ingredients” of your marketing campaign have to interact. Selection, content, targeting timing, conversion, etc. need to be effectively combined and balanced. Marketing campaigns are more or less the tangible outcome of your investments in customer data improvement. Reaching the people you actually want to reach, not spending unneccesary amounts of money on manual rework and pleasing your customer or prospect with an accurate offer. To see Human Inference’s take on this subject, check out the movie on the HIquality data Improver on our website.