HIquality bug cycle

Usually our customers are satisfied with our product suite. At customer support I never receive information about the successful implementations. I got to know our software through the problems that occur, and in almost 15 years of acceptance testing and customer support, I’ve seen all kind of bugs passing by.
Software crashes and never ending loops are nasty. Worse are those bugs that are not that visible in the beginning, but keep on growing in the course of time.
Recently we caught such a bug in our longest existing product HIquality Identify.

HIquality Identify is often used in search applications. Just like police and justice use “descriptions” of a criminal, HIquality Identify uses descriptions of source data to detect the right records in a database. Source records are decomposed to core words and the phonological codes of the core words of streets, names and places are stored in the description table. This table is the base of the search application.

Nulls, nels and nols
Whenever a source record is changed, the descriptions have to be updated as well. A synchronize procedure is used to keep the description table up to date.
Due to a little mistake in this procedure we recently released a version of the Oracle Upgrade pack, that didn’t recognize null values in the database any more. Empty fields in the database resulted in core words with the value ‘null’, and the phonological codes ‘nel’ and ‘nol’.
As a result the scores of evaluations became less accurate, and end scores became too high. The phonological codes of the core words are used as indexes. These indexes are used to pre-calculate the maximum number of evaluations. Since more and more of these fields are changed to nul nel and nol, after several months, instead of search results time-outs occur, stating that not enough relevant search data was entered. 

Did you want to know this?
In a short time we figured out which customers received this special software release, and our consultants visited them for an upgrade. In the end all of them upgraded without knowing the actual reason or what kind of harm potentially could have been caused.
As a customer you want problems to be repaired and bugs to be fixed, without knowing every single detail. Is this nol-worm kind of virus something you would have wanted to know about? Probably not. In case you will have nightmares about nulls, nels and nols in your database, you can contact me any time at Human Inference’s Customer Support, where I cope with all those kind of things that can go wrong in software.

This week, the latest results have been presented in The Netherlands on Transparency in the Banking area. And although some institutions score really good, others really need to take it at least one mile further to get a good or even fair score.

We agree with the recommendations of the report that compliance regulations can help/force in being more transparent, e.g., the SEC in the USA is enforcing more detailed information than their Dutch peer, the AFM. And also for Basel II the financial institutions need to know who they are dealing with in the end. The phrase - in the end – makes it even more difficult for the CSR, because not only the ultimate legal entity is now needed, but additional details per region and per sector are required.

In our daily practice in implementing Customer Data Integration (CDI or MDM for Customer Data) projects, we face these challenges at our customers. They are absolutely willing to provide the right figures, however it’s far from a trivial task. There are many underlying systems that were never created to aggregate this kind of information in an easy way sufficient for reporting the CSR. There is a huge demand on bridging the gap between these systems in an non-intrusive way. To combine individual records in and across systems in so-called Golden Records, so on these can be used both for compliance and transparency on your social responsibility.

]]> http://datavaluetalk.com/data-governance/know-your-customers-improving-your-corporate-social-responsibility/feed/ 0 http://datavaluetalk.com/mdm/the-miracle-of-customer-data-integration/?utm_source=rss&utm_medium=rss&utm_campaign=the-miracle-of-customer-data-integration http://datavaluetalk.com/mdm/the-miracle-of-customer-data-integration/#comments Mon, 24 Aug 2009 13:43:37 +0000 Holger Wandt http://datavaluetalk.com/?p=1193

The more a company knows about its customer’s wishes, needs and habits and the more that company is able to tailor its proposition accordingly, the greater the value it will eventually provide for its customers. We all know that there are countless examples where defective, fragmented, or just plain poor customer data cause unnecessary costs, decrease in revenue, employee dissatisfaction or frustation, damage of the corporate image and many other unsdesirable or painful consequences.

Customer data quality and integration problems impact every area of the value chain of organisations. Far too often companies have a multiple view of their customers. Customer Data Integration (or MDM for Customer Data) is the key to providing companies with a single view of their customer. According to Gartner, Customer Data Integration (CDI) is a combination of technology, services and processes to deliver an accurate, timely and complete view of the customer across multiple channels, lines of business, departments and divisions drawing customer data from multiple sources and systems.

I think that the real “miracle” of CDI lies in the automated, intelligent matching of customer records. Mind you, I’m not questioning the importance of the various CDI-processes (for example, I think that the post of my colleague Ramon de Noronha on the creation of “golden” records is majorly important), I’m just saying that -whenever the integration of customer data is an issue- intelligent, automated matching is the key prerequisite for success.

The quality of your customer data integration solution is only as powerful as the quality of your matching engine. If this statement intrigues you, I strongly advise you to read the white paper “High Precision Matching at the heart of Customer Data Integration“. Enjoy!

The term Golden Record is closely related to Customer Data Integration or MDM for Customer data. It refers to the “single truth” which has been created or calculated from all those duplicate customer records from different systems. This post is not about finding or tagging all those duplicate records. There all kinds of ways to find them using advanced statistical methods, fuzzy matching etc.

But what do you once you have found the duplicates. How do you create the best possible customer data out of all gathered elements?

First of all we have to define what is meant by the Golden Record. We at Human Inference use the acronym ACCU, short for Actual, Correct, Complete and Unique. Ofbviously, we want one unique record. That’s why we use matching or identity resolution software. But Actual, Correct and Complete are less absolute, they can be interpreted in a subjective manner. You can have never-ending discussions about it, build the most complex business-rules ever etc. But I prefer to start with simply determining the superlative of Actual, Correct and Complete. In other words the most actual, the most correct and the most complete data-element or attribute “wins” and makes it to the Golden Record. Let’s take the following example, two almost identical records are gathered from two different systems (A & B).

The basic rule is that only Correct data will make it into the Golden Record. So, if you can validate data please do so. For instance you can check social security, bank account and credit card numbers using algorithms. You can validate email addresses. Using postal reference data, it is also possible to verify the correctness of addresses. The most difficult is to validate names. Extensive knowledge is needed to check whether names of persons and organizations are valid.

In my own experience and opinion you should always discard incorrect data, or let it be corrected by a data steward. In the end nobody should be in doubt whether a Golden Record has been established using doubtful data.

The next step is to examine attribute (field) by attribute. So using the example from above.

So using validation techniques to distinguish the correct data from incorrect data and determining the length of each attribute in the provided records will result in the following Golden Record:

Mister J.F. (John) Miller

26 Spring Gardens

Manchester, Lancashire, M2 1BB

United Kingdom

Even if you have a lot more of attributes in your Golden Record, this method still works. Determine the correct data and use only correct data. And using the function Length (LEN) to determine the “most complete” data. Most complete simply refers to consisting of the most characters. If the source systems also provide dates for “date entered” and “date last changed” you can use this to determine what the most recent data is. The most recent data is determined by formulas like MIN (“CurrentDate” minus “”Last Changed Date”).

I believe this method will lead to a very usable Golden Record in 90 to 95% of all cases. Only when you have to deal with complicated data, for instance father and son living on the same address and having the same initials it becomes much more complex. I am curious which rules-of-thumb and methods you use when calculating the Golden Record. Please put your ideas in the comments.

“Cross-enterprise MDM adoption will remain extremely rare” says Rob Karel. While Aaron Zornes expects “multi-style MDM to increase”, he also expects “multi-hub connectivity from best-of-breeds”. To put it into other words, as another analyst does “most firms will deploy two or more MDM technologies for quite a time”. Definitely “users will struggle to make MDM work unless they address some really hard challenges”. One of these hard challenges is in my own opinion how to create an MDM ecology or environment in which multiple business entities can be maintained in an efficient and proper manner. E.g. how do you cope with MDM for customer data (CDI) and MDM for product data (PIM), this requires deep knowledge of the specific business requirements and needs. It will certainly need well-educated Data Stewards. As predicted by the analysts: “data governance frameworks get attention” and “data quality metrics redefined”.

One of the analysts expects that “MDM costs go up as skills decline”, but in the current situation with scarce financial resources we will see hopefully more “pragmatic improvements to data quality in 2009″. I my opinion we will certainly see that “the data quality market will grow as customers recognize it as a cheaper precursor to MDM”. “SaaS, cloud, and open source software will not affect the MDM market that much for the foreseeable future”, because it can bring extra risks and costs to organisations. Let’s not forget that most (commercial) organisations are unfamiliar with implementing and supporting open source software. If “IBM, Oracle overcome MDM obstacles” than we can expect to see that in 2009 “mega-vendors still dominate”. And that MDM to reach more industries especially if MDM will focus more on different business drivers; “expect a messaging shift from MDM vendors to promote how MDM can mitigate risk”. As long as SOA will need MDM more and more, you can expect to see that “the MDM market consolidation will continue” and finally a “market stabilization by 2012″.

The added value of an integrated customer view depends strongly on the quality of that integrated customer view. Every organization that is seriously planning to create a single customer view should ask itself the following question: “What determines the quality of my customer view and so the accompanying level of added value?”

Prior to answering this question we need to take one step back. Why does not every organization have a single customer view? The cause lies in the fact that many organizations have their customer data spread across multiple systems all facilitating separate business processes. Additionally customer data is often highly polluted, fragmented and incomplete.

So it appears that the data itself plays a crucial role in the lack of an integrated customer view. Or more accurately, the better the data – the better the customer view. And the better the data matching of customer records across separate systems the better the integrated customer view.

So Data Quality and Matching (Identity Resolution) determine in large parts the quality of the integrated customer view and the added value that it delivers. Take a look at this demo showing a step-by-step approach how to build a single customer view and get a better idea of the role of Data Quality and Matching within this process.

No earth-shattering changes occurred in the latest Forrester Research Wave for Customer Hubs, according to analyst and report author Ray Wang. However, Forrester notes that the market remains in the early-adoption stage for full-blown customer hub solutions. Wang defines the goal of the customer hub segment’s goal in that it “operationalizes the acquisition, distribution, and management of customer information for the use in other systems.” Wang notes that the market is broadening and organizations — especially those with high-volume B2C data — will find that vendors have solutions geared for a company’s every need.
“The good news: Solutions have matured and work well in heterogeneous environments,” Wang writes. “The down side: Enterprises remain challenged with defining data governance and data quality policies while optimizing systems for an information supply chain.” Forrester bases its evaluation upon a product’s current offering, its market presence, and the strategy of the vendor producing it. Additionally, the research firm requires that customer hub vendors provide 20 customer references of live deployments. Wang points out that while some solutions are being implemented, a significant number of customer hub purchases remain on the shelf — either not yet deployed or remaining stagnant as part of a broader product suite.
Wang’s report shows Initiate Systems and Siperian leading the vendor pack. “In a virtual dead heat, both best-of-breed vendors widen the gap among their closest competitors by offering improved data stewardship capabilities, richer hierarchy management, stronger industry support, and greater support for third-party tools,” he writes. Wang refers to Siperian as “the smartest kid on the block,” praising the vendor’s expertise in data acquisition, data cleansing, relationship and hierarchy management, event management, reference data management, data stewardship, and architecture. As for Initiate, Wang says that the vendor has delivered the most significant research-and-development gains in the past 18 months and also has the largest number of productive live customers.
IBM, Dun & Bradstreet’s Purisma, and Oracle Siebel UCM follow close behind in the Leader zone. Wang notes that IBM’s dot on the board has gotten bigger, saying that customer data is a clear strength for the company. Wang also writes that D&B’s recent acquisition of Purisma has helped the organization to bridge gaps in its offerings and go to market with a strong, global B2B solution. Additionally, he points out that Purisma scored in the top rankings for satisfaction in the reference surveys.
Wang says there are a lot of alternatives for companies to sort through in the customer hub market: Just behind the leaders on the Wave report are Sun Microsystems with its open-source master data management options, Oracle CDH, Scotland-based VisionWare, SAS Institute’s DataFlux, and SAP. “Customer hubs make sure CRM is successful and that’s why it is so important to evaluate the [technologies] underlying the CRM processes,” he explains. ” ‘Is the data helping me understand how to cross-sell and upsell? And how do we target our customers?’ ” He goes on to say that CDI vendors seem to have thought through every customer scenario an organization might face. “At this point in the market I think the technology is ahead of the customer,” Wang says.