HIquality bug cycle

Usually our customers are satisfied with our product suite. At customer support I never receive information about the successful implementations. I got to know our software through the problems that occur, and in almost 15 years of acceptance testing and customer support, I’ve seen all kind of bugs passing by.
Software crashes and never ending loops are nasty. Worse are those bugs that are not that visible in the beginning, but keep on growing in the course of time.
Recently we caught such a bug in our longest existing product HIquality Identify.

HIquality Identify is often used in search applications. Just like police and justice use “descriptions” of a criminal, HIquality Identify uses descriptions of source data to detect the right records in a database. Source records are decomposed to core words and the phonological codes of the core words of streets, names and places are stored in the description table. This table is the base of the search application.

Nulls, nels and nols
Whenever a source record is changed, the descriptions have to be updated as well. A synchronize procedure is used to keep the description table up to date.
Due to a little mistake in this procedure we recently released a version of the Oracle Upgrade pack, that didn’t recognize null values in the database any more. Empty fields in the database resulted in core words with the value ‘null’, and the phonological codes ‘nel’ and ‘nol’.
As a result the scores of evaluations became less accurate, and end scores became too high. The phonological codes of the core words are used as indexes. These indexes are used to pre-calculate the maximum number of evaluations. Since more and more of these fields are changed to nul nel and nol, after several months, instead of search results time-outs occur, stating that not enough relevant search data was entered. 

Did you want to know this?
In a short time we figured out which customers received this special software release, and our consultants visited them for an upgrade. In the end all of them upgraded without knowing the actual reason or what kind of harm potentially could have been caused.
As a customer you want problems to be repaired and bugs to be fixed, without knowing every single detail. Is this nol-worm kind of virus something you would have wanted to know about? Probably not. In case you will have nightmares about nulls, nels and nols in your database, you can contact me any time at Human Inference’s Customer Support, where I cope with all those kind of things that can go wrong in software.

Yesterday, I attended the Capgemini Cloud Computing Conference in Utrecht and I had the opportunity to listen to Andy Mulholland, their Global CTO. His presentation was by far the most visionary of the program and I would like to share some of the remarks that struck me as interesting. Andy set out to explain how business is changing its agenda. The traditional inside-out approach is changing to an outside-in approach: From opportunity to business intelligence and not the other way around.

He further stated that the ability to make money in the cloud is vastly larger than the ability to save money. I liked this statement, because I’ve heard the “cost saving argument” with regard to cloud computing far too often. Or, as he comments in his book Enterprise Cloud Computing: “The WOW about cloud computing isn’t about on-demand information technology….. It’s about on-demand business innovation”. Of course, Andy did not give all the answers in his presentation , but he’s trying to find way to to resolve the true potential of this new means of delivering IT-resources.

The part I especially liked, was when he said that new business is about creating opportunity by supporting the development of people and expertise. Good stuff! I’m definitely going to read his book…..

Have you ever tried to get contact details in and out of a CRM system, and ended up with a bigger mess? I have. The concept is easy: store all information about prospects and customers in one system, allowing you to have your communication efforts streamlined.

Reality, however, is harder: contact details entered on your website should be fed to the system automatically. Sending your periodic newsletter should be based on the details in your CRM system. Not to mention dealing with information on bounces. Integrating your CRM system(s) with mass mailing, campaign management and self service portals is helpful, but for some reason the major means of transporting lead and customer information still seems to be Excel… Leaving you with the necessity to mass import results, new contacts and changed information.

What you really want is the ability to throw whatever information you have at the system, and let the system determine if you already know someone and take care of adding information to already existing contacts. In addition you do not want to spend time on maintaining information you have. Actually, wouldn’t it be nice if you would be able to have your website automatically add leads to your CRM system (e.g. via the Salesforce Web2Lead function), but without adding duplicates?

The good news is that this can be done. There is tooling out there that keeps your contact data up-to-date, prevents adding duplicate records and makes dealing with contact data easy.

Everybody who has ever been on holiday in France has probably had a neighbour named Gaston, Jacques, Louis, Claire or Françoise . We are used to those first names, they evocate the “France profonde”, sleepy villages at the end of a road, films of Pagnol or Rohmer. Walks along the Seine in de shadow of “Notre Dame” in the spring. Coffee at a terrace of the Boulevard Saint-Germain where an obsequious garçon, named Marcel, is looking at your girl friend or wife in a way you dot not really appreciate. This particular image of France is in danger. In a few years our total frame of reference could have disappeared.

Nowadays French parents let their imagination go freely when they are choosing first names for their children. Looking at recent entries in the civil registry, you will find rather unusual first names like Bulle, Héribert, Loeva, Hermès, Evolène, and Argan.
These first names have all kind of origins. For example, they can be a combination of first names (Timéo, which is derived from Timothée and Théo),or they are different writing forms of known first names (Lilou becomes Lee-Lou). We can also find names from Greek or Celtic mythology or even from literature, like Arwen, a character from the novel Lord of the Rings.

This interest for uncommon first names will of course have consequences for the processing of French data, especially if you take into consideration that these “new” first names, with a frequency less than 3000, are now in the majority. But this diversity will not necessary be a curse. Maybe we will be delivered from ambiguous names, of which we never know whether it is a first name or a surname. Consider the 3 most common surnames in France are Martin, Bernard and Thomas.
But don’t worry; in order to keep the challenge going when you process French data, names like Jacqueline-Germain, Jean Marie Marie Luce, Louise Alexandrine will of course not disappear entirely.
And next time you will be in France, enjoying a salade Niçoise with a cold glass of rosé, overlooking a harbor, where small fishing boots are dancing on the lazy waves, you just will have to get used to the fact that the waitress’s first name is not Marie but Fanchon or Eole.

Source: Le Parisien 19-02-2010 and “Les 4 000 plus beaux prénoms rares “, de Stéphanie Rapoport, chez First, 8,90 €

In many cases an inductive method of data cleansing is the way to go. With the right tools and expertise you can inspect, transform and cleanse entities in a database and reach high levels of data quality without the need to use external reference data. In some cases, however, only working with the internal data and inductively identifying and fixing data patterns is not sufficient. Let’s take a practical example: a bank needs to report on a particular segment of its clients to German bank supervisor BaFin – the Federal Financial Supervisory Authority aka Bundesanstalt für Finanzdienstleistungsaufsicht. The bank apparently has done its homework and has created a central database containing all entities needed for the compliance check. Moreover, the bank has worked out a rather complex set of rules how data must be processed and corrected. One of the most important anchor points in this specific framework is the separation between B2C and B2B entities and for the latter the exact identification of the correct legal form. But what if you cannot trust this identification?

After having profiled the data I quickly found thousands of conflicts, e.g. records with the legal form GmbH (limited company) were at the same time tagged as B2C customer. So, which entry is correct? And: if it is a B2B entity, can we be sure that the legal form GmbH is correct? Especially as the bank has set different rules for processing GbR, GmbH, KG, GmbH & Co. KG, AG, KGaA, just to name a few legal forms…

This is when cooperation with a specialised data provider is needed. And it is actually an advantage if this is not the first time you work with external reference data of this partner. Knowing the providers and the specifics of the various data models in use is a prerequisite for successfully enriching data. Not to forget having an excellent, fault-tolerant matching engine that helps you to link corrupted or wrong name entities to the reference database! In the case of the bank, using the right data provider and matching its content to the customers in the compliance database resolved more than ca 90% of conflicts related to the wrong legal form. And this of course is the cornerstone for being compliant!

The term SOA is introduced already many years ago by Roy Schulte and Yefim Natis - also from Gartner. And even at the time of the introduction of SOA one could start a dispute about the differences between SOA and CBD – Component Based Development. And there are similarities between that dispute and the dispute between WOA and SOA. My take on it is that already from the beginning of ICT or software engineering one is trying to conquer complexity and still remain the benefits of flexibility or agility. The more the technology becomes mature, the more we are able to tackle this paradox.

David Cearley’s take on this strategic technology is that the coming 5 years the standards and technologies still significantly. That web-based solutions will improve that much that global-class solutions that can fulfill enterprise computing needs – the web as the ultimo service oriented architecture. In the data value space we see already an increasing demand and supply of web-based solutions that seamlessly fit in such service oriented solutions – think about all the Cloud and SaaS initiatives. Technologies like – and the global adoption of – SOAP, REST and AJAX definitely gave the right boost to transform SOA into WOA.

Nice sidetrack on SOA vs WOA is that in particular The Netherlands WOA is not a disease anymore (Dutch collective noun for all Sexual Transmitted Diseases). In the past analyst misinterpreted the relevance of Service Oriented Architecture per country on the hits they got on Google, because of that The Netherlands was longtime seen as leading country on Service Oriented Architecture. With WOA that miscalculation can be solved immediately. Value of data – always fun.

Number 6 in Gartner’s top 10 Strategic Technologies for 2009 is all about Social software and networks. Their focus is mainly on the benefits of using the growing virtual social networks for commercial reasons. For Data Value I see 2 observations:

1. The data value within and accross social networks. Within because it becomes harder and harder to find the right person, e.g. LinkedIn and search on Henk de Ruiter you will see actual multiple entries for the same person, and … no wonder you will find actually the same person via the ID Henk deRuiter. The cause for these duplications are as for any other datasource. Users (the actual individuals) didn’t know how to use the system (e.g., make a new ID per company that you worked for) or the systems cannot handle the linguistical and cultural challenges of the world. Since a couple of years we see more and more of these networks in coopetition. Where the networks ‘smartly’ use the ID – firstname + lastname – to collect personal and network information. This is not always correct. The challenge is even more in the synchronization, nobody changes things on all networks and everybody trusts that ‘the systems’ will do the right synchronizations. Initiatives of OpenSocial can help in this area.
   
2. The data within the social networks can be used to enrich your view on your contact. Mind that the data in the networks is not always that accurate, but for the most part, and by combining networks via OpenSocial, you can really bring more value in yur contact data – and that’s the value of enrichment. The moment you automatically link your contact data with the social networks it will be your contact that will enrich your data – every time he or she updates one of it’s networks. As that data is used for multiple purposes it will also give you some guarantee on the content as well. Let’s hope that OpenSocial or any other alike standard can circumvent their technical challenges and will be fully adopted in this area.

A happy new year and be sure to visit DataValueTalk.com in 2009 for continued data value.

Cloud Computing or Cloud Services that’s the ultimate dream we have. It doesn’t bother anymore where services are running, we need a handle to it and it can start raining. Pending on which part of the world you live, you like clouds or not. For someone with lastname “van Holland” it’s almost hard to believe that he needs to trust the clouds!

Number 3 on Gartner’s list on Top technology strategies for 2009 is dealing about Cloud Computing. And a more detailed blog with the challenging title “Delivering Cloud Services: ISVs – Change or Die or both!” on that topic can be found on the blog of Daryl Plummer.

Cloud computing and cloud services will change the data quality landscape. More and more smaller companies can now utilize the analyzing, cleansing, standardization, enrichment and deduplication services that in past were only available om premise for larger companies. By using the cloud, massive parallel computations can be executed to bring solutions for huge volumes of data belonging to large companies. Next steps will be that the cloud will execute data cleansing more efficient. Why cleanse in total 200 million contacts for a dozen companies while the region itself has only 100 million citizens (or households).

Steps we still face in light of Cloud Computing are how to differentiate the services for individual need per customer, and the whole serious issue of privacy and security.