Late 2009 in their report on Who’s Who in Open-Source Data Quality, Andreas Bitterer and Ted Friedman from Gartner, pointed already to DataCleaner as a promising tool. A tool that, in their opinion, could certainly improve by offering more high end Cleansing functions and improve the rather basic User Experience.

Since then, a lot has happened in the DataCleaner space and in the profiling market. Before the launch of version 2 we notified everybody on the acquisition of eobjects.org or DataCleaner by Human Inference. It might be that some of you were curious on what would happen with the functionality, and as stated at that time we would continue with the community and further participate and expand in it. Under the flag of Human Inference we launched the renewed DataCleaner 2.0, where we definitely increased the customer experience with an enhanced user interface together with possibilities to provide filters or filter flows. The filter flows show their benefit if you analyze your data source and want to create new (temporary) data sources based on matching criteria. You can do that either manually, or in a completely automated way to monitor your data.

With Open Source in general, and with DataCleaner in particular we want the community to participate in the functionality of the product. Since long DataCleaner contains the RegexSwap: the community where you can share regular expressions. Why would everybody reinvent the same wheel to build a regular expression on creditcard checks, emails, etc?

Next to regular expressions that can be used to profile data, there is the need on cleansing functions that contain much more business logic that can hardly be covered in a regular expression. For example, to validate of the syntax of an email is correct is something else than validating if there is also a running mail server attached to the domain. Cleansing functions are already part of DataCleaner but there is always a need for other or more advanced functional extensions. To prevent that you need to create them in the ‘DataCleaner’ way we have created an easy extension sharing mechanism.

With the introduction of DataCleaner 2.2 we now introduce the logical next step with the ExtensionSwap. Individuals and companies can provide their functionality as extensions to DataCleaner. We provide an extension store where you can easily select that functionality you were looking for and combine it in your profiling. Some of these extensions will be working on premise, some of them will be easy accessible via the cloud. We’ve already provided a webcast and an example project on how to create and distribute your own extensions – it’s a piece of cake.

As DataCleaner can run in the cloud we can also imagine that people or companies have specific cleansing functions ‘hanging around’ that they want to combine in an easy way in their profiling steps. The same is true for Human Inference where we have our famous Contact related cleansing functions and want to combine that in profiling. For now we provide the HIquality Contacts as a preview and free trial version for everyone. Use it and please provide feedback in case you are missing functionality.

In the evolution of information technology Gartner provided a new term as ultimate goal to reach: Pattern-Based Strategy.

As you were reaching for the final destination in your ultimate journey to transform bits and bytes to real information, again you encounter a new optimum. Pattern-Based Strategy, as described by Yvonne Genovese et al. can be identified as the last era in all the eras of IT-value add. Basically, the level of control identifies in which of the era you currently operate – from tight control and pure automation in the ‘old’ days via augmentation, e-commerce/Web 1.0 and web 2.0 to the highest era called – Pattern-Based Strategy.

In Pattern-Based strategy you make use of the collective intelligence, not only collect data inside and outside your enterprise from those you know, but pro-actively collect information from the crowd, analyse this information and see if you can recognize the patterns on which you know how to respond. Try to be fast on weak signals so that you are the first in that market.

As in Business Intelligence (where you look in your rear mirror to make decisions for the future) and Business Activity Monitoring, within Pattern-Basted Strategies (where you look around and even let others look around while you make decisions) it only works if you can compare data, if you can make information out of your data. In order to make the right company decisions, to steer well, you need to rely on the information that you use. And although we like the new approach behind Pattern-Based Strategy, we as Human Inference, still see enterprises struggle with their Business Intelligence because of their data quality. We see them starting with Business Activity Monitoring and fail because of their data quality. And now we see them start dreaming of Pattern-Based Strategy. Before you act on that dream, solve the data quality – there are solutions available.

For those who already tried to combine information from social communities, the lack of standards there, prevents that you can easily collect that collective intelligence for your patterns. As Human Inference, we value your data, we identified these issues and provide data quality tools to enrich this collective intelligence in order to transform it to trusted information for your strategy.

On 28 january 2010 the next Human Inference Data Quality Summit will be held in the Evoluon in Eindhoven (NL). The theme – Value your data, value your future- is inspired by the idea that investments in data quality have become part of standard business and that vision, strategy and solutions are being synchronized with these investments. As data quality has reached a certain level of maturity, it is time to have an in-depth look at the (near) future of Data Quality.

The program is challenging, comprehensive and entertaining. Keynote speakers include Ted Friedman (vice president Gartner Research), Mathias Klier (professor at the University of Innsbruck) and Sabine Palinckx (CEO Human Inference). Additionally, in the break-out-session a wide variety of theme-related topics will be addressed: maximising the buisnes value of information, guiding a dq-project through migration, data quality maturity, marketing effectiveness and many more….. In short, the Data Quality Summit is not to be missed!

Save the date and register by clicking this link!

• Having the right contact details will reduce waste of natural resources because we bring the deliveries immediately at the right place, and it’s not only the deliveries that can be optimized, we can also avoid that deliveries get lost and natural resources are actually piped for /dev/null !
• By valueing our data through deduplication we can in general avoid to spoil needless energy – both by humans and other resources – and use the sparse energy only for those who actually need it. Here I feel the same remark as David in his blog. There comes a moment in the near future, with an rising energy prices and increasing emission penalties, that that aspect will win in the equation from the actual spoil of goods and human energy.
• Saving resources is now also done by concentrating or centralizing services – optimizing the service per energy unit. For data we see this happening in the Virtualization of data amd Master Data Management technologies. Strong place in your centralizing strategy will be the role of your data quality – that will bring your real value

I encourage you all to think out-of-the-box how data-value can help to make it a better world for the future. But I’m afraid that in this economic climate the short term is ruling and not the long(er) term.

David Cearley at Gartner comes with a fascinating other angle. Basically he sees virtualization also as strategic technology to virtualize the data. And by that twist, data quality and data governance appears annoyingly in the middle of your radar screen. In order to use this strategy for your operational excellence, to eliminate the number of redundant data on your real storage devices, and make a virtual layer between your applications and this virtual data storage, you need to be sure that all your applications can work seamlessly with that virtual data.

To significantly reduce your amount of redundant data you need to make it more agnostic. Your apps will still expect the data in a specific format, while your virtual data is stored in another way. You need tools to convert it quickly – back and forward – from the standard virtual way (e.g. first-name, prefix, last-name, zip-code, thoroughfare, etc) to the specific application way (e.g. name in one field, zip-code, street).

Virtualization of data is not only about deduplication of the data! The challenge is more on how to use the virtual data in your existing applications in a non-invasive way. This can be done by using good data quality tools for your cleansing and deduplication, and for your – back and forward – standardisation.

In general I agree with the top 10 of technological strategies, be there some slight personal priority adaptations, but let’s focus on that in later blogs. The missing point is in my opinion the lack of emphasis on risk mitigation, and I do realize that things changed since October 2008. Which technologies can we adopt to avoid that we provide services, products, at the end money to the wrong contacts, or that we are sure to deliver it to the right contacts. The technology strategy of Master Data Management, Know your customer, Single View of X, or how we call it, will need our attention in 2009!

A couple of days ago IDC published a press release on cloud computing and how it is reshaping the IT marketplace.

Some major points I think are very interesting:

• Cloud computing is “crossing the chasm” and entering a period of widespread adoption
• IDC expects the cloud adoption trend to be amplified by the current financial crisis
• IT cloud services to grow almost threefold, reaching $42 billion by 2012
• More importantly, cloud computing will account for 1/3 of IT growth in 5 years

Jeff Kelly from SearchDataManagerment.com writes about the momentum of SaaS based data quality and integration tools. With quotes from a survey done by research firm Gartner he writes about how “the on demand model is beginning to gain foothold in the industry”.

A few remarkable facts from the Gartner survey:

• 28% of companies have deployed SaaS-based data integration tools
• 24% have SaaS-based data quality tools
• Most companies use a mix of on-premise, outsourced and SaaS-based technology
• SaaS-based software, at this point, often trails its on-premise counterparts
• Gartner advice: “Plan SaaS deployment enterprise-wide….to guard against siloed investments”

Enterprise-wide data quality project requires custom implementation and customisation of the tools to support the specific business processes. This maybe difficult to achieve in the data quality SaaS offerings that are available at this time. On the other hand many data quality functions like rapid addressing, data enrichment or even deduplication happen all across the enterprise and may be well served with on demand offerings.

The current economic crisis will force many organisations to reduce IT expenses and SaaS based data quality will have increasing appeal. In most organisations a mix between on demand services and on premise implementation will be the best model.