This watchlist was created in 2004 from several other lists and at that time it consisted of about 68.000 entries. I use the word entries, because in the years after it became fuzzy if one record is the same as one individual. By the end of 2008 the list had grown to over 1,1 million entries. In 2008 after the American Civil Liberties Union (ACLU) mentioned that the list had passed the 1 million, the government came with an explanation. Although we have recorded over 1 million entries in the database, the net result is that these records correspond to about 400.000 individuals. Terrorist often use different and thus multiple identities, use several (falsified) passports etc. But adding entries with only the first initials and last name, while an entry of the full first names and last name already exists will result in unwanted side-effects.

We all know, as being interested in data quality and identity resolution, that J. Robinson will result into much more matches (hits) than James Robinson. Indeed the number of found matches will sky-rocket and have to be evaluated manually. Might this be the reason, that we see more and more security personnel on airports?

In the latest audit report of the U.S. Department of Justice about this watchlist one other problem was analyzed. While extensive procedures were made for nominating and adding suspects to the watchlist, there is no procedure for removing people from the list. Based on a sample of almost 70.000 entries and investigation of the individuals an astounding number of 35% omissions was found. People who had died were still on the list, people who were no longer investigated upon, cases which had been closed etc. So this watchlist is growing and growing. Resulting in screening personnel who ensnare many innocent travelers as suspected terrorists. And wasting their time and divert their energies from looking for true terrorists. It seems to me that FBI and TSC can benefit from better Data Governance, what do you think?

In many cases an inductive method of data cleansing is the way to go. With the right tools and expertise you can inspect, transform and cleanse entities in a database and reach high levels of data quality without the need to use external reference data. In some cases, however, only working with the internal data and inductively identifying and fixing data patterns is not sufficient. Let’s take a practical example: a bank needs to report on a particular segment of its clients to German bank supervisor BaFin – the Federal Financial Supervisory Authority aka Bundesanstalt für Finanzdienstleistungsaufsicht. The bank apparently has done its homework and has created a central database containing all entities needed for the compliance check. Moreover, the bank has worked out a rather complex set of rules how data must be processed and corrected. One of the most important anchor points in this specific framework is the separation between B2C and B2B entities and for the latter the exact identification of the correct legal form. But what if you cannot trust this identification?

After having profiled the data I quickly found thousands of conflicts, e.g. records with the legal form GmbH (limited company) were at the same time tagged as B2C customer. So, which entry is correct? And: if it is a B2B entity, can we be sure that the legal form GmbH is correct? Especially as the bank has set different rules for processing GbR, GmbH, KG, GmbH & Co. KG, AG, KGaA, just to name a few legal forms…

This is when cooperation with a specialised data provider is needed. And it is actually an advantage if this is not the first time you work with external reference data of this partner. Knowing the providers and the specifics of the various data models in use is a prerequisite for successfully enriching data. Not to forget having an excellent, fault-tolerant matching engine that helps you to link corrupted or wrong name entities to the reference database! In the case of the bank, using the right data provider and matching its content to the customers in the compliance database resolved more than ca 90% of conflicts related to the wrong legal form. And this of course is the cornerstone for being compliant!