I'm currently involved in a large scale CDI implementation for a client, and what we found is that to fix the issues around data quality, across the organization, someone had to take ownership first. I realise the obviousness of that last sentence but this is not as easy as most people might think. The initial response was that this is an IS responsibility, but IS might not necessarily understand all the relevant business rules encapsulated in that data. The problem is the client has multiple systems running inside and across separate legal entities, and herein lays the challenge. An enterprise wide controlling body is required to first identify all DQ related problems and then analyse why the data is in that state. An IS department might initially be able to fix the problems using DQ tools, but the reasons behind those issues might not be resolved and as such will return.
To return to my question - for the tools to work an organisation will first have to take ownership of their data and this is not an IS only function but should be a joint venture between IS and Business.
You raised a very good point. Seldom any data issues can be solved completed with tools/technologies, especially something as complex and high impacting as the customer data. What we have observed in companies that have done this right is that before any technology is deployed, the ownership of and responsibility for the data will need to be defined clearly, and often this is an enterprise wide exercise, even more engaging than implementation of individual enterprise applications such as CRM or ERP. As a side result, we are seeing that companies are start to put in place the role of chief data architect or centralized data management organization. Of course the mere existance of those roles/organizations is not enough to guarantee the success of CDI/MDM project, but at least it is a good start, and an indication that the company is serious about putting the right organization infrastructure to deal with data issues.
This is common problem in many organizations now. CIO wants to do the data quality assessment and assign this task to some vendor, in most cases when the consultant goes to field, nobody takes responsibility of data. IT guys usually worried about extracting the data and loading it to appropriate target system, no one cares about Quality.
One good thing is off late industry is understanding the imporatnce of data after many sleepless nights due to bad data.
I have seen companies struggling with analytics during their quarter/year end results.
Everyone starting to have a look at the data cleansing tools available in the market irrespective of the domain and trying to form a Data Governance team in organization.
Technical people with data quality tools can do more harm then good as they don't always understand the business rules and data relationships. Business people with tools can do harm as they know the data but don't have the SDLC discipline to perform repairs. So collaboration becomes the key to data quality - the people with the tools can collaborate with the people who know the data.
You can do this right now with Enterprise 2.0 collaboration tools - you can take the discussion out of email trails and documentation and into forums and wikis. You need the collaboration to be findable months or years later when people find the same DQ problems. We are starting to see a convergence of bi, metadata and data quality tools and Enterprise 2.0 collaboration functions so the people accountable for the data can communicate with the people who manage the data.
That's absolutely a loaded question, and I'm glad that previous responders have agreed that both the IT and business owners have to work together on solving the issues. Where the business owner understands the business need, they are not aware of internal relationships; the IT organization understands the ERP relationships, but may not understand critical business requirements. That's where the role of ERP business analyst is an absolute asset. The business analyst has specific business functional experience, as well as IT expertise sufficient to help both sides understand the actual data, as well as the consequences of using inadequate, or incomplete tools or scripts to analyze or correct data problems. This person or organization may work in close association with a company's quality or testing organization, as well as the business process owner to support corporate need. What I find works well is a tiger team of IT, business owner and business analyst, depending on the scope of the problem, and you may need a high level corporate sponsor to drive cooperation.