Optimization of Data Cleaning Programs with User-Defined Functions

November 16, 2021

Data has been crucial ever since the dawn of the Internet and it is being collected at every instant. However, in order to extract information from it, data must be cleaned, so that it does not contain errors, inconsistencies and approximate duplicate values. To do that, data cleaning tools are used. Typically, a data cleaning tool can be rule-based or transformation-based. In the first type, there is a set of rules that the data must satisfy. Otherwise, there is a set of repairs provided by the user to be applied over the data. In the second type, the data cleaning tools operate over a workflow of data operations, typically modelled by a Directed Acyclic Graph (DAG). In both cases, external functions may be used, to perform specific data cleaning operations. These functions are typically black boxes to the data cleaning tool, as well as very expensive to execute. We propose to extend the optimizer of CLEENEX, a transformation-based data cleaning prototype that uses external functions. Our extensions focus on the external functions, namely their cost of execution, in order to enhance the overall performance of CLEENEX. The main goal of our extensions is to reduce the impact of executing external functions, namely by avoiding unnecessary executions. Moreover, we provide a method that enables to estimate the cost of their execution. The introduction of these features yields significant execution time speed-up for several cases. We present the state-of-the-art related work, our proposed solution and an evaluation and validation methodology for it.

Authors in the community:

• 

  J

  João Pedro Monteiro Oliveira da Silva Caetano

  ist186452

Supervisors of this institution:

• 

  H

  Helena Isabel De Jesus Galhardas

  ist13078

electrical-engineering-electronic-engineering-information-engineering - Electrical engineering, electronic engineering, information engineering

eng - English

Instituto Superior Técnico