org.apache.pinot.tools.anonymizer

Class PinotDataAndQueryAnonymizer

java.lang.Object

org.apache.pinot.tools.anonymizer.PinotDataAndQueryAnonymizer

public class PinotDataAndQueryAnonymizer
extends Object

The goal of this tool is to generate test dataset (as Avro files) with characteristics similar to a given source dataset. The source dataset is a set of Pinot segments. The tool can be used in situations where actual source data isn't allowed to be used for the purpose of testing (regression, performance, functional, evaluation of other OLAP systems etc). The tool understands the characteristics of the given dataset (Pinot segments) and generates corresponding random data while preserving those characteristics. The tool can then also be used to generate queries for the random data. So if we have a set of production data which you want to use for testing but are unable to do so (because of security restrictions etc), then this tool can be used to generate corresponding anonymous data and queries. Users can then use the anonymized dataset (avro files) and generated queries for their testing. One avro file is generated per input Pinot segment. The tool also randomizes the column names (and table name) so that source schema is not revealed. The user is also allowed to provide a set of columns for which they want the data to be retained as is (not anonymized). User should be careful when choosing these columns. Ideally these should be time (or time related) columns since they don't reveal anything and so it is fine to copy them as is from souce segments into Avro files. Please see the implementation notes further in the code explaining the global dictionary building, and data generation and query generation phases in detail. Also, please see usage examples in AnonymizeDataCommand to learn how this tool can be invoked from command line. Future workadd -- - Add support for partitioning (where dataset is hash partitioned on column) - Potential memory explosion for extreme high cardinality global dictionary columns Please read the design doc for details. - Add support for no dictionary filter columns (Generally this should not happen since whoever is using filter on a column should have created a dictionary for that column. But in case the filter column does not have segment dictionary, then we will not be able to build global dictionary) - Make the global dictionary building phase per column basis to reduce the memory footprint - Add support for generating queries when a predicate was not there in the original table and therefore not present in global dictionary either. The current implementation won't be able to rewrite this predicate correctly. It will substitute the original value with null.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PinotDataAndQueryAnonymizer.FilterColumnExtractor * Filter Column Extractor * *
static class	PinotDataAndQueryAnonymizer.QueryGenerator * QueryGenerator * *

Constructor Summary

Constructors

Constructor and Description
PinotDataAndQueryAnonymizer(String segmentDir, String outputDir, String fileNamePrefix, Map<String,Integer> globalDictionaryColumns, Set<String> columnsNotAnonymized, boolean mapBasedGlobalDictionary) Create an instance of PinotDataGenerator

Method Summary

Modifier and Type	Method and Description
void	buildGlobalDictionaries() * Global Dictionary Builder * *
void	generateAvroFiles() * Data Generator * *

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PinotDataAndQueryAnonymizer

public PinotDataAndQueryAnonymizer(String segmentDir,
                                   String outputDir,
                                   String fileNamePrefix,
                                   Map<String,Integer> globalDictionaryColumns,
                                   Set<String> columnsNotAnonymized,
                                   boolean mapBasedGlobalDictionary)

Create an instance of PinotDataGenerator

Parameters:

outputDir - parent directory where avro files will be generated

segmentDir - directory containing segment

fileNamePrefix - generated avro file name prefix

Method Detail

buildGlobalDictionaries

public void buildGlobalDictionaries()
                             throws Exception

* Global Dictionary Builder * *

Throws:

Exception

generateAvroFiles

public void generateAvroFiles()
                       throws Exception

* Data Generator * *

Throws:

Exception