Package org.apache.pinot.tools.perf

Class QueryRunner

java.lang.Object

org.apache.pinot.tools.AbstractBaseCommand

org.apache.pinot.tools.perf.QueryRunner

All Implemented Interfaces:

Callable<Integer>, Command

public class QueryRunner
extends AbstractBaseCommand
implements Command

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	QueryRunner.QuerySummary

Field Summary

Fields inherited from class org.apache.pinot.tools.AbstractBaseCommand

DEFAULT_CLUSTER_NAME, DEFAULT_ZK_ADDRESS

Constructor Summary

Constructors

Constructor	Description
QueryRunner()

Method Summary

Modifier and Type	Method	Description
String	description()
boolean	execute()
boolean	getHelp()
String	getName()
static QueryRunner.QuerySummary	increasingQPSQueryRunner(PerfBenchmarkDriverConf conf, List<String> queries, int numTimesToRunQueries, int numThreads, int queueDepth, double startQPS, double deltaQPS, int reportIntervalMs, int numIntervalsToReportAndClearStatistics, int numIntervalsToIncreaseQPS, long timeout)
static QueryRunner.QuerySummary	increasingQPSQueryRunner(PerfBenchmarkDriverConf conf, List<String> queries, int numTimesToRunQueries, int numThreads, int queueDepth, double startQPS, double deltaQPS, int reportIntervalMs, int numIntervalsToReportAndClearStatistics, int numIntervalsToIncreaseQPS, long timeout, Map<String,String> headers)	Use multiple threads to run query at an increasing target QPS.
static void	main(String[] args)
static QueryRunner.QuerySummary	multiThreadedQueryRunner(PerfBenchmarkDriverConf conf, List<String> queries, int numTimesToRunQueries, int numThreads, int queueDepth, int reportIntervalMs, int numIntervalsToReportAndClearStatistics, long timeout)
static QueryRunner.QuerySummary	multiThreadedQueryRunner(PerfBenchmarkDriverConf conf, List<String> queries, int numTimesToRunQueries, int numThreads, int queueDepth, int reportIntervalMs, int numIntervalsToReportAndClearStatistics, long timeout, Map<String,String> headers)	Use multiple threads to run queries as fast as possible.
static QueryRunner.QuerySummary	singleThreadedQueryRunner(PerfBenchmarkDriverConf conf, List<String> queries, int numTimesToRunQueries, int reportIntervalMs, int numIntervalsToReportAndClearStatistics, long timeout)
static QueryRunner.QuerySummary	singleThreadedQueryRunner(PerfBenchmarkDriverConf conf, List<String> queries, int numTimesToRunQueries, int reportIntervalMs, int numIntervalsToReportAndClearStatistics, long timeout, Map<String,String> headers)	Use single thread to run queries as fast as possible.
static QueryRunner.QuerySummary	targetQPSQueryRunner(PerfBenchmarkDriverConf conf, List<String> queries, int numTimesToRunQueries, int numThreads, int queueDepth, double startQPS, int reportIntervalMs, int numIntervalsToReportAndClearStatistics, long timeout)
static QueryRunner.QuerySummary	targetQPSQueryRunner(PerfBenchmarkDriverConf conf, List<String> queries, int numTimesToRunQueries, int numThreads, int queueDepth, double startQPS, int reportIntervalMs, int numIntervalsToReportAndClearStatistics, long timeout, Map<String,String> headers)	Use multiple threads to run query at a target QPS.

Methods inherited from class org.apache.pinot.tools.AbstractBaseCommand

cleanup, printExamples, printUsage

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.pinot.tools.Command

call, printUsage

Constructor Detail

QueryRunner

public QueryRunner()

Method Detail

getHelp

public boolean getHelp()

Specified by:

getHelp in interface Command

getName

public String getName()

Overrides:

getName in class AbstractBaseCommand

description

public String description()

Specified by:

description in interface Command

execute

public boolean execute()
                throws Exception

Specified by:

execute in interface Command

Throws:

Exception

singleThreadedQueryRunner

public static QueryRunner.QuerySummary singleThreadedQueryRunner(PerfBenchmarkDriverConf conf,
                                                                 List<String> queries,
                                                                 int numTimesToRunQueries,
                                                                 int reportIntervalMs,
                                                                 int numIntervalsToReportAndClearStatistics,
                                                                 long timeout)
                                                          throws Exception

Throws:

Exception

singleThreadedQueryRunner

public static QueryRunner.QuerySummary singleThreadedQueryRunner(PerfBenchmarkDriverConf conf,
                                                                 List<String> queries,
                                                                 int numTimesToRunQueries,
                                                                 int reportIntervalMs,
                                                                 int numIntervalsToReportAndClearStatistics,
                                                                 long timeout,
                                                                 Map<String,String> headers)
                                                          throws Exception

Use single thread to run queries as fast as possible.

Use a single thread to send queries back to back and log statistic information periodically.

Queries are picked sequentially from the query file.

Query runner will stop when all queries in the query file has been executed number of times configured.

Parameters:

conf - perf benchmark driver config.

queries - query stream.

numTimesToRunQueries - number of times to run all queries in the query file, 0 means infinite times.

reportIntervalMs - report interval in milliseconds.

numIntervalsToReportAndClearStatistics - number of report intervals to report detailed statistics and clear them, 0 means never.

timeout - timeout in milliseconds for completing all queries.

headers - for the query request, e.g. to carry security token.

Returns:

QuerySummary containing final report of query stats

Throws:

Exception

multiThreadedQueryRunner

public static QueryRunner.QuerySummary multiThreadedQueryRunner(PerfBenchmarkDriverConf conf,
                                                                List<String> queries,
                                                                int numTimesToRunQueries,
                                                                int numThreads,
                                                                int queueDepth,
                                                                int reportIntervalMs,
                                                                int numIntervalsToReportAndClearStatistics,
                                                                long timeout)
                                                         throws Exception

Throws:

Exception

multiThreadedQueryRunner

public static QueryRunner.QuerySummary multiThreadedQueryRunner(PerfBenchmarkDriverConf conf,
                                                                List<String> queries,
                                                                int numTimesToRunQueries,
                                                                int numThreads,
                                                                int queueDepth,
                                                                int reportIntervalMs,
                                                                int numIntervalsToReportAndClearStatistics,
                                                                long timeout,
                                                                Map<String,String> headers)
                                                         throws Exception

Use multiple threads to run queries as fast as possible.

Use a concurrent linked queue to buffer the queries to be sent. Use the main thread to insert queries into the queue whenever the queue length is low, and start numThreads worker threads to fetch queries from the queue and send them.

The main thread is responsible for collecting and logging the statistic information periodically.

Queries are picked sequentially from the query file.

Query runner will stop when all queries in the query file has been executed number of times configured.

Parameters:

conf - perf benchmark driver config.

queries - query stream.

numTimesToRunQueries - number of times to run all queries in the query file, 0 means infinite times.

numThreads - number of threads sending queries.

queueDepth - queue size limit for query generator

reportIntervalMs - report interval in milliseconds.

numIntervalsToReportAndClearStatistics - number of report intervals to report detailed statistics and clear them, 0 means never.

timeout - timeout in milliseconds for completing all queries

headers - for the query request, e.g. to carry security token.

Returns:

QuerySummary containing final report of query stats

Throws:

Exception

targetQPSQueryRunner

public static QueryRunner.QuerySummary targetQPSQueryRunner(PerfBenchmarkDriverConf conf,
                                                            List<String> queries,
                                                            int numTimesToRunQueries,
                                                            int numThreads,
                                                            int queueDepth,
                                                            double startQPS,
                                                            int reportIntervalMs,
                                                            int numIntervalsToReportAndClearStatistics,
                                                            long timeout)
                                                     throws Exception

Throws:

Exception

targetQPSQueryRunner

public static QueryRunner.QuerySummary targetQPSQueryRunner(PerfBenchmarkDriverConf conf,
                                                            List<String> queries,
                                                            int numTimesToRunQueries,
                                                            int numThreads,
                                                            int queueDepth,
                                                            double startQPS,
                                                            int reportIntervalMs,
                                                            int numIntervalsToReportAndClearStatistics,
                                                            long timeout,
                                                            Map<String,String> headers)
                                                     throws Exception

Use multiple threads to run query at a target QPS.

Use a concurrent linked queue to buffer the queries to be sent. Use the main thread to insert queries into the queue at the target QPS, and start numThreads worker threads to fetch queries from the queue and send them.

The main thread is responsible for collecting and logging the statistic information periodically.

Queries are picked sequentially from the query file.

Query runner will stop when all queries in the query file has been executed number of times configured.

Parameters:

conf - perf benchmark driver config.

queries - query stream.

numTimesToRunQueries - number of times to run all queries in the query file, 0 means infinite times.

numThreads - number of threads sending queries.

queueDepth - queue size limit for query generator

startQPS - start QPS (target QPS).

reportIntervalMs - report interval in milliseconds.

numIntervalsToReportAndClearStatistics - number of report intervals to report detailed statistics and clear them, 0 means never.

timeout - timeout in milliseconds for completing all queries

headers - for the query request, e.g. to carry security token.

Returns:

QuerySummary containing final report of query stats

Throws:

Exception

increasingQPSQueryRunner

public static QueryRunner.QuerySummary increasingQPSQueryRunner(PerfBenchmarkDriverConf conf,
                                                                List<String> queries,
                                                                int numTimesToRunQueries,
                                                                int numThreads,
                                                                int queueDepth,
                                                                double startQPS,
                                                                double deltaQPS,
                                                                int reportIntervalMs,
                                                                int numIntervalsToReportAndClearStatistics,
                                                                int numIntervalsToIncreaseQPS,
                                                                long timeout)
                                                         throws Exception

Throws:

Exception

increasingQPSQueryRunner

public static QueryRunner.QuerySummary increasingQPSQueryRunner(PerfBenchmarkDriverConf conf,
                                                                List<String> queries,
                                                                int numTimesToRunQueries,
                                                                int numThreads,
                                                                int queueDepth,
                                                                double startQPS,
                                                                double deltaQPS,
                                                                int reportIntervalMs,
                                                                int numIntervalsToReportAndClearStatistics,
                                                                int numIntervalsToIncreaseQPS,
                                                                long timeout,
                                                                Map<String,String> headers)
                                                         throws Exception

Use multiple threads to run query at an increasing target QPS.

Use a concurrent linked queue to buffer the queries to be sent. Use the main thread to insert queries into the queue at the target QPS, and start numThreads worker threads to fetch queries from the queue and send them.

We start with the start QPS, and keep adding delta QPS to the start QPS during the test.

The main thread is responsible for collecting and logging the statistic information periodically.

Queries are picked sequentially from the query file.

Query runner will stop when all queries in the query file has been executed number of times configured.

Parameters:

conf - perf benchmark driver config.

queries - query stream.

numTimesToRunQueries - number of times to run all queries in the query file, 0 means infinite times.

numThreads - number of threads sending queries.

queueDepth - queue size limit for query generator

startQPS - start QPS.

deltaQPS - delta QPS.

reportIntervalMs - report interval in milliseconds.

numIntervalsToReportAndClearStatistics - number of report intervals to report detailed statistics and clear them, 0 means never.

timeout - timeout in milliseconds for completing all queries.

numIntervalsToIncreaseQPS - number of intervals to increase QPS.

headers - for the query request, e.g. to carry security token.

Returns:

QuerySummary containing final report of query stats

Throws:

Exception

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception