org.apache.pinot.tools.realtime.provisioning

Class MemoryEstimator

java.lang.Object

org.apache.pinot.tools.realtime.provisioning.MemoryEstimator

public class MemoryEstimator
extends Object

Given a sample segment, this class can estimate how much memory would be used per host, for various combinations of numHostsToProvision and numHoursToConsume

Constructor Summary

Constructors

Constructor and Description
MemoryEstimator(org.apache.pinot.spi.config.table.TableConfig tableConfig, File sampleCompletedSegment, int ingestionRate, long maxUsableHostMemory, int tableRetentionHours)

Method Summary

Modifier and Type	Method and Description
void	estimateMemoryUsed(File statsFile, int[] numHosts, int[] numHours, int totalConsumingPartitions, int retentionHours) Given a sample segment, the time for which it consumed, numReplicas and numPartitions, estimate how much memory would be required per host for this table Algorithm: Given numReplicas and numPartitions, we can find out total consuming partitions per host, for various numHosts totalConsumingPartitionsPerHost = (numReplicas * numPartitions)/numHosts Given a sample realtime completed segment (with size s), and how long it consumed for (t), we can estimate how much memory the table would require for various combinations of num hosts and num hours For estimating the memory occupied by completed segments- For each numHoursToConsume we compute: If a segment with size s takes time t to complete, then consuming for time numHoursToConsume would create segment with size estimatedSize = (numHoursToConsume/t)*s If retention for completed segments in memory is rt hours, then the segment would be in memory for (rt-numHoursToConsume) hours A segment would complete every numHoursToConsume hours, so we would have at a time numCompletedSegmentsAtATime = (rt-numHoursToConsume)/numHoursToConsume to hold in memory As a result, totalCompletedSegmentsMemory per ConsumingPartition = estimatedSize * numCompletedSegmentsAtATime For estimating the memory occupied by consuming segments- Using the sample segment, we initialize the stats history For each numHoursToConsume we compute: If totalDocs in sample segment is n when it consumed for time t, then consuming for time numHoursToConsume would create totalDocs = (numHoursToConsume/t)*n We create a MutableSegmentImpl using the totalDocs, and then fetch the memory used by the memory manager, to get totalConsumingSegmentMemory per ConsumingPartition totalMemory = (totalCompletedMemory per ConsumingPartition + totalConsumingMemory per ConsumingPartition) * totalConsumingPartitionsPerHost
String[][]	getActiveMemoryPerHost()
String[][]	getConsumingMemoryPerHost()
String[][]	getNumSegmentsQueriedPerHost()
String[][]	getOptimalSegmentSize()
File	initializeStatsHistory() Initialize the stats file using the sample segment provided.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MemoryEstimator

public MemoryEstimator(org.apache.pinot.spi.config.table.TableConfig tableConfig,
                       File sampleCompletedSegment,
                       int ingestionRate,
                       long maxUsableHostMemory,
                       int tableRetentionHours)

Method Detail

initializeStatsHistory

public File initializeStatsHistory()

Initialize the stats file using the sample segment provided.
This involves indexing each row of the sample segment using MutableSegmentImpl. This is equivalent to consuming the rows of a segment. Although they will be in a different order than consumed by the host, the stats should be equivalent.
Invoking a destroy on the MutableSegmentImpl at the end will dump the collected stats into the stats.ser file provided in the statsHistory.

estimateMemoryUsed

public void estimateMemoryUsed(File statsFile,
                               int[] numHosts,
                               int[] numHours,
                               int totalConsumingPartitions,
                               int retentionHours)
                        throws IOException

Given a sample segment, the time for which it consumed, numReplicas and numPartitions, estimate how much memory would be required per host for this table

Algorithm:
Given numReplicas and numPartitions, we can find out total consuming partitions per host, for various numHosts
totalConsumingPartitionsPerHost = (numReplicas * numPartitions)/numHosts

Given a sample realtime completed segment (with size s), and how long it consumed for (t),
we can estimate how much memory the table would require for various combinations of num hosts and num hours

For estimating the memory occupied by completed segments-
For each numHoursToConsume we compute:
If a segment with size s takes time t to complete, then consuming for time numHoursToConsume would create segment with size estimatedSize = (numHoursToConsume/t)*s
If retention for completed segments in memory is rt hours, then the segment would be in memory for (rt-numHoursToConsume) hours
A segment would complete every numHoursToConsume hours, so we would have at a time numCompletedSegmentsAtATime = (rt-numHoursToConsume)/numHoursToConsume to hold in memory
As a result, totalCompletedSegmentsMemory per ConsumingPartition = estimatedSize * numCompletedSegmentsAtATime

For estimating the memory occupied by consuming segments-
Using the sample segment, we initialize the stats history
For each numHoursToConsume we compute:
If totalDocs in sample segment is n when it consumed for time t, then consuming for time numHoursToConsume would create totalDocs = (numHoursToConsume/t)*n
We create a MutableSegmentImpl using the totalDocs, and then fetch the memory used by the memory manager, to get totalConsumingSegmentMemory per ConsumingPartition

totalMemory = (totalCompletedMemory per ConsumingPartition + totalConsumingMemory per ConsumingPartition) * totalConsumingPartitionsPerHost

Parameters:

statsFile - stats file from a sample segment for the same table

numHosts - list of number of hosts that are to be provisioned

numHours - list of number of hours to be consumed

totalConsumingPartitions - total consuming partitions we are provisioning for

retentionHours - number of most recent hours to be retained in memory for queries.

Throws:

IOException

getActiveMemoryPerHost

public String[][] getActiveMemoryPerHost()

getOptimalSegmentSize

public String[][] getOptimalSegmentSize()

getConsumingMemoryPerHost

public String[][] getConsumingMemoryPerHost()

getNumSegmentsQueriedPerHost

public String[][] getNumSegmentsQueriedPerHost()