Package org.graylog2.outputs

Class IndexSetAwareMessageOutputBuffer

java.lang.Object
org.graylog2.outputs.IndexSetAwareMessageOutputBuffer
public class IndexSetAwareMessageOutputBuffer extends Object
A thread-safe and index set aware output buffer implementation.

This buffer class is aware of index sets and calculates the remaining buffer capacity based on the number of index sets per message. If a message has two different index sets, the indexer output will create one message entry per index set in the bulk request against OpenSearch.

To avoid bulk requests that get too big, we reserve one buffer slot per message and index set. The trade-off is that outputs which don't create one message per index set will write smaller batches.

  • Constructor Details

    • IndexSetAwareMessageOutputBuffer

      @Inject public IndexSetAwareMessageOutputBuffer(@Named("output_batch_size") BatchSizeConfig maxBufferSize, com.fasterxml.jackson.databind.ObjectMapper objectMapper)
      Creates a new buffer with the given size.
      Parameters:
      maxBufferSize - the maximum buffer size

  • Method Details

    • shouldFlush

      public boolean shouldFlush(Duration flushInterval)
      Checks if the time of the last buffer flush is larger than the given flush interval.

      This method is thread-safe.

      Parameters:
      flushInterval - the flush interval duration
      Returns:
      true if the time of the last buffer flush is larger than the given flush interval. Otherwise, false.

    • appendAndFlush

      public void appendAndFlush(FilteredMessage filteredMessage, Consumer<List<FilteredMessage>> flusher)
      Appends the given message to the buffer. If the buffer length has reached the configured max buffer size, the given flush consumer is called with the contents of the buffer and the buffer is reset. The consumer is responsible for handling the buffer content.

      This method is thread-safe.

      Parameters:
      filteredMessage - the message to append to the buffer
      flusher - the buffer flush consumer

    • flush

      public void flush(Consumer<List<FilteredMessage>> flusher)
      Calls the given flush consumer with the contents of the buffer and the buffer is reset. The consumer is responsible for handling the buffer content.

      This method is thread-safe.

      Parameters:
      flusher - the buffer flush consumer