Package org.apache.pinot.common.http

Class MultiGetRequest

  • public class MultiGetRequest
extends Object
    Class to support multiple http GET operations in parallel by using the executor that is passed in. This is a wrapper around Apache common HTTP client. The execute method is re-usable but there is no real benefit to it. All the connection management is handled by the input HttpConnectionManager. For access through multiple-threads, use MultiThreadedHttpConnectionManager as shown in the example below. Usage: 
     
    List<String> urls = Arrays.asList("http://www.linkedin.com", "http://www.google.com");
    MultiGetRequest mget = new MultiGetRequest(Executors.newCachedThreadPool(),
           new MultiThreadedHttpConnectionManager());
    CompletionService<GetMethod> completionService = mget.execute(urls);
    for (int i = 0; i < urls.size(); i++) {
      GetMethod getMethod = null;
      try {
        getMethod = completionService.take().get();
        if (getMethod.getStatusCode() >= 300) {
          System.out.println("error");
          continue;
        }
        System.out.println("Got data: " +  getMethod.getResponseBodyAsString());
      } catch (ExecutionException e) {
         if (Throwables.getRootcause(e) instanceof SocketTimeoutException) {
           System.out.println("Timeout");
         }
      } finally {
        if (getMethod != null) {
          getMethod.releaseConnection();
        }
      }
    }

    • Constructor Detail

      • MultiGetRequest

        public MultiGetRequest​(Executor executor,
                       org.apache.commons.httpclient.HttpConnectionManager connectionManager)
        Parameters:
        executor - executor service to use for making parallel requests
        connectionManager - http connection manager to use.

    • Method Detail

      • execute

        public CompletionService<org.apache.commons.httpclient.methods.GetMethod> execute​(List<String> urls,
                                                                                  int timeoutMs)
        GET urls in parallel using the executor service.
        Parameters:
        urls - absolute URLs to GET
        timeoutMs - timeout in milliseconds for each GET request
        Returns:
        instance of CompletionService. Completion service will provide results as they arrive. The order is NOT same as the order of URLs