Package gov.usgs.earthquake.aws

Class AwsBatchIndexer

java.lang.Object
gov.usgs.earthquake.aws.AwsBatchIndexer
All Implemented Interfaces:
Bootstrappable
public class AwsBatchIndexer extends Object implements Bootstrappable
Class to index a batch of products that have already been sent to the AWS hub. Reads a list of products to be indexed. Reads indexer from configuration file. For each product, fetch product information from the get_product AWS endpoint and call indexer.onProduct.

  • Field Details

    • FORCE_REINDEX_ARGUMENT

      public static final String FORCE_REINDEX_ARGUMENT
      Force reindex argument
      See Also:

    • GET_PRODUCT_URL_ARGUMENT

      public static final String GET_PRODUCT_URL_ARGUMENT
      Get product URL argument
      See Also:

    • INDEXER_CONFIG_NAME_ARGUMENT

      public static final String INDEXER_CONFIG_NAME_ARGUMENT
      Argument for indexer configuration name
      See Also:

    • INDEXER_CONFIG_NAME_DEFAULT

      public static final String INDEXER_CONFIG_NAME_DEFAULT
      Default indexer configuration name
      See Also:

    • DATABASE_DRIVER_ARGUMENT

      public static final String DATABASE_DRIVER_ARGUMENT
      Argument for database driver
      See Also:

    • DATABASE_URL_ARGUMENT

      public static final String DATABASE_URL_ARGUMENT
      Argument for database URL
      See Also:

    • INDEXER_DATABASE_ARGUMENT

      public static final String INDEXER_DATABASE_ARGUMENT
      Argument for indexer database
      See Also:

    • INDEXER_DATABASE_DEFAULT

      public static final String INDEXER_DATABASE_DEFAULT
      Default database for indexer
      See Also:

  • Constructor Details

    • AwsBatchIndexer

      public AwsBatchIndexer()

  • Method Details

    • run

      public void run(String[] args) throws Exception
      Description copied from interface: Bootstrappable
      Called by Bootstrap after processing the Configurable interface.
      Specified by:
      run in interface Bootstrappable
      Parameters:
      args - array of command line arguments.
      Throws:
      Exception - Exception

    • getProductUrl

      public URL getProductUrl(ProductId id) throws Exception
      Use getProductUrl template to generate URL. Replace "{source}", "{type}", "{code}", and "{updateTime}" placeholders.
      Parameters:
      id - which product.
      Returns:
      URL with placeholders replaced.
      Throws:
      Exception - Exception

    • getProduct

      public Product getProduct(ProductId id) throws Exception
      Get Product from endpoint.
      Parameters:
      id - which product.
      Returns:
      Product object.
      Throws:
      Exception - Exception

    • processProductId

      public void processProductId(ProductId id)
      Fetch and Index a product. Called from executor service to process product ids.
      Parameters:
      id - which product

    • readProductIdsFromDatabase

      public void readProductIdsFromDatabase(String driver, String url) throws Exception
      Read product ids (as urns) from database and submit to executor for processing.
      Parameters:
      driver - database driver
      url - database url
      Throws:
      Exception - exception

    • readProductIdsFromStdin

      public void readProductIdsFromStdin() throws Exception
      Read product ids (as urns) from stdin and submit to executor for processing.
      Throws:
      Exception - Exception

    • submitProductId

      public void submitProductId(ProductId id) throws InterruptedException
      Submit a product id to the executor service for processing. If queue is too large (500 ids), blocks until queue is smaller (100 ids).
      Parameters:
      id - which product
      Throws:
      InterruptedException - InterruptedException