Package gov.usgs.earthquake.indexer

Class Indexer

java.lang.Object
gov.usgs.util.DefaultConfigurable
gov.usgs.earthquake.product.AbstractListener
gov.usgs.earthquake.distribution.DefaultNotificationListener
gov.usgs.earthquake.indexer.Indexer
All Implemented Interfaces:
NotificationIndexCleanup.Listener, NotificationListener, Configurable
public class Indexer extends DefaultNotificationListener
The indexer receives products from Distribution, and adds them to the EventIndex. This class provides the following configurable properties (in addition to those inherited from DefaultNotificationListener):
associator
An object that implements the Associator interface.
storage
An object that implements the ProductStorage interface.
index
An object that implements the ProductIndex interface.
modules
A comma delimited list of objects that implement the IndexerModule interface
listeners
A comma delimited list of objects that implement the IndexerListener interface

  • Field Details

    • TRUMP_PREFERRED_WEIGHT

      public static final long TRUMP_PREFERRED_WEIGHT
      Preferred weight for persistent trump.
      See Also:

    • TRUMP_PRODUCT_TYPE

      public static final String TRUMP_PRODUCT_TYPE
      Type for persistent trimp
      See Also:

    • PERSISTENT_TRUMP_PREFIX

      public static final String PERSISTENT_TRUMP_PREFIX
      Prefix for persistent trump
      See Also:

    • ASSOCIATOR_CONFIG_PROPERTY

      public static final String ASSOCIATOR_CONFIG_PROPERTY
      Property name to configure a custom associator.
      See Also:

    • ASSOCIATE_USING_CURRENT_PRODUCTS_PROPERTY

      public static final String ASSOCIATE_USING_CURRENT_PRODUCTS_PROPERTY
      Property to associate using current products
      See Also:

    • DEFAULT_ASSOCIATE_USING_CURRENT_PRODUCTS

      public static final String DEFAULT_ASSOCIATE_USING_CURRENT_PRODUCTS
      Default state for associate using current products
      See Also:

    • STORAGE_CONFIG_PROPERTY

      public static final String STORAGE_CONFIG_PROPERTY
      Property name to configure a custom storage.
      See Also:

    • STORAGE_DIRECTORY_CONFIG_PROPERTY

      public static final String STORAGE_DIRECTORY_CONFIG_PROPERTY
      Shortcut name to configure a file product storage.
      See Also:

    • INDEX_CONFIG_PROPERTY

      public static final String INDEX_CONFIG_PROPERTY
      Property name to configure a custom index.
      See Also:

    • INDEXFILE_CONFIG_PROPERTY

      public static final String INDEXFILE_CONFIG_PROPERTY
      Shortcut name to configure a sqlite index.
      See Also:

    • MODULES_CONFIG_PROPERTY

      public static final String MODULES_CONFIG_PROPERTY
      Property name to configure modules.
      See Also:

    • LISTENERS_CONFIG_PROPERTY

      public static final String LISTENERS_CONFIG_PROPERTY
      Property name to configure listeners.
      See Also:

    • LOCAL_REGIONS_PROPERTY

      public static final String LOCAL_REGIONS_PROPERTY
      Property name to configure local regions file.
      See Also:

    • DEFAULT_LOCAL_REGIONS

      public static final String DEFAULT_LOCAL_REGIONS
      Path to local regions file.
      See Also:

    • GEOSERVE_ENDPOINT_URL_PROPERTY

      public static final String GEOSERVE_ENDPOINT_URL_PROPERTY
      Property for geoserve endpoint
      See Also:

    • ENABLE_SEARCH_PROPERTY

      public static final String ENABLE_SEARCH_PROPERTY
      Property name to enable search socket.
      See Also:

    • SEARCH_PORT_PROPERTY

      public static final String SEARCH_PORT_PROPERTY
      Property name for search socket port.
      See Also:

    • SEARCH_THREADS_PROPERTY

      public static final String SEARCH_THREADS_PROPERTY
      Property name for search socket thread pool size.
      See Also:

    • DEFAULT_SEARCH_PORT

      public static final String DEFAULT_SEARCH_PORT
      Default port where search socket listens.
      See Also:

    • DEFAULT_SEARCH_THREADS

      public static final String DEFAULT_SEARCH_THREADS
      Number of threads (concurrent searches) allowed.
      See Also:

    • INDEX_ARCHIVE_INTERVAL_PROPERTY

      public static final String INDEX_ARCHIVE_INTERVAL_PROPERTY
      Configurable property for index archive internal
      See Also:

    • INDEX_ARCHIVE_POLICY_PROPERTY

      public static final String INDEX_ARCHIVE_POLICY_PROPERTY
      Configurable property for index archive policy
      See Also:

  • Constructor Details

    • Indexer

      public Indexer() throws Exception
      Default no-arg constructor. This gets called from the Configurable API. All configuration parameters are set in the "configure" method.
      Throws:
      Exception - If the JDBCProductIndex throws an exception.

  • Method Details

    • getAssociator

      public Associator getAssociator()
      Returns the current associator used to associate products to one-another and products to events.
      Returns:
      The current Associator.

    • setAssociator

      public void setAssociator(Associator associator)
      Sets the given associator as the current associator to associate products to one-another and products to events.
      Parameters:
      associator - The associator to use from this point forward.

    • getProductStorage

      public ProductStorage getProductStorage()
      Returns the product storage component that is used to store products as they are received.
      Returns:
      The current product storage component.

    • setProductStorage

      public void setProductStorage(ProductStorage productStorage)
      Sets the current product storage component used to store products as they are received.
      Parameters:
      productStorage - The product storage component to use from this point forward.

    • getProductIndex

      public ProductIndex getProductIndex()
      Returns the product index component used to index product information as it is received.
      Returns:
      The current product index component.

    • setProductIndex

      public void setProductIndex(ProductIndex productIndex)
      Sets the product index component used to index product information as it is received.
      Parameters:
      productIndex - The product index component to use from this point forward.

    • addModule

      public void addModule(IndexerModule toAdd)
      Adds the give indexer module to the current list of modules used by the indexer to handle products.
      Parameters:
      toAdd - The IndexerModule to add to our list.

    • removeModule

      public void removeModule(IndexerModule toRemove)
      Removes the first occurrence of the given indexer module from the current list of known modules.
      Parameters:
      toRemove - The module to remove.
      See Also:

    • getModule

      protected IndexerModule getModule(Product product)
      This method checks each module's support level for the given product, returning the first module with the highest support level.
      Parameters:
      product - the product to summarize.
      Returns:
      module best suited to summarize product.

    • addListener

      public void addListener(IndexerListener toAdd)
      Adds a listener to this indexer. Listeners are notified when an event is added, updated, or deleted, or when a new product arrives and is un-associated to an event.
      Parameters:
      toAdd - The IndexerListener to add

    • removeListener

      public void removeListener(IndexerListener toRemove)
      Removes a listener from this indexer.Listeners are notified when an event is added, updated, or deleted, or when a new product arrives and is un-associated to an event.
      Parameters:
      toRemove - The IndexerListener to remove

    • notifyListeners

      protected void notifyListeners(IndexerEvent event)
      Send an indexer event to all registered IndexerListeners. Creates a NotificationEvent, with a reference to this object and calls each notificationListeners onNotification method in separate threads. This method usually returns before registered NotificationListeners have completed processing a notification.
      Parameters:
      event - The event that occurred to trigger the notification. Note: An IndexerEvent has a specific "type" to clarify the type of event that occurred.

    • hasProductBeenIndexed

      protected boolean hasProductBeenIndexed(ProductId id)
      Check whether this product is in the index. NOT synchronized to allow multiple threads to access. readProductIndex.hasProduct is synchronized.
      Parameters:
      id - ProductId to check
      Returns:
      true if product has already been indexed.

    • accept

      public boolean accept(ProductId id)
      Override the DefaultNotificationListener accept method, to always process products that may affect event association.
      Overrides:
      accept in class AbstractListener
      Parameters:
      id - the product id to check.
      Returns:
      boolean whether the product should be indexed.

    • onBeforeProcessNotification

      protected boolean onBeforeProcessNotification(Notification notification) throws Exception
      Check whether to skip products that have already been indexed.
      Overrides:
      onBeforeProcessNotification in class DefaultNotificationListener
      Parameters:
      notification - notification about to be processed.
      Returns:
      true to process the notification, false to skip
      Throws:
      Exception - if error occurs

    • onProduct

      public void onProduct(Product product) throws Exception
      This method receives a product from Product Distribution and adds it to the index. Implementation follows from Product Indexer Diagram (pg.10) of ProductIndexer.pdf document dated 09/09/2010. Calls onProduct(product, false), which will not reprocess already processed products.
      Overrides:
      onProduct in class DefaultNotificationListener
      Parameters:
      product - The product triggering the event.
      Throws:
      Exception - if an exception occurs.

    • onProduct

      public void onProduct(Product product, boolean force) throws Exception
      Receive a product and add it to the index. Optionally, reprocessing a product that has already been processed.
      Parameters:
      product - The product triggering the event.
      force - Whether to reprocess products that have already been processed (true), or skip (false).
      Throws:
      Exception - if error occurs

    • storeProduct

      public Product storeProduct(Product product, boolean force) throws Exception
      Stores a product
      Parameters:
      product - Product to store
      force - if should skip already indexed check
      Returns:
      Product if stored, null if not
      Throws:
      Exception - if error occurs

    • summarizeProduct

      public ProductSummary summarizeProduct(Product product) throws Exception
      Use modules to summarize product.
      Parameters:
      product - To summarize
      Returns:
      A product summary
      Throws:
      Exception - if error occurs

    • indexProduct

      protected ProductSummary indexProduct(ProductSummary productSummary) throws Exception
      Add product summary to product index.
      Parameters:
      productSummary - to add
      Returns:
      Summary added to index
      Throws:
      Exception - if error occurs

    • getTrumpedProductId

      protected ProductId getTrumpedProductId(ProductSummary trumpSummary)
      Get the productId referred to by a trump product.
      Parameters:
      trumpSummary - trump product with reference to product id.
      Returns:
      product id, or null if unable to parse product id.

    • getProductSummaryById

      protected ProductSummary getProductSummaryById(ProductId id) throws Exception
      Get a product summary object using its product id.
      Parameters:
      id - id to find.
      Returns:
      matching product summary or null.
      Throws:
      Exception - if error occurs

    • setSummaryWeight

      protected Event setSummaryWeight(Event event, ProductSummary summary, Long preferredWeight) throws Exception
      Update a product summary weight
      Parameters:
      event - the event.
      summary - the summary.
      preferredWeight - the weight to set.
      Returns:
      event with updated summary.
      Throws:
      Exception - if error occurs

    • resummarizeProduct

      protected Event resummarizeProduct(Event event, ProductSummary summary) throws Exception
      Resummarize a product within an event.
      Parameters:
      event - the event.
      summary - the summary.
      Returns:
      event with updated summary.
      Throws:
      Exception - if error occurs

    • checkForEventSplits

      protected List<IndexerChange> checkForEventSplits(ProductSummary summary, Event originalEvent, Event updatedEvent) throws Exception
      Check for event splits (and split them if needed).
      Parameters:
      summary - the summary the indexer is currently processing.
      originalEvent - the event before the indexer made any changes.
      updatedEvent - the event after the indexer made any changes.
      Returns:
      List of changes made during this method.
      Throws:
      Exception - if error occurs

    • splitEvents

      protected Event splitEvents(Event root, Event leaf) throws Exception
      Removes the leaf event (and all its products) from the root event. This method modifies the runtime objects as well as updating the index DB.
      Parameters:
      root - The root event from which all leaf products will be removed
      leaf - The event (with products) that will be removed from the root
      Returns:
      copy of root without the products that have been removed. The indexId property of leaf is updated to its new value.
      Throws:
      Exception - if error occurs

    • mergeEvents

      protected Event mergeEvents(Event target, Event child) throws Exception
      Merges the child event (and all its products) into the target event. If the child event attempts to merge in a product that is the same as one already associated to the target event, the child version of the product takes precedence. Note: This only applies when the target and child product have the same type, code, source, and update time; i.e. the products are duplicates. This method modifies the runtime objects as well as the index DB. The child event is then deleted.
      Parameters:
      target - The target event into which the child is merged.
      child - The child event to be merged into the target.
      Returns:
      the updated event
      Throws:
      Exception - if error occurs

    • checkForEventMerges

      protected List<IndexerChange> checkForEventMerges(ProductSummary summary, Event originalEvent, Event updatedEvent) throws Exception
      Check and merge any nearby events or previously unassociated products that now associate.
      Parameters:
      summary - the summary currently being processed by the indexer.
      originalEvent - the event before any changes.
      updatedEvent - the event after the summary was associated.
      Returns:
      list of any merge type changes.
      Throws:
      Exception - if error occurs

    • getPrevProductVersion

      protected ProductSummary getPrevProductVersion(ProductSummary summary) throws Exception
      Takes a summary return the previous
      Parameters:
      summary - A product summary
      Returns:
      The previous summary
      Throws:
      Exception - if error occurs

    • getPrevEvent

      protected Event getPrevEvent(ProductSummary summary) throws Exception
      Associate products are processed during checkForEventMerges(ProductSummary, Event, Event) and are ignored during this method.
      Parameters:
      summary - ProductSummary
      Returns:
      Event to which a productSummary is associated, or null if not found.
      Throws:
      Exception - if error occurs
      See Also:

    • getPrevEvent

      protected Event getPrevEvent(ProductSummary summary, boolean associating) throws Exception
      Find an existing event that summary should associate with.
      Parameters:
      summary - the previous event.
      associating - whether associating (vs archiving).
      Returns:
      previous event, or null if none found.
      Throws:
      Exception - if error occurs

    • configure

      public void configure(Config config) throws Exception
      Loads parent, specific, and dependent configurations; in that order.
      Specified by:
      configure in interface Configurable
      Overrides:
      configure in class DefaultNotificationListener
      Parameters:
      config - the Config object with settings.
      Throws:
      Exception - if configuration exceptions occur.

    • shutdown

      public void shutdown() throws Exception
      Shuts down the Indexer. The parent shutdown method is called and then all executor services (from listeners) are shutdown in sequence.
      Specified by:
      shutdown in interface Configurable
      Overrides:
      shutdown in class DefaultNotificationListener
      Throws:
      Exception - if exceptions occur while starting.

    • startup

      public void startup() throws Exception
      Starts up the necessary parent, specific, and dependent processes, in that order.
      Specified by:
      startup in interface Configurable
      Overrides:
      startup in class DefaultNotificationListener
      Throws:
      Exception - if exceptions occur while starting.

    • purgeExpiredProducts

      public int[] purgeExpiredProducts() throws Exception
      Checks the index for content that match a configured archive policy. Events are checked first and matched events are removed along with all their products. Listeners are notified of each archived event with an EVENT_ARCHIVED type. Unassociated products are checked next, matched unassociated products are archived and listeners are notified with PRODUCT_ARCHIVE type. Note: Product "age" is determined by when the earthquake for that product occurred and does not reflect how long the product has actually been in the index.
      Returns:
      Int array of size 2
      Throws:
      Exception - if error occurs
      See Also:
      • archivePolicies

    • removeEvent

      protected void removeEvent(Event event) throws Exception
      Removes the given event from the Indexer ProductIndex and ProductStorage.
      Parameters:
      event - event to remove
      Throws:
      Exception - If errors occur while removing the event

    • removeSummary

      protected void removeSummary(ProductSummary summary) throws Exception
      Removes the given summary from the Indexer ProductIndex and ProductStorage.
      Parameters:
      summary - to remove
      Throws:
      Exception - If errors occur while removing the summary

    • search

      public SearchResponse search(SearchRequest request) throws Exception
      Search for products in this index.
      Parameters:
      request - the search request.
      Returns:
      the search response.
      Throws:
      Exception - if error occurs

    • isDisableArchive

      public boolean isDisableArchive()
      Returns:
      disableArchive

    • setDisableArchive

      public void setDisableArchive(boolean disableArchive)
      Parameters:
      disableArchive - boolean to set

    • getArchiveInterval

      public long getArchiveInterval()
      Returns:
      the archiveInterval

    • setArchiveInterval

      public void setArchiveInterval(long archiveInterval)
      Parameters:
      archiveInterval - the archiveInterval to set

    • getArchivePolicies

      public List<ArchivePolicy> getArchivePolicies()
      Returns:
      the archivePolicies