Issue with Instant Search

Permanently deleted user

June 10, 2015 06:10

Hi,

We're having an issue with Instant Search / elasticsearch wherein it is indexing only 40% of the data in a service. Initial scan of the service a few months ago reached 100% and this issue is very recent (last 2 weeks). I have tried multiple services and all have been indexed only to 40%. When I looked through the logs, I found an "UnavailableShardsException" error. For every 100 records, it fails to send 60 through and appears to be timing out and then the indexer moves on to the next 100 records, fails for 60 and so on.

Here is an example of the error:

[2015-06-10 07:23:26,351][ERROR][AMQP Connection amqp-0-9://SomeServer:5672][Indexer     ][GAS.Nutch.Indexer.Impl.ArcGisFeatureIndexer] 59 onprem:SomeServer:http_arcgis_rest_services_geocortex_addresssearch_mapserver_0:63099:0 - UnavailableShardsException[[onprem][0] [2] shardIt, [0] active : Timeout waiting for [1m], request: (org.elasticsearch.action.bulk.BulkShardRequest@650df828)

In the Search Console logs:

[2015-05-25 20:34:43,311][DEBUG][gateway.local            ] [SomeServer] [onprem][3]: allocating [[onprem][3], node[null], [P], s[UNASSIGNED]] to [[SomeServer][X37M2QcYQ9uUTNxnIDksiA][WDCGISAGS][inet[/ip:9300]]] on primary allocation

 

[2015-05-25 20:34:43,311][DEBUG][gateway.local            ] [SomeServer] [onprem][0]: not allocating, number_of_allocated_shards_found [0], required_number [1]

 

[2015-05-25 20:34:43,311][DEBUG][gateway.local            ] [SomeServer] [onprem][2]: not allocating, number_of_allocated_shards_found [0], required_number [1]

 

[2015-05-25 20:34:43,311][DEBUG][gateway.local            ] [SomeServer] [onprem][4]: allocating [[onprem][4], node[null], [P], s[UNASSIGNED]] to [[SomeServer][X37M2QcYQ9uUTNxnIDksiA][WDCGISAGS][inet[/ip:9300]]] on primary allocation

Which seems consitent with the earlier error.  I am having a look at our ports to see if that helps (elastic search seems to use quite a large range of ports in the transport module).

Does anyone have any ideas or insights that might fix this?

Cheers

Sulata

 

 

 

 

 

 

0

• 

  Permanently deleted user

  July 23, 2015 10:59

  Managed to find a work around for this issue. The failure of instant search in this case coincided with a windows update on the server, during which it appears to have deleted 3 of 5 shards, therefore 60% of the indexed data essentially had nowhere to write to. Copying the missing shards from another box running Geocortex was a desperate attempt to fix this without having to reinstall everything worked and indexing reached a 100%.

  -Sulata

  0

Please sign in to leave a comment.