Sites spontaneously unloading from the essentials cache
We have recently begun to experience a high frequency of sites failing to load for clients. Simply reloading the site into the Essentials cache resolves the problem but I can't figure out what is causing the sites to flush from the essentials cache. Any ideas?
-Tim
-
I have also started to experience this as well on my sites. The other issue is sometimes a particular map service won't load either. Even though the REST endpoint is up and I can even see the service when logging into manager. I have been reloading the cache as well which works but it is definitely becoming an issue and is happening more frequently. I am currently on 3.10.0
0 -
I have a support call logged for the very same thing. Of our 4 or 5 main site, usually one goes down every day or two. Reloading the site in Rest Manager fixes it.
Chris
0 -
Have they suggested what the issue might be yet?
0 -
No, not yet. I've just started working with Malcolm, sending him the error msgs and log files. I'll let you know if I find out anything.
Chris
0 -
I should have posted our version number, we are at 3.8. If it would help we can send Malcolm our log files as well.
0 -
Totally grasping at straws here but had a thought on this matter and it might be worth sharing.
My colleague, Nizar, discovered while trying to troubleshoot this issue that the geometry service setting that we had setup for a site that has begun to fail quite often was getting the word "geometryserver" appended to the end of the setting any and every time someone used it (ie an identify). So my thought is could the essentials cache be going stale because of some unintended change to the site xml?
One of our sites just went down and before we hit reload we saved a copy of the site.xml so the next time it goes down we can search the files for differences and see if there are any. Probably way to late on a friday to be thinking of this stuff. ;-)
-Tim
0 -
Had a site cache fail twice since Friday, each time we saved a copy of the site xml and there is no difference at all between the copies so that eliminates my theory. Any news from Malcolm yet?
0 -
I'm not sure if my problem is related, but I'm on 3.10 and just in the past day, I can't get the rest manager to behave at all. I have trouble editing sites and when I do, i get spontaneously logged out (usually when i try to go into the viewer settings). Apps that were working yesterday, won't even load, thy just hang at the "Intializing components. Please wait..." screen.
0 -
We are also experiencing site hangs every couple of days. Site seems to hang with no map or at initializing screen. A site refresh restores it, but I dislike having to be this vigilant about monitoring. We were having the issue on 3.9 and now 3.10.1
0 -
Hi folks,
The Essentials cache works very much like a web browser's content cache. So, if clearing your cache resolves a problem it is likely that the problem is caused by an unexpected response from ArcGIS Server (which we are now caching!) We'd like to isolate the core cause of this problem so that it can be resolved or avoided.
When the sites are offline, are ALL sites offline or just one? Remember that if you have internet access you can always load up the Charlotte sample site, or create a new site based on our services (and/or an ESRI service). If some sites load but not all, we can likely eliminate process-level issues with IIS or ASP.NET. Conversely, if NO sites load we should be examining the IIS process first, consulting the server's Event logs, and addressing any architectural problems that may be interfering with Essentials' normal operation.
If a site is failing to load it should be putting information into the Essentials log file. This is normally located in:
C:\Program Files (x86)\Latitude Geographics\Geocortex Essentials\Default\REST Elements\REST\App_Data\Logs
Before clicking the Reload link to "fix" the site, click on the Site's endpoint link in REST Manager to try to launch the site. Do you get an error message or is it just hung? If you get an error message, we're especially interested to know what has gone into the SYSTEM log files (found above). If it is "just hanging" then try to let it time out to get an error message before refreshing the site. We really need to know what it happening, and the best source of this information will come from the affected computer itself. We are NOT seeing this sort of behavior on any of our test / production machines here, so it is important to collect as much information as you can for us!
Thanks,
-Malcolm
0 -
Malcolm,
We just had it hang at "Viewer Initialization Error". Attached is the log. (I forgot to check the REST endpoint). In our case it is not all sites at once, but one at a time at random.
My read from the log file is that it may be choking on a query from the Address Search workflow. I am attaching both the log and the workflow.
Thanks,
Jim
(https://support.geocortex.com/Data/Sites/1/userfiles/773/parcelsearchaddresscascade.xaml) /Data/Sites/1/userfiles/773/parcelsearchaddresscascade.xaml
(https://support.geocortex.com/Data/Sites/1/userfiles/773/system.2012-10-16---copy.zip) /Data/Sites/1/userfiles/773/system.2012-10-16---copy.zip
0 -
Hi Malcolm,
I have not had this problem for a few weeks but it reoccurred this morning after I used REST Manager to save a change to the site in question.
The browser displays an error that says:
Error
The site with ID "Premises" failed to initialize. Refer to the logs for more details.
The log entry says:
<Event Timestamp="2012-10-17T08:24:13.6742519-05:00" Level="ERROR" Identity="Super"><Message>Cannot serialize Site or SiteConfig: System.Threading.ThreadAbortException: Thread was being aborted.
at System.Threading.WaitHandle.WaitOneNative(SafeHandle waitableSafeHandle, UInt32 millisecondsTimeout, Boolean hasThreadAffinity, Boolean exitContext)
at System.Threading.WaitHandle.InternalWaitOne(SafeHandle waitableSafeHandle, Int64 millisecondsTimeout, Boolean hasThreadAffinity, Boolean exitContext)
at System.Threading.WaitHandle.WaitOne(Int32 millisecondsTimeout, Boolean exitContext)
at System.Threading.WaitHandle.WaitOne()
at Geocortex.Essentials.Configuration.Models.SiteConfig.ConfigureSite(Site site)
at Geocortex.Essentials.Configuration.Models.SiteConfig.ToBusinessObject()
at Geocortex.Essentials.Configuration.Caching.CachingSiteConfigServer.GetSite(SiteRef siteRef)</Message></Event>
<Event Timestamp="2012-10-17T08:24:13.7523814-05:00" Level="ERROR" Identity="Super"><Message>Could not obtain site from cache: System.Threading.ThreadAbortException: Thread was being aborted.
at Geocortex.Essentials.Configuration.Caching.CachingSiteConfigServer.GetSite(SiteRef siteRef)
at Geocortex.Essentials.Configuration.Caching.CachingSiteConfigServer.<>c__DisplayClass2`2.<MakeLazy>b__1(Boolean tryCreate)
at Geocortex.Essentials.Configuration.Caching.SiteRef.get_Site()
at Geocortex.Essentials.Configuration.Caching.CachingSiteConfigServer.GetSiteBuffer(SiteRef siteRef)
at Geocortex.Essentials.Configuration.Caching.CachingSiteConfigServer.<>c__DisplayClass2`2.<MakeLazy>b__1(Boolean tryCreate)
at Geocortex.Essentials.Configuration.Caching.SiteRef.get_SiteBuffer()
at Geocortex.Essentials.Configuration.Caching.CachingSiteConfigServer.GetSite(String id, Boolean clone)</Message></Event>
<Event Timestamp="2012-10-17T08:24:16.2994031-05:00" Level="ERROR" Identity="Super"><Message>The site with ID "Premises" failed to initialize. Refer to the logs for more details.</Message></Event>
<Event Timestamp="2012-10-17T08:24:55.9735632-05:00" Level="ERROR" Identity="Super"><Message>The site with ID "Premises" failed to initialize. Refer to the logs for more details.</Message></Event>
<Event Timestamp="2012-10-17T08:25:41.5386876-05:00" Level="ERROR" Identity="Super"><Message>Site with ID '' was not found.</Message></Event>Only one site was affected.
Steve
0 -
Hi Steve,
Both your logs and Jim's logs contain a thread abort exception, which is helpful to learn about.
In Jim's case, the thread abort appears to correspond with a broken workflow request - the viewer tried to send a workflow to the server but the JSON it sent did not terminate properly (the closing quotes and curly braces were gone). Is there anything prior to this thread abort that would indicate another problem?
Also, can you and Jim examine your IIS access logs around the time of these thread abort errors, to see what else IIS is doing? I'm curious to know if there are any further 500 errors being logged by IIS. If there are, we should enable IIS's exception / error logging so that it will create better output for us to examine.
Regards,
-Malcolm
0 -
Malcolm, the initial log I posted yesterday was the only GCE log from yesterday. Here is the IIS log from yesterday from right around the failure time, sorry they're verbose. (The GIS_External site was the failure.) I'm also including a GCE System log from a failure last week Thursday 10/11 that took place about 8:25 AM on GIS_Internal.
Thanks again.
Jim
(https://support.geocortex.com/Data/Sites/1/userfiles/773/iis_logs_10162012.zip) /Data/Sites/1/userfiles/773/iis_logs_10162012.zip
(https://support.geocortex.com/Data/Sites/1/userfiles/773/system.2012-10-11.zip) /Data/Sites/1/userfiles/773/system.2012-10-11.zip
0 -
I found the entries that correspond to the error IIS log. (I had to take the GMT offset into consideration.)
You can see where I received the message regarding the token expiration, which is where I re-entered the credentials and re-saved the site. That was the trigger for the error message when I tried to browse to the Geocortex endpoint.
(https://support.geocortex.com/Data/Sites/1/userfiles/1516/iislogpremiseserror.txt) /Data/Sites/1/userfiles/1516/iislogpremiseserror.txt
Steve
0
Du måste logga in om du vill lämna en kommentar.
Kommentarer
15 kommentarer