Skip to main content

Documentation for multi-server setup

Permanently deleted user

I've looked in the knowledge base, forums and install doc's but can't find any detailed information on how the multi-server configuration for Essentials is supposed to work.

I've registered a second server with Application Services on our first server and that appears to have worked because if I log into the REST Manager on server2 I get the list of sites from server1.  However, when I try to load a site in REST Manager from server2 I get an error after a very long wait.  If I try to view the REST endpoint for any sites I get an error "the site with ID "test" failed to initialize.  Refer to the logs for more details".  The only error ln the system log file states "could not create site test\site.xml: System.security.SecurityException: Forbidden ---> System.Net.WebException: The remote server returned an error: (403) Forbidden.

I'm assuming that there must be some other folders that need to be shared or access permissions granted through IIS, but I don't see anything in the documenation.  There also doesn't seem to be anything available on how the sharing of sites between servers is designed to work.  Does all administration happen on the Application Services server?  Does the App Services server load-balance requests between registered servers (like ArcGIS SOM balances work across SOC's)?  Can multiple Essentials servers be hidden behind a hardware load-balancer (whether or not they are registered with the App Services server)?

Thanks for any input.

Peter.

0

Comments

3 comments

Date Votes
  • Permanently deleted user

    Hello,

    I'm not sure about the reason for the Security exception here -- our documentation (make sure you have the latest version) details all the setup that is needed for multiple servers on a domain in a generally stock configuration. There are no extra folders to share or additional file permissions to set, given that the post-installer was able to set the required file permissions successfully on each server when it ran initially.

    However, it is always possible that additional domain security policies have been applied in your case that prevent the stock installation from working as it should. These policies can vary quite a bit from organization to organization, so it is difficult to address all possible situations in our documentation. If you would like us to take a look in person, please respond to the email sent to you from Support and we will set something up.

    However, it is recommended that in a distributed install you only administer your sites from the server where they actually reside. This is the main Application Services server (Server 1). There is no benefit to administering them from another server, and this causes extra network overhead, as the files must be written back to the main server anyway. It is important to realize that the distributed installation is *not* a fail-over solution -- there is still a single place where all configuration files are stored, and the main Application Services server represents a single point of failure, just as in a single server installation. Only the processing load is distributed between the multiple servers.

    Given this, if you do desire to place multiple installations of Essentials behind a load balancer in a fail-over setup it will require some manual intervention. The contents of the 'Sites' folder on each machine must be kept synchronized at all times, so that a new machine can be brought up as the main Application Services server (run the post-installer again on each machine and point it at the new server) and still have access to all of your configuration. The network overhead of doing this automatically is the main thing that prevents us from offering this service in a distributed installation at this time -- the servers would be spending all of their time copying files to each other, instead of serving user requests.

     

     

    0
  • Permanently deleted user

    Thanks Jonathan,

    I'm not sure what caused the errors either, but it finally seems to have resolved itself.  I was eventually able to see the sites through the REST endpoint and could edit / save a site from the second server (although it was quite slow compared to working on the app services server).

    Your reply answers a few of my questions but raises a couple more. 

     

     

    1) if we do not use a load balancer and just send all requests to the app services server, will it balance the load between all registered servers in the same way that ArcGIS's SOM will load balance (round robin?) to all SOC processes or does it only send requests to other servers when the app services server is under excessive load or does it actually load balance at all?  

     

     

    2) if we put a load balancer in front of the Essentials servers, are we better off running 'stand alone' servers, each acting as their own App Services Server? 

     

     

    3) Assuming servers are set up as in 2) will there be issues where requests from the client must be directed back to the same Essentials server each time?  We have encountered this problem with ArcGIS when using a callback function on the client - the first request gets routed to server A by the load-balancer and the client gets back a URL to a map image; the client makes a second request for the map image but the load-balancer sends this request to server B but the output file does not exist on that server.  Will this same type of situation occur when generating maps/reports with Essentials (synchronous and/or asynchronous)?  A synchronous print request gets back a URL in the JSON response - when the user clicks on that URL, I assume they must have to go back to the same server as the original request or the URL will be invalid, just like in the AGS scenario. 

     

     

    4)  Is it possible to specify where Essentials generates it's map/report output (to a UNC shared folder)?

    Is there any documentation on the 'ideal' setup for a multi-server, failover/load-balanced setup?  We need to make sure we do this right during initial implementation and don't have to go back after the fact to modify a live system.

    Thanks,

     

    Peter.
    0
  • Permanently deleted user

    Hi Peter,

    Here are some answers for you:

    1) Unlike the ArcGIS Server Manager (SOM), the Application Services component of the REST Elements is "behind" the bulk of processing rather than in front of it. So, it's responsible for storing your sites and how they're secured, and won't delegate any processing to any of the machines connected to it.

    2) If you have a load balancer in front of multiple instances of the Essentials REST elements, then running individual standalone servers will provide a more robust configuration, at the expense of extra work required to keep all of the nodes synchronized. Furthermore, additional configuration will have to be made in order to ensure that any "stateful" requests (print jobs, generated reports, saved sessions) will not end up being routed back to the wrong server (as you describe in your question #3). You can either use a sticky-session load balancer, that will try to send requests from the same client to the same server every time, OR configure your Essentials instances to have additional addresses that will resolve to a unique name. Essentials will respond to either the shared name (via the load balancer) or its unique name (when you're picking up a print job).

    3) Printing will always result in a return trip from Essentials to your browser, whether synchronous or asynchronous. See #2 above. Any other type of operation that does not have an immediate result (reporting, printing) or produces an output file also does a return trip. This requires that either your browser always goes back to the same Essentials application (either through sticky sessions, unique URLs, etc) or that all Essentials applications share the same Application Services connection.

    4) File output is handled by an ASP.NET filter that's part of Essentials, so the notion of "output directory" and "output URL" found with ArcGIS Server doesn't necessarily apply. So, at this time it's not possible to specify a shared output path; however, if you have multiple Essentials application connecting to the same Application Services, they'll all be able to retrieve the same content when a PDF is requested.

    There is currently no white paper or “best practices” document on setting up Essentials in a high-availability, fault tolerant manner, primarily because there are so many variables and permutations that are environment specific.  However, there are some broad guidelines and real-world examples we strive to share on this forum when specific scenarios are presented to us.

    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post