Essentials Environments - Best practices

Overview:
This KBA will attempt to cover some of the best practices when setting up various Essentials environments (For example Dev, Test, and Production).  It was written with Essentials 4.6 in mind, but most of the principals should apply to older versions as well.  

Solution:

Pre-install considerations

Before installing, you should consider the following:

Q1) Do I need high availability?  If so, do I want high availability on all tiers?
Q2) Will anyone besides me be developing sites in Essentials?  Do I need to consider a change control process?
Q3) Will I be using Instant Search in any of my environments?  If so, do I have enough space?  Will my data be on SSDs or HDDs?
Q4) Do my sites need security?  If so, do I need to use Identity server, or can I use Windows integrated?  
Q5) Does my network contain a DMZ?  Where should I install Essentials?  

Once you know the answers to these questions, you are ready to proceed with your install.  It's difficult to determine what a best practice install is, since all environments and deployments are different, but here are a few recommendations based on some of the possible answers above:

A1) If you are planning on using high availability in Production, your test tier should also be a mirror of production, so you can see how the site will preform in a load balanced highly available environment.  Your development tier does not necessarily need to be highly available.  Make sure that you are using 3 or more nodes for Production in order to prevent split brain in the event of a network outage.  Do not cluster your tiers together, you should have an independent Dev, Test, and Prod cluster (so that if there is an issue with the Geocortex Core service on one of the tiers, there is no chance of it impacting the other tiers).

A2) If an issue is discovered in one of your production sites, you should have a documented and agreed upon way for you and/or your team to handle it.  If you are fixing the issue in your development instance, and your colleague is fixing the issue directly in production, you may find yourself overwriting each other's work.

A3) Instant search requires a lot of disk space and RAM to run effectively (See this page in our docs for specific recommendations).  You should review these requirements and make sure that you can meet them in any of the tiers that will be using Instant search.  To save resources, it is acceptable to use Global Search in development, and Instant search only in Test and Production (you can choose to use instant search only in Prod, but it's a good idea to test the functionality before pushing to Prod, if your resources allow you to run Instant search on both tiers).

A4) If you are planning on securing your sites, you will want to test the security in either Dev or Test before pushing to Prod.  If you are using Windows integrated security, the process should be pretty seamless, however if you are using an out-of-the-box identity server, you may find that your permissions become orphaned when you migrate from one server to another.  This is because the permissions are signed from one Identity server, and the signatures don't match when they land on the next tier.  You can edit the Identity server issuer seed in the Post installer on your servers to make the issuer seed identical, or you can edit the site.xml when you move it to the new server, and use a find and replace operation in a text editor to replace the old one with the new one.  

A5) Depending on your network architecture, you may need to be aware of where your server is placed on the network.  Some organizations have DMZs (demilitarized zone), which keep the internet facing servers separate from the internal servers.  Where you place the Essentials servers depends largely on the following items:

• Where is your ArcGIS server?  Can you reach the map services you need from the network segment you are on?
• If not, is your IT team willing to open ports, or create firewall rules so you can access the information that you need?  
• If you are splitting the tiers (for example, perhaps Prod is in the DMZ, but test and dev are internal) can you migrate the site.xml file from one server to another?

Best Practices for running a 3-tiered environment

1) Automate as much of the deployment as possible.  For example if you elect to use Identity server, and need to change the issuer seed in the site.xml file before moving it to the next tier, consider scripting the process, so that it executes the same way each time, and there is no chance of making a mistake, or missing a step.

2) Have a documented process of exactly what is involved when moving a site from one tier to the other.  Do not deviate from this process.  It will vary depending on the needs and policies of your organization.  

3) Discourage changes directly to the production environment.  In order to ensure that changes are not made in production, you can use the Programs and Features control panel (in Windows) to change the Essentials install, to completely remove the Essentials REST Manager component to prevent this from happening.

4) Try to do the bulk of your changes in development, and if possible, no changes in Test and Prod.  If for example, you make a change in test to tweak a setting, and then push it to prod, the next time you are making changes in dev, you may overwrite the change you made in test, and cause issues in prod (since testing the change you made last time was not necessarily part of this test plan).  A good rule of thumb is to push changes up the tiers, never down.  

5) Maintain version consistency across your environments.  Your Dev environment should be running the same version of everything (Windows, Essentials, Core, GVH, etc) as your production environment.

6) When upgrading any software, the order should be Dev > Test > Prod.

7) Test only one thing at a time.  For example, if you are testing a new workflow, don't do an upgrade, or push a viewer change at the same time.  It makes it more difficult to troubleshoot if you don't know which new change has caused your issue.  It's also a good idea to allow a few days between pushing changes before pushing new changes (to give the previous change a chance to "burn in").

8) Have a roll back plan for your production environment, whenever you make any changes.  

9) Have back ups in place before pushing changes to any of your environments.  This can be made part of your automated process of pushing changes from one tier to another.

10) If you have many sites, and make a lot of frequent changes, consider using a versioning system like Git or Subversion to store your sites, so you can keep track of the changes, and roll back if needed.