If a map service coming from ArcGIS Pro cannot be connected in WebOffice author standalone and a corresponding error message (Error: WebOffice has returned an invalid response) is displayed, further analysis should be done in the WebOffice log file.
If the following message is logged in the WebOffice log file, the following steps have to be performed:
java.lang.IllegalArgumentException: NaN is not a valid double value as per JSON specification. To override this behavior, use GsonBuilder.serializeSpecialFloatingPointValues() method.
According to the log message, the reason for the problem is that on at least one layer the service did not set the visibility range correctly in ArcGIS Pro. Scale numbers are expected, but no double values are returned.
At the beginning it should be analyzed which layer of the service is affected. To do this, open the corresponding service via the ArcGIS REST Services Directory and go to the All Layers and Tables section.
On this subpage all layers and tables are listed. Then simply search for the entry "NaN" using the browser search function (<STRG>+<F>). If this value "NaN" appears at a visibility range (Min Scale / Max Scale) at Labeling Info of a layer, then open the corresponding project in ArcGIS Pro and switch to the Labeling Properties at the layer. If no numerical values are set at this point in the Visibility Range area, this problem is being caused.
To fix the problem due to the "NaN" values, numerical values for the Visibility Range (maximum scale / minimum scale) must be set in the Labeling Info properties of the layer. The values <None> are therefore not allowed.
Afterwards the service has to be republished or overwritten and for the WebOffice project the resources have to be reloaded.
After this adjustment these messages should not appear in the log file anymore.