I have an ArcGIS Portal setup with IWA or SAML SSO authentication. Is it possible to use the VSP Engine Tools post installer in this scenario?

Yes, you can install the tools when using authentication via Windows Active Directory and IWA (Integrated Windows Authentication) enabled in IIS for your ArcGIS Web Adaptor, or if using other SSO authentication frameworks such as SAML.

In this case you usually do not need to provide credentials during the post installation process if you are already signed-in and your environment is properly configured. For example, you will need to have your installation target site URL configured within the trusted site zone in the internet options security settings dialog along with user authentication logon settings like 'Automatic logon with current user name and password'.