It sounds like you're embedding Application Configuration settings somewhere that they shouldn't be.
The solution is to move these configuration settings to an external configuration file and load your settings dynamically at runtime. An external configuration file will not form part of the signed assembly and therefore won't ruin your signed distribution.
It is worth noting that the .NET System.Configuration classes do not work very well with assemblies and especially badly for VSTO assemblies (see multiple problems reported here on SO: 594298). As such I would recommend that you would either roll your own configuration class or read up heavily on the ConfigurationManager.OpenMapperExeConfiguration method to override the configuration file being targeted.
Additionally If you're currently using ClickOnce then this project sounds like it would be better suited to deployment via a Windows Installer Package.