Preferences: Behavior: The right side selects which formats use Calibre viewer.
If there is no tick, The handling drops to the OS (default for that format)
So in fact you only need 1 Shortcut key.
For all else, there is the Open With PLUGIN (by Kiwidude

): Add as many custom viewer choices as you wish