Another SoundSource (and Rogue Amoeba) fan here, too 
It may be overkill for what you want, @matt08, but if you do choose to invest in it, I use a combination of Shortcuts and Keyboard Maestro macros to do just what you want (and more!)
Best wishes,
Steve