Issue with quiet siri

So I hate having Siri talk to me when I hold down the button but now I have to tap the Siri floating thing at the bottom when prompted for input instead of just being able to respond like I would if she read the prompt to me. Is this normal for others? Is there a way around it?