Sadly modern kitchens with all of the hard surfaces bounce noise all over the place (most unpleasant) not a lot you can do about that.
The newer grinders will grind a shot in seconds.
I can still carry on a conversation whilst my Mazzer is running and my hearing ain't that flash.
No solution to your particular problem, just personal observations.
What's the old saying "you cant make an omelette without breaking eggs"