If that is the finest your Breville Smart Grinder will grind, I'm sure you'll always be wondering if that is the reason - I know I would.
You could try updosing to compensate. Do you have scales that can measure in grams? if so, how many grams are you dosing to? add 2-3grams and see if things improve.
I'd say you've nailed it - there are shim kits available for the Breville Smart Grinder that will raise the lower burr closer to the top bur at all settings, meaning you will loose a little on the coarse end, and gain the same amount on the fine end (the fine end is where the god shots lie). Call Breville, ask for a shim kit and they will send you one in the mail.