It would suggest the grinder is the issue but I'd do some tests with machine to rule that out. Try simply running water through the machine and timing the blank shot, both first thing and after you have made a coffee. Also make 3-4 cups of coffees in a row without adjusting the grinder and see if the shot times still vary, or if they level out. It could be that you are adjusting the grinder back and forth with a slight delay in the shot times due to a lag in changing the grind to seeing a difference in the shot.
Might also be worth giving the grinder a good clean out, as it can build up with oil from the beans. You can buy special "stuff" that you run through the grinder but just giving it a good brush out and washing the hopper out with detergent could help.
I find my SB grinder can hold a fair bit of ground coffee in the spout part after grinding and that I need to give it a good tap to get it to fall out. You might be getting a whole lot of yesterdays ground coffee coming down in the first shot, which could be quite stale and thus affect the extraction.