No need to seive them out Alan :)
I find that this adds to the character of the cup. I actually go in search of unscreened beans for this particular reason. They roast unevenly but the result is akin to roasting the same been 3 or 4 different ways then blending them.
If anything....remove a few of the lighter beans once roasted and likewise the dark ones.
Broken beans come from either the processing or from the bottom of the bag from which they were packed in. There is an industry exceptable % of blacks and brokens in the green coffee industry.
Nescafe make most of there coffee from brokens and chips.
Its not going to kill you Alan :) Go for it and report back with your thoughts of the roast .