Maybe your kerf on the tool’s too big or if you are using a lead in/lead out they are too big for the hole. I made a tool for this so it just blasts the hole and doesn’t move and have all those holes with that tool on it’s own tool path
Use no Pierce Clearance (it’s added to the Lead In) and a small radius and lead in. Also, if you specify the Sweep Angle to 90 degrees it will use minimum space.
If you don’t mind a small divot in the edge of the hole you can also use no Lead In or Lead Out and turn Pierce Clearance off.