try slowing your bit speed down, & use a backer behind your work piece for starters.
If you're using a drill press, try not to drill w/as much downward pressure. Let the drill bit do the work, & you be the speed. The harder you push down as you're drilling, you'll have the problems you are.