I don't think that will hurt them. More than likely they are transient, prob stopping by for food and moving on. Next year there will be more buck in place of the 5 they killed this year. May not be as big or mature, but then again they might. Takes more than 1 year to pattern dear and get a feel for the herd
Ever spit it out rather than swallow it?
Enjoy it while you can. One day you might be complaining about NOT getting random pop-ups in the morning.