I suspect this is caused by rounding errors. Often when clicking on a point the point moves, no matter how steady the mouse is held. (The user wants to select a point to use Ctrl + Arrow keys)
I verified that the mouse position had in fact not changed, as I had already inherited from the tool.
Solved by checking if the mouse has moved in overridden EndDrag() and then calling CancelOperation if it hasn't.
Worth looking at to see if it can be fixed?