In an archery contest, the "upshot" is the shot that decides the match, so Sir Toby is saying that he cannot continue this practical joke on Malvolio to its conclusion. But it's hard to see just what that conclusion would be. Malvolio is already considered mad and confined to a dark room. What could be worse?
Tudor boy archer by Robert Peake, 1596