Okay, then you need to become a Master Of Time.
Use your superpowers to view everything in slow motion. (Yes, this sound stupid, but bear with us....)
Before you throw the ball the final time, you need to have made sure you caught it in a good/easy position to throw from. So try to work out when you last had control over the ball, and whether at that point you can try to control the hole position. If so, all good. But if not, you have another chance when you catch the ball. There should be a split-second of time before you throw it again where you are holding it. In this microsecond, you can gently twist your grip, or your hand position so that the hole is visible, or aligned forwards-backwards. Then you can have a little bit more control over the throw.
Also, think about how much/little "wrist flick" you are giving on the throw. Maybe you need more, maybe less. Experiment.
All hard stuff, but hopefully that gives you something to think about.