There's a feature called cubic deadbands on the PID controller that was implemented a while ago and doesn't have an UI. Might want to give that a try, maybe it'll subdue the shakes without relinquishing complete control. You'll find it in the UAVO browser under StabilizationSettings. In DeadbandWidth, you define how wide the deadband is, and in DeadbandSlope, you define the slope of the deadband, a flat slope (0%) means it doesn't do anything in the deadband, 50% means you get half the response. The transition between the deadband and the active region is smoothed.
There was a graph on Desmos that illustrated it, but I can't remember the URL.