The RTKF is the Kalman filter that essentially models the torque and the bias terms, latter which is essentially an integral. Raising Q2 will use more of the sensor value to correct the model guess, but how far that's valid is anyone's guess, since rate is a linear term and torque a derivative based on actuator range. Raising Q3 speeds up how fast the bias/integral rises and falls, too much isn't always a good thing.
That's why eventually, having actual torque values would be interesting, i.e. taking realtime telemetry from the ESCs, remapping it to actuator range (probably based on hover throttle and some curve mapping), and then do an inverse mix to get roll, pitch and yaw torques.
Also, raising Q2 will introduce noise intro the torque term, if the gyro is noisy, which gets then amplified by the LQR.
But anyway, as you see, there's tons of parameters to mess around with, so it's a surprise it works largely with the average powered quads. It was quite a pain in the ass to get a good set to begin with
--edit: That ratcheting you see in ActuatorDesired, specifically the second to last image, is the LQR trying too hard. The interval should match the command rate of your remote. That's why the Q1 and Q2 values were that low as they're shipping as default. If you raise Q1/Q2 or drop R, this will get worse.
--edit: You could also try eliminating the ratcheting another way. We have control smoothing implemented. It's ManualControlSmoothing in the StabilizationSettings UAVO. Normal blunts the edges using some predictive action, whereas Linear is plain linear interpolation. The duty cycle controls over what period of the control interval for both options. I should probably finally expose that option in the UI.