raemin often a magnetometer with calibration or offset problems will due well in some directions and not others.
That is, if it correctly reads north when the quad is facing north.. that's no guarantee it reads east when the quad is facing east. And if it's off by 10-15 degrees in its orientation with however it's facing, that's enough to "orbit" or toilet bowl.
errorDecay and braking time constant are two parameters inside the loiter code; basically loiter transitions from position hold, to a "leash mode" where the quad follows a desired position, to quasi-attitude mode smoothly as you move the stick more. It's necessary to control how much work the position control system does in each of these states, and how quickly the position control system takes back over after you release the stick.