My guess is that XCTU also uses the Energy Detect (ED) command. It runs for a specified period of time, 1s in our case, and them returns the "raw" data, the max energy for each channel. Not sure what kind of "filtering" XCTU can or would do. XCTU does show the raw data and it is very stable compared to mine.
Is there a different command that XCTU Spectrum Analyzer is using?
2.4 Pro DigiMesh, Latest FW