Thank you for posting the screenshots.
So clearly there are some crest factor issues as you have a significant leading edge spike in the waveform.
In the digitizer mode, if you up range the voltage, what is the max voltage of the spike? On the full screen graph, you can zoom in and place cursors too.
I suspect that when using the Freq or VAC modes and low ranges, the spike is saturating the range.
But higher ranges that tolerate the spike, don’t have sensitivity for the much lower amplitude remaining signal.
Not sure the origin of the spike. Calm it down if you can (reduce inductance if the spike corresponds to current load changes).
With TSP and Digitize, you could extract the freq info from timestamps between two peaks for example.
Can your use case support idea of not scanning conventionally, but closing a channel and then calling a TSP function?