Perhaps your wheels are descentrated. When I use her off-road I have to centrate the front fork because it needs urgently a reinforcement. You can do it yourself pushing with your foot. If your handlebar is straight and the the front wheel not is time to get down and kick the fork(softly of course)