If I fly under FL only, when flying from airport A to airport B, when do I need to change QNH settings to airport B?

If you fly under the published Flight Levels, you are flying below the Transition Level. Then the altimeter setting you use is QNH, which is the most common altimeter setting for VFR flights. You can adjust the QNH based on the airports you overfly. If the flight is over a very large distance the QNH could change significantly. It is essential to check the QNH at the airport you are overflying, or flying close to, and use this QNH setting. Here is a task for you. Pick two airports close to you and compare their QNH values reported.