If the following information is incorrect, someone please correct me, I have only learnt this stuff in the last couple of weeks from an ATC perspective, and I am trying to apply to the situation what I can remember without re looking up the regs (ive been having to learn alot in a short period of time lately so I may have some figures mixed up.)
A change of 1 hectopascal is equal to roughly 30 feet. Therefore changing your QNH by 1 hpa will cause you to be about 30 ft out, if you are out by 5 hpa then you will be misreading by 150ft, which can obviously cause an issue. You are correct in stating that you should be looking outside when VFR, especially during landing, but it can effect things like the altitude you fly your circuit at, which will affect other aircraft in the circuit as well obviously due to altitude differences (say if call joining downwind they will expect you to be at 1000ft, not 850ft or 1150ft and they may not spot you) . If you are talking to atc, and giving them your level, or are cruising at a cruising level then if your QNH is set wrong you wont be at the correct altitude which is also bad :P In saying all that, you have to bare in mind though that a VFR altimeter is only legally required to misread by + or - 100ft, so you cant always guarentee even with the right QNH your altimeter will read accurately anyway.
Its alot more relevent if you are IFR though (Im assuming your not, but for informations sake I will say this anyway) where your altimeter really does need to be correct due to things like your lowest safe altitudes, cruising levels, traffic avoidance, approaches in IMC conditions etc etc. Due to this IFR altimeters have to be more accurate and are only allowed to misread by + or - 60 ft (obviously you would want as close as possible, but they are the legal limits).