I have seen people struggle learning to fly, no doubt there are some people who should not. Both young and old, oldest I've seen start from scratch was 65, he had a battle but struggled through and used to fly his own Auster. I know another 65 year old who struggled through, bought himself an aircraft with his retirement money then died in that aircraft shortly there after. You can still enjoy aviation without actually flying. I found models great fun, drones are one way of experiencing the 3D world without leaving the ground. I have only flown with a tablet, it is an amazing way of looking at the world. I'm sure the imersive world of 3D goggles would transform the experience.
Everybody is different, not everybody is suited to piloting an aircraft. Ultimately it's something you have to decide for yourself, if others who have an idea of such things suggest maybe it might not be for you then to be true to yourself you should take that into consideration when making your mind up.
Remember the "Grass is always greener on the other side", piloting an aircraft is not the be all and end all of the universe.