Honestly the hardest part of flying is the time and money. I feel guilty about how much time and money I've spent on just burning avgas. A one hour flight is half a day. Usually by myself. I used to fly several days a week, mostly by myself. Some people like to go a couple of times, but just riding around in the back of the plane gets boring, even when going to cool places. Time spent away from family. Then there's the money. I could have bought 500 acres and had a great hunting preserve. Instead I've got a log book.
I believe in no regrets. If flying fits into where you are in life, do it if you can. You don't want to be 10 years down the road regretting it.
A lot of people start and never see it through. Expect challenges and frustrations. Health, weather, maintenance, instructor schedule, family plans, etc. all work against you. Sometime it stretches out. The longer it takes, the more hours it will take. 60-80 hours is not unreasonable for a private when you have other things going on. Then another 10-15hrs for back country training.