Prices are definitely high right now, but there is no telling when they might slow down.
The availability of an adjacent parcel is something that doesn’t come around often. You are very smart to be seriously considering purchasing it. The rarity of this opportunity is something to keep in mind.
I agree with some of the others that a 50% debt to income ratio on your primary residence is quite high and taking on any more debt should be done so very carefully. Because of this, I do not recommend you roll this new purchase into your existing mortgage. If you do that and fall behind on payments you could lose everything. If they are separate, you only lose the 5 acres.
Overall, I wish I would have taken some more chances with raw land in the past. Just have eyes wide open if you choose to pursue it.