To the MC, ling family is just a business partner, they provide him food and shelter while he cultivates. Interactions with the ling family is unneeded and uncalled for. He is going to the village when he can cultivate without the herb mountain help. By proving yourself to not be an idiot might also invite unnecessary problems.
He's already gathering hate from the engagement (the 2 Du kids), if those 2 Du kids know that he is not a fool, they might try to eliminate the MC at an earlier date to prevent Ling Yushi from loving the MC more.... He was left alone cause he is "an idiot" and Ling Yushi would probably not fall in love with an idiot, hence left alone.
Especially in this case, everyone has high hope for the MC to call down a thunder (Which is unfair for MC, seeing how he can't call it down whenever he wants yet), but he has failed to call down the thunder which causes disdain and disappointment to others. Without expectation, there would not be disappointment.