Everyone is originally convinced that he's some sort of doll, rather than a human being, and at some point his retainer, Matthias, realized that El might have been aware since infancy. BUT NOTHING HAPPENS! Nothing at all, no one ever questions how smart El is, why he was so emotionless, or how he can use ice magic.
I give her leeway on making Harutora promise to become her shikigami while they were children (she was taught this way by her family and probably expected Harutora to understand what she was asking of him since he should have been taught this as well). What I don't like is that fact that she created a shikigami to befriend Harutora, lived a double life with him, and used that shikigami's demise as a toll to bind Harutora to her. If it was just this I could be mollified but she's abusive on top of that. The storytelling plays it as humor but many of the scenarios don't warrant her to take such offense. Natsume constantly uses her status to order him around, ruins relationships she sees as a threat to her romantic pursuits, has beaten him because of misunderstandings (which is a trope I personally hate because these characters don't have the right), all the while establishing their relationship as childhood friends (sometimes just Onmyouji/shikigami) and then punishing him like a wronged girlfriend when he's in a situation she doesn't like.