Lower expectations is a great tip.
I find that the more a group does things, the more everyone chills out. It's like the expectations come from a fear of being judged and from uncertainty. When everyone has information from the last ten events then you don't need to stress anymore, because everyone knows how this one will go and they've all judged one another already.
It helps to remember that you are competing with: no event.
If there are other parties happening and you're trying to make a better one, by all means, go all out. But mostly people in their 40s aren't going to many house events, so they're just happy to be somewhere with people. They don't care that you didn't decorate or sweep the floor or prepare an elaborate meal. You made soup and they're thrilled.
I'd also add that first-event nerves (on host and attendee sides) can be an uncertainty problem. No one wants to misunderstand the dress code, social code, etc. Once people have been together, there are now group norms that assuage that (aka "I know what's acceptable to wear and talk about").