Honestly, I don't think I've ever felt guilt about what I ate, unless I knew it was the last of the leftovers or someone else's chocolate.

My "I really shouldn'ts" are more likely related to waste (I know I've got something at home), amount (if i get that main, I won't have space for desert) or cost (I really, really should not visit that cheese stall because I always spend more than I intend to). "I really shouldn't" isn't necessarily related to how we look to others. I really shouldn't isn't necessarily a (self)judgment, but can be a statement of practicality.

But like you, I grew up with food as something communal and joyful. We like to eat, we like to share food, meals are a family thing. I bake when stressed, i like giving food, I go to my parents with homemade marmalade and come back with crab stock.