Why doesn’t my profile picture or my friends profile pictures show in the e-mail?

I love the fact that you can have a profile picture that is sent with the e-mail. But every time I send a message out or someone sends a message to me, there is no picture. It is just the picture of the default white outline of a man with the gray background.  We are not friends on facebook so I understand why their or my facebook profile does not show but our outlook profile picture should show right? We are contact so that is not the problem... I tried everything I could think of so I thought you may know how to fix it. Thanks for your help!