2. Soy is a phytoestrogen. While not increasing estrogen or decreasing a man's testosterone, it mimics estrogen in the body. Too much of this will cause men to grow 'man boobs' and gain visceral fat easily. Eating foods high in indole-3 carbinol such as brocolli and avocado can help your body 'ignore' the effects of these phytoestrogens.