Looks more like lichen than a fungus. Whilst in itself it does no harm, it can harbour pests. If it annoys you then it is removable with a strong jet of water (not so strong that the bark is damaged though.)
jr43101, I am finding, eventually these trees do get effected by it. Probably as Palustris says, from secondary problems. I have watched healthy looking trees get more and more and then become week and begins to fall. I was thinking of spraying bleach water on them to get rid of some. it seems hopeless for the ones that already show effects.
Your tree probably has some other issue going on besides the lichen. I agree with Barb...eventually it will succumb to some other disease or insect attack and eventually die. The only trees I have that I see this on are diseased trees and look somewhat "off" or just not vigorously healthy. I have a sugar maple in the back yard that has the lichen growing on it and it has dead branches all over it and it is the last one to put out leaves in the Spring. this year I thought it was dead it took so long to emerge from dormancy. Eventually I will have to cut it down and sooner than later, which hurts since I got it from my Grams who has now passed on.
Actually, a lichen is not one organism, but a symbiosis between a fungus and algae or cyanobacteria (S?). So, the fact that your tree has a fungal infection is in itself cause for concern. I believe about 90% of all soil borne infections in trees are fungal. Fungus can kill trees, and often does. It's often symptomatic of a tree weakened some other way, for a fungal infection to take hold.