the only damage you'll do is if you're running the car at full tit all the time, however even that will be minimal. The turbo, if it is a leak in the boost circuit (pipe work feeding from the air filter, turbo, intercooler and inlet manifold) will just be boosting that little bit harder to achieve the same power. Drive it, but don't go racing
until you get it sorted.
If you're seeing oil vapour collecting it sounds as though your boost pipe may be split or perished, or the jubilee clip (if there is one?) could be loose.
It's sounding like a boost leak, much like my split intercooler, which I've had whilst I covered 1,200 miles this week.