Baiae is a luxury resort, for lack of a better description, with its fair share of hedonism. Because of the rampant debauchery, Seneca was not impressed with it. As he writes,
I left it the day after I reached it; for Baiae is a place to be avoided, because, though it has certain natural advantages, luxury has claimed it for her own exclusive resort.
He thinks places such as Baiae should not necessarily be shunned, but rather "the wise man or he who is on the way toward wisdom will avoid as foreign to good morals."
The wise man will prefer
to select abodes which are wholesome not only for the body but also for the character. ... We ought to see to it that we flee to the greatest possible distance from provocations to vice. We should toughen our minds, and remove them far from the allurements of pleasure.
He later opines
We too have a war to wage, a type of warfare in which there is allowed no rest or furlough. To be conquered, in the first place, are pleasures, which, as you see, have carried off even the sternest characters. If a man has once understood how great is the task which he has entered upon, he will see that there must be no dainty or effeminate conduct.
He then shows disdain for steam and dry saunas by saying "perspiration should flow only after toil."
The attack on luxury and pleasure continues!
The soul is not to be pampered; surrendering to pleasure means also surrendering to pain, surrendering to toil, surrendering to poverty. Both ambition and anger will wish to have the same rights over me as pleasure, and I shall be torn asunder, or rather pulled to pieces, amid all these conflicting passions.
I had to read the above quote a few times to ensure I understood clearly. When he says the soul should not be pampered by surrendering to pleasure, I assume he means "giving yourself to pleasure; to seek it." And when he says "surrendering to pain ... toil ... poverty" I assume he means we avoid pain, toil and poverty.
In this whole vein of thought is the ideal golden mean. I do not think we have to live like monks and nuns and completely shun luxury resorts, nor do I think wise people or sages will necessarily seek them out and yearn to stay at such places. More specifically, lets suppose a practicing Stoic needs to travel to Las Vegas for a business meeting. He would not refuse and tell his boss that for moral reasons he can't step one foot in Las Vegas. But perhaps he would go and then exercise restraint in Sin City. He doesn't have to drink to excess, visit strip clubs or gamble, nor would he choose to.
Similarly, when it comes to pain, toil and poverty, the practicing Stoic may do painful and toilsome things because perhaps he fears them. In which case, he would chop wood or pile rocks (or push a big one up and down a hill for eternity) or do something that might cause him pain. While doing so, he would therefore see that there is nothing to fear in pain and toil. Similarly, he might live like a pauper for a week or month or longer, eating base foods and sleeping on the floor. At the same time, a practicing Stoic is not required to constantly live a strict monk-like life. He does not have to seek extreme pain or toil either. He only engages in such activities to see that there is nothing to fear or to avoid about them, if, in fact, Fate decided that his whole life would follow such patterns.
The goal of avoiding pleasures and embracing pain is to gain real freedom.
I have set freedom before my eyes; and I am striving for that reward. And what is freedom, you ask? It means not being a slave to any circumstance, to any constraint, to any chance; it means compelling Fortune to enter the lists on equal terms. And on the day when I know that I have the upper hand, her power will be naught.
If you treat yourself with more pain, toil and poverty and less pleasure, then you will be adapted for any lifestyle. But if you live a soft, luxurious life full of pleasure, you will not be ready nor adapted for harsh circumstances which Nature may send your way.
Animals whose hoofs are hardened on rough ground can travel any road; but when they are fattened on soft marshy meadows their hoofs are soon worn out. The bravest soldier comes from rock-ribbed regions; but the town-bred and the home-bred are sluggish in action. The hand which turns from the plough to the sword never objects to toil; but your sleek and well-dressed dandy quails at the first cloud of dust.
Seneca gives this parting advice to Lucilius.
Vice, Lucilius, is what I wish you to proceed against, without limit and without end. For it has neither limit nor end. If any vice rend your heart, cast it away from you; and if you cannot be rid of it in any other way, pluck out your heart also. Above all, drive pleasures from your sight.