Things of themselves cannot touch the soul at all. They have no entry to the soul, and cannot turn or move it. The soul alone turns and moves itself, making all externals presented to it cohere with the judgements it thinks worthy of itself.
With the passage from Book 5:19, Marcus will go on to repeat this same thought another 17 times throughout his Meditations (see end of post for complete list).
This idea, that all our troubles, fears and anxieties come from within - they originate from in our own head - is crucial to the philosophy of Stoicism. Pierre Hadot's commentary on Marcus Aurelius' Meditations is entitled The Inner Citadel and his chapter on the discipline of assent includes the same. The discipline of assent is nothing more than training your mind to be an inner citadel - a mental fortress that is strong against any attack and that the only way it could fall is from within.
Hadot fully explains (p. 106-107),
In order to understand what Marcus Aurelius means when he says that things cannot touch the soul and are outside of us, we must bear in mind that the word "soul" could have two meanings for the Stoics. In the first place, it was a reality made of air (pneuma) which animates our body and receives the impressions, or phantasiai, from exterior objects. This is often what Marcus means by "soul." Here, however, when he speaks about "us" and about the soul, he is thinking of that superior or guiding part of the soul which the Stoics called the hegemonikon. It alone is free, because it alone can give or refuse its assent to that inner discourse which enunciates what the object is which is represented by a given phantasia. This borderline which objects cannot cross, this inviolable stronghold of freedom, is the limit of what I shall refer to as the "inner citadel." Things cannot penetrate into this citadel: that is, they cannot produce the discourse which we develop about things, or the interpretation which we give of the world and its events. As Marcus says, the things outside of us "stay still"; they "do not come to us"; rather, in a way, "it is we who go toward them" (XI, II).
(see also Citadel p. 41, 105, 108)
from page 41 in The Inner Citadel:
the dogma according to which our troubles come only from our judgments, and that things do not penetrate within us (IV, 3, 10), recurs eighteen times in the course of the Meditations, sometimes repeated almost word for word, and sometimes in slightly different form (V, 19; VI, 52; VII, 2; VIII, 47; IX, 13; IX, 15; XI, 11; XI, 16; XII, 22; XII, 25; IV, 7; IV, 39, 2; V, 2; VII, 14; VII, 16; VIII, 29; VIII, 40; VIII, 49).