Nature gives all and takes all back. To her the man educated into humility says: 'Give what you will; take back what you will.' And he says this in no spirit of defiance, but simply as her loyal subject.
Marcus clearly notes what matters: truth and justice. It does not matter what others criticize. If it doesn't stand up to the test of truth and justice, it deserves to be criticized and 'brought to justice.' But if others are going to criticize truth and justice, then don't forget how petty these people can be - what useless things they pursue and care about. They only cheat themselves of virtue.
Truly to amor fati you must love what nature gives and takes.
(see also Citadel p. 121, 265, 217)