Given a (nice) group G, we are interested in how fast the length of a group element can grow when we apply powers of a given outer automorphism of G. If the group G is free or the fundamental group of a closed surface, classical train-track techniques give a complete and precise picture. This can be extended to automorphisms of all negatively curved (a.k.a. Gromov hyperbolic) groups G, using Rips-Sela theory and the canonical JSJ decomposition. Very little seems to be known beyond this setting. We study this problem for a broad class of non-positively curved groups: "special" groups in the Haglund-Wise sense. In this setting, we prove that: (1) the top exponential growth rate of any automorphism is an algebraic integer; (2) if the automorphism is untwisted, then it admits only finitely many growth rates, and each of these is polynomial-times-exponential.