Background: Information about the domains of physical activity (PA) that are most prone to decline between late childhood (11 y), early adolescence (13 y), and mid-adolescence (15 y) may support more targeted health promotion strategies. This study explored longitudinal trends in nonorganized PA, organized PA, active transport and active chores/work between childhood and adolescence, and potential sociodemographic moderators of changes. Methods: Data were sourced from the Longitudinal Study of Australian Children (n = 4108). Participation in PA domains was extracted from youth time-use diaries. Potential moderators were sex, Indigenous status, language spoken at home, socioeconomic position, and geographical remoteness. Results: A large quadratic decline in nonorganized PA (−48 min/d, P < .001) was moderated by sex (β = 5.55, P = .047) and home language (β = 8.55, P = .047), with girls (−39 min/d) and those from a non-English speaking background (−46 min/d) declining more between 11 and 13 years. Active chores/work increased between 11 and 13 years (+4 min/d, P < .001) and then stabilized. Active transport increased among boys between 11 and 13 years (+6 min/d, P < .001) and then declined between 13 and 15 years (−4 min/d, P < .001). Organized PA remained stable. Conclusions: The longitudinal decline in PA participation may be lessened by targeting nonorganized PA between childhood and adolescence. Future interventions may target girls or those from non-English speaking backgrounds during this transition.