Not too young. An experienced teacher advised me, when they are young, to simply follow their lead, let them play or tell stories, and show them things little by little, depending where their head is at.
Kirsten, use a variety of tools at this age to fill a 45-minute lesson: you walk with them saying step, step, stand and show the symbols of quarters and half-notes to represent these; later run, run to represent eighth notes. Clap a simple rhythm with long and short elements, see if they can match yours. Match a steady metronome beat, or twice as fast, or in-between. Can they hum the same pitch as you hum?