The talk begins in 2005, with the unexpected discovery that massive early-type galaxies were extremely compact at redshifts z~2. It is now thought that the remarkable size growth of these galaxies from z=2 to z=0 is largely due to minor mergers; the size distribution of early-type galaxies at high redshift is arguably the most direct evidence of the importance of late mergers in the assembly of massive galaxies. Recently the attention has turned to the question how these compact cores of present-day ellipticals were formed, and what the properties were of their star forming ancestors. Using the 3D-HST survey, combined with extensive Keck spectroscopy, we have constructed a straightforward, self-consistent description of this core formation phase that differs conceptually from other recent studies. Our description has surprisingly wide applications: as part of the analysis, we show that star formation apparently leads to a universal relation between size growth and mass growth of the form d(log r) ~ 0.3 d(log M), and we derive a simple model for the growth of galaxies over the past ~10 billion years.