At zero-temperature, the rigidity of many amorphous systems can be determined by constraint counting, which compares the number of degrees of freedom of a system to the number of constraints. For instance, under-constrained systems are typically floppy. Examples are polymer networks and vertex models for biological tissues. However, these under-constrained systems can be rigidified by the application of external strain, and we recently developed a generic analytical theory for predicting the elastic material properties of such systems in the athermal (i.e. zero-temperature) limit. Here, we extend this theory to the finite-temperature regime close to the athermal transition point, where we show that all under-constrained systems behave in the same generic way. For instance, in the limit of infinitely stiff springs, where elasticity is purely entropic, isotropic tension t and shear modulus G scale with temperature T and isotropic strain e as ∼ T /|e|. Furthermore, we show that for finite spring stiffness, entropic and energetic rigidity interact like two springs in series. This also provides a simple explanation for the previously only numerically observed scaling t ∼ G ∼ T^(1/2) at e = 0. Our work unifies the physics of systems as diverse as polymer fibers & networks, membranes, and vertex models for biological tissues.