Rayleigh-Taylor and Convective Instability of Mantle Lithosphere
Insofar as mantle lithosphere would be more dense than asthenosphere if warmed to the same temperature, it is gravitationally unstable. Thus, if thickened mechanically by horizontal shortening between two more rigid plates or blocks of lithosphere, the instability should enhance that thickening and cause it to grow more rapidly
[e.g., Houseman et al., 1981]. We have been studying how various properties of layers sharing properties with lithosphere control growth rates of instability. This work has considered both Rayleigh-Taylor instability, for which a layer is intrinsically unstable because of its density, and convective instability, for which diffusion of heat can erase perturbations to the density structure due to lateral temperature differences. For cases relevant to the earth, diffusion of heat is slow and can be neglected, though one exception arises when chemical differences introduce differences in density that limit the role of temperature differences.
We found that when strain rates vary with a power of stress, not linearly with stress, so that viscosity is non-linear (or non-Newtonian), instead of exponential growth, perturbations to the thickness of a gravitationally unstable layer do not grow exponentially, as they do for Newtonian viscosity [Houseman and Molnar, 1997]. Instead, growth is very slow at first, but then grows rapidly, super-exponentially. If such a process occurs in the earth, perturbations might go undetected for millions (or tens of millions) of years, and then abruptly the lower lithosphere would sink as blobs and descend into the asthenosphere. Consideration of external forcing [Molnar et al., 1998] and diffusion of heat in convective instability [Conrad and Molnar, 1999] do not alter this super-exponential growth. Moreover, in regions where crust extends and thins, non- Newtonian viscosity enhances localization of the straining, and this localization might help account for intracontinental volcanism far from plate margins [Harig et al., 2010]
Because viscosity varies with temperature in the earth, simple rules for characterizing an unstable layer must be modified. We found that using the ratio of density difference to viscosity integrated over depth, an “effective buoyancy” of the layer, allows a scaling of temperature-dependent viscosity for Rayleigh-Taylor and convective instability [Conrad and Molnar, 1999].
To examine the effect of the top boundary condition, Harig et al  considered a free top, and hence with no shear stress, and exponentially decreasing viscosity with depth in the unstable layer. Our target was the Sierra Nevada, beneath which two high speed zones suggest downwelling plumes that are more widely spaced than might be expected for an unstable layer not free to move at its top. Harig et al.  showed that for a family of exponentially decreasing viscosity structures, wide spacing of downwelling plumes is possible.
In all cases our goal is to find scaling laws that allow realistic values of parameters to be inserted, and resulting dynamics to be compared with what we observe for the earth. For example, the range of exponential decay of viscosity with depth in the unstable layer studied by Harig et al.  overlaps, if only barely, the likely exponential decay in cold mantle lithosphere. Similarly, the radipidity with which mantle lithosphere seems to have been removed from beneath the Altiplano of the Andes and beneath the Sierra Nevada can be used to make field tests of laboratory-based flow laws of olivine, and in both cases they pass those tests [Molnar and Garzione, 2007; Molnar and Jones, 2004]
The inclusion of a light layer above an unstable layer, analogous to crust over mantle lithosphere, can profoundly alter the flow pattern when the composite layer is compressed horizontally and made to thicken. In 2000, Houseman, Neil, and Kohler showed that for some ratios of thickness of the crust to entire lithosphere, and for some ratios of density and viscosity in the two layers, blobs of thickened mantle lithosphere do not sink beneath the region where the crust is forced to thicken, but beneath the adjacent region. In an extension of this work, Molnar and Houseman  carried out numerical experiments to examine the full parameter range and constructed a simple theory based on linear stability to explain results. By analogy with the earth, for some crustal thicknesses, densities and average viscosities, downwelling flow could occur on the flanks an active mountain belt, not beneath it. Regions where mantle lithosphere has been removed, and crust forms a large fraction of the lithosphere might remain susceptible to further shortening because of thin lithosphere, but the convective instability associated with continues thickening might then preferentially remove mantle lithosphere from the regions there thickening occurs, thus helping to sustain thin lithosphere in such regions [Molnar and Houseman, 2004].
There seems no doubt that ancient lithosphere differs from young lithosphere by being chemically different, such that ancient (Archean) lithosphere is intrinsically less dense than younger lithosphere. As a result, the lower density of ancient lithosphere is stabilized by its buoyancy. Nevertheless, both laboratory experiments and analysis of marginal stability show that for appropriate parameters, a chemically less dense less layer over more dense substratum when cooled sufficiently will become unstable [Cottrell et al., 2004; Jaupart et al., 2007]. Blobs of the cold light layer will sink into the hotter, more dense layer, but the process will be oscillatory. When the blobs warm they will rise back to the surface. We are pursuing further analysis of this oscillatory convection.
Conrad, C. P., and P. Molnar (1999), Convective instability of a boundary layer with temperature- and strain-rate-dependent viscosity in terms of 'available buoyancy,' Geophys. J. Int., 139, 51-68.
Conrad, C. P., and P. Molnar, The growth of Rayleigh-Taylor-type instabilities in the lithosphere for various rheological and density structures, Geophys. J. Int., 129, 95-112, 1997.
Cottrell, E., C. Jaupart, and P. Molnar (2004), Marginal stability of thick continental lithosphere, Geophys. Res. Lett., 31, L18612, doi:10.1029/2004GL020332.
Harig, C; Molnar, P; Houseman, GA (2008), Rayleigh-Taylor instability under a shear stress free top boundary condition and its relevance to removal of mantle lithosphere from beneath the Sierra Nevada. Tectonics, 27 (6) , Art. No. TC6019, doi: 10.1029/2007TC002241.
Harig, C., P. Molnar, and G. A. Houseman (2010), Lithospheric thinning and localization of deformation during Rayleigh-Taylor instability with nonlinear rheology and implications for intracontinental magmatism, J. Geophys. Res., 115, B02205, doi:10.1029/2009JB006422.
Houseman, G. A., D. P. McKenzie, and P. Molnar (1981), Convective instability of a thickened boundary layer and its relevance for the thermal evolution of continental convergent belts, J. Geophys. Res., 86, 6115-6132.
Houseman, G. A., and P. Molnar (1997), Gravitational (Rayleigh-Taylor) instability of a layer with non-linear viscosity and convective thinning of continental lithosphere, Geophys. J. Int., 128, 125-150.
Jaupart, C; Molnar, P; Cottrell, E (2007), Instability of a chemically dense layer heated from below and overlain by a deep less viscous fluid. J. Fluid Mech., 572 433-469, doi: 10.1017/S0022112006003521.
Molnar, P; Garzione, CN (2007), Bounds on the viscosity coefficient of continental lithosphere from removal of mantle lithosphere beneath the Altiplano and Eastern Cordillera. Tectonics, 26 (2) , Art. No. TC2013, doi:
Molnar, P., G. A. Houseman, and C. P. Conrad (1998), Rayleigh-Taylor instability and convective thinning of mechanically thickened lithosphere: effects of non-linear viscosity decreasing exponentially with depth and of horizontal shortening of the layer, Geophys. J. Int., 133, 568-584.
Molnar, P., and G. A. Houseman (2004), Effects of buoyant crust on the gravitational instability of thickened mantle lithosphere at zones of intracontinental convergence, Geophys. J. Int., 158, 1134-1150.