Both Nylon element and all-metal prevailing torque nuts work by elastic deformation of a portion of the nut against the male thread.
For the Nylon element nut the deformation is caused by the male thread impressing into the Nylon as the Nylon passes over the male thread.
For an all-metal element the nut's thread is typically manufactured either slightly out-of-round or under size, necessitating elastic deformation of the nut as it's threaded onto the male thread.
The operational principle of both is that the interference fit prevents relative lateral movement in the threads - that is movement perpendicular to the thread axis, Lateral movement is a contributor to loosening under vibration.
Depending on what you read and believe, the Nylon element may also provide some damping.
Nylon element nuts have temperature limitations.
All-metal versions may cause galling, or may damage coatings.