Wednesday 2 May 2012

Maya normals locking and what it does to tangent space

Apparently locking normals in Maya is no simple matter. There are several problems with using MEL, PyMel or methods accessed through the UI:
  • Locking normals might modify them. If you locked a normal before, locking it again might set it to something you didn't want.
  • Edge smoothing is re-calculated. Any vertex that has at least one hard edge, will make all the edges that are connected to it, hard. So for example you have a vertex with two soft edges and two hard edges connected to it. Lock that vertex's normal and suddenly all the edges are now hard.
Edge softness is still important even when you lock normals. With locked normals, changing edge softness will make no apparent difference in the viewport. However Maya will still look at edge smoothing to determine if vertices should share tangent and bi-normal vectors. This has  impact on normal maps and also on tri-stripping on graphics cards.
The only reliable way I found of locking normals is:

  • Store edge softness in an array
  • Save all the normals into an array
  • Set all the normals to what they are using this array
  • Lock all the normals per vertex, per poly
  • Restore edge softness