I have finished a "speed sculpt" in 3D Coat last week, and after a not-so-speedy retopo (I always underestimate the effort that goes into a good retopo ), I baked my highpoly voxel sculpt to the low poly mesh in 3D Coat and started painting. I was actually positively surprised to find almost zero baking errors on my first try, which went against my expieriences with 3D Coat.

What I didn't realize at the time was that the settings where set to "Unreal Engine preset". I did this sculpt in part to test out the new 3D Coat 4.5 (which has some amazing improvements btw... PBR FTW), so of course I lost all my settings from the prior version.

Now, when I realized that, I immidiatly exported the halfpainted model to .obj and tried import to Unity, which currently is my engine of choice. Even before the import I could see that the normal map looked different then what I was used to (tangent space normal maps in Unity usually have "flat colors" for flat areas... this one, optimized for Unreal engine, had a gradient over flat areas).

Needless to say the model has weird shading in Unity. Though interestingly, the missing baking errors are still apparent, while a second model where I tried to remove baking errors for an hour with additional gemometry ans shifting vertices still has lots of it, even though the shading of the normal map is now correct (as it has been baked with the options set to "Unity preset").

This might be a very specific question about very specific tools and engines, but really behind it are more general questions:

Why do normal maps baked for Unreal look so different to the ones baked for Unity? Why do the models tangents/normals themselves seem different? And why the heck do I get a whole lot more baking errors when baking for Unity, when the bake for Unreal seems to be quite fine without a lot of tweaking? Is this a 3D Coat problem, or is the Unity model importer to blame (tried obj and fbx, same errors on the model)?

I tried for hours to get the low-error baking result from the Unreal preset with tangent space maps optimized for unity, but that doesn't seem to work. My next step is to download Unreal 4 (which I have planned to do for ages by now, but never found the time) and see if I can import the 3D Coat baked model to that engine without problems (and with the same low baking error result that the painting room in 3D Coat indicates)... if that works, I definitely have to think about switching engines, because baking errors have been a MAJOR PAIN IN THE *** for a long time now when importing to unity.

I know from other engines that their importers had less struggles with 3D Coat exported obj files than Unity, so if Unreal 4 does the same painless import, it might just become my new engine of choice.

Gonna miss the asset store though....

EDIT:

Spent yesterday evening downloading and installing UE 4, and had a first test run. I am quite impressed so far. Most interesting fact ist that indeed, I was able to import the model as FBX without any of the baking errors that plague the version of the model built for Unity. Really cool, a clear 1:0 for Unreal. Still missing all my third party components and fighting with the editor, but saving hours of bake correction time on each model is massive for me.

Anyway, question is still valid: why? Why the different looking normal map, and why does it have such an influence?