Assimp skeletal animation with Blender

[VeAr]

Hi,

this might not be an LWJGL issue or bug, but generally Assimp related. I spent now a lot of time trying to import and export animated models and animations with Assimp, unsuccessfully. This post can serve as a kind of a warning to other people if they try to use Assimp for exporting models.

Animations are not implemented in the Assimp FBX exporter. I looked at the Assimp github source, and this is missing, its not documented anywhere either. The armature can be exported with FBX and it imports ok into Blender, but not the animation. The Blender 2.79 FBX importer is not the best either, but it opens the model, with matching armature, but rotated bones. The FBX model (even just static model, no armature) exported from Assimp can't be opened in Maya or MAX, but Blender handles it.

The armature exported wtih Assimp Collada exporter imports wrong into Blender.

Exporting animation with Assimp crashes the JVM. Its a C++ exception, somewhere in the destructor of the AIScene class. So practically the whole export process completes, but crashes at releasing the scene.

The best result to export an armature to Blender with Assimp is by using GLTF2 exporter. The armature imports perfectly into Blender.

But trying to export the animation, even if empty, will either return with an error code, or crash the JVM, in the destructor of AIScene. Sometimes the file with animation is written, but it is wrong in Blender.

So in the end i had to resolve to my own PSK/PSA exporter/importer to get animated models and animations into Blender and from Blender. There is a patched Blender PSK/PSA importer source available from GitHub,  and its possible to write a matching exporter for that in Java.

Its a petty that Assimp supports so many formats, but can't export animations with either of the formats.

I see two possibilities to have animated model import/export based on a modern format:

1. Make a direct wrapper for the FBX SDK. Only the FBX SDK can read/write FBX files perfectly. Blender can read/write FBX files not perfect but good enough, so this could work, when Blender-specific stuff/bugs are also considered. The drawback here that its another binary library, so if there is some error, it will crash the JVM. Its a black box just as Assimp is.
2. Make a GLTF2 exporter/importer in Java. The format is pretty readable and open, much simpler than FBX in structure. The existing Java GLTF2 importer/exporter does not look optimal, but would be a good starting point. Its still a question if the Blender GLTF2 animation importer is usable enough.

All in all, the animated model import/export situation does not look too good around Java.

[spasi]

I can't comment on the quality of Assimp's FBX support, you may want to open an issue with them. But if you could provide a sample that reproduces the crash (code + data to import/export), I'll gladly have a look to verify that the problem is indeed within Assimp and not something that LWJGL does wrong.

[orange451]

To verify, does LWJGL work correctly with IMPORTING fbx models? I have plans to use it in the future, but if it's not working I may look for a different solution.

[VeAr]

I can't comment on the quality of Assimp's FBX support, you may want to open an issue with them. But if you could provide a sample that reproduces the crash (code + data to import/export), I'll gladly have a look to verify that the problem is indeed within Assimp and not something that LWJGL does wrong.

I can't post the complete exporter, there are too many dependencies, but the relevant part to export animations is this:

protected void createAnimations() {
		if(model.animSets.isEmpty())
			return;

		if(!exportArmature)
			return;

		// create buffer for animsets
		AIAnimation.Buffer anims = AIAnimation.callocStack(model.animSets.size());
		PointerBuffer animPtrs = MemoryStack.stackCallocPointer(model.animSets.size());

		Quaternionf tmpQuat = new Quaternionf();

		for(int i=0; i<model.animSets.size(); i++) {
			ModelAnimationSet animSet = model.animSets.get(i);

			AIAnimation anim = anims.get(i);
			animPtrs.put(i, anim);

			anim.mDuration(animSet.duration);
			anim.mName(getAIString(animSet.animSetName));
			// keys per second
			anim.mTicksPerSecond(animSet.numKeys / animSet.duration);

			// create buffer for channels
			AINodeAnim.Buffer channels = AINodeAnim.callocStack(animSet.numberOfBones);
			PointerBuffer chanPtrs = MemoryStack.stackCallocPointer(animSet.numberOfBones);
			for(int j=0; j<animSet.numberOfBones; j++) {
				ModelBoneAnimation mba = animSet.boneAnimations.get(j);

				AINodeAnim nodeAnim = channels.get(j);
				chanPtrs.put(j, nodeAnim);
				// animation is bound by node name, no need to have ref to the node itself
				nodeAnim.mNodeName(getAIString(mba.boneName));

				AIVectorKey.Buffer posKeys = AIVectorKey.callocStack(mba.numKeys);
				AIQuatKey.Buffer rotKeys = AIQuatKey.callocStack(mba.numKeys);
				AIVectorKey.Buffer scaleKeys = AIVectorKey.callocStack(1);

				for(int k=0; k<mba.numKeys; k++) {
					AIVectorKey vecKey = posKeys.get(k);
					vecKey.mValue().set(mba.data.get(k*7 + 0), mba.data.get(k*7 + 1), mba.data.get(k*7 + 2));
					vecKey.mTime(mba.keyInterval*k);

					AIQuatKey quatKey = rotKeys.get(k);

					tmpQuat.set(mba.data.get(k*7 + 3), mba.data.get(k*7 + 4), mba.data.get(k*7 + 5), mba.data.get(k*7 + 6));

					// w comes first
					quatKey.mValue().set(tmpQuat.w, tmpQuat.x, tmpQuat.y, tmpQuat.z);
					quatKey.mTime(mba.keyInterval*k);
				}

				// fill in the positions, 
				nodeAnim.mPositionKeys(posKeys);

				// fill in the rotation key
				nodeAnim.mRotationKeys(rotKeys);

				// scale is constant 1
				scaleKeys.mTime(0);
				scaleKeys.mValue().set(1,1,1);
				nodeAnim.mScalingKeys(scaleKeys);

			}
			// fill the channels
			anim.mChannels(chanPtrs);
		}
		// set the animations into the scene
		scene.mAnimations(animPtrs);
	}

The part for model export:

int options = 0
								|Assimp.aiProcess_FindInvalidData
								|Assimp.aiProcess_ValidateDataStructure
							;
			if(convertRightHand) {
				options |= Assimp.aiProcess_ConvertToLeftHanded;
			}

			Assimp.aiExportScene(scene, format, targetFile, options);

The JVM crash reports this stack-trace:

C  [assimp.dll+0x16ca7]
C  [assimp.dll+0x3de15]
C  [assimp.dll+0x40b61]
C  [assimp.dll+0x421fa]
C  [assimp.dll+0x420e1]
C  0x0000000002968c67

If i take out the "Assimp.aiProcess_ConvertToLeftHanded" processing, then there is no crash, but the JVM process still exits prematurely:

Process finished with exit code -1073740940 (0xC0000374)

This is the stack-trace when run under 32-bit JVM:

C  [assimp32.dll+0x14fc6]
C  [assimp32.dll+0x37a01]
C  [assimp32.dll+0x3b749]
C  [lwjgl32.dll+0x1dc4]
j  org.lwjgl.assimp.Assimp.naiExportScene(JJJI)I+23
j  org.lwjgl.assimp.Assimp.aiExportScene(Lorg/lwjgl/assimp/AIScene;Ljava/lang/CharSequence;Ljava/lang/CharSequence;I)I+43

There is always a chance that i got something wrong, but i don't see what would be wrong with the above code. As it crashes in destructor, i would suspect that Assimp still tries to release something that is not filled.

To answer the other question on using FBX to import animations, technically the FBX importer works, and the imported animation data looks good, but Assimp adds its own translation, rotation and scale nodes on top of the bone hierarchy, those transforms must be considered. If you can, then add an additional "fromRootToModelSpace" matrix to your models, and apply that after the animation. I tried to multiply it together with root bone transform, but when converted to quaternion the axis of rotation for the bones is lost. I suppose the animation would still play, but if the engine depends on orientation of bones, then there are problems. Best is to keep bones oriented straight along an axis. Perhaps there is a way to get around/correct this, but my knowledge is not enough. Good thing about the PSK/PSA format is that it works with quaternions, so the whole export/import process is more transparent, but that format has other limitations.

[spasi]

I was able to export animations to binary .fbx, ascii didn't work. Some questions:

- Are you building the AIScene from scratch? I used Import -> Copy -> modify scene -> Export.
- Using the stack for all allocations seems dangerous. Are you sure the memory is still valid when aiExport is called?
- Do the bone nodes exist in the scene? (those referenced by nodeAnim.mNodeName)

I'm not sure why you think Assimp is crashing in a destructor. You could try using a debug build of Assimp, the JVM crash log will then contain the name of the function that's crashing.

[VeAr]

I was able to export animations to binary .fbx, ascii didn't work. Some questions:

- Are you building the AIScene from scratch? I used Import -> Copy -> modify scene -> Export.
- Using the stack for all allocations seems dangerous. Are you sure the memory is still valid when aiExport is called?
- Do the bone nodes exist in the scene? (those referenced by nodeAnim.mNodeName)

I'm not sure why you think Assimp is crashing in a destructor. You could try using a debug build of Assimp, the JVM crash log will then contain the name of the function that's crashing.

I used a dll symbol lister to find out the function name it was crashing in. Might be wrong about that, but sometimes the file is written when it crashes, so makes sense that the crash happens in clean-up.

I construct the scene from scratch, i already tried copying the constructed scene, the export crashes with the export of the copy too, the copying itself does not crash.

I will try importing an empty scene, or export the armature only, load it back, and try to add the animation to that.

The whole export is inside one stack definition:

static {
// reserve 30Mb, the model is ~1.5Mb
		Configuration.STACK_SIZE.set(1024*30);
	}

// ...
// create stack for allocations
try(MemoryStack stack = MemoryStack.stackPush()) {

// construct scene

// export scene
Assimp.aiExportScene(scene, format, targetFile, options);
}

Would you recommend using calloc instead of callocStack, keeping track of allocated objects and free-ing them after, but isn't that what the stack does? My code does not allocate/release another stack, so i assume all buffers are created in the declared stack.

The nodes all exist. They are referenced by name, so i assume its a lazy binding between animations and armature. But i added code to check the names, and all are there.

I'm getting a slightly different stack trace with FBX:

C  [assimp.dll+0x16ca7]
C  [assimp.dll+0x3de15]
C  [assimp.dll+0x10816]
C  [assimp.dll+0x383620]
C  [assimp.dll+0x37c6ca]
C  [assimp.dll+0x37c82d]
C  [assimp.dll+0x379a1d]
C  [ntdll.dll+0xa362f]
C  [ntdll.dll+0x5ecc]
C  [assimp.dll+0x379995]
C  [assimp.dll+0x37b86c]
C  [assimp.dll+0x37bc4e]
C  [assimp.dll+0x37c964]
C  [assimp.dll+0x379a1d]
C  [ntdll.dll+0xa35af]
C  [ntdll.dll+0x4aaf]
C  [ntdll.dll+0x88a6]
C  [KERNELBASE.dll+0x55299]
C  [assimp.dll+0x37aa0d]
C  [assimp.dll+0x258e27]
C  [assimp.dll+0x24cfd0]
C  [assimp.dll+0x24cc1d]
C  [assimp.dll+0x40aaa]
C  [assimp.dll+0x421fa]
C  [assimp.dll+0x420e1]
C  0x00000000028e8c67

[spasi]

I construct the scene from scratch

This might be (part of) the problem. The aiScene struct has internal data (mPrivate) that must be initialized properly and I don't think you can have that without importing something first.

The whole export is inside one stack definition:

static {
// reserve 30Mb, the model is ~1.5Mb
		Configuration.STACK_SIZE.set(1024*30);
	}

Would you recommend using calloc instead of callocStack, keeping track of allocated objects and free-ing them after, but isn't that what the stack does? My code does not allocate/release another stack, so i assume all buffers are created in the declared stack.

The stack is meant to be used for small, short-lived allocations. That's why it defaults to 64kb. What you did can work, but then you'd waste 30MB in each thread that touches the thread-local stack.

I'm getting a slightly different stack trace with FBX

I have uploaded a debug build that might help here: https://s3.amazonaws.com/build.lwjgl.org/nightly/windows/x64/assimp-debug.zip

After experimenting a bit, it looks like there's indeed an issue with destruction. Assimp uses the delete[] C++ operator to destruct array struct members and that's not compatible with simple arrays allocated with malloc or the LWJGL stack. For example, if the array type is a class with a custom destructor, C++ runtimes usually add a header to the allocated memory that stores the array size. The size is used on delete[] to call the destructors of each array element. Afaik there's no specified way to emulate new[] with plain C malloc. We might have to add C++ allocation methods for such structs, which requires custom JNI and is going to be a pita.