Using STB for Unicode

[kenzierocks]

I'd like to support arbitrary characters in my game, but I'm having trouble figuring out the best way to do that with STB. I obviously can't load the entire UTF-8 codepoint table for any font I would like to use, but is there a better solution that doesn't involve knowing what characters I want at startup time? I was thinking about caching some bitmaps somehow, but I'm not sure if I can continue building a bitmap with an arbitrary order of characters that aren't all added at the same time. Does anyone have a good idea or font renderer with lots of Unicode support?

[abcdef]

I don't really know what issue you have but this demo should help out

https://github.com/LWJGL/lwjgl3/blob/master/modules/core/src/test/java/org/lwjgl/demo/stb/Truetype.java

[spasi]

If you need to support a wide range of unicode codepoints, then some kind of caching is probably required. You can use an STBTTPackContext for that (see the functions stbtt_PackBegin and stbtt_PackFontRange). You first initialize the font texture with some common unicode ranges. Then, whenever you encounter a codepoint that's not in the texture, use stbtt_PackFontRange with the original PackContext (do not close it with stbtt_PackEnd). The output STBTTPackedchar data will let you know what part of the texture to update (e.g. with glTexSubImage2D).

[kenzierocks]

So I can use the context over a long period of time without it being too much of an issue?

[spasi]

Yes, so for a particular font you'd have the following state:

- The pack context (a small struct).
- A ByteBuffer with the raw pixel data (as big as your texture).
- The OpenGL texture handle (an integer).
- A data structure that knows which codepoints are currently rendered in the texture.

[kenzierocks]

So I've been working on implementing what Spasi suggested, but I've run into a few troubles with that. My current work can be seen in this file, but I've been getting native errors in various places and NPEs in others. Would someone mind analyzing my code?

[spasi]

The code looks reasonable, but it'd be very hard to find issues without actual stacktraces or crash logs. Could you provide those please?

One problem I did find is in the convertRangeListToSTBTTRanges method. You're using STBTTPackRange.calloc() in the loop and then buffer.put(stbRange). This is inefficient and you're also leaking memory (no .free() on stbRange). Note that STBTTPackRange.Buffer is a packed array of structs, you already have the memory required and don't need to allocate more. You can use either:

STBTTPackRange stbRange = buffer.get();

or use the buffer as a flyweight, i.e.:

buffer.position(i);
buffer.font_size(this.fontHeight);
// etc.

in both cases, the buffer.put(stbRange) call at the end of the loop should be removed (that method does a memcpy from the struct value to the struct array).

[kenzierocks]

Thanks for all your help so far spasi! I've fixed a lot of errors, but now I can't get FontTest (in com.techshroom.emergencylanding.library.debug) to show me the characters. I see that the packed char offsets are really small, around 0 or 0.5. What do I need to multiply by to get the texture offsets?

[kenzierocks]

I've now debugged a little further, and it seems that I just don't get updated data in pixels, it seems to not write any data at all to the buffer. I must have calculated some value wrong, and I'm not very good at debugging C code so I can't look at the method while it's running and figure out what's going on.

[spasi]

I think the bug is in:

this.pixelsTexture.onUpdatedPixels(
    new Vector2d(ch.xoff(), ch.xoff()),
    Rectangle.fromLengthAndWidth(
        ch.xoff2() - ch.xoff(),
        ch.yoff2() - ch.yoff()
    )
);

You should be using x0(), y0(), y1(), y2() instead.

Also noticed that the renderString method is quite inefficient. You should move the STBTTAlignedQuad allocation (and the corresponding free) outside the loop.

The other problem is how you handle "codePointPacked" and the "singleBuffer". You're again copying data around for no reason, but to fix it requires some refactoring. You should aim to have a single STBTTPackedchar.Buffer for all your codepoint ranges, instead of one for each set of contiguous codepoints. Then you would use that single buffer directly and only need a simple map from codepoint to character index, to use in stbtt_GetPackedQuad (you now always use 0).

In order to do this, you'd probably need to allocate a big STBTTPackedchar.Buffer upfront (with capacity equal to as many codepoints can reasonably fit in your texture, at the given font size). Then change convertRangeListToSTBTTRanges to use a slice of that big buffer in chardata_for_range, instead of allocating a new one every time.

[kenzierocks]

Thanks for your analysis, I'm still getting used to managing my own memory in full. I'm still not seeing any data being written into the pixels ByteByffer, which is pretty strange considering how it's always there.