Vertex color plugin

[red sand]

Hello! My name is Blue and I am working on a plugin that adds vertex color support to BlockBench.

I have written about 100 lines of javascript in my life, and I just started learning how to use BlockBench yesterday. But I could not tolerate Blender for another second, so here we are.

I put together a checklist of items (not really in order) that need to be done for this plugin to start working:

• add vertex color & mix mode into vertex data
  • mix mode indicates how the texture color and vertex color are mixed together to produce the final albedo color.
  • Mix mode is a per-mesh value while the color is on every vertex.
  • make sure it is saved into the bbmodel file.
  • The properties API looks promising
• add "vertex color" and "texture + vertex color" preview display modes
  • Or, possibly just make the current "textured" mode show the vertex colors as well.
  • I have no idea how to do this
• add vertex color paint tool to the paint workspace
  • I imagine it's simple enough to add a new tool that is basically just the weight paint tool, but painting colors instead of weights
• add export options that include the vertex color data
  • I imagine just adding new codecs one by one, rather than rewriting the existing export options.
• add dropdown to change the mix mode for vertex colors on a mesh

My questions, to more experienced plugin devs, are:

1. How do I do monkey patching in a plugin?
2. is there a way to override the viewport rendering without monkey patching?
3. Is it better to replace the built-in codecs, or create new ones that support vertex colors? For example for a gltf export, should the plugin replace the existing gltf codec implementation, or should there be a new "gltf with vertex colors" option in the menu?
4. where should the mix mode dropdown / option be placed? Or does something like this already exist for multiple textures on a mesh?

[still basin]

Super interested in this plugin coming out. I tried doing it myself but gave up at interface stuff. (trying to enable wireframe mode when selecting the vertex brush. Lots of UpdateCanvas() logic iirc)

Feel free to rip whatever from my attempt if you'd like. - https://github.com/Luther-Gray/Vertex-Paint Though there's not really anything to get functionally.

I'm not sure I understand the technical wording you're using 100% but I'll explain what I think I understand.

Technically, you shouldn't need anything third party because Blockbench is built with Three JS and that uses Vertex Colors. - https://threejs.org/docs/#Material.vertexColors

GLTF also supports vertex color data out of the box. I think that includes alpha as well, but double check that.

Blockbench technically(afaik) already is messing with Vertex Color tooling with the Weight Painting implementation they tackled. That layer, the vertex colors rather than the texture, seems to be treated as a different "material".(?? I don't know threejs.)

But that functionality is locked in the "Mesh" section of blockbench rather than the paint section. So you can't access your color palette, softness settings for the brush and opacity. The only unique setting you'd likely want is exact falloff settings maybe, since softness doesn't do that with enough precision to control the falloff direction between 2 points.

Everything I needed to make my brush and options came from here /blockbench/js/texturing/painter.js.

And there's a const meshVertexMaterial here - /blockbench/types/custom/canvas.d.ts

Mix Mode and all of that is stuff you can directly grab from the normal Blockbench Brushes.

[red sand]

Sorry yeah, tbh the notes are more for myself than for others. Once I get the plugin working I'll properly explain what it all means lol

But I'm glad to see there is interest in this, it means a lot that I won't be the only person benefitting from this work

At this point I need to just dig around the plugin API as well as the BlockBench source. Once I've done that I can put together a complete plan for how the whole thing is going to work.

[red sand]

Welp, I can't even seem to get properties to work. Here is the code so far:

let vertex_color_property;

function onaddmesh() {
    console.log("The user added a mesh");
    Mesh.all.forEach(m => {
        m.forAllFaces((face, key) => {
            console.log("Face " + key + " had vertex colors " + face["data_vertex_colors"]);
            face["data_vertex_colors"] = "hello";
        })
    })
}

Plugin.register('vertex_colors', {
    title: 'Vertex Colors',
    author: 'bluesillybeard',
    icon: 'icon',
    description: 'Adds vertex colors to Blockbench',
    version: '1.0.0',
    variant: 'both',
    await_loading: 'true',
    onload() {
        console.log("Hello I am a vertex plugin");
        vertex_color_property = new Property(Face, "string", "data_vertex_colors", {
            default: "hi",
            merge: (instance, data) => {
                instance["data_vertex_colors"] = structuredClone(data["data_vertex_colors"])
                console.log("vertex color merged")
            },
            copy_value: (instance, target) => {
                target["data_vertex_colors"] = structuredClone(instance["data_vertex_colors"])
                console.log("vertex color copied")
            },
            export: true
        });

        Blockbench.on('add_mesh', onaddmesh);
    },
    onunload() {
        vertex_color_property.delete();
        Blockbench.removeListener('add_mesh', onaddmesh);
    }
});

The documentation seems mega outdated, but importantly nothing is getting saved to the bbmodel file

Does anyone have any idea of why this might not be working?

It does save the data in the object (that's what the log near the top is for) but the bbmodel file remains empty of a vertex colors field

Btw I put the plugin up on github link

[red sand]

Went through the source code, realized I should change the property so it's applied to MeshFace. Bam, now it works (:

From now on, I am going to completely ignore all documentation and dig through the BlockBench source for information instead.

[red sand]

I'm gonna ask again since I've run into a roadblock. How does a plugin do monkey patching? Is there documentation or examples of it?

Ofc the other option is to abandon the plugin idea and turn this into a contribution / patch set

The roadblock here is, the vertex colors need to be synchronized with the vertices of a face, in the same way as uv. SO basically everywhere the vertices are changed, a patch is needed to make sure the vertex colors stay updated as well.

[red sand]

Well I came up with something. It feels so absolutely completely wrong, like it's not supposed to work but somehow it does

let vertex_color_property
let old_mesh_face_extend

Plugin.register('vertex_colors', {
    title: 'Vertex Colors',
    author: 'bluesillybeard',
    icon: 'icon',
    description: 'Adds vertex colors to Blockbench',
    version: '1.0.0',
    variant: 'both',
    await_loading: 'true',
    onload() {
        console.log("Hello I am a vertex plugin");
        vertex_color_property = new Property(MeshFace, "object", VERTEX_COLOR_DATA_NAME, {
            default: {},
            merge: (instance, data) => {
                instance[VERTEX_COLOR_DATA_NAME] = structuredClone(data[VERTEX_COLOR_DATA_NAME])
            },
            copy_value: (instance, target) => {
                target[VERTEX_COLOR_DATA_NAME] = structuredClone(instance[VERTEX_COLOR_DATA_NAME])
            },
            reset: (instance) => {
                instance_vertex_colors = {};
                for(let vertex_key in instance.vertices) {
                    instance_vertex_colors[vertex_key] = [1, 0, 1, 1, 0]
                }
                instance[VERTEX_COLOR_DATA_NAME] = instance_vertex_colors
            },
            export: true
        });

        old_mesh_face_extend = MeshFace.prototype.extend;

        MeshFace.prototype.extend = function (data) {
            old_mesh_face_extend.bind(this, data)()
            console.log("I was extended again")
        }
    },
    onunload() {
        MeshFace.prototype.extend = old_mesh_face_extend;
        vertex_color_property.delete()
    }
});

Am I supposed to do monkey patching this way? I'm such a JS noob, I have no idea why this seems to work

Went through the whole thing in a debugger, it does in fact work. So new JS knowledge on my part:

• use bind to call a method on an object if that object doesn't have that method
• using this within a function works if that function was called like a method on an object
• properties are actually variables that can be reassigned at will

My C++ brain cannot comprehend this madness but I'm happy I figured it out

[red sand]

The vertex color seems to be stored and loaded correctly, hooray! The next step is to have the colors show up in the preview

Is there a correct way to add new preview modes / modify existing ones?

[red sand]

Dug around Blockbench code for a few hours. As far as I can tell, the view modes are entirely hard-coded, and a lot of serious monkey patching will be needed to add a vertex colors preview mode

I would love to be proven wrong, but it looks like I gotta more or less replace the NodePreviewController.updateGeometry and NodePreviewController.updateFaces methods

Well, I'll sleep on it. Let me know if there is a decent way to add new preview modes.

[frail root]

I'm looking at this now, I think I can show you a proof of concept

It looks like each view mode is just switching which THREE.JS Material is used

[red sand]

Yeah, the issue is it's a giant if-else chain checking agiainst strings. So I'd need to add a new if to that chain - idk how to do that with a JS mixin patch thing without replacing the whole method

Not the end of the world, just annoying and probably not future proof

[red sand]

Either way, I would super appreciate a proof of concept since I really don't know what I'm doing

As far as I can tell the overall process looks like this:

• add a new option to the preview selection dropdown
• create a new THREE.js material
• modify updateGeometry to use that THREE.js material if the vertex color preview is selected
• create a new vertex_color attribute that the material can read the colors from
• make sure that vertex_color attribute has the correct data in the right layout so THREE.js renders the right colors on the right vertices of the right faces

[frail root]

Let me try and make that proof of concept (Just doing the custom view mode with a new material, making that material respect vertex color is a you thing)

[red sand]

Yeah that by itself would be a huge help

[frail root]

const MyCustomMaterial = new THREE.MeshBasicMaterial({color: 0xff0000, wireframe: true});

Plugin.register('blockbench_new_viewmode', {
    "title": "Test View Mode",
    "author": "Kai Salmon",
    "description": "Testing monkey patching view modes",
    onload() {
        const originalUpdateFaces = Mesh.preview_controller.updateFaces;
        Mesh.preview_controller.updateFaces = function(element) {
            let {mesh} = element;
            if (Project.view_mode === 'my_custom_view_mode'){
                mesh.material = MyCustomMaterial;
                this.dispatchEvent('update_faces', {element});
                return
            }
            return originalUpdateFaces.apply(this, arguments);
           
        };
        this._originalUpdateFaces = originalUpdateFaces;

        // Add view mode action to View menu
        this._action = new Action('my_custom_view_mode', {
            name: 'Enable My Fancy Custom View Mode',
            icon: 'fas.fa-gem',
            click: function() {
                Project.view_mode = 'my_custom_view_mode';
                Canvas.updateViewMode();
            }
        });
        MenuBar.addAction(this._action, 'view');
    },
    onunload() {
        Mesh.preview_controller.updateFaces = this._originalUpdateFaces;
        this._action.delete();
    }
});

So that's the full plugin, although the bit you care about is probs just

  Mesh.preview_controller.updateFaces = function(element) {
            let {mesh} = element;
            if (Project.view_mode === 'my_custom_view_mode'){
                mesh.material = MyCustomMaterial;
                this.dispatchEvent('update_faces', {element});
                return
            }
            return originalUpdateFaces.apply(this, arguments);
           
        };

The trick to "How do I inject into that lock if-else chain" is you don't

[red sand]

yeah I figured it would end up being something like that (:
Thanks for the help!

[frail root]

You can look forward to my upcoming Plugin "A view where it's a wireframe but it's red for some reason". I think the community is gonna love it