#Rendering Icon at BlockPos on the Clients Screen [Closed]

60 messages · Page 1 of 1 (latest)

spiral oar
#

I tried to work the matrix and vec3d calculations myself for a while, but its just to much for me.

Minecraft 1.20.6, Java

public class FloatingIconHudOverlay implements HudRenderCallback {

private final List<FloatingIconSupplier> iconSuppliers;
private static final ArrayList<BlockEntity> blockEntitiesToRender = new ArrayList<>();

public FloatingIconHudOverlay(List<FloatingIconSupplier> iconSuppliers) {
    this.iconSuppliers = iconSuppliers;
}

public static void addRenderedBlockEntity(BlockEntity blockEntity) {
    blockEntitiesToRender.add(blockEntity);
}

@Override
public void onHudRender(DrawContext drawContext, float tickDelta) {
    if (!MinecraftClient.isHudEnabled())
        return;

    var player = MinecraftClient.getInstance().player;
    if (player == null) {
        return;
    }
    var world = player.getWorld();
    if (world == null)
        return;

    // Render a 2D icon (sprite or texture, whatever is easier) for each BlockEntity in the blockEntitiesToRender list
    // it should be rendered on the hud, to make it visible through other blocks, when the player is looking into the direction of the block
    // The icon should always show centered on the block and always face the camera flat (like any on hud icon)

Any help would be really appreciated, I am getting desperate.
(I tried about 20 approaches, but conversions from camera vectors to matrices etc. to get the different calculations inside mincraft rendering systems are just to much.
Tried e.g. using the same logic as the textrenderer does in EntityRenderer for the label, but I cant get it to work for non-text)

Glad to help in reverse with non-render related code

primal raven
#

Maybe inject into wherever minecraft renders block entities so you can borrow the correctly-transformed matrixstack

#

That might not work for everything though

spiral oar
#

tried that first, there is no way (that I could find) to get the lightning to work correctly, as the rendering engine will work with the culliing of blocks. The only thing I got to work was a white cube that could be seen through other blocks. It was white because I abused the transparent rendering layers to achieve even that.
I think using the hud renderer and do the rest with manual conversions of the correct vec3d to matrix would be the better approach. Sadly I just cant work it out

novel kelp
#

I would recommend looking into any GitHub that would implement this, probably mods with waypoints

#

I think I did something like this a long time ago but with text not texture, I'll try to find it

novel kelp
# 
private static void renderText(WorldRenderContext context, Vec3d pos, String text, float scale) {
    MinecraftClient client = MinecraftClient.getInstance();
    if (client.player == null) return;

    MatrixStack matrixStack = context.matrixStack();
    TextRenderer textRenderer = client.textRenderer;

    // not sure if this is required
    RenderSystem.enableBlend();
    RenderSystem.defaultBlendFunc();
    RenderSystem.disableDepthTest();
    RenderSystem.disableCull();

    // save state
    matrixStack.push();

    // get relative
    Vec3d cameraPos = client.gameRenderer.getCamera().getPos();
    matrixStack.translate(-cameraPos.x, -cameraPos.y, -cameraPos.z);
    matrixStack.translate(pos.x, pos.y, pos.z);

    // rotate text
    matrixStack.multiply(client.gameRenderer.getCamera().getRotation());
    matrixStack.multiply(RotationAxis.POSITIVE_Y.rotationDegrees(180.0F));

    // setup
    matrixStack.scale(-scale, -scale, scale);
    Text renderedText = Text.of(text);
    float textWidth = textRenderer.getWidth(renderedText);
    float textHeight = textRenderer.fontHeight;
    float xOffset = -textWidth / 2f;
    float yOffset = -textHeight / 2f;
    VertexConsumerProvider.Immediate immediate = client.getBufferBuilders().getEntityVertexConsumers();

    // render
    textRenderer.draw(renderedText, xOffset, yOffset, 0xFFFFFFFF, false, matrixStack.peek().getPositionMatrix(),
            immediate, TextRenderer.TextLayerType.SEE_THROUGH, 0x00000000, 15728880);
    immediate.draw();

    // restore state
    matrixStack.pop();

    // reset
    RenderSystem.enableCull();
    RenderSystem.enableDepthTest();
    RenderSystem.disableBlend();
}

this is pretty old but maybe it'll be helpful

spiral oar
novel kelp
#

Worst case scenario you can give a Unicode a texture then render that Unicode

spiral oar
#

hm not sure if that would work with more than 5 pixels, but it is an interesting idea. Maybe I can then inject into the text renderer and get everything about the position from there ... hmmm.... will try that. Its a hack, but honestly I dont care anymore as long as it works ^^

spiral oar
#

Reasearched a bit more and tried to re-do something I tried 3 months ago for my issue. But as I might not even need the matrix conversion if I do it like this, I thought I would give it another go and provide the code I tried as a reference:

private Vec3d projectToScreen(DrawContext context, Vec3d blockPos, Entity camera) {
        // Calculate the relative position of the block to the player
        Vec3d relativePos = blockPos.subtract(camera.getPos());

        // apply camera yaw
        double z = relativePos.z * MathHelper.cos(camera.getYaw());
        double x = relativePos.x * MathHelper.sin(camera.getYaw());


        // apply camera pitch
        z *= MathHelper.cos(camera.getPitch());
        double y = relativePos.y * MathHelper.sin(camera.getPitch());


        double fov = 2;
        double fovX = x / z * fov;
        double fovY = y / z * fov;

        // Project the 3D position to 2D screen coordinates
        double screenX = context.getScaledWindowWidth() / 2.0 + fovX;  // + relativePos.x * 10; scaling factor
        double screenY = context.getScaledWindowHeight() / 2.0 + fovY;  // - relativePos.y * 10; scaling factor

        // Check if the projected position is within the screen bounds
        if (screenX >= 0 && screenX <= context.getScaledWindowWidth() && screenY >= 0 && screenY <= context.getScaledWindowHeight()) {
            return new Vec3d(screenX, screenY, 0);
        }
        return null;
    }

But, since the code produces rather random screen positions its obviously not working as it is. I tried to follow https://forums.minecraftforge.net/topic/75508-how-do-i-convert-a-3d-point-to-screen-coordinates/ maybe someone finds the error. Might also be the completely wrong approach as well (I hate coding in a field I have not enough context to even know whats reasonable 😐 ).
Next I will try the hack one with textRenderer and UniCode

Forge Forums
How do I convert a 3D point to screen coordinates?

I'm trying to make a mod that renders entity health bars in an external window (same width and height as the game window), not the main game window. To do this, I need to convert an entity's position (x, y, z) to 2D screen coordinates (x, y) so I can render the health bar on the external window. ...

bitter bramble
spiral oar
# 
@Mixin(WorldRenderer.class)
public abstract class WorldRendererMixin {
    @Inject(method = "render",
            at = @At(value = "INVOKE", target = "Lnet/minecraft/client/render/WorldRenderer;renderWorldBorder(Lnet/minecraft/client/render/Camera;)V",
                    shift = At.Shift.AFTER))
    private void onRenderPostWorldBorder(float tickDelta, long limitTime, boolean renderBlockOutline, Camera camera, GameRenderer gameRenderer, LightmapTextureManager lightmapTextureManager, Matrix4f matrix4f, Matrix4f projectionMatrix, CallbackInfo ci) {

        for(var entry : FloatingIconHudOverlay.blockEntitiesToRender.entrySet()){
            var blockEntity = entry.getKey();
            var screenPos = TrapBlockEntityRenderer.project3Dto2D(blockEntity.getPos().toCenterPos(), matrix4f, projectionMatrix, camera);
            FloatingIconHudOverlay.addRenderedBlockEntityPosition(blockEntity, screenPos);
        }
    }
}
# 
public static @Nullable Vec2f project3Dto2D(Vec3d worldPosition, Matrix4f modelViewMatrix, Matrix4f projectionMatrix, Camera camera) {

        var cameraPosition = camera.getPos();
        var cameraRotation = camera.getRotation();
        var in3d = worldPosition.subtract(cameraPosition);

        var wnd = MinecraftClient.getInstance().getWindow();
        var quaternion = new Quaternionf((float) in3d.x, (float) in3d.y, (float) in3d.z, 1.f);

        Matrix4f m = modelViewMatrix.rotate(cameraRotation.invert()); // this
        var product = mqProduct(projectionMatrix, mqProduct(m, quaternion));
        modelViewMatrix.rotate(cameraRotation); // undo that

        if (product.w <= 0f) {
            return null;
        }

        var screenPos = qToScreen(product);
        var x = screenPos.x * wnd.getWidth();
        var y = screenPos.y * wnd.getHeight();

        if (Float.isInfinite(x) || Float.isInfinite(y)) {
            return null;
        }

        return new Vec2f(x, wnd.getHeight() - y);
    }

    private static Quaternionf mqProduct(Matrix4f m, Quaternionf q) {
        return new Quaternionf(
                m.m00() * q.x + m.m10() * q.y + m.m20() * q.z + m.m30() * q.w,
                m.m01() * q.x + m.m11() * q.y + m.m21() * q.z + m.m31() * q.w,
                m.m02() * q.x + m.m12() * q.y + m.m22() * q.z + m.m32() * q.w,
                m.m03() * q.x + m.m13() * q.y + m.m23() * q.z + m.m33() * q.w);
    }

    private static Quaternionf qToScreen(Quaternionf q) {
        var w = 1f / q.w * 0.5f;

        return new Quaternionf(
                q.x * w + 0.5f,
                q.y * w + 0.5f,
                q.z * w + 0.5f,
                w);
    }
#

FloatingIconHudOverlay

    @Override
    public void onHudRender(DrawContext context, float tickDelta) {
        if (!MinecraftClient.isHudEnabled())
            return;

        var player = MinecraftClient.getInstance().player;
        if (player == null) {
            return;
        }
        var world = player.getWorld();
        if (world == null)
            return;

        var cameraEntity = MinecraftClient.getInstance().cameraEntity;
        if (cameraEntity == null)
            return;
        for (var entry : blockEntitiesToRender.entrySet()) {
            var blockEntity = entry.getKey();
            var screenPos = entry.getValue();
            if(screenPos == null)
                continue;
            var blockItem = new ItemStack(world.getBlockState(blockEntity.getPos()).getBlock().asItem());
            var guiScale = MinecraftClient.getInstance().getWindow().getScaleFactor();
            context.drawItem(blockItem, (int) (screenPos.x / guiScale), (int) (screenPos.y / guiScale));
        }
        blockEntitiesToRender.clear();
    }
bitter bramble
#

try dividing the resulting x and y by the gui scale (client.getWindow().getGuiScale())

spiral oar
#

maybe this will help as context. An extract from the GameRenderer renderWorld function that is used to call the WorldRendererMixin:

        double d = this.getFov(camera, tickDelta, true);
        Matrix4f matrix4f = this.getBasicProjectionMatrix(d);
        MatrixStack matrixStack = new MatrixStack();
        this.tiltViewWhenHurt(matrixStack, camera.getLastTickDelta());
        if ((Boolean)this.client.options.getBobView().getValue()) {
            this.bobView(matrixStack, camera.getLastTickDelta());
        }

        matrix4f.mul(matrixStack.peek().getPositionMatrix());
        float f = ((Double)this.client.options.getDistortionEffectScale().getValue()).floatValue();
        float g = MathHelper.lerp(tickDelta, this.client.player.prevNauseaIntensity, this.client.player.nauseaIntensity) * f * f;
        if (g > 0.0F) {
            int i = this.client.player.hasStatusEffect(StatusEffects.NAUSEA) ? 7 : 20;
            float h = 5.0F / (g * g + 5.0F) - g * 0.04F;
            h *= h;
            Vector3f vector3f = new Vector3f(0.0F, MathHelper.SQUARE_ROOT_OF_TWO / 2.0F, MathHelper.SQUARE_ROOT_OF_TWO / 2.0F);
            float j = ((float)this.ticks + tickDelta) * (float)i * ((float)Math.PI / 180F);
            matrix4f.rotate(j, vector3f);
            matrix4f.scale(1.0F / h, 1.0F, 1.0F);
            matrix4f.rotate(-j, vector3f);
        }

        this.loadProjectionMatrix(matrix4f);
        Matrix4f matrix4f2 = (new Matrix4f()).rotationXYZ(camera.getPitch() * ((float)Math.PI / 180F), camera.getYaw() * ((float)Math.PI / 180F) + (float)Math.PI, 0.0F);
        this.client.worldRenderer.setupFrustum(camera.getPos(), matrix4f2, this.getBasicProjectionMatrix(Math.max(d, (double)(Integer)this.client.options.getFov().getValue())));
        this.client.worldRenderer.render(tickDelta, limitTime, bl, camera, this, this.lightmapTextureManager, matrix4f2, matrix4f);
#
  • I also updated the code above to include the gui scale
bitter bramble
spiral oar
#

@bitter bramble also if you have a donation option for any mod you released or something like that, I would love to show my appreciation for your help and buy you a coffee. You have no idea how many hours I already struggled with this and I am now light years ahead of any of my attempts, even though its not perfect yet ❤️

bitter bramble
#

I'm unsure you have the right matrix. I got it from the top of the MatrixStack. Your mixin doesn't look identical to mine.

spiral oar
# bitter bramble I'm unsure you have the right matrix. I got it from the top of the MatrixStack. ...

I used what you did in https://github.com/Natanaelel/regex_search/blob/main/src/client/java/natte/re_search/render/HighlightRenderer.java#L147 as a baseline, but in minecraft 1.20.6 with yarn mappings the render call looks a bit different.

GitHub
regex_search/src/client/java/natte/re_search/render/HighlightRender...

Contribute to Natanaelel/regex_search development by creating an account on GitHub.

#

You can see what the render method is called with in the last code snippet above, or if you have time we could do a quick call and I can show you a stream

bitter bramble
#

sure

spiral oar
#

Rendering Icon at BlockPos on the Clients Screen [Closed]

heady heath
#

Like you mentioned the class that was mixed before in 1.20.1 is no longer there in newer versions

vernal badge
#

not digging into the whole thread, but i did it with WorldRenderEvents when i was doing it without mixin

heady heath
vernal badge
#

they changed recently, might have been the last one that still had the buffers stood up. didnt want to deal with that myself

heady heath
#

Ah yeah okay thank you 🙂

vernal badge
#

AFTER_ENTITIES may work well, depending on need, or AFTER_TRANSLUCENT

heady heath
bitter bramble
spiral oar
# 
package ....mixin.client;

import com.llamalad7.mixinextras.sugar.Local;
import net.minecraft.client.render.Camera;
import net.minecraft.client.render.GameRenderer;
import net.minecraft.client.util.math.MatrixStack;
import org.joml.Matrix4f;
import org.spongepowered.asm.mixin.Mixin;
import org.spongepowered.asm.mixin.injection.At;
import org.spongepowered.asm.mixin.injection.Inject;
import org.spongepowered.asm.mixin.injection.callback.CallbackInfo;

@Mixin(GameRenderer.class)
public abstract class GameRendererMixin {
    @Inject(method = "renderWorld",
            at = @At(value = "INVOKE", target = "Lnet/minecraft/client/render/WorldRenderer;render(FJZLnet/minecraft/client/render/Camera;Lnet/minecraft/client/render/GameRenderer;Lnet/minecraft/client/render/LightmapTextureManager;Lorg/joml/Matrix4f;Lorg/joml/Matrix4f;)V",
                    shift = At.Shift.AFTER))
    private void onRenderPostWorldBorder(float tickDelta, long limitTime, CallbackInfo ci, @Local MatrixStack matrixStack, @Local(ordinal = 0) Matrix4f projectionMatrix, @Local Camera camera) {

        for (var entry : FloatingIconHudOverlay.blockEntitiesToRender.entrySet()) {
            var blockEntity = entry.getKey();

            // invert to remove the bobbing matrix
            var modelViewMatrix = matrixStack.peek().getPositionMatrix().invert();

            var screenPos = TrapBlockEntityRenderer.project3Dto2D(blockEntity.getPos().toCenterPos(), modelViewMatrix, projectionMatrix, camera);
            FloatingIconHudOverlay.addRenderedBlockEntityPosition(blockEntity, screenPos);
        }
    }
}
# 
public static @Nullable Vec2f project3Dto2D(Vec3d worldPosition, Matrix4f modelViewMatrix, Matrix4f projectionMatrix, Camera camera) {

        // copy of camera postion data to not modify it
        var cameraPosition = new Vec3d(camera.getPos().x, camera.getPos().y, camera.getPos().z);
        var cameraRotation = new Quaternionf(camera.getRotation());

        var in3d = cameraPosition.subtract(worldPosition);

        var wnd = MinecraftClient.getInstance().getWindow();
        var quaternion = new Quaternionf((float) in3d.x, (float) in3d.y, (float) in3d.z, 1.f);

        Matrix4f m = modelViewMatrix.rotate(cameraRotation.invert()); // this
        var product = mqProduct(projectionMatrix, mqProduct(m, quaternion));
        modelViewMatrix.rotate(cameraRotation); // undo that

        if (product.w <= 0f) {
            return null;
        }

        var screenPos = qToScreen(product);
        var x = screenPos.x * wnd.getScaledWidth();
        var y = screenPos.y * wnd.getScaledHeight();

        if (Float.isInfinite(x) || Float.isInfinite(y)) {
            return null;
        }

        return new Vec2f(x, y);
    }

    private static Quaternionf mqProduct(Matrix4f m, Quaternionf q) {
        return new Quaternionf(
                m.m00() * q.x + m.m10() * q.y + m.m20() * q.z + m.m30() * q.w,
                m.m01() * q.x + m.m11() * q.y + m.m21() * q.z + m.m31() * q.w,
                m.m02() * q.x + m.m12() * q.y + m.m22() * q.z + m.m32() * q.w,
                m.m03() * q.x + m.m13() * q.y + m.m23() * q.z + m.m33() * q.w);
    }

    private static Quaternionf qToScreen(Quaternionf q) {
        var w = 1f / q.w * 0.5f;

        return new Quaternionf(
                q.x * w + 0.5f,
                q.y * w + 0.5f,
                q.z * w + 0.5f,
                w);
    }
#

All credits go to Natte, who provided the original code and helped me with adjustments to get it to work ❤️

#

Note: do not use the standard DrawContext.drawItem/drawTexture methods. They are integer based, which causes slight jittering

# 
 // same as context.drawTexture but with float instead of int precision
    private static void drawTexture(DrawContext context, Identifier texture, float x1, float y1, float width, float height, float u, float v, int regionWidth, int regionHeight, int textureWidth, int textureHeight) {

        var x2 = x1 + width;
        var y2 = y1 + height;
        var z = 0.F;
        var u1 = (u + 0.0F) / (float) textureWidth;
        var u2 = (u + (float) regionWidth) / (float) textureWidth;
        var v1 = (v + 0.0F) / (float) textureHeight;
        var v2 = (v + (float) regionHeight) / (float) textureHeight;

        RenderSystem.setShaderTexture(0, texture);
        RenderSystem.setShader(GameRenderer::getPositionTexProgram);
        RenderSystem.disableCull();

        Matrix4f matrix4f = context.getMatrices().peek().getPositionMatrix();
        BufferBuilder bufferBuilder = Tessellator.getInstance().getBuffer();
        bufferBuilder.begin(VertexFormat.DrawMode.QUADS, VertexFormats.POSITION_TEXTURE);
        bufferBuilder.vertex(matrix4f, x1, y1, z).texture(u1, v1).next();
        bufferBuilder.vertex(matrix4f, x1, y2, z).texture(u1, v2).next();
        bufferBuilder.vertex(matrix4f, x2, y2, z).texture(u2, v2).next();
        bufferBuilder.vertex(matrix4f, x2, y1, z).texture(u2, v1).next();
        BufferRenderer.drawWithGlobalProgram(bufferBuilder.end());

        RenderSystem.enableCull();
    }

is just a copy of DrawContext.drawTexture, but with all integers switched to float to be precise enough

spiral oar
heady heath
heady heath
#

It's supposed to be rendering at the Emerald block...

NVIDIA_Share_eCMo0l1UhW.gif
heady heath
#

So I'm not sure why the large shift between our versions

bitter bramble
heady heath
heady heath
#

Its so jittery for me, let me try and get an other one

vernal badge
#

looks like you are using the wrong pitch rotation calculation

upbeat panther
bitter bramble
bitter bramble
heady heath
spiral oar