#PixelMap - Minecraft top-down mapper

42 messages · Page 1 of 1 (latest)

weak depot
#

https://git.aposoc.net/McTwist/PixelMap

My initial project started back in 2010, and then was replaced in 2016 with this project. The project is a Minecraft world mapper creating top-down pixel-perfect images as efficiently as possible, supporting any Java Edition down from Infdev up to latest release, and Bedrock Edition at least tested down to 1.2. It can process 3 900 chunks/s (Ryzen 3800XT) and generate either a single big image or several smaller "region"-images. It is designed by default utilize all cores available, effectively increasing throughput significantly with reduced diminishing returns. Additional features included are night time, slice, color mode, gray scale, pipelining and custom block format. Provided are both a CLI and a GUI version.

I have the last couple months tinkered a lot with the build system to make it easier for anyone to build themselves. I have also optimized it further, with its predecessor being able to process 500 chunks/s (AMD FX-8350) to its initial version working 1 500 chunks/s (Ryzen 3800XT), got stuck at 23 hardware threads (EPYC 7702P) before I redesigned the threading routine to have close to no limitation on hardware threads, as long as there is sufficiently enough data to process.

The testing system was sadly more of an afterthought, but when integrated with cmake it made it a lot easier to locate bugs that popped up while I was optimizing and moving around features. Test data is generated in a separate repository and downloaded on demand. Packaging is also done with cmake, automatically publishing packages on demand or automatically.

Note: the performance numbers above are only to show progress over a span of 10 years with different types of hardware, so should be taken with a bit of salt.

#

I still got several features I want to add, and probably a bunch of bugs I have not caught up to, and while testing is a chore, I am trying to add more unit testing as I go, and more integration testing is added if new versions are released. I honestly do not mind what people think about this type of project, as it is fun to tinker with it, add more features or try out new techniques for optimization, but it would be fun if people found it interesting or maybe useful.

limber finch
#

why isn’t this on github?

weak depot
#

Self-hosting is self-owning.

#

I plan to mirror it to at least codeberg~~, but might do github too~~.

coarse sequoia
#

you should put an example picture in your readme

#

so people know what they can expect

weak depot
#

Sure thing. That was also my next step. Do you think it's better to link to a page with all the images, or just put the image inside the repository/wiki?

ripe valley
#

also, great work

weak depot
#

Mirrored to codeberg and added screenshots.

#

And put on github as well.

#

I also made an add-on library which PixelMap can load and replace the rendering engine. This type is using Lua, so the speed is significantly slower than native C++, but I just wanted to make it as a proof of concept. I recently updated it to support LuaJit, which gives a 30% speed boost.
https://git.aposoc.net/McTwist/PixelMap-Lua

#

If more screenshots is desired, I also have my data repository which contains both world data and resulting images, used for integration testing. It's ordered, but not pretty from a user standpoint:
https://git.aposoc.net/McTwist/PixelMap_data

weak depot
wide quiver
#

so this is like

#

taking a top down screenshot of your mc words?

weak depot
#

Yes, as the description says.

weak depot
#

Updated so it can be compiled on Mac, but nothing have actually been tested.

weak depot
#

Checked, fixed some bugs, and now it runs rather fine on Mac. Need to give it some worlds before it is declared fine, but the test suit fails to compile, so maybe start there.

weak depot
#

It now automatically builds for Mac too.

weak depot
#

Note: Creation of the images and copying of the files needed to display the map is all C++. The actual usage of this is in a web browser. (Almost want to use WebAsm to use C++ all the way yamikek )

weak depot
weak depot
#

I got rid of the GitHub mirror. The reason should be self-explanatory. ||ICE||

weak depot
#

I'm trying to modify the GUI so it will be more visually unique. What I have is some sizes/indents and colors. It's difficult only due to finding a design that works, and is not too "programmery generic".

I also am trying to make it compatible with OS themes, like dark/light mode. It won't work until I upgrade to SDL3.2.

#

Btw, I've seen that many mappers/viewers have enabled a way to display and pan the world in-app, and wonder if that is something people want me to add.

weak depot
weak depot
#

Playing around with caching, meaning if a world chunk/region hasn't changed, it will not try to parse and render it again. Example is a world taking usually 4.2s to process, now takes 0.5s if it hasn't changed.
This is necessary for live updating. Sadly, web view makes this much harder, so will have to tinker with that more to get it working properly.

weak depot
#

I just now finished tests for parallelized PNG encoding. It basically uses the default filter and merges together several rows up to a certain size, before it compresses it to a synced chunk (Up to 128KiB). It then does some manual removal of adler32 and generates a total checksum over all chunks and combines them together before putting it into a custom IDAT. The combination is way faster than a regular zlib on default (6) compression is, and when compressing each chunk in its own core, it'll result in an efficient performant boost. For instance, an image from a Minecraft world of 32k x 32k was done in about 167 seconds, but when using parallelized and chunked (instead of compression each row), it'll result in 112 seconds (including the parsing) (will do proper testing later). It has not been timed, but it is going somewhere from 67 seconds to 12 seconds, on 8 cores.
In addition, the file size went from 267MB to 172MB, probably due to more data to compress, and therefore more efficient, even though the most basic filter was being used.

I do not know if this would result in less PNG decoders from working, but I did try it on some, and while libpng itself took some real time reading them, they was still valid.

I also tried using libdeflate to generate images, but that sadly generated invalid compression. I suspect it has extra features that zlib does not support.

weak depot
#

And it has been pushed.

weak depot
#

1.21.11 was released. It contains the PNG encoding boost and size reduction.

restive kiln
weak depot
#

Not that it's bad. You did this within 2 weeks. I have worked on this since 2016.

#

But I also see that this is far from finished. You've not even started parsing the chunks, gathering the blocks, and translating their values, depending on the version you pick.

#

PM can parse any MC world from infdev up to 1.21.11, as well as Bedrock down to the conversion from Pocket Edition.

#

You got loads of work to do, but I suspect you're done with it, considering you worked on it 4 years ago.

weak depot
#

Next feature is visualizing biomes. It just took this long because of writing down the colors is a bit of a chore.

weak depot
#

Seems like MC 26.1 will be released 24th, so will prepare an update which will not since beta include an independent version system. Haven't decided yet, but probably it will be 2.0.0, so I can easier just release updates whenever.

weak depot
#

I just found out that the biggest change for 26.1, is the sudden folder re-location. JE just got rid of the 15y/o DIM<id> system and opt for namespaces instead. BE moved the worlds entirely from %LOCALAPPDATA% to %APPDATA% with some additional user attached to them (This was back in 1.21.120, it seems). Not gonna be hard, but it's a bother to have more to think about regarding backwards compatibility.

weak depot
#

And version 2.0.0 is released.