#tooldev-general
1 messages · Page 124 of 1
The error may contain information that indicates what's wrong.
(something my users rarely realize 😄 )
Are the skills included in those files?
Trade site needs some elemental ailment immunity ^_^
trade site should require authentication, and ||sacrifice in-game currency to perform n-amount of searches|| 😅
May I ask you something? After reading the conversation regarding the trade site I would like to hear your thoughts.
Do you think it would be feasible to just display the price information without the listing for some items?
When checking e.g. a div card in most cases the desired information is the price and maybe the history. Both information (price and history) could be displayed without a request to the trade, right? Without checking the code this information is displayed, even when the listing are not loading, so you got from a different source?
Checking with your macro is just very convenient and I am guilty of using it for ritual rewards too.
where would you get price and history if not from trade
is ninja reliable if the river is that bad
@grave wren Idk, what the source is but it is shown before the trade request is even loaded, therefore I assumed it is not calculated on the fly with the request
I want to remove one request (20 items) with the next update, will be loaded on demand now
More of a priceglance than a pricecheck 🙂
ooh neat woke trade macro now recognizes stacks when using . as thousands separator 
Sent a probe item into the river, poe.trade is just 8-9 minutes behind atm.
Official site has the item listed almost immediately, but doesn't yield results soooo... 😄
i conclude that poe.trade is the devil, thanks for testing
where is the spectre lua on POEdb generated from?
@worthy cape @golden bane
-- Data\Spectres.lua
minions["Metadata/Monsters/Maligaro/SecretDesecrateMonster"] = {
name = "Flickershade",
life = 3,
energyShield = 0.5,
fireResist = 0,
coldResist = 0,
lightningResist = 0,
chaosResist = 0,
damage = 1.05,
damageSpread = 0.2,
attackTime = 1.005,
attackRange = 25,
accuracy = 1,
weaponType1 = "One Hand Sword",
skillList = {
"SecretDesecrateMonsterTeleport",
"SecretDesecrateMonsterEarthquakeTriggered",
"SecretDesecrateMonsterMultiSlash",
},
modList = {
},
}```
copied that into spectres.lua
copied the skills portion into spectre.lua
is there another piece i'm missing?
the name shows up in POB and that error shows up when I select it.
@primal schooner There is a guide on Reddit somewhere for adding spectres
is the 3_0 folder a deprecated file location? because the current CF fork I have installed doesn't have those files in the same location
they are just in \data and \data\skills
additionally, that's the exact steps I did (save the 3_0 folder, because that's not where the current CF POB has those files) in the guide, and that's the error I'm receiving
Ooh neat, the PoB Dev Mode console has a built-in memory leakusage counter
@velvet fog Hi, quick question. Do you happen to have any updates in regards to the Model Viewer on PoeDB? It does not seem to work anymore
ho ho, models
Out of curiousity, what are you using it for?
(I've put in a fair bit of work in parsing and trying to apply the engine materials, rendering PoE meshes is a fair bit more involved than the static textures you may typically have)
@sonic gull there are some model generating issues in new version, so close the function
is the community pob still using proprietary windows runtime?
@velvet fog Anything fun in the model formats, or mostly around textures?
Model exporter reports error and i dont have much time on it right now
im mostly using it(the models) to gather Screenshots to use as Materials for the Path of Paper (dnd) session which im hosting
river having a bad time?
@simple ravine poe.trade update time from stash price change to live search notification was 21:53:59.06 -> 22:06:18.71
So "somewhat"
oof
@sonic gull I will try to fix that this weekend if I have time
@valid eagle https://www.pathofexile.com/forum/view-post/23790007
): have to specify items you have on bulk exchange now, guessing a way to limit price checkers?
flipper tools, not price checkers
Yes, although we have the full source code available now. For one of the next releases, we're trying to rebuild the main .exe and updater from the source code we obtained
Please support utf8 @golden bane
I know that's an often requested feature, but it just doesn't seem feasible without switching the whole graphics stack
China forum had modified binary version can display chinese utf8
I'm not too keen on reinventing utf-8 support. Maybe someone else is, I wouldn't know how to do it anyways
Will source code be public?
Yes, I just didn't have time yet to set up repos and permissions for our maintainers of Openarl's SimpleGraphic
Great news
But we have up-to-date sources for SimpleGraphic.dll, Path of Building.exe and Update.exe
I don't know how much we will have to change to get it to build with our pipeline, but I can get you the code now if you're interested
No hurry, we had used china version for years
Thanks, i appreciate that.
@sonic gull https://poedb.tw/us/model?id=Argus seems many model export failed, but some had worked
Not in front of computer too
Is there currently an algorithm for poe that reads a screenshot of the game and returns the hovered item mods?
So basically the information that you would get from ctrl+c an item but by processing a screenshot?
This would be useful for adapting current price-checking tools to poe players in streaming platforms such as Geforce Now and potentially Stadia. Indeed it would be better if the streaming platforms supported the exchange of information between both clipboards, but it doesn't seem like they are going to implement this anytime soon.
From a technical standpoint, as resolutions vary and the backgrounds of the info panels is no opaque, it's going to be an OCR hellscape.
People used to extract market data from Elite Dangerous stations by analyzing screenshots, the hit rate wasn't exactly stellar even with lots of training.
Compression artifacts from streamed gameplay probably won't help either.
what would motivate one to use screenshots over clipboard (besides the QoL perhaps)?
@static dust Apart from the "you don't have clipboards in streaming", you also don't have access to the Advanced Mod Descriptions.
A Ctrl-C export bakes stats together.
regarding the streaming: sorry, I just noticed.
regarding the Advanced Mod Descriptions: you mean, if you hold down ALT..? Yeah, I can see that
@static dust You could perhaps write a script that ctrl+c the items opens the steam browser, navigates to a url and posts it there so it can be acessed. The problem would be that this task is really slow on streaming and it would take the user screen.
I tried working with opencv just to get the text of the top rectangle of the item that contains its name so that at least non-rares, like currencies, fossils, cards, etc could be quickly priced, but I wasn't even able to get that, it should be possible it's just that my rectangle detection algorithm and ocr is probably really bad.
You might find some useful scaffolding in https://github.com/Marginal/EliteOCR
just curious.. does anyone know, if there is any significant player base playing via streaming platforms?
I would imagine hardcore players (and perhaps players in general) having a hard time with the inevitable delay
it seems to be getting more popular, some people do it to play without stuttering and other issues that are hard to figure out how to fix locally
I'm half considering trying it, I'm having stuttering issues again this patch
I play with it, I'm like in a considerable non-optimal ping of over 150ms and it's fine most of the time, I don't perceive the delay exactly cause there is just too much happening on the screen, it's mostly noticeble when moving items in the stash, an issue that sometimes happens is a small dsiconnection that freezes the streaming for about 2s, you can still control the character you just don't see it. But in optimal conditions I doubt it happens.
@burnt terrace what service are you using?
Geforce now.
I am using the Nvidia Geforce now, as my computer just cannot handle the game really well any more
my computer can run Cyberpunk 60fps but apparently not PoE
im motivated to try it out myself. would be cool to play on a neat ultralight laptop as well
yeah, its very useful since I am on a laptop as well. There are a couple annoyances, but nothing I cannot get around
@golden bane in reference to the code in the most of each of the skills for spectres abilities, can you explain the 2nd column of values in the levels= array?
[1] = { 75, 0, 26, baseMultiplier = 0.75, levelRequirement = 1, statInterpolation = { 1, 2, 1, }, },
[2] = { 75, 0, 26, baseMultiplier = 0.75, levelRequirement = 19, statInterpolation = { 1, 2, 1, }, },
[3] = { 75, 1, 26, baseMultiplier = 0.75, levelRequirement = 20, statInterpolation = { 1, 2, 1, }, },
[4] = { 75, 200, 26, baseMultiplier = 0.75, levelRequirement = 84, statInterpolation = { 1, 2, 1, }, },
},```
the 0 0 1 200
That should be the damage effectiveness modifier
would that be the Deals 101% of damage value found in poedb
It would map to that, yes
Each value there maps to an entry in stats or something, and statInterpolation indicates the kind of interpolation?
and therefor at level 84, the damage effectiveness is 300% (200% increase) ?
So you have static 75% skill_physical_damage_%_to_convert_to_cold, linearly interpolated active_skill_damage_+%_final, and static 26 arrow_projectile_variation?
(note that I have no idea what I'm talking about, just asking questions)
I don't know this off the top of my head, but the values in statinterpolation are something like opcodes iirc
Squinting at that, active_skill_damage_+%_final is linearly interpolated from 1 at level 20 to 200 at level 84.
given that we have this:
stats = {
"skill_physical_damage_%_to_convert_to_cold",
"active_skill_damage_+%_final",
"arrow_projectile_variation",
should it even be scaling?
shouldn't it just be 1 (101%) for all levels 20-83
according to w/e comes from the ggpk (which is where I'm assuming these breakpoints are coming from)
What are you consulting for that value?
poedb?
Only chuan can answer how those spectre level previews are generated.
well, POB has the same breakpoints (at least for lvl 84 and non-level 84)
Idk why there are tabs for spectre model level 68,73,78,82,84 on POEDB
I'm assuming those were breakpoints in days past that have just been left (as new ones were added for current coding)
other than level 84, the information for all POEdb model abilities at each level scaling tab (as far as i've seen)
Hey how does path of building get around cloudflare when making requests to get-items?
import requests
url = R'https://www.pathofexile.com/character-window/get-items?accountName="druuu"&character="dru_ritual"&realm="pc"'
accountNames = requests.get(url)
print(accountNames.text)
resp [403]
use a proper user agent header
Where "proper" includes your tool name and contact information, for when it misbehaves.
just tried, and it works without too
I guess you're not adding the POESESSID cookie
Still, the way to "get around" any protection is to adhere.
launch:DownloadPage("https://www.pathofexile.com/character-window/get-characters?accountName="..accountName.."&realm="..realm.realmCode, function(page, errMsg)
This is from POB. I'm not really familiar with LUA but I don't see any class for launch or DownloadPage
and it doesn't seem to be something native to LUA, so where is it defined
Probably defined in the runtime provided by the EXE.
@rare gulch Defined in Launch.lua
thanks so much
@burnt terrace I use OCR on in-game screenshots to get the list of map mods... it works quite well, maybe my approach could be adapted to your needs?
- scan for a solid horizontal line of the correct border color (rare yellow, unique orange?)
- scan line downwards until end of text box found (pixels are no longer dark enough (brightness < 20) or of a known text color)
- OCR the box, matching against a list of permissible text (stats, mod names, etc.)
Ugh, materials are a mess, ran a probe for all the stage dependencies: https://i.imgur.com/knxT7wO.png
It's Lua, not LUA (it's a word!) and Lua only has support for prototypal inheritance
I just noticed there are back-arrows, ugh.
@golden bane Today I learned how to debug PoB and how Lua works 😄
https://github.com/PathOfBuildingCommunity/PathOfBuilding/issues/2020
@worthy cape did you enjoy looking at the spaghetti?
If I had to pick a time of day to look at this, the hours of 3-7 AM were probably the best 😄
It was much nicer once I got around hooking up a debugger instead of spamming the console, but got sidetracked by VSCode showing the wrong variable for nested locals.
any word on how long these rate limits are intended to stay, if not permanently?
Actually starting to make good progress on PathOfServices, successfully got the whole OAuth redirect stuff working, cookie based authentication, SignalR working (with authentication as well, turns out it automatically authenticates by default so thats dope), and the core framework for my single page vue app functional.
Now that all the boilerplate is done, I feel like I can actually start making progress on actual functionality and whatnot. Yay!
Is there any data file that provides the name of the unique boss in each map?
WorldAreas.dat
in character-window/get-passive-skills api, it had jewel items, but how to know what the jewel's nodeId
@velvet fog Jewel's x property is an index to the jewelSlots array in passive tree json
@obtuse citrus any status update on pypoe for 3.13?
I'm pretty much on break in regards to pypoe & wiki updates right now, so unlikely anything till march
"patches welcome" 😉
@broken cloud thanks
@velvet fog Not sure if the spectre dude has talked to you about this, but what kind of interpolation are you using for the skills of sentries like https://poedb.tw/us/Redemption_Sentry ?
The Deals 101% of Damage/Deals 300% of Damage stats of MeleeEyrieArrow has interpolation type 2 which PoB considers linear interpolation/extrapolation, while your spectre examples seem to treat it as type 1 (static).
@velvet fog btw the buff consuming bug is back again, this time for helmet mods:
(second stat is consumed by the buff, and therefore shouldnt be rendered)
heya this might be a dumb question, but I was wondering if anyone made a readily usable script/program that can convert the current-day .smd/.ast/.fmt filetypes to ones that can be used in a regular modelling program (whether it's .fbx/.obj/.abc and so on doesn't matter) 
i found some scripts from years ago, but they don't seem to be working anymore
Short answer, not quite. Long answer, not by far.
Raw geometry from meshes is reasonably well understood with submeshes and material indices.
Skeletons and animations are somewhat mapped out but I don't think anyone has attempted skinning, I know I screwed it up good in my previous^2 codebase.
Materials are quite well-understood but do not translate well into a traditional static scene kind of world, as the primary way a VFX/texture designer controls the appearance of surfaces is to hook into the renderer via programmable effect graphs (noodle soup-like blueprints) in materials.
I'm currently messing around with reimplementing some of the renderer and trying to compile materials into HLSL shaders but it's nowhere close to done.
I had an .obj exporter for the raw geometry way back when, chuan has something similar for poedb's model viewer, and poeloremaster has his own implementation feeding into whatever evil drives his web UI.
For static meshes you're probably "OK" with a straightforward export but will miss out on a lot of material properties that come from effects, I haven't really looked much at what kind of materials regular doodads have.
Ahh thank you very much for the answer, and makes sense yeah, especially concerning skinning & materials, I'd assume even more so for the latter since the gradual switch to a PBR pipeline
I was curious because I wanted to make some 3D fanart, and for a lot of things there aren't that many easily usable references, it's much easier to have the original model just right besides what I'm working on etc.
Especially for smaller things & characters like Dominus, where for example I'm not sure whether his blue head is his face, or a mask
Do you have a link for your .obj converter
i can only find an old archive of some converted models, and an .rs script I'm not entirely sure what to do with it
I don't know which iteration of the codebases I hacked that up in, heh.
.rs sounds like Rust, so probably my poe-rs but I don't know where in that mess it'd fit in
Ah, probably this one - https://gitlab.com/zao/poe-rs/-/blob/master/tools/src/bin/poeconvert.rs#L30-38
Yee I was about to link it, that one yeah
This is very broken now as it predates the tech patch where the filesystem was all changed around.
ahh we'll see
thanks for your help regardless :D in all honesty I was surprised there isn't a singular tool that deals with this, as there are for many other, less popular games, but it looks like it's for a good reason
Looking at some .fmt files like Art/Models/MONSTERS/Dominus2/dominus.fmt they seem to have since also updated the file format version to 6 from my supported 4.
It's definitely on my roadmap to figure out any updated mesh formats again, but right now I'm quite stuck in the material soup.
There's no rush hahah, it's good work that you continue to put out :D
Can't you just squint at this reference? 😛
ah yes the one hex atziri's shield doesn't protect against
:')
I honestly can't wrap my head around how people figure out how to convert that to workable filetypes, I occasionally read the magic that people do on the Xentax forums, but it's just witchery to me hahahah
What are things that you're stuck on concerning materials
I can't program to save my life, but I do know some things about PBR/shaders
maybe i could be of help with that
It's mostly a matter of glue I believe.
I've got graphs that hook into different stages of the renderer, which read a bunch of parameters and inputs and produces some outputs, all sourced from artists-controlled constants or stage input/outputs.
I don't know much at all how the stages fit into a frame and the drawing of geometry, and there's some interesting circular dependencies.
I drew out all known stages: https://i.imgur.com/knxT7wO.png
Here's an example of part of the celestial nebula hideout background: https://i.imgur.com/453f3qZ.png
The building blocks that an artist has is graphs like that that achieve some sort of incremental effect on the surface, like adding a bit of fresnel rimlights, shifting UVs over time, distortion based on textures, things like that.
Or heresy like that celestial graph 😄
Ah oh jesus you're mapping out the entire rendering engine flowchart, that's mega brilliant
hmm, I know how to mostly recreate shaders in other engines (such as Cycles in Blender), but that doesn't include game-engine specific things like per-pixel effects
If you're curious to the workflow and the tooling that the VFX folks at GGG uses, they had a live demo at Exilecon - https://www.youtube.com/watch?v=KxXJn1DOuzw
A talk by Ivan Leong, a senior visual effects artist at Grinding Gear Games. Recorded at Exilecon 2019.
If that flowchart is how it works in the PoE engine, that's some very smart programming
Was kind of fun watching that video after blindly understanding some of the formats, the tools looked so familiar 😄
Ahh yeah i've seen that one :D it gave me some ideas on how to "cheat" effects like the normally expensive refraction on the crystal within other programs
they've got some really smart people working there
base_fixed_blackc_RefractionAlpha_addnormal_FresnelAddTex_expand_MaskedAddTexEmissive_Fade_Time_Tex_projectiveuv_expand__pixel:6493:10: warning: implicit truncation of vector type [-Wconversion]
The generated shaders are not exactly tiny, I'm guessing there's quite a bit of glue and boilerplate in there.
Ahhh yeye
The only thing I'm not sure about what it does is the expand__pixel function
but the rest seems like the good/standard workflow
or is it projectiveuv_expand
which would make more sense
I'm guessing the filename is just the individual graph names glued together as part of generation, this is from the game logs so I don't have the actual "file" to look at.
Would be much easier then, but I don't touch the game client.
Yee
Hmm, maybe you could contact people like Durante from ReShade, or Boris Vorontsov from ENB, they do this kind of dissecting almost for a living
In any way, I'm not quite sure what the end-game would be for someone who would like to use assets outside of an interactive viewer.
Even if I get a somewhat approximate renderer going, that won't help that use case much.
Yeah
I mean I'm fine with just the models & textures as they're only for reference use anyway
Shaders can usually be approximated in other programs should someone want to go that far
What was the cause of this issue with textures again? I feel like I remember people tangling with this before. Did we come upon a solution?
path of this
should be Art/Textures/Interface/2D/2DArt_UIImages_InGame_AtlasRegions_AtlasTrees_1.dds
wait no thats it
hold on im confused, My dir is a mess
2 secs
worksforme, except that xnview loves to sometimes not render images correctly on first load
going to do some conversion before viewing & check back. maybe xnview's dds viewer is confused
although it works for map items
I've had it occasionally garble images on load, but tending to fix itself later.
xnviewmp, that is
Ok, I believe that it has to do with some special formatting that ggg does that the exporters you used takes into account, but I did not
because when I export w/ libggpk2 to png I get a fine image
but when I try to manually convert the dds to png, i get a "corrupt file" error
Is there some processing/formatting of the dds that has to be done before exporting?
The .dds files are compressed with Brotli
That shouldn't matter since the tech patch.
Everything between the tech patch and texture streaming should be either a plain DDS or a symlink.
Right, forgot about that 😅
Everything since texture streaming should be a .dds.header with a proxy texture or a symlink; or a plain DDS texture.
What are the 4 magic chars & the top of the file spec before the dds header?
version, fullsize width, fullsize height, payload size, followed by that much payload.
payload is either a tiny DDS file to be used as a low-LOD placeholder or a *Art/blargh type symlink.
Cubemaps and other non-tex2d DDS files don't have proxies.
Depends on what you mean.
*.dds are plain DDS files without any external compression but the Oodle from the bundles.
*.dds.header consist of that fixed header and payload, if you want a contained proxy DDS there, you need to strip the fixed header.
That DDS (and DDS.header) as exported from a recent version: https://zao.se/~zao/poe/2DArt_UIImages_InGame_AtlasRegions_AtlasTrees_1.7z
looking. Somewhat confused.
So those chars only appear in *.dds.header files? or am i misunderstanding?
Or that is a part of dds spec and i misunderstood entirely
Hey, I need serious help getting to endgame/bossfights, iv got some T15/16 maps but im struggling understanding how i should be going about it, if someone can help me i can stream my screen
@lean spindle The actual DDS files named *.dds are as vanilla DDS as possible, there's nothing special about them at all. They were more special in the past as they could be both Brotli compressed or textual symlinks.
Not the channel for that, friend. @mellow ice
Try out the 'help' categories for assistance
Ok. icic
The new texture-streaming related *.dds.header are present for all regular texture DDS file paths. They always contain the fixed header, and then follows either a tiny embedded DDS file, or an * followed by a path to the actual texture it references.
Also, the dds you sent me is failing to display correctly in the same way as before. Most strange. Almost definitely on my end, I suppose
AH
I understand
That actual texture in turn should have a .dds.header and possibly a corresponding .dds file.
I just had a vision of symlink hell, but ty for explain. This makes sense now
An update of xnview fixed viewing problems. Now to figure out why the image format conversion lib I'm relying on is throwing a fit.
sometimes its the simple things, it seems
Tech patch upgraded pretty much all textures to BC7, so if it doesn't support those "new" formats it might get a bit upset.
Ah, I see. This gives me something specific to look for, tyty
@compact isle today's trade slowness was strange because Russian was affected too. are you making some internal maintenance and decided to tmp block APT?
@civic crane https://www.reddit.com/r/pathofexile/comments/l5ev9b/rip_trade_site/gkuveft/?context=10
602 votes and 255 comments so far on Reddit
bex comment confirms they blocked APT
Comments also mention Poe Overlay as well. Could be the other tool causing the issue but yours got hit in the collateral.
I think the comment is suggesting poe overlay as an alternative, not saying it's part of the problem. poe overlay community fork still works fine for me
doesn't parse to me like the app is an issue just that lots of people use the app?
it parses to me that they want it to have rate limiting or something
it has rate limiting?
does it? I only use Overlay Community, which does
Awakened PoE Trade does have rate limiting
"We’ve temporarily blocked it to fix the issue and spoken to the developer to let them the know the problem."
= they think it's doing something bad, and the bad thing can be fixed
apt had an update 4 days ago which affected how bulk trade requests are made. Might simply be a bug which manifested heavily today for whatever reason. I, for example, got the automatic update to that version today.
yes, APT rate limits and checks the rate limit state to make sure it doesn't go over
Seems unlikely that APT would suddenly bring the server to its knees at 9am in the morning on a weekday when it hasn't been a problem for the first week and a half.
on the topic of rate limits, they're ridiculous as is. Need a better long term solution than steadily increasing the rate limit every time there's a problem. Idk how many times I've refreshed a couple of search tabs and accidentally gotten myself throttled for 60 seconds because I can click refresh more than 3 times within 5 seconds.
And how'd that work out for you?
The river is unusable. Literally and figuratively. The only way you can get whitelisted is if you're big enough and if you're big enough to get whitelisted you're competing directly with ggg's own trade site.
yep
certainly they ought to invest in better infrastructure, I don't disagree
I'm including architecture in that
probably hard to invest in new development with closed borders atm, but hopefully it's something on the long term roadmap
instead of continually punishing users / tool devs
I think the main issue is it's hard to measure the ROI
how much business are they really losing by having a bad trade experience
probably not that hard to see the impact when they block Awakened or whatever
if it stays blocked long enough, yes
my guess is they'll come up with some bandaid and unblock it in a day. but that just kicks the can down the road.
i am a bit amazed at their post saying they didnt expect a 1x% increase in traffic
for a huge release
doesn't sound good for the future
I didn't think they said what % capacity they were actually at before launch. I think the 1% was just part of the example how you can be "fine" operating at 99% and then suddenly not fine at 100%+
but it is a good question, if they were close to max capacity, why didn't they address it sooner
in the case of the database overload at least, they didn't see it coming seemingly
@civic crane what have you learned? Is this really as simple as "too many people using trade macro"? Why wasn't this seen earlier last week?
unfortunatly yes, it's just many people. there can be introduced one or two setting, maybe some cache for bulk price checks, but you generally don't think about it when api just works (like why would you implement client cache insted of server cache)
but blocking one trade macro isn't going to fix anything, people will just switch to another like the older versions (which are still working).
how do they selectively block your app anyways?
user agent
and noting that its due to a third party tool only brings out reddit experts on the case, blaming APoET for everything regarding trade
so you're considerate to use a unique user agent instead of spoofing chrome's 🙂
yeah see that gets you in trouble
how so
isn't that what poe trade macro does?
I'm sure the pattern of usage of apoet is unique as well so they'd catch on
doubtful
i dont think they have the resources to spend to analyze that
it's default electron's UA user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) awakened-poe-trade/2.9.0 Chrome/87.0.4280.88 Electron/11.1.0 Safari/537.36 I decided not to change it, because you know... there is Cloudflare in front
I'm not advocating for anything that would make their job more difficult
but a real solution will need to be found
and this isn't it
yeah, im just saying that blocking it is doing exactly nothing
it is essentially the same as not responding to any queries or closing down the api etc
all it would take is one person from the community to fork APT and change the user agent
all it does is shift the blame onto someone else
instead of smiting people, theyd do well to figure this out once and for all
But Reddit has a new culprit now so all is well
in other news, it's snowing!
wow that thread hurts to read
is truly painful, yeah
tldr - who messed up?
Combination of things, really
GGG has had issues with their trade system & databases since league start
This compounded w/ the fact that Awakened PoE trade doesn't do local caching for repeat searches (and also the fact that its spiked massively in popularity) led to ggg having increased troubles w/ trade systems
culminating in them blocking Awakened PoE trade atm
at least this is how I understand it
overall, increased traffic that ggg was unprepared for
and this is their bandaid 'fix'
Awakened Poe, got it
hey snos, as they say... all pr is good pr
time to make some memoize mimosa
because there was no practical reason to, now we have Ritual window with rewards. anyway, for how long to cache (default value)
until application restart would be a fairly safe bet
couple of minutes to account for people finding another broken interaction, prices spiking/dropping and it getting nerfed causing prices dropping/spiking
/s
but yeah app restart sounds good
i remember reading somewhere that you considered disabling auto search
i'm sure people will have a problem with that, but u know..
thatd be good aswell, as for a lot of things poe.ninja is a safe indicator
rasmus updates poeninja every 15 minutes
Yeah I don't think it was an issue before this league, really
do you already cache poe.ninja prices btw? too lazy to check, but id assume so as it shows up instantly
Local caching doesn't have to be time-based
just x most recent searches ought to do it, I think
and clear on app restart, yes
this reddit thread needs popcorn
@simple ravine the reddit thread needs

It's extra amusing when they crack open the "distributed system" term without having a clue
tbf it's pretty hard to make an accurate assertion even with (some) degree of technical expertise, but its alright to make an assumption i guess
I think maybe having a "search" button that you click w/in APT instead of the "hover here to search" and "search on each query field change" could also reduce number of trade api requests made by a hefty sum, at relatively low labor and user experience cost.
with complex systems it can be hard to make an accurate assertion even if you own the code
Isn't this what cloud architects like you do all day? 😉
imho, regarding apoet, hover to search should be fine, if i was a betting man id betthat most people are checking currencies/fragments multiple times back and forth and hitting the limit because they have goldfish memories (i dont do it i swear)
scaling always easy when there's no existing technical debt that you have to work around, but that's never the case
yes, hence why this is kinda amusing
if APT returned poe.ninja prices for items instead of the current listings, it would be unusable for a large number of people
poe.ninja prices tell you ballpack, this thing is worth small/med/large amount
but they don't tell you enough to properly list the item for sale
but there is no need to search the bulk trade every time you check a fragment price
yes but that's not even the common use case
without statistics i have no way to know, my sample is 1 and most ritual guides tend to focus on how people overemphasize spending on currencies/fragments
and even in that case, if you're listing fragments and poe ninja tells you they're worth 4c each and there's 10 people listed at 3c, you're gonna sit there and never sell and wonder what's up
but the point is apoet doing the search automatically
if you want to sell it, you are free to move the mouse over the search button
hmm, so proposal is differentiating between price check of "is this worth somethign" vs get current listings
I don't like the idea of making "get listings" be manual, that's just inconvient and doesn't stop anyone
but I would suggest having a separate keybind to get listings and point people to price check as the default
maybe even put a heavier rate limit on get listings
its already sorta manual
right
you do?
I'm proposing, that for filtering based on item stats
you dont search every time a box is checked
what do you do?
you search when they hit a button in that instance
have to mouse over hte item you want to search for and then hit the key
so i mouse over a fragment, hit the search key and get listings
do i unconsciously perform something else?
If you move your mouse below the basic info it grabs initially, i think it then searches listings. or I'm hallucinating, one of the 2
well the author knows for sure
in any case, having to manually confirm sending a query would cut back a lot even after implementing a cache
Agree, I think its worth doing
if cached, show immediately, if not show button
How often do you think people are searching for the same item? If you're mapping and price checking every item in a ritual, most of those items are new.
my memory is so bad
or do you mean cache ALL of poe.ninja's data
i price-check the same item multiple times
which isn't hard to do
when tinkering
My memory is good but short.
what is memory
@wind garden currently talking about locally caching previous queries made by the user
something you'll gradually lose after 30s
ya I don't think that would be sufficient
im just randomly trying out stuff i think would produce the biggest result
apparently
and then i have to reconfirm previous tries because im not sure if this one was highest
I think caching poe.ninja data would help a lot with price checking uniques and bulk items
Yes, I think any item that is non-random-rolled would benefit from serverside caching
but that takes a bit more work
"is this essence one that's worth having at all?"
but poe.ninja hits dont go through to trade api do they
as opposed to the "how much is the cheapest listing for this item?" query
so thats a separate issue
There's either that kind of fetch, checking the general range of an unique (again ninja territory), and then the tricky bit of rares where poeprices.info (or whatever the guesstimate source is) rarely helps.
But in the last case, you have to manually trigger a search for mods, don't you?
I'm somewhat sure that if you cache rares, you'll decrease the total volume of queries a fair bit as well
price checking rares definitely shoudln't be automatic
@long edge The point is that several types of lookups don't need the trade site API, despite hitting things like the exchange unconditionally now.
you pretty much always have to check 1-2 mods to lock in
yeah thats what i was getting at earlier
(I forget if APT culled down on the exchange lookups, it was a while ago I saw it mentioned)
jUsT iNStAlL mONgO dB
If I was forced to bet, I'd say they're probably using a centralized server that every user talks to using the trade page. The standard approach to fix this problem is to use a cascading net of hierarchically linked servers where first the central servers talks to some proxy/cache trade servers, who again talk to some proxy servers, who again (... repeat for as many layers as you need), where only the outer leaf nodes are actually exposed to users, so you can scale the actual searches and such very easily.
This trivializes search performance. For listing items, the central server still needs to be able to cope with the total amount of stuff users want to list, but that doesn't appear to be a problem for this game. For searches, end users only talk to one of the leaf node proxy servers and you can scale as many hear as you want to satisfy search performance.
PoE has an exceptionally trivial case of this problem as well: It's not a problem if one server thinks an item is still listed for a few seconds longer (as there is going to be a small delay until all these servers have sent and received their messages). From the user's perspective this will result in an additional whisper while the item is being traded to another seller from time to time, but it doesn't lead to data incorrectness issues or anything like that.
lord jesus wtf this guy
you love to see it
is this really a bot? quite aggressive
just trying to help lol
xD
you know what would be cool is a "tribute <=" filter function
could save some queries
how would that work
like BaseType == "Stacked Deck"
Tribute <= 1100
then it'll show on the filter rule in the ritual menu
tribute doesnt behave like most currency, as you can't save it
just for the menu though
I don't think I understand
Your item filter has rules added to it for tribute so that it works in the ritual window
on top of the existing stuff
I've had higher stack sizes of the same item in the same window offered for less tribute.
As an aside, I am super curious whether people have been writing down the base tribute costs they're seeing for div cards and uniques. Could provide some insight into the game's internal concept of rarity "tiers" (which might or might not actually be tiers)
only for basic currency mostly
my approach is just to figure out what a chaos orb costs baseline
and then price everything else off of that
Are you seeing any variance in the tribute cost for a single chaos orb?
Or is it the same every time ?
There is definitely variance, maybe 10-20%.
yep, variance on everything
Disabling initial search and setting debounce time to infinity was the first thing I've done with my PoE Overlay (now community fork) when these options became available. For me it's more convenient even w/o considering the API spam part.
To quickly assess whether the item is worth anything or not the poe.ninja data is enough. Getting more precise pricing requires selecting specific important stats (or minimum stack size for stackable items) manually anyway, and for obvious junk items you don't make a query at all.
Care would need to be made around Vaal rituals for data-sourcing, as they used to be able to both inflate and deflate by a larger amount, and since patched only deflate.
Request time wise or time till you see recent items(timeliness/actuality of the data from the latest change id)?
I updated a price and zoned at 18:33 with a live-search running on poe.trade, it's 18:57 and it has yet to trigger.
19:06, item appeared
I googled a bit and apparently Tesseract should be able to segment by line by itself, but I'm not quite sure how to achieve that. People are linking research papers left and right and I'm a dunce. Anyone perchance has any experience with it? Ultimately I'll do the segmenting by myself (indicated by the visual representation there) aswell but I wanted to compare results/performances. Related example image of testing phase:
for example doing it without invert results in 1.21 runtime
so i might get significant gains if i dont segment myself for no reason, or if my segmenting is less expensive
Maybe they took that reddit guy's advice and added a network of layered cache services 😂
jokes aside though, it's starting to itch a little in the solutioning-fingers after that reddit debate and "problem" the other day
can't see any performance issues for the river in the graphs
yeah no issues on our end for the past 12 hours
I'll dig a bit deeper but these analytics have been accurate to problems in the past
I only have poe.trade to test with, but as mentioned above, it took from 18:33 CET to 19:06 CET for an item to appear on their trade site livesearch, 33 minutes; compared to the 8-12min it has taken earlier days.
I see a dip in your requests but no dip in our response times
How does CF factor into things, do you have any stats on whether they do funny things?
they don't have great analytics unfortunately
Current whoop latency - 10.5min
can totally confirm
@compact isle what data rates do you have in peak hours vs average etc? Curious
e.g. how much does the data weigh you create per second or hour or something
those aren't super easy for me to collect for specifically the public stash API
hmm, more like how much data do u pump into the databases or something?
that's a strange metric that I don't collect
there does seem to be a strange discrepancy between PHP's processing time vs nginx's total request time
i'm just in the mood for some tinkering, but wanted to know like... how much data is being added / updated to understand the magnitudes
the api needs a higher max item cap
we could try higher payloads
my implementation is basically fetch new data in a loop w/o doing any processing and its impossible to keep up
high nginx total request time suggests that it take around 2s for the response to be completely streamed out
not sure how cloudflare interferes here, and if that 2s is to cloudflare and then extra time to you
yeah I'm keen to reduce that
i think if it returned 10x amount of stashes it would be easier to catch up
also easier to spot issues when it takes too long
that means 10x the PHP processing time which would be... about 5 seconds per request alone
yep it would be a very expensive change
might be a good optimization target
Can one (and do you) look at the IDs from where you're in the river at the time, vs. what's known to be at the front of the river?
(by occasionally sampling at the tip)
Or is that normally not a problem when you're caught up?
(I guess that individual ID parts (shards?) do not necessarily correspond to update counts, but still)
you can't get caught up...
time to first byte around 2s, if at that point you rip off change id out of first 500 bytes or what not and send request for that change id
you still can't cathc up
are you guys using https or http?
the discrepancy could be TLS handshake to cloudflare
most requests take no extra time but some seemingly randomly do
unless you hit different edges
the request time does get larger for both of you during peak but it's much much worse for poeapp
for around the same number of requests per minute
tried both, no real difference
the holy trinity of apps
ninja also has less requests but doesn't exhibit this request time issue as much as you @polar island
I am glad that I don't have to be that up2date like @polar island
less things to worry about
it is a metric fuckton of data to keep up with
metric fuckton. now we're getting closer to something, Novynn 😛
imo adding some auth to make more requests is the only "easy" possibility. But that would also stress the server further
poeapp / xyz don't have any request restrictions
10x stash tabs pls
oh
yep I'll bring up increasing the limit
maybe with parameters
I'm interested for us to support requesting from individual shards too which is 5x the throughput for free
I think I made one of those comments a while ago, i recollect someone in here wasn't super happy heh
but you technically kinda could do something like that today, by having a very high number for all but one shard in that id
yep the idea wasn't internally liked but I'll push for it
but yeah, you'll likely just end up with the same issue
of course, eventually nothing we do will be sustainable for third parties to keep up
let's hope there are some trade changes before then 😄
unless people are smart enough to detect that shards require varying RPS
inject the river straight into my veins
i still want some numbers, because I like the problem
here are 400 map stashes that people public with no intention of selling maps
injects
sounds like a solid solution
would be nice if i could inject a specific person's map stash into my veins
@simple ravine At least 30.
yeah we're pretty sick of waiting for OAuth stuff to be allowed out lol
it's easier to ask for forgiveness than permission 
not when we're waiting for legal ToS update stuff :p
what's the worst that could happen 
last I heard it was a matter of weeks and not months so yay progress
and that was some months ago
😄
Any decade now.
maybe oauth isn't a thing anymore when they allow it
@earnest radish oh yeah map stash search is a mess, I wish I could say it is an easy fix but boy is it not
mm legal
reminds me (not happily) about our public sector customers
cloud and gdpr and swedish privacy laws. mmm...
I keep getting pulled away from OAuth stuff to put out server fires so you can thank snosme for any delays 👍

i dont understand data talk so i'll sidetrack for a bit, what is the channel to request additional filtering for the frontend part of the official tradesite? id like bulk-feature of minimum amount but im loathe to hack the frontend filtering myself
luckily all we needed to do was add 3 layers of caches in a complex network of AWS proxies that only hit our centralized server if they absolutely must

easy
minimum amount of... stock? That already exists

mmmm wording, id like to apply it to item bases, gems etc, basically results from non-bulk market
we don't track stock for non-bulk items
then again we have to rip that whole system out at some point so maybe we can sort something like that out
so i guess if i want to see it soonish ill have to do it myself, pullshair
such is the developer way
i just want to play the game
someone watches good tv shows
You can play while programming
I need a project manager and an UX person 😄
UX persons are rare
Also self-curse temporal chains for more hours in a day.
I would sometimes need an int stacking build
I actually have an UX person I could ask for feedback, but that means getting to the point where I have UI, potential users, and something resembling a product.
What's this for , if you don't mind ?
My long-term hideout editor project.
Right now I'm shaving the material yak.
@minor charm Base idea is to have an editor that has a bit richer representation of a hideout, where you can group decorations, more intelligently placed decorations, have prefabs, hideable/freezable layers, stuff like that.
A project manager (or analogue) would help with spreading myself way too thin doing things that are interesting but are not furthering the core idea.
Like trying to render things correctly.
and then import it in-game ?
Yeah, then have exports to the vanilla .hideout format.
ahhh nice one 😄
You could have alternative decorations to replace MTX-only ones, export a light-weight variant if you like FPS more, the sky's the limit.
export to autocad would be nice so i could build a nice hideout in my backyard
There's a lot of interesting room to improve on what is currently available in-game, and I'm sure as heck are not the person that's good at it.
@long edge STL files to 3D-print your hideout 😉
now were talking
now I'm interested
real life hideouts
holy smokes 👀 now that's an idea
can it print all the pretty particle effects
Did I mention the VR support I had a few iterations ago?
damn
"any decade now" is probably a reasonable timeframe here
printing custom hideouts would be quite profitable for ggg i reckon
shipping anything out of NZ is a pain
How would the Celestial Nebula hideout look irl?
Why limit yourself to hideouts - print iconic events in PoE history, like when <streamer> gets oneshot to Standard.
The Celestial Nebula hideout groundtiles can go hecc themselves.
Have you seen the material for those?
the kinks can be ironed out along the way, im sure there would be no big issues
no 😄
Part of it - https://i.imgur.com/453f3qZ.png
Most other fxgraphs are rather straightforward in what they do. This one ehm...
interesting
@chrome topaz can you add me on discord?
sure
I uninstalled Skype as soon as I possibly could 😄
Right now I'm trying to sort out shader source code generation, but the game eats time like candy.
and switched to teams
😄
nah surprisingly we managed to convince everyone to be on discord or slack
I had to install Teams to be able to talk to our Contract Manufacturer 😦
Teams isn't bad tho, I just feel like they often have server problems
trade would feel right at home there
I sit 5-7 hours a day in Teams meetings. No issues
Teams probably doesn't have any trade macros
maybe it should, zensei wouldnt have to sit 5-7 hours a day then
no, but 14
it could Watch_Dogs-style look up someone's information so you can assess if you want to use your tribute on them or not
Now I have to admit, I only made it to A1 in current league, and have no idea lul
wait is this how people do ascii art
trade could be temporarily mended by hiding ritual rewards (like they are at the start) and slightly increasing tribute count. that way it will be a gambling simulator so not any different from the crafting part of the game
pl
am i rich
Slowly coming together :3
Ive got a solid bead on the implementation I will be going for on the user rank system, which I think will be a four part system.
- Vouches, how many users have vouched this user
- Complaints, how many complaints they have (can click this number to go to that users profile and see all complaints filed on them by other users)
- Distance, how many steps it takes across the network to get from you to them in trust. Will just say '---' if its greater than 8
- Rank, a number from 0 to 100 which represents their rank via a hidden ranking algorithm. Im opting to not expose the algorithm for this and will do it via a Stored Proc periodically under the hood. The main reason being so that individuals cant min/max and attempt to 'game' the ranking system. The rank is based on all the various info for points 1 to 3 of course, so users are encouraged to focus on all three.
Thoughts?
why not just integrate with TFT discord
I will LITERALLY never trust your user ratings.
No offense.
With TFT you get join dates going back years and people with thousands of vouches.
Could be a good idea to account for value of items used in the service. Since a user could do 100s of low cost trades, and then get 1 negative on multi-exalt service.
Join dates and vouches also mean newer people are left in the dust a lot of the time so TFT isn't that optimal. Should be getting worse and worse for new people to gain rep
I actually don't care if newer people are left in the dust. That's high risk high reward. Those players should be forced to offer services at lower prices just like the vets did.
When the barrier to entry is a new discord account that's how it should be quite frankly.
Oh yikes, I wasn't aware of any drama.
I used them extensively in harvest
and had no issues
There's also the PoE trading discord which also has similar systems.
@earnest radish maybe use the other trade discord then?
@rapid pike thats what the other three vectors are for, and you should see that the rank of the user is pretty reflective of the other three scores, it is purely meant to be used as something to sort the results by, the other three vectors will also be visible
Also I already chatted with the TFT team about integration and they declined to expose their stuff, they require integrated apps to be TFT branded. I wanted to make my app open source such that anyone else could host their own instance of it (on that note, TFT absolutely can host an instance of my app and integrate it into their system, which we already chatted about and I plan to expose tooling to make it easy for them to do that)
Basically my goal is to have something like a group of friends, a private league community, etc could spin up their own instance of the app of their own for use within their circle, and then I will be hosting my own as well likely which I expect will be the "primary" one folks use, since theres a bit of effort required to host your own instance (you need to host a front end web server and a backend API connected to a DB, and probably come up with your own Ranking algorithm if you wish, though thats technically optional! Leaving everyones Rank value at null is valid and will still work)
@grave wren already accounted for in my ranking algorithm, Rank will probably be logarithmic in form, such that its easy to get your rank up to a decent value, but then it curves out. Theoretically only one user in the system should have a ranking of 100/100. It should, I believe, work out to be a bell curve I think, if I am reading these papers Ive got right.
it's impossible to fix trade anyways so making it (or a part of it) more accessible is always welcome
regardless, there will be people who game any system if its left without overseeing, and abuse can happen in any situation of power, so youre kinda forced to choose an option with least risk for yourself in your opinion
i had a hard time falling asleep after failing segmenting algorithms, so i kinda got the novel idea that instead of doing ocr i could request ggg to allow us to ctrl+c harvest crafts. what are the chances of that happening?
There's implementation-heavy solutions to services trading that avoid the "ha-ha your item is now mine" scenarios. WoW supposedly had the enchant-window, where you offer an item for someone to apply an effect to, but at no point do they have it in their possession.
yeah i meant it more like from a perspective of a tool dev
i dont think ggg relly wants you to trade services so theres that
Indeedily 🙂
A lot of the design of neo-Harvest is keeping some semblance of personal crafting power, while rendering bulk sales impossible.
Can you Ctrl-C anything that's not in an inventory?
(inventory being a more generic thing like a stash, inv, crafting box, 2D place)
Chat hovers, I guess.
There's no precedent for Ctrl-C things in the world, even items on the ground, right?
I'd estimate your chances to approximately zero ^_^
dont have poe open but i would imagine that pressing ctrl-c on an item on the ground would just open my inventory (bound to c), because i vaguely remember trying it at one point
that would mean that the chances of me successfully finishing this project are slightly higher than appealing for a change
thats nice
its hard to be stupid
Implementing real-world OCR is very educational, if nothing else.
every search query is chock full of research papers with formulae i dont understand, all i want is a "this is how and why you do it dum dum", not "heres a link to my doctorate"
yeah, image processing was interesting at least
i realised how i could do image comparison more efficiently than pixel by pixel
and last time i read on it (when i was trying to do an atlas overlay and gave up shortly after) it just sounded like some complicated mumbo jumbo because i hadnt tried processing anything myself
Ctrl-C on ground items, even with Key Pickup disabled, does nothing.
why cant we have nice things
Gah, now I can't remember which setting I used to have and none of them feels right 😄
what does key pickup even do
If enabled it won't pick items up unless you hold your highlight or pickup key (both default to Alt, I've rebound highlight to F).
Main game effect it has is that you can actually move around loot instead of accidentally picking shit up and dying.
Separation of binds is to avoid it showing all the irrelevant loot when pressing the button to pick things up.
hm, i think mine was disabled, as i usually hide everything if theres such a fight
default is z for that i think
Yep.
Fun side effect is that whenever I want to look at advanced mod descriptions I'm typing a stream of f into any available chat or filter box 😄
I use Z for visibility sometimes, and then forget to enable it for maps upon end <_<
Heyo. An trade api question here. If I would like to get only items publised today, how would I do that? I see that there is a indexed key in the trade_filters, but however I try to set it, does not seem to change the response in any form. I can not find a comprehensive guide for the trade api anywhere
The api based on the trade site. The one from here https://www.pathofexile.com/api/trade/search/Ritual
❤️ u are the best, thank you very much. did not know that that's how u should set it
The "guide" tends to be "whatever the trade site does to itself" 🙂
i tried with timestamp and everything
your best bet is doing the search manually and checking the network for query
Yea, forgot about that. Guess i got to used to documentations lately
Postman 8 sucks butts, pass it on~
im thinking of not training my own tesseract data because thats a whole other bag of "poorly documented for schmucks like me" i dont want to get into now.
rather than that, i'm thinking of using the first found word (for example: Reforge, Augment, Randomise, Set, Add, ..) to then find similarities within a subset of harvest crafts.
am i underestimating the performance impact of my own such implementation and is it instead worth the effort to get a properly trained dataset? I am aware that my filtering is basically what an OCR might do in the first place, but if its not too expensive id rather go with it than needlessly overwork myself on a simple project
@long edge just use Capture2Text. You’re making it too hard 😛
capture2text uses tesseract anyways
It’s what trackpete used in his old old exp/h tracker https://github.com/trackpete/exiletools-xph-macro/blob/master/poe_ocr_xph.ahk
Yes I’m sure it does, but my point is that if you’re having to consider training etc., you’re way in the weeds
The only remaining “challenge” is to either aggressively crop the image based on where you know the tooltip to be, or to figure out how to filter the text output to find what you care about
thats what i asked about didnt i tho
Sorry not sure what you mean. I came in late to the discussion so apologies if I’ve misunderstood what you’re looking for
in any case from user perspective it doesnt feel good to grab even one more app, let alone two. nah im grateful for the suggestion, but it seems that running everything through c2t cli is not that different to a correctly set up tesseract
Just trying to save you the time of training tesseract and reading dissertations
You can always distribute c2t with your application
yah, all good, i appreciate it. but i think id get the same result if i copied over the tesseract related files
the only issue is that my segmentation isnt good
otherwise it neatly grabs almost everything if correct image preprocessing settings are used
Curious how c2t does? They are likely doing some serious image preprocessing
That’s the “hard” part
You can't leverage something like that the line spacing ought to be equal?
Or can you anchor it to the surrounding frame dimensions?
You also don’t need the OCR output to be perfect. Even the simplest of string comparisons could identify which craft that is based on the output shown there. Something more advanced would be even better
thats what i was getting to there
Yeah that’d work well
but sometimes small L might be construed as I or ] or )
The box around the count and the level indicator, and the whole frame could probably serve some purpose?
I’m wondering if there’s some sort of lucene based solution here. Going to dig
see, i have no idea what lucene is
Some edit distance shenanigans?
@worthy cape im trying to make it work out of the box for any resolution
Yeah basically just edit distance
i guess i could even out the lines somewhat by detecting which segmented parts have only 0s in them and based on that it could sometimes work
The art is of fixed aspect ratio and layout, so it's one thing that's constant.
Char-by-char edit distance would likely have 100% accuracy
Lucene is a big-ass library for text search and related algorithms.
Yeah you don’t need lucene
Edit distance (Levenshtein distance) computes how close a string is to another if you would make changes to get from one to the other.
sometimes because at that point if i have to be perfect i suppose id have to use some other complex algorithm because if you have "type" then its going to have less zeroed-out lines below it than "ziz"
Take your match, match it against all possible descriptions, likely to find a closest match.
would it work out of the box
or is some additional filtering like i mentioned preferred
There’s realistically no way I see there being any ambiguity if you just use an edit distance implementation
Just steal one from SO in whatever language you’re using.
i agree with that and understand it somewhat, but isnt that what ocr does (depending on ocr implementation)?
usability is a big factor so if you ahve to wait 10 seconds to grab all crafts it doesnt feel good
OCR against a known set of strings vs. OCR to accurately identify arbitrary text in a font are thankfully differently scoped.
Elite Dangerous' market tool was scary as it had to find actual numbers for prices.
And possibly unknown commodities.
OCR handles the first transformation (image -> text). A naive Edit distance comparator is what we’re proposing to handle the second (text -> uniquely identified craft)
yeah which is why im wondering what direction is most optimal to go for me, ocr against a known set of strings or edit distance (terms which you two so conveniently told me right now)
ill give it a go then with the naive edit distance
I’d go the edit distance route. It’d take you 2 minutes to implement I think.
so 2 hours for me, gotcha
I have no idea if there's OCR-by-example, I guess that's what the fancy modern training is about.
mucking with it is pretty hard because documentation is lacklustre
people simply know how to do it
In this case it’s hardcore not worth imo
What are you writing all of this in so far. Python? Or just calling tesseract from cli?
whispers to themselves: "I will not get baited into trying this. I will not get baited into trying this. I will not get baited into trying this."
thats the magic part, C#, im using a wrapper for opencv (opencvsharp) and a numpy port (numsharp)
so i have a whole set of problems on top of actual problems
Great, c# is my primary language. Just a sec.
You probably want to use the Damerau-Levenshtein Distance Algorithm
I just got the worst idea. Determine the dimensions of the craft list items, load Fontin, draw the known strings matching how the game lays them out, and compare the images 😄
(don't do this)
apparently if i was doing it in python, id be able to set a OpenCv Mat format to a numpy as an NDArray dataset, but i couldnt get it to work (probably steps are wrong or smth, who knows) so I hacked in this. theres a bunch of overhead that i gave myself for choosing c# :)
great, ill give it a go
It’s the most flexible option. Beats hamming and plain leven for sure because it allows transpositions
Idk you can try hamming and plain leven and see which is most accurate. Ultimately I think all 3 will be near 100%
the dataset im working with could probably be done with imagecompare at this point anyways :P
just grab binary projections and compare them to stored data
this is more fancy and could work going forward more easily tho
Like I’ve said, my hunch is that you can afford to fuck it up real bad and still achieve 100% accuracy here
Which is why I led with “just ship it with c2t bundled”
after im done with the smoke break ill go ahead and give it a go, but my intentions after im done with everything is to get samples and run simulations based on various image settings and check results/performances
yeah, but honestly you don't need to do much processing to get pretty accurate results from tesseract out of the box
you can get a solid variety of data from twitch streams too
Images to test with I mean
Is the UI the same for the horitcrafting station vs. in-grove?
yeah
Only a few minor differences
i thought of asking people to dm me sshots
You can probably find hella screenshots on Google images tbh
For size stability you could automate resizing PoE, taking a screenshot at every possible resolution 🙂
I don’t think any of the craft strings have really changed
How are you handling cropping?
Or are you using openCV to hunt for the currently selected craft?
this is a testing case, but its a sample of how it could look ingame. my initial idea is to have hotkey to show overlay and select crafts you want to "convert to text"
similar to how APoET handles the delve grid i guess
its abysmal, im just throwing controls on there whenever i need new ones, once im satisfied with everything and am sure that i can make it work 99% of the time, ill just create another project with cleaner code and deal with actual gui
i made this so i could test out various processing as i am completely new to everything in that field
You may not even need one. Can’t you just pull the region under the current cursor position?
Even if they have the cursor obstructing a word it’s going to be nigh impossible to lose accuracy after comparing by edit distance, from what I can gather
ultimately id like to have extended functionality like formatting etc
not a simple ctrl+c get
Ah
but perhaps its unneeded?
i wanted to write it so i can press F9, get crafts, input prices and paste in a channel
saves me 2min
Tbh you could automate all of that besides price input
Oh you mean grab multiple crafts with a single key press
I see
yeah like the overlay here can read from 5 different crafts, but maybe you want to exclude 3
Yeah I could see both being useful. Sometimes people will want to bulk list, sometimes it will just be a single craft
the lowest i got tesseract to process without horseg so far is ~0.5s
Ooof
which is why i want to run simulations on tessdata_fast, tessdata_best, older version of tesseract with --oem 1 or whatever it is, various image processing things etc
and tesseract page processing settings itself
Btw should we go to breakout? This is my first time ever participating here. No idea how it usually works
were not really offtopic nor is anyone discussing anything else here so i think were fine
Word
additionally, if you have 10 crafts in horticrafting station, you cant show them all at once, but youd want to export them all at once, so it saves you that texteditor opening perhaps
if i include a overlay/gui like apoet has
but i reckon itd be a pain to make a nice overlay like that
Wouldn’t you have to force the user to scroll for you too?
(Sometimes)
Btw re: speed I think if you get this up and running, optimization is certainly possible afterwards. Wouldn’t kill yourself over it prematurely
the user is in control i suppose, wouldnt want to automate everything, even this way im probably on the edge of ToS
Were it my project, I think I’d focus on single crafts just to get it working before handling bulk. That’s just me trying to KISS though
Nah I don’t think you’re anywhere near the ToS line.
OCR has been a thing in the PoE tooling community for years and years.
im sure im pretty safe as long as i dont interact with the game with the data
but still you never know
Even if you interact, you’re still fine as long as the user is creating inputs for each action
the game is overrun by bots again, I think GGG has higher priorities
from usability perspective it also makes sense to make the user in control in my opinion, similarly to how apoet does (or used to do) the query for you automatically and i certainly dont want it to do it for me
automating extracting creates another set of problems if someone wants to (for whatever reason) export from various non-horticrafting stations. when do you overwrite, when do you add to current list
its simple to create a list of saved listings, but id rather not overcomplicate things
in any case, creating the boxes for cropping could be done on a predetermined resolution case, with an optional detecting algorithm for niche cases, but that'd be after everything is said and done
im getting a dns origin error, is it still the same as this snapshot from 2017: https://web.archive.org/web/20170222130728/https://www.csharpstar.com/csharp-string-distance-algorithm/
how hard would it be to get the "distance" and probability between an item x and another item y by lets say utilising only harvest crafts
like you ctrl+c an item, ping ping ping its x amount of crafts
that way i could choose my desired outcomes and play the game with even my brain
A major problem with everything Ctrl-C is that you do not have affix information, just baked stats.
ah, figures
(it doesn't export advanced mod descriptions-type data, so you don't have tags nor do you have the exact affixes)
I think I mentioned this before, but reading an item tooltip is more painful as it's partially transparent.
yeah, and it has different fonts if i remember correctly
or at least italics/fontsizes etc
probably still fontino
Just look at the mess leaking through there, unless you put it just right in an empty stash tab.
Heh, I hadn't noticed, but the separators in the popup are coloured according to rarity.
so you'd have to have an affix calculator to process the item first which is what poeapp was complaining about
meh, it was a nice idea
does an affix calculator exist out in the open?
(I'm asking, 'cause I wrote one last expansion)
perhaps i was a bit out of order with the ~0.5s timings
i made step by step stopwatches that exclude creating/showing visuals
so either its much more efficient without any optimization or (more probably) im a retardo that doesnt know how to measure performance. or both.
nvm string outputStr = result.GetText(); where Page result = tessInstance.Process(bmap, pageSeg); is the culprit
nvm it's probably waiting for the ocr to do it's stuff before it can return the text
You and your silly region 🙂
wow
Always fun to try to tell if a spelling is with a capital or miniscule letter.
If you're wondering, the crafts seem to all be outlined in HarvestCraftOptions.dat, but the spec has changed since last time.
i pulled them and cleaned from poedb
I guess that they might be updated as there's none that mention life-force in the list.
ideally id be pulling it (on user request) from repoe or something but we can see how that'd work out
Needs someone sorting out PyPoE's spec first then, assumedly.
not sure how to automatize it to remove myself from the equation
yeah, all the best to people who work on that, but thats my goto source for grabbing anything up to date that's properly formated
i guess https://poedb.tw/us/json.php/Harvest/HarvestSeeds could work but who knows for how long
fyi - an affix calculator can be a bit of a rabbit hole for affixes that can show up on items but not spawn naturally on them, like delve and incursion mods. These mods will not be linked to the items via RePoE
In general for an item though, you could just string search against the bank of all stat translations, which does exist in ggpk (after removing numbers)
ill leave that to the big boys and play with my legos over here
i went down that path when i wanted to do something similar (synthesis merge calculator/predictor) and quickly gave up
(it'd be nice to have it as a library tho, hint hint)
I had seen a competitor for tesseract recently https://github.com/JaidedAI/EasyOCR. Not sure how it compares in inference time
At the very least it has CUDA support
i switched back to a fast set of trainingdata and now i get below .1second runtime on that. i initially avoided it because i had no idea of the efficiency of edit distance for this use case
its python only :(
Oh apologies, missed that detail about your project
yeah most OCR tooling will be python-centric
thanks for the link, but still its way above my belt and this project. i noticed that most machine learning things are in python (hard to miss really)
Sorry disappeared for a while. Glad to see it’s working well.
You’d probably be better off including the crafts in a JSON/etc file with the executable you distribute. The crafts don’t change frequently enough to warrant the liability that the source you pull from is down/malformed/etc.
yeah that link you gave me was a major contributor, me implementing it on my own would be really inefficient
yeah json/txt is a definite, but im thinking of cases where in the future ill inevitably not be keeping it up to date myself. i guess a simple info would suffice for those 3 people whod be using it anyways
My stuff relies on having access to a game folder, but I also use actual assets. I still have the DAT spec as a JSON file, so that someone can update it without having to touch the executable.
Putting the relevant descs outside is nice in that way.
Delicious noodle soup: https://i.imgur.com/Tar3i0p.png
(I've got the scale of nodes a bit off compared to actual tooling and haven't merged stage input/output nodes, but this is starting to be readable)
@worthy cape what is this? Some kind of texture processing?
@subtle gyro It's an "fxgraph" from one of the "mat" materials.
In this particular case: Metadata/Effects/Microtransactions/Spells/raise_spectre_celestial/graphs/CelestialMaterial_Projective.fxgraph
Materials in PoE have a bunch of textures and constants associated, a main effect graph and several graph instances. These hook into the render pipeline to supply texturing, UV animation, brightening, and other VFX.
The CelestialMaterial ones are for the slightly animated background you see through celestial MTX effects and also in the Celestial Nebula background (actually groundtiles).
Nodes in the graph correspond to shader code fragments stored elsewhere in the pack which are used to generate regular boring vertex/pixel shaders.
These graphs are exposed to and edited by artists to do VFX and twiddle the look of things.
I'm treading water in the deep end of the PoE pool 😄
just break it down into small units, TDD it and hope for the best
yes, it's starting to resemble a light read before bed :))))
as long as we get closer to 3d printing its fine
How do you pull tags for item bases from the ggpk? Looking in BaseItemTypes.dat the Tags column doesn't have the complete list. Is there a reference to the abstract item type somewhere that I should be looking at?
@oak estuary https://redd.it/55dsgn
This is excellent, thanks.
Summaries and affinities are ❤️
does anyone happen to know offhand where exilence next (https://github.com/viktorgullmark/exilence-next/) writes its snapshots? i've rummaged around a bit in the source but nothing leaps out at me, and there seems to be little information in any docs
Worst case, run it with Process Monitor and look for CreateFile?
yeah, i hoped to avoid anything too involved cus the data i want to edit its more of a minor annoyance than anything. was hoping someone might know offhand
C:\Users\Lars\AppData\Roaming\exilence-next-app
Whole bunch of web:y junk in there like IndexedDB, Local Storage, databases.
hey wait a second, this machine doesn't have a user named lars
jk, thanks for your assistance
much appreciated
how much time it will take to run N requests against rate limited api (assuming requests are fired as soon as possible)? has someone already done this algorithm?
N/rate
wish it was that easy, but when there are multiple rules like poe api has
Oh, I thought you meant in general. Let me loot at the specific rate limiting in place in this API.
Which API? Trade?
yep, sliding window rate limit (in pinned posts)
Ok, so 60:60:60,200:120:900
Maybe I’m dumb, but I don’t see how it is possible to hit 200 requests in 120 seconds given the first rule that limits you to 60 requests in 60 seconds.
Regardless, it looks like the formula I wrote above works fine. You cannot exceed a rate of 1 req/s. Your ETA is N / 1
Ah, I guess the second rule is meant to punish people who continue to pound the API after hitting the first rate limit.
Doesn’t change the answer ofc.
Yeah I’m stupid it’s 1 req/s lol not 60. Edited.
@polar island still intent on bringing the app down?
n: 15, window: 60s
n: 30, window: 300s```
21 request will be completed in 70s
What rates are you looking at? Just share the ones you mean if you aren’t referring to the standard 60:60:60,200:120:900
his is a thoretical 5:10:X,15:60:Y where doesnt matter what timeouts are as he doesnt want to hit them
I see
and then asking how fast for 21
21 / min(5/10, 15/60) = 84s
ahh but you dont need to trigger the second one twice
so you only need to stop at 15, then the next 60s period you can request another 5, and 5, and 1
Yeah, true.
So it’s ~71 seconds.
I see why you asked for an algorithm now.
You can do it with dynamic programming pretty easily
whyd you need an algorithm tho, dont you get a state response?
He’s asking for his own purposes not his code’s, from what I gather. Just wants to know how long something will take.
ah, fair enough
for W:X:?,Y:Z:? assuming W<Y and X<Z and W/X >= Y/Z it should be somthing along the lines of floor(N/Y) * Z + floor((N - floor(Y/N) * Y)/W * X)
right?
That’s not a general solution but works here.
It’s a solution for any inputs that require 1 Z period followed by multiple X periods.
well it would just be ordered list where you dont care about timeouts, and it wouldnt make sense to have limits that when ordered dont follow the above right?
Wait nvm I’m dumb again?
Yes
Yeah I guess that works fine, but now handle an arbitrary number of rules and you’re fked
You’d have to use a summation over the rules
With recursion too
eh shouldnt be that hard with a recursive function
wont have a nice formula but should be a simple function
function limitThingy(sum, N, limits):
if limits.size() < 1:
return sum
return limitThingy(sum + floor(N/limits[-1][0]) * limits[-1][1], N - floor(N/limits[-1][0]) * limits[-1][0], limits[:-1])
``` mesh of languages but you get the idea
where limits is an ordered array 2d, you grab the last element to do math, reduce N by the remaining requests, and then you pass down the entire array but the last element
reducing the number of rules by 1 each time
I don’t think this handles N < W
Eg n=4 for our previous example
N=4 takes 0 seconds (instant requests)
you could send them all instantly and thus they would add "0" time, based on @civic crane "compleated in 70 seconds"
There needs to be some comparison between each rule and N at each step
if N is less then the size of limits is 0, so add a base case for when the array is empty
I didnt have a base case so it woulda gone on forever anyway, good thing I didnt try and run my fake code anyway
@civic crane the formula and the pseudo code function work for you or seen any issues?
yup, with only edit from sum + floor(N/limits[-1][0]) to sum + (ceil(...) - 1) . Still have to verify more cases, and take into account current state of rate limit 🙁
oh btw the current ratelimit doesnt exactly work with mine becouse its a nonsensical limit (200/120 < 60/60) as youll always hit the 60:60:60 before you hit the 200:120:900
isnt that only an example or something tho?
as in, when i was doing a project when conq rolled around the limits were closer to what im currently seeing by inspecting ggg trade
nice equations
poemap.live is using oauth to authenticate the user, does anyone know if there is any public documentation on this? Mainly if its possible to register other applications, if there is sandbox instance and if there are other apis available with that token?
Great thanks!
IIRC it's already done or close to done, but bottlenecked by legal issues
or seasons
well, they are producing a lot of content with a relatively small team, so I guess something like oauth might not be high on the priority list
I mean it's been in development for 2 years heh
i think its nice that there is at least some sign of this.
yeah i mean its only 2021
sorry, 3 years
@subtle gyro #tooldev-general message
@subtle gyro thanks, if I get a promising prototype done i might try. No reason to bother people if its just an idea right now 😄
i had an idea which relied on oauth giving stash/char info
so i prototyped and then was forced to hack alternatives once i realised i based it on an illusion
Waiting for blizzcon on February 19 to push poe to perform something
perhaps another hasty announcement
the problem back in 2018 when discussing this was that the IP based throttle was in effect, meaning that you couldn't create an application that relied on users' data
you'd effectively get throttled very quickly if your application got used by more than a couple of users
that was a while ago though, so hopefully that'll be addressed in a good way
how bad is the river atm? still getting pinged after selling a woke orb half an hour ago

Ug, trade always breaks on the weekend
Constantly getting whispers for things sold over 30 min ago
if you think it's bad now wait till tomorrow
Anybody here interested in dev'ing a super-simple overlay that ties to a website I'm working on? PM me
Ah, so not legal trouble. They are waiting on their legal team to update their ToS.