@compact isle did my rambling help you find that? ๐
#tooldev-general
wind garden
compact isle
it was Jonathan that found it ๐ I'm working on other stuff atm
wind garden
aw shucks heh
compact isle
just relaying what's going on
wind garden
well good to hear
mortal bone
makes sense
mortal bone
you pratically doubled your load with reminder text
bitter wave
Does the generated information for the items/JSON change from one update to the next? (Assuming no experience was added to the Gem/currency was used to modify the item)
cosmic saffron
did it also export an array of all the definitions in every single item or something?
desert roost
Better to have it fixed and rolled out than to inform people that the issue was found and they're "working on it".
compact isle
no display information changes don't update
which can be really annoying if you're indexing
nothing that crazy @cosmic saffron, just unnecessary work when the website doesn't even use them
compact isle
does quite a lot of things actually
not important for the game client since 0.005s is nothing when you can only hover over one item
but for us... ยฏ_(ใ)_/ยฏ
cosmic saffron
while you were busy playing the game I was mastering the art of hovering over all the items
compact isle
no please
bitter wave
If the information doesn't change from one update set to another, could you cache that information (using whatever unique ID the items have to track them) in JSON format so that updates only need to grab the updated JSON data rather than regenerate it for each query?
compact isle
have mercy
yes, but caching wouldn't be so useful because it wouldn't be used often enough
anything we cache would be evicted almost instantly
bitter wave
Might be that I'm not entirely sure how the river/streamed updates work, but if you kept the update stream in JSON format (for web/API use) rather than binary, would that be any faster once multiple systems are indexing? (As far as the entire system load involved in serving the requests)
compact isle
load from requests isn't a problem
eh, I'm sure we'll continue to look into ways to improve it
mortal bone
The reminder text simplying increases the time it takes for each item to go out, from what I understand. This wouldn't really have much to do with caching the actual river since the river already has a list of items and such that need to go out, but it isn't being filled fast enough.
wind garden
It makes sense that the reminder text increased the processing time beyond what it could keep up with. What I wonder about is will further increase of players and clients effectively do the same thing again one day
at which point it would be much harder to fix
mortal bone
It isn't much time but when you want to process hundreds and thousands of items it is too much
compact isle
yes Retik, which is why it's something we have to look into even once this initial problem is solved
mortal bone
I know this isn't a solution right now, but why not dumb the raw data to the public?
wind garden
@compact isle is there any chance an API could be added to request the most recent change_id? It would be really helpful for users who don't care about history to just jump to the end instead of pulling everything old. Would help reduce the load.
mortal bone
You probably can't answer that, but it is something to think about. It would cut down on your processing time
bitter wave
He said the raw data is binary right now
compact isle
^ not useful to you guys at all
mortal bone
Right, but we can be given the tools to make something like that useful
bitter wave
I was trying to say rather than (Gather Binary Data for changeset).ConvertToJson(), convert the Binary to JSON outright (to be the stream that is used by the API) and then just gather the JSON
compact isle
which is more than we're willing to give away from what I understand
mortal bone
Right, which is what I was saying.
There could be a way in which we are given the data in such a way that we share the processing load
compact isle
that's what we're doing with JSON lol
yes Zinsho, but then we'd have to constantly reindex / expire than JSON when items are changed etc
mortal bone
JSON is pretty processed stuff even at a .001s timer per item you are looking at extremely slow times
We can all sit on the outside and give suggestions, but we really can't do much but speculate. You guys know your system better than the rest of us
waxen ridge
Of course we all like to help
compact isle
not sure if we'd get much performance out of packing the data into a different format to be honest
a nice suggestion of course ๐
mortal bone
json is a pretty light payload already
bitter wave
The only performance benefit would be in a Binary->X converter being faster than Binary->JSON. But that still wouldn't fix the majority of performance if 60% is gathering the data.
mortal bone
The idea would be that you wouldn't have to gather all of the data, but you would be able to convert it in such a manor that the public could pull down
hashes are pretty damn fast, so you could generate look up tables since items don't really change all that often (some mods at least)
compact isle
knowing the backend, unfortunately not really an option ๐ฆ
mortal bone
Yeah, that is the hard part lol
we have no real way of knowing how to help, but we can spitball
wind garden
@compact isle my suggestion for data serialization is always google protobuf. Fast, efficient, it's magic.
mortal bone
isn't it basically json
compact isle
nah, it's a json header that describes the format then binary data
(not exactly json but ya)
kind of boils down to the problem of us using our website display information for bulk data tho
eh
mortal bone
Sounds like you guys need some dedicated api servers ๐
bitter wave
and/or a separate dataset that is trimmed down to only what is needed for bulk API rather than all the website display information (Not sure how much would be trimmed down but if that's part of the problem then I'm sure at least some could be).
waxen ridge
It's better to build up than trim down
mortal bone
It sounds like that is what is happening, but it is building up too much
cosmic saffron
that's usually dealt with by a parser that discards art info for example
so each request would generate a bunch of different subsets of the json, which individually has less load but if you allow too much customization it might clog up once it's going over the same data to generate different formats for all the requests
wind garden
right
mortal bone
this is completely different
wind garden
but couldn't the requests be distributed across an array of more servers?
that could be expanded to meet demand
cosmic saffron
that's still going to be centralized because the data's coming from game servers
wind garden
but he just said the data is in raw binary and has to be processed
but they won't let us share the processing load
so instead of processing everything 1000x times, and sending it to 1000x clients
send it to an array of your own servers
and let them distribute the work
then have a proxy that services requests
cosmic saffron
what, to the clients?
wind garden
and sends those requests to the array for processing
compact isle
yes that's a solution
wind garden
clients, meaning all of us requesting from the APIs
compact isle
something that would be interesting
if you treat the parts of the change id as separate streams
then you could concurrently request each shard's data (from a consumption point of view)
mortal bone
you guys would need to distribute a client that we could use
or someway to connect to the peer-to-peer network
wind garden
could the stream's change_id be made deterministic? meaning there's only one stream
compact isle
if you pass a high number for all shards but one then you'll only get that shard's changes / change_id
mortal bone
that would split the load pretty well
4 servers splitting the work of one wouldn't be too bad
compact isle
I'm meaning from the client's perspective
will get you the top ~5k items on the first shard
wind garden
that was pretty quick
compact isle
yeah, a good way to increase items per second
mortal bone
Ah, just loading all the shards into one?
wind garden
not sure I understand but it looks like Emmitt does heh
mortal bone
basically each shared gets a go message, and it returns all the data once they are done
wind garden
did you just come up with a really large id yourself
mortal bone
4 times lol
wind garden
so that's the maximum id value
and passing that to the server gets the tip of the stream?
compact isle
well no
wind garden
ya don't think that's right
mortal bone
the first value would be used like the normal stream
compact isle
it's just guaranteed to not have any stashes in it
so when the server compiles the stashes it has after the change id
it will only serve stashes from the first shard
as it has no stashes on the other shards with a change_id greater than max uint64
mortal bone
so basically all 4 shards are working on the same data pool?
compact isle
no, change ids are unique across all shards
mortal bone
ah ok
mortal bone
right because now you can 0-*-*-*-*, *-0-*-*-*, etc client side
mortal bone
Yeah, it shouldn't take too long
@wind garden basically you can run 5 instances of your processor concurrently
by setting the max id of the change_id for each shard
Shard 0: 0-9223372036854775807-9223372036854775807-9223372036854775807-9223372036854775807
Shard 1: 9223372036854775807-0-9223372036854775807-9223372036854775807-9223372036854775807
Shard 2: 9223372036854775807-9223372036854775807-0-9223372036854775807-9223372036854775807
Shard 3: 9223372036854775807-9223372036854775807-9223372036854775807-0-9223372036854775807
Shard 4: 9223372036854775807-9223372036854775807-9223372036854775807-9223372036854775807-0
bitter wave
0 and 1 are the same
mortal bone
rip
bitter wave
and 2 should be 3...
bitter wave
you went 0,0,1,3,4
mortal bone
yep
bitter wave
But point is made. And since each shard uses it's own change IDs, you won't duplicate the retrievals
mortal bone
should be good
mortal bone
looks to be that way
bitter wave
sounds like all changesets are split, but I'm betting there's no guarantee that your stash would be on shard 1 on every change.
mortal bone
@wind garden adding more shards basically parses out to be null, so you get the start for every shard
"next_change_id":"2151-4130-2564-4088-1240" after adding a 6th
wind garden
i see
at least the input is sanitized hehe
servers explode when you give it > 5 shards ๐
bitter wave
if nothing else issues like today where it's not keeping up you could see if it was any individual shard that was slow, or all 5 (since the 'fast' ones would keep sending new changesets and the slow ones would stay on a given changeid for longer)
mortal bone
input has to be 5
compact isle
which shard you're on is determined by the stash's internal id
wind garden
well isn't that a way to scale things too?
couldn't you add more shards in the future?
bitter wave
not sure it's a uINT64 though, just changed shard 5 to *808 rather than *807 and it didn't reset
mortal bone
makes sense
compact isle
we can add more shards in the future so that method would have to be flexible enough for that
just talked to Jonathan and he said he'd prefer people just treated it as a string blob so that there's no dependencies on the number of shards
but whatevs ยฏ_(ใ)_/ยฏ
mortal bone
could you allow for inputs < max_shards
wind garden
so if I take a current change_Id, 26964087-28958182-26932736-31198706-29232635, I can split it into 5 while masking each shard with 9223372036854775807, and it will pick up from where I left off, right?
compact isle
I mean, passing in max int is easier on everyone
mortal bone
true
compact isle
yes Retik
wind garden
awesome
wind garden
has the reminder text been removed from the stream?
things are loading fast for me here
compact isle
once it's built
wind garden
must just be the faster internet at work
compact isle
it's faster now because we put the webserver we were testing on back into the rotation
looks to be 10-25s
mortal bone
yeah
bitter wave
But should be at least a bit faster assuming you weren't processor/network bound since you can run 5 instances
(Once you update to account for that)
mortal bone
concurrency is the best
wind garden
now if xyz just doesn't update poe.trade to do that, I can have a competitive advantage for an hour or two! mua ha ha ha
mortal bone
It shouldn't make too much of a difference
wind garden
it won't once the fix is complete
in the current state I could likely get ahead of poe.trade
if i'm not already
bitter wave
What would you need to be able to figure out if you are or not? Someone to put up a new stash/item to see when it appears?
mortal bone
process all the shards at once vs one shard 5 times
bitter wave
No, to figure out if he's ahead of poe.trade
compact isle
you start getting just a few items once you're catching up
then 0 when you've caught up
mortal bone
oh guess work
he is basically sniping items, so if he gets there before anyone else then he knows he is ahead
wind garden
@bitter wave just put one of your items in a public stash tab while having a real time search on poe.trade open
see who sees it first
mortal bone
wouldn't you always see it first
wind garden
usually
but if i was behind I wouldn't
if we're at the same place, i'm likely a bit faster
but I don't know by how much
mortal bone
Right, but if you are both at the top, then you win
bitter wave
Stone of Lazhwar, 1 alch, player: MaeriDrain, going up as soon as I can tab back over
wind garden
but ya, that's how you measure the delay
mortal bone
retik, do you store the information or you just parse and dumb?
wind garden
ya i wrote a bunch last night that will pull all his non-currency data
compact isle
Retik is going to be a crimelord of Wraeclast one day
mortal bone
nice
wind garden
then i add my search criteria dynamically with current prices minus some undercut
compact isle
when he retires he'll be a Beachlord
wind garden
i ignore anything below 15c
so at 15c i look for anything 10c or less, at 40c, anything 20c or less, and 80c anything 30c or less
mortal bone
right
wind garden
it was actually working too
but because of delay stuff was already sold
found a crown of eyes for 10c when going rate is like 40
mortal bone
hmm, sounds like I need to do some work lol
wind garden
it's been a fun project because it gets more advanced every new league
i'm always adding on to it
mortal bone
Yeah, if only GGG didn't mind these clients getting out lol
bitter wave
hrm... might be fun trying to build one for that in powershell... only question is how quickly it retrieves the data
wind garden
for anyone with a memory injector it wouldn't be hard to make a trade bot
I've encountered them in the wild
i think within 2-3 generations of CPU, it will be possible for game devs to make games that are cheat-proof
wind garden
ehh i think it would be a bit harder than that
mortal bone
Everything you need is a chat command
compact isle
you could get pretty far with OCR / parsing the logs / issuing commands
wind garden
as of intel's 6600 processor they have something called a memory enclave that allows processes to allocate isolated memory that only they can access
mortal bone
/trade <player name>
wind garden
so it's just a matter of time before windows and other operating systems utilize that feature
mortal bone
item data can be copied to the clipboard
compact isle
quick someone ban emmitt
wind garden
how do you know when someone joins your party?
wind garden
didn't think they included that
bitter wave
could do it in powershell for sure. And then just use filesystemmonitor with onmodify to trigger the script
wind garden
i already do that for my message relay app
i'm reading the client.txt and relaying messages to my phone
just open a file stream and you're good to go
wind garden
i am surprised it doesn't have a rollover size
l2log4net
or if it does rollover, it's a large value
wind garden
only 256mb after 2.5 years?
mortal bone
ye
wind garden
i was sure mine was much larger than that
wind garden
i clear the file periodically though, because it takes too long to read through when it's that large
mortal bone
2 years for this one
bitter wave
It normal that my log has all kinds of debug about being unable to find resource files?
mortal bone
2016/12/03 16:42:22 68434703 10f [DEBUG Client 7944] [WARMUP READ] Cannot find resource file: Art/Models/Items/Weapons/OneHandWeapons/Daggers/Dagger7.ao
that?
bitter wave
yeah
mortal bone
idk
bitter wave
Started on 11/18, never happened before that in my log
wind garden
ya i've seen it a lot
wind garden
odd that it wouldn't be logged
mortal bone
makes sense why it wouldn't be
wind garden
why?
mortal bone
because I could automate the trade process with logs + ctrl c
wind garden
oh right
mortal bone
read party invite -> move mouse to accept
you would have to do a little ocr at this point
mortal bone
not that you should
wind garden
last thing I need is bots to compete with
mortal bone
just convering my ass lol
I am surprised you don't compete with bots honestly
HFT tends to lend itself to bots
Although, currency is probably easier to farm
bitter wave
yeah, poe.trade is still running slow. Says 2 minutes ago on my item that I added ~10-15
wind garden
@compact isle ran into a problem, doing requests concurrently gets me throttled immediately by the server
so if i space my requests out, then i'm running them serially
which is no faster than just requesting all at once
dang is it faster though
about 3-5k items per second
bitter wave
what did it used to be?
wind garden
poe.trade was getting 62/second earlier
mine was about 100/second
i actually think it's still faster running them in serial
bitter wave
When it throttles, what does it return as the json?
wind garden
{"error":{"message":"You are requesting stashes frequently. Please try again later."}}
ohhh i see what was happening
nope
nevermind
bitter wave
not faster?
wind garden
no i mean why i'm running too fast
i thought I wasn't delaying like usual but I was
normally i have a 1.5 second delay between requests
but when i have 5 threads that aren't synchronized
they can all request at once
bitter wave
so have to bump it to every 7.5s or so? Or have to stagger
wind garden
unless they remove the throttle I think i'd need to serialize them
put a lock around the loop
mortal bone
you wouldn't have to serialize
just stagger enough that you don't get hit by the limit
wind garden
that's basically serial though
if I can only do a request every 1.5 seconds
and before i could request all 5 shards
mortal bone
is that the lowest you can get?
wind garden
now i can only retrieve 1 shard every 1.5 seconds
i'll try lowing it, but I think so
i think I remember novyn saying it was 1sec minimum
i had it at 1.5sec to be safe
ya just got throttled at 1sec
it actually makes it easy to handle the data though getting smaller chunks more smoothly
instead of having to wait 10 seconds to get new data I get data all through those 10 seconds
wind garden
mmk
wind garden
well dangit
then it was fater to do it all at once >.>
as long as I'm being throttled
compact isle
ah throttling lol
wind garden
5*1.5 > 5
compact isle
ye
wind garden
although this will be faster if it ever slows down again
for request times that are > 7.5 seconds
OR i'll be getting throttled at 1.5 seconds still
what's the actual throttle time?
2 seconds?
mortal bone
iirc 45/min
compact isle
if you go over 2 requests per second it'll lock you out for 60s
wind garden
ya it's a lot less forgiving that that
mortal bone
ah
wind garden
i can't do 1 request per second even
mortal bone
are those the same times for pulling from character data?
https://www.pathofexile.com/character-window/get-items this end point
compact isle
no
compact isle
using the weird shard method or back to normal now?
wind garden
normal
compact isle
awesome
we were testing poe.trade before and it seemed fine
crisis averted for now at least
fossil cobalt
MSRP was 2ex the other day
wind garden
ya the market is still equalizing
fossil cobalt
Anyone know why POE-TradeMacro would be ignoring mods here and on any item?
mortal bone
Wrong key? I am not familiar, but the crafted mod may also throw it off
mortal bone
maybe the clipboard data changed?
do you have the most recent version?
I am not too familiar with the tool, but there could be many different things wrong
fossil cobalt
I grabbed the recent version, this was also happening with the version that came out on launch
on breach launch
AHK v 2.0-a075
wind garden
is there a price check macro for poe.trade?
ctrl+c and item and it gets an approx price?
torpid mesa
Poe trade macro yeah
desert roost
@wind garden Please look at the Pinned message for #price-check๐
wind garden
cool thanks
wind garden
@chrome topaz I think I'm seeing a bug on poe.trade. If you search Hegemony's Era right now, there's a staff listed as 482.22 dps. All other staffs closet to the same stats are around 390dps. They're not even close to each other. I think the dps on that staff is wrong.
hmm I had "Normalize Quality" off
think that was the issue, probably had 20% qual while the others don't
compact isle
just as a note to everyone, it looks like flasks don't include their utility stat on the website at the moment
compact isle
rip website
mortal bone
yea
wind garden
why has it been down so much lately?
wind garden
people buying stash tabs?
mortal bone
when they go down the site does
mortal bone
api I think
needs contact with the realm for items and such in threads
It looks like automod missed the last item showcase
mortal bone
google?
thorn coral
You cannot find your googles ?
compact isle
the server backend is down, all the website can do is cry
compact isle
things will come back up but the trade api has been temporarily disabled
maiden abyss
how temporarily? ๐
wind garden
wait this is more severe than just needing a reboot?
mortal bone
it broke lol
compact isle
waiting on new binaries to be built, so not too long hopefully
yeah some backend issue with public API + quad tabs
wind garden
ah
compact isle
Chris estimated around an hour for the fix
wind garden
Novynn, have you noticed this morning the trade API slowing down a bit?
things were back to 8sec or so
when they had been at 1.6sec or less
might have just been heavy load
compact isle
we were investigating why the website seemed to be slow in general before everything went down
wind garden
hmm so possibly related
i was noticing things taking longer to show up on poe.trade as well
compact isle
glad it was super fast immediately after the patch though ๐
wind garden
omg
i just looked at the new tabs
there's a quad tab???
that exists?
well played GGG, take my money
compact isle
ya lol, I was surprised too
wind garden
the question is, do they have upgrades
compact isle
unfortunately not
wind garden
or better yet, could i compress 4 premium stash tabs into one quad
I just hate scrolling through pages and pages of tabs
maiden abyss
this^^
cosmic saffron
can't wait until we get 2048 tabs
compact isle
I got excited because I thought you meant the game
desert roost
Would you happen to know if quad tabs will be coming to the Guild Stash as well Novynn?
compact isle
I would not know sorry
wind garden
api still down?
compact isle
yep
wind garden
well i just spent $60 on a supporter pack I probably didn't need, oh well
here's hoping they making divination card tab a bit more space efficient
4 cards per page is a little wasteful
desert roost
Don't worry, GGG needs it!
@wind garden Have you tried using the filter function on the div tab?
desert roost
Makes me not mind it having be so large.
desert roost
You can have it filter by completed sets, unowned sets, etc
wind garden
speaking of filter
everyone complaning about not having map tabs
what I really want is a sort button
desert roost
Yeah, sorting by tiers would be a really nice QoL.
wind garden
totally
especially with the quad tabs
can just make a quad map tab
then click a sort button
mab
wind garden
bam**
good thing there's a filter ๐
i need the stash api back so I can continue trading ๐ฆ
@compact isle be sure to let us know asap, if it's anything like earlier this week, there's a killing to be made
because people keep lowering their prices because their stuff isn't selling and they don't know why
then when the api comes back everything becomes visible
my friend got a 6L chest drop the other night and sold it for 10c, which was the going rate at the time
next night they were 70c
compact isle
ya of course
website in general still seems slower than usual atm though so investigating that as well
chrome topaz
- ddos the API 2) people lower prices 3) ??? 4) profit!
wind garden
it just might work....
cosmic saffron
maybe let people draw dividers into the quad stash so they can ctrl+click into specific regions
compact isle
doing a 1 hour countdown now, api will be back up after the deploy is done
wind garden
lol
desert roost
Sir, please, you're scaring the api. โฌโโฌ๏ปฟ ใ( ใ-ใใ)
...
Well, that could get completely butchered.
wind garden
and countdow finally starts
compact isle
I get my information too early lol
chrome topaz
so it will be fixed in an hour?
wind garden
30min
wind garden
did you read our discussion the other day about concurrently updating from each of the shards?
chrome topaz
yeah. are you using this?
wind garden
i implemented it with the ability to turn it on or off
i had it disabled becaues updates were only taking 1.5 seconds
but catching up, might be useful
update time has to be slower than 7.5 seconds to be useful
mortal bone
Yeah, only 2 requests/1s though
chrome topaz
just get more IPs
wind garden
i think the 45 requests /min is correct
mortal bone
45/min is the default api
wind garden
i get throttled at 1.5 sec intervals but not at 1.6sec
mortal bone
which is really annoying
the character tab stuff throttles after 45 requests
no matter how fast
wind garden
what api were you thinking of for 2/sec?
mortal bone
idk, he helps run it lol
wind garden
i know but i'm going on real usage
wind garden
i have one
wind garden
it works up until faster than 1.5 sec intervals
yes, and i log it
it's all quite clear
chrome topaz
I'm not actually doing it
mortal bone
oh
wind garden
i'll start single threadded and see what the speed looks like
last time the api was restarted things were actually really fast
oo 35% pack size, not too bad
compact isle
to be honest with the speed now concurrent will most likely be slower
and I was saying 2 per second because that's what the code says
but it might lie who knows
mortal bone
easy to test
wind garden
interested to know what emmit finds
compact isle
oh
it does the ratelimit hit then checks if it's limited
so technically that would make it a max of 1 per second
wind garden
ya
compact isle
I think, whoever made this abstracted everything way too much
yep, it rate limits 1 before it should
lol
mortal bone
basically you would have to sync your clock with the server clock to make sure you didn't hit more than once a second lol
60s lockout timer, right?
wind garden
ya
compact isle
yeah so talking it over with Paul it seems that the ratelimit rules are when you get ratelimited not how many you can do before you get ratelimited
mortal bone
if you can't do more than 1.5s then it sounds like the standard website stuff is getting in the way
that would be the 45 request mark
compact isle
by limited I mean restricted
wind garden
the 60sec timeout?
compact isle
ye
mortal bone
at 2/s you get the timeout
compact isle
^
wind garden
1.1
closer
503 now instead of 403
and we're off
3sec per pull
still better singlethreaded
compact isle
not bad not bad
compact isle
that's the idea
compact isle
ye, the intention is that it forces you to put a sleep in your consumer
wind garden
right
too bad it's not rate limited per/shard
so if you were to request 0-0-0-0-0 faster than 1.5sec you would get rate limited. but if you pulled each shard concurrenctly every 1.5 seconds you wouldn't
mortal bone
the api support compression?
wind garden
yes
mortal bone
hell yeah
wind garden
gzip
oh that's dumb
Tabula Rasa is registering that it's rarity is "Rare"
instead of Unique
mortal bone
lol
wind garden
ha
mortal bone
lol
wind garden
me too
mortal bone
I can do 1/s forever though
mortal bone
1/s
if (end_query - start_query) < requests_per_second:
time.sleep(requests_per_second - (end_query - start_query))
else it just continues
wind garden
that's exactly what i'm doing
mortal bone
I'll wait a bit longer
wind garden
in the grand scheme of things, 500ms doesn't really matter
mortal bone
just got limited
wind garden
at 1s?
chrome topaz
shouldn't that be seconds_per_request instead of requests_per_second
compact isle
staff accounts aren't limited ๐ผ
wind garden
ya it works for a while then gets limited
wind garden
mm the rarity thing was my fault
i thought the API sent it as a string and not an int
mortal bone
Request 10: took 0.503000020981 seconds
fetched
Sleeping for 0.496999979019 seconds
Request 11: took 0.167999982834 seconds
limited
Sleeping for 0.832000017166 seconds
definitely something fishy there
wind garden
what do you mean?
mortal bone
I was limited after waiting for the second to finish
wind garden
isn't that limited coming from your code?
when iti throttles you, it's for 60 seconds
chrome topaz
maybe it's not precise enough
wind garden
oh, are you determining whether it's limited by how long it took to complete?
chrome topaz
how come your requests are taking 0.5s?
I haven't seen it drop below 1s for a long time
mortal bone
getting empty stashes
http://api.pathofexile.com/public-stash-tabs?id=9223372036854775807-9223372036854775807-9223372036854775807-9223372036854775807-9223372036854775807
mortal bone
I am writting a tester, so I don't really need real data
well, was writting one
looks like 1/s with some delay for your ping works
looks like actual data queries are taking ~2s
mortal bone
way too many things to pull down
wind garden
<NextChangeID xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ShardIds>
<long>27638327</long>
<long>29694522</long>
<long>27615737</long>
<long>32000672</long>
<long>29982694</long>
</ShardIds>
</NextChangeID>
use those
wind garden
getting really bad fps lag
wind garden
my app isn't finding any deals anymore....suspicious as to whether there's just none out there or if something changed
compact isle
the only other things changed in this patch was the whole item mods being sorted wrong thing
wind garden
ya it seems fine
just weird that there's more trades during the day than at evening
maybe more snipers to compete with in the evening
velvet fog
item stat ordering, how it works?
wind garden
evening in the us
void wolf
evening in us equals morning in eu
wind garden
guess I don't know where the majority of the player base is
wind garden
ah ha!
I knew I had to have introduced a bug causing me to not see any deals
Stupid rarity check
I added rarity as a search criteria. but adding "any" to the rarity enum messed things up. My default search rarity was normal instead of any. so it was only matching normal items >.>
wind garden
it's gotta be frustrating to have people always blame poe.trade for not being up to date
ignorant masses
heavy vortex
is there a guide or anything similar on how to iterate over the skilltree json to build a tree?
mortal bone
Just like you would any other json object, there are a few main sections though
heavy vortex
hmm ok yeah. I guess i'll just start having a play with it
mortal bone
This may help with the structure. It's in class form instead of json
A lot less to read over
Main, and Root are the same thing. It is left for backwards compatibility
void wolf
cs looks like c#
outer jay
Cs is c#
heavy vortex
cool
chrome topaz
@compact isle do you know if internal api rate limits have changed? https://github.com/xyzz/acquisition/issues/371 seems 45 tabs 60 seconds no longer works
jovial bear
hey @chrome topaz I think i've encountered an issue with poe.trade, i thought it had to do with the api problems recently but even now, i don't find my item. I now see why : with this search http://poe.trade/search/owaaoheteriduk I don't see my boots (note that the "crafted" option is on "either". But with http://poe.trade/search/taretazatabesi (just switched the life mod to the crafted equivalent), the boots finally show up. I don't know if it's a known issue or not. Thanks for your time ๐
chrome topaz
it's working as intended, your life mod is not explicit so it's not showing
torpid mesa
he would need to use total life to see it
jovial bear
oh ok... I just ... can't stop to think that I miss trades because people are making the same mistake as me and not finding my item =x
torpid mesa
people prob do that all the time, there are a lot of mod options there
chrome topaz
well if you open the mod box and search maximum life the [total] mod is the first one showing
jovial bear
x)
chrome topaz
it's a reminder for all of us
mortal bone
@chrome topaz It seems like the with internal api you get 45 requests then are locked out for 60s
I have tried 1/s, 2/s, 3/s, and you still get locked out after 45 requests
wind garden
idk what's happening with the API righ tnow
it was really delayed for a bit
20+ seconds per request
now it's back down to 2-3sec
and poe.trade is really behind
several minutes
desert needle
are you seeing network issues connecting to the site?
I'm seeing some general softlayer issues here and there, and lots of people are posting in the sub about instance problems, lag spikes, etc.
wind garden
network has been really unstable since the last patch
bad fps/latency problems
i haven't tracked how it's affected the API but I'm guessing they're related
chrome topaz
make stash api fast again
desert needle
2. 96-95-25-206-static.hfc.comcastbusiness.net 0.0% 10 5.7 12.6 1.9 49.2 14.7
3. 96.120.104.165 0.0% 10 19.4 31.3 12.3 75.1 21.4
4. xe-3-2-1-sur02.michiganave.dc.bad.comcast.net 0.0% 10 15.3 29.4 10.9 92.2 25.7
5. s01d1.northport.fl.westfl.comcast.net 0.0% 10 15.3 26.6 15.3 65.6 14.8
6. be-33657-cr02.newyork.ny.ibone.comcast.net 0.0% 10 28.5 26.2 20.2 32.1 4.0
7. hu-0-10-0-1-pe03.111eighthave.ny.ibone.comcast.net 0.0% 10 21.4 23.7 18.6 28.5 3.2
8. xe-0-0-1.bbr01.tl01.nyc01.networklayer.com 0.0% 10 25.8 24.5 20.0 33.3 4.0
9. ae5.cbs02.tl01.nyc01.networklayer.com 40.0% 10 23.5 24.9 21.1 29.2 3.5
10. ae0.cbs01.eq01.chi01.networklayer.com 30.0% 10 53.2 49.5 42.2 58.5 6.6
11. ae1.cbs01.eq01.dal03.networklayer.com 80.0% 10 67.8 64.8 61.8 67.8 4.3
12. ae33.bbr02.eq01.dal03.networklayer.com 0.0% 10 57.8 59.4 53.3 67.9 4.1
13. ae6.dar01.dal06.networklayer.com 0.0% 10 56.8 58.2 54.2 65.1 3.6
14. po1.fcr01.sr02.dal06.networklayer.com 0.0% 10 84.5 64.9 52.9 89.8 12.6
15. 7c.c9.f617.ip4.static.sl-reverse.com 10.0% 10 79.1 61.3 53.1 79.1 8.0```Pushed 28192564-30283035-28156268-32636593-30583040 : 6018 items; fetch took 80.1773109436 API rate 75.0586410192 items/s
Change id: 28192609-30283108-28156305-32636659-30583093
Pushed 28192609-30283108-28156305-32636659-30583093 : 5801 items; fetch took 39.0494060516 API rate 148.555396523 items/s
Change id: 28192658-30283153-28156357-32636703-30583150
Pushed 28192658-30283153-28156357-32636703-30583150 : 5338 items; fetch took 54.7960891724 API rate 97.4157112419 items/s
Change id: 28192702-30283217-28156410-32636758-30583225
Pushed 28192702-30283217-28156410-32636758-30583225 : 5957 items; fetch took 114.121814013 API rate 52.1986094556 items/s
Change id: 28192767-30283291-28156479-32636811-30583272
it was really slow for a while
desert needle
mad amounts of packet loss coming out of the softlayer us network to dallas
desert needle
mtr
(multiple trace route, there are tools to do it for linux, winmtr for windows, etc.)
wind garden
k
mortal bone
between 1.5 and 5s for me to pull data
desert needle
softlayer's VA gateway seems to be flapping all over the internet too, I keep getting routed through NY to get to it and I'm in DC, should be a straight hop out to the IAD network
getting up to 90% loss between NYC and WDC nodes on softlayer
first response already has someone showing the same problems with softlayer
mortal bone
prob
tribal quartz
anyone know how to fix the offline skill tree save as feature overwriting other builds?
annoying as fuck
and the save feature as well
even if I choose the build I want to overwrite it'll still overwrite the one that was open when I opened the program
woeful sphinx
the "save" and "save as" buttons at the bottom work on the currently opened build (not on the build selected in the build list, those can be saved via context menu). maybe that is your problem?
tribal quartz
but it assumes I'm trying to save to the build that it had open when I started the program
even though I had already reset and made a new tree
random functionality change that fucked it all up I guess?
woeful sphinx
if you reset and make a new tree, that is still done in the build you have opened
it was a functionalitiy change, yeah. we changed to a document based model because what we had before was really weird
tribal quartz
the previous one seemed normal
woeful sphinx
to make a new build, you can click on "new top level build"
tribal quartz
save as is usually used if you want to save your current document as something new
but all it does is rename
I can understand the save feature doing it but save as shouldn't
tribal quartz
it replaces an existing build with the current one and renames it
instead of adding the new one to the list
woeful sphinx
that only happens for me if the build was created and not yet saved (which is a bug i just noticed)
tribal quartz
lemme check
I have a ranger tree
that I created and saved
btw it didn't switch ascendancy to ranger even though I picked ascendancy nodes
I'll reset make a new tree and save as
nvm it just reset the tree
the already saved tree
fuck me sideways this new iteration is just bad
woeful sphinx
it resets the tree of the opened build, which is saved in another state (reseting is just a form of changing the document, so it has unsaved changes). if you then do save as, you have a new build and the old one old one is reverted to its saved state.
for me that's the expected behaviour
woeful sphinx
ok, that's something that other document related programs don't have, as they have no file list. i think showing the current and not the saved state there makes more sense, but that's something where there are different opinions
woeful sphinx
well, i wouldn't be opposed to that idea. but that would create a much bigger outcry
lapis drift
why?
woeful sphinx
now the most heard issue is that changes to a build "overwrite" its state because you need to discard them to get the saved state back (when what they really wanted to do was creating a new build). with automatic saving, the saved state would just be lost because saving happens implicitly.
lapis drift
yes, because there also should be a "+" button
woeful sphinx
there is a "new build" button
lapis drift
well it's hidden under the File menu and i bet most users have no idea it even exists
woeful sphinx
its on the bottom of the build list
(well, its named "new top level build")
maybe it could be duplicated next to the reset button so it's more obvious
frail knot
@compact isle was there a change made that doesn't allow equipped items to be listed on the forum anymore?
i'm trying to use acquisiton to post a 6l im using. anything in stash will get updated to forum, my equipped items wont
heavy veldt
Can I ask here for help with the loot filter? How do I make it work?
To show yellows and up and items that will vendor for a fair amount of currency?
desert roost
Are you looking to make your own? Using an already made filter would certainly be easier.
heavy veldt
https://www.pathofexile.com/forum/view-thread/1246208
I was using this one that a friend of mine linked. Do you have one you'd recommend shael?
ebon oasis
neversink is a good one
wind garden
i really need to make my own index
relying on others just never quite gets what I need
pulsar moat
poe.xyz.is > poe.trade
wind garden
@chrome topaz i need a way to get stat data about items. And sadly you don't support any APIs ๐ฆ (that I know of)
poe.ninja has some that I'm using but they're sometimes out of date
i think he's using the bottom 10% of prices averaged. I'd like to get bottom 1%
torpid mesa
Pull it from GGG they have the API everyone uses
wind garden
pull what?
the stash API?
that's what I mean, I'd need to make my own index that keeps track of history in order to get the stat data I want
chrome topaz
the stat data would probably kill me
wind garden
do you host poe.trade yourself?
i really don't think it would be that bad for me to do though. I like trackpete's idea of using ElasticSearch for the backend. Then I could get the list of popular uniques from poe.ninja. Then I only need to index those uniques and not bother with all the rare/magic items.
@chrome topaz you were having issues getting throttled the other day weren't you?
did you find out what rate you had to pull at?
@compact isle I'm getting throttled at while running at 2sec intervals
0.5 calls/sec
this does look like what xyz was asking about
it looks like the new max is 12 calls/min
that's.....sad
is that even enough to keep up with the stream?
ok what the eff
throttled again even at 5sec
this isn't enough to keep up
floral dawn
there wasn't any changes to the rate limit on public stash tabs
wind garden
how do you know?
well dang, it's not throttling me here at work
did they throttle my IP specifically????
@compact isle help ๐ฆ
compact isle
the ratelimits haven't changed
compact isle
well we can see an IP trying to do it too often
are you sure you're not running another one from the same IP?
wind garden
there we go
all fixed
my client now checks to see if it's on the same subnet as my router. If it is, it will connect to the duplex tcp channel of my main server and stream the data from it rather than requesting it from GGG's server.
torpid mesa
Nice
chrome topaz
@wind garden it was an issue with acquisition, not poe.trade
wind garden
I see thanks
wind garden
@chrome topaz is there any way you could make "Fixed price" more obvious or distingushed between the normal price thingy?
like red letters with a red box outline lol
i get messaged all the time by people offering low offers and say they didn't see the fixed price
pseudo ocean
^
it's super annoying
i don't know if it's in the results from the API but when i put Fixed Price, i'd LOVE if people knew they can't do the "can i give you 10 jeweler's 15 chromes and 2 vaal orbs" thing
desert roost
@chrome topaz Is there an oversight with the Blight enchants? I seem to be unable to search by the explicit enchants in the drop down menu.
dapper thunder
poe.ninja is not working atm?
ebon oasis
working for me
chrome topaz
@desert roost the blight enchants should be up now
desert roost
Thank you!
wind garden
how does poe.trade know if people are online?
wind garden
website troubles?
tulip gale
@wind garden is poe.ninja yours?
wind garden
no
tulip gale
I skimmed through chat a bit, @wind garden are you developing a "deals" tool? could you tell me more please?
wind garden
i have my own app but it's not public
tulip gale
I'm just curious about your tech stack, not the usage of the app itself
wind garden
mmk
wind garden
it just processes the public stash api
i can set criteria to search for
if it finds an item in the stash api stream, it notifies me so I can make an offer on it
tulip gale
so you keep a history to "know" a good price, or you just set that yourself?
wind garden
i use the poe.ninja api to get MSRP prices for things
tulip gale
oh, TIL poe.ninja exposes API
wind garden
not publicly
tulip gale
is the dev here?
wind garden
no
tulip gale
ah
tulip gale
I see, awesome
wind garden
@compact isle any idea what's with the website?
compact isle
it's down while something gets sorted out in the backend
I'm sure there'll be a post about it at some point
wind garden
mm just came up
compact isle
yay
compact isle
yay posting items on the forum is faster now
shouldn't be any more timeouts with acquisiton and stuff
torpid mesa
Nice
wind garden
why is there so much more delay between when you make a change for your stash and when the update goes out?
it used to be when you changed zones and it was almost instant
now it's like minutes later
compact isle
as far as I know nothing has changed with that
bold edge
@wind garden i have also noticed that..probably the servers are going crazy
wind garden
@chrome topaz have you notice poe.trade not sorting by price properly? For example, right now the prophecy From The Void is selling for 1.5ex and 95c. Even at the selling exchange rate for ex to chaos of 71, that 1.5ex is 106c. So the 95c should be getting listed above the 1.5ex but isn't.
chrome topaz
good point
wind garden
@compact isle the trade api is really delayed right now
by at least 10-15minutes
i'm getting whispers for things that have long been sold already and I can see them still listed on poe.trade
i checked my stash api service and it's up to date and going fast
pulling ever 1.6sec
i'm assuming poe.trade is caught up as well, so it would seem the delay is coming from GGG's server
wind garden
actually, i think it is an issue with poe.trade after all
live search on poe.trade picks up a chance almost instantly
but the normal search is way delayed
chrome topaz
normal search can be delayed sometimes yes, normally for about 30s-5m
also funny thing, I noticed some people listing currency for really weird rates, http://currency.poe.trade/search?league=Breach&online=x&want=6&have=6
I guess they meant to select chaos instead of exalt
pulsar moat
Yeah that happens a lot. For some reason setting a stack of ex as ~b/o xx/xx <other currency> trips people up
A few people do put up 'WTS 1 ex for 1 ex' as a joke though now and then
ancient smelt
Does somebody know if the Procurement Dev is also here?
long stratus
Could anyone help me with the trademakro for poe? My issue is I don't know how to disable the ctrl+a shortcut. i tried to alter some script parts but yeah, I failed. I used the ctrl+a shortcut for marking everything in chat but now I can't could someone give a suggestion on how to disable that. Thanks in advance!
timid fable
@long stratus there is an options UI, you don't need to edit any AHK scripts. Right-click the tray icon and choose PoE (Trade) Item Info Settings
also the latest version changed ctrl-a to ctrl-e by default so it doesn't interfere with selecting text
long stratus
it seems the dev itself changed this out
it was ctrl+a but now its fixed to ctrl+e
ty for the reply though ๐
timid fable
np, you might want to check out the options anyway
long stratus
Just did, hole cow no more manual scrolling in the stash tab. This makro is just amazing ;D
holy cow*
dire trail
hey fam! is there any way to parse certain wiki pages in a comfy way? unless i'm mistaken, it seems like my best bet is manually parsing a html response then fishing for what i need
i'm making a discord bot command atm, which would return gem stats (of specific levels of a gem)
lapis drift
there is api http://pathofexile.gamepedia.com/api.php
see the documentation on how to use it
also all items (except div cards for now) use SMW, you can use SMW api to get data from pages
lapis drift
dire trail
oh this is awesome
suddenly i have more than just that command in my mind. thanks a lot
oblique karma
Hey, anybody knows if you can get banned by using Poe-Trada macro?
lapis drift
no, it's safe to use
long stratus
I would kick GGG in the nuts if they banned for this...
srsly its a really neat tool that saves time
waxen ridge
They have no idea if you're using it. You're safe.
gray osprey
It is POE policy that macros are okay as long as it doesn't A. hack(aka break the game and do things you can't do otherwise) B. press more than one button at once with a single key press, it's okay.
oh, it's just a game overlay, yeah game overlays are fine. I mean, you'd be banned by using Steam otherwise, because of their overlay. ๐
@oblique karma yeah it's fine
oblique karma
Thanks ๐
chrome topaz
@compact isle online api has been returning 0 players for about 3 hours 40 minutes now
long stratus
is poe.trade tango down or something?
haughty night
Well, that really sucks. :/ Can't sell anything. nice
chrome topaz
it's fixed now, by the way
haughty night
awesome, thanks for the heads up @chrome topaz and the awesome work you do for this game !
chrome topaz
@compact isle is it possible to disable "Login protection enabled. You must wait 3600 seconds before logging in."? my online-bot cannot login into the poexyzis account. I assume somebody banned it again
compact isle
it's not banned
chrome topaz
can that login protection thing be removed?
chrome topaz
that's a good question
I think it tries to login while the website is half-down, then crashes
then supervisor restarts the process and it tries to log in again
I found a lot of errors where it failed to find <input name="hash"> element
but I don't see any login form redesign or anything that could cause this
chrome topaz
@compact isle i'm pretty sure i've waited for more than an hour now
and it still says the same thing
"Login protection enabled. You must wait 3600 seconds before logging in."
compact isle
looks like you triggered the 20 logins per 24 hours rate limit
and you keep getting limited because you're accessing the login page itself so many times
from the same IP
velvet fog
Is online-bot open source?
wind garden
what's with poe.trade?
desert roost
I'm thinking it got ddos'd
still nexus
ddos'd to kill then xyz (maybe) fell asleep is my presumption
tulip gale
sorry for a dumb question, but, how often the initial "next_change_id" is updated on the request with no id provided
I mean when hitting the endpoint, I am given the same initial state with next_change_id":"2151-4130-3306-4050-1240", shouldn't this differ to provide the current_state and then send new deltas?
@compact isle mentioning you just in case you could clarify a bit, If this is unacceptable, I won't do so in the future, sorry.
compact isle
all good @tulip gale, just don't expect an immediate answer outside of work hours ๐
the next_change_id will change when one of the old tabs that it contains on Standard gets updated and there's room for another one that would've previously been in the next set of changes
tulip gale
so it requires a bit of time to catch up with the "realtime"
if I understand it correctly
compact isle
well, it seems like people overcomplicate the whole process when it's actually really simple
tulip gale
I am testing it with 2s between requests
and in 15 mins I am still getting next change id
compact isle
every public stash tab on our end has a single "change_id" param. When something in the stash changes it gets assigned the latest change_id (shared across all stashes)
so all the stashes at the start are just ones that haven't been updated in ages, but it still gives their current state
it'll take hours to catch up to the top of the river
prisma prism
hi, can anyone link the docs of public stash tab api?
compact isle
prisma prism
thx!
compact isle
no, but if you don't care about the current state of standard tabs you can start later on
tulip gale
how can I get the more recent change id? or is there a way to get a change id that doesn't have any data yet
compact isle
well a change id that doesn't have any data quickly fills up with some
you can get a recent one from http://poe.ninja/stats if you want
prisma prism
can i somehow filter the changes that server returns? like league-specific