Can two different changeIds cointain the same items?
#tooldev-general
deft jolt
mortal bone
You would have to process the data in a queue
Either way you have to process them in the order you get them
simple ravine
well, if he starts grabbing the data in next_change_id, before he completed the current stream of changes
he may lose out on some changes.
deft jolt
hm
mortal bone
Ah, true
simple ravine
It all depends on what you're doing. I assume your goal is to snipe items as quickly as possible?
Otherwise, there seems to be very little reason to chase milliseconds by concurrently processing river data
deft jolt
then again, the extent of my API use is only to find specific items, not to index them
and i've got some code running that prevents exact duplicates from showing
simple ravine
Well, I don't really want to get into a distributed computing science class here, but... I urge you to deep diver into what the consequences are of what you're doing 😉
desert needle
I'm not completely clear on whether or not the backend generates all of the content for a change_id upon request before sending, or if it chunks/streams that data from generation while it's happening
I suppose it would be pretty easy to see by simply calling a next_change_id page immediately and seeing if there are duplicates
but certainly the intent of the design is that you fully complete one page before requesting another, whether or not it matters depends on implementation
simple ravine
"If the number of stashes returned is zero, you get back the same change ID you passed in (a hint to keep trying until the endpoint has some tabs for you)."
from the documentation
desert needle
that happens when you're ahead of the stream
simple ravine
So the ids are generated ahead of time
deft jolt
I do get duplicate changeIds from time to time
But generally they are just ignored and then another request is sent
desert needle
presumably you'd want to multi-request at a point in time when the stream is more active than your ability to grab in one request, which does happen early in leagues
simple ravine
but i'm also asusming that it's like a linked list, with multiple pointers...
i.e. I can get change_id X if I create a requset at time A, meanwhile you get change_id Y if you request at time B?
desert needle
i.e. when you're getting back 250+ tabs in one stream you might be behind
if you start from 0 on two different nodes you will get different change ids
simple ravine
right
desert needle
but doubtful that will improve resolution
simple ravine
so that leads me to believe that you will have to follow the linked list, otherwise you'll be missing out on changes
desert needle
theoretically yes
simple ravine
i really have no idea how they've solved this technically, but it's actually quite fascinating
desert needle
I haven't looked at it in a long time but initially it seemed clear the change_id data was cached in some way
I'm guessing there is a database table or similar that tracks which changes have been sent by which change ids
simple ravine
it does seem to be a computed id based on the request you've made
deft jolt
interesting
desert needle
I have an old e-mail that explains it a little bit but it's nda'd
simple ravine
and one should also take into consideration that it's always current data, meaning that the stuff changes all the time, meaning that change_id=0 response may be different from each request... and consequently all other changes have the same characteristic
deft jolt
however, i have not lost any of my own items when testing/debugging
simple ravine
Well, again, I am not going to attempt to prove any of this, as I'm just speculating. It's up to you how you use this information @deft jolt .
deft jolt
yeah, i know
desert needle
there is some indication that they may be using a behind the scenes system to actually add stashes by id to change_id triggers
deft jolt
As far as I'm concerned, it works fine as is
desert needle
which wouldn't scale with thousands of consumers but the original design didn't expect very many people to use it, so that kinda makes sense
deft jolt
For the purpose im using it for
simple ravine
Well, the current solution doesn't seem to be 'scaled' much, as you can see a big variance of response time from the API endpoint
desert needle
yeah can confirm some sort of similar thing to what I said above
the list of stashes in a change_id are updated when the backend stash is saved
(or at least that's what I was told during the first stash tab api beta, could've changed since then to scale differently)
simple ravine
Ok, so assume the following:
App A and App B
- Both requests change_id=X at the same time. They receive identical response.
- App A waits 1 second, and request the next_change_id
- App B waits 10 seconds, and request the next_change_id
during a time period where 1 second would not fill up a "full payload"
How these two apps now are diverged would be interesting to visualize
desert needle
theoretically they both get the same payload (though randomized)
mortal bone
They would get the same payload over time
simple ravine
Are you sure?
desert needle
sorry, my response was assuming no changes were made after t1
better would be:
at t1, app A requests
at t10, app B requests
at t15, app A requests
at t15, app B requests
At that point they should have the same data
mortal bone
The apps get data that they don't have yet
simple ravine
Ok, let me rephrase so we're on the same page.
Let X = specific change id
T01 App A requests X. Receives payload with Change Y
T01 App B requests X. Receives payload with Change Y
T02 App A requests Y. Receives small payload (few stashes).
T30 App B requests Y. <-- What will App B receive? (change id)
Assuming that there are more stash changes between T02 and T30
elder hearth
is it possible to receive few stashes? (as long as they are the last current instance)
simple ravine
yes
elder hearth
🤔 but when that ID is released to the river isn't it locked for new entries?
simple ravine
that's the interesting part... no it seems like it's not
the change_id seems to be a from->to temporal pointer
elder hearth
that is a problem i think, trackers may miss some items
simple ravine
it is normally not a problem, as long as you process the current request before making and processing next change
deft jolt
Say I got an extremelly powerful computer where processing the data takes about 0.1 sec
Network speed is also top-notch, takes about 0.5 sec to download the data package
desert needle
hah, processing the data isn't important, it's ggg's time to send data, but yeah, assume it's instant
deft jolt
Yeah
simple ravine
CAP Theorem ❤
deft jolt
So if I were to download the whole data package and process it in under a second, would I miss out on items compared to a consumer that waits 10 seconds between each pull?
simple ravine
no
deft jolt
I suppose what it alll boils down to is how the changeIDs are generated
simple ravine
I would assume that the change_id is a pointer, much in a linked list
so it doesn't really matter
what matters in your case, is that you're not overwriting a stash with old data after storing newer data
when doing multi-threading
deft jolt
You mean I am overwriting new data with older data?
simple ravine
No, I mean that it may be a possibility in case of a race condition
mortal bone
It is 5 nodes calculating what has changed since the last time you requested items
simple ravine
Can you elaborate, @mortal bone ?
mortal bone
5 nodes have all of the stash tab data (each number is a node in the change id). They know what they have sent you and send you the difference between what they have sent you and the current state of the system
The api is basically a diff
simple ravine
"each number in the ID is a nice" - what do you mean with 'nice'?
simple ravine
With node, you mean a shard?
mortal bone
Yeah, sane thing in this case
simple ravine
Ah, so if they scale the service out, we'll observe longer change_ids
mortal bone
Yes
simple ravine
I can recall that I saw quite short change_ids a long time ago, and they've become longer, so that makes sense
elder hearth
61606646-65054765-60958520-70862647-65851290
61606649-65054765-60958521-70862647-65851290
61606650-65054767-60958522-70862654-65851290
this means some nodes did not change while i was requesting?
mortal bone
Yes
simple ravine
That actually makes total sense
mortal bone
Some of the shards didn't have any different data
deft jolt
At the start of the 3.0 beta:
5515-5686-6169-6549-5538
vs now:
80812-81981-72617-99989-84552
That makes this less mindblowing.. but still cool heh
deft jolt
I suppose for the extent of my usage, creating a new request every 1 seconds is perfectly fine
shadow flax
I have a tool in python where I can put in the quality numbers of my quality items and it gives me all combinations that make 40 with 4 or less items. Is this of any interest to anyone or has it already been done? Difference with some tools are it only goes for your desired items. Like one specific stash you have, instead of all combinations with everything in every stash, as long as you manually put in the numbers.
simple ravine
Does it group by item category?
shadow flax
only manually, haven't got it working with stash API, but want to improve it to work with stash API so I can choose armours from stash named "Quality" for instance. Don't know where to start with that, any help is welcome
same person, new picture
simple ravine
I'm assuming that you're fetching your personal stashes.
My suggestion is that you group them by item category (gems, flasks, armour, weapon)
shadow flax
mhm, no idea how to fetch using python, API, sessions... Don't know which libraries to look at even :/ or essential functions...
elder hearth
how does your tool "visualize" it? i mean after you have the combination
if i where going to make it i would make using javascript/browser extension @shadow flax
so you could just browser your tabs
simple ravine
that's a pretty good idea.
a secondary idea would be to create a GUI application, that perhaps does other things as well
shadow flax
interesting, displaying in PyCharms run window now
elder hearth
@radiant tangle Yein
shadow flax
Not interested in GUI, makes it too big to handle I feel on a hobby level for now
simple ravine
@elder hearth Heh, I thought that was you.
shadow flax
no, but want to reach that level eventually
did you see my message, you have the code there
simple ravine
@elder hearth Yeah, I noticed. The other avatar you had was quite similar, hence me thinking it was you. Oh well. Fun!
elder hearth
this one right? lol
simple ravine
I can't remember
shadow flax
@elder hearth Any good place to learn POE API for browser extension as well as a good place to learn coding browser extensions?
elder hearth
inb4 they all look the same
shadow flax
right now I manually transfers the numbers on my items one category at a time, so for now no api is involved :/
elder hearth
@shadow flax I don't think you need any API for browser extension, you just log in and do your stuff with javascript based on what page you are
shadow flax
huh, any good starting place for javascript interacting with browser, not sure on the detecting browser info part
to get started calculating
starting place as in a good tutorial in learning that stuff
doing it because I want the tool, and like to learn, definitely up for better ways of doing it
the step of using browser extension or stash API would be learning though, as I don't know it well enough
elder hearth
well, if you already started something in python and have an easy way to point out what combination you need to grab then i think you have the following options with python:
1 - Log in your account using python (may involve cookies/password and shit)
2 - Make your tool to get you tab items via public stash by forcing an update on that tab and then your tool will get that info from the river
or you can do it via browser extension like i said
simple ravine
If you assume the user uses Chrome, you can always grab the session cookie from the cookie store.
shadow flax
2 and browser extenstion sounds most interesting
simple ravine
It involves a little bit of decryption, but completely doable.
shadow flax
for login purposes @simple ravine ?
elder hearth
2 is the easiest but it is a bit of "workaround"
simple ravine
@shadow flax yea
shadow flax
yeah, got premium ones so no problem there. Might want to do both the workaround and browser actually...
elder hearth
if its for learning porpuses, dealing with json with python is easy to learn
i liked it a lot
json and python structures (dicts) are similar
simple ravine
Personally, I would do this kind of thing through the "logged-in" way
Publishing your items on the stash river just to get the data would be a huge pain
shadow flax
yeah, when I see the json in browse it all looks doable, just don't know the next steps to do it ^^
true @simple ravine
also clutters the river :/
elder hearth
@simple ravine he can set his tool to listen to name account, char name and stash name, no biggy
poe.ninja also gives you the last IDs
so it is
start program,
go to PoE, put some shit inside that tab, change instances
and your program will receive it immediatly
shadow flax
god that sounds exiting!
how mad would people be on me for putting up public items just to calculate the 40% recipe? With no intention of selling?
elder hearth
people don't even care, they look for priced items, not unpriced
if you get msg about your unpriced items they will lowball you
shadow flax
yeah, good point
simple ravine
https://www.pathofexile.com/character-window/get-stash-items?accountName=<accountName>&tabIndex=0&league=Standard&tabs=1
start there, imo
shadow flax
yeah, but how to do that in the python code?
shadow flax
[tangent]: is it beta_standard or standard_beta or third?
so some kind of listener in python to specific API?
isn't it gonna eat my CPU having the listener going forever during gameplay? or simply not have it go forever and just when I wanna look it up...?
obtuse citrus
for fetching websites https://docs.python.org/3/library/urllib.request.html#module-urllib.request
elder hearth
@shadow flax he linked the webpage api, not premium stash.
@simple ravine how ppl "login" on that link? or you need to send cookies to that?
simple ravine
@elder hearth yes, you need to send a cookie to that. Much like you'd do in Acquisition/Procurement. Either let the user specify user/password, the php-session-id or grab the cookie from the browser cookie store on disk.
compact isle
sending a cookie that you grab manually is the easiest way
POESESSID on the .pathofexile.com domain
elder hearth
when i used Acquisition i used php_session, not a big deal to press F12 and get that
shadow flax
so for premium I would use latest from poe.ninja, from there go forwards until meeting my requirements then stop and calculate 40%
?
simple ravine
@compact isle For sure, but not the most user-friendly way for the people who are non-technical.
compact isle
if you're writing a script to do something, isn't that by definition technical?
simple ravine
Hehe, sure for scripts - I would completely agree. I was perhaps generalizing it a little.
compact isle
if he wanted to distribute it then sure
but the overhead of handling the login flow is a pain
elder hearth
@shadow flax yes, until your program gets a tab that matches account name, char name, stash name
then you run your thing
shadow flax
would I have to always manually update to latest from ninja?
simple ravine
The smoothest way (user experience context), in my opinion, is to grab the cookie from the user's browser cookie store. If he or she isn't logged in, ask them to login using the browser.
elder hearth
poe.ninja got an api too
shadow flax
ooh
elder hearth
@shadow flax you can grab the latest change_id just requesting poe.ninja api
shadow flax
right 😃 interesting
elder hearth
@shadow flax try the stash thing, looks a bit easier if you get the login thing going
simple ravine
@compact isle Feature suggestion: Path of Exile OAuth with scope/claims. Imagine the possibilities.
solemn crypt
sweet
elder hearth
😱
simple ravine
@compact isle Happy to beta test with you.
shadow flax
I'm gonna go sleep, thanks for all the help! Hope to come back with further improvements later on
compact isle
hah, we'll get there eventually
solemn crypt
soon™
compact isle
that's pretty much anything web related
simple ravine
What are the scopes/claims you're planning to add?
compact isle
act 8 takes place in Sarn
solemn crypt
i'd say software dev in total 😄
compact isle
well for a start I was just going to do a basic profile scope (for verifying accounts) and then just scope each api endpoint
maybe convert some data that people scrape into apis as well
simple ravine
I think that would be a great idea. Using unofficial ways of acquiring data isn't really ideal world for anyone. If you then choose to dogfood those APIs, that'd be up to you, however it'll ensure developers that the APIs are reliable and won't change without notice.
solemn crypt
any plans on switching to trending graphene or you guys prefer to stick with good old rest?
simple ravine
You mean GraphQL?
solemn crypt
yeah, just different implementations in languages (graphene goes for python)
compact isle
GraphQL seems cool but our data isn't really centralized which makes it harder on our end
simple ravine
Graphene is perhaps an implementation of the specification GraphQL
compact isle
definitely a nice thing to consume though
solemn crypt
@simple ravine you're right
simple ravine
GraphQL is very nice, especially as it could reduce the load of the backend system overall as well - as you can build fetching strategies that is optimized, rather than grabbing stuff over and over using REST, meanwhile you're probably requesting data that is not going to be used.
So it's a win-win.
@compact isle Another suggestion that has been discussed, is additional "static" APIs for item and modifier data. I'm sure this has been suggested before, but would like to "+1" that. I'd be happy discuss that further if interested.
compact isle
unfortunately for you, I'm in the boat of no official data until it's discovered in game naturally!
😄
solemn crypt
😄
@compact isle well, i guess you guys atleast have some kind of spa? that would be a good point to start from (back to graphql comment)
simple ravine
It'd help trading tools such as poe.trade to index however.
compact isle
also when it comes to modifier data, that should probably be represented in-game before we make an API to access that kind of thing
so you have to convince those people first
some kind of spa?
simple ravine
Let's say it'd help democratize the space somewhat.
Right now, it's quite a big hurdle to get over to get to that level, which is quite unfortunate to people who aspire to create altneratives
solemn crypt
i mean single point app, well.. hard to explain it at 4am 😄 nvm
compact isle
oh, not really
I mean @simple ravine it'd probably be better for us to just host those kind of thing ourselves in the long run
that's just my personal opinion though
simple ravine
Probably, it would. There's a macro-economic aspect to it, which is beyond the technical discussion, which is the more auxiliary tools and apps that gets published around a game, the more relevancy the game gets on top of its own efforts. One great example is Riot Games and their APIs to their game League of Legends. They took it a notch further, and encourage people to build tools on it, and have a yearly(?) reward, where they acknowledge the best tool of the year etc.
compact isle
I don't think our company wants that kind of relationship
simple ravine
No.. no relationship with Riot Games of course.
solemn crypt
😄
compact isle
I mean relationship between tool developers and the company behind the game
simple ravine
It was merely an example on how a game developer leverages 3rd party developers to boost their own relevancy in the game jungle.
Sure.. I mean it's your company and game. But what they're doing, in my opinion makes sense.
compact isle
yep, and I can agree with that but I'm not the one that needs to be convinced
there are benefits as well as downsides, and I think the downsides are hard to overcome for an indie studio
as well as any official API becoming another dependency to maintain
simple ravine
Oh I agree, official APIs is something you don't want to just smack up and be done with. Unless you have a glowing beta label on it
solemn crypt
well, on another side there're lots of things you have to implement by yourself then (spent working hours, hire new devs etc.)
simple ravine
I'm a co-owner of an online business, with around ~90 staff, and we do around 2 million transactions a day, with a decent amount of 3rd party integrations. The tech team decided to release some of the internal APIs to some 3rd parties.
It was not a fun discussion around the way they just did it without planning.
compact isle
well, you have to remember that GGG is a game company not an online API provider
if dev time can be spent on improving the game, then that takes priority
or in web devs case, on the payment / backend systems
simple ravine
Right, same here - not a game company, but... similar. We're not in the API business either
solemn crypt
right, but hey, where's an AH then? 😃
simple ravine
But the integration, after some hurdles, made sense. It just had to be properly managed.
AH?
compact isle
hah
solemn crypt
@simple ravine auction house (trading system apart from /trade channel)
compact isle
lol @radiant tangle
compact isle
AH talk is above and beyond me I'm afraid 😄
elder hearth
love that Ranger-Cyclone weapon skin, i would skin transfer NPC to weapons
simple ravine
@compact isle Looking at a couple of interviews, it seems to me that the hierarchy at GGG is not super traditional. Is that correct? Seems to me that someone read the Valve manifesto 😃
compact isle
I mean it seems pretty traditional to me
simple ravine
Do you have a CEO, CTO and other CxOs, product owner(s) etc?
compact isle
aren't product owners a SCRUM thing?
solemn crypt
hmm.. i agree with @simple ravine , at least from my pov (almost 6 yrs in game dev company)
but i guess that's related to the company size
simple ravine
@compact isle Nah, that's a Scrum Master, or a project manager
compact isle
well we don't run multiple projects, so I'm not sure how relevant those positions would be
simple ravine
So when you see public appearances of Chris, he labels himself as lead developer.
compact isle
we don't operate on scrum or anything like that
solemn crypt
well, product owners, feature owners
simple ravine
Waterfall? :trollface:
compact isle
well I mean, most of scrum is just dancing and rituals
the important part is to keep shorter cycles
which we do with expansion releases / patches anyway
solemn crypt
dancing around the boards 😄
simple ravine
We structured it as having a product owner that is part of the management team, planning features with the design team, backend team and front-end team, based on feedback from the executives.
compact isle
that's effectively how it works, just with some people having multiple roles 😄
simple ravine
It's quite intersting, because our companies are of similar sizes, if I understood it correctly
compact isle
well a lot of our bulk is from our support staff which are kind of their own department
then art sort of has their own department as well
simple ravine
Same here, around 50% is CRM, VIP and first-line-support
probably slightly more.
But we positioned them in different offices, different timezones, and for some compliance reasons
So does that mean that Chris is till involved as a developer?
solemn crypt
oh, btw, while i'm here and have a chance to ask a member of ggg team 😃 @compact isle are you guys doing anyth to separate web from the game servers itself? i mean, when maintenance is going on game servers, all of your web resources also on stub.
compact isle
oh man, Chris does so much
solemn crypt
@simple ravine i don't think so, he's acting mostly as a "face" of the company and, i'd say, he is more "lead manager" rather then "lead developer" 😄
compact isle
yeah @solemn crypt we're working on reducing / removing downtime from the website during deploys
solemn crypt
but, ofc, i may be wrong
simple ravine
That is both cool and troubling at the same time 😃
But I know how it is, I am still involved in development. Similar journey, starting a company, grow a team and then refusing to step away from the details, even though the management parts required of you in combination makes your workhours 16 hours.
Hat off.
@elder hearth Hey, you made a script that would simulate mod-rolling, right?
Wondering if you would want to sanity-check my procedure for weighted randomization
private static IMaterializedModifier RandomizeNewModifier(this Item item, IReadOnlyList<IModifier> modifiersToUse)
{
var totalWeight = modifiersToUse.Sum(p => p.GetSpawnWeight(item));
var pickInt = Randomizer.Next(0, totalWeight);
var i = 0;
var pickedModifier = modifiersToUse.SkipWhile(v =>
{
i += v.GetSpawnWeight(item);
return i >= pickInt;
}).First();
var values = pickedModifier.Stats.ToDictionary(p => p.Stat, p => Randomizer.Next(p.Min, p.Max));
return pickedModifier.Materialize(values);
}
It's C#, but let me know if it's not understandable.. would be interesting to compare methods
compact isle
I miss LINQ so much
simple ravine
You can't use it at GGG?
compact isle
we don't use C#
elder hearth
@simple ravine looking at it
simple ravine
.net core / netstandard 2.0 - cross platform and all that
@elder hearth the two last rows are not part of the actual picking of modifier itself
compact isle
that's completely out of my domain, man
simple ravine
How is this even possible?
I thought the Added Physical Damage to Attacks was 9-15 max.
compact isle
is this in beta?
simple ravine
No, standard
elder hearth
essence?
simple ravine
I just put it in my inventory on character p_o_l_i_c_e (account name thezensei) if you wanna take a look for yourself
no, i rolled this using alteration orbs
simple ravine
wat
elder hearth
with essence
simple ravine
yes, with essence
solemn crypt
yep, that's with deafening of contempt
simple ravine
but I am pretty confident I didn't use essence on this ring
elder hearth
then you broke the game
simple ravine
probably.
mortal bone
Dev talk :)
simple ravine
@mortal bone It's related.
elder hearth
oh new gems, Path of More incoming?
@simple ravine so about your C# code
modifiersToUse is a List with already filtered mods by your base item tags? (like "body_armour", "two_handed_weapon", etc?
simple ravine
@elder hearth Yes
elder hearth
ok
@simple ravine are you aware that there are some mods that adds tags to your item, and you will need to reevaluate your modifiersToUse (they affect GenerationWeight_Values)
just making sure if you have that before your code
elder hearth
can you say how you randomize it (pickInt), in Python i do
random.choices[modList, modWeightList]
and it is already gives me the mods
mod*
totalWeight is it a list or Int?
simple ravine
what is the structure of modList and modWeightList in this case?
var totalWeight = modifiersToUse.Sum(p => p.GetSpawnWeight(item));
This will get the total weight of all the applicable mods. (int)
var pickInt = Randomizer.Next(0, totalWeight);
This will randomize a number between 0 and the total weight of all mods. (int)
elder hearth
modList = ['ModIdHere', 'SomeOtherMod', 'MoreModsHere']
modWeightList = [1000, 500, 250]
simple ravine
Oh, okay so python has a weighted randomizer built in
elder hearth
yes
simple ravine
numpy?
elder hearth
random
simple ravine
ah ok
mortal bone
Base random library yeah
simple ravine
I'm using a cryptographic randomizer for mine
elder hearth
i was reading about having the best randomizer and so, but what i read so for the random one looks decent, also don't know what kind of randomizer GGG uses
what algorithm or something
simple ravine
probably something equivalent to that
based on using System.Security.Cryptography;
elder hearth
so with my example, you would randomize from 0 to 1750?
simple ravine
yes
elder hearth
🤔 from 0 to 1000 you got 1001 numbers? or am i missing something?
simple ravine
well, I should perhaps have 1 to 1750
elder hearth
ok, so make a cumulative weight list, and stop when you reach that number?
1000, 1500, 1750
simple ravine
what would the difference be in comparison to the current method?
elder hearth
sorry its a question
simple ravine
I'm not familiar with cumulative weight list, and how it works, sorry
elder hearth
you are doing that i guess
i += v.GetSpawnWeight(item);
i gets 1000
then 1500
then 1750
?
simple ravine
well it doesn't sort the list based on weight
elder hearth
no need to sort
simple ravine
ok, yeah then I guess it's that heh 😃
elder hearth
.First()
first what? xD
simple ravine
var i = 0;
var pickedModifier = modifiersToUse.SkipWhile(v =>
{
i += v.GetSpawnWeight(item);
return i < pickInt;
}).First();
This will loop until i >= pickInt and pick that item (.First())
simple ravine
SkipWhile will skip all the items until criteria is met
and then chained with First to pick the first element
(edited, found a typo)
elder hearth
looks fine for me then
if you already handle GenerationWeight_Values with GetSpawnWeight i don't see anything that would cause an issue
@simple ravine GetSpawnWeight handles already rolled/crafted ModGroups too, right?
simple ravine
yes
well, not GenerationWeight as of right now, but it will, I am adding it in a second
private static int GetSpawnWeightImpl(this Item item, IModifier modifier)
{
var tags = item.Tags.ToList();
var mod = modifier.SpawnWeights.FirstOrDefault(c => tags.Any(e => e == c.Tag));
return mod?.Weight ?? 0;
}
velvet fog
all shields got new spawn mods
simple ravine
On class Item:
public IReadOnlyList<string> Tags =>
Explicits.SelectMany(c => c.Modifier.AddsTags).Concat(Base.Tags).Distinct().ToList();
item.Tags is with other words tags added by adds_tags on existing explicit mods and then concatenated with the base items tags.
elder hearth
i see
i need to work on my lambda expressions 😢
seems like it does wonder and i'm not using them much
simple ravine
lambdas ❤
elder hearth
@velvet fog poecraft.com dev said that there are some mods (Hybrid +life +ES, in Str armour for example) in .dat file but we can't roll or reproduce that in beta, do you know something about that?
@simple ravine do you know about chaos spaming an item with Multimod + Cannot chance prefix + cannot change suffix? 🤣
elder hearth
i watched DemiGodKings videos
elder hearth
but the can add mods if you have open something
so when you roll 4, 5 or 6 mods, you need to subtract already existing mod count
velvet fog
@elder hearth I don't sure what you mean
elder hearth
demigod used some chaos before he hit 6 mods roll and then added that Fire Resistance mod
@velvet fog let me link reddit post
brownymaster post
obtuse citrus
he's pointing out that's probably a bug or not updated in the data
velvet fog
I don't get problem on that, checked the tags: str_armour, body_armour, armour
elder hearth
@velvet fog he is saying that he can't roll that ingame
but i'm not playing beta
i can't test
simple ravine
@elder hearth Brain super fried, mind reminding me how I apply generation_weight to final spawn weight?
velvet fog
I don't have beta access too, can't test it
simple ravine
private static int GetSpawnWeightImpl(this Item item, IModifier modifier)
{
var tags = item.Tags.ToList();
var selectedMod = modifier.SpawnWeights.FirstOrDefault(c => tags.Any(e => e == c.TagId));
var factor = modifier.GenerationWeights.FirstOrDefault(p => p.TagId == selectedMod.TagId)?.Weight / 100 ?? 1;
return selectedMod?.Weight * factor ?? 0;
}
I am assuming this is wrong?
elder hearth
@simple ravine its just a multiplier %
if it exists like 65, you get your spawnweight (1000 for ex)
1000 * 65/100
new spawnweight = 650
simple ravine
is it based on if tag exists at all on item, or on the modifier in question
oops, that is complete nonsense code, one sec
elder hearth
@simple ravine lets say your item have a spell damage mod (so it added 175 tag or "has_caster_mod")
the next time you try to sum up your weights, when it reach mods that have
"GenerationWeight_TagsKeys": [176, 175, 0],
"GenerationWeight_Values": [100, 65, 100],
for example, it need to match that 175, so you get that 65 value, then multiply with the normal spawnweight
simple ravine
Can it have more than 1 matching?
elder hearth
i think you can, like having %Spell damage + %phys damage (so you will have 175 and 176 tags) but you only grab the first item that matchs anyway
simple ravine
why only first?
elder hearth
its how tags works, you return your left most matching value
Find the Tags on the entity.
Match the Tags against the mod's list of tags.
For the found mods get the associated spawn weighting of the tag.
If the spawn weighting is zero, discard the mod, otherwise use it.
If multiple tags apply, the left-most tags are most important, where the right most tags are the least.
this is about regular spawnweight
not generation
its the left most on your new Mod list, not your Item Tags list
simple ravine
right, i was under the assumption that the first is the only one that matters
elder hearth
the first is the only one that matters
the first that matches?
elder hearth
well, its right...
simple ravine
private static int GetSpawnWeightImpl(this Item item, IModifier modifier)
{
var itemTags = item.Tags.ToList();
var selectedTag = modifier.SpawnWeights
.FirstOrDefault(c => itemTags.Contains(c.TagId));
var factor = modifier.GenerationWeights
.FirstOrDefault(p => itemTags.Contains(p.TagId))?.Weight / 100d ?? 1;
return Convert.ToInt32((selectedTag?.Weight ?? 0) * factor);
}
would that be correct?
that is given 1 item and 1 modifier, what would the final weight be
elder hearth
you may get float values, like 25 * 65 /100
simple ravine
right, is the factor and selectedTag in this function correct though?
actually, selectedMod should be selectedTag
ugh, it's getting late
so in pseudo it does this:
selectedTag = find the first tag (if any) in modifier's spawn tag list that has a match in item's tag list
factor = find the first generation weight (if any) that has a match in item's tag list and divide it by 100, otherwise 1
elder hearth
selectedTag gets the weight value, right?
simple ravine
spawn weight, yes
elder hearth
seems right
simple ravine
selectedTag = { TagId = "some_tag", Weight = 1000 }
ok, let's see if this works
before I get 3 hours of sleep xD
elder hearth
😢
@simple ravine when you get to a point were you can simulate some items we can compare results to see what we get
simple ravine
yeah, that was my thought as well
elder hearth
did you add mastercraft/essences/corrupted mods?
fuck essences, i hated doing that lol
elder hearth
lol isnt that related to some NZ language or something about NZ?
i can't remember
what is that spelldamagejewel?
simple ravine
if I understood it correctly it's something like "tales of the sea"
my logic apparently think it's perfectly fine to have a jewel mod on a steel ring
elder hearth
oh ok, you are rolling steel ring, how did you get that mod 😐
simple ravine
I think I need to take the Item Domain into consideration
compact isle
yup
simple ravine
@compact isle I'm not gonna get a nasty seize and desist letter and a bunch of lawyer fees if I use talamoana.com will I?
compact isle
hah I doubt it
simple ravine
catch me if you can, i'm in sweden! 😛
(although, I need an excuse to pop over to NZ)
velvet fog
ModDomainsID=1 for item
elder hearth
how many possible mods?
simple ravine
var totalMods = mods.Count(p => p.GetSpawnWeight(item) > 0); = 203
Steel Ring ilvl84
does that seem correct?
191 with prefix or suffix only
elder hearth
len(baseitem.basemods)
191
simple ravine
06:28
elder hearth
😱
simple ravine
@elder hearth I can't find any 'domain' information in the .ot-files. But I guess not too hard to map to Item Class though
elder hearth
@simple ravine yeah, there is no domain info there.
What i do is set domain to 1 by default and while i'm getting the tag from .ot files i look for texts like 'Jewel', 'Flasks' and change when necessary
simple ravine
aye, i think i will do that using the ItemClass information
elder hearth
I think i read from InheretsFrom metadata
simple ravine
yea
elder hearth
Are you going to make a lookup table for ItemClass?
simple ravine
I did embed the ItemClass information onto the BaseItem data
It would be really nice if the logic could be purely data driven, but I doubt I can do that right now without having to make a few assumptions
simple ravine
Something tells me that's not right
@elder hearth If you have time, would you like to run your thing to see how much it would take to craft the above mods on a Steel Ring
elder hearth
@simple ravine ilvl 84?
simple ravine
yes
elder hearth
ok
mortal bone
that seems slightly off lol
simple ravine
....slightly
mortal bone
slightly
elder hearth
@simple ravine I still don't simulate inside values rolls like 70-79 life (when crafting you pretty much will divine it anyway if its a top tier item)
i will try to get some tiers/mods (and above)
simple ravine
sure, that's perfectly fine
elder hearth
for all resistances 10%, can i get 'of the Kaleidoscope' (AllResistances3 - 9-11%) or AllResistances4+?
simple ravine
I'd choose AllResistances4
elder hearth
ok
what about pyshical damage?
Gleaming Req. Lv. 35 Adds (5-7) to (11-12) Physical Damage to Attacks
Annealed Req. Lv. 44 Adds (6-9) to (13-15) Physical Damage to Attacks
simple ravine
T1
elder hearth
ok
elder hearth
@simple ravine looks like you are Augmentating regardless of previous mod (if phys/life or resist exists or not)
but this looks hard to hit anyway :|
i let it run until it reachs 10 rings and it is still running
simple ravine
Maybe I'm not crazy afterall
private void AttemptReachGoal(Item item, Dictionary<Type, int> cost,
IReadOnlyDictionary<string, IReadOnlyDictionary<IStat, int>> desiredModGroupValues, CancellationToken ct)
{
while (!ct.IsCancellationRequested)
{
if (item.Rarity == ItemRarity.Rare)
Roll<ScourOrb>();
if (item.Rarity == ItemRarity.Normal)
Roll<TransmutationOrb>();
if (item.AllDesiredSatisfied(desiredModGroupValues)) break;
if (!item.SoFarSoGood(desiredModGroupValues))
Roll<AlterationOrb>();
if (item.AllDesiredSatisfied(desiredModGroupValues)) break;
if (!item.SoFarSoGood(desiredModGroupValues)) continue; // <-- added this (edit)
if (item.Explicits.Count == 1)
Roll<AugmentationOrb>();
if (item.AllDesiredSatisfied(desiredModGroupValues)) break;
if (!item.SoFarSoGood(desiredModGroupValues)) continue;
Roll<RegalOrb>();
if (item.AllDesiredSatisfied(desiredModGroupValues)) break;
if (!item.SoFarSoGood(desiredModGroupValues)) continue;
while (item.Explicits.Count < Math.Min(6, desiredModGroupValues.Count))
{
Roll<ExaltedOrb>();
if (!item.SoFarSoGood(desiredModGroupValues) ||
item.AllDesiredSatisfied(desiredModGroupValues)) break;
}
if (item.AllDesiredSatisfied(desiredModGroupValues)) break;
}
void Roll<T>()
{
_actions.Apply<T>(item);
cost.Increment<T>();
//PrintStatus(item, cost, desiredModGroupValues);
}
}
ah yes, there is a little bug in there
elder hearth
To craft
10 rings
2T1 Life/Add Phys + all resist (t2+) using alt regal
Trans: 1300
Alts: 1948571
Augs: 24640
Regals: 1300
Scours: 1299
😐
i will debug and see what makes it hard, maybe too many suffixes to get that lucky regal
simple ravine
ok, i'll run multiple and average them as well
simple ravine
meh, hit a snag... rerunning now
elder hearth
To hit any desired prefix/suffix when rolling 1 mod with alt:
4000/159650 (2.5%)
To hit suffix with 1 mod already desired:
2000/98400 (2%)
To hit prefix with 1 mod already desired:
2000/61250 (3.26%)
To hit Life with regal if you have Adds Phys:
1000/((98400-6000)+(61250-6000)) = 1000/147650 (0.67%)
To hit Adds Phys with regal if you have Life:
1000/((98400-6000)+(61250-8000)) = 1000/145650 (0.68%)
going to run a ilvl 48 Diamond Ring, same stats
simple ravine
so.. it's not as slightly off as it may have seemed from the get-go
elder hearth
@simple ravine yes, but because we are being too specific too and ignoring all the rest, just for testing purpouses
I'm sure we are getting rid of Attack speed, Accuracy, +Str/+All, also +%Specific resistances
elder hearth
for suffix?
simple ravine
IncreasedLife7
AddadPhysicalDamage6
IncreasedEvasionRating7
AllResistances5
IncreasedAttackSpeed1
ChaosResist6
It's gonna cost 100 mirrors lol
elder hearth
i'm not experienced enough to run multi-threading here 😢
i would need to refactor my code to suport it i guess
simple ravine
It's running multiple items in parallel. You don't want to run multiple threads for the same item.
Well I guess you can use multiple threads to filter those mods, and calculate the weights etc
var strategy = new AltCraftingStrategy(allMods);
var queue = new ConcurrentQueue<int>(Enumerable.Range(0, 1000));
var costResults = new ConcurrentBag<IReadOnlyDictionary<Type, int>>();
var tasks = Enumerable.Range(0, 8).Select(t =>
{
return Task.Run(() =>
{
while (queue.TryDequeue(out var _))
{
var result = strategy.Execute(new Item(baseItem, 84) { Rarity = ItemRarity.Magic },
desiredModifiers, CancellationToken.None);
costResults.Add(result);
}
});
}).ToArray();
while (tasks.Any(c => c.Status == TaskStatus.Running))
{
Console.SetCursorPosition(0, 0);
Console.WriteLine($"Queue: {queue.Count,-5} Concurrency: {tasks.Length.ToString().PadRight(10)}");
Console.WriteLine("\r\n");
var result = costResults.SelectMany(c => c).GroupBy(c => c.Key, c => c.Value).ToDictionary(c => c.Key, c => c.Average());
result.ToList().ForEach(c => { Console.WriteLine($"{c.Key.Name,-20} {c.Value,10:###,###}"); });
await Task.Delay(1000);
}
await Task.WhenAll(tasks);
when i needed to run alt regal with Regalias i started my script 4 times, and at the end i just added each result :p
elder hearth
nice!
simple ravine
So it's been running for 5 minutes with those 6 mods, and still no results whatsoever.
Ok, let me create a Chaos Spam strategy and see how that'll work out instead
simple ravine
let me upload the stuff to github, and I'll show you
Line 46 - 92, essentially
Not sure if that's all greek to you @elder hearth , let me know if you'd like me to explain it
elder hearth
@simple ravine the desired mods are currently all inclusive?
simple ravine
desiredMods = the required outcome
elder hearth
ok
simple ravine
With your chaos spam routine, do you exalt as well, or just chaos spam?
elder hearth
you can exalt it, because you will get 4 to 6 mods when spamming, sometimes you will get room for it
simple ravine
right - i'm just curious if you're doing it when you simulate
elder hearth
the only thing i simulated so far with exalteds were regalias
simple ravine
Mind sharing your code?
elder hearth
the alt-regaling logic?
simple ravine
yeah, just curious how your code looks like
elder hearth
ok, it was my first simulation so it contains some workaround for some not implemented things like calculating stats (in this case energy shield, i needed it to know what did i roll)
simple ravine
sure 😃
elder hearth
i'm commenting
@simple ravine https://pastebin.com/jKRmAHiX
mod crafting is that way because it is not fully implemented
regalias.get(986, 0) < 100
986 is the max ES possible, i was running until reach 100 regalias
simple ravine
the underlying library/framework, is that something you've made as well?
elder hearth
i used PyPoe for json files
simple ravine
baseitem.craftmod('StrMasterFireResistanceCrafted2')
no I mean the craftmod function etc
simple ravine
oh hehe
simple ravine
probabilities would be intersting to compare
simple ravine
yea, i just picked a random function above
velvet fog
Essence Mod could have higher ES on Regalias?
elder hearth
which PoE version?
def __rollrare(self, roll_type):
if roll_type == 'reroll':
self.__clearmods('regular')
self.rarity = self.RARE
if roll_type == 'essence':
affixes = random.choices([4, 5, 6], [65, 25, 10])[0]
affixes -= 1
else:
if self.basedomain == 11:
affixes = random.choices([3, 4], [65, 35])[0]
elif self.basedomain == 2:
print('Flasks can\'t be rare.')
return
else:
affixes = random.choices([4, 5, 6], [65, 25, 10])[0]
current_affixes = len(self.prefixes) + len(self.suffixes) + \
len(self.cr_prefixes) + len(self.cr_suffixes)
affixes -= current_affixes
if affixes > 0:
self.__add_affixes(affixes)
meh, it doesnt mark down
@simple ravine
i roll 4 mods 65%
5 mods 25%
6 mods 10%
if they are jewel, 65%, 35%
elder hearth
careful with item.Reset() in the future with those Cannont Change master mod 😛
simple ravine
hehe indeed
I want to find an as elegant method as possible
This code doesn't really adhere to SOLID entirely, nor will it - but I'm going to strive to have it as good as possible, so it's easy to build on
elder hearth
yeah no need to rush
elder hearth
@velvet fog is the poedb guy
velvet fog
http://cb.poedb.tw/us/cb.php I will make gem changes at weekend
prisma prism
@velvet fog do you have uniq changes already?
keen owl
Hey guys, i have a question about running trade API, http://prntscr.com/fl6sqi is this server be able to do it?
prisma prism
Depends on what do you mean by 'it'
If you only want to poll trade api and, for example, send discord message when something you want is found then you don't need any power at all, the cheapest 5$ digitalocean vps will be more then enough
keen owl
Im still learning, but the goal is to collect data and make graphs of unique item prices over time, i dont care about sniping, for now.
mortal bone
You will need a shit ton of space or offload the data somewhere
prisma prism
you will not need much cpu and ram then, not sure about storage though
mortal bone
Depends on how long he wants to run it, but if you just held league, prices and unique name it may be good for some time
prisma prism
but if you will optimise data collection properly, you can save a lot of space (for example store all daily data in separate table and then merge it into a single 'daily' price for each uniq for your graph)
keen owl
Something like that is the plan, i dont plan on saving all the data in 3 month league
But as i said, im not very experience so this will be my learning project 😃
mortal bone
Poe.ninja does kind of what you are trying to do
simple ravine
Or perhaps not use sql-esque database for those type of stuff in the first place.
I'll let you in on a little secret. There are databases that are specifically built for these type of problems.
But they are built to be a cluster, meaning a 1-node setup isn't going to cut it
chrome topaz
mr condescending guy are you going to tell us a name of such database?
simple ravine
I'm sorry, it wasn't meant to be condescending at all. I just got caught up with some things
I'd take a look at Datomic and Druid
We use Druid in production right now for our internal ad-tech solution, and it's great
chrome topaz
I see
simple ravine
They're built to store temporal data, which is essentially what @keen owl was looking to do.
Datomic is more based on events, however it can be interesting to look at it briefly
Perhaps that fist comment came off wrong, but I see a lot of people default to SQL, even in 2017 😃
gritty olive
Some form of a TSDB should work if all you're doing is timeseries data
No need to store the raw logs if you just want a chart
mortal bone
I would go with SimpleDB on AWS before I go for something large like datomic and the like.
No need for a distributed system for a hubby project
prisma prism
i understand all that hype about nosql
but honestly, i don't think this is a case for it
lost wigeon
@keen owl I'm the author of poe-rates.com which is a site that lets you see price trends for unique, gems, div cards and so on. That may be obvious, but the amount of RAM you will have on your server will definitely condition the amount of items you can index and retrieve efficiently. The backend is running on an Intel Xeon W3520 (4 cores / 8 threads at 2.66 GHz+) with 16 Go RAM and 300 Go SSD storage. The indexer and the computing programs are written in C++ with MySQL storage. A bit old school I know, but it works nicely 😃
keen owl
@lost wigeon Awesome website, do you maybe have data for whole league?
lost wigeon
I only keep up to 1 month of data to keep everything running fast enough. I'm not sure I would be able to handle more data efficiently.
keen owl
Too bad, i really want somethnig that can track whole league (only one only sofcore intrests me) so i can see trends and stuff like that, im a trading junkie. poeprices.info does a good job of doing that but sometimes intervals are way too long. example: http://www.poeprices.info/uniqueitempages/Xoph_s_Blood.html.
I guesss im gonna have to build it myself ><
Anyway, thank you everyone for providing ideas, suggestions and advice.
lost wigeon
Sure, good luck with your project and feel free to ask for some advices
gritty olive
@mortal bone tbh I would just go with a SQL based database, it's not like they are doing anything intensive and when it becomes intensive they can move, over-engineering is a real thing 😛
mortal bone
Yeah, I know. I wasn't suggesting anything much more advanced than just a simple database solution
Someone was suggesting cloud based solutions and such
simple ravine
Refactoring and re-architecting tech-debt is also a real thing...
mortal bone
But getting a proof of concept/prototype doesn't require scale.
simple ravine
That I agree with
mortal bone
Even self use or a small number of uses doesn't require much
It is hard to suggest the solution when we only know part of the problem I guess
simple ravine
The way I interpreted it was
- to store time-based pricing information
- data was going to become somewhat heavy, due to entire leagues
Might have misunderstood the situation, but I agree that considering that he's learning, SQL might be the best entrypoint, however some of the other databases out there today aren't that much more complicated, and some also run in single-node settings. If you choose to use cloud-based solutions from azure / google / aws, then you don't really have to worry that much about scale.
The biggest headache we face is to deal with database infrastructure at scale. It becomes a big part of planning and execution ops-wise.
The other part about SQL type of databases are schema changes in production environments, where you have > 10M rows is - you don't really want to do that, and will be faced with data migration situations which require decent amount of work
desert needle
one thing I found incredibly useful when first writing code for the stash tab api (and frankly the forums before as well) was to keep a local mirrored copy, either in flat files or something like kafka, to replay when fixing bugs/etc. on conversion to whatever schema. Much much faster than trying to download it all again.
simple ravine
That is a good piece of advice. If one want to take it further, taking a couple of sample-files to use in domain logic unit testing offline, and then further - integration testing on the latest data to detect any anomalies (this can be a little tricky, but useful if you have a live production site that people depend on)
chrome topaz
@compact isle hi, I noticed that beta stash api started returning images with URLs relative to the domain (i.e. starting with /), how should I link to these images so that they are displayed?
velvet fog
They don't set-up a beta cdn for hosting beta images
compact isle
looking into solutions for that
compact isle
you can now load images with the betaapi subdomain
chrome topaz
okay, thank you
mortal bone
@compact isle same for skill tree icons?
velvet fog
seems only for /Art/2DItem/
mortal bone
Hmm, damn
compact isle
why would you want skill tree icons? o.o
velvet fog
'Website APIs' forum?
compact isle
sshh
velvet fog
Is it a surprise in 3.0?
compact isle
nah I was just going to move the developer docs to a forum so that I can improve on it easier
mortal bone
Easier beta icon download when people mess up their installs lol
sharp wharf
Is there any possibility for a FPS lock besides vsync? I'd like my GPU to have lower load overall and vsync gives me fps drops out of hell x)
like some starting parameter or even in the options menu
Reason: There is some instability when streaming with dips here and there since I can only use NVENC as encoder. I cannot play/stream in fullscreen; OBS will either not grab the game or have some stuttering issues. For this an FPS limiter would be more than great.
visual halo
does anyone know, if path of building takes the multiple hits of spectral throw into account in the damage calcs? can't seem to find it anywhere
it just multiplies the hit dps with the attack speed, but not the hits per projectile
sharp wharf
limited it with nvidia profileinspector now, but that can't be a solution for forever. lesser knowledgeable users will have trouble with that
fair oak
uh oh! i've just build an indexer with couchdb
works nicely but it's ugly.
moving forward with scikit-learn & classification for determining a price for a specific item
how does poe.trade knows how an item is sold or removed ? is the stash id unique and if an item is removed from it, will it appears in the next change id ?
also, is the item id unique for each item ?
(basic question but ive never played with GGG data)
deft jolt
When you add an item to your public stash, the whole stash will be pushed through the api. And when you remove an item, again, same stash but without the item. Poetrade indexes nearly every item out there. Parsing and saving all that data may take a lot of resources. However, just parsing the items without saving is low effort, you could technically even run it off your smartphone these days.
Can't say about stash/item ids though, haven't messed around with those much. I guess poetrade would make some use out of those
simple ravine
You will have to do delta checks
fair oak
can I delta check with item id ?
simple ravine
yes
item id will stay the same, unless:
- it is being crafted on
- stacked / unstacked (for stackable currency)
- changed owners
fair oak
ah perfect, thank you for the clarification
simple ravine
np
fair oak
couchdb is so much fun
fair oak
did you write an indexer and/or used a "document" storage database for these big json blob ?
simple ravine
I've done some preliminary tests so far, working on some other things right now, but what I'd do is separate each item into it's own Document
fair oak
i was thinking the same since it simplify the view by a lot
simple ravine
are those milliseconds or seconds?
fair oak
seconds.miliseconds
simple ravine
yeah, that isn't fast heh
fair oak
i'll use ujson and see if it gets better
actually, it's python-couchdb that is slow, i'm around 0.20s (still slow) with no insert
simple ravine
0.2s for the entire change-result or 0.2s per stash inside of the result from ggg's API?
fair oak
0.2s per stash - i've modified my code to create one document per item instead of one document per next_change_id
desert roost
@timid hemlock Heya, is this a known issue? I was trying to use the Shared Items feature, but editing an item and then attempting to save results in this error http://puu.sh/wtlu1/e23ef677d2.png
timid hemlock
@desert roost Well it is now! Fixed in the next update.
foggy vortex
Anyone else find that poe trade isnt working?
simple ravine
works fine.. actually super fast right now
simple ravine
what is your account name?
simple ravine
lol
foggy vortex
app is just not working.
simple ravine
which app?
foggy vortex
poe trade.
I am pretty sure it just only ever works for around 90% of players.
And we just take turns getting fucked.
simple ravine
which league?
foggy vortex
Standard
simple ravine
poeapp isn't indexing your crap either
jedininjman -> account doesn't exist according to pathofexile.com
simple ravine
simple ravine
@chrome topaz doesn't like you 😛
foggy vortex
Prob not
cause I message him whenever his app isnt working
so he hears from me a lot
Not his fault though. It's just completely fucking ridiculous ggg havent added actual trade support. Like this game would not playable without its 3rd party apps. Completely absurd situation.
simple ravine
And still, they want to keep an official arms-length away from 3rd party developers, relationship wise
foggy vortex
I want xyz just shut his shit down and we can allwatch this game fucking crubmle.
simple ravine
I think he gets enough traffic for it to pay a decent chunk of change
foggy vortex
and yet ggg has done fuck all.
simple ravine
it's the same intelligence as when they say "we're not going to use BI tools to understand how to improve this game, but we'll use our guts"
that's how you'll be left behind in the dust heh
lethal flume
chrome topaz
^ yep
foggy vortex
Cool it would be awesome if ppl searched that way.
I find most of them search like this
wher oyu notice, none of my stuff shows up
@chrome topaz any idea why the currency trade interface does not show my listings?
It is the same for gear, I search for affixes my for sale gear has, and nothing comes up.
summer wadi
It doesn't make sense why it would show up on one but not the other.
foggy vortex
@summer wadi yea which is why I have been trying to message the dev for the past few hours
Also its not the first time.
foggy vortex
@lethal flume I am online.
and have been for hours.
and even messaged is thing to indicate I am online
two hours ago
Dont get me wrong, I equally appreciate your tone and helpful insight though
Especially the helpful insight.
Half of the other ppl I message are offline so I think it might just be bugged out. But it seems many offline ppl appear as online normally so who knows.
chrome topaz
I think it's on ggg side, i don't see you in online api @foggy vortex
foggy vortex
that it then? Its been this way for last 5 hours atleast.
chrome topaz
@foggy vortex what's your ign that you are playing right now on?
foggy vortex
account: jedininjaman
char: myfreedomandmybowthats
chrome topaz
did you just log on?
foggy vortex
Sev min ago yes, I was on the last four hours though.
chrome topaz
yeah the api's broken, i'll ask them to fix it
foggy vortex
@chrome topaz ty. Fyi whats the best way to contact you? This happens a lot but I dont want to annoy you or spam you on forum / discord / etc
it is completely pointless for me to pay this game without trade so this is imporant to me.
lethal flume
The ladder online and Public online checks are separate right?
chrome topaz
it's because poeapp sets users as online when they update their tabs
iirc
the checks are separate
lethal flume
Cool
Then @foggy vortex start power leveling so you show up on any ladder. 95+ in sc and 90+ in hc.
foggy vortex
@lethal flume Im good. I pretty much only play as a merchant and play random builds
chrome topaz
you could also use acquisition to set online status
foggy vortex
if i level to 95 would it fix this issue?
chrome topaz
no, because I actually remembered I removed ladder check a while ago
lethal flume
rip
foggy vortex
guess im installing more 3rd party shit just to play this game.
chrome topaz
it took like 30 minutes to refresh so it was really unreliable and contributed a lot to the "i message online players but game says they are offline" problem
foggy vortex
@chrome topaz ive always wondered what your stance on obvious proce manipulators are?
Like Husky Dingo is an obvious one. Dude never trades and has pages of listings.
Hes been doing it forever.
chrome topaz
report them to ggg
foggy vortex
ok
simple ravine
Hehe... couldn't resist the worst pun of the day.
@chrome topaz why would GGG do anything about them?
chrome topaz
it's their game
foggy vortex
I installed acquisition .6 and logged in, It saays online in bottom right corner
is that what you menat?
simple ravine
Anyone else having rotated monitors? I'm having some latency issues on the flipped ones
chrome topaz
there's a tutorial somewhere on youtube, you need to put your online link in
foggy vortex
I made thread listed one item in it and entered the thread number
entered the thread number in aquisition
lethal flume
chrome topaz
you don't actually need to create a forum thread if you list items with the api
foggy vortex
jesus. Just in time for me to get off and make dinner
thanks for help all, was it the aqusition stufff that put me online?
or ggg just turn server on
pseudo ocean
Hello i am making a XYZ killer can someone help
chrome topaz
probably acquisition
deft jolt
is my script not working or is the stash api janky?
deft jolt
it is working fine now
though
http://cdn.poe.ninja/api/Data/GetStats nextchangeid has jumped back quite a bit
so mightve not been just me
simple ravine
jumped back?
deft jolt
i used that to start up my script, got trades from way back
should be in the 60mil range
i guess
deft jolt
yes
deft jolt
ive no idea what happened, but it started at around 2 hours ago
It's highly likely something happened on poe.ninja's side
And since my tool decided to grab a changeid from there, it messed mine up aswell
simple ravine
oh, you don't store the ID yourself, rather get poe.ninja's latest
deft jolt
I do, however in case of a hickup it grabs one from poe.ninja
And it overwrote mine, so yeah
simple ravine
ouch 😦
I'm gonna take a break from my utility tool and start working on an indexer as well
deft jolt
Utility tool?
simple ravine
Multi-purpose. It'll be a crafting simulation, trade-macro/item-info, and sniping tool
deft jolt
Oh nice, would be cool to see em all bundled up
simple ravine
yeah... I also wanted a bit richer experience than the current item-info/trade-macro ahk scripts provide
elder hearth
Some people are summoning @timid hemlock to add this
https://www.reddit.com/r/pathofexile/comments/6jl4qh/discussion_doryanis_fist_new_ability_lightning/
fair oak
making an indexer is easy
try adding machine learning to it to predict the price of an item
yolo regression
simple ravine
and what features are you adding?
fair oak
i wish i've indexed at least 3 month of data
simple ravine
which are categorical and which are numerical?
I don't see any machine learning algorithms in the link you posted. It's just a linear regression model
fair oak
yeah, well, i must admit that behind the buzzword, it is a model that i'm planning to use (and this model is probably the most basic one beside classification model)
it's considered machine learning but yeah, it's only some basic math/stats 😛
simple ravine
so you don't use any support vector machines, neural networks, bayseian stuff etc?
fair oak
nah and i don't know how to use tensorflow yet
i'll do simple stuff for now such as predicting the price of a "prophecy wand" and i'll throw up additional variables such as item properties
simple ravine
are you doing this on items or just currency?
heh, allright
it's been discussed at length, and some people already tired this, fyi - but don't get discouraged though
what i am trying to say is that throwing some obvious parameters into a regression model, will probably not be good enough for a good probability/accuracy
fair oak
yeah... i'll see how it goes, maybe i'll try the classification model to estimate the value of the item.. it'll be easier
simple ravine
you'll spend more than 80% of the time cleaning and preparing your data, and 20% sciencing the shit out of it
fair oak
and i'll end up playing the game instead, right ? 😃
fair oak
i wonder what sgdb poe.trade is using
simple ravine
sgdb?
fair oak
database
simple ravine
I know as much as he's using a lucene based (elastic search), but not sure if he stores it in a canoical database as well
fair oak
i wonder what is the optimal setup for indexing GGG feed
simple ravine
u said it was easy...
fair oak
ex: an indexer that is able to track stashes removed items
i find it easy to store the data in couchdb or postgres but updating/manipulating it in an effective and efficient manner is more challenging
simple ravine
you'd probably want to move the logic as close to the data as possible - if possible, manipulate the data in a stored procedure
fetching, mutating and updating might not be your best choice
prisma prism
i'd try to use relational db for the storage first time. in postgres there is also jsonb type so we can have table with basic fields (ids etc) and put all variable stuff into jsonb field
Not sure about performance though, but it may be enough
velvet fog
it's work for small scale, for me, in garena taiwan trade search, only have 3.5m to 4m items for standard league
using Elasticsearch https://www.elastic.co/ will better, but im lazy
opaque hatch
hey guys, What is the current best way to search for an item listed in poe.trade
doesnt need to be the fastest, like an item sniper just one that can put in certain values and search, obviously I can write a script to fill out the form using post and get the response back but wondering if there is a better way
before I started was hoping the link up the top would be something like poe.trade&search=rare+ilvl65+dmg43 wtc but its seemingly random words hiding it
chrome topaz
you can use GET instead of POST
opaque hatch
@chrome topaz when you make the get request does the server just return the entire page source?
ancient nymph
@opaque hatch use live search
opaque hatch
@ancient nymph what do you mean by this? the live sreach still uses the random letters in the name. What I am trying to do is submit a query by filling in certain fields and then return the result.
the idea is that I can in the end use the ctrl + c data from in game to automatically search for rares of similar values to find its worth
ancient nymph
you dont need to search poe trade for that, just highlight t1 rolls
opaque hatch
?
ancient nymph
3 Tier rolls are usually worth something, unless its a base no one uses like assassin garb
3 tier 1 rolls*
opaque hatch
I think we are on different wavelengths here
ancient nymph
no, you are trying to compare items on poe trade to find value??
opaque hatch
yes
ancient nymph
yea using the tier 1 count is a good heuristic to bypass that, unless you just trying to learn
opaque hatch
yes sort of learning
but
the core focus of this
is to SEARCH poe trade for similar items, I have a few ideas from some other functionality probably nothing that hasnt already been done.
ancient nymph
i think you can dothat why highlighting hte tier 1 rolls of hte item then searchign those, otherwise you get too limited search
opaque hatch
knowing if it had 5 t1 rolls tells me nothing about its value in that market without further research
ancient nymph
yea you just search those, if you add in the other rolls, you might find just a few items and it skews your valuation
opaque hatch
ok yeh I se what you are saying now
so back to my original thing I am hvaing trouble figuring out what the best approach to getting information back from poe.trade. Since the links are randomly generated or at least appear to be it sems difficult to send information off and recieve information back. Like i need a combo of GET and POST to achieve the result
ancient nymph
you can scrape the table for data, althought if i remember right theres a api i think
elder hearth
PoeTradeMacro's advanced search uses something like that and builds an url with selected fields
ancient nymph
o i got it now @opaque hatch just download the poe api yourslef and search it yourslef, instead of using poe trade, that would be easier
elder hearth
You can take a look at poetrademacro source code to learn how to build their URL
opaque hatch
I dont think I ahve the power to scrape through the entires worlds stash tabs ahah. @elder hearth Yeh I am going to have a look at that now cheers
yep it seems they ahve figured out hwo to build the link or at least generate the page
ancient nymph
@opaque hatch http://pathofexile.gamepedia.com/Public_stash_tab_API
its not a large file
elder hearth
@opaque hatch what is your goal? Are you a user or dev? If user just use poetrademacro or http://poeprices.info
opaque hatch
wannabe dev
ancient nymph
man what language are you doing this in @opaque hatch
it would be easy to query that json in python, they already give all the keys
opaque hatch
probably python or c# if I wan to make a gui for it
i have been looking at that page you sent me, so my understanding is that file contains everything
ancient nymph
yep
ancient nymph
you will just have to keep updating the api file yourslef
opaque hatch
yeh
the file is only 4.12mb so pretty managable
well the one I just tested was anyway
simple ravine
@chrome topaz the url structure for searches - those are server-side generated IDs, I presume?
they seem too short to be something else?
velvet fog
it's md5/hash for queries
simple ravine
right, so it's nothing you can generate client side, similar to skill-tree query
deft jolt
@opaque hatch a simple way of getting a poetrade search url
def get_example_pt_url():
payload = {
"name": "Grand Spectrum Viridian Jewel",
"rarity": "any",
"league": "Legacy",
"has_buyout": 1,
"online": "x"}
r = requests.head("http://poe.trade/search", data = payload)
return r.headers["Location"]
@deft jolt thanks sardel sorry Just seen it now. Will check it out when I get home 😃
thanks that was exactly what I wanted mate!
not I just get the source of that link and webscrape it I am assuming
opaque hatch
So i have most of the script working thanks for Sardel, however I am having issue specifying the different mods avaiable into the payload. Been fiddling around the inspect tab for an hour or so now.
simple ravine
hmm, funny how translatoin texts says "Supported Skills deal {0}% increased Elemental Damage" on opal rings
meanwhile in game it says "{0}% increased Elemental Damage"
any idea why that is the case?
opaque hatch
nope haha
I am still trying to figure out how to add mods into that search payload.
"mod_name": "# Life Regenerated per second",
"mod_name": "63",
"mod_name": 2,
Are the only combinations I can think so so far, I just keep getting 500 error. I have figured out every other field except for this one
I ahve also tried with prop_mods
I figure I need to do something with an onclick request or something to actually add the property to the page.
simple ravine
ah, figured it out - RePoE was initially grabbing "wrong" translations
deft jolt
@opaque hatch
payload = {
"name": "vaal regalia",
"rarity": "rare",
"league": "Legacy",
"online": "x",
"mod_name": [
"(pseudo) (total) +#% to Cold Resistance",
"(pseudo) (total) # Life Regenerated per second",
"#% increased Energy Shield"],
"mod_min": [10, 1, 100],
"mod_max": ["", 10, ""],
"group_type": "And",
"group_min": "",
"group_max": "",
"group_count": 3}
empty fields are important
opaque hatch
ah ok
its funny I just actually moved on to using the stash api
becuase I could not figure it out
opaque hatch
this might be a dumb question, and I am pretty sure I know the answer just want to check, I found out that the file size returned from the stash api is roughly 0.5mb. Does this mean that everytime I request it I am downloading 0.5mb. Thereofre if I caleld it once every 2 seconds and ran it for an hour it would download roughly 0.5 30(per min) x 60(per hour) = 900mb per hour?
I got that here
deft jolt
more or less yes
opaque hatch
hmm, ok how on earth do people run these sniping tools on their home computers
deft jolt
unlimited network plans
opaque hatch
haha ok
deft jolt
or you could just get a server for around $4 a month
opaque hatch
and i guess I would also be correct in saying that if I only ran to tool every say 10 seconds I would miss data, and potentially items
deft jolt
no, you wouldn't miss anything
however
if you're going slower than the data is being generated
you might end up getting delayed trades
opaque hatch
as far as I can tell the json api only returns the last 20 accounts to update their public stash
so lets say only ran the script every 10 minutes and 500 character stashes got updated wouldnt that mean I miss 480 of them?
deft jolt
if you follow the nextchangeid from the api, no
opaque hatch
Oh wait hang on I think I see my mistake now, I am using the most current changeID always insterad of using the one from the previous json file
yep i get it now, I am currently just grabbing the most recent one
so really you could build an entire database of everyones stashes from the moment you start he script as long as you kept it running. Which I guess is exactly what poe.trade is
deft jolt
Ye
simple ravine
wait you don't have unlimited network on your broadband connection?
simple ravine
is that common in Australia with other words?