#tooldev-general | Path of Exile 1 & 2 | Page 32

desert needle Jun 7, 2016, 8:28 PM

#

that's a direct pastebin so I don't know if it's kept up to date or if that is necessary

mortal bone Jun 7, 2016, 8:28 PM

#

would anyone actually be interested in seeing that?

feral flame Jun 7, 2016, 8:28 PM

#

Ah, nice! Thank you very much 😃

desert needle Jun 7, 2016, 8:28 PM

#

I think 99.999% of all people just use Neversink's and don't care about editing :x

loud umbra Jun 7, 2016, 8:37 PM

#

While his is popular, I think that number may be a tad off. There's really no way to tell how many people use their own or modify publicly available ones for their own use though.

mortal bone Jun 7, 2016, 8:40 PM

#

We need a community hub for item filters

#

Something like the mod nexus

desert needle Jun 7, 2016, 8:41 PM

#

that'd be cool

loud umbra Jun 7, 2016, 8:44 PM

#

How would you sort it / make it searchable in a way that someone could find one meeting their needs? Defining the properties of a filter such that people can find those best meeting their needs and requirements seems like the hard part.

desert needle Jun 7, 2016, 8:45 PM

#

might just be something to browse by images until you see a style you like

#

I forget how much complexity can go into a filter, I literally just want sound for basic drops and some basic highlighting

deft tundra Jun 7, 2016, 8:45 PM

#

uhh how like the minecraft texturepacks use this one scenery with all the blocks placed

mortal bone Jun 7, 2016, 8:46 PM

#

It would be a big project, but I think it would be really cool

deft tundra Jun 7, 2016, 8:46 PM

#

so you'd want ppl to drop like the different tiers to showcase the colors, that's be a way

mortal bone Jun 7, 2016, 8:46 PM

#

You would have to create a parser

#

for item filters

#

You would want to same data to be shown for every filter

desert needle Jun 7, 2016, 8:47 PM

#

http://bschug.github.io/poedit/poedit.html like that one

mortal bone Jun 7, 2016, 8:47 PM

#

Obviously you couldn't show it handling everything because that is too much, but you could have people add a custom filter

#

as they were browsing

desert needle Jun 7, 2016, 8:48 PM

#

with a next-prev to go through filters, something really simple like that might be good enough to start

mortal bone Jun 7, 2016, 8:48 PM

#

Yep

deft tundra Jun 7, 2016, 8:48 PM

#

omg that is good. thanks Trackpete

loud umbra Jun 7, 2016, 8:49 PM

#

Would also be cool to support multiple types for one filter. So someone could upload their normal, MF, animate weapon variations.

mortal bone Jun 7, 2016, 8:49 PM

#

I haven't fully fleshed this out, but I might be able to come up with some concept 'art' next week

#

Well, it would be more like an account they log into and upload each filter

#

with some sort of versioning system

#

You could have a web version and a client version which could manage your filters for you

desert needle Jun 7, 2016, 8:52 PM

#

doesn't html5 support local file modification?

mortal bone Jun 7, 2016, 8:52 PM

#

Yes

#

should

#

the whole HTML5 desktop stack does somewhere

desert needle Jun 7, 2016, 8:53 PM

#

a web page where you could go, find a filter, click a button, approve something, and have it automatically place it, that would be amazing

#

I just keep my filter up in notepad++ and copy/paste over it every few months but for most people messing with filters is such a pita

#

I wonder if the default item filter is saved in the poe ini file

mortal bone Jun 7, 2016, 8:54 PM

#

I don't think the default is saved

desert needle Jun 7, 2016, 8:54 PM

#

item_filter_loaded_successfully=neversink.filter```

#

in production_Config.ini

#

it probably only loads that on poe startup, but wow that could be amazing if you automatically install and choose it

mortal bone Jun 7, 2016, 8:55 PM

#

I think you have to click 'reload' in game

#

I am not sure you would want to force overwrite with a bad filter

desert needle Jun 7, 2016, 8:56 PM

#

yeah but at least for players who have poe closed and download a filter, then start it up, you could potentially activate it if they chose

mortal bone Jun 7, 2016, 8:56 PM

#

Maybe? I am not sure how that works

desert needle Jun 7, 2016, 8:56 PM

#

a lot of people seem to struggle with the steps of finding the right directory and naming the file appropriately

#

but auto selecting it in the ini for future loads would just be icing

mortal bone Jun 7, 2016, 8:57 PM

#

it is an easy test

#

you can force load

#

on restart, let me test otherwise

#

Hmm, you can't while the game is running. It looks like settings are loaded once on startup

desert needle Jun 7, 2016, 9:01 PM

#

not too surprising

#

can it detect new .filter files while it's running?

mortal bone Jun 7, 2016, 9:04 PM

#

Yep

#

you have to reopen the options menu though

desert needle Jun 7, 2016, 9:08 PM

#

interesting

mortal bone Jun 7, 2016, 9:08 PM

#

I didn't test opening/closing menu after changing in config

#

still doesn't load the filter just repopulates the filter list

#

It is still viable enough solution to either not have the game running or close/open options and select the filter

wind garden Jun 7, 2016, 9:17 PM

#

wait wait wait.....

#

you just tell elastic serach to index and object, and it does it?

#

and then you can magically query it without any other work?

mortal bone Jun 7, 2016, 9:18 PM

#

you make an object and you say hey I need this...you got it? and it will say hey here's the stuff, bro or na, I aint got that

wind garden Jun 7, 2016, 9:18 PM

#

i should really re-read trackerpete's blog on creating an indexer

#

if it's this easy, i might as well make my own

#

and then trackerpete says, why not just use mine

mortal bone Jun 7, 2016, 9:19 PM

#

it is really straight forward

wind garden Jun 7, 2016, 9:19 PM

#

see here's the thing, i'm convinced that there's a delay on the bigger indexers because they have to process every item

#

even if it's just a few seconds

#

seconds matter when you're trying to snipe an item

mortal bone Jun 7, 2016, 9:20 PM

#

You have to do some amount of processing

wind garden Jun 7, 2016, 9:20 PM

#

if you're indexing, yes

mortal bone Jun 7, 2016, 9:20 PM

#

If you are just reading the river then no

wind garden Jun 7, 2016, 9:20 PM

#

ya

#

i'm just getting the latest, checking if it contains an item i care about, and throwing away the rest

#

as lean and fast as it can be for my application

#

so i think i'll use the exiletools apis for doing analytics on price/volume

#

still pretty cool though

woeful sphinx Jun 7, 2016, 9:25 PM

#

Anyone here that can fix bugs with gem data in the Wiki?

#

http://pathofexile.gamepedia.com/Elemental_Proliferation_Support N/As for "less Damage" in table
http://pathofexile.gamepedia.com/Vaal_Spark "Fires 0 Projectile in a random direction"
http://pathofexile.gamepedia.com/Warlord's_Mark N/As for "add. Stun Chance" in table

mortal bone Jun 7, 2016, 9:28 PM

#

That would be the guy that runs PyPoe

#

I think he updates the wiki with a bot

#

I don't know if he is on discord

woeful sphinx Jun 7, 2016, 9:31 PM

#

There is at least some guy here that was talking about PyPoe a lot

compact isle Jun 7, 2016, 9:45 PM

#

good lord I finally caught up with all the chatter in this room

#

certainly more active than the IRC 😄

wind garden Jun 7, 2016, 9:58 PM

#

hooray

#

you see the bit about the empty stash updates?

compact isle Jun 7, 2016, 10:01 PM

#

yeah, not too sure if it is meaningful though

#

it might just be preferring sending them from updates with items or something

#

the amount of accountName: null entries is concerning though

wind garden Jun 7, 2016, 10:02 PM

#

there were a LOT of empty stash updates

#

it felt like millions

#

certainly hundreds of thousands at least

compact isle Jun 7, 2016, 10:05 PM

#

hm

mortal bone Jun 7, 2016, 10:14 PM

#

The null accountName stuff is probably from changing account names

chrome topaz Jun 7, 2016, 10:18 PM

#

i think it's from remove only tabs

mortal bone Jun 7, 2016, 10:19 PM

#

That wouldn't make sense for an account name to be null though. If it is public then there should be an account name attached to it

chrome topaz Jun 7, 2016, 10:20 PM

#

that's what Jonathan told me

mortal bone Jun 7, 2016, 10:24 PM

#

interesting...

compact isle Jun 7, 2016, 10:34 PM

#

xyz is correct

wind garden Jun 7, 2016, 10:35 PM

#

@desert needle what's the purpose of the API key if you can query without one?

#

with regards to your apis

#

or is it only http://api.exiletools.com/stats/_search that doesn't require an api key

wind garden Jun 7, 2016, 11:54 PM

#

What version of eladticsearch are you using?

#

elastic**

deft tundra Jun 8, 2016, 12:15 AM

#

does the community wiki have a library for all the alert sounds and other sounds you can put into the loot filters?

old beacon Jun 8, 2016, 12:34 AM

#

according to http://www.pathofexile.com/forum/view-thread/1260664
theres 9 sounds, but i dont know if the wiki has little play things to sample sounds

deft tundra Jun 8, 2016, 12:37 AM

#

only the 9 okay thanks though.

old beacon Jun 8, 2016, 12:43 AM

#

ill whisper you a short lootfilter to test the sounds with readily available currency

desert needle Jun 8, 2016, 1:38 PM

#

@wind garden - only the main item index requires an api key (though the other ones I think might be out of date and pointing at an old index)

#

I plan to do away with those at some point, just haven't gotten around to it.

#

Right now I'm using Elasticsearch 2.3

#

Retik: you just tell elastic serach to index and object, and it does it? <-- lol no. The base item JSON is not usable at all.

#

I have hundreds (thousands?) of lines of code that converts that item JSON into an actual document that can be indexed properly

#

the way the base item JSON is set up is designed purely for display by their javascript app on the web page and is TERRIBLE 😉

#

they aren't in data pairs, bizarre sub arrays are used, etc.

wind garden Jun 8, 2016, 3:16 PM

#

@desert needle the query you gave me the other day doesn't seem to be working for me

#

i just get error 400 bad request

#

{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "attributes.league": {
            "value": "Prophecy"
            }
          }
        },
        {
          "term": {
            "attributes.rarity": {
            "value": "Unique"
            } 
          }
        },
        {
          "term": {
            "shop.hasPrice": {
              "value": "true"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "uniqueNames": {
      "terms": {
        "field": "info.fullName",
        "size": 1000,
        "order" : {
          "avgPrice[50.0]" : "desc"
        }
      },
      "aggs": {
        "avgPrice": {
          "percentiles": {
            "field": "shop.chaosEquiv",
            "percents": [
              50
            ]
          }
        }
      }
    }
  }, 
  "size":0
}

desert needle Jun 8, 2016, 3:46 PM

#

you're running it wrong somehow

#

<copy paste query here>
'```

#

just gave me results

#

😮

wind garden Jun 8, 2016, 4:14 PM

#

i don't know what i could be doing wrong

#

i'm posting that query to http://apikey:DEVELOPMENT-Indexer@api.exiletools.com/index/_search

#

tried with and without header for contenttype = "application/json"

desert needle Jun 8, 2016, 4:19 PM

#

what are you using to post it?

wind garden Jun 8, 2016, 4:20 PM

#

c# WebClient object

#

it's the exact same thing i used to get the stats query from http://api.exiletools.com/stats/_search

#

don't know why it would be different

desert needle Jun 8, 2016, 4:21 PM

#

is it possible the c# WebClient doesn't support basic auth?

wind garden Jun 8, 2016, 4:21 PM

#

hmm no, but it's possible that basic auth isn't enabled by default

desert needle Jun 8, 2016, 4:21 PM

#

you can also try just doing a get on the search endpoint

#

curl http://apikey:DEVELOPMENT-Indexer@api.exiletools.com/index/_search?pretty for example

#

that will help you isolate if it's a post problem or something else going on

#

you should get basic results with an empty query

wind garden Jun 8, 2016, 4:23 PM

#

hmm that fails too

#

could be auth, but seems strange that i don't just get an auth error

desert needle Jun 8, 2016, 4:23 PM

#

Elastic Search expects some headers which WebClient does not pass. I used HttpWebRequest class and it worked fine without any errors since.

wind garden Jun 8, 2016, 4:24 PM

#

oo

#

let me try

desert needle Jun 8, 2016, 4:25 PM

#

that was a random stack overflow from 2012 about elasticsearch, could be way out of date

#

let me know next time you try and I'll see if anything shows up in the server side logs, 400 is somewhat unexpected regardless

#

I see a few 400's in the logs but they're from a bunch of random ip's

wind garden Jun 8, 2016, 4:26 PM

#

i'm at work right now

desert needle Jun 8, 2016, 4:26 PM

#

ah

wind garden Jun 8, 2016, 4:26 PM

#

so probably bouncing off intel servers

desert needle Jun 8, 2016, 4:26 PM

#

so it's rotating, gotcha

#

I see those requests then and can see the 400 bouncing off the load balancer, lemme check the reverse proxy

#

reverse proxy is passing the request but the API gateway is erroring

#

time="2016-06-08T12:10:00-04:00" level=info msg="Attempted access with malformed header, no auth header found." origin="127.0.0.1:30849" path="/index/_search" 
time="2016-06-08T12:24:04-04:00" level=info msg="Attempted access with malformed header, no auth header found." origin="127.0.0.1:37606" path="/index/_search" ```

#

your gateway might be stripping basic auth, you can try just using a direct auth header

#

Authorization: DEVELOPMENT-Indexer

#

some api clients have problems with header injection so the basic auth is the default but adding that header should work

#

1099``` at least the auth header works from curl 😃

wind garden Jun 8, 2016, 5:13 PM

#

lol that was it

#

client.Headers[HttpRequestHeader.Authorization] = "DEVELOPMENT-Indexer";

#

does it matter whether i use DEVELOPMENT-Indexer or my own assigned apikey?

desert needle Jun 8, 2016, 5:14 PM

#

as long as you're not hammering things, feel free to use the development one

#

there's like a 500req/min overall limit on that key but it averages way way under that

wind garden Jun 8, 2016, 5:16 PM

#

nah shouldn't be hammering it or anything close

#

i tried using the .Net elasticsearch libraries

#

the documentation is nearly unusable

#

they have documentation for v1.x

#

but v2.x is bascially just a diff of apis they changed

#

they don't have any documentation that's been updated with examples for 2.x

#

but i'm pretty sure they're intended to be used for elasticsearch applications that you own on both ends

#

they're not designed to interface with someone elses APIs

#

because i have to have mapping between your objects and my code objects

#

i did have one question though

#

when i tried using elasticsearch-net it was sending requests to /index/objectname/_search

#

instead of /index/_search

#

any idea why?

#

i was calling client.Search<ObjectName>("index")

#

i think in your case, "ObjectName" should be "item"

#

if i recall from the mapping request

desert needle Jun 8, 2016, 6:07 PM

#

ah, so

#

the way I have things set up with an API gateway is kinda weird

#

you don't actually have access to the index and type parameters

#

by sending a request to the /index/_search URL, the backend is automatically selecting the internal index (poe) and type (item)

#

some elasticsearch clients require a bit of fudging to work

#

if they require you to specify the index and type, they won't work

#

sometimes you can fudge it by specifying an index of index and not specifying type

#

(because it just rewrites the URL)

#

so in your case, it sounds like giving it an index URL of api.exiletools.com and an index setting of index should make it rewrite everything to api.exiletools.com/index

wind garden Jun 8, 2016, 7:28 PM

#

Ya I don't think it will work

#

Because it inserts type before search

desert needle Jun 8, 2016, 7:31 PM

#

type should not be required by a client

#

you could try leaving the index blank and setting type to index heh

wind garden Jun 8, 2016, 7:31 PM

#

What happens if I send a request to /poe/item/_search?

desert needle Jun 8, 2016, 7:31 PM

#

you'll get a 404 'cuz that's not a valid url

wind garden Jun 8, 2016, 7:31 PM

#

K

#

The other issue is type mapping

#

Your json doc is huge

desert needle Jun 8, 2016, 7:32 PM

#

api.exiletools.com/index internally gets forwarded to an NGINX reverse proxy server which makes some header modifications and forwards it to an API gateway proxy, which converts /index to /poe/item/ and forwards to the elasticsearch server

#

oh, the docs are a better starting point for the mapping stuff

#

http://api.exiletools.com/info/indexer-docs.html

#

there are some links in there to some scripts that allow you to look at parts of the mapping too

#

but that is a better starting point for understanding the overall scheme (and basic queries and look at the returned documents)

wind garden Jun 8, 2016, 7:34 PM

#

I'm talking about serialization classes for the json

#

Maybe mapping was the wrong word

desert needle Jun 8, 2016, 7:36 PM

#

the documents aren't that big, they're smaller than the original documents iirc

#

but the elasticsearch type mapping is indeed huge

#

has to be, since every mod needs its own type/etc.

#

even so I use editors that load the type mappings and it's pretty seamless

wind garden Jun 8, 2016, 7:40 PM

#

Is that all in the repo?

#

I wonder if I could easily port it to c#

#

I'm not even sure if I need to do all this

#

I guess I'm just a little unsure about how to make my own queries

desert needle Jun 8, 2016, 7:41 PM

#

https://gist.github.com/trackpete start from there 😃 (ignore the skilltree ones)

#

you shouldn't need to load the type mappings at all, that's generally only done for fancier tools (I use Elastic Sense)

#

but tbh once you start writing some queries the mapping becomes easy, it's mostly intuitive

#

so like if I want to search for jewelry shit, I just look at the docs above then go "oh, ok, attributes.baseItemType : Jewelry"

#

simple boolean matches are very very easy

#

more complex aggregations require spending some time familiarizing yourself with the official docs (which are very very good)

wind garden Jun 8, 2016, 7:50 PM

#

How do you determine trade volume?

#

Man I wish poe.trade made "fixed price" more obvious

#

So many people making stupid offers on fixed price items

#

They offer 60c for an item listed fixed at 95c

#

Not eve close

desert needle Jun 8, 2016, 7:52 PM

#

you can use aggregation searches to determine volume

#

for example, if you want to see items added by hour, you would use a histogram aggregation against the shop.added field with filters against the items you want (league, rarity, etc.)

wind garden Jun 8, 2016, 7:54 PM

#

Hmm k

#

What units do you use for price?

#

I saw things listed at like 400

desert needle Jun 8, 2016, 7:54 PM

#

you can see an example of that in the code for the dashboard http://exiletools.com/dashboard (Though I need to update this for the new idnex)

#

shop.amount and shop.currency contains the original price information

#

shop.chaosEquiv contains a fixed rate conversion to chaos equivalent values at ingest time

#

chaosEquiv is intended for sorting and general searching purposes, not as a pure exact price point

#

(since rates obviously fluctuate)

#

http://exiletools.com/rates that has the default rates

wind garden Jun 8, 2016, 8:01 PM

#

There isn't a way to pull latest rates is there?

#

I have to manually set it every morning in my app

desert needle Jun 8, 2016, 8:01 PM

#

I don't even bother to try, for one thing it would require re-indexing every document, but mostly it's that there's no such thing as a rate people actually agree on

#

but it doesn't really matter since I don't remove the actual price data

#

just means you might have to do some hoops if you want to find items worth 1ex or more but priced in any currency and you disagree with the rates, you just have to build your own conversions

wind garden Jun 8, 2016, 8:02 PM

#

I just only look at chaos and ex and I set a ratio

#

Don't bother with smaller conversations

#

Conversions

desert needle Jun 8, 2016, 8:04 PM

#

you can do some more complex stuff on your own if you want using this feature https://github.com/trackpete/exiletools-indexer/issues/109 but 99.9% of the use cases will be solved by filtering/sorting/etc. on chaosEquiv then extracting the original price data

wind garden Jun 8, 2016, 8:14 PM

#

k

chrome topaz Jun 8, 2016, 8:16 PM

#

@wind garden how more obvious can it get?

wind garden Jun 8, 2016, 8:16 PM

#

heh, idk. maybe people are just stupid and don't care

chrome topaz Jun 8, 2016, 8:17 PM

#

probably

wind garden Jun 8, 2016, 8:17 PM

#

i just know 90% of the offers i get on valuable items are 50-60% of my asking price

#

and i have them marked as "fixed price"

#

@desert needle what does the 50% price refer to? Is that the average price of the bottom 50% of prices?

desert needle Jun 8, 2016, 8:28 PM

#

in that query I gave you, it's a percentile calculation

#

the 50th percentile means 50% of all items fall at or below that price

#

as opposed to an average which will get skewed by outliers

#

I usually use 10-15th percentile to determine an actual value

wind garden Jun 8, 2016, 8:29 PM

#

k

#

can you do multiple percents by changing the "percents": [50] line?

desert needle Jun 8, 2016, 8:30 PM

#

correct

wind garden Jun 8, 2016, 8:31 PM

#

[10][20][30]?

#

or is there a delimiter

desert needle Jun 8, 2016, 8:31 PM

#

it's an array, see https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-percentile-aggregation.html

wind garden Jun 8, 2016, 8:32 PM

#

k thx

wind garden Jun 8, 2016, 9:16 PM

#

@desert needle what do you use for interval with the histogram? The shop.added is repoted in datetime format

#

here's the last query i tried. It works without the histogram aggs

#

http://pastebin.com/ux06CiJ6

#

nevermind!

#

i found date_histogram

wind garden Jun 9, 2016, 12:26 AM

#

@desert needle Done!

#

https://github.com/Retik/WhatToFlip/releases/tag/1.0

#

at least version 1.0 heh

#

it's prety basic but i think it does what i wanted

#

hmm, major oversight i just realized

#

i'm using a 6h histogram bucket

#

but i should really do 24h to ensure that I get everything worldwide

#

there we go

#

https://github.com/Retik/WhatToFlip/releases/tag/1.1

desert needle Jun 9, 2016, 1:48 PM

#

sounds cool 😮

wind garden Jun 9, 2016, 3:24 PM

#

@desert needle need some help with a query. MinPrice is super inaccurate because of how far off your ratio for chaos:exalt is. Can I aggregate based on a conditional statement like this?

shop.currency == "exa" ? shop.amount * 57 : shop.amount

desert needle Jun 9, 2016, 3:27 PM

#

you can do something like that (I think I linked some fields to use yesterday), but it's a little overkill. The important thing isn't that the ratio for chaosEquiv be accurate, it's that it's fixed. People very very rarely price items at say 150 pure chaos, so if you know the ratio is 40:1, but it's fixed internally at 80:1, you can still compare something priced at 20 chaosEquiv and 80 chaosEquiv by understanding that one is real and one is converted

#

you can't actually do scripting (value * x) though because I have that disabled

#

anyway the trick is not to use chaosEquiv for DISPLAY other than informationally

#

use it for sorting and analysis, then convert

#

https://github.com/trackpete/exiletools-indexer/issues/109 you can do multi currency searches like this, but not sure how that will help with aggregations

#

if you really really want to track individual currency types you need to do statistics on each currency type then merge them in with your own rates

wind garden Jun 9, 2016, 3:30 PM

#

i just wanna get the cheapest for each item at the current time

#

but it's not possible because it doesn't know what the cheapest is

#

one item could be posted for 130c and another for 2ex and it would think the 130c was cheaper

#

cause in its mind 2ex = 160c

desert needle Jun 9, 2016, 3:32 PM

#

yeah you'd have to get the cheapest for items priced in chaos and items priced in exalts then compare them

#

elasticsearch 5 is supposed to have a completely rebuilt scripting engine that is a lot more secure and less overhead, might be able to enable that kind of live analysis query then

wind garden Jun 9, 2016, 4:14 PM

#

so ggg doesn't send currency and amount correct?

#

they just send the note

desert needle Jun 9, 2016, 4:14 PM

#

yep

wind garden Jun 9, 2016, 4:14 PM

#

those are properties that you parse?

desert needle Jun 9, 2016, 4:15 PM

#

the note is processed for buyout tags

wind garden Jun 9, 2016, 4:15 PM

#

what do you do if their note is something like ~b/o 1 exa 24chaos

desert needle Jun 9, 2016, 4:15 PM

#

or the stash name

#

that's not supported, it becomes 1ex

wind garden Jun 9, 2016, 4:15 PM

#

k

desert needle Jun 9, 2016, 4:15 PM

#

people do that sometimes because they're idiots, but poe.trade doesn't support it either

#

they have to do 1.25ex /etc.

#

and lots of people fuck up currency anyway 'cuz they're idiots

wind garden Jun 9, 2016, 4:27 PM

#

yup

#

my app just found a guy selling The Brittle Emperor for 1.7 chaos

#

he's all oh uhhh wrong price

desert needle Jun 9, 2016, 4:28 PM

#

lol

mortal bone Jun 9, 2016, 4:28 PM

#

Lol

desert needle Jun 9, 2016, 4:28 PM

#

probably meant to select ex

wind garden Jun 9, 2016, 4:28 PM

#

yep

#

happens a bunch

desert needle Jun 9, 2016, 4:28 PM

#

apparently there is a recent surge of items being priced in alterations

wind garden Jun 9, 2016, 4:28 PM

#

another thing ggg could improve on

desert needle Jun 9, 2016, 4:28 PM

#

as that's the first one in the UI

loud umbra Jun 9, 2016, 5:08 PM

#

@desert needle I know you've done ladder stats before. Was curious if that information is in a publicly accessible updated database for the current league? Or do you just run a script when you want to get data of ladder characters for that?

desert needle Jun 9, 2016, 5:08 PM

#

oh, god 😃

#

snapshot stats are available by pulling the whole ladder

#

there is some really crappy random program I wrote 2+ years ago that may show current snapshot stats

loud umbra Jun 9, 2016, 5:09 PM

#

Was mostly curious about items and skill trees.

#

Ah.

desert needle Jun 9, 2016, 5:09 PM

#

ah, yeah, the item and skilltree stuff keeps going in and out of my radar

#

https://github.com/trackpete/exiletools-indexer/issues I hven't really done anything with them since February

#

the last couple of times I shared some results the response was very lukewarm, so it just didn't seem worth investing more effort into

loud umbra Jun 9, 2016, 5:11 PM

#

Ahhh OK. I think you were the one who used to post them on Reddit, right?

#

My memory is terrible.

desert needle Jun 9, 2016, 5:12 PM

#

ya

#

http://exiletools.com/blog/2016/03/22/report-ascendancy-and-skilltree-analysis-in-perandus/ was the most recent one

#

I haven't done an item analysis in a lot longer

loud umbra Jun 9, 2016, 5:13 PM

#

Yep that's exactly the one I was thinking of.

desert needle Jun 9, 2016, 5:13 PM

#

that item analysis one was super complex then GGG made everything private

loud umbra Jun 9, 2016, 5:14 PM

#

Forgot about that. Not even sure if there's enough data then depending on how many people manually enabled it again...

desert needle Jun 9, 2016, 5:16 PM

#

yeah, I got a fair amount of data from accounts that are public and skimmed the names from the stash tab api, but it's still a very small sample

#

and doing that kind of analysis doesn't really yield much beyond "interesting" - I can't come up with any way to actually use it to inform meaningfully

#

not much value in knowing that "shield charge is the meta" etc.

desert needle Jun 9, 2016, 5:48 PM

#

anyone in game at the moment that has a Mortal fragment that can copy/paste the clipboard text from it for me?

wind garden Jun 9, 2016, 5:48 PM

#

not mortal frag

desert needle Jun 9, 2016, 5:48 PM

#

and/or one of them fancy new keys?

loud umbra Jun 9, 2016, 5:48 PM

#

Sorry I'm not home atm :<

wind garden Jun 9, 2016, 5:48 PM

#

woh

#

my discord was behind a bunch

loud umbra Jun 9, 2016, 5:48 PM

#

Won't be home for 6 hours

wind garden Jun 9, 2016, 5:49 PM

#

i can try and buy one

desert needle Jun 9, 2016, 5:49 PM

#

I can try global chat in game I guess

wind garden Jun 9, 2016, 5:49 PM

#

try messaging Darkee

#

he's probably doing uber by now

desert needle Jun 9, 2016, 5:49 PM

#

need to make sure my parsing code works on 'em

wind garden Jun 9, 2016, 5:50 PM

#

this for general usage or something specific?

desert needle Jun 9, 2016, 5:50 PM

#

working on a new backend for my price macro

wind garden Jun 9, 2016, 5:51 PM

#

nice

#

see i reeeeally wish xyz had apis for his site

#

i could pull stat data for currency

#

and use it to get up-to-date ratios for exchange rates

#

do you index currency?

desert needle Jun 9, 2016, 5:52 PM

#

in theory that can be done with the public stash currency data in the exiletools index

wind garden Jun 9, 2016, 5:52 PM

#

ya was just thinking that

desert needle Jun 9, 2016, 5:52 PM

#

I think poe.trade supports posting currency trades in other ways (haven't looked at it)but the stash data should have some

wind garden Jun 9, 2016, 5:53 PM

#

it should have enough from stash api that it's accurate

desert needle Jun 9, 2016, 5:53 PM

#

baseItemType:currency and price will be there in the exiletools index

wind garden Jun 9, 2016, 5:53 PM

#

within 1c i'd imagine

#

darn

#

just missed a really nice roll on a cheap carcass jack

desert needle Jun 9, 2016, 5:57 PM

#

"query": {
  "bool": {
    "must": [
      { "term": {
        "attributes.league": {
          "value": "Prophecy"
        }
      }},
      { "term": {
        "shop.hasPrice": {
          "value": "true"
        }
      }},
      { "term": {
        "attributes.baseItemType": {
          "value": "Currency"
        }
      }}
    ]
  }
},
"aggs": {
  "currencyName": {
    "terms": {
      "field": "info.fullName",
      "size": 100
    },
    "aggs": {
      "percentile": {
        "percentiles": {
          "field": "shop.chaosEquiv",
          "percents": [
            1,
            5,
            25,
            50          ]
        }

      }
    }
  }
},
size:0
}```

#

that's very rudimentary

#

but has some value as a starting point

#

          "doc_count": 533,
          "percentile": {
            "values": {
              "1.0": 53,
              "5.0": 56,
              "25.0": 58,
              "50.0": 58
            }
          }
        },```

#

that look right?

#

          "key": "Divine Orb",
          "doc_count": 296,
          "percentile": {
            "values": {
              "1.0": 5.95,
              "5.0": 7,
              "25.0": 8,
              "50.0": 8.5
            }
          }
        },```

#

etc.

#

@wind garden are you using the RT Stash API? 😉

#

should give you an edge

wind garden Jun 9, 2016, 6:04 PM

#

ya i am

#

in what way were you thinking it would give me an edge?

#

i've been manually entering conversion rate every morning

#

this will just save me some meffort

#

what would be sick is if i could create an automated trading platform 😛

desert needle Jun 9, 2016, 6:05 PM

#

oh, I meant the exiletools realtime stash api - but I was thinking you were doing live searches, I just remembered you're consuming the stash api yourself

wind garden Jun 9, 2016, 6:05 PM

#

ya

#

i tried creating a trade bot earlier but poe has some sort of weird overlay over the stash area

#

makes items show up differently depending on their location in the grid

#

probably intentional

desert needle Jun 9, 2016, 6:06 PM

#

that and it's against the TOS and GGG will ban you if they figure it out ;x

wind garden Jun 9, 2016, 6:06 PM

#

i know 😃

desert needle Jun 9, 2016, 6:06 PM

#

I've seen what appear to be some functional trading bots

wind garden Jun 9, 2016, 6:06 PM

#

oh ya

#

they message me all the time

#

it's stupid

#

offer horribly low prices on multiple instantly

#

multiple items

#

i got messaged for two different items at the same time within a second

#

from the same person

desert needle Jun 9, 2016, 6:07 PM

#

lol

wind garden Jun 9, 2016, 6:08 PM

#

but that's a far cry from being able to automate the execution of a complete trade

desert needle Jun 9, 2016, 6:08 PM

#

yeah

#

the bots I've seen people talk about tend to be pretty obvious too

#

please put the item in the first grid in the trade window etc.

wind garden Jun 9, 2016, 6:09 PM

#

omg

#

"plz ban me, i'm making it so easy"

desert needle Jun 9, 2016, 6:10 PM

#

seriously

loud umbra Jun 9, 2016, 6:10 PM

#

Never encountered one before. That's pretty funny. :P

wind garden Jun 9, 2016, 6:11 PM

#

right now my app lets me set prices as a threshold before they get displayed

#

but i have to update prices constantly as the market changes

#

well, daily

#

if i can combine the data i get from your APIs, i can automate that too

#

can get like 5% price, make it real tight spread

#

and then set a threshold for profit margin

loud umbra Jun 9, 2016, 6:12 PM

#

How has it been going so far. Do you find it effective?

wind garden Jun 9, 2016, 6:13 PM

#

i've made 5ex since sunday

#

found 2 ex, at 7ex now

loud umbra Jun 9, 2016, 6:13 PM

#

Very nice

wind garden Jun 9, 2016, 6:13 PM

#

not bad for very little effort

#

the nice thing is i can look for big ticket items and keep playing

#

as opposed to doing currency trading, which is super time intensive

#

i just end up sitting in my hideout trading jewelers for 1c profit a trade

#

tedious

#

i can make a lot but i don't actually get to play the game 😛

loud umbra Jun 9, 2016, 6:14 PM

#

Yeah I didn't play much last league due to currency trading

wind garden Jun 9, 2016, 6:14 PM

#

this just dings at me when there's a good deal on an item

loud umbra Jun 9, 2016, 6:14 PM

#

Pretty much took up all time.

wind garden Jun 9, 2016, 6:14 PM

#

right click it and it copies the message to clipboard like poe.trade

loud umbra Jun 9, 2016, 6:15 PM

#

Setting it to look for specific items you specify?

desert needle Jun 9, 2016, 6:15 PM

#

yep, that's the way to go

wind garden Jun 9, 2016, 6:15 PM

#

yes

#

i have a list of about 10 items

desert needle Jun 9, 2016, 6:15 PM

#

plus part of the bonus by doing the processing yourself is that you can customize the message to not look like a bot

wind garden Jun 9, 2016, 6:15 PM

#

i can enable and disable them through right click menu

#

well

#

i customize the message to look exactly like poe.trade

desert needle Jun 9, 2016, 6:16 PM

#

but if you're only doing 10-20 items and want to avoid indexing the entire stash tab stream, you can look into using my api for that 😃

wind garden Jun 9, 2016, 6:16 PM

#

since that's what people expect

#

well for this i don't need to keep history

desert needle Jun 9, 2016, 6:16 PM

#

https://www.reddit.com/r/pathofexiledev/comments/4kv3mt/real_time_stash_api_filtered_web_socket_based/

wind garden Jun 9, 2016, 6:16 PM

#

for statistics though i need history

#

which is what i'm using your index for

desert needle Jun 9, 2016, 6:17 PM

#

it basically sends out matches even before they are indexed

#

also notifies you if items are removed/etc. as well

#

so if you're slow you can get a notification that the item is gone before you waste time

wind garden Jun 9, 2016, 6:18 PM

#

ooo nice

#

i do keep a local log as i'm running so i don't get duplicates

#

that way if they update their stash multiple times i dont' get spammed

desert needle Jun 9, 2016, 6:19 PM

#

nod

#

that api I linked only updates on modifications to save on spam

#

so if they change the price from 1c to 2c you'll get an update, or if they remove it, but if they just update something else in the stash you won't get anything

#

someone was using it earlier in the week to wreck the market on vagan daggers

#

I noticed it when I posted a vagan dagger and they immediately messaged me with my message format

#

then like 20s later I started getting spammed by poe.trade stuff

#

I was like "oh, it works!"

#

then I checked the active filters and sure enough someone was monitoring vagan daggers with it, felt cool

loud umbra Jun 9, 2016, 6:21 PM

#

So what you're saying is we should use yours instead right? :P

desert needle Jun 9, 2016, 6:22 PM

#

lol, well, if only a few people use mine then I don't have to spin up more server threads

#

but in theory it will always be as close to realtime as you can reasonably get

#

without having to process the stream yourself

#

typical time is <2s from stream update to receiving the socket emit

#

downside of course is that all my shit is for people who want to mess around, I don't bother making good ui's :p

#

though I've considering making an AHK macro to do that, complete with auto copying the messages in

#

otoh this whole HFT culture of getting 1000 messages is super annoying

wind garden Jun 9, 2016, 6:27 PM

#

dang

#

i should create a search for vagan daggers

#

what do they sell for?

#

and how often do you pull the stash tab api data?

#

GGG support told me 45 times per minute

#

so i put my loop at 1.5s intervals

desert needle Jun 9, 2016, 6:29 PM

#

I make a new stash tab api request 1s after finishing the previous request. Processing is async, so it depends on GGG's server response time. It's typically about every 2s

#

Thu Jun  9 14:29:25 2016 . [8338748-8965491-8235824-9813852-9119154] Downloaded 0.13MB of compressed JSON (0.65MB raw) in 0.832616 seconds (158.39KB/s)
Thu Jun  9 14:29:25 2016 . [8338748-8965491-8235824-9813852-9119154] Added 17 stashes to Kafka and indexed stats for 1465496964fkpmk to Elasticsearch in 0.049736 seconds
Thu Jun  9 14:29:26 2016 $ Full update consumed in 1.892655 seconds
Thu Jun  9 14:29:26 2016 ! Processing API URL: http://www.pathofexile.com/api/public-stash-tabs?id=8338748-8965495-8235827-9813855-9119161
Thu Jun  9 14:29:27 2016 . [8338748-8965495-8235827-9813855-9119161] Downloaded 0.11MB of compressed JSON (0.48MB raw) in 0.676714 seconds (162.81KB/s)
Thu Jun  9 14:29:27 2016 . [8338748-8965495-8235827-9813855-9119161] Added 16 stashes to Kafka and indexed stats for 1465496966tytze to Elasticsearch in 0.039073 seconds
Thu Jun  9 14:29:28 2016 $ Full update consumed in 1.724227 seconds```

wind garden Jun 9, 2016, 6:30 PM

#

they acctually blocked my IP when i had a bug that was causing me to poll super fast

#

they unblocked me once i fixed the bug and said it wouldn't happen again 😃

desert needle Jun 9, 2016, 6:30 PM

#

lol

#

you must've done something crazy, normally they rate limit via nginx and just send you a back off response and block for ~5min

wind garden Jun 9, 2016, 6:30 PM

#

it was in a catch loop

#

the catch wasn't sleeping at all

#

so it looped again and failed

#

then the server started throttling me

#

so it loops even faster

#

basically there was no sleep on failure

desert needle Jun 9, 2016, 6:31 PM

#

(re: vagan daggers I guess they were super super popular at start of league especially, I saw three listed for 15c and one for 15alch so I listed one for 8c just for shits and had over 50 messages by the time I finished the trade, if I was greedier I would've forced all those people up in price)

#

ah yeah

#

I have a few back off protocols for things like empty updates, 500's, time outs, etc. as well

wind garden Jun 9, 2016, 6:32 PM

#

ya

#

i added a 10sec sleep on timeouts

desert needle Jun 9, 2016, 6:32 PM

#

mostly because I hated looking at logs full of 500's every 1s

wind garden Jun 9, 2016, 6:32 PM

#

i also time how long it took to pull the last request

#

so if it took 700ms i sleep for 1500-700

desert needle Jun 9, 2016, 6:32 PM

#

ah, interesting

wind garden Jun 9, 2016, 6:33 PM

#

so it's pretty consistent interval

desert needle Jun 9, 2016, 6:33 PM

#

when it gets later in the league I will probably drop the interval

#

at the end of Perandus I was down to like once every 10s

wind garden Jun 9, 2016, 6:33 PM

#

ya not much data

desert needle Jun 9, 2016, 6:33 PM

#

because sometimes 30+s would go by without an update, just not worth it

wind garden Jun 9, 2016, 6:36 PM

#

can you explain a bit how "term" works in the queries

#

i read the doc but don't really understand it

#

term or terms

#

i guess i understand it int he context of the bool match

#

but in the aggs

#

what is this part for

"aggs": {
        "uniqueNames": {
            "terms": {
                "field": "info.fullName",
                "size": 100,
                "order": {
                    "minPrice": "desc"
                }
            },

desert needle Jun 9, 2016, 6:38 PM

#

term is the specific type of aggregation query

#

it basically says "show every different term that matches this field" where "term" is the value of the field

#

so in that case, it says "list every different item name"

#

(with a return set limit of 100"

#

if you changed that field it would show the unique listed values of whatever field you chose

wind garden Jun 9, 2016, 6:41 PM

#

ah i think i get it

desert needle Jun 9, 2016, 6:42 PM

#

in SQL terms it is similar to DISTINCT

#

i.e. SELECT DISTINCT FROM X

wind garden Jun 9, 2016, 6:42 PM

#

what do you use for currency names?

#

chaos and exa?

#

i tried this and got 0 results

desert needle Jun 9, 2016, 6:42 PM

#

I don't modify those, so it's whatever is in the data

wind garden Jun 9, 2016, 6:42 PM

#

{
    "query": {
        "bool": {
            "must": [{
                "term": {
                    "attributes.league": {
                        "value": "Prophecy"
                    }
                }
            },
            {
                "term": {
                    "shop.hasPrice": {
                        "value": "true"
                    }
                }
            },
            {
                "term": {
                    "shop.currency": {
                        "value": "exa"
                    }
                }
            }],
            "must_not": [{
                "term": {
                    "attributes.rarity": {
                        "value": "Magic"
                    }
                }
            },
            {
                "term": {
                    "attributes.rarity": {
                        "value": "Rare"
                    }
                }
            },
            {
                "term": {
                    "attributes.rarity": {
                        "value": "Currency"
                    }
                }
            },
            {
                "term": {
                    "attributes.rarity": {
                        "value": "Gem"
                    }
                }
            }]
        }
    },
    "aggs": {
        "uniqueNames": {
            "terms": {
                "field": "info.fullName",
                "size": 100,
                "order": {
                    "minPrice": "desc"
                }
            },
            "aggs": {
                "minPrice": {
                    "min": {
                        "field": "shop.chaosEquiv"
                    }
                }
            }
        }
    },
    "size": 0
}

desert needle Jun 9, 2016, 6:43 PM

#

oooo

#

in shop.currency I do a lookup conversion to normalized names

#

you can see this by doing an agg term query on that field 😉

#

"aggs": {
  "NAME": {
    "terms": {
      "field": "shop.currency",
      "size": 100
    }
  }
},
size:0
}```

#

will show you all the possible currency names

wind garden Jun 9, 2016, 6:44 PM

#

i must be doing something wrong

#

i did exactly that

#

got 0 results

#

oh wait

#

i still have my stupid bool filter

desert needle Jun 9, 2016, 6:45 PM

#

you're looking for this: {
"key": "Exalted Orb",
"doc_count": 240871
},

wind garden Jun 9, 2016, 6:45 PM

#

i got it now

#

so in reference to earlier conversation. Do you have scripting disabled entirely?

desert needle Jun 9, 2016, 6:46 PM

#

yes

wind garden Jun 9, 2016, 6:46 PM

#

for security i'm guessing

desert needle Jun 9, 2016, 6:46 PM

#

yeah basically they don't really have any breakers on it currently, so someone could submit a bunch of requests to like multiple every number by every number a million times and crash the index

wind garden Jun 9, 2016, 6:46 PM

#

ah

#

k something confusing

#

when i search for items with currency as Exalted Orb

#

sorting by minPrice of shop.chaosEquiv

#

i get REALLY big numbers for cheap items

#

      }, {
        "key" : "Wurm's Molt Leather Belt",
        "doc_count" : 1,
        "minPrice" : {
          "value" : 8.0E8
        }
      }, {
        "key" : "A Mother's Parting Gift",
        "doc_count" : 1,
        "minPrice" : {
          "value" : 79920.0
        }
      }, {
        "key" : "Driftwood Wand",
        "doc_count" : 1,
        "minPrice" : {
          "value" : 79920.0
        }
      }, {
        "key" : "The Scholar",
        "doc_count" : 1,
        "minPrice" : {
          "value" : 79920.0
        }
      }, {
        "key" : "Leather Belt",
        "doc_count" : 1,
        "minPrice" : {
          "value" : 16000.0
        }

#

ya i checked the shop.amount it's 1/80th that number

#

so it's multiplying the ratio, but the shop.amount seems way to high

desert needle Jun 9, 2016, 6:54 PM

#

check where you're getting minPrice from?

#

that is a temporary alias in the original query, maybe it's giving sketchy results

wind garden Jun 9, 2016, 6:55 PM

#

    "aggs": {
        "uniqueNames": {
            "terms": {
                "field": "info.fullName",
                "size": 100,
                "order": {
                    "minPrice": "desc"
                }
            },
            "aggs": {
                "minPrice": {
                    "min": {
                        "field": "shop.chaosEquiv"
                    }
                },
                "minAmount": {
                    "min": {
                        "field": "shop.amount"
                    }
                }
            }
        }
    },

#

it seems to work when i filter for Chaos

#

but not for exalt

#

ohhhh

#

i know what's happening

#

these are items that aren't usually priced in exalts

#

and when they are they're stupid people pricing them for HUGE numbers

#

like who pays an exalt for wurm's molt

merry oracle Jun 9, 2016, 6:57 PM

#

Could have +1 charge implicit to be fair

wind garden Jun 9, 2016, 6:57 PM

#

true

#

but 80000c?

#

that's 999ex

desert needle Jun 9, 2016, 6:57 PM

#

that kind of shit happens all the time, yeah, it's why I always try to use percentile bases in the 5-10 range

wind garden Jun 9, 2016, 6:57 PM

#

ya

#

dang so how do i filter that

#

what i want to do is the original query

#

and take the names from that query and query their chaos and exa min prices

desert needle Jun 9, 2016, 6:59 PM

#

you just have to run a couple serial queries

#

sometimes it's a lot easier to run a few serial queries than trying to combine everything 😃

wind garden Jun 9, 2016, 6:59 PM

#

ya

#

you have an example?

desert needle Jun 9, 2016, 6:59 PM

#

if you're super bored, sure, take a look at the unique item price reports

#

https://github.com/trackpete/exiletools-indexer/blob/master/www/html/js/es-uniques.js

#

oh, it looks like those are all parallel

#

guess I don't have any public serial code, but basically you just run query 1, parse the results, then execute on that

#

you can also actually return data in the query

#

so, say you wanted to sort data by chaosEquiv but actually pull prices from the result

wind garden Jun 9, 2016, 7:02 PM

#

is that what they call pipeline queries?

desert needle Jun 9, 2016, 7:02 PM

#

just set the size:1000 or something to return 1000 results

#

then you get the aggregation data and the individual items

#

then you iterate through the items to match out their specific information

#

pipeline aggs are a little different, I haven't used them much tbh, they're still fancy and new

#

[2:58 PM] Retik: and take the names from that query and  query their chaos and exa min prices``` for this, though, I'd just literally loop through the results of the first query

#

though I'm sure you can find a way to do it with sub aggregations if you try hard enough

wind garden Jun 9, 2016, 7:05 PM

#

you mean do a new query for each of the 100 results of the first query?

#

isn't that intensive on your end?

desert needle Jun 9, 2016, 7:05 PM

#

https://www.elastic.co/guide/en/elasticsearch/guide/current/_filter_bucket.html

#

actually I think that's what you're looking for

wind garden Jun 9, 2016, 7:07 PM

#

does filter exclude or include?

#

looks like it includes

desert needle Jun 9, 2016, 7:13 PM

#

well here's a crazy example with nested aggs

#

📎 nestedaggs.txt

#

this will start with a boolean to filter only unique items in prophecy with prices

wind garden Jun 9, 2016, 7:14 PM

#

hmm

desert needle Jun 9, 2016, 7:14 PM

#

then it creates a term aggregation by distinct item name (limiting to 10 results for simplicity)

wind garden Jun 9, 2016, 7:14 PM

#

let me try

desert needle Jun 9, 2016, 7:14 PM

#

then for each of those items, it will do a term aggregation of all the currency types that item is tied to

#

then for each of the currency types, it will do percentile prices of the value for that currency type tied to that item

wind garden Jun 9, 2016, 7:15 PM

#

hmm

desert needle Jun 9, 2016, 7:15 PM

#

you will get some very strange results by filtering the currency though

wind garden Jun 9, 2016, 7:15 PM

#

ya

desert needle Jun 9, 2016, 7:15 PM

#

since the low percentile of Exalted Orbs for an item might be 5 Exalt

#

but it's a 1c item

wind garden Jun 9, 2016, 7:15 PM

#

i get every currency type that way

desert needle Jun 9, 2016, 7:15 PM

#

but 5 Exalt is the lowest price in exalts

wind garden Jun 9, 2016, 7:15 PM

#

i really want to filter with the first query

#

and then do another aggs within that

desert needle Jun 9, 2016, 7:16 PM

#

a way to limit the data is to simply add a boolean range filter to the beginning

#

and say only return results with > 1c prices or something

#

or only return items priced in chaos or exalts

#

the base setup is kinda hard to grasp, but basically the filter determines what documents the first aggregation takes place on

#

it's just limiting the internal document set

#

think of it like slicing out a portion of the index to analyze

wind garden Jun 9, 2016, 7:18 PM

#

seems like this should be simpler

desert needle Jun 9, 2016, 7:18 PM

#

if you want to filter further based on actively analyzed things, you either need to do filter based aggregations or pipeline aggregations or multi queries or just do nested aggregations and manage the data better

#

lol

wind garden Jun 9, 2016, 7:18 PM

#

i just want to take the results of one query and then filter out those results that have shop.currency == something

desert needle Jun 9, 2016, 7:19 PM

#

maybe I'm not tracking

wind garden Jun 9, 2016, 7:19 PM

#

so the query i've been using aggregates the top 100 items sorted by chaosequiv

#

within those 100, i want to filter out those which have shop.currency equal to chaos or exalt

#

so i should have like 80 of one and 20 of the other

desert needle Jun 9, 2016, 7:26 PM

#

having a hard time wrapping my mind around this 😉 Do you mean you want the 100 most expensive items that only have prices in chaos/exalt, or do you mean you want the lowest price in chaos and lowest price in exalt of the 100 most expensive items? (where expensive is based on the lowest known chaosEquiv price)

wind garden Jun 9, 2016, 7:27 PM

#

most expensive 100

#

then within those 100, get just those results who's price is in exalts or chaos

#

and give me their min prices

desert needle Jun 9, 2016, 7:31 PM

#

but you don't want to filter by exalt/chaos? that would simplify things greatly to start

wind garden Jun 9, 2016, 7:31 PM

#

no because i'll get the 100 most expensive in chaos and 100 most expensive in exalt

#

but what i want to get is 100 most expensive in equivChaos

desert needle Jun 9, 2016, 7:32 PM

#

if you just add { "terms": {"shop.currency": [ "Exalted Orb", "Chaos Orb" ] }} to your boolean must filters, you'll only operate on documents that have a price in exalt or chaos

wind garden Jun 9, 2016, 7:33 PM

#

but i don't want both at the same time

desert needle Jun 9, 2016, 7:33 PM

#

yeah, if you apply that during the filters than your aggregation wil match on all documents

#

you'll get stuff like { "key": "Skyforth Sorcerer Boots", "doc_count": 14, "minPrice": { "value": 1440 } },

#

that means the lowest priced Skyforth that has a price in chaos or exalt is set to 1440 chaosequiv (of which there are 14)

wind garden Jun 9, 2016, 7:34 PM

#

ya

#

not what i want though

#

it needs to aggregate based on min chaosequiv

#

but then filter the results of that based on chaos and not exalt

#

i tried something like this

#

"aggs": {
"uniqueNames": {
"terms": {
"field": "info.fullName",
"size": 100,
"order": {
"minPrice": "desc"
}
},
"aggs": {
"term": {
"shop.currency": "Exalted Orb"
}
"minPrice": {
"min": {
"field": "shop.chaosEquiv"
}
}
}
}
},

#

but that doesn't work

#

i want this, but only those who's currency is exalted orb

"aggs": {
        "uniqueNames": {
            "terms": {
                "field": "info.fullName",
                "size": 100,
                "order": {
                    "minPrice": "desc"
                }
            },
            "aggs": {
                "minPrice": {
                    "min": {
                        "field": "shop.chaosEquiv"
                    }
                }
            }
        }
    },

#

this query will give me 100 results

#

100 items more importantly

#

of those 100 items i want to know the min price of just the items that have currency in chaos

desert needle Jun 9, 2016, 7:37 PM

#

oh

wind garden Jun 9, 2016, 7:37 PM

#

the agg needs to be specifically on "the 100 most expensive items in poe"

#

which is why i agg on chaosequiv

#

but for each item there's a bucket of those items listed

#

some are in chaos, some are in exalts

#

i want the min price for the items in chaos

#

and the separate min price for items in exalts

#

so i can convert the latter to chaos and take the min of the two

desert needle Jun 9, 2016, 7:39 PM

#

the problem with that is the chaosequiv rates discussed before, if there are 100 of an item priced at 100 chaos but 2 of those items priced at 2ex, the average equiv price will be 100 chaos - so it will show up as cheaper than a bunch of other stuff priced at 2ex which will have chaosequiv average of 160 chaos

wind garden Jun 9, 2016, 7:39 PM

#

that's ok

desert needle Jun 9, 2016, 7:39 PM

#

you'd probably just be better off doing a query for the highest priced items in pure chaos, then the highest priced items in pure exalt

#

that basically solves all the complexity

wind garden Jun 9, 2016, 7:40 PM

#

chaosequiv gets me in the ballpark though

#

ya those items in exalts can scew it

#

skew**

#

but not by much

#

within the top 100 most expensive items the least expensive is like 5 chaos

#

there's less than 100 items that are, on average, priced higher than 5c

desert needle Jun 9, 2016, 7:41 PM

#

see if you can figure something out using shop.price.Chaos Orb and shop.price.Exalted Orb

#

the only problem is that you might get null results unless you add some cross filtering

wind garden Jun 9, 2016, 7:42 PM

#

what is shop.price.Chaos Orb?

#

or are you just referencing shop.currency whos value is chaos orb

desert needle Jun 9, 2016, 7:44 PM

#

each document has a dynamically set field called shop.price.[currency] set to the shop.amount

#

so if a document has a shop.currency:Chaos Orb and shop.amount:5 if will have a field shop.price.Chaos Orb:5

wind garden Jun 9, 2016, 7:45 PM

#

now sure how that helps me

desert needle Jun 9, 2016, 7:46 PM

#

well, this works:

#

  "uniqueNames": {
    "terms": {
      "field": "info.fullName",
      "size": 100,
      "order": {
        "minPrice": "desc"
      }
    },
    "aggs": {
      "minPrice": {
        "avg": {
          "field": "shop.price.Chaos Orb"
        }
      }
    }
  }
}, ```

#

but min doesn't, because the min value will be null because documents with a price in Chaos Orb don't have that field

#

not sure but this seems to work

#

  "uniqueNames": {
    "terms": {
      "field": "info.fullName",
      "size": 100,
      "order": {
        "minPrice": "desc"
      }
    },
    "aggs": {
      "minPrice": {
        "min": {
          "field": "shop.price.Chaos Orb",
          "missing": 1
        }
      }
    }
  }
}, ```

#

according to that, the lowest price in chaos of the most expensive items in the game are

#

          "key": "Courtyard Map",
          "doc_count": 2,
          "minPrice": {
            "value": 59.99
          }
        },
        {
          "key": "The Vinktar Square Courtyard Map",
          "doc_count": 2,
          "minPrice": {
            "value": 55
          }
        },```

#

not sure if that's quite right

#

change it to do the minprice based on Exalted Orbs though and you get:

#

          "key": "Atziri's Acuity Vaal Gauntlets",
          "doc_count": 6,
          "minPrice": {
            "value": 18
          }
        },
        {
          "key": "Skyforth Sorcerer Boots",
          "doc_count": 14,
          "minPrice": {
            "value": 18
          }
        },```

#

which looks right

wind garden Jun 9, 2016, 7:50 PM

#

but they're different items in the buckets

#

you should be getting < 100 results for each

#

err no

#

it'll be 100

desert needle Jun 9, 2016, 7:51 PM

#

no, you will always get 100 results in this case

wind garden Jun 9, 2016, 7:51 PM

#

but they should be the same items in each case

#

and if you throw out the ones that are null, it'll be less than 100

desert needle Jun 9, 2016, 7:51 PM

#

no, it makes sense - the most expensive items that also have prices in chaos will not be the most expensive items priced in exalts

#

because the uniqueNames aggregation is top level, it will always keep fillling up that result set until it gets 100

wind garden Jun 9, 2016, 7:52 PM

#

hmm

desert needle Jun 9, 2016, 7:52 PM

#

by saying "only show me items with a price in chaos" you are effectively removing many of the top 100 most expensive items from the return set

wind garden Jun 9, 2016, 7:52 PM

#

true

#

but i know there's items priced in chaos more than 50 though

#

this says most expensive item in chaos is courtyard map at 55

desert needle Jun 9, 2016, 7:53 PM

#

that's the minimum of the most expensive

wind garden Jun 9, 2016, 7:53 PM

#

still

#

take cospri's will for isntance

#

it's normally priced at 3ex

#

but i've seen it priced at 180c

#

wouldn't it show up above this item?

desert needle Jun 9, 2016, 7:54 PM

#

yes, that query needs some massaging

#

I'm not sure that missing:1 is the right thing to do

wind garden Jun 9, 2016, 7:54 PM

#

it seems close

desert needle Jun 9, 2016, 7:55 PM

#

if you use an avg you get results that make sense

#

          "key": "Cospri's Will Assassin's Garb",
          "doc_count": 90,
          "minPrice": {
            "value": 182.5
          }
        },
        {
          "key": "Taste of Hate Sapphire Flask",
          "doc_count": 80,
          "minPrice": {
            "value": 129.8
          }
        },
        {
          "key": "Reach of the Council Spine Bow",
          "doc_count": 215,
          "minPrice": {
            "value": 127.85714285714286
          }
        },```

wind garden Jun 9, 2016, 7:55 PM

#

what query gave you that?

desert needle Jun 9, 2016, 7:55 PM

#

the problem is we need a way to get the minimum without selecting null values as the minimum

#

      "minPrice": {
        "avg": {
          "field": "shop.price.Chaos Orb"
          }
      }
    }```

#

that's returning the average of the chaos orb prices

#

The missing parameter defines how documents that are missing a value should be treated. By default they will be ignored but it is also possible to treat them as if they had a value.

#

but it doesn't seem to be working right for me

wind garden Jun 9, 2016, 7:56 PM

#

shouldn't need missing

#

if it ignores them

desert needle Jun 9, 2016, 7:57 PM

#

yeah, but it's not ignoring them :/

#

I dunno, there's something weird going on

#

that agg shouldn't be returning null's

#

with min set

#

at the end of the day, it seems like pipeline aggregations might be the way to go on this

#

my elasticsearchfu is not enough to figure it out right now 😃

merry oracle Jun 9, 2016, 7:58 PM

#

Would this result help?

#

err, how do I paste code here? haven't done that before with discord

wind garden Jun 9, 2016, 7:59 PM

#

3 ` marks above and below

desert needle Jun 9, 2016, 7:59 PM

#

if it's short, use yeah that

wind garden Jun 9, 2016, 7:59 PM

#

if i remove the missing

#

i get some that are null right

desert needle Jun 9, 2016, 7:59 PM

#

that's what I got, yeah

wind garden Jun 9, 2016, 7:59 PM

#

i would expect to see the same items for both chaos orb and exalted orb

#

but i'm not

desert needle Jun 9, 2016, 8:00 PM

#

I wouldn't expect that.

#

High priced items will tend to be priced only in exalts.

#

Items that fluctuate at 2ex or lower tend to be priced in either

wind garden Jun 9, 2016, 8:00 PM

#

ya but i would expect to see those items with null for the chaos agg

#

oh wait

#

it's because i'm ordering based on that

merry oracle Jun 9, 2016, 8:01 PM

#

                    "key": "Taste of Hate Sapphire Flask",
                    "doc_count": 18,
                    "minAmount": {
                        "value": 3.5
                    },
                    "minExalt": {
                        "value": 3.5
                    },
                    "minPrice": {
                        "value": 175
                    },
                    "minChaos": {
                        "value": 175
                    }

                }

wind garden Jun 9, 2016, 8:01 PM

#

it doesn't know how to order with the value null

merry oracle Jun 9, 2016, 8:01 PM

#

Would that kind of result be useful for you?

wind garden Jun 9, 2016, 8:01 PM

#

maybe i can add another

merry oracle Jun 9, 2016, 8:01 PM

#

Obviously exalt/chaos price might be null depending on how things are priced

wind garden Jun 9, 2016, 8:02 PM

#

dang, why can't i agg both

desert needle Jun 9, 2016, 8:02 PM

#

I'm pretty sure pipelines is what you want

wind garden Jun 9, 2016, 8:02 PM

#

why doesn't this work

desert needle Jun 9, 2016, 8:02 PM

#

Since the filter bucket operates like any other bucket, you are free to nest other buckets and metrics inside. All nested components will "inherit" the filter. This allows you to filter selective portions of the aggregation as required.

wind garden Jun 9, 2016, 8:02 PM

#

"aggs": {
        "uniqueNames": {
            "terms": {
                "field": "info.fullName",
                "size": 100,
                "order": {
                    "minPrice": "desc"
                }
            },
            "aggs": {
                "minPrice": {
                    "min": {
                        "field": "shop.chaosEquiv"
                    }
                }
                "minPriceExalted": {
                    "min": {
                        "field": "shop.price.Exalted Orb"
                    }
                }
            }
        }
    },

#

doh

#

no comma

#

thereeeee we go

desert needle Jun 9, 2016, 8:03 PM

#

did that work? I thought you couldn't nest aggs in there becauseof the sorting or some shit 😮

wind garden Jun 9, 2016, 8:04 PM

#

i think it'll work

#

"aggs": {
        "uniqueNames": {
            "terms": {
                "field": "info.fullName",
                "size": 100,
                "order": {
                    "minPrice": "desc"
                }
            },
            "aggs": {
                "minPrice": {
                    "min": {
                        "field": "shop.chaosEquiv"
                    }
                },
                "minPriceExalted": {
                    "min": {
                        "field": "shop.price.Exalted Orb"
                    }
                },
                "minPriceChaos": {
                    "min": {
                        "field": "shop.price.Chaos Orb"
                    }
                }
            }
        }
    },

merry oracle Jun 9, 2016, 8:04 PM

#

That's exactly what I did to produce the response I posted

wind garden Jun 9, 2016, 8:04 PM

#

this is what i'm using

merry oracle Jun 9, 2016, 8:04 PM

#

Should get you the lowest chaos/ex price on the most expensive items

wind garden Jun 9, 2016, 8:04 PM

#

        "key" : "Shavronne's Wrappings Occultist's Vestment",
        "doc_count" : 99,
        "minPrice" : {
          "value" : 400.0
        },
        "minPriceExalted" : {
          "value" : 5.0
        },
        "minPriceChaos" : {
          "value" : null
        }

#

        "key" : "Wasteland Map",
        "doc_count" : 65,
        "minPrice" : {
          "value" : 9.0
        },
        "minPriceExalted" : {
          "value" : null
        },
        "minPriceChaos" : {
          "value" : 9.0
        }

#

i can massage the data on my end to throw out the nulls

merry oracle Jun 9, 2016, 8:05 PM

#

you should be able to then handle proper chaos:ex comparisons after processing the results to get more useful comparisons

desert needle Jun 9, 2016, 8:05 PM

#

sweet

#

I can get back to doing some work and get drunk tonight at happy hour dreaming of never thinking about elasticsearch again ;x

wind garden Jun 9, 2016, 8:06 PM

#

        "key" : "Carcass Jack Varnished Coat",
        "doc_count" : 297,
        "minPrice" : {
          "value" : 115.0
        },
        "minPriceExalted" : {
          "value" : 1.5
        },
        "minPriceChaos" : {
          "value" : 115.0
        }

#

thats exactly what i wanted to handle

#

1.5ex is 85c

#

which is much lower than 115c

merry oracle Jun 9, 2016, 8:07 PM

#

This is my first foray into ES and it's fascinating

#

Very different from good ol SQL

wind garden Jun 9, 2016, 8:07 PM

#

ya, i'm getting better at the queries

#

thanks a bunch for the help @desert needle and @merry oracle

desert needle Jun 9, 2016, 8:09 PM

#

😄

merry oracle Jun 9, 2016, 8:10 PM

#

Sure thing! Gonna have to request a proper API key soon so I don't beat up the dev key

#

@wind garden You'll want to add this to your boolean filter if you care about only items that haven't been sold:

#

{
                "term": { 
                    "shop.verified":{
                        "value": "YES"
                    }
                }
            }

wind garden Jun 9, 2016, 8:21 PM

#

hmm

#

i don't think i do though

#

i'm just looking for statistical prices

#

if it's been sold, even better, because someone actually bought it at that price

merry oracle Jun 9, 2016, 8:21 PM

#

Makes sense then

#

you could of course reverse the filter to only find items that are sold

#

just set the value to GONE

wind garden Jun 9, 2016, 8:26 PM

#

woot!

#

carcass jack 65c

desert needle Jun 9, 2016, 8:26 PM

#

yep, and you can also consider adding some filters for updated recently

#

it's rare, but it does happen that the prices fluctuate massively

wind garden Jun 9, 2016, 8:26 PM

#

i love honest traders that sell it to the first guy who messaged them

#

rather than inviting you to group and then kicking you when they get a better offer

desert needle Jun 9, 2016, 8:26 PM

#

for example, Abyssus was <10c for the first two days of this league, then spiked to 2ex after Mathil's videos

#

same thing happened with Varunastra last league, they were 1c for like two weeks before shooting up

merry oracle Jun 9, 2016, 8:27 PM

#

I'm mostly interested in prices over the last few days. Look at what average b/o for things that are selling is, as well as the highest b/o that's sold recently

#

Would help a great deal with pricing

desert needle Jun 9, 2016, 8:28 PM

#

elasticsearch makes it easy with a range filter of updated:now-2d/d will automatically filter it to items updated in the last two days for example

#

I use three days for most of the price report/filter/macro stuff by default

#

so doing GONE in the last 3 days is also super good

#

either way, time to go to happy hour wuwu

wind garden Jun 9, 2016, 8:29 PM

#

i should use that

#

wait

#

does your indexer include items sold since the start?

#

am i including old data?

#

hehehe

#

Carcass Jack Varnished Coat, MinPrice = 65

#

that's me!

#

@desert needle still here?

#

@merry oracle do you know how you would filter like that using shop.updated:now-2/d/d?

wind garden Jun 9, 2016, 9:09 PM

#

i think i figured it out

#

i'm filtering with this

#

"filter": {
                "range": { 
                    "shop.updated": { 
                        "gte": "now-1d",
                        "lte": "now"
                    }
                }
            }

#

and i added this

{
    "term": {
        "shop.verified": {
            "value": "GONE"
        }
    }
}],

#

so as i understand it, that should include items sold in the last day

merry oracle Jun 9, 2016, 9:30 PM

#

That looks right. Just got home from work, so I can double-check it

#

Rather than adding a filter, you can simply add it to the boolean query under the list of "musts"

#

You can use this json:

#

{
                "range": { 
                    "shop.updated":{
                        "gte":"now-1d",
                        "lte":"now"
                    }
                }
            }

#

so basically the same, but no filter needed

wind garden Jun 9, 2016, 9:43 PM

#

hmm, wonder if i get the same results

#

ya looks the same

desert needle Jun 10, 2016, 1:50 PM

#

and no need to add the lte:now that will just happen

wind garden Jun 10, 2016, 2:56 PM

#

Ya I removed it later

#

I feel like a master of queries now. I did a bunch more reading and think I have a much better understanding of it

#

I figured out how to do what I wanted to originally yesterday by splitting things into buckets by currency type

desert needle Jun 10, 2016, 3:13 PM

#

very nice

#

that's one of the things that has amazed me about elasticsearch, it has some incredible power the deeper you go

wind garden Jun 10, 2016, 3:36 PM

#

i got a problem

#

for some reason Shavronne's wrappings isn't showing up in the list

#

http://pastebin.com/qcV6FJrT

#

that's my query

#

i checked the json results

#

it's not in there

#

are there just none in the last 24hrs with a price in your index?

#

i know it was there previously

desert needle Jun 10, 2016, 3:40 PM

#

46

#

with price updated in the last day

#

must be something with prophecy twitching it

#

shows up if you change it to Standard

#

can't explain it offhand unless the prices in prophecy are just cheap

#

oh

#

it's because you're doing a min

#

and someone posted one for 1c

#

you gotta change that min to percentile 😉

#

otherwise the calculation will get thrown off by asshats

#

                    "min": {
                        "field": "shop.chaosEquiv"
                    }
                },``` that part

#

"defaultMessage": "@VitaIsEarthbending I would like to buy your Shavronne's Wrappings Occultist's Vestment listed for 1 Chaos Orb (League:Prophecy, Stash Tab:\"~b/o 1 chaos\" [x9,y0])", teehee

pseudo ocean Jun 10, 2016, 3:50 PM

#

😄

desert needle Jun 10, 2016, 3:50 PM

#

"verified": "GONE", + "shelfLife": 1456,

wind garden Jun 10, 2016, 3:50 PM

#

got it

desert needle Jun 10, 2016, 3:50 PM

#

wonder if he actually sold it

#

though 24min seems too long

pseudo ocean Jun 10, 2016, 3:50 PM

#

if it were me, i'd get the whisper, say 'sold already', then move it out of the wrong tab i placed it in

desert needle Jun 10, 2016, 3:50 PM

#

probably just got spammed like crazy and realized he had accidentally put it into the wrong stash tab

pseudo ocean Jun 10, 2016, 3:51 PM

#

happens often

#

yeh

desert needle Jun 10, 2016, 3:51 PM

#

looks like he kept it though, didn't re-list it or that record would've been updated

#

man, so much spam for the poor guy

#

"stashName": "~b/o 1 chaos" woops

#

it was a 6L too

#

"allSocketsGGG": "B-R-G-B-B-B",

merry oracle Jun 10, 2016, 3:52 PM

#

lmao

#

He probably just dumped it somewhere in a hurry. Oops

desert needle Jun 10, 2016, 3:52 PM

#

anyway yeah that's why you gotta use percentiles, min will definitely trip on a lot of weird shit like that (never use max or avg for the same reasons)

#

I would love to see that guy's chat log heh

wind garden Jun 10, 2016, 3:56 PM

#

how do i sort by the percents again?

#

"order": {
"avgPrice[5.0]": "desc"
}

#

ike that?

desert needle Jun 10, 2016, 3:59 PM

#

something like that

#

I had to google it last time 😃

wind garden Jun 10, 2016, 4:01 PM

#

Isn't working :/

merry oracle Jun 10, 2016, 4:17 PM

#

            "terms": {
                "field": "info.fullName",
                "size": 100,
                "order": {
                    "avgPrice.5": "desc"
                }
            }

#

That should do it

#

Combination of docs and a bit of guesswork to figure that out

wind garden Jun 10, 2016, 4:40 PM

#

sigh

#

i left some code from my other project in my github repo that checked your ip so it could compare if it was on the same subnet as my indexer service

#

someone sifting through it found it and immediately thought it was doing something maliciouos

#

removed it from the repo, but goes to show how reddit loves to dig up any dirt it can find on you

#

@merry oracle that worked, thanks

pseudo ocean Jun 10, 2016, 4:44 PM

#

what is a good interval to reindex the data from http://www.pathofexile.com/api/public-stash-tabs?

#

without spamming the poe.com servers

wind garden Jun 10, 2016, 4:45 PM

#

@pseudo ocean about 2sec should be fine

#

i had mine at 1sec intervals and GGG support got on my case 😃

pseudo ocean Jun 10, 2016, 4:45 PM

#

haha, seems a bit overkill 1sec

wind garden Jun 10, 2016, 4:45 PM

#

my app searches for good deals on items, so speed is important 😃

pseudo ocean Jun 10, 2016, 4:47 PM

#

does anyone have a rough number of how many items I should have in my database after indexing all stuff?

#

annoying that there isn't a total parameter or so in the response so that I could know what comes

wind garden Jun 10, 2016, 4:48 PM

#

mmm @desert needle would have to tell you that. i don't actually keep a record of history

#

i can tell you the rate at which items are coming in though

#

at least on Prophecy league

pseudo ocean Jun 10, 2016, 4:49 PM

#

that would be nice info too 😛

#

and what do these things mean in the item name? "<set:MS><set:M><set:S>Roth's Reach"

wind garden Jun 10, 2016, 4:54 PM

#

I forget what those are for, someone told me once upon a time

#

i filter them out though, they're not needed

pseudo ocean Jun 10, 2016, 4:55 PM

#

yeah, I just remove them at the moment. No idea what they are meant for

wind garden Jun 10, 2016, 4:55 PM

#

@desert needle any chance I could get you to change your ratio for chaos:exalt to 60 instead of 80? Even on standard league the price is 67 right now. I'm not sure if you use different rates for each league but it hasn't been 80 for a long time.

pseudo ocean Jun 10, 2016, 5:06 PM

#

well 300k items so far, god damn how many items are for sale in this game...

wind garden Jun 10, 2016, 5:12 PM

#

it is pretty crazy

#

on exiletools there are stats with a graph of how many are coming it per day

#

i thought he had a dashboard, but i can't find the link

#

i think he's making changes though, there's a number of broken links

desert needle Jun 10, 2016, 5:15 PM

#

I don't adjust the ratios during the league unless they go wildly off kilter

#

in a couple days it will likely be 80:1

#

then based on previous leagues probably hover around there for the rest of the league

wind garden Jun 10, 2016, 5:15 PM

#

hmm

#

i just can't think of a way to get accurate % bins without scripting

#

or even remotely accurate

pseudo ocean Jun 10, 2016, 5:16 PM

#

i don't know. the value of chaos is REALLY strong atm

wind garden Jun 10, 2016, 5:16 PM

#

like i said, even standard is 67c

#

and it's been around forever

#

i can't see it going up too much higher

desert needle Jun 10, 2016, 5:16 PM

#

brjann - I have 5562558 verified items in my index atm

pseudo ocean Jun 10, 2016, 5:17 PM

#

no one needs fusings, either, with prophs

desert needle Jun 10, 2016, 5:17 PM

#

again, the ratio is for sorting/comparison/etc. not for permanence

#

it doesn't need to be accurate

#

because you have access to the exact data for accuracy ;x

wind garden Jun 10, 2016, 5:17 PM

#

i knowww but how can i get good averages 😃

desert needle Jun 10, 2016, 5:17 PM

#

you just have to extract chaos and exalt separately 😄

wind garden Jun 10, 2016, 5:18 PM

#

hmm

desert needle Jun 10, 2016, 5:18 PM

#

the other thing to keep in mind is that adjusting the ratios fucks with history

wind garden Jun 10, 2016, 5:18 PM

#

ah that makes sense

desert needle Jun 10, 2016, 5:18 PM

#

the entire point of the exiletools index is to have that theory of history

pseudo ocean Jun 10, 2016, 5:18 PM

#

thanks trackpete for thenumber

desert needle Jun 10, 2016, 5:18 PM

#

so today something posted for 1ex and a month from now for 1ex, that has the same chaosEquiv for comparison purposes across that time range

#

though obviously items priced in chaos fluctuate and cause weird data issues

#

I've actually considered making an imaginary fixed number for sorting instead of using chaosequiv

#

but ultimately it's a very complex issue

#

if I only cared about "now" I would definitely reindex the ratios every night or something

#

@pseudo ocean also "<<set:MS>><<set:M>><<set:S>>Roth's Reach" that crap is weird localization junk

#

just strip everything with <<set:.*?>>

#

you will never need it

#

it's for the popup/mouseover on the web site

#

also yeah my dashboard hasn't been pointed at my new index, I really need to add that to my to-do pew pew

wind garden Jun 10, 2016, 5:47 PM

#

shoot

#

this isn't working well

#

i'm splitting chaos and exalt bins for the 1, 2, and 5%

#

but the values can vary wildly

#

take carcass jack for example

#

it's almost always priced in exalts

#

and the few times it's priced in chaos it's really cheap

#

60-70chaos

#

while in exalts it's around 2ex

#

i can't take the min of the two, because saying the 1% bucket is 60c is not at all accurate

#

what i really need is the number in each bin

#

and scale them accordingly

#

hmmm....

#

guess i answered my own question

#

actually that's not right either

#

taking a 1,2, 5% bin of chaos is totally different than in exalts

#

there could be only 5 items in chaos, but 200 in exalts

#

that's why i need to convert exalt to chaos before i do the percent aggs

merry oracle Jun 10, 2016, 6:02 PM

#

And it doesn't look like there's any good way to handle the arithmetic to convert using a custom chaos:ex ratio without using scripting

#

Any answers I can find about doing arithmetic in elastic search all say to use scripting

wind garden Jun 10, 2016, 6:03 PM

#

ya

#

guess we gotta wait until ES 5.x for that

#

trackpete has it disabled for security reasons

#

scripting that is

#

you know what's the worst? trying to get a query to work for a long time and then realizing finally that it's broken for a totally different part of the query that you thought was working

merry oracle Jun 10, 2016, 6:12 PM

#

Haha

#

In a previous job I rewrote a massive (and incredibly slow) SQL query that was used for reporting. The original query took upwards of 40 minutes to complete. After rewrite it took about 20 seconds.

#

But I ran into so many issues where I screwed something up during the rewrite

wind garden Jun 10, 2016, 6:15 PM

#

i can't seem to get this percentiles bucket aggregation to work

desert needle Jun 10, 2016, 6:16 PM

#

haha yes that happens to me all the time

wind garden Jun 10, 2016, 6:16 PM

#

here's my query

#

http://pastebin.com/fMMWhLD7

#

it looks good from what i can tell

desert needle Jun 10, 2016, 6:16 PM

#

I just tear things down to the smallest pieces

wind garden Jun 10, 2016, 6:16 PM

#

it works without the averagePriceExalt at the bottom

desert needle Jun 10, 2016, 6:16 PM

#

are you getting any actual error message out of the api server?

wind garden Jun 10, 2016, 6:16 PM

#

don't think so

#

just bad request 400

desert needle Jun 10, 2016, 6:17 PM

#

ah, yeah, sorry about that, I really need to re-do that infrastructure

#

            "shard": 0,
            "index": "poe20160607",
            "node": "lKR6I0EVR-mSRGB3u-Q9ow",
            "reason": {
               "type": "search_parse_exception",
               "reason": "Missing required field [buckets_path] for aggregation [averagePriceExalt]",
               "line": 128,
               "col": 4
            }
         }```

wind garden Jun 10, 2016, 6:17 PM

#

but i have buckets_path

#

does that mean it can't find the path?

desert needle Jun 10, 2016, 6:17 PM

#

"buckets_paths": "uniqueNames>avgPriceExalt", typo?

#

changing that to a period doesn't fix it anyway tho

wind garden Jun 10, 2016, 6:18 PM

#

this is from the doc

"sum_monthly_sales": {
            "percentiles_bucket": {
                "buckets_paths": "sales_per_month>sales", 
                "percents": [ 25.0, 50.0, 75.0 ] 
            }
        }

desert needle Jun 10, 2016, 6:18 PM

#

oh

#

I've never seen this before so I'm just muckin

merry oracle Jun 10, 2016, 6:18 PM

#

buckets_path vs buckets_paths

#

that's probably the issue.

desert needle Jun 10, 2016, 6:19 PM

#

               "type": "parse_exception",
               "reason": "Parameter [percents] must be an array of doubles, type `ArrayList` provided instead"
            }
         }```

merry oracle Jun 10, 2016, 6:19 PM

#

The error says singular path, the definition is plural

desert needle Jun 10, 2016, 6:19 PM

#

if you change it to bucket_path

merry oracle Jun 10, 2016, 6:19 PM

#

ah

wind garden Jun 10, 2016, 6:19 PM

#

typo in the doc???

#

lol

#

omg it is

desert needle Jun 10, 2016, 6:19 PM

#

oh, this is a pipeline agg, that's why I haven't seen it

wind garden Jun 10, 2016, 6:19 PM

#

still doesn't work though

#

oh percents

#

it works!

#

woot

#

weird, percents metric aggregation didn't need double

desert needle Jun 10, 2016, 6:21 PM

#

hahaha

#

            "percentiles_bucket": {
                "buckets_path": "uniqueNames>avgPriceExalt",
                "percents": [1.0,2.0,3.0,5.0,10.0]
            }
        }```

#

yeah that works for whatever reason

#

or at least the query returns

wind garden Jun 10, 2016, 6:22 PM

#

well

#

for some reason the data it's not right

desert needle Jun 10, 2016, 6:22 PM

#

          "doc_count": 15,
          "avgPriceExalt": {
            "value": 25.333333333333332
          },
          "avgPrice": {
            "values": {
              "1.0": 1313.6,
              "2.0": 1347.1999999999998,
              "3.0": 1380.8000000000002,
              "5.0": 1448,
              "10.0": 1520
            }
          },
          "minPrice": {
            "value": 1280
          },```

wind garden Jun 10, 2016, 6:22 PM

#

i think it's doing something i'm not expecting

desert needle Jun 10, 2016, 6:22 PM

#

you might have to change the other percents arrays to doubles

wind garden Jun 10, 2016, 6:22 PM

#

i did that

#

it's not actually using that in the pipeline though

#

it' susing avgPriceExalt

#

it should be taking the average of the bucket though right?

desert needle Jun 10, 2016, 6:23 PM

#

well, let me know if you run into any further errors that you can't figure out and at least I can throw you the back end error response

#

I can't help with the pipeline stuff as I haven't looked into it at all yet 😃

merry oracle Jun 10, 2016, 6:25 PM

#

It looks like averagePriceExalt is completely separate from the aggregation as it's showing after the aggregate results

wind garden Jun 10, 2016, 6:25 PM

#

oh snap

#

lollll

#

ok so that doesn't do at all what i thought it did

#

it's averaging the price of ALL items

#

i'm using the wrong buckets

desert needle Jun 10, 2016, 6:26 PM

#

ah hah! 😄

wind garden Jun 10, 2016, 6:26 PM

#

well at least i got a pipeline agg working

#

learned something

#

i need a bucket agg that will do percents

desert needle Jun 10, 2016, 6:27 PM

#

f'ing GGG clipboard text for items: Elemental Damage: 11-23 (augmented) thx, so useful

wind garden Jun 10, 2016, 6:27 PM

#

historgram takes an interval

merry oracle Jun 10, 2016, 6:28 PM

#

What's the data you're trying to produce?

desert needle Jun 10, 2016, 6:31 PM

#

Elemental Damage: 29-58 (augmented), 3-5 (augmented), 16-249 (augmented) even better heh

wind garden Jun 10, 2016, 6:32 PM

#

@merry oracle ok so percentiles will give me the values at which those percentiles fall

#

basically an average

#

what i want is to get the actual bucket of those items that are the bottom 1%, 2%, 5%

merry oracle Jun 10, 2016, 6:34 PM

#

Do you want anything in the bottom 5%? the 1, 2 ,5th percentile will be specific items, so you need to specify a range somehow

#

Or at least close

wind garden Jun 10, 2016, 6:35 PM

#

i want to take the avg of those buckets

#

specifically, the average on shop.price.Chaos Orb

#

and shop.price.Exalted Orb