Serializing a json response where name: value pairs are not provided. | Discord Gophers | Page 1

fallen copper Oct 24, 2023, 2:55 PM

#

I have a poorly structured json response that is returned by a science api.
It basically returns a list of Fields and then separately an array of string arrays which list the data from those fields in the order of the fields returned.

I am having trouble marshalling the result into a struct, I'm assuming, because there are no json tags to attach to the data struct (since it's just an array of string values, despite providing a response model of the data in that array.).

type Response struct {
    Fields []string        `json:"fields"`
    Data   []ScienceThings `json:"data"`
}

type ScienceThings struct {
    Name                 string
    UiName               string
    Diameter             float64
    Density              float64
    Albedo               float64
    Perihelion           float64
        ...

I seem to be able to get the result and the .Data but am having zero luck marshalling it into the data object to then output it in my test example - before I start going to town on actually using it. I'm using the typical GoFiber stanzas to get this, namely...

  var SbObjects Response
    json.Unmarshal(body, &SbObjects)
    if err != nil {
        fmt.Print(err.Error())
        os.Exit(1)
    }

but the result is the proper model but just with blanks and 0's even though it does list the correct number of records. How do I marshall the non-name/value paired json into the ScienceThings struct? My google-fu has failed to find a good solution here.

neat panther Oct 24, 2023, 3:04 PM

#

fallen copper I have a poorly structured json response that is returned by a science api. It ...

have you tried some debugging with fmt.Println?

fallen copper Oct 24, 2023, 3:06 PM

#

Yes. I get a response back fine into the endpoint via curl and of the proper reponse length. The issue is that the long []Data does not get marshalled into the struct from its json array.

Normally, with json tags, this happens almost automagically, I'm just not clear on how to indicate to Go how to serialize that json array if there are no tags for those fields.

neat panther Oct 24, 2023, 3:07 PM

#

are you sure that ScienceThings will have PascalCase named fields?

fallen copper Oct 24, 2023, 3:08 PM

#

ScienceThings has NO fields, that is the problem. It is a long array of Data which in itself just an array of strings.

#

Honestly, you would think NASA could write a proper API endpoint. 8-/

neat panther Oct 24, 2023, 3:08 PM

#

then make it strings

fallen copper Oct 24, 2023, 3:11 PM

#

I guess what I am trying to do is marshall that array of rsponse string in []Data into the scienceThings struct. So, without defining the struct in the manner I've defined it, how does one do that? I mean [][]string gives me the data but how to marshall it to the struct object?

#

(btw, thanks for your help @neat panther

full dust Oct 24, 2023, 3:17 PM

#

You might need to do it in separate steps where you first marshal them into the string slices, and then map them to the ScienceThings afterwards unfortunately.

fallen copper Oct 24, 2023, 3:18 PM

#

Ughhhh... was afraid of that. i was hoping there was a nice shortcut for that since I imagined it might be a common abstraction in Go land.

#

So, to summarize... marshall into the string slices, then iterate over those to get them into the struct?

full dust Oct 24, 2023, 3:20 PM

#

I believe so! I'll look into trying it another way and then can respond back if it works also!

fallen copper Oct 24, 2023, 3:21 PM

#

Oh wow... cool! Thanks! I can give you the actual endpoint I am poking if that's any help. It does work except for the data actualy, you know... getting into the struct... 🙂

#

It's a nasa jpl endpoint.

#

(or if you know of anythin to point at for me to figure it out, that'd alsdo be helpful... )

full dust Oct 24, 2023, 3:23 PM

#

That sounds great! Thanks!!

fallen copper Oct 24, 2023, 3:23 PM

#

It's a crapload of data so I put a 5 limit on it at the mo...

const jpl_comets = "https://ssd-api.jpl.nasa.gov/sbdb_query.api?fields=full_name,name,diameter,density,albedo,q,ad,per,rot_per,i,n,ma,G,H,K1,M1,t_jup,w,tp,a,e&sb-kind=c&limit=5"

#

And the fiber funciton is simply:

func GetJplObjects(c *fiber.Ctx) error {
    response, err := http.Get(jpl_comets)
    if err != nil {
        log.Fatal(err)
    }
...

#

etc...

full dust Oct 24, 2023, 3:59 PM

#

I built something that should be able to parse the data for you, but I don't know if it's the best way to do it. Basically, I am first marshaling the data into a struct that reflects the format that it's delivered in. Then I'm converting it to a map where the key is the name and the value is the field value. Then I'm marshaling that map and then un-marshaling that into the final struct.

I have a Go Playground example that I'll post. I just have to update the names of things really quick.

#

https://go.dev/play/p/yr7TsAkAeEF

Go Playground - The Go Programming Language

fallen copper Oct 24, 2023, 4:00 PM

#

Oh wow! Lemme check...

full dust Oct 24, 2023, 4:00 PM

#

It's not clean by any means, and I didn't do any error handling lol

#

I found this which may be better than marshaling and unmarshaling, but didn't read it much yet https://stackoverflow.com/questions/26744873/converting-map-to-struct

Stack Overflow

Converting map to struct

I am trying to create a generic method in Go that will fill a struct using data from a map[string]interface{}. For example, the method signature and usage might look like:

func FillStruct(data map[

fallen copper Oct 24, 2023, 4:01 PM

#

And thanks! I really just need an idea of how this would be done in Go. It's ashame there is not something more like pattern matching like in Rust or elixir... 🙂

full dust Oct 24, 2023, 4:02 PM

#

It was fun! There still could be and maybe I just haven't stumbled upon that yet 🙂 I still have a ton to learn!

fallen copper Oct 24, 2023, 4:02 PM

#

mapstructure, eh? Hmmm...

full dust Oct 24, 2023, 4:03 PM

#

Yeah I'm not sure about that yet either! I've used mapstructure as a tag when using Viper before

#

But never that library

fallen copper Oct 24, 2023, 4:04 PM

#

Ahhhh.. I see what you did. Clever. I thnk I was trying to make json marashalling work in some way which made me not try an approach like yours. Thanks tho!! i think that gives me the clue I need. I can jsut get the thing working and then swing back to refactor and see if there are other ways later.

#

Thanks Kolby!! Suer nice of you to help out like that!.

full dust Oct 24, 2023, 4:05 PM

#

If you find a better way, you should definitely let me know too! 🙂 And it was my pleasure!

fallen copper Oct 24, 2023, 4:06 PM

#

The key issue in most of this would be creating the map to then use mapstructure (or json). I gues mapstructure might work if I iterate over the returns fields and then create a map, to then use mapstrcuture. I(which might be more advantageous) but your solution seems like it'll work off the bounce.

#

Srsly thank you. Been grinding on this ofr a god two hours and was getting super frustrated. Merci1

full dust Oct 24, 2023, 4:12 PM

#

That's the worst when you get stuck on something! I'm always happy to help, or at least help find someone or something that can help if I can't

fallen copper Oct 24, 2023, 4:15 PM

#

OMG, actually you can make the whole thing work with the json unmarshalling... this is the key thing that you need to do in the struct. THANK YOU SO MUCH FOR this pointer.

#

type JplResponse struct {
    Fields []string   `json:"fields"`
    Data   [][]string `field:"fields"`
}

And the entire code works perfectly after that.

#

I have to admit I do not understand Go struct tags that well, but this was obviously the important bit.

#

I'll post in the Go Playground so you can see... It works perfectly now. THANKS SO MUCH!

#

@full dust
Check this:
https://go.dev/play/p/DTz7UQPmjSk

Go Playground - The Go Programming Language

full dust Oct 24, 2023, 4:27 PM

#

Oh I didn't know that you only wanted to put it in the JplResponse struct!
I thought that you wanted to create a struct that represented the field names and their value and I could have answered that much quicker lol

That's amazing that it's all working now though!!!

fallen copper Oct 24, 2023, 4:30 PM

#

(Doh... spoke too soon. I had referenced JPlObjects.Data... it stull sdoesn't have the name/value pairs. Lol... close though!)

#

I do... I think your solution or mapstructure is the way to go. You need to create a map... i think you're right.

full dust Oct 24, 2023, 4:32 PM

#

That was the only way that I could think to join them together to be able to map them to the struct without having to write a giant switch statement or something. But there could be other elegant ways to do this that I'm not sure about as well.

fallen copper Oct 24, 2023, 4:41 PM

#

The approach totally makes sense. I am also wondering (since that aws an intermediate response object which made sense if I could marshal direcly from json, and it is part of a getting it into a larger small body object struct, that it might be better jsut leaving it as the map, sooooo let's see... 🙂

full dust Oct 24, 2023, 4:59 PM

#

Leaving it as a map could be good! The only thing would be that they are still all strings when they are in the map, and I converted them to the different types similar to what you had in your struct when I marshaled them! If you are okay with having them as strings, then that could be good way to handle them too 🙂

#Serializing a json response where name: value pairs are not provided.