#OAS Generator - Gleam clients from Open API Specs

1 messages · Page 1 of 1 (latest)

tough vortex
#

https://github.com/CrowdHailer/oas_generator

Generate API clients from Open API (Swagger) Specs.
I have used this for several clients. I recently shared the github one.

I'm pretty happy with the API's that are generated. When the API is defined with basic features then it's mostly what I would write by hand.
It also allows me to discover the API from within my editor. The language server will autocomplete on all the fields that exist in a returned entity.

The generated code makes public a <operation>_request function to build requests and a <operation>_response function so that you can use it with any API client. This is most relevant for the separation between front and back end where API clients have different signatures.

I'm keen for people to try it out and here opinions on if the generated code is "good enough".
Try both generation and try any clients that were generated.

@split perch this was the code you were looking for.

GitHub

Generate HTTP clients from Open API specs. Contribute to CrowdHailer/oas_generator development by creating an account on GitHub.

edgy olive
#

Love to see any OAS generator. Do you support any templates or server side controller generation?

tough vortex
#

Nothing on the serverside. What do you mean by templates?

calm palm
#

Impressive

graceful tapir
#

Maan I've been working on an openapi parser and then generator for the past few days
Sniped

#

though I first wanted to generate server stubs instead

#

Great job on this!

edgy olive
tough vortex
#

What API's do you have that you want to use.

graceful tapir
#

so In the meantime I've created my own openapi decoder lol

graceful tapir
# tough vortex What API's do you have that you want to use.

What I was about to work on today is actually resolving references so that I could ingest a spec that's spread across multiple files. My end goal is to generate type safe Wisp (or whatever else) handlers where you can just plug in the implementations.

tough vortex
tough vortex
graceful tapir
#

I started writing something using Wisp recently and was thinking about docs and Swagger, then realized that Gleam is a language where there's probably no good way to automatically generate docs. So I thought to just switch it around - make the docs the source of truth

tough vortex
#

I always think the code should follow the docs, there's always two sides to the API. If not docs to code which code should the docs derive from. and even if you derive the docs from the server code you will need to have a docs to code generator for the client.

fair depot
#

If I have a spec on YAML format how do I generate the gleam client? Do I convert to json first using some tool like yq ?
The oas library README states "The oas library provides a decoder that is designed to be used with JSON or YAML."

graceful tapir
#

oas is just a decoder that should work for pretty much any data format. The thing is that there's no Gleam libraries to load YAML, or at least I haven't found any

#

But yeah, I think you have to convert to JSON right now

fair depot
#

Thanks, I will try that.

calm palm
#

That's a library for writing YAML rather than parsing it

#

YAML is a very complex format so parsing it is challenging and so far one one has been very interested in it

split perch
#

Fantastic, thanks

tough vortex
#

Published 1.0 to hex. Might have some bugs but I think in most cases the type checker running over the generated code will catch them. Also I think even if I fix the bug's I hope to keep the API constant. hence 1.0

fair depot
#

So, probably showing my total non-understanding of Decoders now, when I try to execute the generator on a spec I get the following error:

error: Could not decode file 'priv/se.json'

cause:
  0: UnexpectedFormat([DecodeError("another type", "Dict", ["components", "schemas", "SchoolUnitInfoResponse"])])

My guess is that I will have to write my own decoder for the type "SchoolUnitInfoResponse" ?
main:

pub fn main() {
  case generator.build("priv/se.json", ".", "skolan", []) {
    Ok(_) -> Nil
    Error(reason) -> io.print(snag.pretty_print(reason))
  }
}

Original yaml spec: https://api.skolverket.se/skolenhetsregistret/skolenhetsregistret_v2_openapi.yaml

tough vortex
#

Just eyeballing it, this looks a little suspect.

          properties:
            type:
              type: string
              example: 'schoolunit'
#

I don't think type should be nested

fair depot
#

Thanks, I just converted the yaml to json with yq :
cat skolenhetsregistret_v2_openapi.yaml | yq -o json > se.json
I will check.

tough vortex
#

1.1 released to support inline objects for Responses

tough vortex
#

1.2 release that support true as a schema value. These always validate and are decoded to a Dynamic value. useful if your payload is too dynamic to be worth parsing at the boundary.

#

git status

fair depot
#

I found out that my yaml was missing some type: object lines for data: attributes. And now I can generate schema and operations.gleam. lucyhappy I have one problem though and that is that the call to utils:set_query(query) get different types in query for different requests :

Type mismatch

Expected type:

    List(#(String, String))

Found type:

    List(#(String, Option(a)))

So sometimes the call is using the first (which also http/request takes) and sometimes I get the second type.

tough vortex
#

it should always be the second.

fair depot
#

After reporting my code generating problems to the designers of the api spec. they made a major re-organization of it. lol.
The API is kept intact but the spec was extensively cleaned up. oas_generator in the service of quality improvement. lucyhappy

split perch
#

I was surprised by the amount of deps installed by this package. Specially a lot of JS related stuff. But this doesn't require node right?
Wonder if all these are necessary.
Too many deps in package is trouble. I often run into dependencies conflicts.
I also would prefer if this doesn't return a Snag type (that is better as an application choice).

tough vortex
# split perch I was surprised by the amount of deps installed by this package. Specially a lot...

Currently I run it on the node environment, using midas_node.
Potentially I could replace all that with simplifile as it's just reading files and reduce deps.
As I always install this as a dep dependency, it's not a priority for me to reduce the number of deps. I'd probably accept the PR to reduce them if the API remained the same.
Returning snag as I always run this as a dep task and print the error again I don't have a usecase to switch on error type. Are you using oas generator in your application at runtime?

fair depot
#

I noticed that enum strings are just aliased to String. Would it be possible to create an gleam enum type and have decoder/encoder?

#

Also when I am running on an Erlang target I do not have a Midas runner. If would be nice to just having oas_generator to exclude generating stuff in the main module (things after "// GENERATED ----") and just rely on operations and schema modules.

tough vortex
#

I think the enum thing would definetly be welcome.
I do like having the combined top level functions in a module separate to the operations. maybe that's not a good design. I'm open to opinions on how best to lay it out

#

@fair depot are you able to share the client you are generating? I'd like to start adding generated clients to the readme

fair depot
#

I think that the combined top level functions very well could go into a separate module but maybe not appened into the "main" one?
So modules schema -> operations -> transactions? (and utils).
I will push something to github once I have it in a presentable manner.

tough vortex
fair depot
#

My knowledge on snag and error handling is somewhat limited ( appr. = 0) but I had a comment on the migration to simplifile. I am using an erlang target for my generator and reading files works "fine and dandy" as it is now. The thing with midas is that there is no erlang runner as for now, but it is not too much work to implement the "higher order functions" yourself.

#

I have digressed a bit and am looking into converting the yaml to json, using cymbal with decoder pr and glam. It seems a bit tricky to decode yaml so maybe I am digging into a rabbit hole. (But there is a lot done already by the PR author).

GitHub

Build YAML in Gleam! Contribute to dinkelspiel/cymbal development by creating an account on GitHub.

tough vortex
calm palm
#

I think you would use the same schema for both without modification

#

We don’t have a yaml parser and they’re challenging to implement, so might get a bit stuck there

fair depot
tough vortex
#

@split perch thanks for the PR. Just pushed to hex.

fair depot
#

I have another openapi spec. , this time on json format, that seems to challenge the generator a bit, or oas/decoder actually:

src/oas/generator.gleam:1214
[DecodeError("another type", "Dict", ["paths", "values", "get", "parameters", "*"])]
error: UnexpectedFormat

Link to json openapi: https://api.skolverket.se/planned-educations/v3/api-docs/--v3-active

tough vortex
#

Yeah the openapi spec is large. pieces of it that don't show up in the specific API specs I've used are quite probably not implemented.
In terms of how to fix your issue i'd try and narrow down the specific parameter that isn't being decoded from the spec.
Then I'd certainly accept a PR to handle one more case if we can link to detail in the spec that says it should be valid.

tough vortex
modern veldt
#

will there be a way to generate an OpenAPI spec from a Gleam Wisp server, or is that out of scope for the future?

calm palm
#

Probably going the other way would work better

#

Write a spec and generate a server and you fill in the business logic

normal summit
#

The problem i would have having to write a openapi specification and generating a server stub, would be that it does not follow my structure. I would almost certainly have to give up some pattern or code style in order to be able to keep generating the stubs when the api changes.

Also i would much rather write gleam than yaml/json

calm palm
#

You would not be able to follow your pattern either if you want to generate the spec from the web app also

#

You'd need to use a convoluted pattern that enables enough reflection or static analysis, and it would likely also have a performance cost.

normal summit
#

Since gleam doesn't have any reflection (or atleast not enough to do the usual annotation of endpoints and inputs and outputs) i would think that generating the spec would be completely separate from the actual endpoints/inputs/outputs. So i don't see how that limits my patterns in any way.

calm palm
#

If there's no link between the server and the spec you're just writing a spec but with extra steps that make it more challenging

normal summit
#

Yes, but atleast it would solve my problem of me being able to create my endpoints the way i want, and i don't have to think about generated files i can't edit.

tough vortex
#

The scope of oas_generator is (and will remain) spec -> code. I hope to see more projects in the wild because I think solving talking to the outside world is worth a few experiments and different approaches.

fair depot
tough vortex
graceful tapir
#

remember I told you I was working on parsing the spec too? Did it with the new decode API, can send you the decoders

#

probably doiesn't match yours 100% but a good place to start with the update

fair depot
# tough vortex <@1115284075977134080> are you able to share the client you are generating? I'd ...

I have uploaded the first WIP version. For the YAML spec I am able to decode it with a slightly modified version of the decoder PR for cymbal now. The statistics api which has a json spec I am not yet able to decode.
https://github.com/karlsson/snafe

GitHub

Gleam client for the Swedish National Agency for Education OpenAPI - karlsson/snafe

fair depot
#

Small update, for the statistics api spec. it turned out it is the spec that is the problem and not the oas_generator nerdglasses .
Some type was "String" instead of "string" and almost all responses were using a common schema that ended with "object" although there was a lot of different types to be parsed below that.

#

I sent some info to the development team....

tough vortex
#

Nice. I'm thinking somewhere I need to keep track of the diff's made to public specs.

tough vortex
tough vortex
tough vortex
fair depot
#

Maybe this is shown somewhere, but it seems that I can get away with a rather simple midas task-runner for erlang (or generic httpc really) for the oas_generated midas tasks since they all seem to be a Fetch task. Very nice!

fn map_http_response(
  response: Result(Response(BitArray), httpc.HttpError),
) -> Result(Response(BitArray), t.FetchError) {
  case response {
    Ok(response) -> Ok(response)
    Error(httpc.InvalidUtf8Response) -> Error(t.UnableToReadBody)
    Error(httpc.FailedToConnect(..)) ->
      Error(t.NetworkError("Failed to connect"))
  }
}

fn run(task: t.Effect(a)) {
  case task {
    t.Fetch(request, resume) ->
      resume(map_http_response(httpc.send_bits(request)))
    _ -> t.Abort(snag.new("Expected a fetch instruction"))
  }
}
tough vortex
# fair depot Maybe this is shown somewhere, but it seems that I can get away with a rather si...

There isn't a description of how to do this anywhere. Though it's a positive sign that you were able to work it out.
Indeed for the OAS libraries all effects are fetch's so it's fairly trivial to implement.
Two things of note

  1. The reason I use Midas is to encode OAuth flows in which case the Follow effect is also uses.
  2. I'd change your error message from expected a fetch, to something like I don't handle {{effect}} in this environment
#

Just released oas_generator 1.3.0 based on oas 3.0. Produces the same output for my API client for messagebird

fair depot
#

I tried to update but it seems that there are some dependency conflict. oas_generator now have gleam_json dep >= 3.0 while midas (and oas I think) still have < 3.0 ? Maybe you should push latest oas_generator to github as well...

tough vortex
#

oops, well the code is now pushed to github

#

oas_generator can't rely on 2.x and 3.x versions of gleam_json because of the removal of json.UnexpectedFormat(..)

#

Pushed a new version of midas with relaxed dependencies

fair depot
#

I am panicking now 😅 .

src/oas/generator.gleam:635
#("pattern must start with '/' and be valid url", "/v2/contracts/{organizationNumber}/{educationProviderCode}", ...)
runtime error: panic

could not find all parameters in match

stacktrace:
  oas/generator.gen_request_for_op src/oas/generator.gleam:636
  oas/generator.-gen_fns/4-fun-1- src/oas/generator.gleam:1035
  gleam/list.map_loop src/gleam/list.gleam:409
  gleam/list.flat_map src/gleam/list.gleam:747
  oas/generator.gen_operations_and_top_files src/oas/
...

I can see that the gather_match function in oas has changed to require a complete url while it was only the path part before, but the input from the spec is still only the pathpart:

@@ -754,8 +770,8 @@ pub fn query_parameters(parameters) {
 }
 
 pub fn gather_match(pattern, parameters, components: Components) {
-  case pattern {
-    "/" <> pattern -> {
+  case uri.parse(pattern) {
+    Ok(Uri(path: "/" <> pattern, ..)) -> {
       string.split(pattern, "/")
       |> list.try_map(fn(segment) {
         case segment {
@@ -773,7 +789,7 @@ pub fn gather_match(pattern, parameters, components: Components) {
         }
       })
     }
-    _ -> Error("pattern must start with '/'")
+    _ -> Error("pattern must start with '/' and be valid url")
   }
 }
#

Can it be a valid url if it starts with '/' and not with http(s)?

#

If I revert to the previous version of gather_match it works.

tough vortex
#

A path starting with / should still parse as a valid URL. that change was made to correctly clean query parameters

fair depot
#

OK, so it something else in the patter that is not a valid URL, like the parameters with { } braces?

feral ledge
#

maybe they need to be url encoded?

fair depot
#

Most probably, but the problem is that this is what the openapi spec. define. Anyway, I was able the verify that this is the problem (on erlang target). Maybe it works on js target since there is a pure gleam implentation there.

#
4> gleam_stdlib:uri_parse(<<"/v2/contracts/{organizationNumber}/{educationProviderCode}">>).
{error,nil}
5> gleam_stdlib:uri_parse(<<"/v2/contracts/organizationNumber/educationProviderCode">>).
{ok,{uri,none,none,none,none,
         <<47,118,50,47,99,111,110,116,114,97,99,116,115,47,111,
           114,103,97,110,105,122,...>>,
         none,none}}
fair depot
#

If you just wan't to strip off the query part, maybe:

pub fn gather_match(pattern, parameters, components: Components) {
  case pattern {
    "/" <> pattern -> {
      string.split(pattern, "?")
      |> list.first()
      |> result.unwrap("")
      |> string.split("/")
      |> list.try_map(fn(segment) {
....

?

feral ledge
#

could do split_once

fair depot
#

nope

feral ledge
fair depot
#

sorry yes! maybe

#

the problem is that split_once will error if there is no query part and then I have to do some fancy unwrap and if ok then I heve to take the first part of a tuple. My gleam wrangling is not ready for this. 😄

#

I will leave it to @tough vortex to come up with some beautiful solution.

feral ledge
#

maybe we could open an issue on the stdlib? i feel like the behaviour of this should not be platform dependent

fair depot
#

I am not sure that the behaviour on js target is different, I guessed since I thought @tough vortex was running on this and that it worked ok for him.

Newbies take on split_once :
string.split_once(pattern, "?")
|> result.unwrap(#(pattern, ""))
|> fn(a) { a.0 }
|> string.split("/")
Not sure it is more readable.....

tough vortex
#

Hmm I'm testing this

uri.parse("/v2/transcript/{transcript_id}")
  |> echo

and getting an ok result

Ok(Uri(scheme: None, userinfo: None, host: None, port: None, path: "/v2/transcript/{transcript_id}", query: None, fragment: None))
frail crystal
tough vortex
#

I'm not getting that error though on the JS target. version 0.60.0 of stdlib

fair depot
#

I get Error(Nil)for the Erlang target for the same.
I can see in the comments for uri.parse:

  // TODO: This is not perfect and will be more permissive than its Erlang
  // counterpart, ideally we want to replicate Erlang's implementation on the js
  // target as well.
#

So I guess the JS implementation permits more. Question is if {}are valid characters in the URL. If not uri.parseis probably not the right function to use or if it is then the Erlang implementation is wrong.

feral ledge
tough vortex
#

yeah path templates are different to uris

tough vortex
#

published as 3.0.1

fair depot
#

Thank you! Yes it works sjust fine now.

tough vortex
lean terrace
#

Are there any docs, examples or existing projects showing what t.fetch, t.do, t.try, t.Done, base_request, and handle_errors are supposed to look like (for example when used with gleam_httpc or gleam_fetch)? or at least the type signatures they're supposed to have? maybe it would be nice for the generator to output a placeholder type signature for those functions with a todo body if they dont already exist in the project? cause at the moment i'm pretty confused trying to fill in the gaps haha

fair depot
lean terrace
#

ohhh, the t. stuff comes from midas... I tried to make up functions/types for that which worked haha. would be nice if midas was mentioned somewhere in the readme/docs/anywhere lol

tough vortex
#

This is a fair point.
I think I've not written about the midas bit because I'm not sure I want to use it. How ever I haven't come up with a better way of composing request/response in a way that works with and without promises.
It's possible to just use the code in the operations file, i.e. delete the top level module if you don't want midas.

tough vortex
#

Published 1.4.0 of OAS generator, this now handles nested objects by creating intermediate custom types.
It doesn't yet do any better job of explaining how to use the library :-p

tough vortex
#

Published 1.5.0 of OAS generator, this handles additionalProperties by creating dicts, it also handles nested objects directly in request specifications'
It also tells you how to use it.

#

Also I'm a bit stuck on having better error messages when decoding dictionaries. Any pointers/help appreciated. Original post here #general message

fair depot
#

I just pulled in the latest oas_generator (1.6.3) and I get some compile errors for missing functions in my utils.gleam file:
utils.object, utils.dict and utils.dynamic_to_json.
Are there any examples (or description) for what is expected in those?

#

They are called from the generated schema.gleam

tough vortex
#

Actually it's missing dynamic_to_json. If your app doesn't use it can implement with a todo.
I'll share a full implementation next week

tough vortex
#

This is how i tackle dynamic_to_json

pub fn dynamic_to_json(d) {
  let assert Ok(json) = decode.run(d, decoder())
  json
}

fn decoder() {
  {
    use <- decode.recursive
    decode.one_of(
      decode.dict(decode.string, decoder())
        |> decode.map(fn(data) {
          let entries = dict.to_list(data)
          json.object(entries)
        }),
      [
        decode.string |> decode.map(json.string),
        decode.bool |> decode.map(json.bool),
        decode.int |> decode.map(json.int),
        decode.float |> decode.map(json.float),
        decode.list(decoder()) |> decode.map(json.array(_, fn(x) { x })),
        decode.new_primitive_decoder("Nil", fn(x) {
          case x == dynamic.nil() {
            True -> Ok(json.null())
            False -> panic as "unexpected json type"
          }
        }),
      ],
    )
  }
}

It seems wasteful am I missing anything clever

feral ledge
#

why do you panic?

tough vortex
#

Ahh wait this actually doesn't work.
dynamic.nil != json.null

tough vortex
#

I think I need a decode.nil function

#

This was previously
x == dynamic.from(json.null)
I can't recreate that without ffi

tough vortex
#

This is the solution I've gone for

fn decoder() {
  {
    use <- decode.recursive
    decode.one_of(
      decode.dict(decode.string, decoder())
        |> decode.map(fn(data) {
          let entries = dict.to_list(data)
          json.object(entries)
        }),
      [
        decode.optional(decode.string)
          |> decode.map(json.nullable(_, json.string)),
        decode.bool |> decode.map(json.bool),
        decode.int |> decode.map(json.int),
        decode.float |> decode.map(json.float),
        decode.list(decoder()) |> decode.map(json.array(_, fn(x) { x })),
      ],
    )
  }
}
#

Just wang in an optional on one of the types.

tough vortex
tough vortex
#

Switch the anonymous object to be named based on the hash of their types and not an internal number. The number approach was unstable as new types added. Now the same anonymous type always gets the same name

#

Also is it possible to reexport constructors
i.e.

pub const MyThing = internal.Anon123
tough vortex
exotic hearth
#

Is the readme out of date? I can't run the petstore example and I notice that the signature for generator.build has changed.

#

I get unknown content type: application/octet-stream (after needing to manually create the folder src/petstore)

exotic hearth
#

It feels like I'm missing some prerequisite work to use this tool that I am not aware of.

tough vortex
#

Ahh, it could be out of data. I didn't realize there was that content type in the pets example.

#

sorry about that.

exotic hearth
#

I'm not exactly sure what that means. This seems like exactly the tool I could use for my current project, as I have an api schema with 782 routes that I need to work with. I don't need to interface with all routes immediately, but I will have to hit most of them over time. I don't want to do this manually 😅

tough vortex