Discussion on API doc generation | Dagger | Page 1

fierce shell Sep 17, 2022, 7:54 AM

#

I'm trying to generate API docs using the docusaurus gql plugin. I'm assuming the API schema is in api/*.gql. Docusaurus fails with this error

[ERROR] AggregateError: Failed to find any GraphQL type definitions in: ../api/*.gql;
 - Syntax Error: Expected ":", found Name "String".
  - Syntax Error: Expected ":", found String "This directory with the file at the given path removed".
  - Syntax Error: Expected ":", found Name "HostVariable".
  - Syntax Error: Expected ":", found Name "HTTPRemote".
  - Syntax Error: Expected ":", found Name "Secret".

#

When I try to lint specific schema files, there are similar errors

#

✖ 1 error detected
vikram@dagger:~/public/dagger/website$ ./node_modules/.bin/graphql-schema-linter /home/vikram/public/dagger/api/git.gql
/home/vikram/public/dagger/api/git.gql
1:13  Cannot extend type "Query" because it is not defined.  invalid-graphql-schema
23:11 Unknown type "Directory".                              invalid-graphql-schema

✖ 2 errors detected
vikram@dagger:~/public/dagger/website$ ./node_modules/.bin/graphql-schema-linter /home/vikram/public/dagger/api/directory.gql
/home/vikram/public/dagger/api/directory.gql
26:5 Syntax Error: Expected ":", found String "This directory with the file at the given path removed".  graphql-syntax-error

✖ 1 error detected

#

Is there a base Query type defined in a different location?

fierce shell Sep 17, 2022, 8:53 AM

#

The only schema that successfully generated so far without errors (see above) was the one from examples/...netlify, here's a quick sample of how it looks

fierce shell Sep 17, 2022, 9:59 AM

#

^^ with a docusaurus plugin

#

I will also try a few others and update here

naive river Sep 17, 2022, 4:25 PM

#

Those files probably have typos, they have never been actually parsed by any tooling before

#

so you may just be catching those human errors

#

How can I reproduce? I can help fix the typos

clever fog Sep 17, 2022, 4:26 PM

#

repro instructions 🙏

#

I'm guessing this one: https://www.npmjs.com/package/docusaurus-graphql-plugin

npm

docusaurus-graphql-plugin

Docusaurus plugin generating Markdown documentation from a GraphQL schema.. Latest version: 0.8.0, last published: 3 months ago. Start using docusaurus-graphql-plugin in your project by running npm i docusaurus-graphql-plugin. There are no other projects in the npm registry using docusaurus-graphql-plugin.

#

looks like we need to point it to

 // can be a path, a glob or an URL
 schema: "schema.graphql",

#

On cloak branch of dagger/dagger repo:

yarn add graphql 
yarn add docusaurus-graphql-plugin

I made my website/docusaurus.config.js plugins section look like this:

plugins: [
    "docusaurus-plugin-sass",
    "docusaurus2-dotenv",
    [
      "docusaurus-graphql-plugin",
      {
        // can be a path, a glob or an URL
        schema: "../api/*.gql",
      },
    ],
  ],

npx docusaurus docs:generate:graphql

got errors that look like they could be worked on!

[ERROR] AggregateError: Failed to find any GraphQL type definitions in: ../api/*.gql;
 - Syntax Error: Expected ":", found Name "HTTPRemote".
  - Syntax Error: Expected ":", found Name "Secret".
  - Syntax Error: Expected ":", found Name "HostVariable".
  - Syntax Error: Expected ":", found Name "String".
  - Syntax Error: Expected ":", found String "This directory with the file at the given path removed".
    at loadFile (/Users/jeremyadams/src/dagger-fork/website/node_modules/@graphql-tools/load/cjs/load-typedefs/load-file.js:35:19)
    at async /Users/jeremyadams/src/dagger-fork/website/node_modules/@graphql-tools/load/cjs/load-typedefs/collect-sources.js:172:25

naive river Sep 17, 2022, 4:46 PM

#

2 things you can do to fix a bunch of those errors

#

fields should be of the form name: Type but in some cases we forgot the : resulting in name Type. Adding the missing column will fix those

#

You need to add one file let’s call it root.gql:

type Query {}

#

And 3. you need to load api/*.gql as a whole. If the tool doesn’t support that, you can concatenate them all into a single file then load that

clever fog Sep 17, 2022, 4:50 PM

#

Also found these errors when trying these schema files

{
  // can be a path, a glob or an URL
  schema: "../examples/**/schema.graphql",
},

npx docusaurus docs:generate:graphql

[ERROR] Error: Unable to merge GraphQL type "Netlify": Field "deploy" already defined with a different type. Declared as "SiteURLs", but you tried to override with "Deploy"
    at mergeType (/Users/jeremyadams/src/dagger-fork/website/node_modules/@graphql-tools/merge/cjs/typedefs-mergers/type.js:26:19)
    at mergeGraphQLNodes (/Users/jeremyadams/src/dagger-fork/website/node_modules/@graphql-tools/merge/cjs/typedefs-mergers/merge-nodes.js:38:73)
    at mergeGraphQLTypes (/Users/jeremyadams/src/dagger-fork/website/node_modules/@graphql-tools/merge/cjs/typedefs-mergers/merge-typedefs.js:67:64)
    at mergeTypeDefs (/Users/jeremyadams/src/dagger-fork/website/node_modules/@graphql-tools/merge/cjs/typedefs-mergers/merge-typedefs.js:13:22)
    at makeExecutableSchema (/Users/jeremyadams/src/dagger-fork/website/node_modules/@graphql-tools/schema/cjs/makeExecutableSchema.js:72:58)
    at mergeSchemas (/Users/jeremyadams/src/dagger-fork/website/node_modules/@graphql-tools/schema/cjs/merge-schemas.js:32:63)
    at getSchemaFromSources (/Users/jeremyadams/src/dagger-fork/website/node_modules/@graphql-tools/load/cjs/schema.js:56:46)
    at Object.loadSchema (/Users/jeremyadams/src/dagger-fork/website/node_modules/@graphql-tools/load/cjs/schema.js:19:12)

#

Maybe I need to exclude the gen stuff

website ➤ ls -1 ../examples/**/schema.graphql                                            git:cloak*
../examples/alpine/schema.graphql
../examples/netlify/go/schema.graphql
../examples/netlify/ts/gen/core/schema.graphql
../examples/netlify/ts/gen/netlify/schema.graphql
../examples/netlify/ts/schema.graphql
../examples/python/hello/schema.graphql
../examples/yarn/schema.graphql

#

if I try it with just alpine/schema.graphql I get

[ERROR] Error: Cannot extend type "Query" because it is not defined.

Unknown type "Filesystem".

Unknown type "Query".
...

Guessing that since core is not available in this tree, I'm missing those

#

(okay going to run and move a bunch of stuff my old house)

naive river Sep 17, 2022, 4:59 PM

#

clever fog if I try it with just `alpine/schema.graphql` I get ``` [ERROR] Error: Cannot e...

that should be fixed with my 2nd point (one liner root.gql)

#

there should not be any Filesystem type in api/

#

sorry just realized we’re talking about 2 different directories

#

@fierce shell there are 2 different apis here, the current one and planned new api

current api: core/*.schema.go (strings embedded inside go types). Not sure how to get that into docusaurus cc @jade flower

#

planned new api (not yet implemented): api/*.gql

#

both would be useful to document

#

but probably 1. is the priority

#

so we need to figure out a way to extract the schemas from those go files, I bet there is tooling for that already

clever fog Sep 18, 2022, 7:08 AM

#

took a swag at this:

From website directory:

yarn add graphql-schema-linter
npx graphql-schema-linter ../api/*.gql
###fix things, run above command, repeat until just non-critical errors from linter, about 30 mins###
npx docusaurus docs:generate:graphql
###you get md files in website/docs/api!###

naive river Sep 18, 2022, 7:34 AM

#

yay!

#

how does it look?

clever fog Sep 18, 2022, 7:35 AM

#

https://github.com/dagger/dagger/pull/3135

clever fog Sep 18, 2022, 7:36 AM

#

naive river how does it look?

I haven't looked at the pages all rendered pretty yet. Got to move them to right place, etc.
I'm going to open an issue for finishing the API schema work. The linter demands more tribute!!

#

here's one

#

#

#

linter wants all those Descriptions filled in 🙂

#

This kind of stuff. Linter is not messing around 😆

/Users/jeremyadams/src/dagger-fork/api/directory.gql
3:15  The `id` argument of `directory` is missing a description.                                                                                                                                                          arguments-have-descriptions

9:1   The fields of object type `Directory` should be sorted in alphabetical order. Expected sorting: contents, diff, directory, file, secret, withCopiedFile, withDirectory, withNewFile, withoutDirectory, withoutFile  type-fields-sorted-alphabetically

12:14 The `path` argument of `contents` is missing a description.                                                                                                                                                         arguments-have-descriptions
15:10 The `path` argument of `file` is missing a description.                                                                                                                                                             arguments-have-descriptions
18:12 The `path` argument of `secret` is missing a description.                                                                                                                                                           arguments-have-descriptions

naive river Sep 18, 2022, 7:51 AM

#

That looks great 😍

clever fog Sep 18, 2022, 7:51 AM

#

Here's the todo https://github.com/dagger/dagger/issues/3136

naive river Sep 18, 2022, 7:52 AM

#

The linter only wants what’s best for us. Whatever pain it inflicts, it’s out of love.

clever fog Sep 18, 2022, 7:52 AM

#

https://tenor.com/view/the-simpsons-care-bear-threatening-intensive-care-bear-gif-4055068

Tenor

Intensive Care Bear

▶ Play video

naive river Sep 18, 2022, 7:52 AM

#

https://tenor.com/view/tough-love-breaking-bad-better-call-saul-gif-10622603

Tenor

tough love

▶ Play video

#

ha ha almost picked that one

bronze marsh Sep 18, 2022, 5:30 PM

#

@naive river @jade flower would it be useful to add a hidden cloak schema command in the meantime that outputs the unified schemas from core until we move those to go:embed ?

jade flower Sep 18, 2022, 6:10 PM

#

I don’t think that’s necessary, most things should work using the introspection API

bronze marsh Sep 18, 2022, 6:14 PM

#

jade flower I don’t think that’s necessary, most things should work using the introspection ...

not sure if the docusaurus graphql plugin can generate docs based on the introspection API. We could craft a query to the API to dump the schema, but those would be separate steps. Having something like cloak schema would allow us to get the schema without starting the API server and running scripts to get it

bronze marsh Sep 18, 2022, 6:15 PM

#

bronze marsh not sure if the docusaurus graphql plugin can generate docs based on the introsp...

just did a quick search in the repo and doesn't seem to mention anything regarding introspection (https://github.com/Zhouzi/docusaurus-graphql-plugin)

bronze marsh Sep 18, 2022, 6:24 PM

#

bronze marsh just did a quick search in the repo and doesn't seem to mention anything regardi...

seems like it allows fetching from a URL but apparently it needs to be the raw schema, no the introspection type result

bronze marsh Sep 18, 2022, 6:26 PM

#

bronze marsh seems like it allows fetching from a URL but apparently it needs to be the raw s...

correction, seems like it could work, since it's using UrlLoder from graphql. Checking now (https://github.com/zhouzi/docusaurus-graphql-plugin/blob/main/packages/docusaurus-graphql-plugin/src/index.ts#L68)

GitHub

docusaurus-graphql-plugin/index.ts at main · zhouzi/docusaurus-grap...

Docusaurus plugin generating Markdown documentation from a GraphQL schema. - docusaurus-graphql-plugin/index.ts at main · zhouzi/docusaurus-graphql-plugin

bronze marsh Sep 18, 2022, 6:47 PM

#

yep, it works.

import { UrlLoader } from '@graphql-tools/url-loader'
import { printSchema } from 'graphql'

const loader = new UrlLoader();

const [{ schema }] = await loader.load(`https://swapi-graphql.netlify.app/.netlify/functions/index`, {});

console.log(printSchema(schema));

naive river Sep 18, 2022, 6:47 PM

#

so this tool can be developed with the regular Dagger SDK 🙂

bronze marsh Sep 18, 2022, 6:48 PM

#

TL;DR @clever fog@fierce shell if you have a cloak engine running you can use the UrlLoader in the docusaurus graphql plugin and it will automatically generate docs for the available schema (including extensions if any) cc @naive river 😉

naive river Sep 18, 2022, 6:55 PM

#

we could have a docusaurus plugin that you just install the usual way and boom it works

#

embedded dagger sdk would take care of starting the engine

fierce shell Sep 18, 2022, 7:18 PM

#

I am also looking at spectaql which supports doc generation "From a live endpoint using the introspection query."

#

@clever fog regarding th eplugin, I tried both the one you mentioned and also this one https://www.npmjs.com/package/@edno/docusaurus2-graphql-doc-generator#plugin-loaders

#

The benefit of the second one is that it allows you to define the output directory, so it's not necessary to move the generated docs from website/ to docs/ separately.

#

@bronze marsh if I wanted to test the introspection API what should be the endpoint I provide to the URLloader?

bronze marsh Sep 18, 2022, 7:29 PM

#

start the cloak API and http://localhost:8080/query

fierce shell Sep 18, 2022, 7:31 PM

#

Thanks

#

yarn add @edno/docusaurus2-graphql-doc-generator graphql
yarn add @graphql-tools/url-loader

// docusaurus config 
plugins: ["docusaurus-plugin-sass", "docusaurus2-dotenv",  ["@edno/docusaurus2-graphql-doc-generator",  {loaders: {UrlLoader: "@graphql-tools/url-loader", options: { subscription: ""}}, schema: "http://localhost:8080/query", rootPath: "../docs", baseURL: "api"}]]

npx docusaurus graphql-to-doc

#

produces

Documentation successfully generated in "../docs/api" with base URL "api".
36 pages generated in 0.404s from schema "http://localhost:8080/query".

#

#

#

Interestingly, this also generates the sidebar and doesn't seem to care about the lint errors

fierce shell Sep 18, 2022, 9:07 PM

#

I have also tried SpectaQL. The generated output from introspection is pretty good. It also includes examples. However there are two issues:

It generates HTML, not Markdown. Therefore it has to be integrated into Docusaurus as a separate set of static pages. Since it's generated separately, it also does not use the Docusaurus theme, although it allows full customization via its own CSS files. We would need to put in some extra work to have it fit with the remaining pages.
Again because it is a static page collection, (a) the sidebar links cannot be auto-generated and (b) the sidebar links open in a new tab. (a) could be done via an additional script and (b) could perhaps be justified that it's OK for reference docs to open in a new tab

#

#

#

Do you want to see all 3 generator options "live"? I have uploaded the website static build to https://vikram-dagger.github.io/ -> please have a look and lmk your thoughts

naive river Sep 18, 2022, 9:13 PM

#

Do you have a strong preference @fierce shell ? Or are you not sure

#

If you could open an issue summarizing options and your recommendation, that would be great. We can call for feedback there, then @jade flower should make the call

fierce shell Sep 18, 2022, 9:15 PM

#

I would go with option 2 (edno plugin) at the current time because (a) it integrates directly with the theme/sidebar nav without needing any extra work (b) the output directory is configurable so we don't need extra scripting to move generated files around (this is minor) (c) it seems to have a reasonable set of customization options in its docs

#

SpectaQL is "nicer" but requires more resources to get it integrated neatly with the rest of the docs. Default plugin could also be fine except that we need to move the generated docs from their initial resting place to a new one. Regarding this last point -> @clever fog is there a specific reason we have moved our docs directory outside the website directory? The default location is website/docs.

#

@naive river, yes I will open an issue as per your suggestion and update here once done

clever fog Sep 18, 2022, 10:09 PM

#

fierce shell SpectaQL is "nicer" but requires more resources to get it integrated neatly with...

was like that when i got here 😂 @prime gust might know

#

one benefit has been easy github browsing of docs without "website"/docusaurus files in the way

naive river Sep 18, 2022, 10:15 PM

#

the name website is confusing

#

(it’s not the dagger.io website)

fierce shell Sep 19, 2022, 8:05 PM

#

Yes, I agree - I also thought it referred to the dagger.io website initially

#Discussion on API doc generation