Packages, Templates

[hoary thunder]

As for the registry protocol: I think it would likely be designed in such a way that an advanced registry with custom auth, package enumeration, and so on could exist (with a backend server) while still supporting more basic registries that are just file servers / a CDN.

The index.json thing will probably need to be replaced down the road as it gets too large at some point.

The index.json is not actually used under normal operation. There is no check in the CLI that a package exists before fetching it.

It's used for typst init and maybe for some other stuff like hints.

But since the CLI only fetches from packages.typst.org, it's still safe (as long as you trust packages.typst.org).

[fallow seal]

Just wondering, not necessarily feature requests:

• Why do the PDF docs hosted by Typst not open in the browser like they do on CTAN?
• Why do templates get to have thumbnails, but packages don't?

[placid sky]

Packages thumbnails can be interesting but not necessarily, some packages add only functions or features and they can't show what they do directly like a template

[fallow seal]

True, but not having any thumbnails at all I think makes the page look less appealing than it could be

[floral crater]

On the first point: IIRC this is controlled via the Content-Disposition header. I'm guessing this isn't easy to toggle in the software hosting the website?
-# On Firefox, the download attribute being set on the element with a URL will also behave like a file download, according to mdn. Though one can set browser.download.open_pdf_attachments_inline in about:config to force the inline behavior (regardless of content-disposition or download attribute).

[hoary thunder]

We don't host any documentation, it's all GitHub links, so I don't think there's anything we can do about that

[fallow seal]

Right, I checked again through many READMEs and I couldn't find any docs leading to typst.app, so I must have been delusional when looking at the URL then.

[manic zenith]

Why do the PDF docs hosted by Typst not open in the browser like they do on CTAN?

We don't host any documentation, it's all GitHub links, so I don't think there's anything we can do about that

It seems like it is possible to get GitHub links to PDFs to render in the browser. At least, this one renders fine for me using Firefox:

https://raw.githubusercontent.com/mewmew/cheat-sheet-typ/master/cheat-sheet.pdf

Edit: hmm, the same link doesn't work when I try it in Chromium. So it seems to be browser dependent

[fallow seal]

Oh, that's interesting. I guess just linking to https://github.com/mewmew/cheat-sheet-typ/blob/master/cheat-sheet.pdf would give the most consistent experience.

[manic zenith]

For me, the prefix doesn't work in Firefox when using https://github.com/mewmew/cheat-sheet-typ/blob/master/cheat-sheet.pdf

But it does work when using https://github.com/mewmew/cheat-sheet-typ/raw/master/cheat-sheet.pdf

(i.e. use /raw/ instead of /blob/)

Haha, but /blob/ works in Chromium. And /raw/ in Firefox. ^^ lol

[fallow seal]

Oh, you mean like opening in the browser's PDF viewer, not GitHub's one?

[manic zenith]

Yes, exactly.

[fallow seal]

On Chrome, blob still opens in GitHub

[manic zenith]

I now got the GitHub preview one to work in both Firefox and Chromium (with /blob/). The issue was NoScript for me in Firefox.

However, the /blob/ preview version is just an image? For instance, links doesn't work in the PDF.

[fallow seal]

Weird, I thought there used to be an option in the sense "open full PDF"

[manic zenith]

I guess you have to click the "download raw file" link for that.

[fallow seal]

Apparently that "open full PDF" option I was thinking of is just to load more pages, but it's still images like you said.

[manic zenith]

right

[crimson sleet]

Just append ?raw=true and you get the in browser view

[fallow seal]

That still directly starts the download for me, on Chrome at least.

[fresh hornet]

Since we currently see downtime for the package registry we should reopen the discussion of custom independent registries again.

The current work around is to locally install / manual cache the packages the you need.
As mentioned by @lone granite in #discussions message this is not possible for ci/cd. It is also very uncool to be forced to do that. It feels a bit like a lock in. It means that even when not using the webapp, you are still dependent to an external service that you can not change or choose (when using packages). Mirrors are also not possible.

The web app already allows private packages for pro users. I don't have pro and don't use the web app but it sounds a bit like a custom registry, but hosted on and restricted to typst.app. Typst on-premise with self hosted and Distribute packages and fonts across your organization sounds even more like a custom registry.

Any chance we can find a solution for that in the future? What are our options. Of course the solution should be safe to some extend, but custom registries will always come with some safety issues to some point. But that should not be the reason to kill that idea completely.

[hoary thunder]

There is no intent to lock anybody into anything. The whole package infrastructure was just built on a minimal time budget. We're open to supporting custom registries, but would like to do this the right way (with a properly specified protocol that can be extended to use cases that require auth, etc.).

[fresh hornet]

I see. I guess this is a situation were everyone is not 100% happy with the current solution. You guys did your best and created a good working (besides aws issus) package infrastructure. I am very thankful for that work.

As grateful as I am for the current package infrastructure (I really am), I might be forced at some point to build a script tool (like pip but for typst) by myself. Should I do that, maybe there are some learnings from it that could go in an ofiicial way.

[sleek halo]

like pip but for typst

This already exists, typship and utpm are two tools that support it (not sure if it's released in utpm yet though). It's not a registry, but you can e.g. typship download <git repo> -n preview -c <release-tag> to install from Github. (Unless you mean making a proper registry tool on top of either of these.)
There's also this showcase of mine from a while back—not that I'd seriously recommend it, but it is prior art and I do use it in CI 😉

As for official support, it just takes time.

[fresh hornet]

Oh. Interesting. utpm looks quite interesting. I woll take a loot at it.

[fresh hornet]

Do you happen to know if there is any kind of read documentation for utpm at all? I cant find anything besides a list of commands in the readme...

[crimson sleet]

usually, for a cli, the list of commands / the help command is gonna be the documentation if u will

[fresh hornet]

Yeah. But when it comes to adding dependencies to a config file, I would expect at least an example how that could look like.

[fresh hornet]

@sleek halo you are a contributor to utpm right? While trying it out, it always crashed. On every command I tried. Before I dig into it, I wonder: Do you happen to know, if utpm creates a build kind directory with package storage for every workspace?

[mild carbon]

ps there's #1266702763539304461

(not very active though)

[sleek halo]

tbh, I'm currently betting on typship. The docs are also not great (thus e.g. #32) but I think it does what I need for the package template and my "distribute private packages" needs.

(But if there's more, let's definitely move to the other thread)

[placid sky]

Hey 😠 (you are right 😔)

Like every year I don't have time unfortunately :((

And there aren't many contributors

[placid sky]

I'll answer tomorrow I'm a bit drunk (sorry) 😭

I have plans for it but like I said: no time

[mild carbon]

Story of my life

[fresh hornet]

Is it okay for the pople that host packages.typst.org if other/alternative typst clients download from there?

[main thicket]

This already happens and as long as your clients don't ship with a DOS built in it probably fine

[floral crater]

otherwise you can target / mirror https://github.com/typst/packages

[fresh hornet]

They definitively wont do that. But I will make sure to use the downloads as less as possible.

[fresh hornet]

I dont plan to self host a repo so that is not a viable option for me.

[floral crater]

"otherwise" as in if you would be generating lots of traffic 😅
but since thats not the case your initial idea should be fine

[sleek halo]

I think it is also good etiquette to use a user agent that identifies your tool, so that you can be reached if anything is a problem.

[fresh hornet]

That is a good hint!

[hoary thunder]

With an identifiable user agent, it's fine with us!

[fresh hornet]

Alright! Thanks for the confirmation. I will make sure to do that.

[fresh hornet]

I am working on a package manager prototype for advanced package management. I am currently working on the config format. I would love to get some feedback (any kind) on the design I have so far.

The following config adds some dependencies to the project (including official ones and git ones), patching/overwriding one dependency and applying a patch to another one.

[dependency.cetz]
"1.1.1" = { git = "URL", rev = "asdasd" }
"1.1.3" = {official = "cetz", version = "1.2.3", apply-edit = "local/fix.patch"}

[dependency.cetz-plot]
"1.1.1" = {official = "cetz-plot", version = "1.1.1"}

[dependency.algorithm]
"1.1.1" = {official = "algorithm", version = "1.1.1"}
"1.1.3" = {official = "algorithm", version = "1.2.3"}

[[patch]]
old = { official = "cetz", version = "1.1.1" }
new = { git = "URL", rev = "asdasd" }
target = { official = "cetz-plot", version = "1.1.1" }

[main thicket]

is this for typst proper or is this going to be a third party project, if it's the latter I would like to know if you considered looking at either utpm or typship to contribute this instead of creating another competing project.

To answer the actual question, I'm not entirely sure what to make of the patch format, the one at the bnottom and the one apploed at the topic is presumably not the same?

[fresh hornet]

This is intended to be a third party project. I have looked at both projects. Typship seems to have a completely different approach. utpm looked like it could be something similar. But when I tested, it was super buggy and crashed with panics on mostly all commands that I tried. Furthermore there are pretty much no docs.
Besides, as far as I know, my approach is a bit different from the base. I indent to have a local build dir for each project which enables proper patching/overwriding of dependencies. The things I have in mind would likely not be a small addition, but more of a rewrite of the base. I do not know if my ideas even work or if I can contribute to utpm on a skill level. That is why I decided initially not to contribute to utpm.

[fresh hornet]

Not really. The first "apply-patch" applies a git like patch file to a dependency. The block at the bottom replaces a dependency with another one (most likely another version of it).

[main thicket]

I see, how exactly will this work from a user perspective? Surely they can't just run normal regular typst if a patch is to be applied

or will they simply set the package dir?

[fresh hornet]

The idea is to have a custom package dir for each project. Since I use and most other people that I know use vs code, I can simple place a s code config file in the project and no manual config is needed.

[placid sky]

The last PR is covering all problems you mentionned (on utpm)
Try to use the dev-install branch if you want to take a look, I need to review some comments made on before merging it

But I get why you didn't want to contribute it directly

[fresh hornet]

I skimmed the PR, but could not find anything about dependency patching or a project wise package dir. Could you give me a hint where I need to look for that?

[placid sky]

It doesn't talk about patching a project, I was talking about bugs and docs 😔

[fresh hornet]

Oh. Alright. Great to see those! But sadly not what I need in the end 😔 .

[sleek halo]

Did you see @deep pewter's typship#29? It's an issue proposing something similar (at least it seems so, superficially). Personally, I'm looking for something a bit different (not managing dependencies for individual projects/documents, but package registries), so if you think this fits your vision, also taking typship#34 for inspiration would be nice 🙂

(I also would love if there were more synergies between the projects in this area, but I get that right now we're still more in an exploration phase, where the directions of the tools just differ too much)

[fresh hornet]

I did not. Thanks for pointing that out. It looks a bit like what I want. Although I think it would be hard to integrate the patching aspect into it, since it introduces a lot of difficulties and seems a bit out of scope at this point.

[sleek halo]

yeah, it was more meant as extra inspiration than a nudge to contribute 🙂

[fresh hornet]

When I find the time to finish my prototype, I will share the ideas from it (incl. the code) with you and we might finde a common thing that we can merge all together.

[deep pewter]

patches can be run once before a tarball (npm, github, typst universe) is extracted to the fs.

[fresh hornet]

Sure. But what do you do if two packages depend on cetz:4.2.0 and you want to patch the cetz version for one of them? That makes it difficult. You need to have two version of cetzt:4.2.0 (one patched and one original). Now you need to patch the package that depends on it too, because it can not keep its name. This issue can occur recursive. My idea was to save the modified packages as a new package (with edited non duplicate name) and automatically edit the packages that should use this one.

Not sure it is clear what I mean.

[deep pewter]

typship#34 looks interesting. We may have a typship-index repository like crates.io-index, https://github.com/rust-lang/crates.io-index. but this might look like some feature supported and maintained by official typst, an index maintained by official universe.

[deep pewter]

I think you can use some tricks like alias, to create a patched cetz (in workspace-wide). like that:

[dependencies]
cetz = "0.3.0"
cetz-patched = { version = "0.3.0", patch = "local/fix.patch" }

[fresh hornet]

But how to handle patches/overwrides for dependencies. What if I want to change the cetz version that cetz-plot is using? What if I use also zap and do not want to overwrite that cetz version?

[deep pewter]

then we also need a patched cetz-plot? My idea are all learned and borrowed from cargo, https://stackoverflow.com/questions/66289830/import-rust-package-with-alias-in-toml.

[fresh hornet]

Yes. We need to create a duplicate of cetz with another name and edit cetz-plot to use that one.

[deep pewter]

Patching sounds a bit risky in typst tho. If "cetz" and "cetz-patched" both have a state named "some-internal-state", they actually share the state during typst evaluation. only package owners know whether state can be shared safely.

[fresh hornet]

Ohhhhhh noooo. I did not think about that.

[deep pewter]

So we are lucky if such "runtime conflict" won't happen. so in principle we should PR cetz to fix some bugs ovo.

[fresh hornet]

The states itself can be renamed. But I forgot about imports (and states) where the name of the file is not hard coded. I can not replace those too.

When I think about it. The sates are not the issue. For patching we could simple add some uniq value infront of each state def for each dependency. But imports that are expressions break patching. We cant just add something to them and it works.

[fresh hornet]

But it seems that is a minimal issue. Only one package in the univese seems to not hard code it.

[main thicket]

I generally recommend that internal states for packages are keyed by their version, name and some garbage to ensure it doesn't conflict with users code

[placid sky]

As utpm is downloading packages, is UTPM/{version} fine? Or do you want something specific?

[main thicket]

That should be fine, I use something similar for tytanic

I just copied what typst uses

[hoary thunder]

That works

[kindred hollow]

Over the Christmas break I ended up writing up some thoughts and suggestions around the design of dependencies in Typst that hopefully addresses the security concerns in https://github.com/typst/typst/pull/5579#issuecomment-2545034781 while also expanding on the idea of typst.toml being used for larger Typst projects as opposed to just packages. I thought I would add it here as I hope it could be a useful starting point for a design discussion around the implementation of non-Universe packages when the time for that comes.

Though I am a keen and long-time user of Typst, I am a bit of a tourist in the Typst development community, so my apologises if this is not the right forum for such a discussion.

Typst Projects and Dependencies Design Discussion: https://typst.app/project/RO6NOpDmMvjVoYDk2WAXyq

[sleek halo]

I don't have enough time to go into as much depth as you have, but I'm also interested in this topic (from the "using packages not on Universe" angle, less the "declaring dependencies" angle) and have some comments.

First, I'm a fan of trying things out in "user space" where possible. That doesn't work for the @dep syntax, but infrastructure around the [dependencies] table in typst.toml could be added as [tool.dependencies] for third party tools to pick up.

(a problem with this is that the typst-kit crate for parsing typst.toml files requires the package section; removing that requirement would make tool sections for documents/projects more feasible, but that's a simple change)

Related are these feature requests on the typship third party package manager: typship#29, typship#34

A big part of what you want is probably the convenience of not having to use third party tools, but the design space here is very big (just compare your proposal and the two issues—all very different), so I think it's more realistic and also the safer approach for Typst to see first try one or two approaches outside the compiler and learn from them. Web app integration is not possible this way, of course.

Second, some of what you want can be done by adopting conventions/patterns instead of adding features to Typst. Instead of [dependencies] in typst.toml, a file like deps.typ can also be used to centralize dependencies. This gives reuse of dependencies between different entry points, and to some extent makes imports static (at least as static as what you describe in 3.d.v).

Similarly, the [project] table could be emulated by a project.typ file that contains some variable definitions. Imo thinking about possibilities like this encourages separating the different aspects in the proposal: What is the actual core that can't be achieved using conventions, and has to go into the compiler (or a tool)? To me, the answer is resolving non-Universe packages and enabling dependency overrides, so personally I would look at whether there is a design that can achieve these (together or separate) without touching all the other aspects as well.

[kindred hollow]

Thank you for taking a look!

I agree that I was probably a bit overly enthusiastic including suggestions such as the [project] table in the proposal. These are really intended to be seen as potential additional suggestions that are separate from the main topic, but are included to sound out whether the proposal fits organically within Typst — [project] is an entirely optional part of the fleshing out of typst.toml such that it goes beyond just packages. This ties in with the distinction between documents and projects in the proposal.

I agree that this table in particular is easily replaced with the existing scripting functionality of Typst and should only be included if would be considered idiomatic, in which case its inclusion could help guide users towards an idiomatically correct/preferred setup.

I also use the deps.typ trick today, but to me this seems as more of a workaround due to the limitations of the current implementation. Similarly, while I think projects such as typship and utpm do a great job exploring the design space, I would also argue that a first-party implementation would be very valuable, and probably necessary for use in the WebApp. I don't think the discussion around a first-party implementation needs to wait for 3rd party tools to be further developed. One of the best features of Typst is that it ships in a single binary and is easy to embed and install — that all goes away if important features such as non-Universe packages are relegated to third party tools.

The idea of resolving dependencies statically, either entirely or partially, I think could potentially be a very good design choice for Typst and would alleviate the concern around data exfiltration, which exists in principle even for Universe packages today.

If time permits, I will have another go at the proposal and reduce the emphasis on the [project] table, which I agree detracts from the discussion at the moment.

[main thicket]

Do you guys think we could resolve this? https://github.com/typst/packages/issues/586

I think, just like they said in their comment, adding the field to the manifest is backwards compatible assuming noone has added weird non-standard fields to their manifests. I looked at the package-check and it seems to only check known fields, not reject unknown fields though, bit of a bummer if that bites us.

How do we feel about a separate display-name field? Should this be added, there's some merit to it, considering that the target audience of Typst aren't necessarily programmers, but I also feel like ther will ahrdly be a difference bewteen the display name and the identifier name.

90% of the time it'll be spaces instead of dashes and title case

[sleek halo]

I'd be in favor; I specify a kinda display name as the title in my PDF manuals, and that field would reduce manual adjustments. The use case in the latest comment (before tinger's) resonates in a similar manner.

[cerulean tangle]

Is it planned to let packages use Typst for their README at some point (not necessarily in the next release)?

[main thicket]

I think that's a logical step once package documentation is also typst based

[cerulean tangle]

That makes sense, especially because then both the README and the documentation would probably be the same thing

[main thicket]

Yeah I reckon there would be a dedicated way to host a docs page with reference and examples separate from the github readme

[cerulean tangle]

One of the issues I have with the current README.md for packages is that many packages use the same README for their GitHub repo as for the package itself, meaning the Typst Universe page ends up describing what Typst is and how to build the package, which most users don't care about.
So anything that prevents authors from being able to reuse the same file is an improvement imo

[main thicket]

Yes but that's not easily avoided because the universe page doesn't show anything other than the readme AFAIK

[mild carbon]

I would also hope there's a way to compile the manual for a package from the cli

[cerulean tangle]

My hope is that one day (maybe when we move out of the preview namespace), Typst Universe no longer considers README.md files and instead displays the package documentation, which could have a "main" page that would serve the same purpose as a well-made README today

[main thicket]

Yes that's what I mean

I believe this will happen even before the non preview namespace

I think this is the logical conclusion to the typst docs themselves being selfhosted already