How to filter JSON Data based on schema (with complete type checking)

[digital thistle]

How do I filter

const data = {
    id: 1,
    title: "Hello, world!",
    body: "This is a test post.",
    createdAt: new Date(),
    updatedAt: new Date(),
    visible: true
};

with schema

const schema = {
    id: true,
    title: true,
    body: true,
    createdAt: true,
    updatedAt: false,
    visible: false
};

and still get full type checking and correct filtered data based on the schema

so end result of the filteredData type is

{
    id: string,
    title: string,
    body: string,
    createdAt: Date,
}

(with the visible and UpdatedAt gone)

[visual crown]

xibread#0

Playground Link

Preview:```ts
const data = {
id: 1,
title: "Hello, world!",
body: "This is a test post.",
createdAt: new Date(),
updatedAt: new Date(),
visible: true,
}

const schema = {
id: true,
title: true,
body: true,
createdAt: true,
updatedAt: fals
...```

You can choose specific lines to embed by selecting them before copying the link.

[digital thistle]

Thank you so much, will be trying once I get back home

[tropic echo]

Is this related to a MongoDB query?

[tropic echo]

https://www.mongodb.com/docs/manual/tutorial/project-fields-from-query-results/#return-all-but-the-excluded-fields

[digital thistle]

Its postgres, I can do something similar with select but I want to grab and use all the data in server, and filter it through a function before passing the public datas to the api

May I know how I can do that through a filter function?

I cant seem to get the types working if its through a function

_ _
Code:

export type Schema = {
    [K in keyof any]: boolean;
}

type Filter<T, U> = Omit<T, Extract<keyof U, { [K in keyof U]: U[K] extends false ? K : never }[keyof U]>>
export function filterData<T>({ data, schema }: { data: T, schema: Schema }): Filter<T, typeof schema> {
    const filteredData = {}
    for (const key in schema) {
        if (schema[key] === true && key in data) {
            filteredData[key] = data[key];
        }
    }
    return filteredData as Filter<T, typeof schema>;
}

const data = {
    id: 1,
    title: "Hello, world!",
    body: "This is a test post.",
    created_at: new Date(),
    userId: '',
    visible: true
};
const schema = {
    id: true,
    title: true,
    body: true,
    created_at: true,
    userId: true,
    visible: false
} as const
const filteredData = filterData<typeof data>({ data, schema: schema });
console.log(filteredData);

filteredData type:

const filteredData: Filter<{
    id: number;
    title: string;
    body: string;
    created_at: Date;
    userId: string;
    visible: boolean;
}, Schema>

here is what I tried but with some errors and types not working (still showing visible as an available option even though it filters correctly in runtime)

[visual crown]

@digital thistle Here's a shortened URL of your playground link! You can remove the full link from your message.

relrelway#0

Playground Link

Preview:```ts
type Schema = {
[K in keyof any]: boolean
}

type Filter<T, U> = Omit<
T,
Extract<
keyof U,
{
[K in keyof U]: U[K] extends false ? K : never
}[keyof U]

function filterData<T>({
data,
schema,
}: {
data: T
schema: Schema
}): Filter<T, typeof schema> {
c
...```

You can choose specific lines to embed by selecting them before copying the link.

[digital thistle]

heres the playground link above^

[visual crown]

xibread#0

Playground Link

Preview:```ts
type Schema = {
[K in keyof any]: boolean;
}

type Filter<T, U> = Omit<T, Extract<keyof U, { [K in keyof U]: U[K] extends false ? K : never }[keyof U]>>
function filterData<T, U>({ data, schema }: { data: T, schema: U }): Filter<T, U> {
const filteredData = {}
...```

You can choose specific lines to embed by selecting them before copying the link.

[digital thistle]

Will try once I get back - thank you

[tropic echo]

Personally I'd rather use a library like lodash.

https://lodash.com/docs/4.17.15#pick

[digital thistle]

oh never knew that existed 😭

i'll take a look, thank you!

[heavy bramble]

https://mobily.github.io/ts-belt/api/object#selectkeys

https://ramdajs.com/docs/#pick

https://selfrefactor.github.io/rambda/#/?id=pick

https://remedajs.com/docs#pick

[digital thistle]

How can I possibly use this with the schema?

[tropic echo]

Do you have to use that 'schema'?

[digital thistle]

I use that schema to select wether the data should be included given to the api

How should be the schema be then?

[heavy bramble]

What's wrong with just a list of strings of the keys you want to keep?

[tropic echo]

The lodash .pick function just expects and array of strings corresponding to the properties you want to keep.

[digital thistle]

How do I make an array type-safe that are pulled from this type

[heavy bramble]

const keysToKeep = ["id", "name"] as const satisfies (keyof YourType)[]

But the pick function already handles that for you if you pass in an array directly

So you can just do _.pick(obj, ["id", "name"]) and it'll give a type error if the keys don't match what's in the obj

[digital thistle]

Ah okay makes sense, Thank you so much - will be implementing this now and see how it goes

[tropic echo]

This is my attempt to make the thing work, as you had it originally. It can be simplified probably, but you might be able to use it to learn something. I'd still recommend using a library like lodash though, it's just easier and more correct.

[visual crown]

slashrug#0

Playground Link

Preview:```ts
type Schema = Record<string, boolean>

type Filter<T extends {}, U extends Schema> = {
[P in keyof T]: P extends string
? U[P] extends true
? T[P]
: never
: never
}

function filterData<
T extends Record<string, unknown>,
S extends Schema

(data: T
...```

You can choose specific lines to embed by selecting them before copying the link.

[digital thistle]

Thank you! I'm definetely considering using a library

The ts_belt equivalant would be https://mobily.github.io/ts-belt/api/object#get? Or which one is it - Since I saw the benchmark and saw that ts_belt is way quicker than lodash

[heavy bramble]

Nah the equivalent function in ts-belt is selectKeys

I linked it above

[digital thistle]

ah okay sorry got it

Thank you!

I should only filter using the function when I need to use it right? If so then I might wanna make the array as a constant so I can use it anywhere in my codebase. How do I do it while still getting type-safety?

[heavy bramble]

But yeah there's a bunch of libraries like ts-belt/lodash/ramda/rambda/remeda, all with slight differences in performance, typescript compatibility, and range of utilities

[tropic echo]

I'm not sure ts-belt is still maintained or maintained well enough tbh.

[heavy bramble]

Just use the as const satisfies (keyof Type)[] I showed above if you need that

[tropic echo]

lodash has been around for ages and known to be fast and solid hence my recommendation.

[digital thistle]

Alright will take all the recommendations, thank you!

[heavy bramble]

I've found lodash's typescript support more lacking than the other options personally

But it's down to preference for the most part anyway

[digital thistle]

I'll use ts-belt for now since its faster and i'm sure not much maintenance is needed for a general library like it, but thank you so much all^^

[tropic echo]

Ts-belt has a very low star count on Github and a steadily decreasing download count on npm, and it was last updated 7 months ago. All these sings are telling me it was a good attempt, but it was just not going to be adopted at a wide scale, and it's now being abandoned. Not a good idea adopt libraries like this in a new project, objectively.

[heavy bramble]

I'm not vouching for ts-belt specifically

I prefer the ramda derivatives myself

[tropic echo]

Me too 🙂 Just wish it was faster.

Wdym derivates?

[heavy bramble]

Like rambda/remeda

[digital thistle]

Should performance be in account for just filtering small items?

[tropic echo]

Nah I'm just overthinking it so next time you choose a lib you know some more things to consider.

[digital thistle]

Fair enough, thank you!^

[heavy bramble]

Also mb you need a readonly in there too

[visual crown]

littlelily#0

Playground Link

Preview:```ts
type Post = {
id: number
name: string
description: string
}

const keys = ["id", "name"] as const satisfies readonly (keyof Post)[]```

You can choose specific lines to embed by selecting them before copying the link.

[digital thistle]

still the same error but I suppose this works too?

export const PostsPublicSelect:(keyof Posts)[] = [
    'body', 'created_at', 'id', 'title', 'userId'
]

[heavy bramble]

Nah that overrides the type of the array

Probably not a great idea in this case

[digital thistle]

Ah how do I fix it then? Apologies for asking much, not a typescript advanced person

[tropic echo]

Why not?

[heavy bramble]

Type inference on the returned object from pick

A const array will let the function infer what the output object looks like exactly

[heavy bramble]

Also it should just work, I'm not sure what's going on with your ide

Check out the playground, that works fine

Or is Post not a type but a class instead in your code?

[digital thistle]

error starting from satisfies onwards

[heavy bramble]

What does Posts refer to?

[cyan surge]

@relrel I know it's not directly a answer to your question but if it's an option for your case, have a look at the zod library. It'll allow to do exactly what you want:

const schema = z.object({
  id: z.string(),
  name: z.string(),
  age: z.number().optional().default(10)
})

const typedObject = z.parse(untypedObjectYouGetFromDB)

// from here on, typedObject is fully typed with IntelliSense and linter support etc.

You also get your TS types out of the box to avoid double-definitions: type TypedObject = z.output<schema>

[heavy bramble]

Can you copy that line along with the definition of Posts into a playground?

[digital thistle]

Seems like its not what I am looking for 😿

[cyan surge]

and well, picking and omitting is also there: const partialSchema = schema.pick({ id: true, age: false})

[digital thistle]

just looking to filter

not to use zod (I use zod for other purposes dw)

[digital thistle]

Sure gimme one sec

[cyan surge]

Ah ok, then ignore what I said 😄

[digital thistle]

No worries! Thank you^

[digital thistle]

Okay uh so i'm not really sure on how to copy the definitions of Posts into the playground since i'm pulling Posts from Prisma

But i'm sure its not the type thats causing the error since the comma error starts from satisfies onwards

[heavy bramble]

Hmm

There's nothing wrong with what you wrote that I can see

[digital thistle]

works in the playground, but not in my ide

let me take a look rq, apologies

[heavy bramble]

Restarting vscode might help, idk

[digital thistle]

weird eh

[heavy bramble]

Or there's a syntax error on a previous line

[digital thistle]

so right after const it shows an error

[heavy bramble]

Does as const satisfies readonly string[] error?

[digital thistle]

Right so i'm blaming my ide for this since it works on the playground - apologies

[tropic echo]

Could it be an outdated version of TS (with no support for satisfies)?

[heavy bramble]

np, there's not a variable called satisfies defined in that file right?

[digital thistle]

not at all

[digital thistle]

ah right

[heavy bramble]

That's a good point

[digital thistle]

Ok yeah thats definitely it

How do I fix that exactly?

[heavy bramble]

Use 4.9 or above

Preferably just 5.1.6

Change the version in your package.json and npm install

[digital thistle]

gotcha will try

Still the same error

thats weird

[heavy bramble]

Restarted the ide after updating ts?

[digital thistle]

did try restarting ts server but not ide

will try now

Ok that was a bad beginner mistake on my side

works now flawlessly

[heavy bramble]

Ok cool c:

[digital thistle]

Right okay it works now

[heavy bramble]

Didn't realise it's only been a year since 4.9 released, feels like it's been longer than that since satisfies came out

[digital thistle]

Whats the benefit of using

const keys = ["id", 'body'] as const satisfies readonly (keyof Posts)[]

over

export const PostsPublicSelect:(keyof Posts)[] = [
    'body', 'created_at', 'id', 'title', 'userId'
]

exactly?

[tropic echo]

Did you end up with ts-belt or lodash in the end?

[digital thistle]

Havent got that sorted :)

Will do in just a moment

[tropic echo]

Ah, just bc I wanted to try this exact thing.

[tropic echo]

I mean this ^

[heavy bramble]

The benefit of the former is that TS holds onto the information of exactly which keys you provided, so if it's passed to a function like pick, the output type will be more specific and give you an object type with the specific keys you put in the srray

[tropic echo]

But I think it depends on how the library is typed.

[digital thistle]

Ah okay makes sense

[digital thistle]

Which one do ya'll recommend?

I'm sure performance will not be a big issue here but definitely future-proof

[tropic echo]

If this is a serious project I'd go with tried and tested, otherwise won't really matter.

[heavy bramble]

lodash is a good entry into those kind of libraries. ramda and libraries based on it are my preference but maybe a little more advanced I guess (probably still easy enough to use for you)

[digital thistle]

serious or not, i'm not sure. But this project is definitely towards the businesses side so don't want things to slow down anytime soon

[tropic echo]

Yeah if you wanna learn some fp principles while you're at it then ramda et al are good.

[digital thistle]

so ramda?

or lodash

[tropic echo]

Is this a project you're getting paid to deliver? Is it making you money/will it make you money?

[digital thistle]

a small project, might be making it a SaaS thing in the future

[tropic echo]

Or is this a random side project you're doing to learn?

[digital thistle]

i'm not filtering much anyways so performance wise, not much hit

[tropic echo]

Seriously overthinking this tbh imo 😅

[digital thistle]

Ok yeah true let's just go with tsbelt as the 2nd safest option

its gna be easy to migrate to lodash if its ever needed anyways

[heavy bramble]

ts-belt has more difference in naming conventions and such than lodash and ramda do

So probably not as easy to migrate

[digital thistle]

they're both requring the same type of arguments

object, and keys to pick

[heavy bramble]

Well yeah, it's just the same function with a different name

[digital thistle]

mhm

[heavy bramble]

I just mean the names are more similar between lodash and ramda

[digital thistle]

lodash seems safer, ts-belt seems way performant

eh

lets just take lodash then

[heavy bramble]

Sounds good to me

[tropic echo]

@heavy bramble totally right about this ^

Although I tried it with ts-belt

[heavy bramble]

Most of those pick functions use the built in type utility Pick for the return type, so they'll all act the same there

[digital thistle]

I have another concern that would definitely help but its more related to prisma/json/zod - should I make a new thread or just ask it here

[heavy bramble]

and if Pick is just called with Pick<T, keyof T> then you don't get a very useful output

[heavy bramble]

May as well ask here, I can probably answer

[digital thistle]

Thank you so much for helping me with the picking issue^^

[digital thistle]

Alright so I have a JSON field in my database but I would like for it to typed using zod preferably on server-level (doesnt have to be enforced database-side)

what i’m doing currently is omitting that field and replacing it with the typed one but its messy and would love to see an easier approach to this

[heavy bramble]

That sounds fine to me, either that or make a Zod schema for that whole table and pass your returned object from the db through .parse, then the resulting object will have the json field properly typed

[digital thistle]

ok yea theres no easy way to do it sadly

[heavy bramble]

If you just want a schema for that one column though, then I'd write it like this:

const post = await prisma.post.findFirst(...)
const { json, ...rest } = post
const parsedJson = jsonSchema.parse(json)
const parsedPost = { json: parsedJson, ...rest }

[digital thistle]

yea thats a big hassle

more of a messy one

It's fine then

[heavy bramble]

You can abstract it into a function

[digital thistle]

How would one usually work with json in database

[heavy bramble]

You don't really in a clean way unless you ignore validation

But that's not a great option either

[digital thistle]

Now that I think about it, validating a json field is also kinda smart to ensure its shape

Alright i'll figure this out later

I'll try work on the lodash pick first for now :)

Thank you so much! Appreciate the help

!resolve

[digital thistle]

Sorry to reopen, but how do I make a filterData function

export function filterData(data:Array<unknown>, schema) {
    return data.map((data) => pick(data, schema));
}

with the correct return type

!reopen

schema being:

const schema = [
    "id",
    'body',
    'created_at',
    "title",
    "userId"
] as const satisfies readonly (keyof Posts)[]

[tropic echo]

I guess you'd use a generic.

[heavy bramble]

depends how far you want to go with the types. I've given two options in this playground, a basic version, and an enhanced version. the basic is much easier to write, but gives less specific output, whereas the enhanced version gives much better type output (try hovering over each result and youll see)

[visual crown]

littlelily#0

Playground Link

Preview:```ts
import { pick } from "lodash"

type Posts = {
id: string
created_at: Date
userId: string | null
title: string
body: string
visible: boolean
}

export const filterDataBasic = <TObj extends object, TKey extends keyof TObj>(data: readonly TObj[], schema: readonly TKey[]) =>
...```

You can choose specific lines to embed by selecting them before copying the link.