Custom content collection loader cache

[coral mica]

How can I create a caching mechanism in my custom content collection loader so subsequent builds don't need to process entries that haven't changed?

[next knot]

Hi @coral mica. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:

---

To implement a caching mechanism in your custom content collection loader so that subsequent builds skip processing unchanged entries, you should use the digest feature of Astro’s Content Loader API. Here’s how you can do it:

1. Generate a digest for each entry:
   When processing your data, compute a digest (a string that changes when the data changes, e.g., a hash of the entry’s content).

2. Set the digest when storing entries:
   When you call store.set() to add an entry, include the digest property. Astro will only update the entry if the digest is different from the previous build. If the digest matches, the entry is not re-processed, enabling efficient caching between builds.

3. Example usage:

for (const item of feed.items) {
  const data = await parseData({
    id: item.guid,
    data: item,
  });
  const digest = generateDigest(data); // Implement your own digest function
  store.set({
    id,
    data,
    rendered: {
      html: data.description ?? "",
    },
    digest,
  });
}

The format of the digest is up to the loader, but it must be a string that changes when the data changes. This can be done with the generateDigest function. The entry will only update if the digest does not match an existing entry with the same ID.
Astro Content Loader API: DataEntry
Astro Content Loader API: DataStore

Summary:
By using the digest property in your custom loader, Astro will automatically cache entries and only re-process those that have changed, making your builds faster and more efficient.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@coral mica Kapa.ai is still learning and improving, please let me know how I did by reacting below.

[coral mica]

@next knot The problem in my case is that the parseData function is too expensive, so I want to check the cache before calling it.

[next knot]

Let me check the knowledge base for you again. One moment! :books:

---

I'm sorry, but the knowledge sources do not provide a way to check the cache or retrieve the previous digest before calling the parseData function in a custom content collection loader. According to the documentation, the store.get() method allows you to retrieve an entry by its ID, which includes the previously stored data and digest. However, the recommended workflow is to use parseData before storing the entry, and there is no explicit mention of accessing the previous digest before parsing the data.

Here’s what is documented:

• You can use store.get("my-entry") to retrieve an existing entry, which will include its digest if it exists.
• The digest is used by store.set() to determine if an entry should be updated, but the documentation does not describe a workflow for skipping parseData based on the previous digest, since parseData is intended to validate and parse the data before storage.

If your data source provides a way to determine if an entry has changed (e.g., a lastModified timestamp or an ETag), you could potentially use that information before calling parseData, but this approach is not described in the official documentation.

Summary:
The knowledge sources do not contain enough information to describe a supported way to check the cache or previous digest before calling parseData in a custom loader. You may retrieve previous entries with store.get(), but the intended workflow expects parseData to be called before storing or comparing digests, and there is no documented method to avoid this step based on cache state alone source.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: