#VPS hosting external sync

12 messages · Page 1 of 1 (latest)

faint micaBOT
#

Hi @gritty hazel. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:

#

Convex can be self-hosted on your own infrastructure, including a VPS environment managed by tools like Proxmox VE and Pterodactyl. The self-hosted setup is straightforward: you can run Convex using Docker Compose, and it exposes the necessary ports for API, HTTP actions, and dashboard access. You’ll need to handle your own routing, backups, and monitoring, as the self-hosted version is single-node by default and does not come with built-in support for distributed scaling or high availability out of the box. If you need to scale beyond a single node, you would have to modify the backend code yourself, which is non-trivial and not officially documented for production use at scale Self-hosted: Everything You Need to Know, Self-hosted README.

For your use case—where high real-time data throughput is not required, but consistency between services is important—Convex is well-suited. Convex provides a fully serializable consistency model, ensuring strong data consistency and transactional guarantees for your application state. This is particularly useful when you need to keep background tasks and data in sync across services, even if those services are not highly interactive or real-time Robust conflict and consistency handling.

#

When it comes to syncing background tasks with external APIs, Convex supports actions, which are serverless functions designed for interacting with external systems. The recommended pattern is to use triggers or the scheduler to enqueue actions after database changes, allowing you to call external APIs or perform background work. This approach is suitable for integrating with external services (like game server APIs) and ensures that your background tasks are coordinated with your database state. However, actions are not part of the core sync engine and should be kept as small and focused as possible for scalability Scheduler, actions, and the sync engine, Best way to have other systems react to convex data changes.

Summary:

  • Convex can be self-hosted on a VPS, but you are responsible for operational aspects and scaling.
  • It provides strong consistency guarantees, making it suitable for syncing state between services.
  • Background tasks and external API syncs are handled via actions, which can be triggered by database changes.
  • For non-real-time, consistency-focused workloads, Convex is a good fit, but be aware of the single-node limitation unless you are prepared to extend the backend yourself.

If you need high availability or distributed scaling, the hosted Convex Cloud is recommended, but for your described requirements, self-hosting should suffice.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

#

@gritty hazel Kapa.ai is still learning and improving, please let me know how I did by reacting below.

#

Let me check the knowledge base for you again. One moment! :books:

#

I'm sorry, but the knowledge sources do not contain information or code examples specific to dispatching and tracking backup tasks within Proxmox VE, nor do they cover direct integration patterns between Proxmox VE and Convex.

However, I can summarize how Convex generally handles background tasks and external API syncs, which you could adapt for your use case:

  • Background Task Dispatching: In Convex, you would typically use a mutation to record a task (e.g., a backup request) in the database, and then schedule an action to perform the external work (such as calling the Proxmox VE API to start a backup). This pattern allows you to track the status of the task in your Convex database and update it as the action progresses or completes.
  • Tracking Task State: You would store task state (e.g., "pending", "in progress", "completed", "failed") in a Convex table. The action can update this state via mutations as it interacts with Proxmox VE.
  • Resource Polling: For polling server state or resources (CPU, memory, disk), you could schedule periodic actions that fetch this data from Proxmox VE and write it to your Convex database via mutations. This enables your frontend or other systems to query the latest state from Convex.

Best Practice:
Convex documentation recommends keeping actions small and using them only for the non-deterministic parts (like external API calls), while storing and tracking all important state in the Convex database for consistency and observability. This approach is scalable and allows you to monitor progress and results from your app or dashboard The scheduler, actions, and the sync engine, Zen of Convex.

#

Example Pattern (Pseudocode):

// convex/tasks.ts
import { mutation, internalAction } from "./_generated/server";
import { v } from "convex/values";

// Mutation to create a backup task
export const requestBackup = mutation({
  args: { serverId: v.string() },
  handler: async (ctx, { serverId }) => {
    const taskId = await ctx.db.insert("tasks", {
      serverId,
      type: "backup",
      status: "pending",
      createdAt: Date.now(),
    });
    await ctx.scheduler.runAfter(0, internal.tasks.performBackup, { taskId, serverId });
    return taskId;
  },
});

// Action to perform the backup (call Proxmox VE API)
export const performBackup = internalAction({
  args: { taskId: v.id("tasks"), serverId: v.string() },
  handler: async (ctx, { taskId, serverId }) => {
    // Call Proxmox VE API here (pseudo-code)
    // const result = await proxmoxApi.startBackup(serverId);
    // Update task status
    await ctx.runMutation(internal.tasks.updateTaskStatus, {
      taskId,
      status: "completed", // or "failed" based on result
    });
  },
});

// Mutation to update task status
export const updateTaskStatus = mutation({
  args: { taskId: v.id("tasks"), status: v.string() },
  handler: async (ctx, { taskId, status }) => {
    await ctx.db.patch(taskId, { status });
  },
});

This pattern allows you to dispatch, track, and update the status of background tasks in Convex, even though the actual API calls to Proxmox VE would be implemented in the action.

Resource Polling:
You could use a similar scheduled action to periodically poll resource usage from Proxmox VE and update a "server state" table in Convex.

#

Conclusion:
While the knowledge sources do not provide Proxmox-specific code, they do recommend syncing all important task and server state data to the Convex database for consistency and observability, using actions for external API calls and mutations for state updates. This is the recommended approach for handling such cases in Convex The scheduler, actions, and the sync engine, Zen of Convex.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

#

@gritty hazel Kapa.ai is still learning and improving, please let me know how I did by reacting below.

#

Let me check the knowledge base for you again. One moment! :books:

#

The knowledge sources do not contain information specific to Proxmox VE or its UPID-based task tracking. However, they do describe a general pattern for handling long-running external tasks that require polling for completion, which you can adapt for your use case.

Pattern with Convex:

  1. Start the Backup and Store the UPID:
    Use a Convex mutation to initiate the backup via Proxmox VE’s API. Store the returned UPID and set the task status to "pending" in your Convex database.

  2. Schedule a Polling Action:
    Use Convex’s scheduler to run an action that periodically polls the Proxmox VE API for the status of the task using the UPID. This action can reschedule itself (e.g., every few seconds) until the task is complete.

  3. Update Task Status:
    Each time the polling action runs, update the task status in Convex (e.g., "in progress", "completed", "failed") based on the response from Proxmox VE.

Example Pattern (based on knowledge sources):

// Pseudocode based on Convex patterns
export const startBackup = mutation({
  args: { serverId: v.string() },
  handler: async (ctx, { serverId }) => {
    // Call Proxmox VE API to start backup, get UPID
    // const upid = await proxmoxApi.startBackup(serverId);
    const upid = "example-upid"; // placeholder
    const taskId = await ctx.db.insert("tasks", {
      serverId,
      upid,
      status: "pending",
      createdAt: Date.now(),
    });
    await ctx.scheduler.runAfter(0, internal.tasks.pollBackupStatus, { taskId, upid });
    return taskId;
  },
});
#

export const pollBackupStatus = internalAction({
args: { taskId: v.id("tasks"), upid: v.string() },
handler: async (ctx, { taskId, upid }) => {
// Call Proxmox VE API to check status
// const status = await proxmoxApi.checkTaskStatus(upid);
const status = "in progress"; // placeholder
await ctx.runMutation(internal.tasks.updateTaskStatus, { taskId, status });
if (status !== "completed" && status !== "failed") {
// Reschedule polling
await ctx.scheduler.runAfter(5000, internal.tasks.pollBackupStatus, { taskId, upid });
}
},
});

This approach follows the recommended Convex pattern for background jobs that require polling external APIs, ensuring the task state is always tracked in your database and updated as the external process progresses [Background Job Management](<https://stack.convex.dev/background-job-management#tracking-status-in-a-table>), [Zen of Convex](<https://docs.convex.dev/understanding/zen>).

**Summary:**  
- Store the UPID and task state in Convex.
- Use scheduled actions to poll Proxmox VE for task status.
- Update the Convex database as the task progresses.
- Reschedule polling until the task is complete.

This pattern is robust and aligns with Convex’s best practices for integrating with external systems and tracking long-running tasks.
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------