#API | Export / Import data

1 messages · Page 1 of 1 (latest)

north anchor
#

Trying to export ALL data and customization from old upgraded (prev. versions) instance to a clean v1.14 instance (both self-hosted Docker).

Goal is to export Everything (objects and workspace customation) possible.

I have the API keys generated and was able to launch "Playground" and also downloaded "Bruno" API client app and was able to connect to both instances.

Is there some documentation I can follow to bring all objects and data properly using API?
If CSV method is used, the data-model has to be created beforehand and I rather bring all that using API rather than mess up while recreating it...
Thank you!

whole hull
#

Not entirely sure about the customization part like views, roles, favorites, etc.

#

One thing in mind is that IDs will change so you can't really rely on them but on fields with unique values

north anchor
north anchor
whole hull
north anchor
#

Basically, I am trying to copy all the custom fields and their data from one instance to another. Obviously, source and target UUIDs won't match on same objects and I understand that. So, how to move a custom field using API? If you would elaborate the steps, that would help tremendously!

whole hull
#

Ah, so recreate

#

Let me check

whole hull
#

If you want to do it programically, I'd do it this way:

  • read all objects using /objects from REST metadata API from both instances and compare them
  • parse JSONs and remove all non-modifiable objects (list is below), then standard fields from all objects from both JSONs (ID, Position, Search vector, Note Targets, Task targets, Timeline Activities, Attachments, Favorites, Name, Creation date, Created by, Deleted at and Last update) as they're created by default when object is created
  • read all objects from first instance one by one, check if it exists in second instance, if not create it (either save responses in other variable or append to variable containing all objects from second instance to have object ID needed to create relation fields)
  • then read object if so, check what fields it has, recreate missing fields in said object from one instance to another using REST API (or GraphQL API for relations)
  • once all objects are recreated properly, use GET /{{objectNamePlural}} REST API to get all records data in batches of 200 records from first object and save it to some variable, check if there's nextCursor, if so, add it to URL and repeat it until nextCursor is null
  • when all records are retrieved from one instance, prepare a POST /{{objectNamePlural}} REST API request to another instance in batches of 20 to speed the process and not overwhelm the server

One problem I see is restoring relations between objects as record ids are different so probably a different approach would be needed in last step, especially if there are few objects related to each other and I'm not entirely sure how to do it in generic way

#

Also, recreating relations in notes and tasks should be left as last as they need a different approach since they have noteTargets and taskTargets

#

List of non-modifiable objects
[
'messageParticipants',
'blocklists',
'favoriteFolders',
'calendarEvents',
'attachments',
'workflowRuns',
'workflowAutomatedTriggers',
'messages',
'timelineActivities',
'workflows',
'calendarChannels',
'calendarChannelEventAssociations',
'viewFilters',
'viewFields',
'calendarEventParticipants',
'messageChannels',
'workspaceMembers',
'messageFolders',
'connectedAccounts',
'viewFilterGroups',
'noteTargets',
'views',
'workflowVersions',
'taskTargets',
'viewSorts',
'messageThreads',
'favorites',
'messageChannelMessageAssociations',
'viewGroups',
]

#

Example REST POST API /object request body

{
  "nameSingular": "pet",
  "namePlural": "pets",
  "labelSingular": "Pet",
  "labelPlural": "Pets",
  "description": "test",
  "icon": null
}
#

Example REST POST API /fields request body

{
  "type": "TEXT",
  "objectMetadataId": "836903fe-31f8-49d2-957a-61a1c28c8e2e",
  "name": "test",
  "label": "Test",
  "description": "",
  "icon": "",
  "defaultValue": null,
  "isNullable": true
}
#

Example GraphQL metadata API request body to create a relation field

mutation CreateOneFieldMetadataItem($input: CreateOneFieldMetadataInput!) {
  createOneField(input: $input) {
    id
    type
    name
    label
    description
    icon
    isCustom
    isActive
    isUnique
    isNullable
    createdAt
    updatedAt
    settings
    defaultValue
    options
    isLabelSyncedWithName
    object {
      id
      __typename
    }
    __typename
  }
}

const variables = {
    input: {
      field: {
        description: 'Shows the person behind newest update',
        icon: 'IconRelationOneToMany',
        label: 'Updated by',
        name: 'updatedBy',
        isLabelSyncedWithName: true,
        objectMetadataId: `${sourceObjectId}`,
        type: 'RELATION',
        relationCreationPayload: {
          type: 'MANY_TO_ONE',
          targetObjectMetadataId: `${workspaceMemberObjectId}`,
          targetFieldLabel: `Updated by ${objectName}`,
          targetFieldIcon: 'IconUsers',
        },
      },
    },
  };
north anchor
# whole hull If you want to do it programically, I'd do it this way: - read all objects using...

Thank You very much and I sincerely appreciate the detail you have shared. I'll work on implementing this process and see how far it goes.
Also, does the team / project have any plans to provide a tool or automation for such migrations from other systems and from older versions of Twenty (if requried)? Something like a object / field mapping tool that could facilitate export / import (I'm thinking in lines of sql Workbench or similar tool allowing table / field exports...)
Importing using CSV files may be ok for small customiztion where only few drop-downs (and their lists) or few custom fields. If there is a significant customization, a field/object mapping tool would help right? Or, is it the project's goal to offer the software and situations like these beyond basic imports are team's consulting services?

whole hull