XCP is slow

[analog gale]

Hi, we are verifying xcp to migrate from an NFS server using Nimble Storage to NetApp.

Assumed migration method

1. Synchronize once with xcp copy while the service is running.
   (2. perform a differential sync using xcp sync until the day of the switchover)
2. Stop the service and perform full synchronization using xcp sync
3. check the synchronization status by xcp verify
4. Switch to NetApp

Information on synchronization source directory:
• Total number of files is about 300 million
• Total size is about 40TB
• Many of the files are small in size.
• Directory structure is up to 10 levels.

System configuration is attached

Problems:
• scp copy is very slow, only 100KB/s ~ a few MB/s.
When memory cache is active, it is over 300MB/s.
NFS CPU usage, memory usage, and NW seem to be OK.
Nimble Storage latency is also fine.

• When I perform scp sync, the connection to the NFS server is broken.
Load average of NFS server becomes high (nfsd seems to be running at full capacity).

Question:
• Is there a problem with the expected switching method?
• What is the cause of the above problem in this configuration?

[abstract topaz]

I can't help you on the being slow part, as it was for us as well.
Our migration was over SMB and NFS, about 120TB total and over 11billion files.
but no matter what we used for the copy process it was slow, slow, slow.
I ended up using XCP without the sync option
xcp copy -preserve-atime -v

Keeps the access time, was required for tiering data properly.
and after it was all done
Xcp scan -stats
to compare the source/destination

There was another XCP tool, https://gitlab.com/haim.marko/xcption, but it's closed off and I haven't used it. But it was supposed to be able to run multiple jobs together.
XCPtion is a wrapper utility for NetApp XCP NFS/CIFS file copy/migration utility. Support been also extended to: [robocopy] (CIFS), [cloudsync] (CIFS, NFS, S3, etc) (https://bluexp.netapp.com/cloud-sync-service), rclone (s3, google drive, ondrive and many more) (https://rclone.org/) and ndmpcopy (copy data between netapp volume/sub volume)
XCPtion has the capability to concurrently execute and manage multiple tasks across a cluster of servers in a distributed manner
This is achieved by utilizing Hashi Corp Nomad distributed scheduler.

not sure if this is an option, and it's something I just recently was directed to
https://bluexp.netapp.com/cloud-sync-service

[forest pewter]

Have you verified the networking path between the xcp server and the source and the xcp host and the Netapp?

Have you made the suggested kernel settings/changes on the xcp server?

https://docs.netapp.com/us-en/xcp/xcp-performance-tuning.html

Jumbo frames?

[analog gale]

Thanks for the reply.

Can you give me some info on when you guys made the transition?
Specs of the XCP server, version, number of parallels, how long it took to copy etc.

I also have a feeling that the problem is on the NFS server side.

[analog gale]

Thank you.
Unfortunately, that configuration change has already been implemented.

[abstract topaz]

xcp 1.9.3
server, RHEL 7.9, 32vCPU, 64GB ram
4 of those, all running from different volumes
In total it took 11 days for the first set of files and then 2-3 days each time after up until our cutover, which took about 10 hours