PowerShell and RoboCopy

[viscid pulsar]

Hello, Just moving a question from #powershell-help to here.

I have been given a PowerShell script that takes in a list of Usernames, and hostnames of Primary and secondary machines for that user.

It will run through and copy config files from the primary to secondary machines, usually for bespoke software we have that stores config in the software folder rather than appdata etc...

however the script seems to run serially (one machine then onto the next) as it runs all night into the morning.

Also some machines the script fails at (no error, not even sure if robocopy is throwing fault to be honest, always found robocopy to be solid!) some machines.

thing is if I remove the machine or move to end of list, then it's fine! (this rarely happens, struggling to find cause, although AV picking up a psexec or some recently banned app may be at play, some of these machines belong to devs etc...)

below is an example of what is happening (paths have been changed as don't want spanked by work)

there is a hashtable like below (pointed out elsewhere we have a /E and a /Lev:1 command that doesn't work.

$Jobs = @(

                
                @{
                    Source      = "\\$MachineName\c$\Program Files\appyMcAppFace\Data";
                    Destination = "\\$FlippedMachineName\c$\Program Files\appyMcAppFace\Data";
                    File        = "appy.init";
                    Options     = @("/E", "/XO", "/R:0", "/lev:1", "/log+:$RobocopyLogFilePath\$FlippedMachineName.txt")
                },

that then gets fed to this foreach loop

foreach ($Job in $Jobs) {
                $Source = $Job.Source
                $Destination = $Job.Destination
                $File = $Job.File
                $Options = $Job.Options

                if (Test-Path -Path $Source) {
                    robocopy $Options $Source $Destination $File
                }
                else {
                    Write-Log -Level "[INFO]" -Message "$Source does not exist" -Log $OutputLogFilePath
                }
            }

there are also some if statements for other config files, with different options

if (Test-Path "\\$MachineName\c$\Program Files (x86)\oldApp\X\Config") {
                robocopy /MIR /XO /r:0 /log+:"$RobocopyLogFilePath\$FlippedMachineName.txt" /lev:1 "\\$MachineName\c$\Program Files (x86)\oldApp\X\Config" "\\$FlippedMachineName\c$\Program Files (x86)\oldApp\X\Config" "OLDAPP.configuration-file", "Configuration.xml"

}

just wondering what could be causing the stop (or if its related to robocopy)

but the main thing is to make this faster, as it could be still running when users start at 9 AM the next morning.

Any help appreciated, anything no clear, please ask (I know I can be clear as mud 😀)

[grim brook]

so yeah if it is multiple machines, then yeah you could spin up multiple jobs. only the one robocopy execution itself is multi-threaded.

[viscid pulsar]

Thanks! yeah been looking into the -MT option.. definitely a few things to play with 🙂, I found this in which the main answer is similar to what you have suggested above (except they say runspace rather than background-job, I have played with that for something else with config mgr, spun up several runspaces running scripts, on a server my colleague was using BES on and the server ran out of mem and the scripts failed.. (dunno what happened to BES..) https://learn.microsoft.com/en-us/answers/questions/1360875/whats-the-fastest-way-to-copy-large-folders-from-o

[grim brook]

Runspaces would be more efficient most likely as starting a job starts a brand new powershell process. Runspaces run inside the powershell process. The job api is simpler most likely though. How many machines are we talking? And how many GBs per machine approximately?

[viscid pulsar]

About 300 machines, most jobs are small config so couple of mb at most, but there a few Devs etc... that can sometimes have GBs worth of data

[grim brook]

yeah so i can see having a [runspacepool] for that to speed it up.

[viscid pulsar]

Hmm.. deffo somethings to play with, thanks 🙏

[grim brook]

are these citrix machines? if so why not use the profile feature of WEM?

[viscid pulsar]

They are mostly nutanix VMs and we use Citrix VDA to connect to them, as for why this approach was chosen over others, not too sure...

Tho WEM is added to the list now, at least I can find out if it was considered (we did have Appsense once but that is gone, it would have worked!)

[dim sonnet]

Not sure if it's robocopy erroring
There's extra details you can enable in the log files. I'd use a different filename for each runspace, to prevent overwriting the same file
https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy#logging-options

(pointed out elsewhere we have a /E and a /Lev:1 command that doesn't work.
When File isn't a folder or wildcard, but a filename like "appy.init" -- I don't think those options would apply

I just noticed this order

robocopy $Options $Source $Destination $File

The docs use this order

robocopy <source> <destination> [<file>[ ...]] [<options>]
# so try
robocopy $Source $Destination $file $Options

I don't remember if your array named $Options auto expands, or if you have to splat it like this

robocopy $Source $Destination $file @Options

Oh you can test the return code for errors: https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy#exit-return-codes