PowerShell Remoting at Scale: Jobs, Per-Host Errors, Files, and the Second Hop

The last lesson ran commands on one or two hosts that were always up. A fleet is messier: some hosts are down, some fail halfway, and files have to travel too. Every example runs in the same client container and Docker lab as the last lesson, so start it again if you removed it.

Background Jobs Across Hosts

-AsJob returns at once with a job you collect later. One of these three hosts doesn’t exist:

# asjob.ps1 - run on three hosts in the background; one of them doesn't exist
$job = Invoke-Command -HostName web1, web2, web9 -UserName ops -AsJob -ScriptBlock {
    "$(hostname) answered"
}
"job type: $($job.GetType().Name), child jobs: $($job.ChildJobs.Count)"

$null = Wait-Job $job
$results = Receive-Job $job -ErrorAction SilentlyContinue -ErrorVariable failed
$results | Sort-Object
$job.ChildJobs | Sort-Object Location | ForEach-Object { "$($_.Location): $($_.State)" }
"errors: $($failed.Count)"
Remove-Job $job

$ pwsh asjob.ps1
job type: PSRemotingJob, child jobs: 3
ssh: Could not resolve hostname web9: Name or service not known
web1 answered
web2 answered
web1: Completed
web2: Completed
web9: Failed
errors: 1

The job is a PSRemotingJob with one child job per host, so web9 failed on its own while web1 and web2 finished. The ssh: line is SSH’s own message, written straight to the console by the ssh process that PowerShell starts, so it appears even with -ErrorAction SilentlyContinue, and it may land in a different place in the output on your machine. PowerShell’s own record of the failure is the one error in $failed. By default PowerShell connects to 32 hosts at a time; -ThrottleLimit changes that number.

Which Host Raised Which Error

The script block below fails on both hosts. The question is which error came from where:

# errors.ps1 - which host raised which error
Invoke-Command -HostName web1, web2 -UserName ops -ScriptBlock {
    if ((hostname) -eq 'web1') { Get-Item /etc/hostname }
    Get-Item /no/such/file
} -ErrorAction SilentlyContinue -ErrorVariable remoteErrors |
    ForEach-Object { "found: $($_.FullName) on $($_.PSComputerName)" }

"$($remoteErrors.Count) errors"
$remoteErrors | Sort-Object { $_.OriginInfo.PSComputerName } | ForEach-Object {
    "$($_.OriginInfo.PSComputerName): $($_.CategoryInfo.Category) $($_.TargetObject)"
}

$ pwsh errors.ps1
found: /etc/hostname on web1
2 errors
web1: ObjectNotFound /no/such/file
web2: ObjectNotFound /no/such/file

Remote errors come back through the normal error stream, so -ErrorVariable collects them like local ones and the rest of the results still arrive. Each one carries OriginInfo.PSComputerName, the host it happened on. On a hundred hosts, that is how you build the list of machines to fix.

Copying Files Through a Session

# copy.ps1 - copy a file to a remote machine and back through one session
Set-Content ./app.conf 'port = 8080'
$s = New-PSSession -HostName web1 -UserName ops

Copy-Item ./app.conf -Destination /home/ops/app.conf -ToSession $s
Invoke-Command -Session $s { "on web1: $(Get-Content /home/ops/app.conf)" }

Invoke-Command -Session $s { Add-Content /home/ops/app.conf 'workers = 4' }
Copy-Item /home/ops/app.conf -Destination ./app.back.conf -FromSession $s
Get-Content ./app.back.conf

Remove-PSSession $s
Remove-Item ./app.conf, ./app.back.conf

$ pwsh copy.ps1
on web1: port = 8080
port = 8080
workers = 4

-ToSession copies from the client to the session’s host, and -FromSession copies back. The paths on the remote side are remote paths, and the transfer goes over the same SSH connection as the commands, so no scp or shared folder is needed. The second copy brought back the workers = 4 line that was added on web1.

Implicit Remoting: a Remote Command That Looks Local

# implicit.ps1 - make a remote function callable as if it were local
$s = New-PSSession -HostName web1 -UserName ops
Invoke-Command -Session $s {
    function Get-Where { "this ran on $(hostname) as $(whoami)" }
}

$module = Import-PSSession -Session $s -CommandName Get-Where -AllowClobber
"imported: $((Get-Command Get-Where).CommandType), module type $($module.ModuleType)"
Get-Where
"but here: $(hostname)"

Remove-PSSession $s

$ pwsh implicit.ps1
imported: Function, module type Script
this ran on web1 as ops
but here: client

Import-PSSession builds a local proxy, a function in a temporary script module, for each command you name. Calling Get-Where on the client runs the real one on web1 and returns its output. This is how you use a module that’s installed on a server without installing it locally. The proxy only works while the session is open.

The Second Hop

web1 is a perfectly good remoting client too. So can it reach web2?

# secondhop.ps1 - from web1, try to reach web2: the second hop
Invoke-Command -HostName web1 -UserName ops -ScriptBlock {
    "on $(hostname), now trying web2..."
    try {
        Invoke-Command -HostName web2 -UserName ops -ScriptBlock { hostname } -ErrorAction Stop
    } catch {
        "second hop failed: $($_.Exception.Message)"
    }
    "plain ssh from $(hostname) says: $(ssh -o BatchMode=yes ops@web2 hostname 2>&1)"
    "keys in ~/.ssh on $(hostname): $(@(Get-ChildItem ~/.ssh -Filter id_*).Count)"
}

$ pwsh secondhop.ps1
on web1, now trying web2...
second hop failed: The SSH client session has ended with error message: The SSH transport process has abruptly terminated causing this remote session to break.
plain ssh from web1 says: Host key verification failed.
keys in ~/.ssh on web1: 0

PowerShell only reports that the SSH transport broke. Plain ssh from web1 shows the real reason: web1 has never seen web2’s host key, and it has no private key to log in with, because your credentials stayed on the client. The second connection starts from scratch on a machine that holds none of them. Over WinRM the same wall is the Kerberos “double hop”. The fix is usually not to copy keys onto servers but to run both hops from the client, which a Practice task below does.

What SSH Remoting Can’t Do

# configname.ps1 - endpoint configurations (and JEA) belong to WinRM, not SSH
try {
    Invoke-Command -HostName web1 -UserName ops -ConfigurationName Restricted -ScriptBlock { hostname } -ErrorAction Stop
} catch {
    "$($_.Exception.GetType().Name): $($_.Exception.Message)"
}

$ pwsh configname.ps1
ParameterBindingException: Parameter set cannot be resolved using the specified named parameters. One or more parameters issued cannot be used together or an insufficient number of parameters were provided.

-ConfigurationName selects a named session configuration on the target. Those configurations, including Just Enough Administration (JEA) endpoints that limit what a user can run, exist only in WinRM remoting. That’s why PowerShell can’t match the parameter with -HostName at all. SSH remoting also has no $PROFILE and no sudo inside the session. If you need JEA, set up WinRM on Windows; this lesson doesn’t cover that, or CredSSP, or disconnected sessions. Microsoft Learn covers each one: PowerShell remoting over SSH, WinRM remoting requirements, Just Enough Administration, the second hop and CredSSP and disconnected sessions.

Practice

1. From the three-host job, print each failed host with the type of error, then a retry list that holds only the failed hosts.

# q1.ps1 - name the hosts that failed, and why, then list them for a retry
$job = Invoke-Command -HostName web1, web2, web9 -UserName ops -AsJob { hostname }
$null = Wait-Job $job
$failed = $job.ChildJobs | Where-Object State -eq 'Failed'
$failed | ForEach-Object {
    "$($_.Location) failed: $($_.JobStateInfo.Reason.GetType().Name)"
}
"retry list: $($failed.Location -join ', ')"
Remove-Job $job

$ pwsh q1.ps1
ssh: Could not resolve hostname web9: Name or service not known
web9 failed: PSRemotingTransportException
retry list: web9

2. Copy a folder of three files to both web1 and web2 through their sessions, and count the files on each host.

# q2.ps1 - copy a whole folder to both hosts through their sessions
New-Item -ItemType Directory ./site -Force | Out-Null
1..3 | ForEach-Object { Set-Content "./site/page$_.html" "<p>$_</p>" }
$sessions = New-PSSession -HostName web1, web2 -UserName ops

foreach ($s in $sessions) {
    Copy-Item ./site -Destination /home/ops/site -Recurse -ToSession $s
}
Invoke-Command -Session $sessions {
    "$(hostname): $(@(Get-ChildItem ~/site).Count) files"
    Remove-Item ~/site -Recurse
} | Sort-Object

Remove-PSSession $sessions
Remove-Item ./site -Recurse

$ pwsh q2.ps1
web1: 3 files
web2: 3 files

3. Get the contents of /etc/hostname from web1 and write them to a file on web2, without a second hop.

# q3.ps1 - avoid the second hop: let the client talk to each host itself
$name = Invoke-Command -HostName web1 -UserName ops { Get-Content /etc/hostname }
Invoke-Command -HostName web2 -UserName ops {
    Set-Content ~/from-web1.txt "copied from $using:name"
    "$(hostname) now has: $(Get-Content ~/from-web1.txt)"
}

$ pwsh q3.ps1
web2 now has: copied from web1

4. Run a remote command with -HideComputerName and check whether PSComputerName really disappeared.

# q4.ps1 - -HideComputerName hides the column, not the property
$r = Invoke-Command -HostName web1 -UserName ops -HideComputerName {
    [pscustomobject]@{ Name = 'disk'; Ok = $true }
}
$header = ($r | Format-Table | Out-String).Trim().Split("`n")[0]
"table columns: $header"
"PSComputerName is still: $($r.PSComputerName)"

$ pwsh q4.ps1
table columns: Name   Ok RunspaceId
PSComputerName is still: web1