#Line break appended?

58 messages · Page 1 of 1 (latest)

feral hull
#

I have a powershell script that runs a Get-Content -Raw command and then Write-Outputs with no modifications. In bash, I call the script like this myscript.ps1 -file "myfile.txt" > "myfile2.txt" The issue is that there is a line break appended to the new file. Any idea why?

odd ibex
#

windows?

#

its not because of powershell specifically

#

but why not do the writing in powershell as well?

#

powershell -Command "Get-Content myfile.txt -Raw | Set-Content myfile2.txt -NoNewLine"

feral hull
#

Somehow I still have this extra line break that way.

odd ibex
#

sorry i missed adding -NoNewLine on Set-Content at first

feral hull
#

Oh

#

That does work, thanks. Would still be great to understand why in bash it gets modified.

odd ibex
#

i cant tell you exactly technically but it does that for everything that gets returned into batch.
e.g.

image.png
#

vs in powershell

image.png
feral hull
#

Oh then I can just strip it I guess

odd ibex
#

even output of a single char

image.png
#

you could also modify myscript.ps1 and add a parameter to copy to file
e.g.
myscript.ps1 -file myfile.txt -copyto myfile2.txt

feral hull
#

No, I will be making modifications, I'm not just copying. Was just simplifiing the script to understand the line break.

#

Thanks though, I guess I'll trim the end.

odd ibex
#

gotcha

solar hatch
#

> uses Out-File which uses formatdata
Add-Content / Set-Content just writes the raw string. Usually you want that

south plume
#

Yeah, I think the reason ...

#

Is that you're in bash, running powershell and piping powershell output to a file

#

And PowerShell is formatting, as @solar hatch said, so you get extra whitespace from the formatter

#

But I'm honestly not sure, because I'm assuming your .ps1 file invokes pwsh because of a shebang line and I don't really know how that affects output 😐

solar hatch
#

Somehow I still have this extra line break that way
How are you testing that, specifically?
hexyls or ReadAllBytes to be sure

Meaning don't use Get-Content -Raw or file length as bytes -- both will obscure comparisons
( The combination of utf8 with/without BOM and utf16 can make something appear to be equal when they are not. And vice-versa. )

Are you using wsl at all? That has issues where it overrides encoding as utf16le even if you set utf8 -- giving you unexpected bytes. ( there are details in another thread if it applies)

Here's a more precise test (screenshot)

You might be hitting the case a.txt

  • There's an extra byte
  • But there's actually 0 newlines in it

for that example, a.txt

  • PS5 is writing a single a encoded as utf16-le A single a requires 2 bytes in that encoding
  • and inserted a BOM at the start

I used 5 because I'm assuming you are. If you use 7 you can rely on utf8 being the default.

On 5,

image.png
#

For comparison here's pwsh which gives much smaller files

  • it's utf-8
  • and no BOM

Be careful you don't set your encoding to [text.encoding]::UTF8, that includes a BOM

image.png
#

Instead you want this for the non-bom version.

[Console]::OutputEncoding = 
    [Console]::InputEncoding = $OutputEncoding =
    [System.Text.UTF8Encoding]::new()

Otherwise when piping it might appear like there's an extra newline. when it's really a BOM issue.

odd ibex
#

whatever that is

solar hatch
#

@brittle dirge I was looking at several of your native binary piping threads. I'm unsure if there should be a new issue. -- if it's even possible to alter from pwsh's end? There's old issues but the closes ones were closed or merged

I edited the question a few times, so let me know if it's not clear. This most likely is something quick for you to answer -- If not It's likely my edits

Short version: The case where it breaks is when a non-Pwsh shell invokes pwsh, has implicit STDOUT,
then BASH redirects it with a trailing newline.

The "failed" versions emit a single u+000a .
I confirmed

  • it's one byte.
  • It's not encoded as utf16-le.
  • it's not a BOM, not a CR

A simple fix is using a parameter.

./myscript.ps1 -file "myfile.txt" -Out "myfile2.txt"

mkelement's thread gave me the idea

# They say this works, but they meant within a pwsh shell.
, [Text.Encoding]::UTF8.GetBytes('string') | openssl dgst -sha256 -hmac authcode

# so I tried this from bash: Using an explicit pipe... before execution ends
# (does not help)
./emitOnlyWithDef.ps1 | cat > out.txt

# another idea, also fails
echo './emitOnlyWithDef.ps1 > echo.out' | pwsh -NoL -NoProfile -File -

# this from bash works. 
echo './emitOnlyWithDef.ps1 | Set-Content -NoNew echo.out' | pwsh -NoL -NoProfile -File -

where emitOnlyWithDef.ps1 is

#!/usr/bin/pwsh
$PSDefaultParameterValues['Out-File:NoNewLine']       = $true
$PSDefaultParameterValues['Write-Output:NoEnumerate'] = $true
$PSDefaultParameterValues['Write-Host:NoNewLine']     = $true
,"z"

> pwsh -Nop -Nol -C "./emitOnly.ps1 > out.txt"
> hexyl out.txt
┌────────┬───────
│00000000│ 7a 0a
solar hatch
# south plume But I'm honestly not sure, because I'm assuming your .ps1 file invokes pwsh beca...

But I'm honestly not sure, because I'm assuming your .ps1 file invokes pwsh because of a shebang line and I don't really know how that affects output 😐

I tested that above. If I set $OFS ='' before my implicit output, then it works

file: emit.ps1

#!/usr/bin/pwsh
$OFS = ''
'z'

then from Bash:

> ./emit.ps1 > out.txt
> hexyl out.txt
┌────────┬──────
│00000000│ 7a   
└────────┴──────
#

I imagine there's unexpected side-effects. I'd wrather use a filename as a parameter

brittle dirge
#

sorry what's the repro? best thing is simplified repro steps with expected and actual results

solar hatch
#

the simple case is this file
test.ps1

#!/usr/bin/pwsh
"z"

Then if you run

./test.ps1 > out.txt

The output is "z\n" not "z"

brittle dirge
#

yeah that's likely the formatter

#

Write-Host -NoNewLine 'z' might work

solar hatch
#

Even though > is not executed by pwsh

quartz mountain
#

is ./test.ps1 > out.txt running in bash or pwsh?

solar hatch
#

yes, the > is bash

#

it is not in a pwsh shell

brittle dirge
#

the string still hits Out-Default

quartz mountain
#

with a bash pipeline you can see it's not just > which aligns with pwsh emitting the newline with Out-Default as per the above

$ pwsh -File /tmp/test.ps1 | hexdump -x
0000000    0a7a                                                        
0000002
#

sorry wrong args

#

-C gives the expected order (I always forget which one)

$ pwsh -File /tmp/test.ps1 | hexdump -C
00000000  7a 0a                                             |z.|
00000002
solar hatch
#

For the implicit test, this and those others didn't appear to catch it$PSDefaultParameterValues['Write-Host:NoNewLine'] = $true
it required the explict Write-Host

#!/usr/bin/pwsh
Write-Host -NoNewLine 'j'
brittle dirge
#

I'd be inclined to argue it's expected, if you want to write something specific to the process's stdout it's best to do that. If that still includes a new line then that'd be an issue

#

yeah there's no implicit write-host

solar hatch
#

So if you have a script that's just this

'j'

Does it touch some combination of Out-Default, Out-Host, Write-Host ?

brittle dirge
#

just Out-Default

#

well, Out-Default technically calls something called Out-LineOutput or something along those lines

solar hatch
#

For a single item does it apply a $OFS, or is $OFS used at the very end

brittle dirge
#

but it's not a real command

brittle dirge
#

doesn't apply

solar hatch
#

Let me try again, I was pretty sure that fixed it

quartz mountain
#

IIRC it's also similar to how 'a' | hexdump -C in pwsh also adds the newline to the input data

brittle dirge
#

by default ofs is considered to be a space

solar hatch
#

Yeah that IFS was a false positive. It was write-host