format-default: Object reference not set to an instance of an object. | PowerShell | Page 1

subtle skiff Jan 24, 2024, 10:39 PM

#

Hello pwsh gurus! I have run into some behavior at work that has me scratching my head. I'm trying to debug my app and this is making things seriously difficult. I'm using pwsh version 7.4 in vscode. I have an ordered hashtable that I want to interact with on the integrated terminal. When I try to access a key/property one level down from the top I get a message I have never seen before, and I hope someone here can lead me to a solution. Here's a screenshot.

distant fable Jan 24, 2024, 10:52 PM

#

When using format-* you are breaking the original objects. It is only inteded for interactive viewing in the terminal.

subtle skiff Jan 24, 2024, 10:53 PM

#

I'm not using any format commands. I'm just typing the variable name at the terminal.

distant fable Jan 24, 2024, 10:53 PM

#

ah sorry misunderstood you.

subtle skiff Jan 24, 2024, 10:54 PM

#

No worries.

distant fable Jan 24, 2024, 10:55 PM

#

I have no idea what is happening here. But I'd start with comparing a simple [ordered]@{} in get-member with what you got.

#

Others more knowledgeable than me will probably chime in.

subtle skiff Jan 24, 2024, 10:58 PM

#

Thanks weq. I've done a lot of searching and asking elsewhere but I'm finding nothing to even help get started on this. I can see the top level just fine as seen in the screenshot, and I can list keys and values at any level, but I can't interact with it in order to help debug and develop my code.

distant fable Jan 24, 2024, 10:59 PM

#

I'd try poking this problem into #powershell-help really and seeing if perhaps someone like Patrick would chime in on it.

#

seeminglyscience = Patrick.

subtle skiff Jan 24, 2024, 11:01 PM

#

I put it there as well, but I know that message scroll by pretty fast there. 😄

spark escarp Jan 24, 2024, 11:28 PM

#

Do you happen to enumerate on keys while mutating it, somewhere before the screenshot?

bleak crag Jan 24, 2024, 11:29 PM

#

Can you show us how you constructed the ordered dictionary in the first place?

spark escarp Jan 24, 2024, 11:29 PM

#

do you definitely want that type not a Collections.Specialized.OrderedDictionary ?

#

Asking because [Management.Automation.OrderedHashtable] was missing on an azure function I used the other day, meaning I think it's a recent type?

subtle skiff Jan 24, 2024, 11:32 PM

#

No. I'm aware of that gotcha mutating on keys. Basically it's used for looking up data in the nested hashtables and keys.

spark escarp Jan 24, 2024, 11:32 PM

#

I would check the previous few errors. sometimes they aren't visible, and they get missed because they generated multiple errors in one go
$error[0..4] | Fl *

subtle skiff Jan 24, 2024, 11:34 PM

#

Originally this is created by converting json to a hashtable, but I cast it to an ordered hastable in the process...
[orderedhashtable]$HpasHash = $jsonInput | convertfrom-json -depth 100 -ashashtable

bleak crag Jan 24, 2024, 11:36 PM

#

The cast is unnecessary in 7.4, ConvertFrom-Json -AsHashTable already returns [OrderedHashtable]

#

Can you share a JSON document with which this behavior occurs?

spark escarp Jan 24, 2024, 11:37 PM

#

Should he be using OrderedHashtable or OrderedDictionary ?

bleak crag Jan 24, 2024, 11:37 PM

#

spark escarp Should he be using `OrderedHashtable` or `OrderedDictionary` ?

I don't think it's pertinent

subtle skiff Jan 24, 2024, 11:38 PM

#

I wish I could share more, but I can't show any data.

bleak crag Jan 24, 2024, 11:38 PM

#

I didn't ask for production data 🙂

subtle skiff Jan 24, 2024, 11:39 PM

#

Wasn't aware -ashashtable automatically creates an ordered HT.

#

Good to know.

spark escarp Jan 24, 2024, 11:39 PM

#

do you have any sort of custom classes or arrays that are changed?

subtle skiff Jan 24, 2024, 11:39 PM

#

Not that I'm aware of.

spark escarp Jan 24, 2024, 11:39 PM

#

What does $global:error[0..3] | Get-Error look like ?

#

global part only matters if you're in a debugger break point

subtle skiff Jan 24, 2024, 11:40 PM

#

Reconnecting now to check...

spark escarp Jan 24, 2024, 11:43 PM

#

You could could also try a break when that error throws. a regular EA

function DoStuff { [cmdletBinding()]Param()
    ... stuff
}
DoStuff -errorAction 'break'
# otherwise you could set the preference. careful, it kinda gets messy
$ErrorActionPreference = 'Break'

subtle skiff Jan 24, 2024, 11:46 PM

#

This isn't even in a function. I'm literally just trying to interact with the hashtable on the command line. I'm not getting the results I need when I try to navigate down into nested keys. I want to test interactively to fix my function that isn't returning the correct data.

spark escarp Jan 24, 2024, 11:47 PM

#

Right, I know
The 2nd version might give you the state that's throwing

subtle skiff Jan 24, 2024, 11:49 PM

#

This is the output of the last error. The formatting is pretty terrible.

spark escarp Jan 24, 2024, 11:52 PM

#

I was about to say I'd try this

$error.Clear()
$hpashash
# <errors> format-default not reference
$error.count

So that error says it involves editorservices. it's possible there's a version conflict with the extension

#

You can try previous versions of the extension through the UI here

subtle skiff Jan 24, 2024, 11:53 PM

#

So you're saying if I had access to the hashtable at a powershell prompt outside of vscode, it would work? It's a vscode issue?

#

Interesting.

spark escarp Jan 24, 2024, 11:53 PM

#

Well, that could be a useful test to compare

#

if it's easy to run your code in windows terminal, that would isolate the vscode host

subtle skiff Jan 24, 2024, 11:54 PM

#

OK. One sec...

#

I get the same message outside of vscode. 😦

#

At least the error is a little easier to read from the cmd prompt.

spark escarp Jan 25, 2024, 12:04 AM

#

Okay, editor services is probably unrelated then. I'd check the data types you have.
Maybe you can get it to error on a specific key, as a hint of the cause

$hpashash.GetEnumerator() | %{  
   'key: {0}' -f $_.Key
   if($null -eq $_.Value) { $_.Key } 
}

subtle skiff Jan 25, 2024, 12:08 AM

#

This only outputs one key (Product). But I know there are others. Strange.

spark escarp Jan 25, 2024, 12:08 AM

#

$hpashash.GetEnumerator() | %{  
   ($_.Value)?.GetType() ?? '<missing>'
   $_.Value.PSTypeNames -join ', ' 
   'key: {0}' -f @( $_.Key )
   if( $null -eq $_.Value) { $_.Key | Join-string -op 'is true null: ' } 
   
   $_.Value | Out-Default
}

subtle skiff Jan 25, 2024, 12:08 AM

#

Strike that. Product is the only key at level 2.

spark escarp Jan 25, 2024, 12:10 AM

#

There's couple of values you could try. The first GetType() line doesn't throw if it's a true null value
Does it error if you try to convert back to json ?

$hpashash | Json -depth 10

subtle skiff Jan 25, 2024, 12:11 AM

#

No error converting back to json.

spark escarp Jan 25, 2024, 12:12 AM

#

Are you using other runspaces or using -parallel ?

subtle skiff Jan 25, 2024, 12:13 AM

#

No

spark escarp Jan 25, 2024, 12:13 AM

#

oh, have you tried no profile to rule out other module conflicts?

> pwsh -nop
> import-module yourmodule.psd1

subtle skiff Jan 25, 2024, 12:17 AM

#

Output from your second longer command above.

#

I don't generally use profiles at work, but I'll start a new session with noprofile to make sure.

#

Same result using noprofile.

spark escarp Jan 25, 2024, 12:28 AM

#

What type do you get for product, before it throws? $_.Value.GetType()

bleak crag Jan 25, 2024, 12:29 AM

#

Try $_.Value | Get-Member in place of $_.Value | Out-Default

spark escarp Jan 25, 2024, 12:29 AM

#

~~Err I guess it's a ordered hash~~

bleak crag Jan 25, 2024, 12:30 AM

#

Probably is. It'd help a lot if you could attempt to reproduce the error with non-proprietary test data @subtle skiff

#

Write a mock JSON document with the same "shape" as the real JSON document, but with dummy data

#

(the exercise of reproducing the issue might even help you find the problem)

subtle skiff Jan 25, 2024, 12:34 AM

#

The json version of this object is over 10k lines long. That's a task. Maybe I can find a smaller one.

spark escarp Jan 25, 2024, 12:39 AM

#

What if you remove keys not named Product, save export to json
then use that to import
and see if you still get an error

subtle skiff Jan 25, 2024, 12:41 AM

#

I found another file that is only about 2300 lines that contains the same type of object. Same results when I try to access the Product key though. So it's not a size issue.

#

Not sure what you mean by remove key's not named product.

#

All the data is nested under $HpasHash.Product

spark escarp Jan 25, 2024, 12:42 AM

#

Oh product is the root level essentially

subtle skiff Jan 25, 2024, 12:42 AM

#

There are 7 nested keys under Product.

#

Basically

spark escarp Jan 25, 2024, 12:42 AM

#

if you open the json in VS Code, does it show any errors or control characters?

subtle skiff Jan 25, 2024, 12:45 AM

#

Nothing showing up under the Problems tab for that file except cSpell unknown words.

bleak crag Jan 25, 2024, 1:00 AM

#

Try repeating @spark escarp's snippet from above, but on the entries in the Product table:

$hpashash['Product'].GetEnumerator() | %{  
   ($_.Value)?.GetType() ?? '<missing>'
   $_.Value.PSTypeNames -join ', ' 
   'key: {0}' -f @( $_.Key )
   if( $null -eq $_.Value) { $_.Key | Join-string -op 'is true null: ' } 
   
   $_.Value | Out-Default
}

subtle skiff Jan 25, 2024, 1:03 AM

#

I think I'm on to something...

#

I pulled the json out for each of the 6 sections under Product into their own files and pulled them in as hashtables using the same method as the single file. I'm not going through each one to see which will give me the format-default error.

#

Found one so far, with two left to test.

#

Ugh. It's 3 of 6 giving the format-default error.

bleak crag Jan 25, 2024, 1:11 AM

#

Keep digging down one level at a time, you'll probably find a commonality

subtle skiff Jan 25, 2024, 1:11 AM

#

Nah, I'm not on to anything. False hope. The first section I tested worked fine down through the first three levels, then gave me the same error on a key in the fourth.

#

I don't see any pattern. It errors on keys containing objects, keys containing arrays of objects... but then in some cases I can go one level deeper and it starts working again.

#

🤷‍♂️

bleak crag Jan 25, 2024, 2:27 AM

#

What is the deepest you can go and still encounter the error? That should leave you with the smallet subset of input data that can reproduce the issue

bleak crag Jan 25, 2024, 10:08 AM

#

This is the smallest sample JSON document I can come up with to trip up Format-Default in a similar way (it'll render the ['Child'] value but implicit Format-Default chokes when rendering ['Parent']) :

$broken = '{ Parent: { Child: [ {"":"v"} ] } }' |ConvertFrom-Json -AsHashtable
$broken['Parent']['Child'] # formatted and rendered without issue
$broken['Parent']          # format-default error

subtle skiff Jan 25, 2024, 12:36 PM

#

So missing quotes around keys triggered this? I would think it would show up in the problems tab in vscode. I'll come back later with an update if I can find something today. Thank you!

pastel moat Jan 25, 2024, 12:37 PM

#

it's specifically the empty key. The lack of quotes is fine.

bleak crag Jan 25, 2024, 12:45 PM

#

No, that's not what I'm saying, forget about the unquoted labels.

$broken = '{ "Parent": { "Child": [ {"":"v"} ] } }' |ConvertFrom-Json -AsHashtable
$broken['Parent']['Child'] # formatted and rendered without issue
$broken['Parent']          # format-default error

#

What I'm trying to convey is: the fact that @subtle skiff was able to reproduce it at one level of his object hierarchy but then not with a child member of that object doesn't mean that he wasn't onto something

#

He absolutely was onto something -

#

What I'm trying to get at is the same as yesterday: @subtle skiff you need to find a way to reproduce the issue with mock data - if you can share some JSON data that actually causes this issue, then we can probably find the root cause 🙂

pastel moat Jan 25, 2024, 12:50 PM

#

"somewhere in the formatter's code" 🙂

bleak crag Jan 25, 2024, 12:52 PM

#

then we can probably find the root cause more accurately 😅

#

I do suspect the answer lies in something similar - a property key that's either empty or contains something unexpected

pastel moat Jan 25, 2024, 12:53 PM

#

yep, there's some fairly... interesting code in the formatter

bleak crag Jan 25, 2024, 12:56 PM

#

Yeah, and a lot of unpacking-then-reifying-as-PSObject-and-back-again nonsense during shape detection - that's essentially what causes the error in my example above ^ "" is a valid dictionary key but not a valid PSProperty name

spark escarp Jan 25, 2024, 3:14 PM

#

Do you think this change could be involved?
https://github.com/PowerShell/PowerShell/pull/20913/files

#

In the past, I thought you got invalid key errors on ConvertFrom

pastel moat Jan 25, 2024, 3:19 PM

#

yeah because it was direct to PSCustomObject which cannot have empty noteproperty names

spark escarp Jan 25, 2024, 3:20 PM

#

Oh, Ashash isn't

pastel moat Jan 25, 2024, 3:20 PM

#

yeah

spark escarp Jan 25, 2024, 3:21 PM

#

Do you know what the Via in the ConvertViaCast function stands for?

#

from LanguagePrimitives.cs above

pastel moat Jan 25, 2024, 3:22 PM

#

via as in through or by

#

it's not an acronym, is that what you mean?

spark escarp Jan 25, 2024, 3:22 PM

#

Oh, heh okay

subtle skiff Jan 25, 2024, 3:31 PM

#

A team member pointed out that if he removes arrays in the sections that are producing the errors, the error disappears. So pwsh can't handle arrays in hashtables converted from json any longer??? 😄

pastel moat Jan 25, 2024, 3:34 PM

#

if it's format- errors then it's a problem displaying that, not a problem handling it as such

#

this still suggests a bug in the formatter, but do you need the formatter?

subtle skiff Jan 25, 2024, 3:35 PM

#

I'm not using any formatter.

pastel moat Jan 25, 2024, 3:36 PM

#

you are implicitly if you display in the console as you did in your very first screenshot

#

the formatter is implicitly invoked there by virtue of you getting the console to display the value

subtle skiff Jan 25, 2024, 3:36 PM

#

So how can I bypass that or use a different formatter than can display the data?

pastel moat Jan 25, 2024, 3:37 PM

#

you could write your own handler for it, you shouldn't have to, but... It depends what you want the outcome to be.

You can still assign and otherwise use the values as they are. It's purely a display bug.

spark escarp Jan 25, 2024, 3:37 PM

#

Did yours end up having blank or empty key names? the errors we triggered came from that

pastel moat Jan 25, 2024, 3:37 PM

#

there's a few oddities in the default output formatter that can really catch you out

#

I'd take the erroring chunk and ConvertTo-Json that fragment to get a look at an approximation of the data

subtle skiff Jan 25, 2024, 3:38 PM

#

I didn't find any blank keys.

#

Write my own? Oh brother. 😄
I can barely keep up with my work as it is.

spark escarp Jan 25, 2024, 3:42 PM

#

you didn't have a subset of a json you can share, right? I'd give it a shot

pastel moat Jan 25, 2024, 3:43 PM

#

it depends what you're doing with the value. If you never need to display it in the console...

It'd be still very interesting to see an approximation of the content that caused the problem.

bleak crag Jan 25, 2024, 3:45 PM

#

It's killing me that @subtle skiff can't share his input data 😅

subtle skiff Jan 25, 2024, 3:45 PM

#

How would I even start writing my own handler? Converting to json each time I want to see a value seems like moving backwards.

pastel moat Jan 25, 2024, 3:45 PM

#

yeah, following the path through the formatter is quite fun

subtle skiff Jan 25, 2024, 3:46 PM

#

My boss gave me permission to email the data to my personal machine to test there, so I may be able to share soon.

pastel moat Jan 25, 2024, 3:47 PM

#

Not as simple as an empty key in the data I guess?

subtle skiff Jan 25, 2024, 3:48 PM

#

I searched for "" and the only hits were values, not keys.

spark escarp Jan 25, 2024, 3:48 PM

#

oh well the string "" might not match, there could be invisitble chars in it

#

you might want a regex like

$re=  @'
"[\x00-\x1f]+?"
'@

#

You can get a syntax-highlighted subset like

function showJson { 
   $input
       | ConvertTo-Json -Depth 99 
       | set-content 'temp:\out.json'

  code -g (gi 'temp:\out.json' -ea 'stop')
}

bleak crag Jan 25, 2024, 3:51 PM

#

pastel moat Not as simple as an empty key in the data I guess?

We still don't know that that's the case, it was just the only thing I could think of to trip up Format-Default in a similar fashion 🙂

spark escarp Jan 25, 2024, 3:51 PM

#

# usage is 
$hash | SHowJson
$hash.Product | ShowJson

#

I think this is what they call nerd sniping

subtle skiff Jan 25, 2024, 3:56 PM

#

No results on that regex to find invisible characters.

spark escarp Jan 25, 2024, 4:06 PM

#

Do you get any errors if you use $data | ConvertFrom-Json -AsHashtable:$false

subtle skiff Jan 25, 2024, 4:16 PM

#

Interesting. I tested it on my personal machine and I don't get the error in either vscode or independent terminal. Of course, work is windows and personal is macos, so apples & oranges.

#

Work is pwsh 7.4.0 and personal is 7.4.1 so maybe it was fixed or it's a windows-only issue?

bleak crag Jan 25, 2024, 5:03 PM

#

Still can't tell without sample data 🙂

subtle skiff Jan 25, 2024, 6:44 PM

#

I'm downloading a Win11 vm for virtualbox to test. Where can I find previous versions of powershell so I can install 7.4.0 before 7.4.1 to test?

pastel moat Jan 25, 2024, 6:45 PM

#

should be here https://github.com/PowerShell/PowerShell/releases/

GitHub

Releases · PowerShell/PowerShell

PowerShell for every system! Contribute to PowerShell/PowerShell development by creating an account on GitHub.

subtle skiff Jan 25, 2024, 6:47 PM

#

Thanks!

subtle skiff Jan 29, 2024, 6:59 PM

#

This was evidently a bug in 7.4.0. My team is being put on the rush list to upgrade sooner than the rest of the company.

bleak crag Jan 30, 2024, 11:28 AM

#

No problems with the same input data in 7.4.1?

subtle skiff Jan 30, 2024, 1:54 PM

#

Just confirmed after upgrading to 7.4.1.... behavior is gone!

#format-default: Object reference not set to an instance of an object.