#╰・virtualization

1 messages · Page 1 of 1 (latest)

wraith mica
golden plinth
#

The 3.0 storage/clusters api says it needs
ControllerRequestV3 {
host (string, optional),
password (string, optional),
port (integer, optional),
public_certificate (string, optional),
username (string, optional)
}
However, I get the following error (when supplied only host, user and pass):
{
"errorMessage": "public_certificate is required.",
"errorCode": 10005
}
Public certificate seems to be optional when cluster is added in the UI... Being mandatory here would make automating this slightly more complicated. Emphasis being on more, not slightly 😄

wraith mica
#

@golden plinth , When I tried with swagger, it works fine with just sessionid in headers. Can you please provide more info on which version of ONTAP Tools you are facing issues with? For sample code with Ansible, please refer - https://github.com/NetApp-Automation/ONTAP_Tools_Datastore_Management

GitHub
GitHub - NetApp-Automation/ONTAP_Tools_Datastore_Management: Sample...

Sample Ansible Playbook utilizing ONTAP Tools for VMware vSphere to manage the Datastore. - GitHub - NetApp-Automation/ONTAP_Tools_Datastore_Management: Sample Ansible Playbook utilizing ONTAP Tool...

late goblet
#

Question about Cinder with NFS on openstack, how is the hash formed for /var/lib/cinder/mnt/<hashed_value> from the NFS shares provided to cinder, is there an article that describes this process ? is it a combination of the IP and flexvol name, what if a dynamic DNS name is provided for mount point ?

random reef
#

Anyone running esxi 6.5u3 with ontap 9.12.1?

little hill
#

7.0U3 here with 9.12.1

#

6.5 specific?

random reef
#

yeah.. ihac w/problems..

sinful finch
sinful finch
little hill
sinful finch
#

Ah.

#

All I know is they are kinda similar to Docker in some way or form but a not too much up on it.

late goblet
sinful finch
#

I don't know 😦

white scaffold
sinful finch
#

Oh ok. I don't know who knows the answer tho.

random reef
sinful finch
random reef
#

Cluster update was interrupted by hardware failure.. resuming tomorrow. Mostly looking for a “yes, this configuration generally works” for 6.5U3 + 9.12.1P2 + NFSv3

#

2009564766 if you’re up for having a second look at what’s been done so far

eternal wagon
#

Just NFSv3, no plugins etc is OK AFAIK. It's not really listed in the IMT.

random reef
#

yeah, that's pretty much my thinking too - we haven't changed NFS in years

sinful finch
#

No idea Alex. Looks like that case is kinda all over the place a bit.

#

Incorrect: I see some TCP delays in the ONTAP trace.

#

I'd get traces from both the ESXi host and ONTAP host at the exact same time.

random reef
random reef
#

But oh well, onwards and upwards

golden plinth
# wraith mica <@305431071141396482> , When I tried with swagger, it works fine with just sess...

Yes, this one works as it uses /api/rest/2.0/storage/clusters and this is how I had my clusters registered all the time. However in ONTAP tools 9.12 only /api/rest/3.0/storage/clusters is available where things have changed in the body. Swagger says public_certificate is optional field, but then comes with an error that is missing and is mandatory. In fact I use very similar approach to what is in the role above. You should try and register a cluster to 9.12 tools.

golden plinth
#

This is the error message: {"errorCode": 10005, "errorMessage": "public_certificate is required."}

little hill
#

Ok gang, reaching for straws here, but hoping someone has their ears on!

I powered everything down today to replace my core switches. Cabled everything back up, powered up the first ESXi host, powered on a VM, NOPE. Insufficient permissions. I can vmkping, I can register/unregister, and I can MOVE a VM to a new datastore that I was also able to mount with the same NFS export policy. But no matter what host or what VM, that datastore is not letting me write to it, or power anything on.

Any thoughts?

eternal wagon
#

export rule just r/w 0.0.0.0?

little hill
# eternal wagon export rule just r/w 0.0.0.0?

Failed to power on virtual machine DC02. An error occurred while opening configuration file "/vmfs/volumes/a527c25b-9a57c93e/DC02/DC02.vmx": Insufficient permission to access the file.

#

Was worth a shot.

upper flame
#

I bet that the storage isn't even the one denying access here. It's never the storage's fault ;)

little hill
#

At this point, I’ll try anything. I’m half a step away from having to tear it all down and start over. I’ve lost an entire day and night trying to crack this one

eternal wagon
#

if you do a new Datastore?

#

Could also try and rip everything else out except 0.0.0.0. or create a new one with just 0.0.0.0/0 and change the policy?

#

packet captures?

little hill
#

New host, no vCenter, standard vSwitches, mounted the datastore, registered VM, same error.

#

No idea where this is coming from. I am so completely and utterly lost. Tried everything.

#

It's been boiled down to barebones at this point. It's either something with the vmx config file, or the storage.

little hill
#

Welp, I have no idea wtf was going on, but re-mounting the datastores over NFSv4.1 has allowed the VMs to start booting.

#

Gonna have to dig deep on this one.

fading shard
#

What's the export-policy of the SVM root vol? And on which junction-path is the datastore-volume mounted?

#

These things sometimes happen when NFS VAAI offloading is not correctly configured.

#

Also when you're connecting to a datastore via NFSv3 and NFSv4.1 simultaneously.
I remember Veeam suddenly noticing a new datastore and accessing it via NFSv4.1 to scan if any VMs are inside. While all the ESXi hosts used NFSv3. Because of all the NFSv4.1 locks blocking the files ESXi was going crazy... it couldn't figure out why writing to some files was not possible (NFSv3 does not know about NFSv4.1 locks).

little hill
#

Nothing was 4/4.1 enabled prior, No Veeam, but the svm-root vol is interesting. Def wanna peek at that, will report back

proven narwhal
#

Hi. I just searched about ONTAP/HyperV 2022 integration. No much information. I heard SMI-S wouldn't be developed for 2022. What about SMHV? I could not find any CVD/NVA on HV 2022.
Is there anything I missed?

amber lotus
#

SMHV I'm pretty sure is now end-of-life

#

Give me a few minutes, and i'll get the details and CPQ

amber lotus
#

sorry for the delay, see CPC-00468

golden plinth
high forum
#

Did you happen to "update" the NetApp certificates? are they still valid on the ONTAP side?

golden plinth
#

Yes, they are and I have them issued by our AD CA..

high forum
high forum
golden plinth
#

The cluster.

#

This is the error:
"json": {
"errorCode": 10000,
"errorMessage": "An error occurred while adding Storage System. PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"
},

#

Also the OTVs I deployed on Friday now can add the clusters. The ones that were just deployed popped with the error above. Probably all will work tomorrow or something like that.

#

Aaand for v3.0 API call to register the cluster with OTV public_certificate is a must - either note that in documentation or mak it optional 🙂

#

One can grab the public certificate of the cluster with "openssl s_client -showcerts -connect ontap.cluster.fqdn:443 </dev/null 2>/dev/null|openssl x509 -outform PEM" - just register that with ansible shell and use its stdout as public_certificate 🙂

little hill
#

@wraith juniper

fading shard
sinful finch
#

I'm trying to think who would be good to help with that.

little hill
#

@wraith juniper

sinful finch
#

I poked a couple people. I think Chance is going to close the loop on this.

#

From what I'm told it works but VMware's implementation is still rough but ours is fine as long as you don't expect multi-node trunking.

#

@fading shard

wraith mica
#

My understanding is, currently, you can create multiple NFS lifs on a node with same VLAN. Have another set of LIFs on the secondary node with different VLAN.

wraith mica
#

https://community.netapp.com/t5/Tech-ONTAP-Blogs/The-Top-Five-Reasons-You-Should-Be-Excited-About-NetApp-s-Next-Generation-vVols/ba-p/446808

The Top Five Reasons You Should Be Excited About NetApp’s Next-Gene...

This summer is a great time to be a VMware and NetApp user, especially if you happen to be in the service provider space. Why? This summer NetApp is releasing version 10.0 of ONTAP tools for VMware vSphere. This release is completely rearchitected for the needs of the modern hybrid multi-cloud envir...

fading shard
#

Noooo, why is it gone? 😭

#

Closed it too fast but I read something about a version number 10?!

#

Since OTV has been linked to ONTAP versioning so far... ONTAP 10 this summer?! Start a Hype train!

white scaffold
#

@fading shard It's back now 🙂

wraith juniper
#

Woot! Lets goooooo!

wraith juniper
#

Meaning, the legacy 9.x generation of OTV will still be around for a while.

little hill
#

“SMVI” “ghettoVCB” #triggerwords

craggy wing
#

<triggered>

wraith juniper
#

@little hill if we consider the OTV family tree to have started with VIBE, when was that? 2007?

#

I think SMVI was 2008 and VSC 2009 if my notes are correct.

little hill
#

Yea … ish. I’d have to ask the OG, Matt Robinson, when he first wrote the original script

hollow roost
#

Wow... VIBE. Was a great starting point for customers that had the ability to manage it. We had sales people pitching it to shops with like 1 or 2 admins and it created some awful experiences.

random reef
#

Anyone remember RCU? 🙂

robust moth
#

Sure. Rapid Cloning Utility. But does anyone remember VCB? 😉

high forum
#

Older: Bare Metal Backup: the first version on floppy disk

robust moth
#

hm, oldest I remember is Central Point Backup for MS DOS, back in '93 or so? used it with QIC-80 tapes, accessed through the floppy controller. slow as molasses but easier than shipping around hundreds of floppy disks 😄

wraith mica
#

VCB - VMware Consolidated Backup.

clear palm
#

Anyone know when NVMe datastores on VMWare are supported by ONTAP tools for vSphere and SnapCenter plug-in for vSphere? Just upgraded to latest version, and NVMe SVMs/datastores are not even shown in ONTAP tools.... hmm

fading shard
#

NVMe/FC is supported already with OTV. You need vSphere 8.0 for that.

#

With SCV it's not yet supported.

clear palm
#

Well just upgraded to OTV 9.13 and it does not see the NVMe over TCP datastore... maybe it's only FC that is supported? (I'm on vSphere 8)

fading shard
#

Yes, only NVMe over FC.

woeful ingot
#

So... I upgraded to 9.13.1 in my lab, and decided to let SM enable those anti-ransomware fpolicy rules it keeps nagging me about. They blocked me from making new VMs ><. Turns out .nvram is in the extension blacklist, so maybe don't do that if you keep VMs on an NFS datastore 😉

robust moth
#

I wouldn't enable Anti-Ransomware on NFS datastores anyway. Those should only be exported to your ESX servers, and every attacker who gains access to those is probably smart enough to not use any of those well-known extensions 🤷‍♂️

#

those fpolicy lists try to protect you against automated ransomware tools which encrypt every drive/share they can get their hands on. NFS datastores shouldn't be accessible by clients

little hill
#

NFS datastores for VMware should exclusively be available and exported only to ESXi hosts IP addresses, nothing else.

#

Would also highly recommend using the NetApp OTV plugin to deploy your datastores via vCenter. It will set all these best practices for you.

woeful ingot
#

sure, I'm just concerned that our UI will guide an unsuspecting customer down an unfortunate path.

robust moth
#

@woeful ingot where exactly did the System Manager nag you about these fpolicy rules? I have mine open right now and can't even find where those are configured 😅 (shows that I'm definitely more of a CLI guy 🙈 )

#

ah, found it, that tile was hidden in the "Insights" tab. Had to (re-)enable it

#

I mean, the UI asks you to make sure you have verified all the extensions in the list... but tbh, I guess nobody reads through 3000+ extensions in a tiny textbox and finds the nvram extension in there 😄

woeful ingot
#

and it's presented as a "best practice"

robust moth
#

yeah, I can see a couple of our customers just hitting "enable" there and then wondering why their VMs won't start anymore 🤔

tame cradle
tame cradle
#

Hi all, I trying to figure out what processes are optimzied when using VaaI with NetApp NFS.
After research I only see a few improvements when deploying a template to the same datastore and cloning a powered-off VM.
Is there some more info available?
Thanks!

high forum
clear palm
#

How about NFS 4.x in VMWare 8, is that fully supported yet? I tried it some versions ago and it had some issues where it went offline if we did failovers on the NetApp Cluster... I think it was VMWare that had the issue... maybe there is a recent Best Practice VMware with NFS paper? Also missing this on NVME over TCP (which works great btw. but not fully supported by other NetApp tools...)

high forum
#

Nvme/TCP is actually block based and works great. You will not see that in any nfs document.

NFSv4.1 was supposed to be supported in 7.0 but I think it took a couple updates to get there.

With that said, everything I’ve heard is that unless you need 4.1, it is generally slower in the way it is implemented between ONTAP and ESXi

clear palm
#

Most of the time the convienence of NFS by far overway the performance gains. I still think the way you need volumes for LUNs is an abstraction layer that NetApp should remove as it just makes it harder for the customers to understand what is going on... the classic is that they see available space at four different stages... (aggregate, volume, lun, vmfs) and then of cause there is the ever ongoing issues with unmap 😉 But it looks like they are trying to fix this with the NVME implementation... anyway.. the reason NFS 4 is interesting is of cause the multipathing support.. an annoying issue with NFS3 is that it is "locked" to the IP address is was mapped in ESXi... of you have to move the volume from one controller to the other on the NetApp, you will end up with cross-talk traffic between the controllers...

fading shard
#

The additional latency introduced by intracluster traffic is negligible with current platforms.

tame cradle
#

https://community.netapp.com/t5/Tech-ONTAP-Blogs/Announcing-General-Availability-of-ONTAP-tools-for-VMware-vSphere-10-0/ba-p/447616

Announcing General Availability of ONTAP tools for VMware vSphere 10.0

Overview   Large enterprises and service providers require a scalable, centralized, and automation-enabled platform for storage lifecycle management as the data center landscape grows and evolves with a wide array of workloads.   We are excited to share the General Availability of ONTAP tools for VM...

wooden geyser
#

awesome..

#

does ONTAP tools V10 add in the ability to VMware VM snapshots etc.. or do you still need to use SnapCenter Plug-in for VMware vSphere 4.9

fading shard
#

Backup & restore is still separated from OTV and I doubt that will get merged again.

random reef
#

anyone done proxmox and ontap? 😄

robust moth
#

we have a few customers using it more or less successfully (they mainly do storage via simple NFSv3 mounts AFAIK)

fading shard
#

I have a customer using NFSv4.2 with Promox because of DEALLOCATE support

random reef
#

nice!

#

I'm hoping to get some time over the new year break to build a proxmox lab

sacred atlas
#

What does everyone think about the new licensing for VMware? Does that move more customers to vSAN vs NFS and Block? Does this new licensing model hurt the storage market?

little hill
#

VSAN is a 4-letter word at this point, 10 years in. There's a reason it still has < 10% market share. Situationally, for test beds, it's hella-useful. But if you've already got an enterprise array, you'd be a fool to lean on it for anything other than piddling around.

DISCLAIMER: I'm sure there's a VSAN lover or two out there that adores it, and happy to have that debate.

sacred atlas
#

Oh I know. With part of the new licensing is that they are giving customers free vSAN but anything more than that it’s $210 per TiB. I’m trying to disprove someone that the storage market is going hurt because of this. I disagree with him.

little hill
#

virtual infra architectures are not going to drive the storage industry forward. Next-gen LLMs and AI will.

#

No Data? No AI.

lament tapir
#

I don't think it's going to make a huge impact to the storage market. The hypervisor market, on the other hand...

clever vapor
#

I'm not sure if this is the right category here.... Santa gave me a DS224+, so I finally joined the church of Synology.... I tried to install the ONTAP simulator, but it won't boot ... I see a few "Illegal instruction" and it hangs with "Waiting until daemon ktlsd starts up." wrong CPU (Celeron J4125) in the NAS?

robust moth
#

You could try an older version of the Simulator, maybe the newer versions require specific CPU extensions that your Celeron doesn't have

clever vapor
#

9.11.1 didn't work either, should I go back to even older versions?

sinful finch
#

Probably.

#

In ESX you have to upgrade VMDK version as well. That might be a problem. The hypervisor and VM must support the newer CPU versions.

clever vapor
#

so I should have bought a DS723+ with a Zen1 CPU... looking at the Broadcom deal, I will not start looking into having an ESX host online 24/7 any more, proxmox is good enough

high forum
#

Honestly, I don’t think any of those processors on the NAS devices contain all the elements the cpu needs to run ONTAP sim

#

Unless the NAS has a full Intel or amd.

clever vapor
#

my NetApp-SE has a DS1821+ with a 4core-zen1 CPU, toaster-VM works on his machine, but yeah, different CPU type

maiden yacht
#

Does anyone have any advice for me?

#

I need to virtualize OSx on the M1 chip.

#

More than two at a time.

#

I saw GitHub would rather shuck Mac Minis like corn and put them into custom build sleds than deal with virtualization.

#

That's not a viable solution for me. XD

little hill
#

VMware Fusion an option? They support Apple Silicon in Fusion 13 Pro

#

Like what are you using as the host machine?

maiden yacht
#

Mac mini with m1.

#

The VM also needs to be remotely accessible as if it is its own machine to the network just like it would be on ESXi. It's going to talk to remote infrastructure.

high forum
#

I’ve seen posts about this. I had osx running on an Intel windows (workstation pro) for years. I finally switched to an m2 last year. Only thing I can’t do now is run a virtualized version of esxi

robust moth
#

(kidding, of course, as that is supposed to run on small SBCs like the Raspberry Pi and the Pine64... but hey, maybe an M2 version is not too far off, who knows)

high forum
#

Actually yes. It 100% is not supported on the M2(yet). The instruction set the m2 uses won’t allow it to boot. M1 will never allow it. Doesn’t have the right stuff for esx virtualization

maiden yacht
#

I am not confident that it will... I have tried so far ESXi, Virtualbox (Apparently it wont work for OSx? I had issues getting it to recognize my input devices) parallels (only runs two at a time, also very slow) and right now my biggest success is with Fusion.

#

Although I am not certain I can meet my networking goals with Fusion yet.

#

It needs to lease an IP from DNS, and it needs to be reachable over the network.

#

I think that should be doable, I just need to bridge it to the hosts NIC. Ive just never done this before, I am used to ESXi.

maiden yacht
#

My coworker wants to take another stab at Mac virtualization with ESXi.

cosmic yacht
#

it will all come down to what M series chip is being used. I don't think the M1 has any type of nested virtual support and m2/m3 support ARM 8.4-A nested

maiden yacht
#

We had a disagreement on what chip was in the damn thing (I swore it was M1, my coworker says he explicitly ordered intel) so I am headed in to the office to put eyes on the darn thing. If its M2 I am going to laugh and then pray for Jesus to take me. :V

cosmic yacht
maiden yacht
#

I did it.

#

I got ESXi to begin installing.

#

I had to turn off Secure Boot.

#

I am completely unfamiliar with macs.

maiden yacht
#

Does anyone know if VMWare prevents you from downloading files?

white scaffold
#

In what way?

maiden yacht
# white scaffold In what way?

So yesterday while I was beating my head against this task, I downloaded an ESXi ISO for 7.0.X. However as I tried out new stuff I wanted to try a method that required the ESXi offline zip file. However when I went to download it VMWare's website gave me an error stating that I did not have the requisite permissions. Which makes no sense because I had downloaded that file earlier that day.

So does VMWare restrict how many times my account can download their files?

white scaffold
#

That’s silly, but nothing would surprise me with them lately 😕

maiden yacht
#

I was talking to my team lead about it as I was leaving the office (I am usually last man to leave) and had already wrapped up and had that brain storm as I was leaving, as one does. On Monday I am going to ask them to make a VMWare account and see if it will let them download an offline zip file, and we will specifically choose one to match one of the NVMe Community Apple drivers.

#

Assuming VMWare wont allow me to download an offline depot myself.

high forum
#

If you have the VMware power cli:

Add-EsxSoftwareDepot -DepotUrl https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
Get-EsxImageProfile| select Name,Version,Vendor | Sort-Object -Property name
$src="use version from output"
$dst="my-use version from output"
New-EsxImageProfile -CloneProfile $src -name $dst -Vendor ForMe

Export-EsxImageProfile -ImageProfile $dst -ExportToBundle -FilePath .$dst.zip
Export-EsxImageProfile -ImageProfile $dst -ExportToIso -FilePath .$dst.iso

That will create both the iso and the zip

maiden yacht
#

Why would this get me the ESXi I need when I cant download from their product download center?

high forum
#

The destination site is vmware.com. This is where vsphere goes for updates. Freely available without a login. Been doing this for years to build custom images for installs.

I download the offline bundle (zip) then add in the Netapp vaai vib and update drivers as needed. Got a whole script to do this.

maiden yacht
#

Ooo nice.

junior cove
#

Simple Question: What is the difference between ONTAP tools for VMware vSphere 10.0 and 9.X? Are they used for specific environments? Or is v10 an upgraded version of v9?

fading shard
#

OTV v9.X --> OTV with the known 3x features (VSC, VASA-provider, SRA), basically everything written here https://docs.netapp.com/us-en/ontap-tools-vmware-vsphere/concepts/concept_virtual_storage_console_overview.html
OTV v10 --> currently vVol-only, you can deploy 3x of those VMs so you always have a VASA-provider available in case a host dies; currently REST-API only, will get a GUI in the next version; the other two features are also planned to get included in v10
https://community.netapp.com/t5/Tech-ONTAP-Blogs/Announcing-General-Availability-of-ONTAP-tools-for-VMware-vSphere-10-0/ba-p/447616

If you don't use vVols, currently choose OTV v9. This most likely will also get some more updates, like 9.14 or so.
In the future (maybe 6-12 months, mayber more) OTV10 will supersede v9.

ONTAP tools Overview

The ONTAP tools for VMware vSphere provides end-to-end life cycle management for virtual machines in VMware environments that use NetApp storage systems. It ...

Announcing General Availability of ONTAP tools for VMware vSphere 10.0

Overview   Large enterprises and service providers require a scalable, centralized, and automation-enabled platform for storage lifecycle management as the data center landscape grows and evolves with a wide array of workloads.   We are excited to share the General Availability of ONTAP tools for VM...

bitter tusk
robust moth
#

But we already have ontap 10.x ... it is (was) called "ONTAP GX". I still have versions 10.0.3 to 10.0.5 lying around here, just no FAS 3040 anymore to run it on 😄

little hill
#

<@&1131581028138487969> <@&1132034827655336046> <@&1132035203188142211> We know a lot of you depend on VMware and NetApp to run your enterprise environments. With the recent news, shakeups, and licensing changes Broadcom have implemented for the VMware stack, we're seeing lots of customers entertaining alternatives for a variety of reasons; cost and uncertainty being the predominant ones. For what it's worth, we're looking at all of them too, and we're curious which ones have caught your eye and why? We want to insure we provide proper documentation and support for as many as we can. I'm also going to make some content trying each of them out throughout the course of the year. If you would prefer to DM me your response anonymously, please feel free.

ebon scaffoldBOT
#

**:bar_chart: Are you looking to switch hypervisors? If so, which ones are you evaluating and/or transitioning to? **

🇦 Hyper-V / Azure
🇧 Nutanix
🇨 XCP-ng
🇩 Harvester
🇪 ProxMox
🇫 Xenserver
🇬 KVM / kubevirt
🇭 None, we're sticking with VMware
🇮 Other option not listed
🇯 Undecided

Appreciate Simple Poll? Upgrade to premium :star:

feral citrus
#

Don't want to be an ass, but I just found it funny cause I read that like yesterday.

little hill
#

At least I got VMware right? And not vmWare or VMWare. 🤣

#

Thanks for letting me know!

feral citrus
#

Thats the one where I always have a capitalized W 🙂

lament tapir
#

If Proxmox starts getting more traction, I'm going to purposely randomize capitalization because tradition

little hill
#

teh pRoXm0x

carmine fractal
#

prauxmaux

glossy summit
#

I've PoC'd a number of alternatives, including

  • oVirt (Redhat Enterprise Virtualization upstream)
  • OKD (Redhat OpenShift upstream)
  • Apache Cloudstack

They all have issues. oVirt couldn't even get installed, and RHEV is EOL'd. OKD works fine if you just wanted containers, or NAT'd VMs, but the documentation is horrible - they basically copy-paste it from redhat without updating it. For example, 100% of queries, both official documentation and third-party, state to use nmstate in OKD for configuring bridges networks with VMs - and none of them tell you that nmstate requires a redhat subscription so you can't actually use it in OKD

#

we declined to use Proxmox due to security issue, like all nodes in a proxmox cluster having passwordless root access to other nodes

#

We are a linux shop, and also don't have much budget - that rules out Hyper-V

chilly whale
glossy summit
#

Harvester is currently local hardware only and we vms on 100% external storage, so that rules it out - although a very helpful Project Manager for Harvester informed me that they plan to support 100% vm storage on Netapp Trident around October with 1.4.0, in which case it might be useful for us

#

I could talk about this for a while - replacing vmware is my current project at work, so I'm knee-deep in it

feral citrus
#

Some people also recommend verge.io. Might also be worth looking at.

glossy summit
#

but atm, it's looking like we are opting for basic KVM+Qemu with Cockpit and custom automation for the moment

feral citrus
#

No clue if it's any good tho.

glossy summit
#

because the options are very very thin. The majority of solutions are dead, everyone is abandoning hypervisors in favour of containers

#

and what's more, they are abandoning baremetal

#

so often I ask these questions and just get the reply "deploy it in the cloud" as if that's always possible

little hill
#

One of the poll options I debated including was "We're out. Refactoring everything to containers" but that opens up a whole other can of worms.

glossy summit
#

@feral citrus Hai! I think I've seen you around before :)

glossy summit
#

not to mention that people forget that 99.999% of container infrastructure requires you to have existing infrastructure in place

lament tapir
#

What features are you looking for in an alternative, i.e. live migration, HA, REST API? What's the highest priority?

glossy summit
#

good luck deploying baremetal highly-available kubernetes with no existing infrastructure - DNS, DHCP, NTP, etc.

#

they have to exist somewhere

#

meanwhile, none of that is required to deploy a simple VM in KVM

#

again, you have this chicken and egg problem where everyone develops their products assuming you are deploying it in something like AWS or Azure and can just use their infrastructure

#

which means you can't use it when it is your infrastructure

little hill
#

I do miss the good ol' days of RHEV.

glossy summit
#

and it's a waste of resources to just run 1 service per server when all of your servers are multi-unit rack-mount servers with 1TB of RAM and 500+ CPU cores, just to run a single DNS or NTP server

#

it's the exact reason that virtualization and containerization came to be in the first place

chilly whale
#

Either way Cisco Hyperflex is moving to Nutanix, fwiw

feral citrus
glossy summit
feral citrus
#

me neither

glossy summit
#

and raises complicated issues too - for example, if it's required that all hardware has a support contract, or SLAs for repair or something, can we get that with raspberry pis? lol

sacred atlas
#

You know I asked this right after the acquisition…

glossy summit
#

It seems like there just isn't a good answer TBH - as a community, we've completely abandoned VMs and now we are paying for it

#

the other issue I run into is that so many projects, both mature and fledgling, want to be the diva of your network

#

they want you to configure your network to play nice with them

#

which was an issue we had with Apache Cloudstack - we have hundreds of VLANs, but apache cloudstack required that you configure your network just so, in order to play nice with them

#

which baffles me - why is it so hard to just have a web application that plugs into libvirt to give you details, console access, and live migration, and let you just configure the networking yourself? I've been managing linux for 15 years, I can configure some networking details on virtual machines and manage my own virtual switches

feral citrus
#

I guess that's why VMware is still the only king in virtualization. They just have a complete package.

glossy summit
#

Yup

sacred atlas
#

Seen a lot of uptick in Nutanix ever since the acquistion completed and the new sku's came out.

glossy summit
#
  • Doesn't require existing infrastructure to deploy - can be installed as the first server in your environment
  • doesn't dictate your network layout, lets you configure it to match your existing network
  • Minimum server requirement is 1, and you can deploy HA later
#

OKD was a pain - the minimum servers to deploy OKD was 8!

#
  • 1 server to provide DNS, DHCP, PXE Booting, Ignition, etc.
  • 1 server to act as a bootstrap
  • 3 master nodes
  • 3 worker nodes
    If you didn't deploy it as HA from the very beginning, you could never make it HA later, you would have to completely reinstall from scratch
feral citrus
glossy summit
#

I mentioned that I could have a Proxmox cluster deployed and fully functioning for production in a few days since I'm familiar with it, but the option was declined

median tulip
feral citrus
#

So for your InfoSec team to allow Proxmox you'd need to have an additional VM with hosts the Management UI...

glossy summit
#

tried different versions, all kinds of things

glossy summit
#

@little hill I like your username btw. datacenterdude lol

little hill
#

Thanks! Been my alter-ego since 2011!

glossy summit
#

Nice!

little hill
feral citrus
glossy summit
#

Anyway, I'm hoping to find some gold here, because it's been a tough road. Someone point out a holy grail that I missed please

glossy summit
robust moth
#

AFAIK, the final rebate structurings and partner discounts/levels are not even out yet. We had some opp'ties where the Broadcom pricing came out actually cheaper than the previous VMware ones. So we won't rush anything with switching to something else.
If it turns out we need another solutions (e.g. for smaller customers or whatever) it will be tricky. Many engineers/architects in our company are voicing concerns about Hyper-V since it still doesn't work properly, has issues and/or bugs, or is generally not ready for "enterprise" usage, so they do not want to have to support that 😂 In that case we will probably look at Proxmox

median tulip
#

ten create RFE for Proxmox? 🙂 I think they are going to listen to "users" now even more, as there will be a lot of new one..

little hill
#

The sheer amount of patents and IP that VMware holds is one of the things that is going to hold the industry back, if I'm being honest.

median tulip
robust moth
#

OTOH, even though Proxmox is based on KVM, we have been told by NetApp engineers that the Efficiency Guarantee program (which explicitly lists guarantees for "KVM") does not apply to Proxmox. We're currently trying to figure out what's that all about but it seems to be difficult to get an answer

robust moth
#

again, we have at least one Opp'ty/quote where the actual price is cheaper (something like 32k€ vs. 40k€ so not even insignificant)

chilly whale
#

Though prox also lacks proper shared storage, you don't get snapshots with iSCSI and FC / NVMeOF is straight up not supported

robust moth
#

Proxmox supports NFS though? And it can do qcow-based snapshots IIRC, so basically all of our scripts from 2008 when NetApp didn't have a product to snapshot ESX datastores will still work I guess (i.e. quiesce VM, trigger snapshot, unquiesce VM)

chilly whale
#

NFS has its limitations, you could do Ceph but again that would require you to re-architect your storage infra entirely which not all of us can do

robust moth
#

yeah, but let's face it, every alternative to VMware will have shortcomings and limitations, so if you have to rely on Proxmox then there will be stuff you're not getting anymore (e..g backup software integration, SnapCenter functionality, etc.)

#

or rather, the money you don't pay to VMware anymore will go towards hiring a person or otherwise paying someone else to fix/re-engineer all these things. If you factor this in, I guess most of our customers will still stick with VMware 🤷‍♂️

chilly whale
#

At least for us, it's everything surrounding vSphere that's the problem not vSphere itself, Citrix PVS integration is a hard requirement in addition to a plethera of things

#

But yea, as you've mentioned unfortunately it'll likely have to all be re-architected

robust moth
#

yeah, and re-engineering an entire datacenter virtualization environment will probably cost much more than simply accepting the new Broadcom pricings. I mean there are ways to save there, too (i.e. by using larger CPUs with more cores).

We had a similar situation a few years ago when Oracle massively increased their prices. It was claimed that everyone would move away, e.g. to MS SQL, but this didn't happen (at least not on a larger scale). Why? Because even swapping one SQL database with another is not easy. Now imagine swapping a whole virtualization stack + everything surrounding it? Probably won't happen.

glossy summit
robust moth
#

you could always boot them from FC SAN, as long as you have HBAs in your server (or are using FlexPod 😉 )

#

there should be no difference to the OS between a "local" disk and an FCP LUN

real forge
#

When I say "garbage" i mean the rollercoaster that is the Broadcom acquisition but also some of the previous issues that have come about, while it definitely dates me, the vRAM tax comes to mind

random reef
glossy summit
little hill
#

I'm gonna start loading it onto my gear soon

#

It does appear they have early NVMe/TCP support just not through the GUI just yet

random reef
#

I’ve just built a proxmox cluster too. Setup is pretty simple - few minor UI issues, but nothing horrible

woeful ingot
#

I stood up a proxmox cluster when the deal closed since it gets mentioned so much. The good? We can use an NFS datastore as swing space to help move VMs off of broadcom. We can provide primary storage over NFS, iSCSI or presumably FC SAN. My issues with it were getting my lab VMs to boot on it. VSIM hangs loading kernel, even if I tell PVE/KVM to use all VMW device types. OTS also fails to boot because its virtio drivers won't recognize the proxmox variants of the virtio devices, and even if I get could past those initial stumbling blocks, there are other oddball PVE constraints I'll hit, like only supporting one virtual scsi controller, and only allowing 15 virtual disks. It would be fine for click ops installing vanilla linux and windows, but virtual appliances are a no go.

#

So I'm about ready to declare that one a dead end and spin up XCP-NG.

little hill
#

I’m also looking for engineering opportunities where we can contribute back to some of these projects, similar for how we did with OpenStack 10 years ago

woeful ingot
#

I understand proxmox has a storage plugin framework. That might be worth exploring as they pick up VMW expats

little hill
#

Yup. And if I’m being blunt, at this stage, running VSIM and Select are the last things on my priority list. We’ll get around to that when we get to it

woeful ingot
#

For sure. It’s an abnormally large percentage of my lab vm population. But I’m not a typical user of anything 🙂

little hill
#

I love you testing all this stuff. I wanna put LOD to work and have them put all these things through their paces and see what shakes out. Need to go harass Toby. He’ll love hearing from me. 🤣

woeful ingot
#

That will be interesting to me, since my homelab is a homegrown lab on demand.

little hill
#

In case anyone is just now joining the conversation, please head back up and read the announcement and respond to the poll! #╰・virtualization message

woeful ingot
#

As part of my setup / testing I automated build out of a nested proxmox lab in Ansible. 🙂

little hill
#

I swear if you nested it in VMware… 🤣

woeful ingot
#

I did ><

#

A DC/jumphost, 3 PVE hosts, and a single node OTS

#

Sorry about the giant preview box 😆

wicked lichen
glossy summit
wicked lichen
glossy summit
#

oracle is dangerous to get into bed with. Or to even be in the same bedroom with

robust moth
glossy summit
#

ah, you misunderstand then - you have to boot the VMS

#

and as I understand, Harvester doesn't let you store the boot volumes of the VMS on external storage

robust moth
glossy summit
#

right - but afaik, you can ONLY use VMs with longhorn storage on Harvester, and nothing else

#

so you would have to format your LUNs with longhorn as local storage, and it's be a bit of a pain to manage them all that way I'd think

#

but tbf, I've never managed network storage like that shrug

robust moth
#

you're still thinking inside k8s... longhorn, storage classes, etc. that all doesn't matter for the OS

#

an FC LUN looks exactly like a local SATA disk to the OS

glossy summit
#

it does if we want to deploy VMs through Harvester, and Harvester will only let us deploy on longhorn

robust moth
#

a VMDK on an FC datastore even more so

#

but you cannot boot your nodes from longhorn, can you?

glossy summit
#

idc about booting the actual hypervisor, that will be on local storage

#

I care about booting the vms from external storage

robust moth
#

you said you need local storage to boot the VMs from

glossy summit
#

no, I need external storage to boot the VMs from

#

but you can only use local storage with Harvester

robust moth
glossy summit
robust moth
#

an FC LUN is local storage. Your HBA shows it like a direct attached SAS disk. It shows up as /dev/sda in linux.

glossy summit
#

I'm aware of this

robust moth
#

that's the whole point of an FC san 🙂

glossy summit
#

what I don't know is how scalable it is to carve a bunch of LUNS per node, then format them as part of the Longhorn distributed storage array vs. just being able to say "here's NFS/Trident/Ceph/etc use it"

robust moth
#

Well, everyone I talked to seemed to agree that Longhorn storage is slow as molasses

#

so I would generally advise against it

glossy summit
#

right, but longhorn is the only thing you can use with Harvester as far as I'm aware

robust moth
#

I have not personally tested it yet, so I can't give first-hand experience tho

glossy summit
#

so you can't present raw disks or nfs, or ceph, or anything else to Harvester to consume as storage for VMs - only the Harvester-managed longhorn storage array

robust moth
#

well that would be bad. I mean if they ripped out the CSI interface from k8s then they're technically not even k8s-compliant anymore 🤷‍♂️

#

ah wait

#

no sorry

#

I mixed something up

#

they do more than k8s right

#

but still, if the k8s below has a CSI plugin for storage I don't see why the VMs on top couldn't use that

#

I think longhorn is what they ship with but I'm pretty sure you can attach/use other storage as well. it would be pretty bad if not

glossy summit
#

well, I know I've seen some github issues asking why they can't use existing k8s plugins like csi-nfstorage or whatever

robust moth
#

I mean we have a customer where we're implementing harvester at the moment, I can check with the team that does it but I'm 99,9% sure they are not using Longhorn because they are the ones who have tested it in the past and got very bad performance from it

glossy summit
#

the issue we have, is there's no good alternatives, particularly if we can't pay for it

robust moth
#

I think there are some more free virtualizers if you absolutely need to go the "free-as-in-beer" route (proxmox for example). But I sure hope you're not running enterprise or business-critical environments on a "free" system without some sort of (paid) support in case something goes wrong

glossy summit
#

Nope, there's a number of reasons we decided not to use proxmox

#

we PoC'd OKD and oVirt specifically because there was a route for paid support/upgrading to RHEV/OpenShift, but had way too many issues

robust moth
#

but OKD is k8s only, not VMs?

#

or did they add VM workloads now too?

#

if you only need k8s there's even more free alternatives (rancher, k3s, kubespray, ...)

glossy summit
#

no, we need VMs, k8s is just incidental

glossy summit
robust moth
#

did you look at opennebula?

#

that's something we have deployed for self-service VMs in a (small) cloud environment

glossy summit
#

interesting

robust moth
#

I think it can also run on kvm

#

(we're using it on vmware though because that's what we already have)

glossy summit
#

Thinking I can't find a comparison matrix

#

is their enterprise version only support and hosting, or is it the standard "you are limited to X CPUs, no ldap, no monitoring, etc" bullcrap lots of products pull?

#

is it really necessary to censor that word? facepalm

#

IMO, it's stupid because I'm liable to choose solutions for work that I'm familiar with in my homelab. And if you don't suppose LDAP/SSO without paying tens of thousands of dollars a year, I'm not touching it

robust moth
#

yeah, well, sadly usually "you get what you pay for" is kinda true... if you can't/won't/don'T spend money on something, you basically have to connect all the bits & pieces yourself. It sucks but that's where we are at 🤷‍♂️

#

and if you can't justify to your superiors that an investment in a properly-supported base technology for basically your entire datacenter is warranted, well, I don't know what to tell you (but I know that I would leave that place... fast... it's a disaster waiting to happen)

glossy summit
#

Meh, I agree for some things

#

but SSO should be a basic, not a $50,000/year add-on
https://sso.tax/

The SSO Wall of Shame
The SSO Wall of Shame

A list of vendors that treat single sign-on as a luxury feature, not a core security requirement.

#

in most cases, there are free libraries in the language of choice available, and some projects even actively reject PRs to include them in the FOSS version, just to keep charging money

robust moth
#

I see it a bit more nuanced. SSO can actually be bad for security because it allows a compromised AD account (for example) to take over the virtualization or backup environment. Sometimes, separate users (with different passwords) are actually a good thing, as some sort of air-gap

#

so yeah, SSO might be nice and all, but I wouldn't put it that high on my list of requirements

glossy summit
#

maybe, but there's a sliding scale of security and convenience, and manually managing many users across hundreds or thousands of systems is at the very far end of it

#

not to mention that it generates other security holes

robust moth
#

do you really need thousands of users accessing your virtualization infrastructure?

glossy summit
#

say you have 1000 services, and someone leaves the company, and you remove or disable access to 998 of them

#

but you forget two

#

now you have a potential problem

glossy summit
robust moth
#

that's easily avoided by security audit tools and/or scripting

#

anyway I need to sign off for today. see you later!

glossy summit
#

well, I guess you can go hire the intern to manage all the accounts, because it's not a job I'm signing up for - I'd have stayed in helpdesk if i wanted to do nothing but manage user accounts all day long

glossy summit
wicked lichen
median tulip
# robust moth we have seen different numbers and done some calculations based on list prices b...

looks like he is not only one.. https://www.youtube.com/watch?v=peH4ic7g5yc&

YouTube
ServeTheHome
VMware GUTS Customers with 10x Price Increases

We discuss how some VMware customers are being faced with 10x or more price increases in the wake of the Broadcom acquisition changes.

STH Main Site Article: https://www.servethehome.com/vmware-vcsp-customers-seeing-10x-or-more-cost-increases-under-broadcom/
STH Top 5 Weekly Newsletter: https://eepurl.com/dryM09

------------------------------...

▶ Play video
little hill
#

For those of you that participated in the poll here, I invite you to contribute on this one as well! (non-NetApp folks, please)

robust moth
#

I would but I don't have a LinkedIn account 🙈 (probably the only one in the world without one)

sinful finch
rigid needle
#

Is it normal for there to be this big of a difference between what Ontap shows versus what vCenter shows? Is there any way to get these closer? I've tried refreshing, rescanning, and increasing storage in vCenter with no results.

image.png
clear palm
rigid needle
sinful finch
#

If you have NFS it will report correctly since it's straight from ONTAP.

tight turret
robust moth
#

to quote George Pamboris: "Relax. Take a deep breath. VMware is not going away"

cosmic yacht
#

no, it isn't going away, but it will be unobtainable for a lot of people.

little hill
#

^Current VMware admins playing with their hosts with a renewal on the horizon, remembering the good times.

wooden geyser
#

The biggest issue i see is all the people (brain drain) leaving VMware, i really doubt there is going to be a lot more active development.. Will NetApp release all their snap tools/plugins for other hypervisors ? word on the street is Veeam will support Proxmox very shorty

robust moth
# wooden geyser The biggest issue i see is all the people (brain drain) leaving VMware, i really...

I don't think so. I mean the product will stay, it's just the various bundles and add-on possibilities that they're axing. vSphere, NSX, Aria, etc. will probably all continue to exist for quite a while. As for the official statement, I don't think NetApp released anything publicly but my quote from George Pamboris (PM for all ONTAP and FAS/AFF hardware) above was made by him in front of 400+ customers at an event this week. They are not in panic mode to try and support other hypervisors. They're of course watching and evaluating the situation but don't engage in knee-jerk reactions (which is a pretty good approach to the situation IMHO)

#

most medium to large customers will stay with VMware, and many smaller ones as well. Smaller cloud providers might move away because of the huge entry cost wrt. the core count (3000+ cores minimum IIRC) but those don't usually run SnapCenter anyway

sonic mortar
#

A lot of the folks I've been talking to are still in negotiation mode.. knee jerk reactions aren't really going to pan out cost wise either for lots of shops. But curious if anyone here has had a management team that said yeah we're out and then how you plan to deal with the transition! (maybe to openshift virtualization?)

cosmic yacht
#

our team has already been told to make plans and find a replacement, by end of year, when our contract is up

sonic mortar
#

Ooh - that's not a lot of time to move.

#

Spoke with a team that basically said - to replace everything (not just migrate) is going to take a lot longer than the time left we have on our contract so we'll probably have to renew at least some portion. But that probably depends on how big / complex your environment is.

little hill
#

The data to examine is not which to choose, but the sheer fact that only ~30% of the installed base are no doubt sticking with VMware. The other 70% are some varying levels of “we want out but we’re not sure how”

random reef
#

3x seems to be the a average uplift in pricing from what I’ve read

#

So if they lose 50% of their customer base, they’ll still be ahead

woeful ingot
#

Yup. That aligns with what I’ve seen reported too. 3x uplift is typical. Renew that 4node 128 core cluster and find a new home for everything that won’t fit in that cluster. Current customers control the bleed but would any new accounts sign up for that? Is this a drain the swamp over 5 years scenario? I think in 5 years we’ll have a good view of the new normal. Between now and then… it’ll be an adventure.

robust moth
#

Again, we have seen customer quotes (actual orderable quotes) that were in the same ballpark (+/- 20%) than their existing contracts

sonic mortar
#

I think as the dust settles we'll get a better idea. There are huge uplifts in some reports (and I wonder what they're using - really storage heavy maybe?). Definitely tough to find new customers right now but in a few years people will have forgotten/adapted circa Oracle of 2003..

sinful finch
#

If some product can get a good enough feature set and support it might take off to replace VMware.

cosmic yacht
#

for smaller shops i think there are already options out there, for those not using options like vsan/etc, most of the shops I know using VM simply use it because it's the most common thing around and easiest to get somebody to work on because they already know the product.
and, again, many of those shops are hunting alternatives because their cost is going to skyrocket at the end of their contract.
From the few people I've had hitting me up about helping with migrations, so far, are mostly looking to either Nutanix or Scale(new one for me) for a paid solution or Proxmox for *nix shops and Hyper-V for hybrid

it will be interesting to see what solutions other companies start to provide to fill in the gaps they have compared to VMware right now, or what tech partners will start to build out integrations/addons/etc to work with others

wraith mica
sinful finch
woeful ingot
wicked lichen
wicked lichen
# sinful finch This might be a good alternative for many customers. RedHat is a company that is...

Hey Paul, if you want to find out a little more about OpenShift Virtualization, Red Hat has a learning path set up here: https://cloud.redhat.com/learn/getting-started-red-hat-openshift-virtualization

Red Hat Hybrid Cloud
Getting started with Red Hat OpenShift Virtualization

Having the freedom to run virtual machines (VMs) within a container and on a single platform can simplify VM management and reduce time to production. Using Red Hat® OpenShift® Virtualization can enable administrators to incorporate virtual machines into their containerized workflows without having to drastically alter their existing infrastruct...

sinful finch
white scaffold
random reef
#

I've got my 3 node NUC proxmox cluster up and running some workloads. It's very well documented, and once you get used to some of its.. expectations.. it's quite easy

#

biggest thing I found was that it treats everything it can as block storage, it makes sense

woeful ingot
#

I've decomm'ed my vsphere management cluster to free up 4 nodes for bare metal alt hypervisor testing. PVE is going first. want to see if there is any way to convince ontap to boot on it.

random reef
#

how many physical test boxes do you have at home @woeful ingot ?

#

with a name like madlabber, I gotta assume it's a few 🙂

cosmic yacht
#

heh.. how many physical boxes... 5 desktops, 3 laptops, 1 frankentop.. laptop with no lcd, 2 cisco 220m4 servers, franken dual asus server, dell T630, dozen or so SBC and a new mini-pc that isn't online yet

woeful ingot
# random reef with a name like madlabber, I gotta assume it's a few 🙂

😄 My vsphere lab was 7 hosts, now down to 3. I peeled off all the NUCs and collapsed the management domain into the workload domain, which is now 3 hosts, 2 running an OTS HA pair, and with a single node and the deploy VM living on the 3rd. There are several other NUC class hosts, mostly running KVM OTS variants at the moment, and a "travel" lab nuc that is usually in the backpack when I'm on the road. I wish I could make better use of LoD, but I need custom labs, labs that live more than 7 days, and labs that I'm not locked out of if I don't have my corp laptop in hand.

#

oh, and a DL380 in the garage if I run out of oomph 🙂

neon lodge
#

Dang, that's impressive @woeful ingot. I only have two Dell R730's now, At one point I had 6 running VSAN as well as iSCSI and NFS from 2 Synology DS1813+ and 1613's. Not currently powered on but when they are, I have a pretty decent environment 🙂

cosmic yacht
woeful ingot
#

At one point my lab was a mix of ML and DL series HP, but as impressive as that power bill was, my wife was not amused. I ended up redesigning everything for low power and low noise. So lots of NUCs and XeonD

white scaffold
woeful ingot
#

It's an interesting concession.

little hill
#

Yesterday, I had the privilege of hosting Rich from the 2GuysTek YouTube channel on an episode of NetApp ONAIR. Typically, we have stuck with bringing NetApp people on the NetApp show, but after discovering his extensive evaluation work across a myriad of #VMware alternatives, I selfishly wanted to have a deeper conversation with him about some of his findings.

What we got was a deep, rich conversation full of doubt, hope, and solid recommendations depending on your particular degree of situation you might be in.

My thanks to Rich for not only being on the show, but being vulnerable and willing to discuss his journey and recent work he's done on evaluating Hyper-V, ProxMoxVE, XCP-ng, and several others.

Enjoy the conversation, and if there's any questions, drop them in the comments! 🤘

https://www.youtube.com/watch?v=FApFFOpUbKI

YouTube
NetApp
Virtualization in the Broadcom Era | NetApp ONAIR

Second only to the white-hot nature of AI, the predominant topic of conversation amongst enterprise professionals of-late is VMware, alternative options, and even virtualization itself is being taken to task. So for this SPECIAL FEATURE episode of NetApp ONAIR, I wanted to bring in a special guest who, as a VMware admin and enthusiast, has just ...

▶ Play video
white scaffold
#

Whatever, call me biased, but that was a good episode. It was relevant and genuine.

cosmic yacht
#

I'm still wondering if major vendors (netapp, pure, etc) will start working integrations with other companies outside of vmware. hopefully it will expand well beyond vmware

random reef
#

vmware has the most options, so it's what we've done the most for. There's nothing too magical about plain NFS or block to KVM, proxmox, Nutanix.

#

could we write a site recovery manager for proxmox using snapmirror or metrocluster? probably. But without demand it's a hard one to show ROI for to do so.

little hill
# cosmic yacht I'm still wondering if major vendors (netapp, pure, etc) will start working inte...

This will happen when customers start moving the needle. The absolute best way to get traction will be for our customers to begin communicating their intentions of migrating to a particular platform (away from VMware) to their account teams and/or VARs. Engineering dollars and prioritization require demand. Exploration into potential partnership with some are in progress. No promises, but we're putting things through their paces internally. Side-note, I'm also standing up a 3-node ProxMoxVE cluster and working on hooking it up to a C250 to start developing some best practices. Ideally, they are the same as what's worked for VMware over the years. At the end of the day, it's NFS. We can likely recommend some host/node settings and configuration practices to help make the best of connected NetApp storage.

shrewd stone
#

Hi there, anyone been using XCP-NG extensively and can provide insights on its stability, performance, and usability within a multi-tenant environment, especially with NetApp NFS. I've been testing XCP-NG for the past 2-3 weeks and have been impressed by its large feature set. I find it quite user friendly and very similar to vSphere in alot of aspects. Pricing doesn't seem to be to bad either.

little hill
#

I’m putting ProxMox through the paces currently, but expect I’ll be landing on XCP next. I’ve got a C250 and (3) HP DL380 gen8’s

high forum
#

I’d like to know how XCP-NG is mounting using NFS. I had a customer using XenServer that had a data center failure (sure handler stopped, everything either shutdown-NetApp-or crashed-Dell). After a restart, they found nearly all the VMs were corrupted in some fashion. I could really only attribute this to the way xen default nfs mounts: SOFT! Worse, they (xen) adjusted the timeout to be only 10 seconds. Hence when the Netapp gracefully shutdown, the xen storage repositories got confused with soft mounts and bad stuff happened.
So if anyone is using xcp-ng, please log into a console and run mount. See if nfs mounts are hard or soft by default

shrewd stone
#

You can set NFS mount options within Xen Orchestrator. Good point though @high forum thanks for highlighting that

image.png
high forum
#

I know you can set them. The default, especially these days should be hard! Otherwise it’s simply a ticking time bomb

#

For Citrix/xen. There was a script to change to hard for default. Not supported though

shrewd stone
high forum
#

Thanks. That’s really stupid. Especially if you go to the console and just do “man nfs” and find the soft option

#

Nothing like having holes punched in your data

fading shard
robust moth
#

hehe, here I am thinking you're all talking about a new version of NetApp XCP 😂

little hill
shrewd stone
little hill
random reef
little hill
random reef
eternal wagon
#

Fantastic !

woeful ingot
#

For once a positive change.

granite comet
#

Hi All,

Can a vCenter datastore have more than one IP for performance? Our NetApp that we are using for centralized storage of our VM’s has two port channels leading to the switch that our ESXI hosts/vCenter is connected to, each with their own IP on the NetApp side, however on our datastore we only have 1 of those IPs added. The two port channels on the NetApp do provide redundancy on the NetApp side as the IP can hop to the other port channel if it’s own port channel dies somehow, however I was hoping we could increase throughput and performance. Would appreciate any 2cents. Thanks!

fading shard
granite comet
#

nice! thank you! i wonder why NFSv3 and not v4/4.1?

robust moth
#

ESX can do v4.1 but (at least in the past) was quite buggy and not very stable (especially on takeover/giveback) so I'd stay away from it. Also NFSv4.1 session trunking works only on a single node, not with IPs spread across nodes. speed-wise it doesn't make any difference anyway for ESX

granite comet
#

Hmmm.. seems I will need to do my research on NFSv3 vs NFSv4.1 for VMware datastores, my predecessor used 4.1 and I am just now doing my first ever build outs for tech upgrades of our stacks

robust moth
#

if you use 4.1, at least thoroughly test takeover/givebacks and LIF migrations between nodes to see if it is stable

granite comet
#

That seems like a huge flaw in 4.1 for it to be viable, the reliability of LIF failover

high forum
#

I have yet to see 4.1 work reliably every single time (storage takeover and storage giveback or just a node failure in general). More often than not I still see an ESXi host get “stuck” and the only recourse is to hard-reboot it

#

I have seen it work occasionally but once v4.1 gets under load, all bets for me are off the table (when using VMware)

robust moth
granite comet
#

Roger that! I’ll relay all this to my VMware counter part, I wonder if there was a reason why my predecessor began using v4.1, we did have some older FAS systems that I saw he was using v3 on

#

If there’s no sincere reason for me to use v4.1 I would rather stick with the reliable v3 even if it is older

high forum
#

the crux is support. ONTAP supports pNFS which is mutually exclusive from VMwares session Trunking. As has already been indicated, session trunking in ONTAP is limited to a trunking group per node. ONTAP 9.14 does "support" session trunking but until someone can prove to me it works in a multi netap node cluster reliably under load/pressure there is no way I can suggest it

#

just stick with v3.

random reef
#

Yeah if you find a VMware setup using nfs other than 3, you need to find out if the admin was a nerd knob twiddler who assumed newer = better, or if they had a genuine reason. It’s normally the first

granite comet
#

I thought maybe the vSphere STIG that we have implemented may have required it, but I looked through it and didn’t find anything

high forum
#

The stig absolutely does not require it. The only possible reason someone should even think about using v4.1 is to use Kerberos but that just slows performance so no one uses it.

granite comet
#

Yeah it’ll be interesting to hear why it was used in our last build outs. Guessing it’s what Alex mentioned above, someone thinking newer = better

boreal thunder
#

Saw the poll Nick did back in February, and noticed that HyperV/Azure was fairly high on the list. Anyone actually follow through with that and are using NetApp in Hyper-V/Azure Stack HCI? Trying to find folks with a similar setup to ours has been a challenge

cosmic yacht
#

we're in Azure with 25+ clusters, but it's all CVO

robust moth
#

we will be testing Azure Stack in our lab sometime in the next weeks. But so far we have no experience

granite comet
#

Hi all, a little confused on the wording from nConnect documentation. The highlighted portion initially seemed contracting to me, however is it just talking about supporting multiple TCP connections across one singular physical connection?

0.png
#

Does that still increase performance? like, if i have one 10gb connection from e0c, but now am able to put more than one NFSv3 data LIF on that e0c and connect both to my datastore, wouldn't that still have a 10gb limit?

#

thats what this documentation is saying to my smooth brain at least

image.png
high forum
# granite comet thats what this documentation is saying to my smooth brain at least

of course the 10G limit is still there. The benefit is the ability to use the 10G to its fullest. Most things do not fully use a 10G connection. Allowing multiple simultaneous connections allows for more to pass.

Impact of nconnect on total connections
When an NFS mount is established, a single NFS connection ID is used. However, with the new NFS
option nconnect, it is possible to open multiple TCP connections per NFS mount. For instance, using
nconnect=8 with your NFS mount option creates as many as eight NFS connection IDs to the data LIF
in the SVM. Although this can deliver extra performance benefits, it can also use up more connection IDs than expected

If you plan on using nconnect in environments that generate a large number of mounts, be aware of the
connection ID limits per node for your platform, and plan to distribute connections across multiple cluster nodes to balance connections appropriately. In cases where a single node/data LIF is being used for NFS mounts, connection IDs run out much faster than if multiple nodes are used. Connection ID limits should be factored into scale discussions when architecting the solutions.

You should review https://www.netapp.com/pdf.html?item=/media/10720-tr-4067.pdf and read about nconnect

tight turret
#

If you use ESXi and are Active Directory joined, Heads Up: https://www.microsoft.com/en-us/security/blog/2024/07/29/ransomware-operators-exploit-esxi-hypervisor-vulnerability-for-mass-encryption/

Microsoft Security Blog
Ransomware operators exploit ESXi hypervisor vulnerability for mass...

Microsoft Security researchers have observed a vulnerability used by various ransomware operators to get full administrative access to domain-joined ESXi hypervisors and encrypt the virtual machines running on them. The vulnerability involves creating a group called “ESX Admins” in Active Directory and adding an attacker-controlled user account ...

fading shard
#

I really really hope nobody does that 🙄

tight turret
#

I know more than one tier one university / healthcare org who does do this.

little hill
frank crystal
#

We need a second coming of oncommand shift to deal with this VMware problem… would make mine and my customers’ lives so much better

little hill
#

It’s under way. I can’t share more than that for right now.

#

Reach out thru your TPM private channels for an update

frank crystal
#

I’m in RTP today with Brandon Miller and some others at an EBC…

#

We “kind of” just talked about it 😛

white scaffold
frank crystal
#

@white scaffold too bad…

sinful finch
#

BMilly!

robust moth
#

Are there any details on when ONTAP Select will be supported (or "will work", rather) on vSphere 8 u3? IMT still only goes up to u2. A customer tested it recently and the Select VMs either outright crash or don't work properly (as in "no login prompt" but System Manager works, etc)

sinful finch
#

Contact the IMT team. I bet you could do "report a problem". They used to have a NG back in the day.

atomic umbra
#

Hi! is it possible/supported to use Openshift Virtualization with NetApp Trident (ontap-san) using Snapmirror Active Sync with RWX PVCs?

little hill
#

That’s a lot of moving pieces! @proud kayak is this something you can assist with?

robust moth
#

RWX with LUNs is difficult. What filesystem are you intending to use there?

#

technically it is possible but you need a cluster-aware filesystem on it (GFS2, OCFS2, etc.), but even then the question remains on how/where to run the lock manager that those require

drowsy steppe
#

Hey all, at work we are looking to setup Hyper-V. I want to make sure I'm doing this right from the stroage side and looking for input. I have three nodes, each node is connected to 3 LUNs via iscsi using an initiator group. We clustered the three nodes using WSFC and added the LUNs as cluster shared volumes. Does this make sense to do it that way? We are licensed for scvmm and wonder if there is a better way regarding storage.

little hill
#

Have you been through these yet @drowsy steppe?

#

for iSCSI LUNs, insure you've got the server feature for MPIO installed.
Add-WindowsFeature Hyper-V, Failover-Clustering, Multipath-IO `-IncludeManagementTools –Restart
^We recommend just running that to insure you have the features all installed

drowsy steppe
#

Hi @little hill thanks for the follow up. Yes, after I posted I did another search and found that. I think I’m good

little hill
#

Awesome! Been a while since I've tinkered with H=V but I know the person that writes those docs. Let me know if you run into anything!

drowsy steppe
#

That would be great! Bit of a learning curve from VMWare but the basic idea is there. Thanks Nick

little hill
#

It runs great on core if you know your powershell. But SCVMM does most of the heavy lifting anyway from a mgmt host, similar to vcenter

atomic umbra
drowsy steppe
atomic umbra
wicked lichen
#

I know he's been tagged a few times, but @proud kayak did do a demonstration on some of the Astra business continuity stuff with Snapmirror at Red Hat Summit a few months ago, and I believe the support for protecting VMs in OpenShift is slated for an upcoming release.

scenic rune
#

Figured this is a good a place as any to ask. We're currently standing on a proxmox cluster to replace our VMWare Cluster and have an ONTAP. Love the storage but we're having tons of NFS issues... Has anyone solved the constant time out? Fwiw we have bonded 25gbps DACs from the new cluster to a Nexus9k and same on the NetApp. Same setup with esxi has zero issues. Ports mentioned are dedicated only to talking to the NetApp.

high forum
#

Make sure the nfs Mount is using v3 and is a hard mount.

Might not hurt to go to the proxmox cli and run “mount “ to show what mount options are being used.

Might need to start using iSCSI?

scenic rune
#

We've tried 3, 4, 4.1, 4.2. Will check mount options. Would rather run NFS for thin provisioning if possible.

high forum
#

Don’t use 4.x! That is a stateful connection that may be disrupted during takeover/giveback

What are the default mount options for v3? Post them. Educate us

little hill
#

Currently iSCSI is the only IMT-qualified protocol. We expect NFS and FC to be done any time now, followed by some best practices later this year or early next year. I know this doesn't help right now, but we're going thru the paces ourselves to find the best configurations to run PVE on ONTAP.

robust moth
#

Also check the MTU on all devices if you are using jumbo frames

high forum
#

Good call although that usually manifests different issues. For instance You can mount, but can’t do anything or the mount will start but hang. I’ve haven’t yet come across an nfs mount with a mtu mismatch with normal working access later misbehave

primal wave
#

Hi, I hope everyone is fine today. I'm trying to see if there is anyone who has stumbled over a NFS-Offloading capability outside of VMware API (VAAI). 🙏

cosmic yacht
primal wave
little hill
primal wave
scenic rune
scenic rune
# high forum Don’t use 4.x! That is a stateful connection that may be disrupted during takeov...

That really depends on what your architecture is. We have a dedicated storage network with bonded 25gbps DACs on each server on an isolated VLAN with nothing competing for either the queues or backplane so 4.x is what we use with our current ESXi architecture with zero issues holding the actual VM Disk Images for our production environment with around 70 VMs. Pretty sure we have either 4 or 6 (bonded in pairs) DACs running from the netapp to the same Nexus9k switch.

little hill
#

That can help steer engineering prioritization!

scenic rune
robust moth
#

the NFS 4.x issues have nothing to do with network speed, cabling or bonding

#

NFS v4 has no advantages over v3 (at least for ESX/virtualization), especially on dedicated VLANs. It only brings pain

high forum
#

^^^ this!

scenic rune
#

🤷‍♂️ Never really experienced any pain with it except that it's finnicky to a bad network. 4.2 is pretty solid since they fixed a bunch of the issues from 4.1.

robust moth
#

does VMware support 4.2 now? I thought they were stuck with 4.1

#

but good thing that it works for you. I have never seen an ESX setup that doesn't choke on a lif failover to another node... but maybe it's been improved recently, haven't had any customers using 4.x for a while now

high forum
#

At this point, I do not think I can ever trust ESX with 4.1

little hill
#

If you don’t mind the up-front investment setup, NVMe/TCP with vSphere is sweet

robust moth
atomic umbra
high forum
little hill
wicked lichen
robust moth
high forum
#

Nope. It’s actually something we tested on.

#

There is an argument to allow a lun to be thin. Nothing exists yet for nvme. If I recall, it’s not in the spec!

robust moth
#

so why is my volume then not full?

#

what if I put a dozen 80gb namespaces in a single 100g volume?

high forum
#

It is ONTAP and the VOLUME that you are seeing. the namespace itself doesnt support thin. The underlying volume will still dedupe

robust moth
#

the namespace is just a file though

#

for me this looks exactly like in the LUN case with a non-space-reserved LUN

image.png
fading shard
high forum
#

looks like snapshot space reserved

robust moth
#

I think the KB you linked to is no longer correct

high forum
#

try to change that option

robust moth
robust moth
#

"Space Reservation: false" means thin provisioned

#

the KB says it is fixed to "Space Reservation: true" which is thick provisioned

#

this is all 9.14, and the KB has not been updated for a year...

high forum
#

Show me how you change that option. it is not available on the CLI at all

robust moth
high forum
#

convert thin/thick

robust moth
#

I created a new namespace and this is what it was set to

#

I don't want to convert thin to thick

#

Maybe I cannot change it, but that's besides the point 🙂

#

there isn't even an option to change it to thick provisioned

high forum
#

I have a 9.14 and it defaulted to space res false

robust moth
#

yes

#

that is what I said

#

That's why I said nvme namespaces are always thin provisioned ("Space Reservation: false"). there is no option to change it to thick provisioned ("Space Reservation: true")

#

that KB might be old

fading shard
#

well guess what: The space guarantee for namespaces is the same as the space guarantee of the containing volume. 🤓
https://docs.netapp.com/us-en/ontap/nvme/support-limitations.html#namespaces

cl1::*> vol show -volume test_og1_* -fields space-guarantee,size,available,used
vserver  volume         size available used    space-guarantee
-------- -------------- ---- --------- ------- ---------------
gw2_NVMe test_og1_thick 1GB  872.1MB   100.7MB volume
gw2_NVMe test_og1_thin  1GB  972.5MB   300KB   none
2 entries were displayed.

cl1::*> vserver nvme namespace show -volume test_og1_* -fields size,is-space-reserved,is-space-reserved-honored
vserver  path                                size  is-space-reserved is-space-reserved-honored
-------- ----------------------------------- ----- ----------------- -------------------------
gw2_NVMe /vol/test_og1_thick/namespace_thick 100MB true              true
gw2_NVMe /vol/test_og1_thin/namespace_thin   100MB false             false
2 entries were displayed.
robust moth
#

yeah, makes sense 🤷‍♂️

#

I wish they would put something like "correct as of ONTAP 9.x" into the KBs so that you know if they still apply... I mean I guess it was different before 9.14 otherwise that KB wouldn't exist, but since we have no older systems anymore (or do we?) it would mean setting up some simulators to check that now 🤔

scenic rune
high forum
#

based on the KB I provided, it is correct based on the setup being used. There should 100% be better clarification in the KB about the space-reserved on nvme namespaces.

#

moral of this: when creating a nvme namespace, you do not currently have control if the provisioned space is thick or thin, you rely on the underlying volume

#

I also just did some bit flips. changing the underlying volume from none to/from volume seems to make no change to the space-reservation" flag for the namespace

robust moth
#

also, can you clarify what you mean by "the KB is correct based on the setup"? because as I see it, the KB is not correct (it shows different values for "space reservation" and "space reservation honored", which I couldn't reproduce: for me those are always in sync, just as OG1 showed in his output above)

fading shard
#

I guess if you change the space guarantee of the volume, the namespace will not change. At least that's how I understood TMAC. I have not tried that yet.

robust moth
#

then it kinda makes sense I guess

high forum
#

right. @fading shard I did the same. thick vol thin vol and the namespaces did as we expected (thin/thick based on volume). When I flipped the volumes to all thick and and all thin, the "-is-space-reserved" did not change, so I think it is possible to get that combination from the KB.

#

Like I said, it is probably worth putting much more detail into the KB or generate a new KB about nvme namespaces and how thin-provisioning plays

#

especially that you cannot change it

robust moth
#

yeah I think so too

wraith mica
wraith mica
scenic rune
scenic rune
#

So we're planning to keep VM disks on CEPH and use our NetApp for bulk data storage of our robot generated data.

scenic rune
#

Anyone played with NVMe/TCP with Proxmox yet? @little hill got my interest piqued and I wanted to see what opinions are.

little hill
#

I’ll get Florian and the guys from Credativ in here for some Q&A

scenic rune
#

^^He really is the best

#

I promised my engineering team that I wouldn't dive in to it yet, some Q&A would definitely sate the thirst for performance 😄

wraith mica
scenic rune
robust moth
woeful ingot
#

I did some preliminary lab testing of using a common NFS/SMB datastore to swing VMs over to Proxmox from ESX, and it worked pretty well. Basically configure NFS volume that both can access, create stub VM in proxmox using VMware device types, single file flex clone the vmdk and -flat.vmdk into the stubs directory, qm rescan -vmid <stub vm ID>, connect disks and power on. Fun stuff. ONTAP is awesome.

robust moth
#

you don't even need to flexclone the vmdk descriptor file as that is a few bytes anyway, you can just copy that over. The flat file is the big chunk and there flexclone really helps 👍

frank crystal
#

So when is oncommand shift officially coming back? I have a potential to wipe out 14 Pure Flasharray XLs but it is contingent on OnCommand shift

cosmic yacht
#

why shift specifically? there are other options that will convert

frank crystal
#

because it is fast like a marsupial and not as janky as say starwind or veeam... It is for a mass conversion to speed time... and vmotioning to a new array and then shifting it to a prebuilt hypervisor is epic... used it successfully the first go round

little hill
frank crystal
#

Oh I get it... I'm all NDA'd up... Just thought since this a change to change a major amount of curtains from orange to blue; this might prime the pump of progress a little more >5 million opportunity

#

Literally 14 pure flasharrays displaced

cobalt tangle
#

Hi there, quick question. Is running Exchange on VMware with NFS Data stores supported?

#

I'm pretty sure thats an unsupported use Case

robust moth
#

supported by who? Microsoft?

#

I think Exchange only supports block storage, at least that's how it has been in the past. Nevertheless it works just fine on NFS (and has been for almost 20 years now). If you want to be on the safe side, use in-guest iscsi LUNs. If you want an easy solution, use a vmdk on an NFS datastore

cobalt tangle
#

Yeah but the easy solution seems to lead in some edge cases to a dirty shutdown DB. At least the Exchange Admins told me that..

#

Is there an official NetApp statement about Exchange servers? So I can pressure the Exchange Admins to get the setup right..

robust moth
#

Don't think I have ever seen that. The SQL server doesn't know or even see it's running on NFS. As long as your ESX has a stable NFS connection to the datastore (and NO NFS 4.1!!) it should all be fine

eternal wagon
cobalt tangle
#

From your perspective, when can I call a NFS Connection stable? During our latest ANDU we saw storage latency around 1500ms during the failover. Is that normal or should I be worried?

robust moth
#

but that article talks about mounting the NFS from within the Windows server VM, which is of course not supported. "All storage used by Exchange for storage of Exchange data must be block-level storage" -> a VMDK is block level storage for what the Exchange server is concerned 🙂

eternal wagon
#

check the second one

robust moth
#

(nevertheless we have dozens of customers doing exactly that for over 10 years 😂 )

#

so yeah if you're concerned (or your DBAs are concerned) use in-guest iscsi to be on the safe side

little hill
#

Use iSCSI LUNs. You’ll have a much better experience. I was doing this in 2008 with VMware and NetApp

cobalt tangle
#

🙏 thanks for your input @eternal wagon @robust moth @eternal wagon

high forum
#

Correct. If you want to use something like SnapCenter to protect your exchange vm the vm itself can be on an nfs datastore. The exchange should be on a vm attached iscsi lun. I think that exchange also possibly supports using continuously available shares but I don’t that works with SnapCenter

wraith mica
#

High level overview of VPC with VCF and its integration with ONTAP features. https://community.netapp.com/t5/Tech-ONTAP-Blogs/Private-Cloud-with-VMware-Cloud-Foundation/ba-p/456247

Private Cloud with VMware Cloud Foundation

Private Cloud with VMware Cloud Foundation   Customers often find the gap between the public cloud and their private cloud with VMware. In this blog, let me explore how to make it seamless with VMware Cloud Foundation (VCF) based on announcements made at VMware Explore.   For example, in AWS, the cu...

robust moth
#

Has anyone heard of a NetApp tool called "VMDC", aka "Virtual Machine Data Collector"? Apparently that is a thing that partners should have access to, but I can't seem to find anything anywhere 🤔

cosmic yacht
#

I know of the data collector for vsphere and hyperv, but believe they are part of Data Infrastructure Insights
never heard of a stand alone for it

robust moth
#

ah that might be it. thanks

wraith mica
#

VMDC will be available soon as standalone script right after the shift tool.

little hill
glossy vortex
#

What does this tool do? Similar to rvtools?

little hill
#

It's a data collector "plugin" (for lack of a better word) for our Insights observability product

#

You may also wanna check out this episode with @patent pumice who may be able to fill in the blanks here on some of these questions, since he's in here.

patent pumice
#

If Data Infrastructure Insights (DII - formerly known as Cloud Insights) is the thing in question, @glossy vortex you could say yes like rvtools in terms of what data it collects, but that's largely where the comparison ends. DII does a lot of AIOps heavy lifting (config change analysis & correlation, custom metadata, heterogeneous compute/storage support by design, ML-powered smarts for minimizing alert nosie, etc. etc.) to make admin/ops life easier - the doc release notes of the last couple months can give you a pretty good idea of what I mean: https://docs.netapp.com/us-en/cloudinsights/concept_whats_new.html That said, it does sound like @wraith mica is talking about something different than what I just described in relation to @robust moth's original question.

What's New with Data Infrastructure Insights

New features and functionality available in Data Infrastructure Insights.

glossy vortex
#

I see; yeah we did a poc with oci on-prem some years ago. I really liked the tool but due to costs was scrapped.

patent pumice
#

ahh that's unfortunate to hear! While it does have a cost, the benefits vs that cost are massive. hopefully next time you get an opportunity to put it into production 🙂

fading shard
robust moth
fluid kernel
robust moth
#

looks interesting so far but I guess it needs a while to populate all the metrics and stuff 😄

image.png image.png
robust moth
wraith mica
robust moth
#

but yeah once you log off it's gone so if you want to have permanent access, installing as service is better

#

the topology window sounded interesting but it is still rather empty 🙂

image.png
#

Will be interesting to see what will happen with this tool in the next weeks/months

wraith mica
robust moth
#

Maybe I need a different browser, this runs in my test VM with only Edge installed

wraith mica
robust moth
#

hm. no idea. The logs show nothing helpful I think. What vSphere version are you running?

#

or maybe it just takes a while to scan our lab environment with >1000 VMs and >300 networks 😉

wraith mica
#

I'm running vSphere 8.0.3. I was told VMDC is going out after Shift tool. Let me double check whether we are using the right version.

robust moth
#

okay, I tried on both, 8.0.2 and 8.0.3 vcenter.

robust moth
#

(VMDC download was pulled apparently...)

tame cradle
#

Hi all,
Just deployed OnTap tools for vSphere 10.2 and added vCenter. vSphere is version 7.0.3
The plugin is deployed and showing up in the menu.
We cannot start the update host data, option is there but not clickable, and everytime we right click on a cluster we get the following message:

image.png
little hill
#

Let’s see if we can get @wraith juniper ‘s attention…

wraith mica
tame cradle
wraith mica
# tame cradle Anyone?

Just curious using domain account or local sso account for vSphere? Anyhow, both should be supported. Did you contact support? They can collect and look into the logs.

tame cradle
#

Administrator@vsphere.local

#

Will contact support

inner obsidian
#

Hi all. We are planning a migration from VMware to Hyper-V at NetApp IT. Currently we use 4 NFS datastores exported to multiple ESX hosts per ESX cluster. We are trying to determine if it would be better from a performance standpoint to use SMB 3.0 or iSCSI for Hyper-V volumes (AFF 700). Any advice is greatly appreciated.

faint steeple
#

If you use anything past Windows Server 2019 or ONTAP 9.13, you will need a 3rd party backup technology for automated VM or application consistent snapshots. NetApp unfortunately has not continued support for Hyper-V on Server 2022 or Server 2025, or newer versions of ONTAP, with their SnapManager/SnapCenter line. You may want to evaluate this aspect first to see how it impacts your supported storage topology. If I had to pick from the two options you provided, it would be SMB 3.0, as it is more future proof than iSCSI due to possible SMB Direct support, would have less configuration complexity, and would be easier to manage (IMHO), however both SMB and iSCSI support ODX. I would closely evaluate Server 2025's support for the NVMe/TCP flavor of NVMe-oF, using the integrated initiator, with promised future native initiator updates to support RDMA.

#

Hyper-V has been in a odd place for the last few years with Microsoft telling everyone it was being sunset to push people to Azure, then the VMware acquisition fiasco causing reevaluation and renewed development. Even though Server 2025 is getting modern features, I would wager most of these are likely due to Hyper-V being the hypervisor for Azure Stack HCI, a hypervisor that cuts NetApp out of the storage stack. I am currently a VMware customer on perpetual licensing with dwindling support, and I haven't made a decision yet on which hypervisor to jump ship too, since I am currently supported and waiting for options to shake out and materialize. I can say that every mainstream hypervisor out there has implemented their own storage subsystem, and not all of them even support integrating NetApp. I am hoping NetApp announces renewed support for Hyper-V on Server 2025, with all of the new features, and all of the downline support via SnapCenter.

inner obsidian
fading shard
#

As far as I know support for Hyper-V has already be announced but not sure if it's public yet. There should be support for newer Windows versions for SnapManager Hyper-V in the IMT, renewed support for SMI-S in SCVMM and also talks about possible SnapCenter support.

Also a new way to migrate from ESXi to other hypervisors like Hyper-V without needing to copy data (it's converting the disk formats) should be announced very soon (actually I don't know what they're waiting for...)

glossy vortex
little hill
inner obsidian
cosmic yacht
#

We have used both for HV over the years and they both work well. It's really a matter of preference. I prefer iSCSI simply because of block and easier for us to manage. We are currently in the process of moving most of our MS SQL iSCSI to SMB because of limitations on the hosts, but again, they both work just fine.

inner obsidian
wraith mica
#

For SMB, you can use multichannel, continous availability feature etc. While adding more nodes, the process will be simple. iSCSI can provide you multipathing, igroup, etc. With AFF, you have chocie. If you use ASA, restricted to block protocols.

wraith mica
woeful ingot
#

So this was fun. Spun up a new cluster on 9.15.1 to evacuate my old cluster on 9.15.1. OTV (9.13) works fine on the old one, not on the new one. KB for the error was basically, "IDK, remove and reinstall OTV". After much head scratching, it was zapi being disabled on a new install. D'oh. It works now but I should have gone to OTV10.

high forum
#

No man. 10 is not cool for small installs

Just do

set diag
web ONTAP modify -suspended false

Enables ONTAPi

#

10 requires vasa to be enabled and once you deploy a volume through otv10 it is not trivial to uninstall if needed.

Stick with otv9 if you can. Although with ONTAP 9.15 you may need to use otv10

fading shard
#

Deployment should be easier with 10.3

high forum
#

I hope so. 10.1/2 are not “trivial” in as much as you are guessing at some ova template responses since they are not documented well

woeful ingot
#

9.15.1 is working fine with OTV9.13 after turning zapi back on. Will see if it breaks when I go to 9.16.1 later this month. busy with my esxodus for now.

wraith mica
high forum
#

Still MUST enable the VASA provider even for traditional volumes. That’s nuts! Once you deploy a single traditional data store over nfs you cannot unregister the plugin!

#

If you don’t believe me…try it

robust moth
#

true you cannot unregister the plugin as long as there is a storage backend registered, and you cannot remove the storage backend if there are volumes that were provisioned through it. It doesn't require you to enable VASA though

high forum
#

From what I’ve tested, I could not deploy any data store until vasa was registered

robust moth
#

it works without VASA provider in LOD at least

image.png
#

I guess the unregistrering is some sort of safety or something... I'm sure you can just delete the OTV server and clean up the plugin registration from the vCenter afterwards. Not optimal, but if you absolutely have to get rid of OTV I'm sure there's some way to do that

little hill
#

@wraith juniper Can you confirm the findings here? Is VASA required now?

woeful ingot
#

I'm redeploying my OTV, I think I'll stick with 9 :D.

high forum
#

So I played in the LOD last night. I could not get it to work at all using the stand alone tool (adding vcenter, adding backend storage, associating storage to center). So I removed the association and the storage. Went into vcenter and added the storage. I was able to use it after that. No vasa.

#

Got more playing to do

merry pumice
#

OTV 10.2 error trying to deploy a NFS datastore

ONTAP 9.16.1rc1, vSphere 8u3

I get to the point, where you're supposed to decide NFS versions (used to be radio buttons, now it's a dropdown), but the dropdown is empty!

Is this a known bug (didn't find anything related to this behavior)?
A workaround?

Also: the RBAC JSON file to create users is not yet 9.16-compatible (just have to add it in the JSON, though)

image.png
high forum
#

Make sure it’s licensed and the NFS server is configured and running. I start with manually adding a user to ONTAP called otv, role admin. For now skip the rbac. Make sure the vserver has
-vstorage enabled
-tcp enabled
-udp disable
-v3 enabled
-v4.0 disabled
-v4.1 disabled
-v3-64-bit enabled
-v4-64-bit enabled
(I usually set all those when I set up a SVM for VMware. Need advanced mode for some options. Enable the 64 bit file handles in case flex groups are used in the future)

I know otv should set it, but start off with a policy for your svm to allow the tax hosts

robust moth
#

note that this seems to happen even before selecting the actual storage system (see the steps on the left side) so I would rule out any ONTAP configuration issues at this step...

wraith mica
wraith mica
robust moth
wraith mica
wraith juniper
#

Generally the issue with not seeing a suitable combination of options like unable to select the desired SVM or unable to select the desired protocol ends up being due to the onboarded storage not being configured in a way that ONTAP tools expects. Like missing NFS server on an onboarded SVM for example, or something along those lines.

robust moth
wraith mica
#

Here are my top 3 reasons why I will use ONTAP tools 10.x over 9.x

  1. Single instance of ONTAP tools can manage multiple instances of vCenter servers. In VCF, just deploy to management domain and workload domains can just consume it.
  2. HA option available to support vVol deployments and SRA providing fault tolerance.
  3. Ease of deployment of vSphere Metro Storage Cluster using protect cluster feature.
wraith juniper
robust moth
#

yeah, NVMe/TCP is awesome. Too bad most backup tools in use by our customers don't have integration yet 😅 (yes, I know that SnapCenter supports it, but for some reason people often choose another backup solution 🙈 )

wraith juniper
high forum
#

Unless you’re using SnapCenter to replicate, it’s simply a protection/recovery point. If replicated to another cluster then I might accept it as a backup (second copy)

merry pumice
# wraith mica With ONTAP tools, check the SVM to ensure NFS is enabled. You can confirm from s...

NFS was enabled (including -vstorage)...

I've got 2 equally configured clusters. On one 1 manually configured an NFS datastore, now the OTV wizard immediately offered me NFS versions (though sometimes in a different sequence - NFS 4.1 above NFS 3) and I can select the 1st cluster to create the datastore. The 2nd cluster/SVM was not offered in the drop-down.
On the 2nd cluster/SVM (no manually created datastore yet), it took forever (I gave up yesterday, but tried again just now), but now it seemed to work, but only a datastore with default settings.
When I try to set QoS settings in the create-datastore wizard, the datastore creation fails.

One possible culprit for the discovery failure: I had created a VSCAdmin with the JSON file offered by OTV and found out, it could not even do a correct discovery of the "Storage Backend". I had to change it back to "admin"...

#

OTV 10.2 NFS VAAI Plug-In suggestion:

Is it possible to trigger the start of the vaai-nasd service after rolling out the plugin, i.e.
[root@esxi1:~] /etc/init.d/vaai-nasd start

#

I always have to manually start the service after rolling out the plugin before HW acceleration is recognized...

robust moth
merry pumice
#

OTV 10.2 NFS 4.1 datastore does not work:

It mentions a Kerberos incompatibility:
But as you can see, the Kerberos setting is "Do NOT use Kerberos" ...
???dafuq

image.png
high forum
#

Note: if a datastore is currently mounted, You need to reboot (or somehow unmount/remount the datastore).
Otherwise, going to each node and starting vaai-nasd before mounting should also work without a reboot (no mounted datastores)

#

@merry pumice vstorage DOES NOT require NFSv4 to be enabled at all. What is REQUIRED is the export-policy rule MUST have NFSv4 (I just use protocol=nfs -> this implies v3 and v4)

#

vstorage uses the "v4 nfs protocol" in the export-policy as essentially a back-door to use vaai calls for NFS. NFSv4 being enabled in the SVM is 100% not required.

#

I always always turn it off. too many issues/headaches. V3 is easy and works reliably

robust moth
#

yeah, if you don't actually need it (for Linux hosts, for example) then I also turn it off ... If only to make sure that the VMware admin doesn't accidentally mount a datastore with 4.1 🙂

clear palm
#

A quick question on the "new" Shift software... We have a customer who has some large NFSv3 datastores mapped up to their VMWare environment. They would like to migrate selected VMs from their datastores to Hyper-V... Is it just as easy as setting up CIFS on the SVM to connect to the Hyper-V hosts, and then presenting the existing datastore to Hyper-V? And Shift will then do its magic converting the selected VMs from one to the other? Will you then end up with a "mixed" mess of a datastore where some is VMWare and other is Hyper-V ? Or do you setup a "migration" datastore to avoid this?

vernal vale
high forum
#

Hyper-V users either iscsi luns or smb continuously available (CA) shares on ntfs. Not NFS.

robust moth
#

also you don't need an NTFS filesystem in the backend, WAFL will work just fine 👍 (nitpicking, I know 😉 )

high forum
#

Yeah…I’m not sure that would be supported. Should check IMT

high forum
#

Just use cifs on the backend. Keep it simple. It will work fine. I suspect you may run into user mapping issues possibly other issues using a ca share without cifs.

Just because you can doesn’t me you should.

vernal vale
little hill
#

Just use iSCSI and call it a day. It cleans everything up when working with MS and eliminates any locking or availability issues. Been doing this with virtualized MSSQL/Exchange all the way back in 2008. You could also run into MS support discrepancies.

high forum
#

I agree @little hill ! But there are customers out there that just don’t want to do or deal with something new to them (iscsi).

I just helped a customer fully setup Microsoft cluster and hyper-v all on ca-shares. Works perfectly fine.

Heck, way way back, I was asked to setup a Microsoft cluster to support mssql for an old VMware installation. Back then, didn’t have iscsi they bought just NAS protocols. Did cifs and ca-shares. Again, works great if needed!

little hill
#

All depends on risk tolerance. We went through the same stuff when 10GbE NFS was first intro'ed in VMware.

#

I definitely YOLO'ed my way into 10GbE NFS putting all VMs on a single datastore to maximize dedupe.

#

Notorious flow control bugs in those early UTAs

clear palm
#

But if your vmware is on NFS datastores and you want to go to Hyper-V with iSCSI, Shift will not be able to help you? Isn't the idea with Shift that you don't need to move the data? 😉

vernal vale
#

There is a oncommand Shift best practice guide. I didn’t read the whole document, just flied a little bit over, but it’s described here which volumes nfs exports and cifs shares are needed.

robust moth
# clear palm But if your vmware is on NFS datastores and you want to go to Hyper-V with iSCSI...

shift will convert vmdk to vhdx (and eventually vice-versa). So you would still have to copy the vhdx over to the iSCSI LUN somehow after converting it. There are other ways of course, e.g. if your vmdk is a single flat file with external descriptor file, you can lun create -file-path ... to convert that file to an ONTAP LUN that you can then attach to a VM as RDM (or whatever the equivalent is called in Hyper-V)

glossy vortex
glossy vortex
#

I got a question which hopefully makes sense regarding configuring SMB and Hyper-V. The netapp docs mention to create two data lifs; one per node
Should we also use these for domain communications or do we create dedicated mgmt lif for that purpose?

high forum
#

Personally, I deal a lot in secure environments.

What I do:
create auth svm
Create one lif, with default-management service policy on e0M (can’t serve data this way), auto revert true, broadcast domain wide failover policy.
Create DNS for svm
Create an active-directory svm
Create the domain-tunnel to the auth svm
Create your domain logins (security login create -auth domain…)

faint steeple
#

Hyper-V requires SMB 3.0 which is a multipath aware protocol with recovery procedures baked in. The reason you create two data LIFs for an SVM, one for each node, is so that when your sessions are terminated on one node due to maintenance or fault, there is an active path to the SVM from the client to the node that survives. Each LIF requires an IP, so clients will know about both IPs on the data LIF. SMB sessions are stateful and HA takeover drops sessions on LIF migration. A dropped session is a dead session. The management LIF will flow between the nodes. Yes, sessions will be dropped, but the management life does not use protocols that are multipath aware. All SVMs are created with a management LIF, then you should create one data LIF for each node in your cluster. If you had a 2 node cluster, you would have 1 management LIF and 2 data LIFs.

mental blade
#

Python script for Proxmox Ontap snapshots..
https://www.credativ.de/en/blog/credativ-inside/ontap-snapshots-for-proxmox-ve/

credativ®
chrinorse
ONTAP snapshots for Proxmox VE

ONTAP snapshots for Proxmox VE In modern IT infrastructure, virtualization is the key to efficient resource management. With virtualization, memory, cpu, network and storage resources can easily be assigned to and shared between virtual machines (VM). Virtualization also comes with the advantage of being able to easily change the resources assig...

ripe grove
#

Will there be some future blog post regarding this VCF Netapp option announcement ?

image.png
little hill
#

What would you like to know? @wraith juniper is our resident VMware dude, and can answer any questions you may have.

wraith mica
robust moth
#

Oh maybe you could fix the video in the Proxmox documentation here to not use Mixed security style for SMB please? Every NetApp engineer in the last 15 years has said to not use Mixed security style, now customers start asking us annoying questions about why the official NetApp videos suddenly use Mixed security style 😅

Proxmox VE with ONTAP

Shared storage in Proxmox Virtual Environment(VE) reduces the time for VM live migration, and makes for a better target for backups and consistent templates ...

wraith mica
wraith juniper
fluid kernel
#

Good job Broadcom, all docs.vmware.com links are broken and just points to techdocs.broadcom.com main site 😭
Seems like there's a lot to do for the NetApp Docs team during the next weeks.

clear palm
#

We have a customer with Hyper-V who would like to do backups to our setup with SnapMirror and that part is trivial. We do not have any HyperV in our DC, but only VMWare. Can we some how use OnCommand Shift to recover/convert a Hyper-V VM into VMWare? PS: The Hyper-V datastore is CIFS and our VMWare hosts can of cause mount NFS datastores... We have been looking at different tools that can convert Hyper-V to ESX, but it looks like they convert the whole disk and as far as i know OnCommand Shift can do this with only converting the descriptor of the disk? Does anyone know how/if this can be done without Oncommand Shift?

wraith mica
robust moth
ripe grove
#

Is there a work in future that SDDC from start can deploy management on SAN or that ISCSI will be supported ? Since without ISCSI there is no NetApp NFS Ransomware, Netapp FlexGroups etc.

wraith mica
# ripe grove Is there a work in future that SDDC from start can deploy management on SAN or t...

That's right. Currently, (with VCF 5.2) need to use VCF import tool to use with NFS, VMFS over FC. You can have a vSphere environment without VSAN and convert to management domain or import as workload domain. Note: All the clusters in a vCenter environment will be part of same VCF domain. Need to wait on VCF 9 to see what features it brings.

Broadcom documentation on this topic - https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/map-for-administering-vcf-5-2/importing-existing-vsphere-environments-admin/supported-scenarios-for-importing-vsphere-infrastructure-to-vmware-cloud-foundation-admin.html

glossy vortex
high forum
#

Probably something to do with waffinity. The number of affinities per wafl /aggr. 4 volumes per aggregate would be 8 volumes in an ha pair

#

And the expectation is one lun per volume per the link you sent

little hill
#

Didn’t H-V also drive Continuous Availability (CA) shares? That would lead to multiple redundant constituents underneath

high forum
#

I don’t think as much. The link referred to above was for SAN connections. Going to ca-shares can still use multiple volumes but it is all file based. It suggests enabling odx (offloading). There are a few other tweaks. I’ve run through the ca-shares config one time with a customer. Works great with ms cluster

little hill
#

<@&1131581028138487969> <@&1131580957393166367> <@&1132035203188142211> A year has gone by since we did our [first poll](#╰・virtualization message) on this topic, but we wanted to revisit it a year later and see where you all ended up. Did you make any decisions? Changes? Did you start down a path and revert to VMware after discovering blockers? Fill us in! We're making some product decisions and want to hear from our community about what your priorities are heading into 2025! Check out the poll below and give us your candid feedback about what we can do better to enable you with any of the hypervisors out there!

spice steppeBOT
#
datacenterdude
Question

Are you looking to switch hypervisors? If so, which one(s) are you evaluating and/or transitioning to?

Choices

🇦 Sticking with VMware
🇧 Azure/Hyper-V
🇨 RedHat OpenShift (OSV)
🇩 OSS Alternatives (Proxmox, XCP, KVM)
🇪 Nutanix
🇫 Other
🇬 Still undecided

Ends

Never

Mode

Multiple votes allowed

glossy summit
#

I was really hoping to see market pressures create some real competition in the virtualization space - but to my knowledge, it hasn't.

One of the biggest flaws I see is that so many virtualization "solutions" are designed by people who forget that lots of software as a service is infrastructure that has to be implemented and managed. E.G. DNS, DHCP, Databases, etc.

Yeah, if you deploy in a cloud, you can just use their SaaS, but not if you deploy baremetal. In lots of companies and environments I've worked in, this is a big strength of vmware - if you have an internally routable IP, and at least one physical server, you can bootstrap your infrastructure. You can even host vsphere on the same infrastructure it manages.

Many other solutions come with a huge list of requirements, hardcoded DNS entries that you have to satisfy, databases you have to provide and connect to that can't be hosted on the virtual infrastructure - it really increases the complexity of their deployments.

cosmic yacht
#

what deployments have hardcoded dns or have a list of reqs that are hard to meet, especially one that you cant host on the same infra

glossy summit
#

OpenShift was a big one:
https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/installing_on_bare_metal/user-provisioned-infrastructure#installation-dns-user-infra_installing-bare-metal

api.<cluster_name>.<base_domain>.
api-int.<cluster_name>.<base_domain>.
*.apps.<cluster_name>.<base_domain>.
bootstrap.<cluster_name>.<base_domain>.
<control_plane><n>.<cluster_name>.<base_domain>.
<compute><n>.<cluster_name>.<base_domain>.

Of course, OKD (FOSS OpenShift) had a whole host of other issues too, so... that might not be the worst one

#

Like the fact that OKD just automated the copy and paste of the OpenShift docs, and did zero validation on them - so we built out a cluster on the premise that we can use virtualization with bridged networking, in OKD, since it was seemingly well documented... only to find out after it was deployed, that is relied on an operator that was only available with a paid OpenShift subscription or something, and so unavailable in OKD. And you couldn't just manually create a bridge on the linux host - it's not seen by OKD. It was a mess

#

Proxmox would be a good alternative, but to date, I haven't found or heard of any solutions for the fact that it uses passwordless root access to all the nodes in a cluster. IMO, they should have been using an API or service account based method that doesn't require root having SSH access to nodes, let alone using passwordless ssh-keys

wraith juniper
dense frost
#

We may be glued to VMWare but there is a big right-sizing initiative and a big shift for specific workloads to possibly shift to Nutanix

#

forcing easily a 1/3 reduction in billing to VMWare.

#

But, we have the time and flex to work with it because we locked in rates//pricing for X years when Broadcom began sniffing out vmware.

#

I will say, this put a damper on possibly moving off of our Cisco MDS's as well for FC to Brocade, because apprehension.

inner obsidian
#

Hi All. Good Morning. Had a question regarding multi-channel with SMB 3.0 used for the shared storage in Hyper-V. Currently we have 4x SMB shares on 4 different ONTAP nodes, accessed via LIFs local to the node.

There are currently accessed from Hyper-V cluster as below:

\\LIF_node1\SMB_share_node1
\\LIF_node2\SMB_share_node2
\\LIF_node3\SMB_share_node3
\\LIF_node4\SMB_share_node4

Should we instead use a DDNS record with the SVM name pointing to LIFs on all 4 nodes and mount as below ? Please advise...

\\SVM_name\SMB_share_node1
\\SVM_name\SMB_share_node2
\\SVM_name\SMB_share_node3
\\SVM_name\SMB_share_node4

frank crystal
glossy summit
frank crystal
glossy summit
#

Thank you! :)

frank crystal
glossy vortex
#

Has anyone encountered a Hyper-V SMB issue like this before?

We're working with a customer and have followed the NetApp documentation meticulously. LIF migrations work as expected, and takeover processes seem fine. However, a few minutes after a giveback, the virtual machines enter a paused state and require a reboot to recover.

We've opened a case with NetApp, but I wanted to see if anyone else has experienced this issue or has any insights.

cosmic yacht
#

never seen that. I don't remember ever having an SVM pausing, let alone requiring a reboot/restart to serve data

glossy vortex
#

The hyperv vm's on the data store enter a paused state

tame cradle
#

Hi all,
I deployed 10.3 yesterday in a site with a six-node cluster and running VMware 7.0.3.
Deployement went very smooth and adding vCenter and back-end storage as well.
Did the association.
Opened the plugin, no data present. Did the discovery twice, still no content.
Any ideas?

OTV10.3-4.png OTV10.3-3.png OTV10.3-2.png OTV10.3-1.png
mental blade
#

Proxmox 8.4 released with a new backup API as one of the new features.
https://forum.proxmox.com/threads/proxmox-ve-8-4-released.164821/

Proxmox Support Forum
Proxmox VE 8.4 released!

We are excited to announce that our latest software version 8.4 for Proxmox Virtual Environment is now available for download. This release is based on Debian 12.10 "Bookworm" but uses the Linux kernel 6.8.12 and kernel 6.14 as opt-in, QEMU 9.2.0, LXC 6.0.0, ZFS 2.2.7 with compatibility patches...

frank crystal
#

Anyone using vvols with ontap and Veeam currently? Any musings? Have an unsatisfied customer with Nimble storage using about 400 vvols and is very pained by the thought of going back to VMFS

fading shard
# frank crystal Anyone using vvols with ontap and Veeam currently? Any musings? Have an unsati...

We have absolutely no vVol customers afaik. 😬
Over 1000nodes but most are doing NFSv3, some VMFS (especially with the introduction of SM-as, and also some NVMe_oF here and there).

I've always wanted to try vVols but despite the advertised benefits it was always too complicated, too risky (because of the single VASA provider VM we had earlier) or even buggy compared to easy-peasy-NFS datastores. I think I'll give OTV another year before trying again.

wraith mica
wraith mica
tame cradle
#

Will try!

little hill
#

I also saw reports from Duncan Epping and William Lam this week that the Free ESXi has returned with an embedded non-expiring license. Looking into the details of what that entails

civic summit
tame cradle
#

This worked, but still no data showing up

high forum
#

Just curious…what user are you registering with for vcenter? Is it possible the user doesn’t have the correct permissions?

Another “test”… if you have aiqum installed, use the same user to have aiqum check in with vcenter. After it scans it should at least see the VMs and if not it may be a role issue

#

And I’m not saying this is the issue. Just think something it could be

tame cradle
#

administrator@vsphere.local. Did not want any issues there. Use a different account later

high forum
#

Do you have AIQUM? Did you try scanning from there?

tame cradle
#

Wil try tomorrow

wraith mica
# tame cradle

Did you create a datastore using the vCenter Plugin? Any VMs hosted on that datastore?

tame cradle
tame cradle
wraith mica
tame cradle
#

Hi, Thanks for your reply! watched the video. We do not use VCF, nor FC (NFS only). So it only works for storage created with the OTV plugin? I thought it would do some sort of inventary as well.

wraith mica
tame cradle
#

Ok, will try to create a new datastore (NFS) tomorrow.

boreal thunder
#

Hey All - it's been about... 4/5 months since any mention of SMI-S and SCVMM had happened for Hyper-V.. Nothing was publically posted from Ignite as expected.. There was mention of it being "Announced internally but not publically"... Has anything changed? I'm still not seeing anything available for download regarding new SMI-S Provider releases or updates for WUHU, SnapManager for Hyper-V, etc anywhere.. Just some toolchain around migrations (I think it's called Shift?)

boreal thunder
#

Guess not?

little hill
#

Sorry, @boreal thunder, was traveling all last week and missed that one. Let me go check with engineering and get back to you.

high forum
#

@boreal thunder you might want to visit the Netapp support site today. SMI-S 5.2.7 was released

#

This release is not compatible with Windows Server 2012 or System Center Virtual Machine Manager (SCVMM) 2012.

•SMI-S Provider 5.2.7 supports Windows Server 2025, Windows Server 2022, Windows Server 2019, Windows Server 2016, SCVMM 2025, SCVMM 2022, SCVMM 2019 and SCVMM 2016.

little hill
#

TMAC beat me to it! SMI-S landed yesterday.

Overview

NetApp SMI-S Provider 5.2.7 enables you to manage and monitor storage systems and to manage LUNs and volumes of storage systems, CIMOM configuration settings...

#

For SMHV, contact your NetApp rep or VAR/Partner and we can work thru them to get something approved.

Shift toolkit is currently in Preview with an expected GA soon in Q2

boreal thunder
#

Woo!

shrewd stone
#

awesome been waiting for this as well

boreal thunder
#

Huh - Interesting differences between this and the previous release. Can't install this on my SCVMM server anymore - requires a separate server to host just the SMI-S Provider

#

But otherwise, the release notes don't list any changes beyond what OS's are supported

#

Some wickedly old OpenSSL/Libxml2 vulns I guess

little hill
little hill
#

NetApp added to SCVMM 2025

#

Note: Requires ONTAP 9.15+ and SMI-S v5.2.7

boreal thunder
#

This makes me excited ^_^ @little hill any chance you know or can find out why SMI-S Provider needs to be on a separate server than SCVMM now?

boreal thunder
#

Hrm - guess not

#

lol

white scaffold
#

@little hill 🙂

little hill
#

Let me dig in to this and get back to ya

distant kelp
#

Anyone have any good experiences with migrating from OTV 9. 13 to OTV 10.4 using SRM, I notice documentation talks you thro migrating but not so much on the cleanup of the old 9.13 plugin etc. Interested to hear experiences.

little hill
#

@wraith juniper

wraith juniper
distant kelp
#

To confirm. You'd be best doing the removal of 9.13 after completing the installation of OTV 10.4. No vvols in use so should be straight forward I guess, procedure should be:

  1. Deploy new OTV at each site.
  2. Register Vcenter for each site on respective OTV.
  3. Add local storage to each OTV.
  4. Clear down and reset sra, install new 10.4 sra at each site.
  5. Rescan both sras in srm
  6. Check snapmirror relationships etc can be seen.
  7. Proceed to remove 9.13 plugin from vcenters.
  8. Power off and delete 9.13 OTV at each site.
robust moth
#

vvols are soon a thing of the past anyways... best to migrate away if you're still using them

white scaffold
#

Reddit is all over that one

robust moth
#

yeah. I think we have one customer (out of many hundreds) or maybe two who use them....

winged needle
#

Hey there,
out of curiosity - are there any news regarding Proxmox and block storage (iSCSI, NVMeoF) on OnTap regarding Snapshot support?

little hill
#

News, sure! All protocols are now officially supported and on our IMT, both block and file.

There’s nothing stopping a really sharp Python coder from writing some scripts to trigger dirty snaps of VMs currently.

I can tell you that it’s a side project I’m working on myself. We’ll see where that goes.

Engineering has stated no current plans to build a dedicated integration like OTV for VMware within the PVE ecosystem yet. That’s not to say never, just not anytime soon.

winged needle
#

yeah, ok. That's more or less the way I currently do including Proxmox UI integration. Just wanted to know if there's something official to avoid maintaining such solutions - thanks so far!

winged needle
#

So, currently I think the easiest way for "Endusers" is to integrate such solutions in parallel to the Proxmox UI and fetch the upstream capabilities. Selecting on the capabilities deciding if this should be presented to users. A further tooling then orchestrates the backend stuff and holding the meta data in a dedicated database (currently a sqllite).

ontap_snapshot_iscsi.png
mental blade
#

@winged needle That looks nice, is that something you might have available for the public? or is it still a work in progress?

winged needle
#

Thank! Unlike my ProxLB project, I’m not yet sure about it. I’m currently in a way of too many opensource projects and do not have that much time for maintaining additional tools.

Quality over quantity. Currently, it’s not planned but that might change

winged needle
#

but maybe an idea to integrate this into ProxLB as an additional feature next to DRS and DPM

mental blade
#

ok, thanks for the info 🙂

boreal thunder
#

Hey @little hill - Any updates on my previous question about SMI-S needing to be on a separate server? Figure it's been about 10 days or so so I'd poke around and see what's what

little hill
boreal thunder
#

Hm.. I don't believe it was in the documentation for the last release - unless I'm completely blind

little hill
mental blade
distant kelp
#

Any technical reason why Vcenter HA isn't supported with OTV, specifically OTV10, think it applies to 9 too. Flip over between master and slave should be invisible to anything using it, in theory. If it's not supported is it possible to get fpvr? A lot of organisations will use vcenter HA. Thanks.

robust moth
#

for an HA deployment of OTV it shouldn't really matter. I think what they mean by "vSphere HA is not supported" is that if you have a single-node OTV installation and think that's good enough because you're using vSphere HA for that VM

distant kelp
#

I also wondered that. But the wording made me question it as it calls it out specifically as "Vcenter HA is not supported"

#

Also the HA deployment specs of OTV seems massive. Wanted to try and keep it simple with a small deployment with regular backups.

robust moth
#

IMHO you only really need the HA deployment if you're using the VASA provider, every other config should be fine with a single instance. then again I'm not a vSphere expert so don't quote me on that 🙂

high forum
#

This is exactly what I do as well. As long as VASA isn’t required for vvols(being deprecated by VMware) and I teach the customer not to use storage profiles, then VASA really isn’t needed and a single instance is fine

For what it’s worth, it (10.4) makes deploying nvme/tcp stupid easy:
set the variable on esx (cli or GUI)

esxcfg-advcfg -s 0 /Misc/HppManageDegradedPaths
(reboot needed)

setup the nvme/tcp adapters on both ends and test ping for jumbo frames

Use Otv to create namespaces

distant kelp
#

Otv is literally being deployed to facilitate sra for srm

high forum
#

That too! Not sure if the multi-node is necessary for SRA/srm or not

wraith juniper
little hill
little hill
#

Nevermind, looks like the real @raven mantle just joined! Perhaps they will give us a demo of the new project!

raven mantle
#

Hi 👋 Being in the spotlight is not really my thing. I just hope that BareProx will work and survive in the wild (outside my lab).

obsidian hedge
#

sure it will! 🤘

robust moth
mental blade
#

yes, it's a separate machine you install, you can install a docker image if you want, but its totally standalone and talks to proxmox and ontap to handle backups, and restores, with its own web based gui

robust moth
#

is there a reason why it can't be integrated into the Proxmox UI itself? Doesn't Proxmox support plugins, similar to vCenter? That would be much easier for the admins if they can manage everything in one place

mental blade
#

Might be possible, well see what the future has in sight.. this is a first release just to get it working.. so we can get our fantastic storage the right way with proxmox.. 🙂

obsidian hedge
#

We know that it is requested to create an integrated part into Proxmox UI but we are not there yet.

raven mantle
#

The menus and tabs under “Datacenter” in Proxmox are hard-coded in the pve-manager source code (JavaScript and Perl).
They are defined as static lists or classes when the UI is built and are not designed to load new menu entries or panels dynamically at runtime from external JavaScript files. Unlike some plugin frameworks, Proxmox does not provide an API or plugin system for injecting custom tabs, menus, or sections into the Datacenter area via user scripts.
So that's the main reason. At least to my knowledge.
A workaround is to create a new pve-manager with the menus/"plugins" added in source code. And No! I won't do that.

Maybe I'll get deeper into hocking on to menus and functions in the Proxmox-gui but.. That can be a nightmare every apt-get upgrade.

The application is created in VS 2022 and it's C# MVC. It Can probably run on whatever that handles .NET 8
Currently there is a docker image on docker.io for debian. I'm running the application on a small Debian-vm with docker-compose.

wraith mica
#

For customers staying with VMware and looking for Private Cloud, VCF 9 is now generally available.

#

Interesting features are Boot from NVMe-oF, Memory tiering, etc. VCF Operations and Automation now becomes as required components of VCF.

robust moth
#

What servers do support boot from NVMe-of yet? In the NVMe Boot Spec Task Group, the only server vendors I think are Dell and HP

woeful ingot
#

Also bye bye vVols

#

But the best part is you can now greenfield the management domain without vSAN.

white scaffold
#

@wraith juniper published a blog about it yesterday:
https://community.netapp.com/t5/Tech-ONTAP-Blogs/The-Future-of-vVols-and-the-NetApp-Advantage/ba-p/461642

The Future of vVols and the NetApp Advantage

Many of you have seen VMware's announcement today regarding the general availability of VMware Cloud Foundation (VCF) 9.0. VCF 9.0 delivers many powerful new capabilities as well as changes to existing supported technologies. Among other things, one change you will see is that “starting with VCF 9.0...

woeful ingot
#

Nice

#

I went and did the VCP-VCF. Remains to be seen if it was worth the effort. VMUG license portal has been down for a while.

little hill
# woeful ingot Also bye bye vVols

There's a small teeny-tiny part of me, down in the subcockles, that feels empathy for anyone that bit on the Pure + vVols solution. Because they're in trouble.

woeful ingot
#

But only just a little 😆

lethal shuttle
fading shard
# boreal thunder Hey <@277739185786454016> - Any updates on my previous question about SMI-S need...

"Best Practice:
NetApp recommends using a dedicated Windows Server to install NetApp SMI-S Provider, - do not be installed on the SCVMM server to prevent interference, optimize performance and simplify troubleshooting."
From: https://community.netapp.com/t5/Tech-ONTAP-Blogs/Unlocking-the-Power-of-SMIS-with-Hyper-V-A-Comprehensive-Guide/ba-p/461896

Unlocking the Power of SMIS with Hyper-V: A Comprehensive Guide

NetApp Storage Management Initiative Specification (SMI-S) is a powerful tool for managing and monitoring storage systems. NetApp SMI-S leverages the Web-Based Enterprise Management (WBEM) protocols to provide a unified interface for various management tasks, including the handling of LUNs, volumes,...

little hill
#

@wheat gate Just now ran across your video, appreciate the highlights and hat tips! Feel free to share those in here if you make more! https://www.youtube.com/watch?v=efd1Q6u5WGA

YouTube
Raspberry Pi Cloud
NetApp und Proxmox VE ❤️ Eine geniale Kombination! Jetzt suppor...

🚩 Zur NetApp/Proxmox Doku: https://docs.netapp.com/us-en/netapp-solutions/proxmox/proxmox-ontap.html
🚩 Weihnachtsgeschenke fällig? - https://amzn.to/3VAJN70

#netapp #proxmox #storage

Kapitel
00:00 Einleitung
00:56 Lasst uns starten
02:48 NetApp NFS und Proxmox verbinden
06:57 Active Active iSCSI Multipathing
08:52 NetApp iSCSI und Prox...

▶ Play video
wooden geyser
#

very small 2 node Hyper-V cluster to run off NetApp, 30 VMs and not a lot of IO.. should we use iSCSI or CIFS (SMB3)? ..currently they have NFS4 with VMware.. prefer to keep it as simple as possible

little hill
#

ISCSI is always my default with anything Microsoft, especially VMs, and even more especially if you’re going to be doing any direct to guest LUN mounting.

high forum
#

Iscsi, but please make sure you are setup correctly with appropriate multi pathing!!!

knotty rivet
little hill
woeful ingot
#

neither of the bugs I filed for their ovf importer were fixed. :/. I guess I keep making my own importers. But iSCSI improvements alone are worth the bump.

high forum
#

But can it easily do iscsi boot? The last attempt was hideous

woeful ingot
#

That's something I haven't even attempted.

high forum
#

I can easily tell my dell server (in the BIOS) to boot from iSCSI. The (I think) INT13 handler does its thing and passes off to ESX and installs just fine. When I do the same with Proxmox, it just does build the "stuff" inside linux to accomodate a boot. All the help online is not geared towards the hardware I am using. I am sure I can figure it out if I get time, but rather, it should not be so difficult in 2025 to make it happen...especially with a "new" version

woeful ingot
#

My PVE test boxes are all the old NUCs that used to run my VMW management domain. I’ve had them booting from iSCSI before so that might be fun to revisit.

clear palm
#

Just a quick "rookie" question about VMWare... had a customer with a little 3 host cluster running vCenter on one of the hosts... this node ofcause had memory issues and triggered a purple screen of death... the vCenter was gone and not restarted on another host, so the HA-stuff also didn't happen for the other VMs on that host... is this the way it's supossed to "happen"? Is the only solution to this problem vSphere FT ? Or is the a way to setup VMWare so that it will startup the vCenter on another node?

#

...also for whatever reason the server didn't just reboot automatically, it stayed on the purple screen... is there a way to make it just reboot automatically at this point? (I realize that the server might not boot because of the faulty memory, unless it's disabled...)

robust moth
#

no. The vCenter should definitely restart on another node. Although it might take a few minutes until that happens because of datastore locks, timeouts, etc.
This is definitely how it is supposed to work, but I can imagine maybe the other nodes did not have enough RAM to re-start the HA VMs... have you allowed resource overprovisioning ("Admission Control") in the cluster?

robust moth
#

And the HA is independent of the vCenter host, so even if the vCenter didn't restart on another node for some reasons, the other VMs that have HA enabled should still restart

clear palm
#

Well there are actually 4 nodes in the cluster, but normally one two are powered on because of DPM is enabled... I am just thinking that when vCenter is dead and not restating, there are noone to control things... But I guess the clustering for some reason didn't restart the vCenter... because it was down for several hours... I think we have to do a few tests on this, so make sure it works

robust moth
#

yeah, definitely check the logs for the VMs on why they couldn't restart. There are multiple possibilities.

  • not enough RAM or CPU resources and overcommitment disabled (the "slots"-calculation in the HA settings is weird, it's usually best to just disable admission control)
  • Inaccessible datastores
  • Communication issues between the hosts
  • ...
clear palm
#

It does state the vCenter will try to restart the VM and not the Cluster 😉

Screenshot_2025-08-25_at_10.47.09.jpg
robust moth
#

check this setting

image.png
#

if you don't set this to "disabled", the hosts will only restart VMs if they have enough RAM and/or CPU capacity for the VMs, which can be tight in smaller clusters

clear palm
#

Is was/is set to disabled...

#

Of cause the DNS servers were also placed on this server 🙂 Which of cause causes issues... so I have now created a rule that doesn't allow them to be placed on the same host...

robust moth
#

that shouldn't cause HA issues though? Unless you have mounted NFS datastores via hostname or configured export-policy rules for hostnames?

#

but yeah, definitely check the log files (both the ESX logs of the surviving host and the vmware.log files from the vCenter VM)

clear palm
#

nope we use IP's (we learned the hard way) 🙂

#

But I guess we need to schedule a service window and simply test this again to make sure it works as expected...

clear palm
#

While we are at it... is there a "known" way to add new ESXi hosts to a setup with OTV 10. and SnapMirror Active Sync? When we set this up we used the OTV, and it created the iGroups etc.. now we need to add and replace some hosts... do we still need to add the ESXi hosts to the igroups? Or can OTV help us out?

clear palm
#

I think I figured it out at last.. you have to change the "Host Cluster Relationships"... and it then creates the new igroups for the new hosts...

#

Why does OTV keep on adding other datastores into the Protected Cluster? (We have a few datastores from other Clusters that are not mirrored... ) they are added automatically into the Protected Cluster causing it to go into warning state... pretty annoying

clear palm
#

It boggles the mind how OTV version 10 has besome so complex that it takes about an hour to upgrade... K8s cluster f... if you ask me

high forum
#

I always complain about how long it takes to install. That first power on can take 30 minutes

clear palm
#

Just attempted an update from 10.2 to 10.4... took forever... of cause failed despite following all guides to the letter... I have wasted the good part of 4 hours waiting for it to stop start and reboot...

high forum
#

please open a case! I am usually doing this in secure spaces and cannot upload anything.

little hill
#

Anyone have any thoughts they'd like to share from the VMworld Explore keynote this morning?

white scaffold
#

Well that sounds like crickets 🦗

agile hazel
#

I’m in Vegas, wondering why the NetApp logo is red.

#

Won this yesterday.

agile hazel
little hill
agile hazel
#

I was just in a session and the percentage of orgs was 69%. Crazy…

agile hazel
little hill
agile hazel
#

Stone cold!!

little hill
#

Actually, that's a photo of me looking across the aisle at the Pure booth

agile hazel
little hill
#

Go find the short girl at the entry stand of the booth named Sam and tell her Nick said to give you one

#

I’ll be over there this afternoon and trying to get one myself

agile hazel
#

Sam was MIA.

little hill
#

I got mine… where you at?

agile hazel
#

Lol in an AI session I’ll be down in 5.

agile hazel
#

Got my hoodie, thanks buddy lol.

random reef
clear palm
clear palm
#

I think I might have "figured" it out... I did a test (yes I have too much time on my hands) where I installed 10.2, then enabled VASA and then did the upgrade to 10.4, and it worked first time... once 10.4 was installed you can now disable VASA in the web gui of the OTV... I will give this approach a go at the customer tomorrow...

fading shard
#

Does your customer use vVols?

clear palm
#

Nope does anyone? 😉

fading shard
#

Well then you don't need to enable VASA. Or did I misread that?

clear palm
#

The problem is that on OTV 10.2 you cannot disable the VASA services, and apparently if you haven't registered it, it will fail the update to 10.4... then when you are on 10.4 you can unregister it, and disable the VASA service... strange pice of very large software 🙂 I think it's sad to see that they use CEPH as storage when they could have added a whole ONTAP Select in that space 😉 For what it's worth, 10.4 seems to have a lot of fixes compared to 10.2, and the GUI even seems faster... 🙂

fading shard
#

Ah, I never updated the new OTV.

#

Still hoping it becomes a bit more lightweight. Since vVols are becoming deprecated everything VASA related could get removed in the future.

#

But I guess having a redundant VASA provider has been the main reason for the new architecture. If that's gone I don't really see the need to deploy multiple OTV instances. (Never used SRM, no idea if that needs an HA-capability.)

clear palm
#

I did another update attempt today, after I have registered the VASA provider... ended up at the same result again... and had to revert back. So tomorrow I will attempt a reinstall and hope it will pick up the SMas setup as I add the storage clusters to the new installation....

clear palm
#

OK, just a little update on the OTV update issue... We ended up removing the old 10.2 plugin from VMWare and stop the OTV VM... Then install 10.4 as a new installation. This is what NetApp Support instructed us to... The 10.4 installation completed OK, and we have the plugin... yet the existing SMAs setup isn't "discovered".... we then reported this to support... after a week they have now come back to suggest to tear down the SMAs mirrors and re-configure it... to which I just told them "hell no!"... This is a production system so we will not compromise the system, just because NetApp doesn't tests their software before releasing it... yeah.. a bit disappointed

white scaffold
#

@wraith juniper ^ see above ^

wraith juniper
#

Good morning!

#

You shouldn't have to re-do the SMas mirrors. At least, that is not the intended behavior. You will likely have to onboard both the source and destination ONTAP clusters, then run a manual discovery from the OTV main page in the vCenter UI. Watch for the tasks to complete in the vCenter task list. Once they are done, it should see them.

#

As a side note, moving forward, as we deprioritize vVols and VASA, we are working on slimming down the product. You are correct, the large size and scale of ONTAP tools is there to serve tens of thousands of vVols. As that becomes less of an issue, we will no longer need to be quite so stout.

clear palm
clear palm
# wraith juniper As a side note, moving forward, as we deprioritize vVols and VASA, we are workin...

I am so happy that I totally have ruled out vVols from day one... first time I saw it in action, I remember the long volume names of the UUID and I could only see nothing but pain in a debugging situation 😉 Glad to head that vVol is finally "dead" and you can slim down the OTV... to be honnest it also "bugged" be that if you turned off the vVol services in OTV you get a big red banner in the GUI as it something was "broken"... so at some customer I just configured vVol to get rid of that message 😉

clear palm
#

Long shot here 🙂 Does anyone know if there is a way to re-register the vCenter plugin with OTV 10.3 ? We have tried to do a modify on the vCenter, which didn't help... we would rather not remove it because it has associated storage systems... I seem to remember that the old OTV had a URL https://otvip/regsiter.htm or something like this... but apparently this has changed in version 10...

high forum
#

There is a swagger interface for the REST API

clear palm
#

Yeah I had a poke arround, but I cannot seem to find anything that will kick off the register... wonder if you could do it yourself via the MOB? I have another system where the OTV plugin is installed, so maybe it would be possible to export the records, modify it to fix the new IP etc.. and import it via the MOB... but then again, why doesn't NetApp just have this as a button in their OTV Management GUI? 😉

high forum
#

It looks like it is call "add" and not register

#

​From the swagger interface -> /virtualization​/api​/v1​/vcenters -> Adds a vCenter.

#

And there is a "patch" that says: Update vCenter details (json-patch) Fields supported: username, password, storage-backends, plugin_desired_version and vasa_provider

clear palm
# high forum It looks like it is call "add" and not register

Well if we look inside the OTV Management GUI we have a vCenter there already... and I think (but will have to verify with NetApp) that the "Add" you are refering to is actually this.. and I would rather not remove this vCenter because it has some storage systems "attached" with SMas configured.. and this is excatly what we are trying to retain because OTV from 10.4 and back isn't able to pickup already created SMas relationships... only from 10.5 this is possible, and only if created by 10.5....

high forum
#

There is also a patch function

oblique ravine
#

Fujitsu delivers Proxmox support 24/7 https://www.fujitsu.com/emeia/about/resources/news/press-releases-fsas/fsas-technologies-teams-with-proxmox-to-launch-data-sovereignty-focused.html

Fsas Technologies Teams With Proxmox to Launch Data Sovereignty-foc...

Fsas Technologies, Fujitsu’s advanced data center solutions provider, today announced a reseller partner agreement with the open-source virtualization leader Proxmox, providing European companies an alternative for those who desire more control, flexibility, independence, and increased data sovereignty.

orchid gorge
#

Fsas Technologies* fingergun

novel peak
#

Am I asking too much out of the Ontap simulator? I just want to stand them up and leave them up and use them to develop some sre tools. Seems as soon as I get them setup, a few volumes, mirrors, etc. then for any number of reasons they crash and always have a volume recovery needed.

VMWare Workstation,
Tons of Memory,
Expanded root volumes,
Added disks and extra aggregates,
Disabled snapshots, etc

robust moth
#

yeah they're kind of fragile. You can make the NVRAM persistent with a bootarg, it makes them slower but also more stable

novel peak
robust moth
#

try setting setenv bootarg.vm.vnvram full in the loader

woeful ingot
#

That and super-size the root volume

little hill
static zinc
#

When do we get Proxmox support for ONTap Select????? FPBR has been submitted.

obtuse pecan
silk copper
#

We're also waiting for support for deploying ONTAP Select to Proxmox

static zinc
#

@everyone we need Proxmox support!

white scaffold
#

What do you need?

static zinc
#

we need Proxmox support!

little hill
#

Define “support” because it IS “supported” and all protocols are on the IMT.

If you’re looking for “integrations” like VSC/OTV, we’re still scoping what that would look like.

#

“Support” happened back in Aug/Sept 2024

#

I think what you’re after is the ability to deploy the ONTAP appliance into an LXC container via PVE

#

THAT is more of an integration and part of that work that’s being scoped out.

#

@wraith juniper can relay this feedback into engineering

mental blade
robust moth
#

hmm.... Looks like at least the summary in the README is AI-generated 😂

mental blade
#

yes, som of the documentation got som "external" help.. 😉

little hill
#

I tried to vibe code some python to do something similar myself.

wraith juniper
#

Feedback received. Appreciate everyone's input.

static zinc
#

@little hill More than anything we want to be able to deploy from NetApp Deploy to a Proxmox host. Plan and simple deployments. Single node deployed to VMware via Ontap Select Deploy. We want to be able to do the same to a Proxmox host. Plus have support from NNetApp when we break it.

tulip fog
# static zinc <@277739185786454016> More than anything we want to be able to deploy from NetA...

Exactly @static zinc - a Netapp engineer (Sean Hatfield) we spoke to was able to get the Ontap Select VM running on Proxmox by deploying the VM from the backend (not via Select Deploy), but there remains an issue where Netapp Select Deploy won't talk to the Proxmox host as it would a VMWare host or Rocky/RHEL Host. The issue seems to be mostly adapting Deploy to speak to a Proxmox stack like it does the other "supported" hypervisors, and for Netapp to bless running on Proxmox end to end. At the end of the day, we are looking to be delivered from VMWare, not ready to pay the same amount to switch to RHEL, and Proxmox has better centralized management than Rocky KVM. Please, Netapp, please help deliver us from the cost overruns of RHEL and Broadcom. And @little hill - no interest in running Ontap Select inside a container - just a standard KVM VM.

little hill
#

^^^ @wraith juniper

#

I think this might be something NiMo would be up for…

wraith juniper
#

I think Select Deploy is bigger than a NiMo project. The problem with Proxmox and Select Deploy, as it always is with all things Proxmox, is they ripped out Libvirt and replaced with their own control plane. Yes, Proxmox runs on QEMU/KVM, but the peice that all the integration components talk to is missing. So, we might design something for generic KVM, Oracle Virtualization, HPE VME, etc... But it won't work with Proxmox.

That said, we are not ignoring Proxmox.

I'll talk to the ONTAP Select PM and see what he says.

woeful ingot
# tulip fog Exactly <@736659754012770405> - a Netapp engineer (Sean Hatfield) we spoke to wa...

I use single node OTS in nested virtual labs, so I automated that bringup in ansible on proxmox (without deploy) as part of a larger project to move my homelab from broadcom to proxmox. I have done manual deployment of HA pairs on proxmox so I know they can work as well. As you said its a matter of getting deploy to speak proxmox apis. Until then my main lab clusters are on Rocky. It works fine there, but I would also like to bring those hosts into the proxmox cluster someday.

#

Until then, please get your account teams to file those PVRs. It may feel like it goes into a void, but it documents customer demand and feeds into product planning.

glossy vortex
#

IMT doesnt show nfs support for proxmox 9.0 but does show nfs 4.1 and 4.2 support with Proxmox 8.4 and older

I assume this just means it hasnt been tested or validated by netapp but will work just fine? Also strange it doesnt validate nfsv3

image.png
little hill
#

That validation is in the works

glossy vortex
#

We are preparing a proxmox proof of concept with nfs specifically in mind; looking forward to see how that goes.
Might be good for of our small client use cases and isnt a headache to manage.

little hill
#

I need to correct this, slightly. The current qual under way is for 8.3/8.4. I'm checking to see if we can also consolidate qual'ing 9.0. Bear with me.

#

@glossy vortex

wraith mica
#

Shift 6.0 is now available at Support Tools. AI Chat bot and Storage Manager for Proxmox VE is included as Tech Preview. Storage Manager for Proxmox VE supports NFS (Provisioning, Backup, Restore and Clone) and iSCSI (Provisioning, Backup). Known limitations are NFS DNS not supported with this tool. Need to perform discovery between the operations. Give a try and please provide feedbacks to make it better. If need Storage Manager for Proxmox VE alone as standalone (without Shift tool), it can be downloaded from NetApp Console.

Shift_6.png SM_Proxmox_VE.png SM_Proxmox_VE_Console.png
rotund tendon
#

I'm researching moving hundreds of vSphere 7.x VMs (NFS datastores on NetApp C800s) to Proxmox over the next year.

The Shift Toolkit looks like it could speed up individual VM migrations quite a bit which would be great.

Are there plans to add Proxmox migration (rather than disk conversion) to the Shift Toolkit?

fading shard