#┊・networking

1 messages · Page 1 of 1 (latest)

ornate vault
#

Hi! Do you have any good documentation or blog article on how the feature "Route Caching" does work on ONTAP? I think in the past it was named "fastpath".

pale wagon
ornate vault
#

How does routing now work without fastpath? Just normal route table lookups?

pale wagon
#

Route caching / Routing Table per SVM.

ornate vault
#

okay

#

and route caching is just like a FIB?

#

(and thanks for you really good answers!!)

pale wagon
#

Had to look that one up. :). but yes. looks like.

tidal tundra
#

do the X91152A's have different ASICs across the ports or is it all one ASIC? Just wondering if to use e0a and e0c or if e0a and e0b would still be acceptable?

warm island
idle lava
#

AFF400 ha-pair 4 100GbE interfaces open. Can an ifgrp span nodes/ports in a ha-pair? I have an mlag setup and the dfgw for the target vlan is using VRRP. The plan is for node1 port 0d and node2 port 0d to one switch and node1 port 0c with node2 port 0c the othe switch using active-active lacp in a single ifgrp if possibile. Targeting more on redundacy and failover versus aggreagtion. I had a ticket open and allowed it to close as there was no consenus among the NA team. Thanks for any guidance.
G

warm island
#

so node1 has an ifgrp (let's say a0a), made up of e0d and e0c, and node 2 does as well. I've in the past called node one's ifgrps a1a, a1b, a1c etc, and node 2 a2a, a2b and a2c

#

then you create a LIF with the home port of node 1 a0a-1234 or whatever VLAN ID (if not native), and allow it to failover to node 2 a0a-1234

#

iSCSI LIFs don't failover until ONTAP 9.11 (and then only on the ASA platform), and even then, host multipathing is probably a better choice

#

(imo, obviously we added it for a reason.. but I don't see it worth it)

mossy condor
#

Taking a shot in the dark here 1. Customer is planning on implementing F5 load balancer

Here is the Ask:
We are in process of building the network level AZ for our NetApp so we leverage few options from our end however, it would be great if you can get some experts from your team who can help and clear all our doubts with respect to network level load balancing using external LB. we are basically looking network level failover scenario where out of 3 interface if one lif goes down the other 2 should serve the data .

We want to use F5 LB for multiple LIF’s from same SVM and also create a automation for a SVM failover use same F5 VIP.
https://support.f5.com/csp/article/K6749
https://support.f5.com/csp/article/K12757773
https://www.f5.com/pdf/products/arx-series-ds.pdf

ionic vigil
#

You really don’t need a 3rd party LB here for anything other than front end traffic. NetApp and switch logic can take care of the backend. It sounds like you’re more interested in failover than actual load balancing though. Fair?

warm island
#

Yeah if you have any choice please don’t put an F5 in front of any SMB server, NetApp included. Data consistency is a precious thing and that’s going to mess with it. Can you tell us more about why they want to do it?

steady hawk
#

I had to talk people out of putting Citrix load balancers in front of SGrid because they didn't want to change the DNS TTL

mossy condor
mossy condor
#

So they want what I think is SVM-DR synchronous replication and the ability to script planned failover sending traffic to the other cluster. I have a call with them in 5m. I will get more info

pallid swan
#

If they use the on-box DNS round robin, it also checks load on each node and sends connections to that node specifically.

mossy condor
pallid swan
#

Oh a caching appliance.

#

I mean...AFF's performance is pretty fast anyway so why would you need that?

#

Automated storage tiering
ARX automates the placement and movement of data between different tiers (or classes)
of storage, with each tier potentially comprising devices from multiple vendors.

#

I mean...data is tiered automatically with FabricPool if it's older, and volumes can be on slower disks if needed.

#

To me it seems redundant with all of ONTAP's features.

little berry
#

Adding my voice here. I would suggest if possible they push back the F5 buy/install at least, see if ONTAP's native features can't provide the function they want.

#

It may be worth consulting with our professional services or sales engineering team to change configuration or validate design.

upbeat geyser
#

Be aware: the on box DNS load balancing is not meant to be used in high performance workloads. After this failing miserably with a few thousand simultaneous mounts hitting the on box DNS we inquired with engineering. We were told it is what it is and will likely not be updated to accommodate high performance workloads.

visual river
#

we want to use 4 ports with LACP for CIFS on ontap 9.11. what would be best in this case? create lacp port channel 1 with 2 netapp ctrl1 ports and channel 2 with two ctrl2 ports? or one channel with 4 ports?

#

I'd create LAG on LACP with for example e0a and e0b to create a0a on ctrl1 and same on ctrl2 and then put them in failover group?

#

so two lacp channels

tidal owl
#

You can't create a ifgrp with ports from different nodes. So yeah, create LACP port channels on each node. With as many ports as you have available (using 2, 4 or 8 is best for an even distribution).

#

Use FlexGroups so you can use aggrs from both nodes.
If you want to use the ports from both nodes (active-active) for the same CIFS server you would need DNS load-balancing.

visual river
#

second 2x 10Gb will be for failover

#

thanks bro!

tidal owl
#

Yeah, that's fine too

ionic vigil
#

It's nuts that we'll likely pass 1M routes sometime in the next 12 months in the BGP tables.

steady hawk
#

Frightening how little security wraps BGP and how vulnerable it is.

patent elk
#

idel-time from the result of "nfs connected-clients show -volume". I cannot find definitions on what idle-time is. some clients have never accessed NFS volumes, but showed 4minutes idle time. 3d+ seems the maximum idle time I can find upon the environment here. I also found the other client, the NFS used to be mounted, and already gone now, but the idel-time on this volume still showing 2m+. So, how exactly idle-time is calculated?

lunar portal
#

Question for all the Cluster / Switching Guru's - I've got a FAS2750 and a 2650 that I want to put into a switched cluster using a pair of Cisco 3132Q-V's (yes I know these will probably go EOL sooner rather than later but as a POC they should work decent) since the cluster ports on the two fas units are 10G - SFP+ I presume I just need to use 10G (unless I can use the 40's to split but not needed at this juncture) QSFP transceivers to the 10G SFP+ cluster ports ? Any got you's I need to be aware of ?

ionic vigil
#

10Gb cluster interconnects are plenty for 2xxx series. Everything is Nexus 9k’s now though so just keep that in mind. You can also just use Cisco DAC copper/twinax if you don’t want to do optics/fiber.

pale wagon
#

it'll work off the 10G - I also noticed in the docs that they have to be enabled manually and the RCF doesn't enable them?

HWU shows that the perfered is the 4x10 breakout.

Check HWU for whatever cables you're using if you didn't get them from NetApp.

Also note that after the 27xx and 26xx you max out the 10G ports.

image.png
lunar portal
upbeat geyser
#

You can either use the 4xsfp ports on the switch, which requires a command to enable them (which ultimately disables the first qsfp port) or you can use the 40g to 4 x 10g breakout cables

grim ibex
ionic vigil
#

Hmm. I thought the Cisco 9k's and Mellanox one's were the only supported ones. I know the NVIDIA Spectrum's are on the list but availability on those has been /null.

pale wagon
#

Broadcom = BES

#

there's also 3xxx out there. though none of the ones we support are still avaiable for sale.

lunar portal
#

I drugged up some 3132Q-Vs for a small cost - if you don't mind used/referb they can be found while EOS I fairly ceratin you can still get support on them as well

#

Yea just verified EOSL is 8/31/27 so still have a few years left

raven spear
#

I have a customer that still has a bunch of CN1610s in production. Amazingly, they don't hit EOS until 2025.

steady hawk
#

No but the CN1610's can only take you so far Ontap wise

raven spear
#

They can take you to 9.11. CN1610 support is dropped in 9.12.

patent elk
#

If I run "nfs connected-clients show -volume", it will give me "local_reqs" and "remote_reqs" values, representing "fast-path" and "slow-path" respectively. How could fast or slow path be determined? by ONTAP or what? anyway can we control which one to use? What is the difference between them and indirect/direct path? If somebody can please shed some light on.

pallid swan
ornate vault
#

Are there any plans to allow customer to use more generic types of switches versus the limited supported list . For entry level configs the cheaper BES switches cost as much as a controller pair (around 40k euro) making the business case for an horizontal scale out very complicated while there are plenty of alternatives that could probably do the job . This is in the same vein as the metrocluster compliant switch which allows the use of non NetApp validated switches for highly critical systems .

steady hawk
#

Support Arista

#

Pretty please

ionic vigil
#

I'll let some of the hardware leads weigh-in, but it's not just about the compatibility of things, it's about pinning down supported OS/fw versions that run on the switch. WILL it work? Probably. Will every switch out there get beaten up by QA and their golden image configs hardened, enabling us to claim "full support"? That's just unrealistic unfortunately, short of having a battalion of QA engineers and web designers to keep the versions matrix updated. Cisco alone has thousands of models of switches with hundreds or thousands of versions of IOS/nxOS. You'll find we're always going to err on the side of realistic supportability and stability when it comes to stuff like that, and personally I applaud/welcome that stance.

velvet frost
#

How does what you say relate to the use of non-Cisco-supported SFPs NetApp uses in the Cisco switches? I'd argue that supportability is low when Cisco tells NetApp that they won't continue the TAC case unless the end customer switches to Cisco SFPs (not all end customers do have a couple of these lying around)

steady hawk
#

Nick, where I entirely agree with the stance of erring on the side of caution.

I will toss this out there for chaos
Arista is near a 1:1 conversion on commands and if I recall, has simply one OS to deploy.
Hardware though obviously differs.

warm island
velvet frost
warm island
#

Mmm.

velvet frost
# warm island Mmm.

yeah, exactly 😉 I mean it's fine I guess if NetApp had the respective optics lying around to swap them in during the TAC case, but I know of at least 2 cases where some NetApp engineer asked us (partner) of we could lend them some optics for a TAC case because Cisco refused to continue working on the case as long as non-Cisco optics are involved 🤷

patent elk
#

We have Layer2 network currently is dedicated to VMware Datastores mounted on ESXi hosts. Further, we use IBM backup and restore software to back up VM's.

The software uses DataMover(another VM) to back up VM's from the datastore mounted on ESXi or Backup DB's onto NFS volumes. This process or backup dataflow currently goes through Layer3 network via Layer3 nics which is heavily loaded.

My question is: instead of using Layer3 network, I can reconfigure the backup software and use Layer2 nics on ESXi hosts and the backup servers, thus the backup dataflow will go around the same Layer2 network shared with Datastore. Will this design cause any issues?
If experts here can help to shed some light on the design.

warm island
#

in other news - the CN1610 is now supported for 9.12.1 clusters! It's still low bandwidth compared to the 40 and 100G cluster switches we have now, but it's good until 2025

white rune
#

Hi all!
Just a brainstorm question.
I have a customer with an A250 configured with NFS for VMware.
Connection is a LACP to two Cisco Nexus switches, In VMware we use a NFS VMkernel with two active uplinks, all 10 Gbit.
During the initial installation jumbo frames was not enabled in VMware and on the NetApp.
We would like to change the VMkernel and Broadcast-domain to a MTU of 9000.
Can we do this "live" or must we plan a moment of downtime? I can orchestrate the changes to be within seconds.
Thanks!

warm island
#

will it exceed the NFS timeout for vmware? I wouldn't risk it personally

white rune
#

Hi Alex,
I know how to change the settings in OnTap and vSphere .
Was looking for experience in doing this "on-the-fly".

warm island
#

I know you know how - I'm just confirming that it isn't something we'd suggest doing live. The pause is real, it happens. I've seen 10-15 seconds normally

white rune
#

Thanks Alex!!
Any ideas on how to approach this coordinated between VMware ESXi and NetApp Broadcast domain?

upbeat geyser
#

I’ve done this a few times. Make sure the switch is ready first. Then I do ONTAP (adjust the broadcast domain mtu). Verify the Netapp nfs data lifs can in fact ping each other with jumbo frames then enable the esx side.

Esx generally includes updating the distributed switch to allow 9000 mtu then modifying the vmkernels to allow 9000.

Generally, it just works. If you have some oddities in your network there may be issues.

Always good to make this change in a maintenance windows to cya.

How to test jumbo frames from ONTAP:

net ping -vserver <esxSvm> -d true -p 5000 -destination <ip of Netapp/esx lif> -lif <svm-nfs-lif>

Jumbo frames is up to 9000. With overhead, that max packet size is actually 8972. If you try a packet size of 9000 it will fail.

hoary coral
#

normally on (Cisco) switches jumbo sets 9216, iirc... and i think most other equipment just call it 9000 while allowing for the tagging overhead outside the 9000 ... it's not terribly precise and unnecessarily confusing

#

ideally, you'd have a frame size that is just right for 8k NFS read/writes... or such that larger r/w is split nicely into mulitple frames... but it's rarely worth the effort to tune this exactly

upbeat geyser
#

Different switch vendors are different depending on the way they encode tagging. Cisco is 9216. I think Arista is 9214. Do the right thing for the switch vendors. Look up the correct information.

With the exception (front what I have seen anyway) of the INTEL nic drivers on windows (which use some number over 9000!) all clients are today set to 9000. This is the largest that it can use.

There is still overhead. You can’t use actual test with 9000 directly. You use the anything between 1473 and 8972.

A standard frame at 1500 also has overhead. Anything over 1472 is technically a jumbo frame.

velvet frost
#

Has anyone ever used or configured Arista switches as MCC IP backend ("open networking"), and can maybe share the configs for the correct DSCP/QoS settings? (i.e. this -> https://docs.netapp.com/us-en/ontap-metrocluster/install-ip/concept_considerations_mc_compliant_switches.html#generic-switch-configuration but for Arista)

Considerations for using MetroCluster-compliant switches

MetroCluster IP switches that are listed as supported switches in the link:https://hwu.netapp.com/[NetApp HardWare Universe^] and provided by NetApp are NetA...

knotty rock
# raven spear They can take you to 9.11. CN1610 support is dropped in 9.12.

It looks they extended the CN1610 support to include 9.12. Right?
https://kb.netapp.com/?title=onprem%2FSwitches%2FNetApp%2FDo_I_still_have_support_for_CN1610_switches_reporting_as_%2527limited_support%2527%253F

NetApp Knowledge Base
Do I still have support for CN1610 switches reporting as 'limited s...

The CN1610 FASTPATH operating system transitioned from full support to limited support with the release of NetApp ONTAP® 9.6. No further releases are planned for the FASTPATH operating system …

raven spear
static flame
#

Has anyone heard anything about NVME/TCP for Windows?

tidal owl
#

Currently there's no native driver for Windows. The only Windows-certified driver I know of is the one from StarWind: https://www.starwindsoftware.com/starwind-nvme-of-initiator

Which does not mean it's supported for ONTAP targets. As long as no support shows up in the IMT, it's unsupported. (But might still work, maybe just try it. The free version is "allowed for non-production lab and PoC use".)

NVMe-oF (NVMe over Fabrics) Initiator from StarWind
NVMe-oF (NVMe over Fabrics) Initiator from StarWind

StarWind NVMe-oF Initiator being Windows-Certified is the only solution that offers NVMe-oF connectivity over TCP and RoCE v2 networks for Windows applications

velvet frost
#

There's also one from Pavillion but I don't think you can get it separately from their storage systems

tidal owl
velvet frost
#

ah, nice, that's new then. When I last looked at it, the GUI definitely looked different between the two

patent elk
#

We have a list of ACL rules associated VLAN's protecting the storage networks in Computer Center. In the end of those "permit ..." or "deny..." rules, the bottom of the ACL is "permit ip any any". In effect, everything is allowed to access these networks except what is blcoked by the "deny" statements. We are being ask by Networking team to tight up the access. I am not sure of how we should do that.

I can think of port 443 should be allowed which is needed by NetApp support, and also AWS S3 should be allowed. Am I in the right track? We would not want to restrict any legitimate accesses which may not be known to us.
My question is, what rules should we add / change? any specifics can you think of?

warm island
#

What storage protocols are you running on the system?

upbeat geyser
#

The easiest thing to do is make sure the switch is logging all failures and more importantly successes. You can use the logging information to build your rules. This comes in handy because most ports do not return on the same port. So you have to allow the return path typically on any.

#

Then if you get a failure, you can use that to build a rule that will allow.

patent elk
#

If somebody please comment on following:

If I allow accesses on all those ports listed on page 33-34 (https://www.netapp.com/media/10674-tr4569.pdf) from all company's networks, and block everything else, would that be a right approach at beign with? Also allow all outbound traffic.

Since the storage has already been protected by the company’s firewall, is there the need to also specifically deny the access from internet?

white rune
#

Hi, long shot. Currently doing a new install with NVidia SN2100 switches. The NVidia PAK certificate is not yet there. Does anyone have access and can provide the 5.4 version for the SN2100 switches?

nimble parrot
#

Quick question regarding the service-policies and the way they work... I can see that a simple vserver with iscsi has default-management set, with a whole lot of management-core, menagement-dns-client etc. etc. if we wanted to limit this to specific subnets, does it only affect incomming requests? and if yes, why are there so many different services defined? like management-ntp-client ? And if the service-policies restricts on both source and destination IPs.. how do you distinguish between the two? Basically we would like to limit the ranges from where you can access the management services... and this confuses us a bit 😉

pallid swan
#

Just use another LIF for management.

raven spear
#

Weird question: Does a 15m MPO-LC breakout cable exist? Customer inventory spreadsheet is showing pn as X665205-15 (which might be wrong pn since I was able to find 5m/30m breakout cables in HWU as X66205-5 / 30 and the inventory sheet had those PNs wrong as well), but I can't find the 15m ones in HWU. If you put me under oath, I'd swear the 15m version existed since I recall using them and the customer inventory sheet shows them existing in multiple data centers.

velvet frost
#

Not sure if you can get one from NetApp, but these cables are pretty standard so any shop that sells fiber cables should have them

opal lichen
raven spear
#

yeah, getting them third party is no problem, just trying to figure out if I'm go crazy thinking NetApp offered them

upbeat geyser
#

Just be sure to order MPO Type-B!

#

Type-A is for switch to patch panel (it flips fibers)

velvet frost
upbeat geyser
#

Yep. True. Haven’t used a lot of moo breakout. Just know type a fails mpo-mpo

warm island
hazy creek
#

hi

pale wagon
#

👋🏻

void wigeon
velvet frost
upbeat geyser
#

And that has been deleted

#

This blog post cannot be found, Please check your URL

velvet frost
#

typo, sorry, didn't notice it

ionic vigil
#

Throw this in as a post in #1062049107096633454 because we have engineers that monitor things in those channels. This is more for general discussion!

half sinew
#

Hello We have a Metro IP cluster with AFF A250 version 9.14.1P5 with BES 53248 version 3.9.0.2 switches.
The clusters are more than 20km with CWDM
We tested SFPs for ISL 2 SmartOptics SFP-10GE-ZR-C47 on the BES but the ports do not go up and stay down. Link Link Nominal
Length Length Link Bit Rate
Number 50um 62.5um Length Per Lane
Port of Lanes Vendor Name [m] [m] [m] Serial Number Part Number [Mbps] Rev Compliance

0/9 1 Amphenol 0 0 2M APF21139360163 NDCCGF-N102 25500 F 25GBase-CR CA-S
0/10 1 Amphenol 0 0 2M APF21139360148 NDCCGF-N102 25500 F 25GBase-CR CA-S
0/13 1 SmartOptics 0 0 -- VE2322001320 SFP-10GE-ZR-C47 10300 4.1 Unknown
0/55 4 Amphenol 0 0 2M APF21149367367 112-00574 25500 B0 100GBase-CR4
0/56 4 Amphenol 0 0 2M APF21149367961 112-00574 25500 B0 100GBase-CR4

upbeat geyser
#

Did you check the Netapp support matrix (hwu.netapp.com) and verify the switches/ONTAP version/optical modules?

Did you check the Broadcom site for supported optics for the BES? They have a document there with all supported optics.

Randomly testing optics is generally not a good idea unless they are supported my the manufacturer

opal lichen
#

I hope this isn't any of you

pallid swan
pale wagon
#

/s obviously

opal lichen
#

Vicki Vallencourt showed me her VLANs… and I liked them too!

atomic mica
#

Correct me if I'm wrong, but if traffic comes in a LIF sitting on Controller 1 ports, wanting to access data on an aggregate owned by Controller 2, it would just get routed over the cluster interconnect network and Controller 2 would process that request still. Have a coworker who's telling me that unless the traffic comes in on the ports of Controller 2, controller 1 would have to do the processing for it. Sounds very wrong to me

upbeat geyser
#

Access via LIF on node 1 (nblade), through cluster interconnect network (switches or switchless) to node 2, node 2 will process dblade (disk), send results back out cluster network to node 1 to return to sender from originating LIF on Node1 (nblade). I think that is the path

atomic mica
#

okay right, thats how ive always thought of that. IIRC the added latency going over cluster interconnect network with a switchless cluster is rather negligible?

upbeat geyser
#

Originally, it was really not well known. In the OLD days, the best-practice (for VMware anyway) was one LIF per datastore with the idea that if you move the datastore, you can move the LIF and keep locality. After plenty of real-world testing scenarios, the minimal latency introduced by the cluster network appeeared to be neglidgible. It is still recommended whenever possible to acces data direct from the node. When using ONTAP Tools for VMware, this makes sure it happens (unless you move a LIF or move a volume!)

atomic mica
pallid swan
#

It's microseconds. Typically 10-200 usec.

#

And yes, one LIF per volume.

upbeat geyser
#

best practice: 1 LIF per node. Use ONTAP Tools for VMware to deploy datastore. it will automatically mount to the IP where the datastore is

#

If using VMware and the customer has the higher license, what I do is this

  • one LIF per node
  • one (or sometime two) datastores per node
  • In VMware, create a Storage DRS Cluster
    -- DISABLE automation. This allows VMware to determine placement, but will not MOVE anything
    -- moving VMs will inevitably disrupt any snapshot space usage (moving a VM will consume snapshot space!)
  • start placing VMs in the Storage DRS cluster and for the most part, VMware will distribute between the members.

Note: if you make the Storage DRS cluster and then have ONTAP tools rescan, it will detect and you can actually have ONTAP tools place the volume in the Storage DRS Cluster

atomic mica
#

I guess it's time for me to look into ONTAP tools for VMware 🤓 just been doing datastores the old fashioned way, right click and "add datastore" 😅

upbeat geyser
#

Then after everything is mounted, go to the ESX-CLI and
esxcli storage nfs add -H netappIP -v local_name -s /Ontap_path -c 4

#

The limit for connections is 4 unless you modify ESX:
esxcfg-advcfg -s 8 /NFS/MaxConnectionsPerDatastore (sets to 8)

#

ONTAP Tools -> right click -> provision datastore
Will ask a few questions and will use current best practices to deploy datastore! (which cluster, which svm, which aggregate, etc...Will NOT ask for an IP-> will only display aggregates if the node has an IP and an available aggr)

velvet frost
atomic mica
#

good thing i didnt open my fat mouth and say "i think thats actually not right" 😅

pallid swan
dawn crest
#

Sounds like a great question for the exam haha.

velvet frost
pallid swan
#

Sorry I meant @atomic mica .

kind wadi
violet beacon
#

I don't have anything to do a test run with so I'm just confirming this procedure to upgrade the Broadcom BES-53248 switches is pretty spot on and there are no gotchas to know before running through this process - https://docs.netapp.com/us-en/ontap-systems-switches/switch-bes-53248/upgrade-efos-software.html#prepare-for-upgrade. Also is the RCF a necessary task I need to perform?

Upgrade the EFOS software

Follow these steps to upgrade the Ethernet Fabric OS (EFOS) software on the BES-53248 cluster switch.

waxen blade
#

This is the correct process, and RCF is necessary... usually. You can confirm from the current version you have vs what is available on the BC site, just make sure the RCF version is matched to the EFOS version and is on the compat matrix from NetApp via Hardware Universe

Just remember to back up your license files, configs, etc. BEFORE you do anything else.

violet beacon
#

@waxen blade Thank you sir!

upbeat geyser
#

Sometimes you can update efos and sometimes you need to do both efos and rcf. As already indicated you really need to follow the reference table to stay in support.

I always do a side by side compare of the old to new rcf to determine if I really need to clean. If there are just minor updates like the one where they just add the vlan interface for tracking the version, no need to clean

I’ve followed (actually I’ve had the docs corrected a number of times as I find issues) the docs and they seem to work well.

The most important change is in efos 3.12+. After installing 3.12 or higher there is a new way to install the rcf. The latest version of 1.12 does not comment anything out. Instead of copying to nvram:script you copy to nvram:reference-config and then you “script apply reference-conf.scr”

dark void
#

helo i'll write there cuz there's no "general" channel but has anyone successfully imported NetApp's chassis visio from visiocafe into draw.io ?

unkempt horizon
#

I have not. Years ago, however, I was able to import them (somehow) into Network Notepad. I don't know if that will still work, though.

#

LucidChart will also work for this. Perhaps it has gotten better since I last used it in 2021/2022.

velvet frost
#

so, it looks like the nVidia SN2100 cluster/MCC switches are now EOA? That's how I read CPC-00635, however, HWU still doesn't list an EOA data for the switches. Which one is correct?

upbeat geyser
#

Even more hilarious the replacement is X190212.
Not even on the supported switch list yet

Maybe the SN2100 is being retired and replaced with a new nvidia switch?

velvet frost
#

well there were rumors of a different switch vendor at Insight last year, maybe that's what they'll replace it with...?

upbeat geyser
#

Ah hah. It is NDA. Go to partner hub and look for X190212. It will make sense then

#

Not sure why it is NDA since it is supposed to be in the quoter tool (Feb 20).

tidal owl
#

Huh ok, interesting move.
But I won't shed a tear over the SN2100... ☺️

upbeat geyser
#

Looks like the replacement rhymes with Sysco

opal lichen
#

Hey they make our food!

#

(their baked potato salad is solid, and sold everywhere)

velvet frost
#

MetroCluster support again still missing officially (only 36-port configs in HWU)

compact blade
#

can't find pictures of this switch 😦

velvet frost
#

it's the same model that is already being sold, just with fewer ports licensed

upbeat geyser
#

The “new” model isn’t on the site yet. Not in hwu. Not on support for rcf.

velvet frost
#

yeah, but at the same time it's apparently already in the quote tool, and the SN2100 have already been discontinued. In any case, it seems to be a port-on-demand thingy so it should work out of the box with the same RCF files, since you can decide which ports get licensed and which don't (it's not like you only get ports 1-12 and nothing else). Pricing seems decent (about 50% of the 9336 switch) so I guess we'll be filing a few PVRs now 🙂

feral saddle
#

Cisco question (and why I like Mellanox Onyx os more) to enable vlans 2-4094 on a nxos switch...whats the easiest way

#

i was able to enable 2-3967 but 3968+ takes a crap

waxen blade
#

what cisco, cats?
weird, cause 1-1000 is standard and 1025-4096 is extended

#

1001-1024 can't be used

feral saddle
#

i need to do this on nexus 5 and 9ks

waxen blade
#

3968-4094 are internall allocated for internal use, cant use/create/etc any of those

#

not on the nexus

feral saddle
#

ahhh ok that makes sense then...i can create and use vlans up to 3967

waxen blade
#

technically it's 3968-4047 and then 4094, but we never use anything over 3967 because of that

#

welcome to cisco, where unified commands, options and settings don't exist and change with the wind

velvet frost
#

and don't get me started about -foo enabled|disabled vs -foo on|off vs. -foo true|false 😛

tidal owl
#

Parameters which autocomplete and others which don't... Commands where asterisk works and where it doesn't...
Also stuff which can only be changed via modify and not during create...
I could go one but still ONTAP cli is still so much better than NXOS imho.

velvet frost
#

or foo bar show vs foo show-bar ... but yeah, still a better CLI than many others

ionic vigil
#

And in SysMgr, aggrs are tiers

teal smelt
dark void
#

I gave up and have a windows VM with visio for that now. draw.io is way better and free, hope they can get something to work in the future

unkempt horizon
#

@teal smelt I have not. Years ago, however, I was able to import them (somehow) into Network Notepad. I don't know if that will still work, though. LucidChart will also work for this, but I haven't used that since 2021/2022.

teal smelt
#

Thanks!

unkempt horizon
nimble parrot
#

Quick question about the NetApp Multimode MPO optics like the X65405, would that be compatible with the Cisco QSFP-100G-SL4 or QSFP-100G-SL4-S ? What throws me is that Cisco states MPO-12 (12 fibers) where this isn't really stated on the NetApp optics.. they only specify 4 x Optics at 25Gb each... so can anyone clear this up please? 😉

#

...not sure why NetApp doesn't have a 100G Multimode optic that has the LC connector?

velvet frost
velvet frost
#

But to be honest, I have no idea what "SL4" optics are... Short Long Wave? 😄 so SL4 is apparently "Short Link", i.e. 30m over MMF... TIL!

#

But the X65405 is (among others) the Avago AFBR-89CDDZ, which claims to be SR4, so I would go with SR4 on the other side as well

#

you just need to be careful with the MPO cables as there are 3 types (straight, flipped and crossed, or whatever they're called)... You need type B

nimble parrot
velvet frost
#

but if you're going CWDM anyway I would always go with single mode as the signal integrity is better (no modal dispersion )

nimble parrot
#

I quote the NetApp cables that are present in HWU... I just cross my fingers that they will work with the Cisco modules... I am waiting for someone at NetApp to verify this... until now it's been "don't see why it shouldn't" 😉 not sure why they cannot just give a precise anwser 🙂

#

We are forced to use Cisco modules so it's the QSFP-100G-SL4 or the QSFP-100G-SL4-S

#

And the MPO-12 fibers is the thing that I cannot find on the details of the NetApp module...

velvet frost
#

the NetApp cables will work of course, they will sell you the correct ones (unless you go over MPO patch panels that do funky stuff in the background, then all bets are off)

velvet frost
velvet frost
nimble parrot
#

That was the modules they claimed should match

velvet frost
nimble parrot
#

So SR4 will not work with SL4 ?

velvet frost
#

no idea, I never knew SL4 existed until 1 hour ago 🙂

nimble parrot
#

🙂

#

I find it strange that there are not a "step" in between 5M DAG cables and 15M MPO 🙂 we just need to patch over to the other rack, so 5M isn't enough while 15M is too much... so we will most likely try to source some shorter MPO cables

velvet frost
#

but again, if you source shorter MPO cables, make sure they're of Type B

nimble parrot
#

It's the cable clutter that I don't like... (I still have nightmares over NetApp's Quad SAS-Cabling) 😉

velvet frost
#

yeah, okay, I can relate to that (we have had custom made RJ45 cables in our Lab, in lengths between 10cm and 1.5m, in 10cm increments ... sorting them was a nightmare 😄 )

nimble parrot
#

...all our other cables are to within 10CM (Ethernet and Optics) but I am yet to find anyone who can sell me MPO cables in such specific lengths

velvet frost
#

neat!

nimble parrot
#

But I think the SL4 is just a less powerful laser as it cannot reach as long as SR4... but I think we will go "safe" and go with the same at both ends...

tidal owl
#

I think your QSFP-100G-SL4-S is a typo, there is no such SFP afaik.
These two I know exist:

  • QSFP-100G-SR4-S
  • QSFP-100G-SL4

Optical characteristics are the same for SR4-S and SL4, only difference seems to be the max distance (with OM3, OM4, OM5) and the power consumption. Since they are both using the same IEEE 100GBASE-SR4 standard I would simply choose the cheaper one.
Also make sure that your switch supports the SFP, sometimes you need a newer NX-OS / IOS version: https://tmgmatrix.cisco.com/

grafik.png grafik.png
velvet frost
#

that looks like a typo in the matrix though... If they send with 2.4dBm (1.6mW) per lane, they should reach the same distances over the same fibers

tidal owl
#

I think that's just the supported maximum distance. The SL4 SFP is almost half the price of a SR4-S so some components must be of lower quality.

nimble parrot
#

...nr 2 on the list..

tidal owl
#

yeah that's where my screenshots are from 😉

upbeat geyser
#

Lots of stuff here today!

The Netapp X65405 (100g optic) will usually work in most Cisco switches. However! Netapp has been selling the Cisco Branded 100g SR4 optic for use in the Cisco switches they sell!

Only certain platforms/cards support the AOC cables

The Cisco 100G BiDi uses the standard OM4/LC fiber. I’ve heard that if you really want to there is a pvr around that will allow the BiDi to be used on most platforms (not the x1148 in the c/a800 though! No optics allowed on the x1148 in the c/a800)

I’ve heard the BiDi uses multiple lasers with different wavelengths where the MPO simply uses 12 fibers

nimble parrot
#

I was tempted to go with the AOC cables (I guess this is the 15M+ "DAG-like" cables but with fiber?) but just to bee 100% sure we didn't have any issues we choose the SR4 optics with the MPO connector... and the Cisco guys will then order the Cisco QSFP-100G-SR4-S module... And we ordered the 15M MPO/MPO cables just to be sure... but will look at a 3rd party cable which is shorter... again there is a large jump from 5M to 15M 😉 And where possible we like to keep it neat so no cable loops hanging arround the racks...

upbeat geyser
#

Always check hwu. There are some cards/platforms that actually support AOC. My issue is the matrix only shows 15m & 30m as supported. Figure if they have qualified those two, any length should work

#

I’ll be asking next week in person

nimble parrot
upbeat geyser
#

As a partner, we will many times just order third party MPO/MTP cables for custom lengths as needed.

I understand NetApp can only stock so many options

nimble parrot
#

I am in need of some "deep dive" Cisco commands to figure out why we lost link on our MCCIP (9336C-FX2) ISL link. we of cause have two, so no down time, but this has now happened twice within a few months and we suspect the supplier of the lines to have an issue... we have looked at "sh interface counters", "sh interface transceiver"... (this is Cisco original modules "QSFP-100G-ERL") "sh logging"... etc. there are no apparent errors, just looks like the cable was pulled, and after 20 secs. it links up again... this MCC has been in production for about two years now and for the last two months we have seen two of this kind of errors... and both times on the same ports.. so we of cause suspect the supplier of the lines, but we need to be sure it's not the switch/modules that are at fault... any suggestions are welcome...

opal lichen
#

When it’s up, are the signal levels in an acceptable range?

nimble parrot
# opal lichen When it’s up, are the signal levels in an acceptable range?

Yep, it all looks OK, thant's the thing. we strongly suspect that "someone" pulled the wrong cables somewhere... and it happned twice within two months... so signal OK, ethernet packets look OK... only in the "logging" we can see that signal lost... then 20 secs. later it links up again... I was just wondering if there were any other commands on a Nexus switch which was able to diagnose a bit deeper..

opal lichen
#

I don’t have anything else there to offer. But I will say the most interesting issue I was ever part of resolving at my last job was a microwave link being interrupted by a boat crossing the link at random. Took forever to figure out that’s what was causing it.

pale wagon
#

Diag logs?

#

i don't think they're enabled by default on the 9Ks

hoary moon
#

hi all, i use the DNS Load Balancer from ONTAP (on-Box) and searching for log file for troubleshooting

upbeat geyser
#

Log in to system manager. Then modify the URL
Https://name_or_ip/spi

log in with an admin user that is capable of access the service-processor interface

Start looking. The logs may be on one or more nodes in the cluster. All the log files are there and may be a couple levels deep

tacit socket
upbeat geyser
#

Yep.

thorny creek
ionic pawn
#

I'm working on improving our Storage backend for a proxmox cluster. We are using an aff c190 to provide NFS, for vm disk and backup storage. It was setup using NFS v4.2, ostensibly to take advantage of session trunking. I'm questioning the configuration though, because we aren't surviving a controller swap on the netapp appliance. For example, lifs riding a0a on controller 1 get failed over to a0a on controller 2... everything works correctly... but the proxmox vms lose their connection to the vm disks and the VMs have an os-appropriate bed shitting event. I'm not sure what to be looking at here. I feel good about switch config (pair of redundant switches with ports in MLAG), and things are, i think set up using NetApp guidelines from docs. The only thing occuring to me is the amount of time taken for the fail-over and session re-establishment for the NFS mount. 4.1 and 4.2 are stateful iirc, v3 isn't. Would we maybe be better off falling back to v3 and dropping trunking in favor of nothing / nConnect? Just dont' have much practical experience with Netapp and Proxmox yet, but figured enough people have to be migrating over from Vmware that someone might have some thoughts / ideas.

velvet frost
#

I think the general consensus is to use NFS3 with NCONNECT rather than 4.x with Session Trunking, as NCONNECT works much better

ionic pawn
#

I know I've read somewhere that NFS3 should fail over faster...but i'm not finding the source as i look now. It isn't a heavy lift change, so probably worth a shot.

velvet frost
#

Since NFS3 is stateless, the client just re-tries and as soon as the gARP packets were received by the switches, the reconnect will succeed. NFS4 loses state, and the client has to explicitly handle reacquisition of that state after a reconnect, so there's much more protocl (and many more steps) involved that can potentially fail or have issues.

thorny creek
warm island
velvet frost
#

you can actually connect that EPROM to an i²c controller (e.g. RaspberryPi) and read from (or even write to) it. Things like vendor IDs etc 🙂

nimble parrot
#

...and you can code it... in this example you would need the box from fs.com (https://www.fs.com/eu-en/c/fs-box-3389) it can of cause only code it's own cables... and yes it makes a difference... I have had cables which the NetApp wouldn't work with until I coded it to a "Cisco" cable...

FS.com
FS BOX | Others - FS Europe

FS provides FS BOX, free & fast delivery, expert tech support, outstanding warranties.

velvet frost
#

yeah, some of these EEPROMs are write-protected until you send them a "magic sequence" (which of course is undocumented) but some of the cheaper ones just use regular EEPROMs that you can fix with any i²c tool. I actually made a breakout board with an SFP slot for my RaspberryPi to play around with that 🙂

nimble parrot
#

I would love an opensource variant of this 🙂 You are correct, some times I need to reach out to FS in order to code specific cables... and they check you order number etc. I guess one should be able to "sniff" the i2c traffic and maybe "crack" the code... I cannot immagine that it's very complex taking the little eprom into account 😉

#

...and their "box" is online... so they know everything you do 😉

velvet frost
#

there are other vendors who don't require their boxes to be online (or so I've heard)... don't remember the name(s) right now though

upbeat geyser
#

Man, I had a nightmare a few years back. Had a metric tonne of infiniband cables that were not fully coded correctly at the factory. They sent us two devices to reprogram....400+ infiniband cable ends. If I never recode a cable again, it will be too soon (this was in a sedcure area, so the customer got to keep the devices)

nimble parrot
#

So is it just me? 😉 I am trying to "reset" a few old Nexus 3132Q-V because the admin pass is lost... I have a serial terminal on it, and as it boots I hit "Ctrl-C" like crazy and I managed to get a prompt where it asks me if I want to init the switch... I tried that, and it basically reformats the bootdevice, so it now boots into the loader prompt... so far so good... I then try to set an IP, and boot from TFTP "n3000-uk9-kickstart.6.0.2.U6.10.bin"... but this fails with "error: Failed to download image."... on my TFTP server I can see that the file is requested.. ( I tried two different TFTP servers )... have I just bricked the switch? 😉 or is the BIOS just too old for this? "2.7.0" ? I have just tried with another similar switch and it starts up booting the same kickstart image that I am trying to load from TFTP... any suggestions would be nice 😉

waxen blade
#

try ctrl ]
at the prompt type recovery mode equals 1 then admin-password "NEWPASS"

#

that's what i typically use for the 3k series

nimble parrot
#

...an update.. I managed to get the kickstart booted... but when I get into the swich (boot) and "up" the mgnt0 interface, it cannot load anything from my tftp or ftp... looks like the network isn't working... IP looks OK...

#

I saw the Ctrl ] but a bit hard to issue that from a Mac via a terminal to a linuxhost connected via "screen /dev/usb0 9600"... 😉

#

...pretty sure I bricked a switch 😉 from the loader prompt I just tried to load the nxos directly (just to try something)... it loaded, then stopped... and now as I power up the switch it just tells me "Copyright 2013, Cisco Systems"... and then just sits there... nice...

waxen blade
#

can you get to the switch(boot) prompt?

nimble parrot
#

yep, but it's like the network is then broken

#

"sh ip" show the correct IPs.. (from the initial "set ip 10.10.10.1 255.255.255.0...

waxen blade
#

have you tried to set an IP on anything other than the mgmt port

nimble parrot
#

And I remember to do "conf t" "inter mgmt0" "no shut"...

#

Which other ports are there? pretty sure there is only the mgmt0 port at this point

waxen blade
#

it's a 32 port 40g switch isnt it, you have modules for any other port?

nimble parrot
#

there are 4 x SFP+ and the test are 40G.. and one 1G RJ45 for mgmt0

#

If I issue "sh int" it only shows mgmt0

#

...and I have link.. but again at this point it will not respond to ping

waxen blade
#

what version of os do you have, and from the bootloader screen can you do a boot n3000-uk9-kickstart.x.x.x.bin

#

ctrl-] should be hit as the "valid image" line comes up in order to reset the password/config/etc

nimble parrot
#

yet from the first loader> screen I can set an IP with "set ip 10.10.10.1 255.255.255.0" and "set gw 10.10.10.253" then I can boot the kickstart

#

Oh... right now I am working on the "bricked" switch on which I did an init... looks like it formats the boot device, so no images to boot into...

waxen blade
#

password strength-check
switchname 3132a
no feature telnet
ssh key rsa 1024 force
feature ssh
system default switchport
no system default switchport shutdown
copp profile strict
interface mgmt0
ip address 192.168.1.1 255.255.255.0
no shutdown

nimble parrot
#

when I get to the switch(boot) the next step would be to do a "copy ftp: bootdevice:"

#

pretty sure most of these commands does not work when booted into the kickstart? but I can give it a try...

waxen blade
#

the nexus has a bit of a different update than other switches.
depending on the version.
You might need to do the install all nxos

#

and I generall use scp, not tftp

#

delete all the old files from bootflash, except the current running one, just to make sure space is available

#

and install all, not boot: install all.
first command updates bios as well, which might be required

nimble parrot
#

the bootflash has been formated and is clean

waxen blade
#

guessing you don't have a usb port on that model

nimble parrot
#

yes I have... but is not showing up 😉

waxen blade
#

yea, that's weird.
that should show up even without the os loaded, unless it's disabled by the firmware

#

boot usb1:nxos.x.x.x.bin

nimble parrot
#

also in the loader brompt?

waxen blade
#

i have a 3048, so not sure they are the same

#

what bios version do you have

#

and nxos version

nimble parrot
#

2.7.0

#

no nxos version yet 😉

waxen blade
#

which one are you trying to get

nimble parrot
#

9.3.14

#

the kickstart is: n3000-uk9-kickstart.6.0.2.U6.10.bin

waxen blade
#

at the boot loader, for configuring the mgmt0 int
set ip 192.168.1.10
set gw 192.168.1.1
cmdline recoverymode=1
boot (nxos file)

#

boot from whatever device has the boot image

#

on mine, for usb, it's usb1, which is odd

#

usually it's usb0

#

on my mds9710 it's usb2

#

yes.. my mds9710. i have 2 of them. dont ask why, unless you want to buy them 😄

nimble parrot
#

are you sure I can boot the nxos image directly?

waxen blade
#

what if you just do dir

#

should show all devices

nimble parrot
#

only shows the empty bootflash:

waxen blade
#

switch(boot) boot n3000-uk9-kickstart.6.0.2.U6.3a.bin

Booting kickstart image: bootflash:n3000-uk9-kickstart.6.0.2.U6.3a.bin

#

yea.. if your bootflash is empty you'll have to copy over the image, but if the network isn't coming up...
not really sure. i'm sure it's probably something simple

upbeat geyser
#

found this

https://www.google.com/search?q=recover+the+switch+and+reinstall+the+os+nexus+3132q-v&sca_esv=fb6ec2c420ab31bb&rlz=1CDGOYI_enUS849US849&hl=en-US&biw=375&bih=632&aic=0&sxsrf=ANbL-n6Sa6-a8MtU5oljBy7GkXKYhe3pOw%3A1768923398636&ei=BqFvaeXJJvquur8PnoKmkQ0&oq=Recover+the+Switch+and+Reinstall+the+OS&gs_lp=EhNtb2JpbGUtZ3dzLXdpei1zZXJwIidSZWNvdmVyIHRoZSBTd2l0Y2ggYW5kIFJlaW5zdGFsbCB0aGUgT1MqAggAMgUQIRigATIFECEYoAEyBRAhGKABMgUQIRirAjIFECEYqwIyBRAhGKsCSMxBUPETWNwjcAN4AZABAJgBpAGgAaECqgEDMC4yuAEByAEA-AEB-AECmAIFoAK-AqgCMcICChAjGMAGGCcY6gLCAhQQABjwBRi0AhjqAhjbBRieBtgBAcICDRAjGMEGGCcY6gLYAQHCAisQIxigBhjBBhiCBhiDBhgnGIgGGKgGGKoGGOoCGO0FGJAGGPoFGN0F2AEBwgIQEAAYAxi0AhjqAhiPAdgBAcICExAuGAMYtAIY5QQY6gIYjwHYAQHCAgUQIRifBZgDEPEFjdPSeBjOnK66BgQIARgXkgcDMy4yoAfsDrIHAzAuMrgHpALCBwcxLjEuMi4xyAcZgAgA&sclient=mobile-gws-wiz-serp#fpstate=ive&vld=cid:a602a542,vid:JskVWVY5U00,st:105

www.google.com
🔎 recover the switch and reinstall the os nexus 3132q-v - Google...
opal lichen
#

Holy link batman

upbeat geyser
#

Yeah, embedded crap

nimble parrot
#

well that's then thing.. I am able to set the ip in the loader setup... then I am able to boot the kickstart OK... but once inside kickstart it's like the network is broken... I have link etc... (and yet I remember to do the "no shut")

upbeat geyser
#

Goes over exactly this. Booting with 6.x code

waxen blade
#

yea, but he can't get the network to see his tftp

upbeat geyser
#

I think the step that’s missing is after the kickstart, you need to modify the m0 port to no shut then get the nxos image

nimble parrot
upbeat geyser
#

Watch the video! Exactly what’s needed

nimble parrot
upbeat geyser
#

About 6 minutes in

waxen blade
#

switch> enable
switch#> conf t
switch#> int mgmt0
switch(config-if)#> ip address 192.168.1.10
switch(config-if)#> no shutdown
switch(config-if)#> exit
switch(config)#> ip default-gateway 192.168.1.1
switch(config)#> copy running-config startup-config
switch#> sho ip int brief
switch#> sho ip route

nimble parrot
# upbeat geyser About 6 minutes in

...right now I am trying to boot the whole nxos image via tftp... takes awhile.. figure it will most likely fail and I will give your commands a try

waxen blade
#

not sure if it's needed or not

vrf context management
ip route 0.0.0.0/0 192.168.1.1

nimble parrot
#

many of your commands doesn't work in the kickstart 😉

waxen blade
#

they're for after the nxos is loaded

nimble parrot
#

I have not tried "ip address 1.2.3.4" but "ip address 1.2.3.4/24" doesn't work

waxen blade
#

those are the commands i just ran on my 3048

nimble parrot
#

just not sure if it is possible to load the whole nxos from tftp...

waxen blade
#

you can if you use the install all command

nimble parrot
#

there is no install command in the loader.. at least not in mine...

waxen blade
#

bios might be too old, v5 is the newest

#

5.2 i think

nimble parrot
#

this is 2.7 🙂

waxen blade
#

you might need to boot to an older version of nxos, like v6

#

v6 has bios 2.6/2.7 in it

nimble parrot
#

let's see if I have one that old 😉

#

7.0.3 is the oldest in my library

waxen blade
#

i'll check to see what versions i have

nimble parrot
#

I have cisco access but have only registered the later metrocluster switches... pretty sure they cannot run 6....

#

...it's still loading and blinking... so lets see

waxen blade
#

yea, oldest version i have is 7 as well, for the 3132 anyway

#

6.12 for others, but doubt they would work with that switch

#

i'd try to use the oldest one you have

#

have it upgrade the vios/etc, go from there

nimble parrot
#

it's loading from tftp at a rate of 6Mbps.. 🙂 strange how Cisco put such slow CPU/Network on a system that can move way more data on the other ports 😉

waxen blade
#

cool, so tftp started working?

nimble parrot
#

it always worked from the loader prmpt, just not from the kickstart

waxen blade
#

well good luck after it's all updated.
I have to go out and start workin on the roof to my shed. hoping to get it finished this week

nimble parrot
#

thanks for the help... I'll report back if it works 😉

upbeat geyser
#

FYI the process is slightly different for 6 (split kickstart/nxos) and 7+ (combined files)

#

I grabbed that video since you indicated at the beginning version 6

nimble parrot
#

OK, I will have a closer look at the other 3 switches I have.. they whould be identical... if I can "reset" the password with the Ctrl ] thing... then I should be able to copy off the files from that switch and use here...

#

but... I can confirm that booting nxos9.3.14 directly from the loader, does not work 🙂 only took me an hour to figure out 😉

upbeat geyser
#

You might need an earlier version. The nxos version may be incompatible with the bios

#

Take a look at the boot process. What’s the bios version and other things look like? Might be the EPLD

#

What version was the switch running before you did the reset?

nimble parrot
#

All the switches have/had 9.6.0.2 installed...

#

I just managed to stop the boot process on one of the other switches... so I ended up in the switch(boot) prompt where I tried to change the admin password... OK... but you are then supossed to load nxos.. and for whatever reason when in the boot prompt I am unable to see any files on the bootflash:... I can see the used and free bytes, not no files at all.... so I scrolled back in the boot sequence where I was able to see the nxos image loaded... I copied the image name... but nope... it would not boot to anything... very very strange... also all the descriptions on resetting password on Nexus are different... it's either Ctrl-C, Ctrl-B or Ctrl-] ... go figure... Ctrl-] does nothing for me.. it's Ctrl-C which eventually langs me in the boot bios... And I thought NetApp was a bit wierd sometimes... this is right up there...

nimble parrot
#

OK... got it.. to reset the admin password on these switches you need to stop the boot process rather quick with "Ctrl - L" which gets you into the "loader" bios... from there you do a "cmdline recoverymode=1" and you do a boot kickstart.bin... which gets you into the "loader" prompt... here you can do a "conf t", "admin-password mypass", "exit"... then you need to "load nxos..." and you are able to login... I am no sure why the bootflash: is "empty" when breaking the boot with Ctrl-C... If I follow this procedure it all looks nice...

#

..oh an yep we are on 6.0... the image names confused me

upbeat geyser
#

Thats odd. You sure it is a 3132q-v? I am not showing any code for that (6.x) only 7.x+

#

Of course, Cisco may have stopped public availablity of the 6.x code

nimble parrot
#

sh modules shows: N3K-C3132Q-40GE-SUP

upbeat geyser
#

what about show version? usually that shows the model also

#

again...just curious

nimble parrot
#

cisco Nexus 3132 Chassis ("32x40G Supervisor")

#

It is a bit different from the 3132Q-V.. I have the management ports at the front...

#

after udate to nxos 7 is states: cisco Nexus3000 C3132Q Chassis

waxen blade
#

glad you got it going

ionic vigil
#

Sorry I missed this thread earlier. I like to tftpboot switches that need a fresh config. Especially the Cisco stuff. I’ve bricked a Dell switch recently doing it that way because of the HORRID design, but Cisco has never done me wrong

nimble parrot
# waxen blade glad you got it going

I think the main reason was the old firmware 6.0.. but who can blame me... look at the name the firmware file has: n3000-uk9.6.0.2.U6.10.bin looks like 9.6... but obviously not 😉 If looks like when you get to version 7 they add the kickstart stuff into the main image...

upbeat geyser
#

Yeah. That’s 6.0.2(U6). The uk9 throws it off. I thought about asking for the name of the file. Would’ve given it away

nimble parrot
#

Does anyone know if you are able to use a SFP 1G SX Optic module in an SFP+ port on a Cisco N9K Nexus switch? Or should you go with a SFP+ optic and set the speed to 1G instead?

opal lichen
#

In my experience (not Cisco), as long as it’s a supported optic, you can do that, just make sure auto-negotiation follows suit.

velvet frost
#

you can do show interface eth 1/1 capabilities and it shoud show you which speeds are supported:

Ethernet1/1
  Model:                 N9K-C93180YC-FX
  Type (SFP capable):    10Gbase-(unknown)
  Speed:                 100,1000,10000,25000
  Duplex:                full
  Trunk encap. type:     802.1Q
...
#

if it says 1000 under "speed", you should be fine with a 1G optic

#

I don't think you can set the speed of a 10G optic down to 1G (at least not all the time, it might depend on the exact SFP model)

nimble parrot
#

Sadly right now I only have DAC cables in the switches, and they all report 100/1000/10000 😉 I guess I will need to test with a SFP module and SFP+ if the SFP is not usable

velvet frost
#

the "speed" output should be what the port supports, and not related to the currently plugged in SFP or cable (in the output above, nothing is plugged in that port, for example)

#

just checked, it shows the same speed values no matter if it's unused or if an SFP or DAC cable is plugged in. So you should be fine with a 1G SFP

brittle mauve
#

And that would be the case either with optics or DAC's

#

A Cisco N9K-C93180YC-FX would be fine with a 1G optic, just make sure you use the right type of optic on both sides AND the right fiber 🙂

#

@nimble parrot Is it a "genuine" Cisco optic or a generic / netapp one ?

nimble parrot
brittle mauve
nimble parrot
#

A quick question for the Cisco nerds 🙂 I have two C9372PX-E and two C9372TX switches. I would like to set them up in a "mesh" preferably Layer2. The switches are mostly identical, one has only SFP+ ports while the other model has RJ45 10G ports. They all have 6 x 50G QSFP ports. We will be connecting hosts redundantly between two switches (port-channel) but not across switch models. I am not sure if it is possible to create a Layer2 VPC between the two C9372PX and across the two C9372TX, and then between C9372PX and C9372TX... ? I am a bit affraid about STP kicking in 😉 And even if this was possible, what is the best way to handle the keep-alive links? normally I would just interconnect the two mgmt0 ports on the back together, but now we have 4 switches 🙂 so I just connect all mgmt0 ports into one of the C9372TX ports? And yes, I am aware of the Spline/Leaf setup, but since I have no experiance with that or Layer3 switching BGP etc.. I would rather use layer 2... suggestions are very welcome 😉

nimble parrot
#

I got this 4 switch "mesh" up and running... and switching wise it works great. I do have an issue on the two N9K-C9372PX-E switche (the ones with SFP+ ports)... they seem to have an issue when working in the console... it "stalls" and "hangs" in the commandline... just a few secs etc.. but it's very annoying 😉 I have been looking at the resources, CPU, memory and processes and it doesn't look like there is something hitting the CPU... has anyone had this issue with cisco switches? strangely enough the two other switches in the setup is having no issues like this... besides the VPC and portchannel between the switches it is a very simple setup...

idle stump
# nimble parrot I got this 4 switch "mesh" up and running... and switching wise it works great. ...

@nimble parrot haven't seen a hang in the console before, but assuming you're not seeing this in the other switches you have your console baud rates etc set correctly. Are you also getting the hangs in SSH ?
On the mesh setup are you running RSTP or MST ? The STP keep alive is only to check the partner switch is alive and to help determine VPC primary/secondary, those mgmt0 ports won't need to reach all 4 switches.