#How to make network trace useful?

1 messages · Page 1 of 1 (latest)

weary sierra
#

so I am trying to do some serious diagnostics on network failover issues. After some preliminary network trace commands here is my assessment of the network trace command: somewhere between meh and useless. Why?

  1. No decent filters. It appears to be using tcpdump so why not pass the filters on down to it? being able to filter on a single ip or single layer 3 protocol is like a V12 with 11 cylinders missing. We are trying to filter on CFM messages, so only have layer 2 info available. No filtering? Sigh...

  2. Cant save files to SVM. Ok, so if there are no decent filter capabilities, one expects very large files. I can process them later with tshark. But......they are only stored on the node root volume and that volume is miniscule and the files max out at 2G and must be downloaded through a web page. CFM broadcasts every 30 seconds so to capture 2 or hopefully 3 broadcasts created 20 files, 40GB for 1.5 minutes of capture.

  3. Doing #2 filled up the root volume, and guess what happens when you do that? The node crashes. YIKES.

So lets see if I can summarize this: no filter == large files == node crash == serious buyer's regret!

So how can I make network trace useful? What am I missing?

noble marlin
#

rtfm?
::*> network trace start ?
[-node] <nodename> *Node Name
[-port] {<netport>|<ifgrp>} *Port
[[-debug] {true|false}] *Print Debug Info on Console
[ -address <IP Address> ] *IP Address
[ -protocol-port {1..65535} ] *Protocol Port Number
[ -buffer-size {1..512} ] *Buffer Size in KB (default 512)
[ -file-size {1..2147} ] *Trace File Size in MB (default 1024)
[ -rolling-traces {1..64} ] *Number of Rolling Trace Files (default 2)

ashen star
noble marlin
#

i believe i did. There are filters both on IP, physical ports and tcp ports. There are max file sizes. One can limit how many rolling files are created.

#

the fact that network traces are an OS operation under the node makes it pretty difficult to save anything on an SVM, but traces are a cluster command anyway. Root volume sizes can be changed and there are other ways of exporting the file than just the webgui

ashen star
#

the argument was that filtering a single IP or TCP port is not enough, to which I totally agree. The other argument was that you cannot define where the file will be stored

#

and you cannot change the root volume size, at least not by much:The root aggregate only has 5% free space which means there's not much you can do

noble marlin
#

depends on how the root aggregate is setup, of course.

#

i'm not arguing there isn't room for improvement, but i don't find the characterization especially accurate. It's good enough for 99% of what it's used for.

ashen star
#

@weary sierra there's always the possibility of running native tcpdump in the system shell. That should make it possible to add any filters you want

ashen star
#

@weary sierra if you poke around in systemshell a bit you might even find out how to store the dump file on an arbitrary volume in some other SVM 😉

noble marlin
#

i always assume the "f" means "fine" among polite folk, fwiw

#

and i would have mentioned running tcpdump in the systemshell if the conversation didn't get hijacked by a hall monitor

willow stag
#

For the root volume, you can check
storage aggregate show-spare-disks
Look how many available root partitions are non-zero
Do a
system node run -node local disk show -n
Look for any possible unowned partitions

Sometimes there are a handful of root partitions unassigned. If that’s the case you can use node shell to increase the root aggregate and then increase vol0 for the node

weary sierra
#

yeah rtfm indicates an inadequate reading of the original post. in my first point I mention the only two filters available -address and -protocol-port, neither of which address any layer 2 issues. In the second point I address -file-size and -rolling-traces.

It's good enough for 99% of what it's used for.

I would argue it's not. The first thing I wanted to use it for and it CRASHED the node. That makes it not only NOT good enough, but dangerous. For what these devices cost, this kind of non-chalance in programming is not acceptable.

@ashen star I will try systemshell and tcpdump.

neat falcon
#

IIRC, there are two separate ways to capture packets: network trace start and debug network tcpdump start

The latter command absolutely lets you store your traces in a SVM/volume of choice, as well as use filters.

e.g.

::*> debug network tcpdump start -node node_name -ipspace Default -pass-through "-i e0e -w /clus/svm_name/junction_path_name"
#

note that /clus is fixed, but /svm_name and /junction_path_name must be changed to suit your environment

ashen star
frank cliff