How to Obtain Performance Statistics and Event Logs from NetScaler

This article describes how to collect performance statistics from virtual servers and services of NetScaler.

You can collect historical performance statistics of the virtual servers and associated services from the archived newnslog files in the /var/nslog directory. The newnslog files are interpreted using the executable /netscaler/nsconmsg.

NetScaler CLI

You can run the nsconmsg command from NetScaler shell prompt without naming a file, to report events in real time.

• Use the following syntax to read a historical file:
  /netscaler/nsconmsg -K /var/nslog/newnslog -d event

   Displaying event information NetScaler V20 Performance Data NetScaler NS10.5: Build 57.7.nc, Date: May 14 2015, 07:35:21  rtime: Relative time between two records in milliseconds seqno rtime event-message                         event-time 11648 16310 PPE-0 MonServiceBinding_10.104.20.110:443_(tcp-default)  

  Notes: The preceding command uses -d event to request display of major events. The parameter -K (uppercase) is for reading and -k (lowercase) is for writing. If you accidentally use -k then you might overwrite any information.

• To view the time span covered by a given newnslog file, use?? the syntax as in the following example:
  /netscaler/nsconmsg -K /var/nslog/newnslog -d setime

  The current data is appended to the /var/nslog/newnslog file. NetScaler archives the newnslog file automatically every two days by default. To read the archived data, you must extract the archive as shown in the following example:
  cd /var/nslog - This is the command to go to a particular directory from NetScaler Shell Prompt.
  tar xvfz newnslog.100.tar.gz - This is the command to extract the tar file.
  /netscaler/nsconmsg -K newnslog.100 -d setime - This is the command to check time span covered by the particular file, in this example newnslog.100.
  
  "ls -l"command can be used to check all the logs file and time stamp associated with those files

  root@NETSCALER# cd /var/nslog
  root@NETSCALER# ls -l

    wheel    461544 Aug  7  2014 newnslog.1.tar.gz -rw-r--r--   1 root       wheel    191067 Aug  7  2014 newnslog.10.tar.gz -rw-r--r--   1 root       wheel  11144873 Apr 26 22:04 newnslog.100.tar.gz -rw-r--r--   1 root       wheel  11095053 Apr 28 22:04 newnslog.101.tar.gz -rw-r--r--   1 root       wheel  11114284 Apr 30 22:04 newnslog.102.tar.gz -rw-r--r--   1 root       wheel  11146418 May  2 22:04 newnslog.103.tar.gz -rw-r--r--   1 root       wheel  11104227 May  4 22:04 newnslog.104.tar.gz -rw-r--r--   1 root       wheel  11297419 May  6 22:04 newnslog.105.tar.gz -rw-r--r--   1 root       wheel  11081212 May  8 22:04 newnslog.106.tar.gz -rw-r--r--   1 root       wheel  11048542 May 10 22:04 newnslog.107.tar.gz -rw-r--r--   1 root       wheel  11101869 May 12 22:04 newnslog.108.tar.gz -rw-r--r--   1 root       wheel  11378787 May 14 22:04 newnslog.109.tar.gz -rw-r--r--   1 root       wheel  44989298 Apr 11  2014 newnslog.11.gz  

  The log files from NetScaler can be obtained from Secure FTP utility too. Download WinSCP Utility and browse to the particular directory to get the required log files.
  Refer to the following article for information on how to use WinSCP for downloading and uploading files. CTX130305 - How to Upload Files to an FTP Server Using the WinSCP Utility.

• Use the nsconmsg command to only display a span of time within the given file, as shown in the following example:
  /netscaler/nsconmsg -K /var/nslog/newnslog -s time=22Mar2007:20:00 -T 7 -s ConLb=2 -d oldconmsg
  Where

  • s time=22Mar2007:20:00 is start at March 22, 2007 at exactly 8 p.m.
  • T 7 is display seven seconds of data
  • s ConLb=2 is a detail level for load balancing statistics
  • d oldconmsg is display statistical information

  The statistical information provided by the -d oldconmsg parameter is recorded every seven seconds. The following is a sample output:

   VIP(10.128.58.149:80:UP:WEIGHTEDRR): Hits(38200495, 18/sec) Mbps(1.02) Pers(OFF) Err(0) Pkt(186/sec, 610 bytes) actSvc(4) DefPol(NONE) override(0) Conn: Clt(253, 1/sec, OE[252]) Svr(3) S(10.128.49.40:80:UP) Hits(9443063, 4/sec, P[2602342, 0/sec]) ATr(5) Mbps(0.23) BWlmt(0 kbits) RspTime(112.58 ms) Other: Pkt(36/sec, 712 bytes) Wt(10000) RHits(31555) Conn: CSvr(42, 0/sec) MCSvr(20) OE(16) RP(11) SQ(0) S(10.128.49.39:80:UP) Hits(9731048, 4/sec, P[2929279, 0/sec]) ATr(9) Mbps(0.27) BWlmt(0 kbits) RspTime(161.69 ms) Other: Pkt(41/sec, 756 bytes) Wt(10000) RHits(31555) Conn: CSvr(32, 0/sec) MCSvr(19) OE(13) RP(4) SQ(0) S(10.128.49.38:80:UP) Hits(9341366, 5/sec, P[2700778, 0/sec]) ATr(4) Mbps(0.27) BWlmt(0 kbits) RspTime(120.50 ms) Other: Pkt(42/sec, 720 bytes) Wt(10000) RHits(31556) Conn: CSvr(37, 0/sec) MCSvr(19) OE(13) RP(9) SQ(0) S(10.128.49.37:80:UP) Hits(9685018, 4/sec, P[2844418, 0/sec]) ATr(3) Mbps(0.23) BWlmt(0 kbits) RspTime(125.38 ms) Other: Pkt(38/sec, 670 bytes) Wt(10000) RHits(31556) Conn: CSvr(32, 0/sec) MCSvr(20) OE(10) RP(7) SQ(0)  

  Note: The reason the client connection count of the individual services do not add up to the client connection count of the virtual server is because of session reuse between the NetScaler appliance and the back-end service.

Description of the Output

Virtual Server Sample Output

 VIP(10.128.58.149:80:UP:WEIGHTEDRR): Hits(38200495, 18/sec) Mbps(1.02) Pers(OFF) Err(0) Pkt(186/sec, 610 bytes) actSvc(4) DefPol(NONE) override(0) Conn: Clt(253, 1/sec, OE[252]) Svr(3)  

The following list describes the virtual server statistics:

• VIP (IP address:port:state:Load balancing method): The IP address and port of the Virtual IP address as it was configured; State of the virtual server or virtual IP address such as UP, DOWN, or OUT OF SERVICE; Load balancing method, configured for the Virtual IP address..

• Hits (#): Number of requests that reached the virtual server.

• Mbps (#): Network traffic that passed through the virtual server in Megabytes per second (Mbps).

• Pers: Type of persistence configured.

• Err (#): Number of times error page was generated by the virtual server.

• Pkt (#/sec, # bytes): Volume of network traffic in terms of packets passing through the virtual server; average size of the packets, flowing through the virtual server.

• actSvc(#): Number of active services that are bound to the virtual server.

• DefPol (RR): Indicates whether default load balancing method is active. Default load balancing method is used for some number of initial requests to smooth the behavior of the other methods.

• Clt (#, #/sec): Number of current client connections to the virtual server rate.

• OE [#]: Number of server connections from the virtual server in open established state.

• Svr (#): Number of current server connections from the virtual server.

In the preceding output, Svr(3) indicates that when the command collected the statistical sample, there are three active connections for the virtual server to the back-end server, even though there are four services in total. When a client has an "open established" connection to the virtual server, it is not necessary that the client is sending or receiving any network traffic at that point of time when the command collected the information. Therefore, it is common to see the Svr counter lower than the OE[] number. The connections that are actively making or receiving transactions are represented by Svr counter. The Mapped IP address (MIP) or Subnet IP address (SNIP) makes the connection to the associated back-end server, but the NetScaler tracks which virtual server is connected to the back-end server and calculates the counter.

Service Sample Output

 S(10.128.49.40:80:UP) Hits(9443063, 4/sec, P[2602342, 0/sec]) ATr(5) Mbps(0.23) BWlmt(0 kbits) RspTime(112.58 ms) Other: Pkt(36/sec, 712 bytes) Wt(10000) RHits(31555) Conn: CSvr(42, 0/sec) MCSvr(20) OE(16) RP(11) SQ(0)   

The following list describes the service statistics:

• S (IP address:port:state): IP address, port, and state of the service such as, DOWN, UP, or OUT OF SERVICE.

• Hits (#, P[#]): Number of hits directed to the service, Number of hits directed to the service due to configured server persistence.

• ATr (#): Number of active connections to the service.
  Note: Active connections are those which have outstanding request to the service or currently have traffic activity.

• Mbps (#.##): Traffic volume passed to the service in Megabytes per second (Mbps).

• BWlmt ( # kbits): Defined bandwidth limit.

• RspTime ( # ms): Average response time of the service in milliseconds.

• Pkt(#/sec, #bytes): Traffic volume in terms of packets per second going to the service; Average size of the packets.

• Wt (#): Weight index, used in load balancing algorithm.
  Note: If you divide this value by 10,000, then you get the actual configured weight of the service.

• RHits (#): Running hits counter used in Round Robin load balancing algorithm.

• CSvr (#, #/sec): Number of connections to the service rate.

• MCSvr (#): Maximum number of connections to the service.

• OE (#): Number of connections to the service in established state.

• RP (#): Number of connections to the service, residing in the reuse pool.

• SQ (#): Number of connections to the service, waiting in the serge queue.

NetScaler GUI

The log files and various troubleshooting data can be obtained from NetScaler Configuration Utility too. Navigate ?? to System > Diagnostics.

Data Collection Procedure to Troubleshoot NetScaler Related Issues
NetScaler Commands to Find the Policy Hits for NetScaler Gateway Session Policies????