• Log files, storing flow-level measurements.
• Histograms, storing the distribution of a given quantity during a time interval.
• RRD, storing histograms in a RRD database.

LOG Files

log_tcp_complete, log_tcp_nocomplete:

report every TCP connection that has been tracked by Tstat. A TCP connection is identified when the first SYN segment is observed, and is ended when either:

• the FIN/ACK or RST segments are observed;
• no data packet has been observed (from both sides) for a default timeout of 10s after the opening SYN segment (TCP_Singleton_Time), or 5min after the last data packet (TCP_Idle_Time); the actual values of the timeout timers is controlled by the corresponding global constants provided via the -G command line option;

Tstat discards all the connections for which the three way handshake is not properly seen. Then, in case a connection is correctly closed it is stored in log_tcp_complete, otherwise in log_tcp_nocomplete.

log_udp_complete:

reports every tracked UDP flow pair. An UDP flow pair is identified when the first UDP segment is observed for a UDP socket pair, and is ended when no packet has been observed (from both sides) for 10s after the first packet (UDP_Singleton_Time), or 3min 20s after the last data packet (UDP_Idle_Time); the actual values of the timeout timers is controlled by the corresponding global constants provided via the -G command line option.
By default, Skype and chat protocols running over UDP are reported only in a separate file (see LOG_ALL_UDP in param.h).

log_video_complete:

Tstat can produce a log_video_complete file which logs every TCP Video connection that has been tracked. Currently are classified as Video the RTMP connections, TLS connections associated to YouTube, and HTTP connections based on 2 distinguished approaches:

1. HTTP request URL matching (Tstat HTTP engine)
2. HTTP response DPI (Tstat Streaming Classifier engine)

The Tstat HTTP engine (1) classifies relevant video-related HTTP connections (YouTube, Vimeo, generic FLV/MP4, VOD and FlashVideo) mostly based on simple URL matching. The Tstat Streaming Classifier engine (2) classifies HTTP connections based on 2 distinguished approaches:

1. Value of Content-Type information in the HTTP's header
2. Signature matching in the video payload, to identify the video container.

log_http_complete:

Tstat can produce a log_http_complete file which logs information from every HTTP request and response. It is NOT generated by default, so it must be explicitly activated with the corresponding directive in runtime.conf.

log_mm_complete:

reports statistics for the RTP and RTCP flows. The classification process exploits a finite state machine that perform checks of version field, sequence numbers and payload types. In particupar, if in the first UDP packet

• the version field is set to 2
• the payload type field has an admissible values (for RTP or for RTCP)
• the UDP ports are larger than 1024 and are even/odd for RTP/RTCP

the flow is marked as possible RTP/RTCP flow (FIRST_RTP/FIRST_RTCP).
When the second UDP segment of this UDP flow (same IP/ports) is observed, then Tstat double checks if it still be interpreted as RTP/RTCP payload controlling if

• the version is equal to 2
• the same ssrc is present
• the seqno is the expected one
• the payload type is the same as before.

If checks succedes, then the flows is marked as RTP and its analysis may start.

For RTCP flows, a simpler heuristic is used:

• the version must be equal to 2
• the payload type must be a correct one
• the UDP ports are larger than 1024 and are even/odd for RTP/RTCP.

If so, the flow is considered a RTCP flow and its analysis may start.

log_dns_complete:

Tstat can produce a log_dns_complete file which logs information from every DNS request and response. It is NOT generated by default, so it must be explicitly activated with the directive in runtime.conf. Moreover, Tstat must have been compiled to include the LDNS library, as reported by 'tstat -v'. Details on the compilation requirements are included in the Tstat source documentation in doc/README.log_dns.

log_skype_complete:

reports statistics for each SKYPE flow identified using the methodology described into "Revealing skype traffic: when randomness plays with you". Note that records change according to the trasport layer (UDP or TCP) used by Skype.

log_chat_complete, log_chat_messages:

Tstat is able to classify MSN Messenger, Yahoo! Messenger and Chat based on XMPP Protocol such as Jabber or Google Talk. In log_chat_complete are reported statistics for each chat flow while in log_chat_messages for each chat message.

log_tcp_complete, log_tcp_nocomplete

Core/Basic TCP Set

TCP End to End Set

TCP P2P Set

TCP Options Set

TCP Layer 7 Set

0000 0000
^^^^ ^^^^
|||| ||||_____ NPN used, no explicit value negotiated (implicitely SPDY)
|||| |||______ NPN used, HTTP/1.x negotiated
|||| ||_______ NPN used, SPDY negotiated
|||| |________ NPN used, HTTP/2 negotiated
||||
||||__________ ALPN used, no explicit value negotiated 
|||___________ ALPN used, HTTP/1.x negotiated
||____________ ALPN used, SPDY negotiated
|_____________ ALPN used, HTTP/2 negotiated

TCP Advanced Set

Additional Tables

Video TCP End to End Set

Video Information Set

YouTube Information Set

Video TCP Options Set

Video TCP Layer 7 Set

Video Advanced Set

log_http_complete

log_dns_complete

Histogram description

An Histogram represents the empirical distribution of a specific index considering a fixed measurement period. For each measured index, Tstat creates and updates an histogram that collects the hit number of that quantity. For examples, considering the IP packet length, Tstat updates, for each observed IP packet, the counter of the number of observed packets with a particular length. At the end of the measurement period, Tstat saves each histogram in a separate TXT file, reset all the values and then restarts to collect samples. The duration of a measurement period is defined by the MAX_TIME_STEP parameter, which is defined in the file param.h, and by default, it is set to 5 minutes.

Recalling that (see HOWTO) Tstat is able to distinguish between IN-coming, OUT-going and LOC-al traffic and among C2S - Client-to-Server and S2C - Server-to-Client, it follows that, when applicable, it generates histograms according to traffic directions. Histograms names are strictly related both to the direction and the type of measure and as to have a quick remainder of the supported indexes it can be used:

    bash> tstat -H ?
    #name             min  bin_size     max  description
    profile_flows     0    1            5    flows handled
    profile_cpu       0    1            4    cpu load [clock/time]
    chat_flow_num     0    1            7    Number of tracked IM flow
    web_bitrate_loc   0    1            7    Web 2.0 content bitrate [bit/s] - local segments
    web_bitrate_out   0    1            7    Web 2.0 content bitrate [bit/s] - outgoing segments
    web_bitrate_in    0    1            7    Web 2.0 content bitrate [bit/s] - incoming segments
    L7_WEB_num_loc    0    1            7    Number of tracked Web 2.0 flows - local flows
    ...

The following tables report a verbose description of all the supported histograms grouped as:

• IP Layer: statistics related to ip addresses and IP protocol;
• TCP Segments: statistics related to individual TCP segments;
• TCP Flows: statistics related to TCP flows;
• UDP Layer: statistics related to UDP flows;
• Streaming Flows: statistics related to streaming flows;
• RTCP Flows: statistics related to RTCP protocol;
• HTTP Flows: statistics related to HTTP protocol;
• Profile: profiling of the machine running Tstat;