Add log entry type and update docs

README.md

@@ -73,8 +73,8 @@ queries **and** block host names I don't want being accessed. I do
 this by assigning the bogus address "127.0.0.255" to the names in my
 server's "hosts" file, which is used by "dnsmasq" to answer DNS
 queries. That address is a **valid** "localhost" address, so will
-**immediately** fail requests unless you put a http(s) server. And
-I'm sure you can imagine those possibilities.
+**immediately** fail requests unless you put a http(s) server on it.
+And I'm sure you can imagine those possibilities.
 
 The other source of information and block capabilities is "iptables"
 / "ip6tables". I added rules to log "new" packets. The parsing
@@ -122,12 +122,12 @@ Although I think C++11 defines the minimum viable version of C++
 there are places that its still not available, which is most likely
 to happen with oddball tool chains like those provided by Netgear.
 But even my RaspberryPI 2B doesn't support it. Well... maybe with an
-OS upgrade... I'm hoping to support a very wide Linux platform
-coverage.
+OS upgrade... In the end I'm hoping to support the broadest range of
+Linux powered platforms.
 
 I'm doing all of my development with Linux. It may be possible to do
 something similar, maybe with very little modification, on BSD based
-platforms. I don't have them and therefor won't be personally working
+platforms. I don't have them and therefor won't personally be working
 on it. But, unless it makes things really ugly, I'm not opposed to
 contributions on that front.
 
@@ -138,25 +138,141 @@ Installation & use
 
 This is still **VERY** crude and incomplete.
 
-1. Configure the source: the basic setup is hardcoded in the source.
-   I'm just testing right now. ;-) There are three lines near the
-   bottom of "iptraffic.cpp" to be concerned with:
+1. Type "make" to compile. Hopefully it compiles for you.
 
-    * `#define PATH "/srv/backups/iptraffic"`: this is the base path
-      to the work directory. Its prepended to other paths, read on.
+2. Create a configuration file. See "Config File" section below.
 
-    * `ifstream log(PATH "/test.log");`: defines the log file to
-      process.
+3. Run `iptraffic ...`. See "Command Line" section below.
 
-    * `ofstream out(PATH "/processed.log");`: the file with the
-      combined data that's written out.
+4. Review the output.
 
-    * `config.load(PATH "/iptraffic.conf");`: Path to a configuration
-      file that lists networks to consider as "us" and connections to
-      ignore, basically the set that is considered "OK".
 
-2. Type "make" to compile. Hopefully it compiles for you.
 
-3. Run "iptraffic".
+Config File
+-----------
 
-4. Check the output.
+The config file is a sloppy INI style file. Its broken into sections
+that start with a header, which is a line containing a name
+surrounded by square brackets ( [...] ). The content of the section
+stretches to the next header and will either contain name / value
+pairs or specifically formatted text data.
+
+Remarks are lines that start with a pound sign (#). I am not using
+the MS style semicolon (;). Blank lines are also ignored.
+
+Currently there are two sections allowed in the file:
+
+ - **us:** which lists the ip address range(s) that are used inside
+   your network (us). Every address not matching is considered
+   outside your network (them). This is used to determine direction
+   of connection (in/out).
+
+    Each line is a single entry. The entries can be in IPv4 & 6
+    notation and should be in a format that matches your iptables log
+    output. For IPv4 that means no leading zeros. For IPv6 **all**
+    hex digits are usually logged, even the zeros. Address prefixes
+    can be used to define networks and they should end with either
+    "." (IPv4) or ":" (IPv6). With IPv6 it could be useful to end on
+    a boundary other than those denoted by colon.
+
+    Example: "192.168.1." matches the 192.168.1.0/24 network.
+
+ - **ignores:** is a set of TSV records that define what traffic you
+   are OK with happening, which are ignored and not reported in the
+   output.
+
+    TSV records contain the following fields separated by tabs:
+
+    1. Our / internal address or "*" for match all addresses.
+    2. Our port number. 0 matches all.
+    3. Outside / their IP address. "*" matches all.
+    4. Their port number. Zero matches all.
+    5. Domain name associated with the destination. "*" matches all.
+    6. IP protocol used (must be all caps):
+
+        - **ICMP:** pings, trace route, ... Port #s match the ICMP
+          message "type"
+        - **TCP:** the most common network connection (HTTP(s), FTP, ...)
+        - **UDP:** Connectionless IP protocol, used for DNS, NTP, ...
+
+    7. Incoming (1) / Outgoing (0) connection.
+
+    A network connection that matches the filter defined by the
+    fields will be ignored and not reported to the output. In this
+    phase this is basically how we define what's "OK" traffic. Its
+    assumed blocking is handled in the firewall and therfore not
+    showing in the logs.
+
+    example:
+
+    ```
+    *	0	*	443	hackaday.com	TCP	0
+    ```
+
+    Ignores a connection from any internal address to a remote
+    address for "hackaday.com" using the remote port of 443 in the
+    TCP protocol. Basically this ignores (marks OK):
+    `https://hackaday.com/`. **BUT** other connections generated by
+    the content from that connection will still show in the output.
+    So this means things like g00gle analytics, CDNs, external
+    JavaScript and CSS servers, ... will still be reported.
+
+
+
+iptraffic Command Line
+----------------------
+
+The iptraffic command uses the following syntax:
+
+```
+iptraffic -c {config file} \[-o {output log}\] \[{input log} \[ ... \]\]
+```
+
+- **{config file}:** must be specified and for best results must
+  contain at least the \[us\] section with your internal addresses.
+  See "Config File" above. Otherwise the "direction" indication is
+  meaningless and it gets harder to write ignores.
+
+- **{output log}:** if supplied is the file name to write the
+  agregated and filtered content to. If the "-o" option is left off
+  output is written to stdout, in UNIX tradition.
+
+- **{input log}:** One or more input log files can be specified on
+  the command line. They will be processed in the order specified and
+  resulting data written to the specified output or stdout.
+
+**NOTE:** Its assumed that the content of an input log file is in
+chronological order (pretty typical) and that the files are specified
+on the command line in chronological order. Data collected from the
+earlier logs will be used to satisfy needs in the latter logs. This
+affects how DNS names (queries) are mapped to connection addresses.
+Its assumed that the DNS query is answered before the resulting
+connection is made. This is the natural order of things.
+
+This program is mainly intended as a test _jig_. To test both the
+concept and DNS to connection mapping algorithms, which still need
+work.
+
+
+iptraffic Output
+----------------
+
+Here's a sample of an unexpected connection I found being made:
+
+```
+Jun 15 02:06:28 192.0.2.75 -> 74.125.195.188 TCP[5228] mobile-gtalk.l.google.com
+```
+
+This shows my tablet calling home to g00gle for something. The
+internal address was changed to match the "demo" IPv4 network, as
+specified in RFCs. This line consists of the following information in
+order:
+
+ 1. Date / time stamp of the iptables log entry
+ 2. The _us_ (internal) address, assuming your config file is correct.
+ 3. Direction of the connection. "192.0.2.75 called _them_".
+ 4. The _them_ (external) address, assuming your config file is
+    correct.
+ 5. IP protocol & port of the destination
+ 6. The DNS name if a matching DNS query was logged prior to the
+    connection getting logged.

iptraffic.cpp

@@ -167,9 +167,12 @@ struct IPtraffic: public cBaseApp {
 
       /// process connections ///
 
-      if(ln.count>5
+      if((ln.count>5  // old style
       && ln[4]=="kernel:"
       && ln[5]=="ACCEPT"
+      ) || (ln.count>6  // new style
+      && ln[4]=="vmunix:"
+      && ln[6]=="ACCEPT")
       ) {
         conn = ln;
         if(!pre_match(config.us, conn.us)) conn.swap();