Bandwidth: Getting numbers.
Selecting a logtype
You can extract a lot of mail bandwidth usage info with Moxy,
here's how.
You first need to decide what type of information you wish
to extract and configure Moxy to log appropriately. You can
have a look at the Setup and config
section for details but it boils down to 3 options:
- Use syslog or filelog logging if you're
only interested in keeping track of general system use.
You can still extract lots of info (using moxylogalyze.pl,
see below) from these logs and they behave in a manner similar
to regular MTA logging.
- Use gdbmlog logs to have up to the minute bandwidth
usage statistics on a per destination basis (using moxystats,
see below).
- Get the best of both worlds, using sys_and_gdbmlog
which logs to both a gdbm file and the system's syslog facility.
After you've set your selection using the config file's
logtype attribute, start Moxy and send a few emails
Extracting the data
When it comes time to retrieve meaningful data from the Moxy
logs, the method used will depend on the logtype
you are using.
Syslog and filelog
The syslog and filelog log types log information
in plaintext, through the system syslog facility (in which
case the data will go wherever syslog.conf is set
to send mail.*, usually /var/log/maillog)
or to a regular text file, respectively.
You may open these files in any editor and peruse them to
your hearts content. When you want some hard numbers from
these logs, use moxylogalyze. moxylogalyze
is a simple Perl script that can parse Moxy logs and calculate
total bandwidth usage based on a number of criteria. Here
is the output generated for moxylogalyze.pl --help:
Usage: moxylogalyze.pl [--listby PARAM|MATCHES] LOGFILE
Where:
LOGFILE The log file to parse
-l,--listby P List all totals, sorted by P (from, ip or dest).
-v,--verbose Verbose output to STDERR
-V,--version Version info
-h,--help This message.
MATCHES:
--from EMAIL from this sender.
-i,--ip IP from this IP addr.
-d,--dest HOST:PORT mail proxied to this host and port.
--for EMAIL for this user.
Multiple MATCHES may be used (they will be ANDed together).
Note that you can use regular expressions in place of string literals for EMAIL,
IP and HOST:PORT.
Examples:
moxylogalyze.pl --dest psychogenic.com /path/to/logfile
moxylogalyze.pl --listby from /var/log/maillog
moxylogalyze.pl -v --dest ':\d+25' --for 'pat(rick)?' /var/log/maillog
As you can see, you can get data for many many different
scenarios with moxylogalyze. It's important to understand
the exact meaning of the 4 MATCHES arguments.
- --from matches the sender's from email address
as it was stated during the MAIL FROM: phase of
the SMTP transaction. E.g. billy@microsoft.com
- --ip matches the sender's source Internet Protocol
(IP) address at the time the message was sent. E.g. 131.107.3.121
- --dest is the host and port combination
that this email was proxied to. If for instance an email
comes in for two domains - a.com and b.net - a both theses
domains are set to be proxied to c.org port 1025, then a
single connection will be made and this will be reflected
by a single entry in the logs indicating a connection to
c.org:1025 with email for the users 'ua@a.com,ub@b.net'.
- --for matches a recipient's email address. The
log entry in the example used above would match both
--for ua@a.com and --for ub@b.net even
though only a single connection was used (and thus, the
data was sent only once).
The three example uses from the --help output are pretty
self explanatory but here's a little more detailed description:
moxylogalyze.pl --dest psychogenic.com /path/to/logfile
This outputs the number of bytes transfered from any source
to any destination containing the sequence of letters
'psychogenic.com'. Remember that the destination may have
no relation to the recipient email addresses - it is simply
the machine host:port to which this email was proxied.
moxylogalyze.pl --listby from /var/log/maillog
The --listby option allows you to output all
the totals found in the log file. These may be calculated
and sorted by one of from, ip or dest,
as defined above. In this example, the number of bytes sent
to domains Moxy is responsible for will be displayed on a
per sender email address basis.
moxylogalyze.pl -v --dest ':\d+25' --for 'pat(rick)?'
/var/log/maillog
This is a search that shows that you can combine MATCH requirements
- these are logically ANDed - and use Perl regular expressions
in you arguments (the literal strings used in the examples
to date are but a degenerate type of regex). This example
matches all log entries for mail proxied to a machine on anyport
which contains at least one digit followed by the sequence
'25' (eg 12325 but not 25) and destined for at least one user
with an email address that contains 'pat' or 'patrick' (yeah,
the (rick)? can be eliminated without affecting the
number of matches but I'm just trying to demo the use of regular
expressions ;) ).
moxylogalyze.pl assumes you are using the default
log_format (see the /usr/local/moxy/moxy.conf
file for details). If you've modified the configuration, you
can still use moxylogalyze.pl by modifying a single
line in the Perl script. Change the line:
my $Regexp = q|\s*(\d+)\s+bytes\s+from\s+<([^>]+)>\s+\[([^\]]+)\]\s+to\s+\[([^\]]+)\]\s+for\s+'([^']+)'|;
so that the regex matches your output format. You may also
need to change the ordering of the
my ($size, $lfrom, $lip, $ldest, $lfor) = ($1, $2, $3, $4, $5);
lines if you've changed the order of the size, from, ip,
dest and for in the log output format.
Gdbmlog
The gdbmlog log types keeps data in, oddly enough,
a gdbm (GNU database) file. The disadvantage of using gdbmlog
is that log files are no longer human readable. The advantage,
however, is having rapidly accessible and up to the minute
bandwidth usage statistics on a per destination basis. If
having a running total of the number of bytes sent to each
destination host:port combination is what you're intersted
in, gdbmlog is the way to go but how do you extract
information from this database?
The moxystats utility may be used to get current
bandwidth usage for a given host:port combination when using
gdbmlog. The output of moxystats --help
is :
Usage: moxystats -f INFILE [DOMAIN:PORT[S]]
Get bandwidth statistics from moxy gdbm log file INFILE.
-f,--file INFILE Specify the moxy gdbm log file to open.
-a,--all Dump entire contents of gdbm file.
-v,--verbose Print extra info (dates).
-V,--version Output version info.
-h, --help This message.
Thus, to extract bandwidth usage for mail proxied to mail.yahoo.com
on port 25 and to difinium.com on port 1234 you would use:
moxystats -f /var/log/moxy.log mail.yahoo.com:25 difinium.com:1234
and to output statistics on all host:port combinations present
in the gdbm file, use
moxystats -f /var/log/moxy.log --all
Sys_and_gdbm_log
If you choose to use the sys_and_gdbm_log log type,
which logs to both syslog and a gdbm db, you can use either
moxylogalyze.pl or moxystats as described
above.
|
