Tanzim Mogharrarat Log

Introduction

Within tanzim-mogharrarat directory there is a Python script named write_connection_logs_ng.py that you may rename it, but you (support guys) never need to edit it or even open it (and we strongly recomment not to change it unless you know what you do). Every customer has a config file that is located in /home/ISP/tanzim-mogharrarat/isp_config and you put this config file on that ISPs server beside python file with the same name (just differ in extention). for example:
/home/ISP/tanzim-mogharrarat/write_connection_logs_ng.py ==> 93.126.8.2/usr/local/sbin/write_connection_logs_ng.py
/home/ISP/tanzim-mogharrarat/isp_config/naghshejahan.conf ==> 93.126.8.2/usr/local/sbin/write_connection_logs_ng.conf

What is important is that this python and config file on the customers server, but have the same name (one with .py extention, the other with .conf) And after that you should run /usr/local/sbin/write_connection_logs_ng.conf like before (as a client script).

Configuration File

About config file, there are many sections, every section belogs to a log file (that is probably associated with a service type for most customers) and every section, has many optional items. Unless few of them is most probable to use:

Options per Log File(Section)

dir

for example:
dir = /var/backups/archive/adsl

service

service = Internet
OR
service = VoIP
PS1: values Internet and VoIP are not case-sassitive
PS2: default value is Internet, even if service = ... was not provided, it will log only Internet connectios, so you can not write all (internet and voip) logs to a single file

cond_*

Every key starting with cond_ will be supposed as a condition, for example:
cond_ras_desc = ras1
cond_ras_desc = ras1,ras2,ras3
cond_group_name = VoIP-Users,VoIP-Users2

PS: the value for these conditions: {ras_desc, group_name, not_group_name} is multi-str, so you can put multiple values seperated by a comma (other conditions may be added multistr support later)
PS2: Every condition is a filter. If you want to disable a condition, simply remove the line (remove filter) not to write empty value for it.

These 3 config items was probably the essentioal ones. But for some customers we may also need the folowing config items:

cond_not_*

For every column name, you can define a blacklist. For example if you want to remove rows with username of "IN-User" and "IN-User2", you write:
cond_not_username = IN-User,IN-User2
Or exclude rows or some groups, for example:
cond_not_group_name = VoIP-Users-IN,VoIP-Users-IN-New
Or even, for other columns, exclude some specific remote_ip or dnis ... (See all available columns)
cond_not_dnis = 11223344, 22334455

add_phone

add_phone = True
OR
add_phone = False

If add_phone was enabled, in the cases that user's called_id was empty but the user has "phone" attribute (see Admin -> User Info -> Comment -> Phone), it will put phone value instead of called_id in csv file.

PS1: You can write 1 instead of True, and 0 instead of False
PS2: Default value of this option is False, so you can remove that line if you don't want to enable it
PS3: DO NOT enable this option for IN services where all usernames (and then phone numbers) are the same

login_time_from and login_time_to

Default range of login time is from yesterday midnight (00:00) to the last midnight, that is the most-used case. But if you want to get older logs, you can manually set these values in every section, for example:
login_time_from = 2011-06-27
login_time_to = 2011-06-28

columns

Default columns is based on official rules of tanzim mogharrarat (for Internet and VoIP), you can see them here:
/home/ISP/tanzim-mogharrarat/other/Internet.jpg
/home/ISP/tanzim-mogharrarat/other/VoIP.jpg

Default columns for Internet is:
login_time, logout_time, remote_ip, caller_id, mac, username

And default columns for VoIP is:
login_time, logout_time, ras_ip, caller_id, called_ip, called_number, dnis, username, user_id

Other available column names are:
group_name, retry_count, credit_used, duration_seconds, e1, mvts_caller_id

If one of customer want to change columns, you can override default columns by setting this. for example:
columns = login_time, logout_time, duration_seconds, ras_ip, caller_id, mvts_caller_id, called_ip, called_number, username, user_id

Current supported columns are: user_id, username, login_time, logout_time, remote_ip, caller_id, mac, called_number, ras_id, dnis, called_ip, serial, ras_ip

PS: spaces before or after commas does not make problem

alias_*

every key starting with alias_ will specify an Alias or Descriptive name in the HEADER (first row) for a column. for example, if you write:
alias_dnis = Calling Number
in the header, title of column "dnis" will be "Calling Number"

for example, Ferdose want to show user_id AS Serial, meant values are user_id s, but the title is "Serial" (and the reader will not understand that these are user_ids), so for that we write:
alias_user_id = Serial

compress_file

compress_file = True
OR
compress_file = 1
if you want to csv file be compressed.

default_called_ip

This is default Gateway IP for VoIP. For example:
default_called_ip = 90.120.3.3

default_dnis

Similar to default_called_ip, but about DNIS (calling number):
default_dnis = 12345678

default_...

You can specify default value for every column, for example if you write:
default_remote_ip = 127.0.0.1
Then all empty remote_ip's will be replaced with 127.0.0.1

ras_e1_map

If you want to have E1 Numbers in columns, you have to define a mapping between ras_id's and E1 numbers, for example:
ras_e1_map = 1: 44556677, 2:55667788
That means, if ras_id=1, E1 number will be 44556677, and if ras_id=2, E1 number will be 55667788
Don't forget to add "e1" to columns (thats not included by default), for example for a VoIP log:
login_time, logout_time, ras_ip, caller_id, e1, called_ip, called_number, dnis, username, user_id
(e1 is added after called_id)

remove_isp_mapped_users

This is True by default. That means it will remove all rows that associated to a Mapped User of an ISP.
If you want to disable it, write:
remove_isp_mapped_users = False

join_voip_rows

This is True by default. That means it will join (merge) duplicated VoIP rows (with the same unique_id_val and same called_number).
If you want to disable it, write:
join_voip_rows = False

exclude_empty

Sometimes, for any reason, some of Remote IPs are empty (but not all of them or most of them), and the customer want to not log empty-IP items. So in this case, we write:
exclude_empty = remote_ip

And if the customer want to also remove items without caller_id, we should write:
exclude_empty = remote_ip,caller_id

filename

The csv file name will be automatically set by date range, even if you use custom date range (using login_time_from and login_time_to config items), but if you want to manually scepcify file name, you should use thid option, for example:
filename = custom-log.csv
Do not put slashes in this, this is not absolute PATH, this file will be saved in directory scepcified with dir = ...

dates

This option will overwrites any of login_time_from, login_time_to, and filename.
For example: dates = 2011-9-1, 2011-9-10
Or a range using two dots:
dates = 2011-9-1 .. 2011-9-5
That is equal to:
dates = 2011-9-1, 2011-9-2, 2011-9-3, 2011-9-4, 2011-9-5
Or use both comma and ..
dates = 2011-8-28, 2011-9-1 .. 2011-9-5, 2011-9-25
If you use this option in a section, that section will be like many sections, since hase many scv output file, one file for every date.

Note: Zeros before numbers does not matter, for example 2011-1-1 and 2011-01-01 both are correct

General Options

You can provide a section name "GENERAL" (uppercase), and put some general options there. Currently supported options here are:

date_type

Acceptable values: gregorian, jalali, g, j (all in-case-sensitive)
This is date type for SHOWN dates in CSV file (and will not applied to input dates written in config file)
Default date type is gregorian of course. So If you want to have show dates in Jalali, write:
date_type = jalali
OR
date_type = j

sleep

This is sleep time between every log file (ie: log section), and is in seconds. For example if you write:
sleep = 2
it will wait (sleep) 2 seconds after saving every csv file (for a section) to prevent system to be high-load.

compress_file

This is default values of compress_file in sectoins, so if you enable compress_file in GENERAL section, you have not to enable it in all sections, and it will compress files by default (unless you disable it in the log section).

default_called_ip

Like previous, this is default value. For example if a customer has many VoIP log files (many sections for VoIP), while all Gateway IPs are the same, you can write it once in GENERAL section, instead of writing in all VoIP sections.

default_dnis

Like default_called_ip, but about DNIS(calling number)

default_...

Like that described in Section options

ras_e1_map

As described in Section options

remove_isp_mapped_users

As described in Section options

join_voip_rows

As described in Section options

Other Notes

spaces before or after equal sign (=) are not essential.