FYI: ETail - A tail like monitor for chat.log

Lorna

Mature
Joined
May 11, 2015
Posts
27
Avatar Name
Lorna La Arauka
ETail - A tail like monitor for chat.log

Entropia tail is a monitoring tool for the chat. It can read and filter in real time the log and do some actions to alert the user if the string searched is found.

Ussually when I'm sweating, or hunting or chatting I don't read the messages in the other chats and then the mob is dry or I've lost somme offer or my weapon is wasted before I noticed it. I've checked some log monitoring tools, real nice tools, but either too complex, like logExpert, or they haven't all all the things I wanted So I decided to try and do one for a not really concentrated player like me.

The program isn't finished yet, and some cool features like searching for trading offers or exporting the results into csv or json aren't finished yet, but the alert and monitor features are totally functional. ATM you can define a filter (regular expressions are acepted) and if the string is matched you can define the following behaviours:

-Set a color for the line found, useful using two screens.
-Hide a line.
-Play a sound using a wave file.
-Read a user defined text usig the TTS of the OS, (I find this one really useful since you can write down wht the computer will say if the string is found)
-Print an alert into the program window.

Also the filters used can be saved and loaded for each activity, the lines found can be saves into a text file for later processing and the log is rotated every 5MB (this was forced by my mistake selecting the programing language HTA/VBScript).



CURRENT VERSION: 0.09 alpha.
No size limit in chat.log. No log rotation available by now, it will be introduced as an option in the last update before the beta version. Advanced filters are unchanged.


Outdated, kept for older versions users (if any).


==============================================================================
IMPORTANT NOTICE ABOUT ROTATING LOGS
==============================================================================
Due the limited file support of VBScript large files (more than 5MB, about 50k lines) will overload the processor, to prevent this the program will automatically rotate the log and compress the original file in a zip file with the following name format "logfilename_DATE_TIME.zip" in the same folder.
The script will promp for the method of rotation. The reencoding method (answer yes to the prompt) is slower but it will correct any encoding fault and remove any blank (chr(0)) characters, also the resulting logfile will be bigger because it is using Unicode instead of ASCII characters.
Answering "No" to the prompt will just copy and zip the log file, but if it has any character encoding errors, they will not be corrected.

Note that if you reencode the log probably you may indicate to any other processing programs that the charset is Unicode UTF-8. If you didn't reencode the log, maybe you will need to correct any faults, like eliminating blank chars.

==
Recovering a failed attempt to rotate.
Before the compression start, the original log is moved to a temporary work file, named with this pattern "logfilename_DATE_TIME.tmp". This file can be renamed safely to "logfilename_DATE_TIME.log" and then saved in another place, note that this file isn't converted.
==
 
Last edited:
Adress of Etail

It can be downloaded here:
http://lorna.chafalladas.com/etail/

7zipped just extract into a folder of your chioce.
Setup can create shortcuts and will be extracted to "MyDocuments\Entropia Universe\Entropia Tail"
 
Internet Explorer Script Error :
Line 401
Char 9
Error: subscript out of range : '[number: 6]'
code :0

Also I dont like it messing with my log file , it said the logfile is too big when its only 7 mb. also it renames and also creates a zip file and resets my chat.log file to zero.
 
Think twice before you download this.
Just a word of warning.
 
Internet Explorer Script Error :
Line 401
Char 9
Error: subscript out of range : '[number: 6]'
code :0

Also I dont like it messing with my log file , it said the logfile is too big when its only 7 mb. also it renames and also creates a zip file and resets my chat.log file to zero.

Thanks for the feedback.

I will copy some of the readme in the starting post.
The log is copied into a zip file, if you want to restore or use it, just do a backup of the new one and copy the old. I started in VBS/HTA because I thought it would be easier to code and use simple system resources, unfortunately I was very wrong, and VBS handles very poorly text files so it needs to rotate every few MBs. You can merge both files just copying the content of the newly created chat.log at the end of the old one using ant text editor like notepad, notepad++, SciTE or geany.
¿Wich SO are you using? I have only tested in XP32 and W8 x64, I'll try to reproduce the error.

PD.
The error raised after deleting or creating new filters? Sometimes it loses data in the filters array when doing it, but I couldn't find the mistake yet.
 
Last edited:
Think twice before you download this.
Just a word of warning.

Please, explain further. Don't see anything wrong with process explorer or task manager. Although I'm using the obscure htmlhelper dialogs, no bytes are sent or received yet to the net, also didn't detect extrange writing when outputting or using the TTS.

Anyway, it's in early stage, so it will be buggy.
 
Forum account created this month, not MA approved, 7zip file could contain anything...5 posts..etc
Its nothing personal against the poster.
 
mmm 7zip , nothing wrong with it I guess but why not good old zip or rar?

I cant use this unfortunately my log files get pretty massive with the activities I do on Normandie mothership so good luck with it .

I got the error after clicking No to whatever the log message was.
im using 32bit windows XP but honestly as soon as I see any app throw those VBscript errors I generaly trash it .
 
mmm 7zip , nothing wrong with it I guess but why not good old zip or rar?

I cant use this unfortunately my log files get pretty massive with the activities I do on Normandie mothership so good luck with it .

I got the error after clicking No to whatever the log message was.
im using 32bit windows XP but honestly as soon as I see any app throw those VBscript errors I generaly trash it .

Thanks for the info, Ill check it.
Well, I use 7zip as it have better compression ratio than Zip or Rar and its very fast.

Arangar

Forum account created this month, not MA approved, 7zip file could contain anything...5 posts..etc
Its nothing personal against the poster.
Fine with that, I'm not very active on the ingame chats, neither the game forums:ahh::ahh:. BTW any modern antivirus can check 7zipped files, thats why I also put the 7zip with the installer. Also recommend using some safe browsing addons like noScript for Firefox.


UPDATE:

Found the problem. The etail_filters.conf file is from an earlier version and has the wrong number of fields. You can copy a file of the same name stored into the "files" folder over the "etail_filters.conf" in the etail root dir to solve the problem.
 
Last edited:
Well, I use 7zip as it have better compression ratio than Zip or Rar and its very fast.

.zip unpacking is built into windows.

I don't really like when someone packs something with something else, and wants you to download a separate extractor program for it. Especially, someone sends something like .rar in a mail to an end user, the end user then installs something that's either shareware and hijacks windows default .zip handling (and from that Point will nag about 30 day license period) or is downloaded from a source that uses an adware installer program. In once case, winzip made a user confused; he wanted to see what he sent to the other party, but the problem was winzip showed the content of a harddrive directory when clicking on parent directory inside a zipfile.

This is just a general thought about people who pack their things with anything except zip for windows, or possibly tar.gz on *nix.

How do you do the tail functionaily btw? I mean generally. If you get end-of-fil, do you keep Reading until end-of-file is false? In case I want to do something like that myself.
 
Last edited:
.zip unpacking is built into windows.

I don't really like when someone packs something with something else, and wants you to download a separate extractor program for it. Especially, someone sends something like .rar in a mail to an end user, the end user then installs something that's either shareware and hijacks windows default .zip handling (and from that Point will nag about 30 day license period) or is downloaded from a source that uses an adware installer program. In once case, winzip made a user confused; he wanted to see what he sent to the other party, but the problem was winzip showed the content of a harddrive directory when clicking on parent directory inside a zipfile.

This is just a general thought about people who pack their things with anything except zip for windows, or possibly tar.gz on *nix.

How do you do the tail functionaily btw? I mean generally. If you get end-of-fil, do you keep Reading until end-of-file is false? In case I want to do something like that myself.

You got a point there, although 7Zip its getting a name and is open source is likely that many people will be careful enough to suspect something from a not known format. I'll add a common zip file to the downloads.

When I read the file I store in a global variable the size of that file, then before opening the file for reading I check if its size has changed, since it is faster than opening the file and counting the lines, VBScript can't tell the lines of a txt unless you read ALL the lines.
I don't know for sure if it will be the same in other languages, but if I remember correctly, opening a file involved several more operations than checking its size at the low level.
 
the difference is marginal in this case , your 7zip package is 4.15 mb and a plain old vanilla zip file is 4.45 mb .

(7zip files can also be unpacked using rar and thats what I used) , if I had to download another unzipping package , I wouldnt even bother with it for similar reasons to Aia .
 
ETail 0.08 is out

Hi, it's been a while since the last update, but I've finished a new tab, the advanced filters.

Now you can select a channel, an avatar and three strings to search instead of a single string. There is also the posibility of executing a program and pass it parameters like a command shell pipe from the chat window. This version includes a calculator and a timer in order to explain the use of this feature. You can use and edit a lexer to match synonims like WTS, Selling, Sell or Lysterium, lyst, lst. Also you can controol the behaviour of the search parameters to find only near matches or in all the line.

I'm preparing a brief documentation of the use of the new features that I will paste here when finished.

You can find the program here:
http://lorna.chafalladas.com/etail/
 
Last edited:
Update, manual to advanced filters added.

Here is a copy of the text on the etail's page. It explains how to search for several different words in the chat log, use the dictionary and how to run programs from the game chat window.

Entropia Tail Advanced filters usage.

The advanced filters tab have several fields and buttons to interact and create custom filters that can search for strings in determined channels and said by concrete avatars. Supose that you want to buy someting in one planet, and you know that sometimes another player comes to that planet and have good offers, maybe you have that player in the FL, but you don’t want to disturb him or you are in the middel of something, like a animated discussion about the damage per pec of the Opalo in your soc. You can make a filter that exclusively searchs for that player messages in a determinated channel and only if he says something related with selling a paticular item. Below is an explanation on how you can make it.

  • The Channel input text:


By clicking into the corresponding cell a dialog to edit its contents is opened. You can add here a list of comma separated channesl to search for the rest of the parameters. PE if you type “trade,calytrade” the program will search coincidences only in the “trade” “#trade” and “calytrade” chanels.

Match channel

  • The avatar input text:

In the same way you selected the channels, you can select avatars, note that the chat log writes the full avatar name, so you can search for full names to avoid unwanted matches. PE “Lorna La Arauka,Another Avatar Name,Nickname” the first two will search the exact avatars, but the later will match any avatar that has “Nickname” as a part of his avatar full name.

  • Match avatarThe parameters input text:


Those work almost the same, but there are two important diferences. The words you enter can be checked against a dictionary to find sinonims and variants, so you won’t need to type all the equivalent words that players use in the chat, and you can determine how the program will seach for the string entered. This is explained in the word, scope and dict sections of this post.

Match fstparWord or string search:

Here you can tell if the program will search the text you’ve entered as an independent word or if it will be matches inside a string. PE. If you have typed “Lyst” and checked as a search string, and not as word the program will match “Alyster” “Lysterium” and “Lyst”, but if you tel to search as a word, it will only match “Lyst” if is typed like “the Lyst is going down again” but not “the Lysterium is going down again”.


  • The Scope parameter:


Here you can tell the program if it will search for the precedent string in all the line or only in nearby characters. This can be useful if you are searching for prices, since a seller can be offering several products and your desired price can match the one of another thing. PE, one player sells “WTS: Mat1, mat2, Lyst 110%, Mat3, Mat4 105″ if you search for Lyst at 105 and you set the scope to “0” (All the line) the program will tell you that it found your rqeuirements, but is not really true. But if you limit the scope within 6 characters, it will search only the 6 letters after the string “lyst” is found, and the exmaple line will not be marked as found.


The dictionary file:

Here you can edit the equivalents of search terms or select different dictionary files. For example, when you click to edit the file, a list of the used words is shown, you can add new words to the list using tha add button, and you can delete them by just leaving the corresponding field in blank. The dictionary works adding the words listed to the search expression if the words you’ve entered in the params edit form are matched. In this way you can just type “WTS” and the program will add “Selling, sell” or whatever you want if is entered as a sininim in the dict file you are using for the filter.

dict

  • The execute program action:


When you select the actions filter options form by clicking on a cell of the “Action taken” column, a form with a text field and a change executable file button will be presented to you. Here you can type a command that is in the path environment, or select the executable file of it. Then you can customiza a bit the behaviour of the execution.

Yau can type the parameters that the program will use in the text field, but if you want to pass the parameters by tiping them in the EU chat window, you must add the PARSEPARS string after the name of the executable, more detailed info can be found in the “etail_readme.txt” file.

Then you must enter a “command” in a parameter field text in order to tell the program when this action will be executed. Be warned that you must also enter a chanell or list of channels to search and the full name of the avatar that has permision to execute a program in your computer. This prevents that the command is executed if someone types a word that you are using as a command in the chat. Anyway, use this action cautiously, you will be executing programs in second plane and if you don’t know them they could fill your memory or overheat your computer if many copies are running simultaneously.

In the files provided there are two utils stored in the addons folder of etail. ETcalc.vbs and ETimer.vbs. If you type “local” in the channel window, your complete avatar name in the avatar form, “calc” (or calculator or somethng to remember that is a command to calculate arithmetic operations) and in the Action taken form you select run program and copy this text “addons\ETCalc.vbs PARSEPARS”, then you can type “calc (2*9)/3″ in the “local” chat and it will tell(in English ATM) you the result and copy it to the clipboard.

For running the countdown timer script util ETtimer you can type “mytimer” as string parameter and copy “addons\ETimer.vbs PARSEPARS” to the program input field. It will need three parameters, the time to start the alarm in hh:mm:ss, the number of times that the alarm will be repeated and the text that the alarm will read. PE typing mytimer 00:00:05 1 DEFCON 1 will say “defcon one” after 5 seconds, and repeat it after a second.



There is also a “view” icon to check the resultant regex if you want to modify it to your own purposes, remember that the normal filters can use regular expressions if you type them.

I think the other options are self explanatory so you can fiddle with them easily without the need of further explanations.
 
How will this actually work in this situation:
You say in your first post there is a limitation of 5 mb or 50K lines .
I manually split my chat.log file anyway for analysis purposes but to give you an example my latest chat.log file started on the 9th of this month and currently contains 168,000 lines and is 13 mb in size.
What will happen in game when the 50K line limit is hit while actually playing ?
Is this going to cause any lag issues ?
 
How will this actually work in this situation:
You say in your first post there is a limitation of 5 mb or 50K lines .
I manually split my chat.log file anyway for analysis purposes but to give you an example my latest chat.log file started on the 9th of this month and currently contains 168,000 lines and is 13 mb in size.
What will happen in game when the 50K line limit is hit while actually playing ?
Is this going to cause any lag issues ?

Mhh, when I used it with my old 30MB file the only thing was that the processor was a bit overcharged since the program uses around 20% 25% of CPU, but dind't noticed lag.
It seems important so I'll try a workaroud, like a temporal file using the last lines of the main chat.log so you don't have to rotate it.
For the another question, ETAil only does the file size check at the start of the program, when you are using it it will no check the size. It would be very inconvenient.

BTW, the code for file size checking is this
Code:
	FileSize = objFSO.getFile(fich_log).size
	If FileSize > 5000000 Then
		rotateLog(fich_log)
	End if

It starts at line 134.
You can edit the "etail.hta" file to increase the file size or just comment (using ' before a line comments it) or delete that lines with the notepad or any plain text editor like scintilla or notepad++.
 
Last edited:
ETail 0.09 is out

Regarding the concerns of SoReal about the chat.log size I've come with a dirty and cheap solution. The use of an external program to read and parse the last lines of the chat.log file.
Instead of using tail.exe from the ported gnu utils or telling users to download the 2003 Resource kit I did a tiny console program in C++ for reading the last lines and write them to a temporary file "Etail.tail", this file will be emptied every execution of the program. This executable file is named "Filereader.exe", and the source code is included in the packages with criptic comments for compiling in VS2013, Devc++ or Code::blocks included.

Now ETail can parse files bigger than 39MB without lagging, processor overload or other isssues related to the file size.

There are two versions of ETail package with Filereader.exe, the 0.09 one is for 7 or later OS, and the 0.09B is for all windows form XP32. The main difference is that the 0.09 is smaller than 0.09b, but will not run in all windows versions.
 
Last edited:
Etail 0.010 is out

Fixes a bug that caused the program to load the same advanced configuration everytime, now the starting conf will be the last one you saved.
Changed the text in the filters and advanced filters tabs from "Save filters" to "Save configuration" because not only the filters are saved, also the configuration that load that filters at the start is saved.
Changed the Tab "maps". The maps will be an external program that can be called via the advanced filter action "Run program" with some parematers still to be determined, double clicking in the explorer or via some configurable buttons in the maps tab, maybe it will be renamed to addons or utils tab.

The map is still at very very early stage, only Calypso is available and many locations are outdated, I've used the old "Entropia locations" very nice and useful program to get the coords, but since it seems to be abandoned, a lot of locations seems to be out of place. The only way to edit them now is by editing the "Calypso.csv" list of coords placed at "ETMaps\maps" dir.
 
Entropia Tail 0.11 is out

New features in the map, fixed security of the Run program advanced filter. Added locations to Thule and Arkadia.

Run program MUST now have the channel and avatar name filled. The name must be the full name of the avatar (name nickname surname). The map can be updated via chat, you can make a new advanced filter to call the “addLocation.vbs” script, using this instructions:

addLocation.vbs must be invoked with parameters without quotes and separated with spaces. The mandatory parameters are

-Map name, a name from the planets.csv file
-Location, the coordinates provided from the game in the same format provided EG [latitude,longitude,altitude]
-Type of location, there are some predefined types, here is the list, the names are case sensitive:
Revival
Outpost
Teleport
POI
Mission
Estate
Ore
Mob
“POI” stands for Point Of Interest, like a nice landscape, building or similar remarkable in world things, like the Gamer Chick Memorial.
-Descriptive text, a text describing the location must be quoted EG “This is a nice location”.
From here all parameters are optional and can be surrounded by quotes in order to be stored as one single data field, otherwise they will be treated as separate parameters.

So to call addlocation fron the chat in order to add a location, if youv’e defined a run program filter like “addcoords PARSEPARS”, tou must type in your selected commands channel

addcoords Arkadia [lat,lon,height] POI "A nice view in Arkadia"

If all went well you can hear t sucess message, if the coordinate hasn’t been added there be a spoken failiure message. After that you can refresh the map by pressing the refresh map, selecting it at the dropdown menu or via chat if you started the monitoring in the ETMaps.hta program and typing showmap “Name_of_the_map” you want to view.

More details are in the ETMaps readme file.
 
have to comment, as i wrote a similar tool for myself a while back, why is there an issue with file size? Perl and the tail module available (about 26k) has no problems with large files, and a gnu ported tool wont either (dont recall if i tried this or not). tail is happy for files of hundreds of GB in size, no point reinventing a cludge solution.
 
have to comment, as i wrote a similar tool for myself a while back, why is there an issue with file size? Perl and the tail module available (about 26k) has no problems with large files, and a gnu ported tool wont either (dont recall if i tried this or not). tail is happy for files of hundreds of GB in size, no point reinventing a cludge solution.

VBscript needs to read the entire file to access a text line. The only way to read a 30 MB file in less than 60 seconds is very CPU consuming. Instead of making users download the windows 2003 resource kit that has a tail command, make them to search in sourceforge for the gnu windows utilities or installing powershell I chose to write a tiny program that do the read for me. I, myself would download those if it was the case, but I thought that there was no need to make a user to lurk everywhere (because the links tend to change too frequently) to find and download the tools needed, also in this way I avoid license troubles. There is also a way to do a kind of tail in cmd, but uses a lot of memory and needs admin rights to work correctly.

BTW never saw a log greater than 100MB in a system, an AS400 technician told me that they are too heavy and tend to make troubles beyond that size. I supose it's because logs are used by a lot of programs, people and other systems at the same time.
 
Scanned with 21 different Anti-virus systems. All clear

TsjfYW7.png
 
Scanned with 21 different Anti-virus systems. All clear

TsjfYW7.png

Thank you very much :).
I think that my system is clear, but there is always the chance of being infected without noticing, a second opinion is always welcome in these matters.
BTW, the size of the file is because it has a lot of images, maps and tiled maps mainly, it will grow a bit untill all the planets have their own map rendered.
 
Last edited:
Back
Top