Python tools

miathan

Old
Joined
Jan 24, 2008
Posts
99
Location
Sweden
Avatar Name
Alindrina Alli Golden
Added csv output to the skill scanner so it can be used with entropiawiki and other tools.
 

Wistrel

Elite
Joined
Mar 31, 2005
Posts
2,534
Location
The Arctic (GMT)
Society
Entropia Pioneers
Avatar Name
Wistrel (Wisty) Chianti
Just had a read through the descriptions. That sounds super useful (and didn't know about the existence of the json file). I really like the before and after thing and nice to see an open source skill scanner that could be build upon/re-used if needed. Must have taken ages to do the log aggregator! Good effort!!
 

TheOneOmega

Elite
Joined
Oct 5, 2008
Posts
2,649
Society
Freelancer
Avatar Name
Marie TheOne Omega
Super interested to learn about the json file. Is there any information or a link to it on the website, or is it a hidden Easter egg?

In particular, I'm curious about the manner in which it may be used. I assume the only reason for the file to exists would be for people to make automated requests to scrape the text of the file, as your program seems to be doing. Would it be possible to, and would MindArk allow, programmers to request an updated copy of the file every second or so, in order to create a complete record of a hunt/mine/craft (computing the before/after of every looting instance)?
 

Wistrel

Elite
Joined
Mar 31, 2005
Posts
2,534
Location
The Arctic (GMT)
Society
Entropia Pioneers
Avatar Name
Wistrel (Wisty) Chianti
Super interested to learn about the json file. Is there any information or a link to it on the website, or is it a hidden Easter egg?

In particular, I'm curious about the manner in which it may be used. I assume the only reason for the file to exists would be for people to make automated requests to scrape the text of the file, as your program seems to be doing. Would it be possible to, and would MindArk allow, programmers to request an updated copy of the file every second or so, in order to create a complete record of a hunt/mine/craft (computing the before/after of every looting instance)?
I didn't check for sure, but I suspect it is being referenced/used by the javascript in the items table that you can see on your account on the website. I note that table has some dynamic functionality that allows re-ordering based on which column you click so suspect that is enabled by way of the page using the data in JSON format and then presenting that in table form. That's probably its only reason to exist.

As for the only reason for the table to exist, it's been there for over a decade and a half (although think it used to be static HTML). Generally having a comprehensive list of your items outside of the game is handy - especially when you "loose stuff" (what storage box on what planet was it?) and historically people used to copy and paste it into tools that would value the items using resources like entropiaauction (run off the expiring auction items feed that used to be online - not sure it still is... or where it is if it still exists) or PEbay. Not sure if either of the those two are still around... probably not.

So yes, while it isn't the only reason for the data to exist, the items list, in one shape or another has more or less always been used by community tools.

As for if it would be possible to programmatically access the JSON at regular intervals without the need for human intervention... that depends. The key blocker is the sign in process and the maths captcha. Those will be pretty awkward to get round however it might be that once you have signed in once, you can maybe maintain a session via a cookie. I've heard of people simply grabbing a cookie from their browser and reusing it in their program to get access to some sites but that by no means also means this would be possible in the case of Entropia.

As for hunting tracking, there was a log aggregator in the example programs in the GIT repository that is tracking stuff as it shows up in the log and the log is fair game, just a text file on your PC. There would be benefits of being able to get at the JSON without (or with only initial) human intervention though.
 
Last edited:

miathan

Old
Joined
Jan 24, 2008
Posts
99
Location
Sweden
Avatar Name
Alindrina Alli Golden
Yeah, the json file is used by the client side javascript on https://account.entropiauniverse.com/account/my-account/my-items/. That's the reason why it exists.

Regarding automatically garbing this from a script, there are a few caveats. Like @Wistrel said you need to login first (password, captcha and one-time-code) so the easiest way would be to log in normally using your browser and extract the session cookie.

Another "problem" is that you can't access "my items" too often, you will get blocked. You can see this by just refreshing the "my items" page a few times.
So every second will not be possible, but maybe every minute or so (not sure what the timeout is).
Personally I just get the items before and after each hunt.

The last issue is what MA would think of a script like that, they might think it's an abuse and ban you? I haven't asked them, probably good to do first.


My long term goal is that we would have a really good hunting tracker (like "Little Big Mining Log" for hunting :))
The easiest way is to use the log files as much as possible because they are easy to parse.
For loot it's only possible to get that output when team hunting though.
You can upvote this one so we can get loot logs also when solo hunting:
The other option is to OCR, I think LBML uses this a lot.
I made some initial test with Python and Tesseract and I think it would be possible to OCR the loot output from the top left corner.
 

Wistrel

Elite
Joined
Mar 31, 2005
Posts
2,534
Location
The Arctic (GMT)
Society
Entropia Pioneers
Avatar Name
Wistrel (Wisty) Chianti
I seem to remember there was an option to turn off loot window and have loot in chat instead... I've never done this but guessing it would maybe work like team loot does? Not sure... possibly though you still won't get the TT value.

Interesting on the "problem" I actually had the exact same thing happen to me the other night when I was having a cursory look at how plausible it would be to write an auto-login. I wasn't sure if it was a coincidence though as I was doing it quite late and wondered if something had gone into maintenance. I thought also maybe it was because I was accessing the page without a browser but it sounds like, from what you say, you can achieve the same effect by hitting refresh a few times in a browser. That does seem a little odd... I wander wha the threshold for "too many times" or "too fast" is... I mean if it's say only once every 5 mins then that would be good for your thoughts around hunting before and after comparison at least. Would make development tricky though as you'd want to be testing a lot while you got things working right.

Anyhow cheers for the info, good to know it wasn't just me.
 

Ferial

Guardian
Joined
Jul 4, 2012
Posts
326
Location
Sweden
Avatar Name
Chris Ferial Book
First off: very nice looking set of tools.

Second: Automatic fetching of the json is easily achieved by writing a browser plugin, since they have access to all the information the browser has. This is not difficult. It does however pose a security threat if made publically available for that same reason, it has access to everything the browser has access to, and as such could be used for malicious intents.

Furthermore, automating the process more than you have done in your before and after app provides little to no value. It is sufficient to get the information quickly enough when repairing and/or converting shrapnel (Since with an automated fetch those things would require manual intervention anyways, e.g. stop hunting for the period of time that it takes for your script to fetch new data, then convert/repair and once again wait for new data fetch before you can start hunting again).

So in short, automating the process is not worth it, because what you gain in reduced manual intervention is lost in time. In my opinion.

We do however need improved logging in Entropia. With proper logging of what costs are incurred (e.g. tool X inflicted Y damage) (You took Z points of damage to K body part) As well as loot logging we could have some very nice tracking tools available that requires minimal manual intervetion aside from setup.
 
Top