Also, if you haven't read @FunkyBacon's blog on the fallout, I seriously endorse it: part1 part2
The Way it Was
Yesterday's post specifically outlined the classic way that specific client data was accessed without the API. It's still the way most locally-saved features still work, specifically fittings and overview. Whenever you press import/export inside the game client, files are accessed from \My Documents\EVE\. This is why (in my opinion) overview modifiers are still fair game, and the means EFT-warriors can get their fits from EFT into EVE and back again.
This sanctioned method is still perfectly valid, and you too can play with the data. This was how EVE-central got its data until late 2011. Unfortunately, this stream requires users actively acquiring data, and is prone to bad-data stuffing.
The Way it Is
Say hello to contribtastic. This works by scraping the results from every market query you make in-game, then pushing it off to be processed by eve-central. I'm not particularly savvy on the exact specifics, or how things are parsed... but it works off the live cache files as they are generated. This provides a much higher quality, more direct, and higher throughput stream of price data for all the hungry marketeers across EVE. Also, this stream enables more powerful services like EMDR.
This process has not been specifically outlawed by CCP, but absolutely exists in an uncomfortable gray area. And, without warning, CCP has the ability to pull the functionality out, leaving us with the old-way. It won't outright kill 3rd party development, but it would be a serious hamstring.
The Tenuous Relationship
There is an unsteady armistice when it comes to the market feeds for EVE. The rumor I've heard through other player-devs from fanfests say that CCP cannot provide the direct API connection to the market, fearing in-game stability. On the other hand, 3rd party developers need a high-quality stream of market data to keep the logistical gears moving. Also, with the generous rate API server hands out IP bans, I really don't want CCP hosting market data via API. This leaves us with the current operating relationship: utilizing the gray-area for market feeds, while CCP turns a blind-eye.
As this Kelduum scandal runs its course, I am sure there will be calls/threats on the EVE-O forums, once again, to encrypt local cache files. But I hope that CCP Seagull has the foresight to see how dangerous this proposition is to the so-called "Enablers".
Updates as they become available:
- Perhaps cache scraping isn't as gray as expected. According to Reverence, they have blessing from CCP to publish cache scrapers
- Fuzzworks proposal for CCP-provided marked API
- Serpentine Logic's December post on limiting bots: Remove the advantage of botting
- @Noizygamer's redacted post.
- Original source of misinformation: Jitonomic
- Jitonomic condones client scripting. Unquestioningly against EULA
- More info on data collection methods:
- TheMittani article draft: The Machines Are Taking Over
Notes from Entity (developer of eveapi for python and Reverence)
Hi,
Well, there's this post here that is very explicitly granting permission to read cache:
http://community.eveonline.com/ingameboard.asp?a=topic&threadID=734561&page=1#9And there's also the fact I sent my cache decoder to CCP's CTO for review and approval before releasing it, so I can say with absolute certainty that it is currently perfectly fine to decode the cache.
Of course, as with any tool, it can be used for evil as well as good, so in the end it still matters what you do with it.
And CCP is also within their right to revoke the permission if they so choose, but I have not seen any such notice, nor do I really see a reason for them to.
Besides, forbidding it is pointless because such a policy is unenforceable, but that's another story.
Hope that answers your question.
13 comments:
Ideal world, from the way I see it, is to provide a service much like EMDR themselves.
They provide an endpoint, and it's up to people like us to listen to it, store the information and so on, if we want to do anything to it. IF they can make it zeromq, and the same format as emdr, then it'll just work.
It fixes one major problem with having the data directly accessible from CCP, on demand, and that's 'perfect information'
We'd have to have some way of pushing it out in a fashion that mirrors regular user activity, so commonly accessed information goes out regularly, and less common is less frequent.
A few problems:
1) currently EVE API is not xpath friendly. So any marketstat API would need to be done completely differently than the current API (DOM) to remain gdoc friendly
2) We'd still want historical data (which eve-central hosts)
3) CCP hasn't ever been very willing to offer "perfect data" for market info.
4) the EVE API server gets pretty generous with IP bans.
I agree that "perfect world" we'd have a standardized official pipeline of data from CCP... but CCP's API development arm has always been a very low priority. I kind of think it's too important to leave to CCP
Hence why I'm suggesting it the way it is. Most people will still get their market data from an aggregation site, rather than directly from CCP.
The EMDR feed (which I'm suggesting is replicated for this) is a firehose of market data, including both current data and market data (when people look at it). It's not 'perfect' data, except at the exact moment it sends you information on a particular item. I'm not suggesting an API you make requests from. Just a fairly constant low level stream of data that you can subscribe to (by running a consumer on that information)
1: gdocs stuff will pull from a site like eve-central, rather than directly.
2: historical data will be available, but only if you're willing to collect it (or use a service like eve central)
3: It's not perfect data.
4: No bans needed, as you're not making requests, just listening.
I might put in some kind of ip based authentication to listen to the stream, so people can 'license' it (sign up with an eve account, tell it what ip can listen for you.)
To return to 'part one' of this topic. When you analyzed John's purported market tool, this was the only part that came up illegal - and of course it's grey.
So basically none of the individual pieces of John's tool (as far as we can tell) were actually illegal, CCP just decided that the combination was illegal.
So basically a small amount of automation is fine, but more is not. The dividing line is set by (ultimately) Screegs who has a complete tin ear on a number of levels.
Please go back and read the update to The Nosy Gamer's post - http://nosygamer.blogspot.com/2013/02/who-watches-teachers.html - and an edit to your reference here is in order, I think.
Working on it. I only just got Nosy Gamer's update and will be incorporating the findings ASAP
Thanks for the edit - by the way, I only recently discovered your blog and find it fascinating. Please keep up the good work!
It's a fine line, but the wording of the TOS/EULA becomes important here. Please look up the exact words yourself, but to paraphrase "It's against the rules to use tools to acquire items _faster than a human could do_." It's not the use of macros and such per se, but what you do with them.
If you have a macro which just presses F1-F8 and the DCU for you, it doesn't matter because on the server side it would all come in within the same 'tick'; a human doing this work manually would be able compete.
A macro which cycles between 8 clients to 0.01-ISK a market item at a single key press, thus being able to submit 8 market orders at once and essentially neutralizing the 2-second update time for each individual order - that is something a human would not be able to compete with.
Of course this is just my interpretation - but I like to think it's compelling one.
Another thought: One reason why CCP might be accepting of cache scraping as such is that a player has to be on the location and looking at the market items in order to inject the data into the cache in the first place.
It's when you combine a macro looking up every important market item in rapid succession (a market quickbar with every important item and a cursor key-down macro would do), a cache scraper, an evaluation "for best market orders" program, and an input-Macro, that's when you get into botting territory.
Thanks for the input. I agree there is an important line to draw between read and write here. It's difficult to read like a robot in this case (client limitations), but an anonymous aggregator like EMDR gives you a IRL-power-trader-like feed.
I absolutely agree the line is being crossed with bulk input macros. But the environment is a whole lot more gray than I anticipated.
Update: you can actually read the market like a robot using sanctioned IGB hooks.
goon metrics market querier: http://goonmetrics.com/market_scan/
IGB hooks: http://wiki.eveonline.com/en/wiki/IGB_Javascript_Methods#showMarketDetails_Method
The CCP philosophy seems to be near-limitless on READ operations. Whereas WRITE is heavily regulated and discouraged.
I don't know much about what the market trading community does with its scripting, but I am familiar with what CCP Sreegs looks for when looking to ban botters. If your set-up involves anything simulating keystrokes like a cursor key-down macro, that would be the same concept as what mining bots do and would get you banned.
I also know that Team Security hates python injection bots, which are what Questor and H-Bot are, and if those bots are identified the users are permanently banned. I had that confirmed by Sreegs when I was writing about another subject.
Also, look at one other thing. I think all of the approvals come from before CCP Sreegs being hired. He was hired in October 2010 and I don't know when he actually started work.
All of the tools covered in my post (namely, those that hook into market aggrecators like EVE-Central), seem to exist in a half-blessed, unenforcable, "can't really stop them" kind of territory.
The real point I think that needs to be driven home here is: READ is by and large okay, but we will break your hands if you WRITE. This makes me want to be a fly on the wall on CREST meetings.
Post a Comment