Tuesday, January 14, 2014

Everything You Never Wanted to Know: eveapi

This is going to be slightly misleading.  The purpose of this post is to showcase Entity's eveapi Python module, more than the official EVE API.  The ins-and-outs of large scale EVE API work require much more space than I have here, and I am not completely familiar with all the traps and holes.  Instead, this is meant as a first-pass guide to the API and how to leverage it in your custom code.  Also, this will only be about "v2" read-only options, since CREST is something else entirely and mostly a pipedream still.

EVE API Basics

The EVE API provides a read-only portal for apps to get game data.  This can be anything from skill plans, to industrial jobs, to wallet transactions, and more.  The API is accessed using a keyID/vCode combo, which is controlled through your account management page.  There, any feed can be enabled, or disabled, even the key can be set to expire or deleted entirely.  This gives account owners the means to have many APIs for any particular app or service.

A query looks like:


EVE's API returns follow the XML DOM structure.  This is a parsable tree that many code languages have easy methods to handle.  Javascript and Perl are my two personal favorites... Python's XML handler is kinda terrible.

Feeds

There are 4 kinds of feeds.  Account, Character, Corporation, and generic.  Each group has a use scheme, authentication requirements, and similar behavior.

Account Feeds

Account feeds are used to get general information about the key and do not require any special access.  They take a given keyID/vCode combo and return information about that key.  APIKeyInfo will tell you what type of key (Corporation or Character) and what feeds it can access.  Characters will tell you what characters are accessible by the account/key and general info like corporation name and characterID.  These feeds are meant to be used as a validation step.  If you are writing your own app, it's a good idea to validate against these resources, so you can handle errors such as API expiration or invalid key more gracefully.

Character Feeds

I won't break down every feed one-by-one, but Character Feeds are meant to give individual character data.  This can be troublesome if you are given a all-characters key, since the character list will need to be pulled from the Account/Characters feed.  

All of the Character feeds require a 3rd piece of the key, CharacterID.  So, if you wanted all the character sheets of one account, you would have to ask up to 3 times: one for each CharacterID.  

Corporation Feeds

Like Character Feeds, Corporation Feeds require a specific API to access.  These API keys can only be generated by CEO/directors.  The access/information is completely separated from Character Feeds: one cannot access the other at all.

Corporation feeds say they need CharacterID in the documentation, but I believe most don't need it.  Also, the Starbase Detail requires itemID of the tower in question (found in the Starbase List).  As always, check documentation for feed specifics.

Generic Feeds

All other API feeds fall under a generic category and do not even require a keyID/vCode combo.  If you're looking for map info, or a characterID name conversion, or some basic stats, they can be accessed directly without a special key.  Also, cache and limits to these APIs tend to be much more generous.

Accessing Feeds

Authentication is a two factor process.  First, a key must be validated: make sure it's valid, the expected type, and not expired.  Second, to access a feed, it must have the valid "accessMask".  If one is not valid, the API will reject your request.

Every feed has a binary access mask.  To check if you have the right key for the feed in question:

if(API_accessMask & _feed_accessMask_) == _feed_access_mask_: 
    pass 
else:    #API will return HTTP:503 error
Using this snippet means you can access any feed the key gives access to, rather than hardcoding for a specific one-size-only key.  Seriously... I want to crush hands when a tool won't take a 'everything enabled' key

Best Practices

  • Follow cachedUntil guidelines
    • Should re-return same data until cachedUntil expires
    • Can lead to bans if ignored
  • Don't forget User-Agent in your request header
    • Gives admins a way to contact developers who are causing problems
  • Use gzip encoding in your request
  • Check the return header for more info

Using eveapi

Go crawl through some of the API documentation and look at the different returns each feed has.  Though there are some constant themes, each feed has completely different return structures.  If you were to write a new API tool from scratch, you could be faced with a feed-by-feed custom cruncher.  Instead of writing 70-80 custom functions, eveapi lets you get straight to the meat of each feed with a simple set of calls.

How does eveapi work?  It utilizes Python's magic methods to build classes and objects dynamically as they are called.  Though there is no specific auth.char.CharacterSheet() entry in the codebase, it builds the query on-the-fly from the names of each call.  There is a level of elegance and future-proofing that makes eveapi absolutely incredible.

Installing

It's meant as an importable module.  It's pretty easy to copy down a version from github.  Though, to keep a version in your repository current with its current repository, use git submodule.
From there, it's a simple import eveapi from eveapi, and you're ready to rock!  Since eveapi is just a Python module, it makes it very easy to include with a project for deployment where you may not have control of the Python version, like AppEngine.

Working with eveapi

eveapi comes with a tutorial program to show how some of its features work.  Here's a little more ELI5 approach:

1: Create an auth object for the key in question

Here we have 2 objects:
  • auth is a key-specific access token.  Used for any direct Character or Corporation feed access
  • api is the global api token.  Used for generic queries 

2: Validate the key credentials

Though an exception will be thrown when you try to access something you shouldn't, it's best to avoid the exception entirely.  With this code, we pull down the keyInfo, fetch the character list (which can also come from Characters), and validate that we can ask for the Character Sheet.  We could further validate that the account is even active with AccountStatus, but that requires more access than may be given.

3: Pull down relevant information



The art of eveapi is that it accepts the names straight from the <rowset> or any other tag and lets you crawl through with english instead of:
 ...getElementsByTagName(...)[0].firstChild.nodeValue 
--OR--
 for row in rowsets[0].getElementsByTagName("row"): 
   row.getAttribute(...)
Each feed can be accessed by name.  Just combine the /type/APIname without the .xml.aspx

The contents can be accessed by name.  If it's a list, check the <rowset name="[this one]">, if it's a text item, just reference the name.  It's deceptively simple!


Instead of getting hung up in the actual structure of each individual API, each element can be accessed directly, regardless if it's text/attribute/tag/etc.  This leaves more time for contributors to work on building real tools rather than becoming mired in feed-by-feed minutia.

This is by no means an exhaustive guide for EVE API or the eveapi module.  But this should cover the basics for those who may have been overwhelmed by the API feeds previously.

Monday, January 6, 2014

The Pains of Kit Building

As I mentioned in my last post, all the complex corner-cases for production happen in T2 manufacturing.  Where putting together the bill-of-materials was difficult due to T1/T2 linking, the "build kit" step is painful due to intermediate building.

In a sloppy first-pass, it's pretty easy to just say "I'll build everything".  The code is not too difficult: once the parts are linked, it's just a matter of taking second (or third) passes on those products to work down to raw materials.  Unfortunately, it would be a massive waste of all this programmatic power to not allow for a little flexibility.

Problem: I want to be able to build some of the intermediates, but not others

The basic inspiration starts with EVEHQ's Prism tool.  

In this view, it's pretty easy to select each product that can be built and adjust the final bill-of-materials for that product.  Though EVEHQ does support a batch tool, I find it too rigid for my work.  Also, there is a lot of data missing from this view:
  • Build vs Buy cost for intermediates?
  • Build vs Buy time for intermediates?  Added intermediate time will detract from final ISK/hr calculation
  • No flexibility for partial builds
  • Product-by-product adjustment
Though the amateur or intermediate T2 manufacturer may just write off those functions as "too much trouble", I personally worry about these points and use them in my current production practices.  Doing partial builds on components saves me character-days of otherwise wasted production time.  

Solution: Replicate Current Resupply Methodology

Today, when I am picking up production kits, the todo list looks something like:
  1. Figure out what products to build for sale
  2. Pick-and-choose what intermediate products to DIY build
  3. Purchase everything: raw materials, T1, components
Regardless of how I organize my tools, it will require 2 passes.  First pass to list raw materials + intermediates, second pass to figure out additional raw materials.  Though I can write some decision conditions to automate step-2, as a first draft I should stick to just processing it with expected human input.

As a draft set of outputs:

STEP1: Design a "Kit"


STEP2: Adjust intermediate products


STEP3: Get Shopping/kit list(s)

These views give me an intermediate point to deliver programming logic without the overhead of UI design.  Also, it gives me a springboard to play with UI without having to completely redesign the code.  Though I need to tweak the intermediate build step to be more intuitive, I think this is a pretty decent spot to start the next major leg of code development.

What's In The Works

I have spent much of the last week getting the pieces in order and starting work on actual EVE API code.  The hope here is that I can load each contributor into the program and do the BPO math off their actual skills.  This will help define bill-of-materials on anyone, and better define costs on the other end.  

I recently teamed up with a friend from #tweetfleet to start focusing more completely on tool development.  We're splitting the work up to finally solve some of the bottlenecks that have been holding me back from hiring help.  The hope is, by spring, to have most of the alpha tools banged out so we can bring on players to help with the industry half while we work on more ambitious back-end work.  It's gearing up to be a pretty epic year!