Sunday, July 20, 2014

Building Better Spreadsheets - Crius Toolset

Crius releases on Tuesday, and most industrialists are scrambling to replace their spreadsheets (I know I am).  What's worse, the job-cost equation requires live data from the game, which can be difficult for someone relying on spreadsheet feeds.  Fear not!  I have great news!

Crius Feeds In Google Spreadsheets

These functions let you read in the CREST/API feeds of common calls and import them directly into your spreadsheet.  Furthermore, the triggers are set up to automatically refresh the data periodically so your spreadsheet will always be up-to-date.  Though this is not an exhaustive API tool, and still could use some more features, it should be a huge leg up for any spreadsheet jockey.

Most of the feeds are designed to dump the entire feed, and don't offer much filtering in the call.  Instead, they were designed to be used in a reference sheet that could then be leveraged using VLOOKUP() or QUERY().  This might lead to some issues with complexity down the line, so I intend to eventually add some finer calls that will just return single-line kind of data.

Getting Started

Method 1: Clone the Master Spreadsheet

This will give you the tools and triggers, but will not stay up-to-date with the master sheet.  Until I can wrap up the code as a stand-alone Drive app, this will be the most stupid proof way to get a copy:
  1. Open the spreadsheet
  2. Go to File -> Make a Copy
  3. Set the name of your copy (do not share with collaborators)
  4. Remove any extra sheets/calls you need to and start developing your spreadsheet
This method is the easiest to start with, but has the issue that it will not keep current with updates.  

Method 2: Copy-Paste from Github

The codebase is free and open source, and is designed to be copy-pasted into the gdoc script interface.  This method is a little more tedious, but will be easy to copy updates as they come out.
  1. Get plain-text code from the GitHub repo
  2. In your spreadsheet, go to Tools -> Script Editor...
  3. This opens a new window.  Select "Blank Project" from the initialization prompt
  4. Copy the raw code into code.js space
  5. Set the name of the project
  6. Save the changes
  7. Configure the app triggers.  Set get/All functions to 1hr timers

This will give you all the utilities in a fresh, or existing codebase.  Also, configuring the triggers appropriately will keep the data up-to-date automatically.  It's technically optional, but without time triggers, it will require a fresh open to pull fresh data.

Also, as updates come out, you'll be able to drop in the new code.  I expect to keep this project backwards compatible, so each drop in should ADD features.  Though, of course, if you go editing the code, you will need to be more careful about dropping in changes.  

Function List

  • getPOS (keyID, vCode, header_bool, verbose_bool, test_server_boo l)
  • getFacilities (keyID, vCode, header_bool, verbose_bool, test_server_bool )
  • getIndustryJobs (keyID, vCode, header_bool, verbose_bool, test_server_bool )
  • getAvgVolume (days, item_id, region_id )
  • getVolumes (days, item_id, region_id )
  • AllItemPrices (header_bool, test_server_bool )
  • AllSystemIndexes (header_bool, test_server_bool )
  • AllTeams (header_bool, verbose_bool, test_server_bool )
  • AllAuctions (header_bool, verbose_bool, test_server_bool )
The functions are designed to be referenced as simply as possible.  CREST feeds like AllItemPrices and AllSystemIndexes can be referenced without arguments if desired.  Also, the classic API feeds are designed to return as much information as they can, with internal switches to try and use the /corp/Locations feeds if possible.  Also, most feeds come with a "verbose_bool" trigger to add/remove ugly or useless raw ID kind of data.  Lastly, the test_server_bool has been left in the release.  For TQ this value can either be blank or false.

Function guide below the cut

Wednesday, July 16, 2014

Can I Play With MA(C)Dness?

Blame this one entirely on @K162space.  He got me playing with Quantmod in R.  Also, shout out to CCP Quant for showing off the original bits, and CCP Foxfour for the CREST market history.

After all my EVE data experiments, I've had a very hard time finding predictive correlations between datasets.  Even trying to bring in destruction data showed no predictive properties, instead only showing correlation to corroborate market volumes.  Also, I've tried my best to get into Varakoh's analysis, but have never been able to internalize his methods enough to consider rolling it into my own analysis (also getting reliable candlesticks for EVE data is a pain)

Then Blake over at sent me this tidbit:

Quantmod opened a whole new realm of tools I never knew about (full listing here).  The most interesting to me has been Moving Average Convergence Divergence (MACD).  I wouldn't be so bold as to say "use this to make money", but the results are extremely interesting when using IRL stock data.

For those that have never seen an MACD plot, the theory is relatively simple: Different line crosses are signals for buy/sell operations (specifics here and here).  Though the signals are imperfect, and will by no means catch every peak and valley, they can be an excellent sanity check for the longer trend.  For many IRL stocks these trends can have very strong correlation and are a popular tool among amateurs.  It is less useful for the minute-to-minute daytrading or arbitrage, but can be a powerful data source for the long-term investor.

Can this be useful for an EVE trader?  Well....

Let's look at a few plots:
PLEX - YTD - click to embiggen
Tritanium - YTD - click to embiggen
Phenolic Composites - YTD
I had to do a little fudging to get open/close bars to work correctly (though high=close=avg is roughly the same).  The trend information is interesting, but the crossover signals aren't lining up well.  Using the traditional 12,26,9 configuration, many of the crossover signals arrive 2-3 days late.  If you ended up using these charts as-is, I think you'd end up at best breaking even.  Though there are some secondary things you could do like buy and sell in the swings up and down, these charts aren't going to be immediately useful.

I then started playing with shorter windows, and pairing a fast and a slow chart might be a better way forward.  Unfortunately, I'm blindly fiddling knobs right now, but I'm actively hunting down documentation to better fine tune the charts.  I was thinking a 10-15d window might be more accurate, and a 25-30d window would serve well as a "slow" chart.

Also, I think the weakness has a bit to do with the quality of data here.  Where MACD is looking for close prices, we're feeding it daily averages.  This might be a good excuse to finally work on a better snapshot tool using eve-central data.  Though I had a lot of trouble processing down the raw archive, starting up a stupid-script that pulls periodically from the traditional API would be a quick solution that could start crunching in the background.  Once I can get my industry tools refactored, I expect to get back into the data mining game pretty seriously.

In the meantime, be sure to subscribe to my channel on Twitch, and follow on Twitter.  I will be doing some random streams over the next week as I get back into a reasonable routine again.

Sunday, June 22, 2014

Crius stuff - Getting Ahead in Code Development

Crius isn't that far off, and I'm starting to get a little anxious about getting started on new industry tool code.  Though CCP_Foxfour has been extremely outspoken on Twitter and Reddit about API rollouts, it's been harder to get the other base math information needed to make calculations.

Beta Data Dump

Thanks to FuzzySteve and CCP_Nullarbor for dropping this pro-tip in my stream.  You can take a look at the current BPO data inside the Singularity client files:
<sisi client path>\bin\staticdata\blueprint.db
 There you can find a SQLite db of the individual BPO data roughly the same format as will be delivered in the SDE when Crius launches.  I had a little trouble exporting the whole thing because I'm a noob and SQLite TEXT/BLOB rules are weird.  I ended up whipping together a little Python script to dump as csv.  The final version will be published as YAML, but should still be the JSON-string type for extraction.

Just to reiterate, we probably won't have a real set of tables for attributes like before.  Instead, each blueprint will be an individual object with all the attributes inside it.  There's nothing stopping you from expanding/transforming this data back into tables, but don't expect CCP or Fuzzwork to do it for you (EDIT: maybe after release, but will be custom project).

New APIs

The API is dead, long live the API

Until SSO rolls out, we're still going to have the classic-API for private data, and CREST for public/global data.  I will need more time to whip up real guides, and I will probably just update dev wikis rather than post wall-o-text here on the blog.

Classic-API Feeds

  • [char|corp]/IndustryJobs: updated format, will break classic tools
  • [char|corp]/IndustryJobsHistory: updated format, will break classic tools
  • corp/Facilities: will list corp-controlled facilities like POS/Outpost


  • /industry/
    • more info soon(tm)
Personally, I continue to use Entity's eveapi Python module for classic API queries.  Mostly because Python + XML kinda sucks.  Thankfully CREST is JSON and much easier to handle.  I still haven't seen an all-in-one CREST module, so you're gonna be stuck writing handlers for each new API feed.  This shouldn't be too much trouble, since the only things that need to change between requests is address and perhaps HTTP request headers.

Monday, June 16, 2014

Crius Industry - Features You May Have Missed

On Saturday, I had my stream, and crashed +Ali Aras's stream afterwards along with Mynnna, Salpun, and a few others I didn't immediately recognize.  Also, +Chella Ranier dropped into my Friday stream (and I will have her segment isolated into its own video soon).  It's been exceptionally helpful to get all the various perspectives on the industry changes, and we're still only scratching the surface.

If you're relying purely on the dev blogs for info about the changes, you're going to have a bad time.  Mynnna showed me how grossly out of sync I was by missing important information about CCP Grayscale's BP migration on the forum.  Also, I almost missed some of the POS stuff in the most recent devblog.  Lastly, it's worth messing with the UI on all the devices you expect to play EVE on.  It's a pretty big window, and I anticipate issues with shorter form factors such as laptops.

Note, none of the following are set in stone yet.  Provide feedback if you don't like the changes!  It's also worth noting the entire CSM is incredibly active on this Crius front.  Feel free to reach out to your personal favorite(s) and include them in your feedback!

Invention Math Changes

There are three big changes for Invention that might have gone unmentioned elsewhere.  
  • Invention will only consume 1 run off a copy per attempt
  • Invention times are far more dynamic to match up with manufacturing times
    • Copy + invent = 0.5x T2 build time (roughly research = build for T2)
  • Invention results will all be +ME/+TE.  No more negative results
    • Base materials will be increased to keep prices roughly flat
Both of these changes are pretty big for T2 producers.  Reduced copy consumption clears up two problems for most inventors: BPC clutter, and reduce errors with T2 outcomes.  Though copy-time vs build-time is being reduced, this results in a big increase for module copies.  This ends up being balanced out by the one-run/invent feature.  You will still need multiple copies for parallelism, but that supply should last for quite a while.

Personally, I find the invention time balance far more interesting.  The goal is noble, to make invention time a far more meaningful choice, but I think this change misses the mark in its current state on Singularity.  For periodic gaming, I imagine a few different time groups:
  • <=1 hr: multiple per session, up to 6x/day
  • < 3 hrs: once per session, up to 3x/day
  • < 10 hrs: twice per day
  • < 22hrs: once per day
  • n-days - 2hrs: once per n days
Lastly, the ME/TE changes should make the T2 BPO tinfoilers jump for joy.  Until the material quantities balance out (using a modified refining equation, causing weird material costs) and the SDE comes out to better run the bulk numbers, this will be a very hard thing to judge.  BPOs will still have a slight advantage due to the requested runs cost scaling, but I think you will be able to get better ME/TE than most BPOs in the end.  I'm particularly interested in how decryptors will drive the margins, since we probably won't see the big savings factors at 6-8% savings like we saw between -ME steps.

What I Don't Like

Personally, I only care if something is >2hrs, because I won't be able to do more than 1/day at that rate (without mobile app).  Also, I care that jobs hit at 22hrs rather than 24, because it can be hard to keep up with a strictly 24 or 25 hour cycle time.  Though the whole spectrum is interesting if you're going to optimize to share BPs over timezones, as long as jobs are a per-character mechanic, the times can be blocked into the above categories IMO.  

Also, I'd rather that the invention time balance be 0.4x or 0.35x to account for specializing in the science skills.  I'd like to put a tangible benefit for taking some of the obscure science skills to 5.  Lastly, another "between the lines" message here will be wider decryptor-type use, because of the reduced research-time/run.

Currently POS math is broken, so I am not sure how this will wash out in the end.  Also, I still need to play with some of the corners to check the balance.  Lastly, the decryptor math is going to be something worth checking with a fine toothed comb.  On TQ today, decryptors pay for themselves off the ME savings, and the difference between positive steps is much smaller than the difference between negative steps.  I have a hard time imagining that better research/build ratios or lowered datacore costs will drive consumption.

POS Changes

There are a bunch of little things that changed between pre-Kronos and now.  
  • Additional labs/arrays will reduce job install costs
  • "Invention" and "Research" labs have been split:
    • Design Lab: able to copy/invent
    • Research Lab: able to ME/TE research
    • Hyasyoda Lab = meta Research Lab
  • Lowsec love
    • Intensive refining array = mid-grade null outpost refinery
    • Lowsec-only Thukker component array
      • ME better than outpost
      • Built from ghost site materials?
  • EDIT: moon mining/reactions will be available in 0.4 sec systems
As I covered previously, the people crying the biggest tears were solo capital producers.  With the new POS array changes, the playing field becomes much more even between LS/NS.  There's still the risk v reward metric, and a supply chain bottleneck for compressed ore, but the ability to participate in the capital game is still there (even if it's not entirely solo any more).  

Also, I really like the flat differentiation between "invention" and "research" labs.  This makes POS design far more direct for the amateur.  It's still hard to judge the job-cost benefits (in general) because of math and UI elements not being better solidified yet.  I'd also like to see the fitting requirements for both arrays be equivalent (or swapped) and a meta Design lab to match the Hyasyoda offering.  With the BP@POS rules, I'd like the choice to utilize blingy equipment for both Design and Research labs.  

Other Notes

The features are designed for an incredibly wide swath of EVE to participate in industry.  Though I am not wholly convinced it's great for high-powered and cooperative industry, it's a big step up from TQ historically.  Also, a lot of the bigger scale horsepower will rely on +Regner Blok-Andersen getting CREST/API/SDE stuff out for developers to start moving some of the every-day calculations outside the client.

Lastly, I have this grandiose idea that job fees will be the bar that separates the men from the boys.  Though right now, costs still account for <5% of business.  I can see where taxes + fees start to eat your lunch, but it's still a death-by-inches mechanic, and that makes it hard to justify a lot of effort to minimize it.  Also, with the job cost UI lacking a lot of the promised breakdown, it's hard to feel like it represents a meaningful choice yet.  If the costs are going to be this complex, I need to have a fighting chance to make good decisions.

Thursday, June 12, 2014

Crius Industry Stream - June 11th - Initial Impressions

I cheated and dropped in on Ali Arias's day-of stream to get a first look before my first scheduled stream.  Thanks to Salpun and Ali, got to work through the biggest bugs.

The goal of this first stream was just to stretch out and try the new features.  No particular stress test, no grandiose goals... just see how it works, and jot down some first impressions.  Also, it's been quite a while since I've done any significant beta testing, so it was an exercise in getting notes set up out-of-game.

Bug Report

Being so fresh, everyone should expect the experience to be a little choppy.  I'm going to keep my report to the things that might stop you from testing something yourself, or is otherwise a glaring known issue.  To keep up on rollout, I suggest EVE Manufacturing's blog

Numbers Are Wrong

:Math: is not to be trusted right now.  Currently everything is in a "does it work" status.  I ran into a bunch of hilariously wrong numbers, but don't take those values as truth.  I expect that as the mechanics work themselves out, the devs will focus on the actual balance of stuff.  Also, I noticed things like POS bonuses were not applying, but that might be fixed sooner than the rest.

Also, it's worth noting the job cost equations are not properly seeded either.  I have an expectation that the "earn more geckos" event in May was to seed the internal databases with activity levels.  We should see "mostly right" numbers coming to Singularity.  I wouldn't cry fowl about the price of things being screwy due to Singularity prices being 100isk for everything.

Installing Personal Jobs Is Broken

There is a bug that personal industry jobs can't be installed due to wallet weirdness.  I expect this bug to be fixed very soon.

Teams Are Not Usable Yet

The Teams UI is in the industry tool, but bidding/acquiring/using teams is not enabled yet.  Also, utilizing Teams with the math is broken does not give us enough useful data right now.

First Impressions

The new UI is very pretty.  I think it's by-and-large a step up.  There are a lot of little weirdness that I can't tell if it's bug or feature, but I expect the mechanical portions of installing jobs to solidify rather quickly.  I am hoping they smooth out a lot of the asset update hiccups, because last night's build did not like moving things around.

Compared to the old UI, I'm extremely impressed.  Most of the 1-step kind of work is a much improved.  There is a lot of information available for the amateur or tinkerer.  I still worry that high activity industry is going to be obnoxious.  I want to push through the initial novelty and put the new UI through its paces so I can better judge how industry will feel in the long run.

Also, I'd like to put kudos out there for the facility selector.  I liked how it slurped in POS equipment, and shows everything everywhere with a pretty simple HUD.  Again, I am worried how it will play with heavy-activity work, but at first glance it's a big step up from the old UI.

As for things I didn't like; the job UI is huge.  It doesn't fit well in a minimum-height window layout, and will make industry on a laptop kinda obnoxious.  Also, I would like to see a few more readouts for price information.  They are currently hidden in mouseover tooltips, but I would prefer if the estimated value was displayed along with the charges.

Feature Requests

Didn't get a chance to push all of this to the forums like I intended, but I'll go ahead and post my notes.
  • “In use” BP’s should filter to the bottom?
  • Need better means to force refresh on the industry pannel
  • Design Lab: Copy/Invent
  • Research lab: ME/TE research
  • Hyasyoda: ME/TE research
  • Moving BPOs around is painful:
    • Cannot start job with BPO in container
    • Cannot update inventory when BPO moved to another hangar location
    • cannot “drag and drop” BP’s around inventory
  • Facility infopane:
    • Don’t like the text being so small
    • Also don’t like the spacing between icon/text
    • Would like click to highlight in facility list
  • Skill requirements:
    • Shows green book when zero skills are required
    • Does not show affect of skills on outcome (does on mouseover)
  • T2 outcome selection
    • Not a fan of drop down. Perhaps side-by-side icons?
  • Would prefer if ME/TE adjustments happened in circle like the outcome probabilty
  • Decryptor icons are ambiguous. Should show text in BOM window
  • Skills should differentiate between ‘required” and “current”
  • Jobs report: should show materials being used (help differentiate jobs)
  • Jobs report: should still have text of output, not just BP icon
  • Install delay is pretty long (>10s). Should be <1s
  • Invention needs an “outcome” column
  • No way to sort original/copies in blueprint view
    • Would like sort-by-valid-activity
  • Would prefer view to show both mfg/research jobs utilized/remaining

Watch The Stream