The Sublime Art of BPO Fu

I don't know how many readers out there have been ambitious enough to dig into the EVE Data Dump for more than just typeID/typeName conversions.  There is a ton of data in there, and traversing it can be a real trip for the uninitiated.

One segment that has always proven personally daunting has been trying to scrape out BPO data.  Though the queries look easy to start, they quickly become cumbersome trying to handle all the data you actually care about.  Some of the linking you probably want:

• T2 Products: what does the invention step cost?
• T2 Products: what is the source BPO?
• T2 Products: what decryptor(s) should I use?
• T2 Products: default runs yield?
• Can I build the sub-components (capital, T2, T3)?
• What is the product's group/category?

Truly, the "basic" functionality of a materials list is pretty easy to put together.  Simply combine the "base materials" and "extra materials" queries and apply the appropriate math.  Most of the pain comes around T2 blueprints.  So much of the accounting for T2 is interdependent.  Also, since T2 BPOs exist, the attributes are a little screwy when trying to account for the two production paths.

My buddy Valkrr had a pretty decent xml/JSON tool that took care of all the little nuances, but lacked a standalone updater.  Since he has quit EVE for the foreseeable future, I was SOL for recreating his utility as the SDE's update.

BPO_Builder.py command-line script
You will need to download the script + scraper.ini to parse FuzzySteve's MySQL releases.  As of now it requires whatever SDE you want to scrape to be mounted locally.  There are no special options, it just dumps the result files for use elsewhere.  In a future release I'd like to internalize it to the app in question so it can refresh the values at launch time.

The Goal: A Formatted File Of BPO Data

The idea here is you could crawl along BPOs (or their products) and get all the data you'd care to know in one return.  Invention info, dependent builds, linked blueprints, math data... the only thing missing is required skills for the builds (filtered out).

As of this post, the script returns an XML file similarly formatted as above.  I am working to push out a JSON version for lighter-weight release.  The idea here is this script can be run once-per-release to give you a standardized view of all the BPO data you could need for a program.  I'm personally pushing this to replace my gawd awful Perl kit builder, that stored much of this data manually.

The Pain of the SDE

If you are completely lacking in SQL-fu, the SDE can be really obnoxious to traverse.  Thankfully, FuzzySteve is incredibly easy to get a hold of and is an immense help in those circumstances when you're just stuck.  He was instrumental in helping with the T1/T2 BPO mapping.  

The absolute worst part of dealing with the Data Dump is the few little pitfalls out there.  The Data Dump is littered with bugs, and they've been there for a very long time.  Some bugs I ran into:

• Meta Level (dgmTypeAttributes.attributeID=633) is split between valueInt/valueFloat
• fixed with COALESCE(dgmTypeAttributes.valueInt,dgmTypeAttributes.valueFloat,0)
• Though metaLevel is never a float in-game, someone at CCP flubbed about 20% of the items into the wrong type
• Capital T2 rigs (Anti-EM/Anti-Explosive armor rigs) report the wrong metaGroupID in invmetatypes
• Reported bug
• Added manual repair to my tool 
• Mercoxit Mining Crystal I lacking a metaLevel (unlike other mining crystals)
• Reported bug
• Added manual repair

What's Next

This will enable two goals of mine.  First, to be able to further crunch scraper data to answer questions like "How much Tritanium was destroyed".  Second, to enable me to move away from spreadsheets to more sustainable apps.  I'd still like to do something to hook into google spreadsheets, only because of ease of sharing, but trying to build a similar tool for price searching is unsustainable.

Go ahead and give my script a whirl.  Tell me if there's anything you need added to it.  The dump should be quick and easy to use.  It's a bit larger than I expected, but slurping in XML/JSON should be pretty easy to handle.

Promoting Learning

November and December have been a real shift in my playstyle and priorities.  Though I am still chipping away at my old goals of Conquering the T2 Manufacturing Sphere, Aideron Robotics has given me fresh new goals with some much younger players.  Though I am not participating in very much PVP, the bits I get to do are fun, and my work enables the corp to reach above its weight class.

As I said before, my training contributions haven't been very helpful lately.  The summer pushed me in to the 1% echelons and what I personally find helpful these days is not applicable to the general players just looking to get their careers off the ground.

As a half-joke, I started streaming my EVE playing.  Though I don't use the in-client tool (doesn't do anything for multiboxing), I can now show off exactly HOW I do what I do.  Personally, I think this is kind of dull to share, but I've been getting a ton of positive feedback about the streams.

Guides requested in streams:

• How to POS?  Speccing a POS, what mods in what quantities?
• How to spreadsheet?  Your sheets are awesome, I want!
• How to pick a product?
• Where to sell a product?
• How can industry help a corp?

These are all great questions, but each is a pretty big topic.  Though I'm liking the response my infographics have been receiving, they aren't always the right tool for the job.  I tried to work on a T2 production flowchart to explain how you can optimize research/manufacturing in parallel, but I couldn't make it pretty.  

Instead, I have a few people whispering in my ears to do some youtube things.  And though I would love to do a series of classes, rather than the 1-hr rant fests that most game youtube channels tend to be, the time investment is pretty steep.  I will be talking to some of my buddies to see if I can get a video editor at the least so I can offload the part I just have no time for.

And because I've been neglecting my #graphpr0n duties

Since I released my Booster Use Infographic (updated version here) I wanted to see if I had any sort of impact on the game.  Though the post-release spike is pretty considerable, it's hard to isolate the reasons.  One of the Yule Lads handouts was a Synth booster, and I would need better data on fights/deaths to really dig into the impact.  Though it's pretty convincing to say here that it's not exactly seasonal.  It's refreshing to see demand growing over the last year, but only time will tell.

Experimenting with Streaming

Since EVE pushed twitch.tv integration into the game, I figured I would give it a try more as a joke than anything.  Thanks to +Poetic Stanziel finding my post and pimping my feed, I had nearly 20 people in the stream watching as I do my regular shopping trip at the start of my production cycle.

I found it pretty enjoyable.  I went ahead and explained my process to those who were watching, stepping through each of the tools, why they are set up the way they are, and how I manage my projects.  I do feel a little nervous/silly talking to the void with no direct feedback, but it was fun.  I know this is EVE and we shouldn't share our advantages, but I enjoy helping the others out there because it's in my best interest to have more educated peers.

I've joked in the past that I should make a youtube series about spreadsheets and industry.  I just had a hard time imagining how many people want to watch someone click through spreadsheets, but there seems to be a bit of demand.  Producing videos for youtube is a bit more work than I have time for right now, but streaming my work isn't.  I'd like to put together something semi-regular, like a class series, or at least a live walkthrough of what I do and why I do it.


I invite everyone to join me at 1P MST (2000 in game I believe) on Sunday Dec 15 as I walk through my prebuild steps.  I will see about getting the stream cached somewhere (hopefully on twitch) for those that can't make it.  I'm hoping to start the stream about 1hr before that so people can gather and ask questions in chat (I will also accept questions here on the blog and on twitter) and hopefully it will run 1-2hrs and we'll all be enriched.

Stream: twitch.tv/hlibindustry
Twitter: HLIBindustry
Time: 2000 UTC Sunday Dec 15

Hope to see you there.  Invite your friends.  Please don't wardec/gank me into oblivion :)

You Don't Understand Risk

If there's a topic that drives me crazy fastest, it's talking about risk.  EVE is loaded with all sorts of risks and some pretty serious penalties for those foolish enough to ignore them.  As such, everyone has their own little rules about avoiding risk... though I believe a lot of them are completely useless or overly paranoid.

It boils down to a general ignorance when it comes to probability and statistics.  Everyone is obsessed with the lottery odds.  Wins are worth any cost, even if the odds are low.  Losses must be absolutely avoided, even if the odds are equally low.  MMO players will farm the loot-treadmill for a 1% drop, but won't use something that could have a 15% penalty.

This has been the major block to trying to push Combat Boosters on my friends.  They are still hung up on the penalties and are blinded to the rewards.  I went ahead and expanded the infographic to include a breakdown of the actual odds.  This is a bit of a draft, I don't like the new graph colors yet, and I need to push a fully updated version with the edits from Reddit.

The point I wanted to illustrate here was how low the chances of penalty really are, especially for Standard boosters.  With no skills, the odds are 80% that you will incur 1 or less penalties.  Neurotoxin Recovery 5 pushes that figure to 89%.  The odds of actually rolling all 4 are astronomically low in comparison.  In my opinion, if you're flying a ship that should use boosters and aren't, you're losing out on a big opportunity!  This is the same point I make about invention accounting, if you understand the probabilities, you'd be foolish not to participate.   These mechanics are driven by a random number generator (RNG), most players shy away since they aren't in complete control.

Risk aversion drives me the most crazy when it comes to flying a freighter.  There are dozens of guides about how to safely use a freighter, and the advice boils down as such:

1. Don't Use Autopilot.  Fly gate-to-gate.  AFK is death
2. Don't haul too much.  1B is the standard rule of thumb
3. Avoid chokepoints.  Inter-hub shipping goes through some high risk chokepoints
4. Other good ideas:
• Have insta undocks
• Have insta docks
• Have a friendly webber
• Avoid hauling in high traffic situations
• Jita approach (3 gates land on 4-4 with the same vector, ganker's paradise)
• Friday-Saturday prime times
• Inter-hub routes
5. If you can't accept the risk, use a courier service

All of these are good guidelines, but don't let fellow freighter pilots know you have broken any of these commandments.  Personally, I contend the 1B cargo rule.  If you are already engaging in risky behavior (high traffic, gank routes) then the 1B rule is totally reasonable.  If you go through Niarja/Madmilre with more than 1B in unwrapped cargo during prime time, you're probably going to have a bad time.  

Personally, I have set up my routes so I can be a little more lax on these rules.  My Jita<-->Base route is pretty low traffic and doesn't go through a high-risk chokepoint.  I use instas to get in and out of stations, and if I am being particularly ballsy (stupid) I have a webber + skirmish mindlink.  

Yes, I am risking a gank.  Yes, I am probably inviting even more risk bringing this up.  But I think it's important to properly assess a situation and use the right tool for the job.  If I have a heavy haul to do on a weekend, I'll usually ping a courier service to just avoid the risk entirely.  If it's a quiet weekday, and I just need something moved, I'll bring the right equipment to protect that investment.  

Being crippled by fear of risk is no way to play the game.  If you follow every single rule and guideline to avoid risk, you'll either never undock, or never actually have any fun.  The guides are there to help people, but no one takes spaceships more seriously than EVE people.

Better Piloting Through Chemistry

With all the recent PVP I've been doing in Aideron Robotics, chiefly against Russians in Old Man Gang, we've been faced with a higher class of solo/small-gang PVPer than any of us is used to.  Chiefly, the kind that ALWAYS has fleet boosts, and employs pirate implant sets.  These have made for some very tough nuts to crack, and have made defending Heydieles a real challenge.

Our answer has been to respond, as much as we can, in kind.  I've pushed two of my booster characters into the system with a full suite of fleet boosters.  I've been keeping a steady stream of fitted ships on contract where pilots can quickly grab them down and get back in the fight.  With the recent tide of allies and a generous US holiday, we've been able to turn the system in our favor.  Aideron Robotics has recently taken away OMG's POCOs (#1, #2), and really stomped down Caldari challenge to the system.

This isn't enough, and we've been scrambling for more to answer OMG.  There's no way we're going to push pilots into pirate sets and expect them to beat OMG at their own game, but we can leverage Combat Boosters!

The big problem with boosters though is convincing people to try them.  With the really steep penalties for use, and pain of transport/sale, most pilots completely discount them.  Though, if they are properly utilized, Boosters can be a real force multiplier when used in the right roles/ships.

Unfortunately, there are no really good definitive guides on drug use.  Ripard Teg's Fit of the Week segment usually highlights individual boosters when they make sense in a fit, and there are some written guides explaining how they work in wiki wall-o-text fashion, but there isn't a great go-to guide for them.  To push boosters on our greener members, and keep them active in our FC's minds for utilization, we need something better.

Making Infographics

Recently, Aideron Robotics has been pushing a "Making a Better Pilot" series.  Similar to a lot of TEST/CFC propaganda, cute infographics to try and curb bad behaviors or illustrate less-intuitive piloting ideas.  

Since my moon mining flow chart was so well received, especially with the siphon additions, I figured this was a good chance to fill a need.  Also, I'm eyeing Booster production, but the market throughput is kind of anemic, so I figured we could kill a lot of birds with one stone here:  Increase demand in general, improve Aideron performance in PVP, line the industry wallet, and contribute something to the overall meta (which I have been neglecting for the last month or so).

How It's Made

For the graphically retarded, Google Drawing in Google Docs is an absolute life saver.  Pair the image dumps(link) with some text and a little graph magic and GIMP.  The biggest problem I ran into is I wanted a radial bar graph... and had no easy way to make one.  Seriously... why is this so difficult?

What I Wanted

What I Made

The hope was to have a bit of a gauge to illustrate the various grades + skills combinations.  The hope was to illustrate that, with the appropriate skills, the chances of incurring a truly unacceptable penalty was very low.  Unfortunately, explaining probability to the masses is always a frustrating endeavor. Though I think I illustrated the reality pretty decently by pairing my chart with a character sheet view.  

Released infographics after the break!