Wednesday, March 27, 2013

Failing Gracefully

Recently ran into two problems, one in-game and the other in Prosper.  I'm going to try and merge both thoughts into a single blog post.  Might be reading too much of Chella's Low-Sec Lifestyle!

In-Game: The Bad Investment

I've stressed mea culpa on my bad investments before, like the Nomad.  Recently, the ball strapped to my ankle has been a perfect-storm with Redeemers.  First, I repeated the same mistake I made with the Nomad, hopping in as the bubble was crashing.  Second, I managed to buy-in when materials were spiking.  So I'm stuck with a double whammy absolutely murdering my margin.

I cleared my buy-in schedule to try and ride out the turmoil, but it looks like I will be selling out near-cost.  With T1 minerals rapidly deflating, global production volume growing, and sales volume really not keeping pace, black-ops have settled to similar margins as everything else.  If you don't derp the buy-in, there's still money to make... but it's a lot of effort and capital to tie up.  Call me greedy, but if I'm going to plunge in 2-3B on a buy in, I want to see really strong returns for the trouble.  Otherwise there are plenty of smaller bets that are paying off well enough.  Also big buy-ins just make me edgy when it's time to ship kits to the factory... still waiting for Miniluv to gank one of my freighters.

The funny part about all this was I flew in a Gal FW fleet on my main a couple weeks ago, and one of my old friends from Aideron Robotics was jabbing me about my recent industry fails.  Except, that in none of these fails have I lost money.  It's similar to the hard time I give my cube-mate in real life.  He sold of his company stock 6wks earlier than he should have... whenever I give him a friendly jab about it, he reminds me he still took home money... what's there to complain about?

Without risk, the game isn't fun.  I actually get excited for days as I ramp up to a big risky project.  Maintaining a status quo is boring.  In fact, I might be taking on a shiny new big project very soon.  I hope to blog about it as soon as details are hammered out.

Out-of-Game: Shaky API

March has flown by at mach speeds just like Feb before it.  I've found very little time to commit to actually getting any personal work done.  

Seems Poetic Stanziel over at Poetic Discourse has started a data farming project too.  Their post highlighted an issue I was about to have with my own Toaster project.  Though I have parsed down my zKB calls to allow for rather extensible operation, I made no accounting for failing mid stream.

When I set up my eve-central parser, it's set up to take bites by-day (because that's the way the dumps are).  If I fail to pull a day down, that date is not written into the result DB.  When I start the eve-central parser, it first figures out what days are missing before starting to query eve-central.  If I crash for whatever reason in the eve-central parsing, all I have to do is run the script again and it will pick up where it left off (or do some small ammount of DB admin to clean up FUBAR'd data).

With zKB, my bites have to be much finer.  Though I start with the same date filtering as eve-central, I then cut the queries even finer to by-group.  zKB then needs one more cut, since it will only return 200 results per query.  So, in a single day, I could require 100+ queries.  If I start dropping calls catastrophically, I could have big chunks of half-finished data in the ToasterDB.  This would be compounded at restart, leaving me with big chunks of unreliable data.

The solution took several layers:


First, I needed to add a simple try/retry loop.  It's reasonable to expect that any one query in a batch of 100+ will fail, even on my eve-central parser.  So, the first stage was to add a try-wait-try-fail routine to the actual fetch operations.
request = urllib2.Request(dump_url)
request.add_header('Accept-encoding', 'gzip')
for tries in range(0,max_tries)
opener = urllib2.build_opener()
except HTTPError as e:
except URLError as er:
print "Could not fetch %s" % (query)
//place query back in queue//
//fail mode//
Pretty simple copy-pasta code from Stack Overflow.  It tries to use the opener (after setting request encoding), and retries until failure.

3- Strikes

It would be easy enough to fail the retries with a "Try again later, champ" message.  Unfortunately, this is a very bad idea for my zKB tool.  If the connection is going to be shaky, but not totally crash, I need to be able to handle failures gracefully.

This means I have to do the following:
  1. Place the failed query back into the work queue, or have another way to keep track of reruns
  2. Count these failures and only crash the program when things have gone critically wrong
To handle this, I built a class:
class strikes:
def __init__(self, what):
self.strike = 0
self.max_strikes = config.get("GLOBALS","strikes")
self.what = what
def increment(self):
self.strike +=1
def decrement(self):
self.strike += -1
if self.strike < 0:
def strike_out(self):
if self.strike > self.max_strikes:
print "Exceded retry fail limit for %s" % self.what
This isn't the final revision but it gives you an idea.  Each part of will have its own strike counter.  If too many strike.increments() happen, the whole program will fail.  It also allows a cool-down function to reset the strikes if the connection is just wavy.


It's bad enough you failed the query (repeatedly), and this happened so often it's just time to die... but now I have to die in a manner that is useful for retry.  Pardon the pure pseudo code, I haven't written this part yet.

//fetch ALL the current work queues//
//generate fail_<script>.json files//

This is several fold.  First, I intend to make the program multi-threaded/parallel on final release.  This means any sub script inside the wrapper could fail.  When an abort is called, I need to get the data on where all the children were on the queue and dump out the progress.  JSON is just because I can basically write/read the objects as they appear inside the program without having to make human-translations.  

This will mean I also have to change the program init stage.  Where I was originally just asking the DB what is missing, I can also check against fail_*.json logs to restart the program.

Given Infinite Time...

The idea with the retry scheme was to mimic some of what TCP does to manage throughput on a busy connection.  Though I am not worried at all about the actual bandwidth delivering me my data, employing an intelligent fetcher will make all parties run smoother.  

To do this, I would need to make my strikes/sleeps/retries more dynamic.  This would mean every time a certain operation failed, it would save new data about the issue.  I'm not a classical CS guy, but it would be possible to keep a running tally of "time since last fail" or other metrics to categorize each fail until a dynamic steady state was found.

... but that won't get me my data... and is over engineering.  Perhaps when I have more time than sense I'll consider putting some dynamic back-off, but the retry-wait-strikes-fail routine I have already should be adequate to build the DB I'm after... even if that takes several retries.

Monday, March 11, 2013


Spring is creeping in, and I'm about to hit the same wall I hit periodically.  Like clockwork, I am getting stircrazy with my current state of play.  Also, it's about time to reevaluate my goals and progress and take a little time to decompress and start fresh.  It's time for a bit of an EVE holiday.

The confluence is due to a few factors.  First, I've made some less-than-great bets that need some time to shake out.  Second, I am brushing against the acceptable upper limit for work in-game.  Third, I'm really not pocketing the ISK I'd like to be.  The hope here is to extend this work week out from 8d to 14d, allowing me the time to sell this week's products for a higher margin than currently expected.

Now, what does a heavy industrialist vacation look like?  Sadly, it would be unwise to just full-stop the production line while I slack off.  First, there is a baseline of low-effort product that would be silly not to keep up with.  Second, there are some products that require enough lead time that any break from production is effectively doubled (read: freighters).  So, think of it as more of a "half time" break than a full shutdown.

What to do in the break?  I'd like to really get a lot of code done.  I have a few people waiting up to join in open development, but I have given myself some benchmarks to meet before accepting their help.  Half because I need to do the code; it's a learning exercise.  Half because if I don't, I'll be left being the conductor writing pseudo-code, relying on others to do it for me.

I will post soon(tm) about the different levels of commitment for the industry project.  Maybe I can help pull back the curtain, or iluminate how I could be doing it better.

Friday, March 1, 2013

February Progress Report

Totally stealing inspiration from EVERYONE on this one.  Being the end of the month, it's time to review progress and check against goals!

First, graphporn:

Predicted progress:

 Actual progress (to WW05)


Profits and margins slid in February.  Growth in margins was eliminated by the drop in Nomad prices.  Shrinkage in overall returns was due to a mix of issues.  One piece can be attributed to slacking off, I lost some 7+ days in an already short month.  Second due to some stress from being overleveraged due to the Nomad build.  And third, a significant shrinking in margins on ships due to the cost-spike related to the NOTEC announcement.  

Margins are cooling on a lot of my mainstay products.  I want to believe this is due to a very active industrial surge to match with the surge in accounts since Retribution.  Also, ships are becoming a shaky place to operate, since many T1 variants are matching most T2 hulls at significantly lower costs.  The only place left for producers are the ships with hyper-focused roles.  T2 modules still are selling very well though, so much of the cooling from ships is being picked up in modules... but there is a time bottleneck on very-high-volume module production preventing me from really cashing out in that realm.

Goals Checklist:

In-Game Goals:

  • Stockpile 10B ISK: Failed
    • Managed to stock ~8B into the bank, but 4B is slated for resuming JFs soon(tm)
    • Committing another 2-3B to a market-hub experiment in the FW warzone
  • Purchase BPOs for Black Ops: Success
    • Though I intended to get both Dominix/Armageddon BPOs, I settled for 'Geddon x2 for now
    • If Redeemers preform well, will add Sin's in short order
  • Maximize POS/character utility: C+
    • POS allocation is adequate at this time.  Though will be tough if another contractor starts
    • Invention/Copy/Research allocation on track
    • Was not able to start 2nd ship manufacturer due to cycle delays and ISK limitations
  • Start up 9th industry mule: On Track
    • Char #8 is ready to run, but still needs ~35d more training to be complete.  
    • Trying to have more all-4 inventors to spread the invent load out

Prosper Goals

February was the month to finally nail down the database.  And once again, as I do so often, I underestimated the work.  So far, progress has been good, but slow.  The RAW price database (1.5yrs of eve-central data boiled down to by-system "box and wisker" values) will be done this week.  Also, I have started laying the framework for the next DB, kill data. 

My shortfall can be blamed on a lot of factors.  February started with a "manic" verve to get code down, but was quickly fizzled due to IRL work stress.  Also, my method of coding is a lot like a rat working through a maze for the first time.  I spend a lot of time doing it wrong, and I have to gut 70% of the work and try again.  I think I've put myself on a sustainable path to avoid rework, but there were a lot of hours of work, and not a ton of code to show off.  If I can commit more good chunks of undisturbed time, like it's a homework assignment, I can make some real strides and progress down.

I'll discuss the various databases and their goals in next week's "goals" post!

Market Prediction Review

  • Shrink in overall margins: True
    • Margins are shrinking slower than I expected.  
    • "Guns and Bullets" isn't good enough.  I had to diversify into additional modules
  • Blaster rubberband: True
    • As I predicted, blasters crashed and rebounded.  
    • Was perched perfectly to milk the incoming bubble 
  • Ship Margin Shrinking: True
    • Though I was more doom-and-gloom, the ground under T2 ships is shrinking
    • Zealots weren't as good a bet for Feb as I anticipated
    • I bet on the wrong hictor (Phobos)
    • Recons are volatile   
  • What I didn't see:
    • NOTEC caused price bump in ships, but shook out quickly
    • Nomad Price Crash
    • Medium Ammo margins halved
    • Improvement in general module profits and volume
    • Weird volume problems in decryptors.  Need more investigation
I will write a goals/prediction post for March this weekend.  It's nice to set goals and be able to check off boxes!