Paradigms and Perspectives

I've been reading a lot of Merchant Monarchy lately and mulling over our polar opposite perspectives.  I've never had a taste for margin trading, so perhaps our methods are meant to be opposite.

The War on the "No-Lifer"

This is an admirable goal, and one all traders should strive for.  Diddling on the 0.01 ISK game is irritating, and though there are methods to combat botters, playing the small-change game is silly.  

I personally am in the camp that "game mechanics reward this behavior".  Half because trying to play otherwise is doing more damage to your bottom line than anyone else's (on the short scale).  Half because those who think they know better usually overestimate their skill or intuition.  I've been in many a market where someone did irreparable damage to a margin because they were sinking the price on too much volume.  I've cleaned out orders to spike prices.  There is an art to margin trading, and I have never been able to make that playstyle work to my satisfaction.

But the theme is admirable.  The goal in any ISK generating endeavor should be minimal effort for maximum return.  If you're babysitting 100's of orders trying to keep ahead on the 0.01 ISK game, you're gonna have a bad time.

The "Sheep"

Again, our philosophies diverge.  Personally, I try to follow the herd, the mentality being "make hay while the sun shines".  I get the general consensus that being in the herd will lower final returns vs being on the cutting edge, but when it comes to :effort: vs ISK, I have found it's great to blend in with the sheep.

I understand that participating in the herd this way makes me liable to run off the cliff, which I have done.  But my personal philosophy is closer to "fishing" than "herding".  I use my tools to take a look over the whole spectrum of possibility, then zero in on the trend of the week.  I usually miss the peak, but when the margins are this wide, there's still plenty of ISK to be made.  As long as I'm near 5% of the peak, I count it as a win. Mostly because chasing the edge of bubbles would lead me to madness!

Being a manufacturer, I am absolutely playing either a predictive/speculative game or a follow-the-leader catch-up game.  Thankfully, the cycle times on T2 and the costs associated with broad-base production give me an edge when I can quickly switch products while my competitors languish on the wrong product.  Also, having a large operating purse gives me the ability to change directions on a dime, rather than being stuck to worse products.

Another advantage I have is hiding in the crowd gives me the fastest path from product-->ISK.  By participating in the middle of any particular market, I can nearly guarantee my predicted margins and quickly cash out.  Though there are a ton of places I could technically be making more ISK, I avoid them because the market velocity is not on my side.  By hiding in the herd, I can quietly milk ISK out of the market and neither turn heads nor fall victim to price adjustments out of my favor.

To abuse another cliche, "I don't have to outrun the bear, I just have to outrun the slowest friend"

Can't Argue With Winning

At the end of the day, as long as you're making money and having fun playing the game, who can say you're doing it wrong?  This is the nature of the sand box.  My method is (nearly) perfect for my goals.  I'm cashing out billions per week, and my operating purse is at a suitable scale to allow me all the flexibility I need.  

Would I love to be ahead of the curve?  Absolutely!  But IRL is putting a terrible damper on my ability to build new tools.  If I can get better visibility into perspective products, specifically ships, I should be able to improve returns.  Regardless, I need to find something to help ride out the summer doldrums which are quickly approaching.

Also, if I may Pull a Johnson.  Working on a shiny-new-big-awesome-sekret project.... SHHHH, I can't tell you about it yet.

More Notes On Stockpiling

I've written about my disdain for stockpiling before, I've even tried to write a guide.  Recently, I've been mulling a problem whose obvious solution is stockpiling, but I'm trying to resist the temptation.  Again, as a nod to Chella at Low Sec Lifestyle, I'll try a little stream of conciseness to illustrate the problem... maybe there will be a solution at the end.

The Problem: Hictors

I have a love-hate relationship with hictors lately.  On the one hand, they are a great T2 ship manufacturing investment because they serve a specific niche that cannot be replicated in any other hull.  On the other hand, they serve such a laser-focused role, they aren't consumed in extreme volumes.  This has made the margin erratic.  The internal conflict is compounded by the fact that hictors cost significantly more than the rest of the cruiser hulls to produce.  

What I imagine is happening is demand is spiky.  When a supercap brawl takes down a bunch of hictors, logistics guys backfill stocks.  This burst in demand drives the price skyward.  The burst in margin triggers builders to fill the market need to milk out that sweet sweet isk.  The problem is it can take 3-5d+ to go from "that's a good investment" to "cash out" regardless of your preparedness.  The higher the margin climbs, the more people rush to cash-in.  This causes, what I imagine to be, a collapsing wave.  Suppliers quickly outstrip demand and the price collapses, eliminating margins.

Usually, I would write off the product volatility as "too much risk" and ignore it, but I've tasted the sweet success of cashing out while the bubble was high multiple times.  This is a problem that can be solved, if I can just find the right method to approach the product with.  It's inherently obvious that my current practice of work-weeks/kits isn't the right tool for the job.

Stockpile, Stupid

The obvious solution here is stockpile the built ships.  The problem I see is you're fighting two market forces there rather than one.  On the one hand, you're betting on material costs, and on the other, you're betting on the price of the item.  Don't get me wrong, there are places this is a good investment, but I don't think ships are.  There are so many people participating in ship production, their prices tend to be tied strongly to a specific margin.  With material prices slowly sliding, and a personal expectation that component prices will take a dive with Odyssey, this is a terrible way to conduct business.  Lastly, velocity on hictors is TERRIBLE.  The whole thing makes me feel like I'm lining everyone's pockets except mine.

There are places to stockpile.  Personally, I stockpile invented T2 BPCs and at least 4x the T1 BPC variants.  This cuts 2-4d from the kit-to-market time, and the T1 BPCs help me back-fill quickly.  I like blueprints because the cost stays rather solid, and market changes only move the final product price single-digit percent.  Unfortunately, this still leaves at least 3d on the kit-to-market time, which means I need to get better at sniffing out bubbles so I can leap in and cash out quickly.

Lastly, there's a throughput issue to produce these hulls in volume.  First, it takes 2-4d to invent enough for a good batch.  This is matched by the time required to build the components, and add another 2 days for the final build.  In a perfectly-organized project, you can overlap invent/intermediate-build, but in that case it will still take you 6 days to go from "I should build that" to finally on market.  This is too slow for effective market milking.

Better Tools

Really, the answer here is get off my lazy ass and finish the tools I've been talking about for over 6mo now.  With a graph watching historical margin + better reporting into when large fleets get whelped, hictors would be a great place to really put my automated tool through its paces.

In the mean time, I will just need to add hictors to an "asynchronous schedule".  Instead of making them a standard part of my build, I should do a better job to watch prices/margins with my existing tools and pounce on opportunity when it arises.  This should be easy to do with my ammo builders in standby, but when I put them back to work, it will be tough to meet the component build to be able to cash out in big enough numbers.  Perhaps it's time to dedicate a character to full-time component building and extend my kit buys an extra week out.

Failing Gracefully

Recently ran into two problems, one in-game and the other in Prosper.  I'm going to try and merge both thoughts into a single blog post.  Might be reading too much of Chella's Low-Sec Lifestyle!

In-Game: The Bad Investment

I've stressed mea culpa on my bad investments before, like the Nomad.  Recently, the ball strapped to my ankle has been a perfect-storm with Redeemers.  First, I repeated the same mistake I made with the Nomad, hopping in as the bubble was crashing.  Second, I managed to buy-in when materials were spiking.  So I'm stuck with a double whammy absolutely murdering my margin.

I cleared my buy-in schedule to try and ride out the turmoil, but it looks like I will be selling out near-cost.  With T1 minerals rapidly deflating, global production volume growing, and sales volume really not keeping pace, black-ops have settled to similar margins as everything else.  If you don't derp the buy-in, there's still money to make... but it's a lot of effort and capital to tie up.  Call me greedy, but if I'm going to plunge in 2-3B on a buy in, I want to see really strong returns for the trouble.  Otherwise there are plenty of smaller bets that are paying off well enough.  Also big buy-ins just make me edgy when it's time to ship kits to the factory... still waiting for Miniluv to gank one of my freighters.

The funny part about all this was I flew in a Gal FW fleet on my main a couple weeks ago, and one of my old friends from Aideron Robotics was jabbing me about my recent industry fails.  Except, that in none of these fails have I lost money.  It's similar to the hard time I give my cube-mate in real life.  He sold of his company stock 6wks earlier than he should have... whenever I give him a friendly jab about it, he reminds me he still took home money... what's there to complain about?

Without risk, the game isn't fun.  I actually get excited for days as I ramp up to a big risky project.  Maintaining a status quo is boring.  In fact, I might be taking on a shiny new big project very soon.  I hope to blog about it as soon as details are hammered out.

Out-of-Game: Shaky API

March has flown by at mach speeds just like Feb before it.  I've found very little time to commit to actually getting any personal work done.  

Seems Poetic Stanziel over at Poetic Discourse has started a data farming project too.  Their post highlighted an issue I was about to have with my own Toaster project.  Though I have parsed down my zKB calls to allow for rather extensible operation, I made no accounting for failing mid stream.

When I set up my eve-central parser, it's set up to take bites by-day (because that's the way the dumps are).  If I fail to pull a day down, that date is not written into the result DB.  When I start the eve-central parser, it first figures out what days are missing before starting to query eve-central.  If I crash for whatever reason in the eve-central parsing, all I have to do is run the script again and it will pick up where it left off (or do some small ammount of DB admin to clean up FUBAR'd data).

With zKB, my bites have to be much finer.  Though I start with the same date filtering as eve-central, I then cut the queries even finer to by-group.  zKB then needs one more cut, since it will only return 200 results per query.  So, in a single day, I could require 100+ queries.  If I start dropping calls catastrophically, I could have big chunks of half-finished data in the ToasterDB.  This would be compounded at restart, leaving me with big chunks of unreliable data.

The solution took several layers:

Try-Try-Again

First, I needed to add a simple try/retry loop.  It's reasonable to expect that any one query in a batch of 100+ will fail, even on my eve-central parser.  So, the first stage was to add a try-wait-try-fail routine to the actual fetch operations.

request = urllib2.Request(dump_url)
request.add_header('Accept-encoding', 'gzip')
for tries in range(0,max_tries)
try:
opener = urllib2.build_opener()
except HTTPError as e:
time.sleep(sleep_time)
continue
except URLError as er:
time.sleep(sleep_time)
continue
else:
break
else:
print "Could not fetch %s" % (query)
//place query back in queue//
//fail mode//

Pretty simple copy-pasta code from Stack Overflow.  It tries to use the opener (after setting request encoding), and retries until failure.

3- Strikes

It would be easy enough to fail the retries with a "Try again later, champ" message.  Unfortunately, this is a very bad idea for my zKB tool.  If the connection is going to be shaky, but not totally crash, I need to be able to handle failures gracefully.

This means I have to do the following:

1. Place the failed query back into the work queue, or have another way to keep track of reruns
2. Count these failures and only crash the program when things have gone critically wrong

To handle this, I built a class:

class strikes:
def __init__(self, what):
self.strike = 0
self.max_strikes = config.get("GLOBALS","strikes")
self.what = what
def increment(self):
self.strike +=1
self.strike_out()
def decrement(self):
self.strike += -1
if self.strike < 0:
self.strike=0
def strike_out(self):
if self.strike > self.max_strikes:
print "Exceded retry fail limit for %s" % self.what
fail_gracefully()

This isn't the final revision but it gives you an idea.  Each part of DB_builder.by will have its own strike counter.  If too many strike.increments() happen, the whole program will fail.  It also allows a cool-down function to reset the strikes if the connection is just wavy.

Die-Gracefully

It's bad enough you failed the query (repeatedly), and this happened so often it's just time to die... but now I have to die in a manner that is useful for retry.  Pardon the pure pseudo code, I haven't written this part yet.

//fetch ALL the current work queues//
//generate fail_<script>.json files//

This is several fold.  First, I intend to make the program multi-threaded/parallel on final release.  This means any sub script inside the DB_builder.py wrapper could fail.  When an abort is called, I need to get the data on where all the children were on the queue and dump out the progress.  JSON is just because I can basically write/read the objects as they appear inside the program without having to make human-translations.  

This will mean I also have to change the program init stage.  Where I was originally just asking the DB what is missing, I can also check against fail_*.json logs to restart the program.

Given Infinite Time...

The idea with the retry scheme was to mimic some of what TCP does to manage throughput on a busy connection.  Though I am not worried at all about the actual bandwidth delivering me my data, employing an intelligent fetcher will make all parties run smoother.  

To do this, I would need to make my strikes/sleeps/retries more dynamic.  This would mean every time a certain operation failed, it would save new data about the issue.  I'm not a classical CS guy, but it would be possible to keep a running tally of "time since last fail" or other metrics to categorize each fail until a dynamic steady state was found.

... but that won't get me my data... and is over engineering.  Perhaps when I have more time than sense I'll consider putting some dynamic back-off, but the retry-wait-strikes-fail routine I have already should be adequate to build the DB I'm after... even if that takes several retries.

Burnout

Spring is creeping in, and I'm about to hit the same wall I hit periodically.  Like clockwork, I am getting stircrazy with my current state of play.  Also, it's about time to reevaluate my goals and progress and take a little time to decompress and start fresh.  It's time for a bit of an EVE holiday.

The confluence is due to a few factors.  First, I've made some less-than-great bets that need some time to shake out.  Second, I am brushing against the acceptable upper limit for work in-game.  Third, I'm really not pocketing the ISK I'd like to be.  The hope here is to extend this work week out from 8d to 14d, allowing me the time to sell this week's products for a higher margin than currently expected.

Now, what does a heavy industrialist vacation look like?  Sadly, it would be unwise to just full-stop the production line while I slack off.  First, there is a baseline of low-effort product that would be silly not to keep up with.  Second, there are some products that require enough lead time that any break from production is effectively doubled (read: freighters).  So, think of it as more of a "half time" break than a full shutdown.

What to do in the break?  I'd like to really get a lot of code done.  I have a few people waiting up to join in open development, but I have given myself some benchmarks to meet before accepting their help.  Half because I need to do the code; it's a learning exercise.  Half because if I don't, I'll be left being the conductor writing pseudo-code, relying on others to do it for me.

I will post soon(tm) about the different levels of commitment for the industry project.  Maybe I can help pull back the curtain, or iluminate how I could be doing it better.

February Progress Report

Totally stealing inspiration from EVERYONE on this one.  Being the end of the month, it's time to review progress and check against goals!

First, graphporn:

Predicted progress:

 Actual progress (to WW05)

Analysis 

Profits and margins slid in February.  Growth in margins was eliminated by the drop in Nomad prices.  Shrinkage in overall returns was due to a mix of issues.  One piece can be attributed to slacking off, I lost some 7+ days in an already short month.  Second due to some stress from being overleveraged due to the Nomad build.  And third, a significant shrinking in margins on ships due to the cost-spike related to the NOTEC announcement.  

Margins are cooling on a lot of my mainstay products.  I want to believe this is due to a very active industrial surge to match with the surge in accounts since Retribution.  Also, ships are becoming a shaky place to operate, since many T1 variants are matching most T2 hulls at significantly lower costs.  The only place left for producers are the ships with hyper-focused roles.  T2 modules still are selling very well though, so much of the cooling from ships is being picked up in modules... but there is a time bottleneck on very-high-volume module production preventing me from really cashing out in that realm.

Goals Checklist:

In-Game Goals:

• Stockpile 10B ISK: Failed
• Managed to stock ~8B into the bank, but 4B is slated for resuming JFs soon(tm)
• Committing another 2-3B to a market-hub experiment in the FW warzone
• Purchase BPOs for Black Ops: Success
• Though I intended to get both Dominix/Armageddon BPOs, I settled for 'Geddon x2 for now
• If Redeemers preform well, will add Sin's in short order
• Maximize POS/character utility: C+
• POS allocation is adequate at this time.  Though will be tough if another contractor starts
• Invention/Copy/Research allocation on track
• Was not able to start 2nd ship manufacturer due to cycle delays and ISK limitations
• Start up 9th industry mule: On Track
• Char #8 is ready to run, but still needs ~35d more training to be complete.  
• Trying to have more all-4 inventors to spread the invent load out

Prosper Goals

February was the month to finally nail down the database.  And once again, as I do so often, I underestimated the work.  So far, progress has been good, but slow.  The RAW price database (1.5yrs of eve-central data boiled down to by-system "box and wisker" values) will be done this week.  Also, I have started laying the framework for the next DB, kill data. 

My shortfall can be blamed on a lot of factors.  February started with a "manic" verve to get code down, but was quickly fizzled due to IRL work stress.  Also, my method of coding is a lot like a rat working through a maze for the first time.  I spend a lot of time doing it wrong, and I have to gut 70% of the work and try again.  I think I've put myself on a sustainable path to avoid rework, but there were a lot of hours of work, and not a ton of code to show off.  If I can commit more good chunks of undisturbed time, like it's a homework assignment, I can make some real strides and progress down.

I'll discuss the various databases and their goals in next week's "goals" post!

Market Prediction Review

• Shrink in overall margins: True
• Margins are shrinking slower than I expected.  
• "Guns and Bullets" isn't good enough.  I had to diversify into additional modules
• Blaster rubberband: True
• As I predicted, blasters crashed and rebounded.  
• Was perched perfectly to milk the incoming bubble 
• Ship Margin Shrinking: True
• Though I was more doom-and-gloom, the ground under T2 ships is shrinking
• Zealots weren't as good a bet for Feb as I anticipated
• I bet on the wrong hictor (Phobos)
• Recons are volatile   
• What I didn't see:
• NOTEC caused price bump in ships, but shook out quickly
• Nomad Price Crash
• Medium Ammo margins halved
• Improvement in general module profits and volume
• Weird volume problems in decryptors.  Need more investigation

I will write a goals/prediction post for March this weekend.  It's nice to set goals and be able to check off boxes!