Tuesday, October 18, 2016

Up And Down - The OHLC Endpoint Story

This is the first part of a more technical devblog. I will be writing up more specifics in a part 2, but I wanted to talk about the ups and downs behind the scenes with our EVE Mogul partnership. Issues are mostly my failings, and Jeronica, Randomboy50, and the rest of the team have been amazing given my shoddy uptime.

Prosper's OHLC Feed

I forgot to blog about this since the plans for Prosper's v2 codebase have only recently solidified, but we have a CREST markethistory -> OHLC feed hosted at eveprosper.com. The purpose was to run Flask/REST through its paces, but Jeronica over at EVE Mogul whipped up a front-end and Roeden at Neocom has been using it in their trading forays.

SSO Login Required To View

This originally served me well as a learning experience, but keeping a REST endpoint up isn't as simple as originally expected. From Flask's lack of out-of-the-box multithread support, to some more linux FUBAR's below, it's been a wild ride. And now that players are legitimately counting on this resource as part of their toolchain, I figured it's time to get my act together.

The Litany of SNAFUs

What really brought the house of cards down was our move from a traditional hosting service to a full r/homelab solution. Prosper has been living besides some other nerd projects (minecraft, arma, mumble, etc) and this move gets Prosper off the shitlist from the other customers when Wednesday night rolls around and I hammer box generating the show's plots. Unfortunately, for the added performance, we trade being under a benevolent tinkerer; restarts and reconfigs are more common than before. It's a huge upgrade, and I can't thank Randomboy50 enough for the support, but nothing is truly free (except the minerals you mine yourself™).

#nofilter #bareisbeautiful

This need for stability runs headlong into a shitty part of python: package deployment. Though wheeling up and distributing individual python libraries is easy, deploying python as a service is not. There will be a second blog on the specifics, but you're largely stuck with magic-project-deploying scripting out of the box, which can get really hairy if you're not careful about virtualenvs.

Thankfully, work turned me on to dh-virtualenv and though now we're grossly overengineered with a service .deb installer, we now have a properly deployed linux service that should be far more robust going forward. It does mean that there's now "build" and "deploy" steps for updates, but now that we're tied into systemctl the endpoint should be much less likely to go down.

With the last few months of work, I still expect a large amount of reengineering in our quest for a Quandl-like EVE service, but with the installer built in we can upkeep the endpoint with a lot less effort going forward. We are still behind on the ProsperWarehouse rollout, getting scrapers rewritten, but those modules should be a cakewalk to deploy now with ProsperAPI properly built up.

Also, I've worked in a discord logging handler which will be useful for monitoring, but notes on that later ;)

Friday, October 7, 2016

Stoking the Hype Train - September Economic Report

Editor's Note: blog was published about 30mins before CCP released updated notes about alpha clones.  The ban on Alpha-multiboxing means we revise back our expectations on low-end minerals in HS

September's economic numbers were released this week.  And though we got a sneak-peek in our o7 Show Market Brief, having the real numbers released to the public gives us a chance to really review the state of the EVE economy.

I'm going to break from formula this month.  Where previously I've tried to explain the technicals, I would rather take this chance as a primer for the fall/winter seasons with all the juicy features on the horizon.

What's Not In The Report

Sadly, the economic numbers were cut just before some of the biggest news was released.  Though we can see some ripples thanks to Alpha Clones, and some effects in the mineral markets thanks to a barge rebalance, we won't see the waves from the Mining Booster devblog.

Before we pick apart the month-gone-by, I think it's incredibly important to send out some forecasts about minerals:
  • High-end minerals are going to crash thanks to the Rorqual changes
    • 6 exhumer-grade mining drones and a invulnerability button will significantly increase nullsec yields
  • Low-end minerals are going down thanks to Alpha Clones
    • Alpha clones in HS will be very close to "mined minerals are free".  Though total yields should not increase dramatically, costs/risks will fall.
Now, "crash" is a pretty strong term to bandy about, but there will be some very significant moves in all the materials thanks to November's changes.  So much so, that I'd be hard pressed to hold stockpiles in any minerals personally.  Isotopes are a less risky prospect, but the Engineering Arrays could cut POS fuel consumption more than fleet-booster charges will raise it.

Also, if you are the tinfoil-chewing type... Those megacyte volumes pre-devblog sure look suspicious

What Do You See

I said in last month's report that there were some troubling headwinds.  Values were okay, but month-to-month rates were very weak even for the end of summer.  With September's report out, things are looking much better.

Both Net Trade and ISK velocity plots are looking healthy once again.  Ship trade has crossed back over the 1T mark, and minerals have taken a sharp bump thanks to the barge rebalance.  Also, with PLEX about to cross 1.2B, it's interesting to see the net trade values aren't quite matching the slope, pointing to a speculation bubble.

The specifics of ISK velocity are still a little lower than I'd like to see, but with trends pointing positive (and the 30d skew on the calculation) I'm reasonably happy to see EVE warming up for the winter.

Lastly, looking over the PVP numbers, I think it's interesting to see a bump in value destroyed without the kill counts really moving up.  We have seen in past events (Opportunities, Bloody Harvest, The Hunt, etc) where pilots were trafficked in to more combat, and the Purity of the Throne event does not seem to be driving the same activity levels.

Other Signals

The sink/faucet graphs are starting to look better too.  Seeing Active ISK Delta (ISK leaving due to inactive accounts) shrink is heartening.  That's going to be one hell of a statistic to watch in the November report once Alphas release.

And I'm loving this Top5 plot of the sinks/faucets.  Interesting to see things rise and fall with the higher resolution.  The bounty levels post WWB are extremely interesting (would like to see a mission vs rat breakdown).  With levels climbing that fast, I wouldn't be surprised to hear of nerfs/rebalances some time in the next 6mo.

Last but not least, the player vs NPC breakdowns of fees now with citadels is a welcome addition, and we will be looking in to tracking this as a trend in the future.


Though my August outlook was cautiously pessimistic, September has the game back on a good track.  October/November's numbers are going to be the real blockbusters to watch and I hope everyone has put in their EVE Vegas bets.  There's still a lot of upset that can play out between now and November, when Alphas launch.  Nothing was particularly surprising in September's numbers, but the new charts are a welcome addition.

Friday, September 23, 2016

ProsperWarehouse - Building less-bad code

EVE Prosper is first and foremost a data science project.  And though hack-and-slash has got us this far, we need to consider a proper design/environment if we want to actually expand coverage rather than just chase R/CREST/SQL bugs.

There has been some work moving Prosper to a v2 codebase (follow the new github projects here) but ProsperWarehouse is a big step toward that design.  This interface should allow us to open up a whole new field of projects, so it's critical nail this design on the first-pass before moving on.

What The Hell Is This Even
Building a Database Abstraction Layer (DAL).  

Up to now we have used ODBC, but there are some issues with cross-platform deployment, and database-specific weirdness that have caused issues: such as ARM and MacOS support being painful.  Furthermore, relying only on ODBC means we aren't able to integrate non-SQL sources like MongoDB or Influx into our stack without rewriting huge chunks of code.  Lastly, we have relied on raw-SQL and string-hacks sprinkled all over the original codebase, making updates a nightmare.

There are two goals of this project:
  1. Reduce complexity for other apps by giving standardized get/put methods.
  2. Allow easier conversion of datastore technologies.  Change connection without changing behavior
By adopting Pandas as the actual data-transporter, this means everything can talk the same talk and move data around with very little effort.  Though some complexity will come from cramming noSQL style data into traditional dataframes, that complexity can be abstracted under the hood and always yield the same structures when prompted.

How Does It Work?
The Magic of Abstract Methods

I've never been a great object-oriented developer, and I've been especially weak with parent/children relationships.  Recent projects at work have taught me some better tenants of API design and implementation, and I wanted to apply those lessons somewhere personal.  

Database Layer

Holds generic information about the connection; esentially the bulk of the API skeleton.  Whatever Database() defines will need to be filled in by its children.  This container doesn't do much work, but acts as the structure for the whole project under the hood.

Technology Layer

Right now, that's only SQLTable(), but this is designed to hold/init all the technology-specific weirdness.  Connections, query lingo, test infrastructure, configurations.  This is supposed to be interchangeable so you could pull out the SQLTable and replace it with a MongoDB- or Influx-specific structure.  This isn't 100% foolproof with some of the test hooks the way they are built in right now, but by standardizing input/output, conversion shouldn't be a catastrophe.

Datasource Layer

A connection-per-resource is the goal going forward.  This means we give up JOIN functionality inside SQL, but gain an easier to manage resource that can be abstracted.  All of the validation, connection setup/testing, and any special-snowflake modifications go to this layer.  Also, because these have been broken out into their own py files, debug tests can be built into __main__ as a way for humans to actually fix problems without having to rely on shoddy debug/logging.

This adds a lot of overhead for initializing a new datasource.  In return for that effort we get the ability to test/use/change those connections as needed rather than going up a layer and fixing everything that connected to that source.  It's not free, but should be a cost-benefit for faster development down the line.

Importlib Magic

The real heavy lifter for the project isn't just the API object design, but a helper that turns an ugly set of imports/inits into a far simpler fetch_data_source() call.  I would really like to dedicate a blog to this, but TL;DR: importlib lets us interact with structures more like function-pointers.  This was useful for a work project because we could execute modules by string rather than using a "main.py" structure that would need to import/execute every module in-sequence.  This should make it so you just have to import one module and get all the dependent structure automagically.

Without importlib, every datasource would have to be imported like:

Instead now it can look like

A small change, but it should clean up overhead and allow for more sources to be loaded more easily.  Also, this does mean you could fork the repo and build your own table_config path without going crazy trying to path everything.

A Lot Of Work For What Exactly?

The point is to simplify access into the databases.  With a unified design there, we can very easily lay the groundwork for a Quandl-like REST API.  Also, with the query logic simplified/unified, writing direct apps to both fetch/process the data go from 100+ lines of SQL to 2-3 lines of connection.  

By abstracting a painful piece of the puzzle, this should make collaboration easier.  This also buys us the ability to use a local-only dummy sources for testing without production data, so collaborators can run in a "headless mode".  Though I doubt I will get much assistance on updating the Warehouse code, it's a price worth paying to solve some of the more tedious issues like new cron scripts or REST API design with less arduous SQL-injection risk/test.

Monday, September 12, 2016

Fantasy Reflects Reality - August Economic Report

Thanks CCP Quant for releasing the stats AFTER the latest Prosper episode.  Monthly Economic Report - August 2016

Though there isn't anything particularly bad in this month's numbers, it didn't really live up to my expectations outlined in last month's summary.

Tell Me What You See

It's very "second verse, same as the first" looking from July into August.  A lot of the bulk stats look very good.  Net trade and PVP rates are staying steady and moving as-expected ahead of the YC118.8 release with its rebalance to mining.

Looking back at our favorite pairing, PLEX and ISK velocity, things looks good.  We're finally back over the 1B mark, and trending positive. 

PLEX is going to be extremely interesting in the coming weeks thanks to Alpha Clones.  Though I believe most of the positive and negative trends wash eachother out, I think there are some big changes in store for other trends outside of PLEX.  Specifically if more people are logged in and interacting with the game, even with the handicapped-clones, those could put some measurable pressure on mineral consumption and ISK velocities.

Also, most of the discussions I've seen around Alpha Clones center around "free ISK" and though the effects won't be zero (esp for skill trading... free baseclones), I think the pain of exploiting Alphas will tamp down most of the worst worries.  Lastly, we wouldn't be getting the F2P program if Team Security weren't up to sniffing out illicit activity, so I am not overly worried about massive botting rings.

Troubling Headwinds

What has me worried in the short term isn't so much the levels.  Absolute readings from all indicators are looking good.  But looking at the month-to-month rates have me worried.  Looking at the wallet statistics, I'm happy with the total ISK, I'm happy with the sinks/faucets, but I'm not happy about the rate of wallet growth or ISK velocity.  

I expected after WWB+Citadel that players would be able to recoup their lost wealth and start making roads toward regaining normal.  And though the sink/faucet chart says activity is returning to normal, the wallet status graph says total ISK out there to work with is diminished.  The trend isn't reason to raise alarm, but I will be watching these stats going into the winter.

Drawing Conclusions

I'm a big fan of Marketplace (and if I didn't have to work for a living, I'd produce those kinds of stories for EVE).  And if you follow their Friday Roundup, a recent theme with the IRL economic numbers are "it's good... but...".  I have a similar feeling about this month's report; I expected trends to start pointing upward into the fall, but we're just seeing flatness.  We also aren't seeing any fallout +/- from No Man's Sky, which is just a little weird given the bad press post-launch, but high engagement in a scifi property tangentially linked to EVE; I was anticipating some conversion to come back.  

The devs at CCP seem to be happy with the trends, but I remain slightly worried.  Things will most likely be totally fine, but there are some troubling headwinds in the latest numbers to just be wary of.  I fully expect to have more interesting news to report in the September and October numbers.

Friday, August 19, 2016

Switching to Sublime - #DevfleetFriday

This isn't strictly EVE related, but Talk Python To Me has inspired me to post some more dev/meta related content.

To those who have worked with me on code, you'll know at least two things:

  1. I'm a total luddite when it comes to adding helpful tools to my belt (IDE's specifically)
  2. I live and work in Windows when I can
But my new job has me working in Mac and Unix far more often, so my favorite editor Notepad++ is out.  After touring editors and IDEs for a replacement, I have settled on my new home: Sublime 

Sublime New World

"Why Sublime?" was a pretty easy question for me to answer: very cross-platform, and a deep utility library.  In fact, I've pointed my configs at dropbox so I can easily work in the same Sublime across machines.  Also, the cross-language formatting profiles are pretty slick, and was able to get things working the way I wanted without too much pain.

But Sublime really wins with its Package Control.  Nearly anything you would want can be added to the editor in just a couple commands.  And by-and-large, they're easy to set up and use right out of the box.  Also, many of the packages I've dug into are mostly python, so it's been easy to fix small bugs.

Personally, SublimeLinter has been the best thing to happen to my code recently.  Though it took some effort to get working with python3 on the Mac, having the tattle-tale highlights for bad code and bad practices has me writing far more consistent code.  Writing a .pylintrc file is a little obnoxious, and it doesn't validate virtualenv packages (only ones installed in main site-packages), once it's configured it's a real godsend for keeping my code in line.

My one knock against the platform is the need to memorize a large number of shortcuts that aren't inherently obvious.  There are some really slick tricks in the arsenal, but it feels very VIMy to have them handy

Things I Miss From N++

Search Highlighting

N++ has this amazing multi-color highlighting tool that makes traversing large or complicated code blocks easy.  Word Highlight is an okay replacement, but it doesn't do multi-highlight the way I'd like.  

Project Searching

Another very slick N++ feature is its extended search.  Able to quickly see results, contexts, and previous searches in a docked pane was another useful tool I miss having.  Sublime has an extended search, but it pops open in a new document, and doesn't easily fold the way N++ does.

Multi-Pane Interface

This is a total nitpick.  Sublime absolutely does have multi-pane views, and they are actually a step up in the aggregate.  But my one complaint is duplicating a file isn't really obvious.  The "New View Into File" command isn't mapped, and I'd rather r-click a tab and send to pane.

Free or Fee?

My one complaint about Sublime is its licensing.  Though it has a winrar-type "free forever*" mode, it complains semi-randomly at save-time to purchase a license.  At $70, I think the fee is a little steep given how much competition is out there, and its key method (RSA hasn?) is not super intuitive.  

I'm not at all against paying for a thing, but recently many enterprise services have really stepped up their game and spoiled me with their set-and-forget memberships.  Services like XSplit and Adobe just work, and though I pay pretty handsomely for the memberships in the long run, I'd almost rather buy Sublime through a Patreon model.  At the absolute least, I would like an account auth primary key, even if the current key method works as an offline backup.