Up And Down - Maintaining a OHLC Endpoint and Deploying Flask Restful

This is the first part of a more technical devblog. I will be writing up more specifics in a part 2, but I wanted to talk about the ups and downs behind the scenes with our EVE Mogul partnership. Issues are mostly my failings, and Jeronica, Randomboy50, and the rest of the team have been amazing given my shoddy uptime.

Prosper's OHLC Feed

I forgot to blog about this since the plans for Prosper's v2 codebase have only recently solidified, but we have a CREST markethistory -> OHLC feed hosted at eveprosper.com. The purpose was to run Flask/REST through its paces, but Jeronica over at EVE Mogul whipped up a front-end and Roeden at Neocom has been using it in their trading forays.

SSO Login Required To View

This originally served me well as a learning experience, but keeping a REST endpoint up isn't as simple as originally expected. From Flask's lack of out-of-the-box multithread support, to some more linux FUBAR's below, it's been a wild ride. And now that players are legitimately counting on this resource as part of their toolchain, I figured it's time to get my act together.

The Litany of SNAFUs

What really brought the house of cards down was our move from a traditional hosting service to a full r/homelab solution. Prosper has been living besides some other nerd projects (minecraft, arma, mumble, etc) and this move gets Prosper off the shitlist from the other customers when Wednesday night rolls around and I hammer box generating the show's plots. Unfortunately, for the added performance, we trade being under a benevolent tinkerer; restarts and reconfigs are more common than before. It's a huge upgrade, and I can't thank Randomboy50 enough for the support, but nothing is truly free (except the minerals you mine yourself™).

#nofilter #bareisbeautiful

This need for stability runs headlong into a shitty part of python: package deployment. Though wheeling up and distributing individual python libraries is easy, deploying python as a service is not. There will be a second blog on the specifics, but you're largely stuck with magic-project-deploying scripting out of the box, which can get really hairy if you're not careful about virtualenvs.

Thankfully, work turned me on to dh-virtualenv and though now we're grossly overengineered with a service .deb installer, we now have a properly deployed linux service that should be far more robust going forward. It does mean that there's now "build" and "deploy" steps for updates, but now that we're tied into systemctl the endpoint should be much less likely to go down.

With the last few months of work, I still expect a large amount of reengineering in our quest for a Quandl-like EVE service, but with the installer built in we can upkeep the endpoint with a lot less effort going forward. We are still behind on the ProsperWarehouse rollout, getting scrapers rewritten, but those modules should be a cakewalk to deploy now with ProsperAPI properly built up.

Also, I've worked in a discord logging handler which will be useful for monitoring, but notes on that later ;)