In an effort to break out of those bad habits, I set out this weekend to deliver a piece of code to solve my EMD_history problem. A bit ago, I wrote a gdoc script that would pull data from EVE-Marketdata's history API. This was crucial in delivering the graphs I've made for the entire summer, including my big TMDC moon article.
Unfortunately, this script is heavy and poorly suited for the gdoc environment. It was prone to time-outs, and required a lot of massaging to get the data sets I needed. Frankly, it was too weak for my needs. And with CFC's Ice Interdiction scheduled to conclude this weekend, I need to be able to process the data quickly and effectively to get results posted while they are relevant.
Pricefetch sourcecode on github
Despite being more effort than originally expected, a usable script has emerged! I don't have a witty name for it (it's way too buggy for full release), but I can at least now start scraping data from hosts and have it in a local version for getting from data to graph much faster than before.
Todo to make it more robust:
- Make a crash path. Allow the program to resume where crashed
- More robust error catching: not handling some common exceptions (http requests)
- More complete arguments: currently only does The Forge, all items
- Add EVE-central dump crunching for stock charts and finer data
- Make a update path. Check existing database for entries and only fill the missing ones
But today, I have a way to get that data into an intermediate faster and in larger quantities than before. As soon as cache scraping is fixed, I intend to finish sending my alt to the busiest regions and pushing data to EMD for a more complete.
Also, once again, thanks to Fuzzysteve for helping me out with some SQL fu. I added his help to the code snippet gallery
Also, once again, thanks to Fuzzysteve for helping me out with some SQL fu. I added his help to the code snippet gallery
No comments:
Post a Comment