17.5M Entries
40hr estimated parsing time
| Frigate | 905,329 |
| Cruiser | 315,493 |
| Battleship | 77,617 |
| Industrial | 81,642 |
| Capsule | 929,041 |
| Titan | 25 |
| Shuttle | 41,814 |
| Rookie ship | 246,308 |
| Assault Frigate | 110,147 |
| Heavy Assault Cruiser | 30,583 |
| Deep Space Transport | 2,421 |
| Combat Battlecruiser | 170,480 |
| Destroyer | 332,165 |
| Mining Barge | 54,804 |
| Dreadnought | 2,218 |
| Freighter | 1,960 |
| Command Ship | 6,340 |
| Interdictor | 32,956 |
| Exhumer | 24,032 |
| Carrier | 4,873 |
| Supercarrier | 113 |
| Covert Ops | 39,401 |
| Interceptor | 52,546 |
| Logistics | 15,082 |
| Force Recon Ship | 24,132 |
| Stealth Bomber | 97,226 |
| Capital Industrial Ship | 247 |
| Electronic Attack Ship | 5,940 |
| Heavy Interdiction Cruiser | 3,961 |
| Black Ops | 1,129 |
| Marauder | 1,375 |
| Jump Freighter | 861 |
| Combat Recon Ship | 8,368 |
| Industrial Command Ship | 2,986 |
| Strategic Cruiser | 32,309 |
| Prototype Exploration Ship | 265 |
| Attack Battlecruiser | 82,652 |
| Blockade Runner | 11,583 |
Remaining To-Do
- Investigate count bug
- Initial dump is 10x expected values on items?
- Finish "prettying" for release
- Incorporate config files to avoid committing sensitive info
- Work on monolithic program to init/maintain DB's
- Update pricefetch to scrape all regions for full market picture
- Find a way to maintain/release .sql dump of data generated
- mySQL optimization and "bridge" scripts for smaller passes
Progress So Far
I have to thank a bunch of people for helping me get to this point where I have at least a passable crawler and data set to munch on. I would like to get EVE-Central's dumps processed before moving onto the data science step, but we will see what happens.
Extra special thanks to:
I still have a lot of work to go between "working" and "good", but being able to stand upright and get my hands on this data is exceptionally awesome.
Finally, I can put together data like this:
No comments:
Post a Comment