CFC Interdiction Follow-Up

This was an interesting project.  Thankfully, some recent tools made the work go much smoother, and I'm really starting to sink my teeth into SAS JMP.  This piece was a lot of fun to write, since the overall theme from the graphs is "CFC failed to make sustained impact to the market", and TMDC is so widely regarded as only a propoganda hole.

• CFC Caldari Ice Interdiction: Final Impacts
• Imgur Graphpr0n gallery
• Ice Raw data
• Secondary raw data
• Mining raw data

Cutting Room Floor: Mining Report

This has been a topic I've wanted to cover, but haven't had the time to really cover it.  Also, the story is pretty brief.  

The story I wanted to tell was "ice miners, when harassed, will switch products", but the interdiction was not strong enough to force that to happen.  The story instead is a slow and steady decay in mineral prices.  It's interesting to see ice overtake minerals again, but I am not completely sure of the reasons.  When I tried to expand the story, I felt that the graphs were "show"ing more than I could really "tell".

Personally, I've been blaming the anemic market on the end-of-summer doldrums.  Unfortunately, I didn't know how how extensive the lull was.  

Graph Credit Ripard Teg at Jester's Trek

In my opinion, I think this is the most blunt graph to showcase on the state of EVE as a whole.  With no major conflict (CFC v TEST in Fountain drove the post-Odyssey spike) and a general malaise in accepting the status quo, my predicted rebound will probably fall far short of my personal expectations.

If I were to speculate in larger terms, I'd say that the slow slump in mineral prices is due to supply remaining constant while demand shrinks.  I could even further speculate that though the number of participants is declining, that there is some supply floor furnished by bots.  I admire CCP's fight against bots, but in a world where eradication is impractical, you are forced to take them into account at some level.  Unfortunately, I have no way to track bot numbers from any of my data feeds, so this effect is 100% speculative.  If demand picks up, I expect the price to flatten out and slowly rise, since full time mining is boring and not as lucrative as other endeavors, but unless player traffic drastically improves, I expect these swings to be very slow.

It's important to remember that we all exist in a circle of life between PVP and industry.  If PVP isn't destroying, industry has no one to sell to; if PVP outpaces industry, the price to PVP goes too high, and people become risk adverse.  My hopes are as we go into the winter, and through Rubicon, that moon incomes will recover to a level that CFC space starts to look enough to start prodding.  Also, I wonder if CFC leadership is intentionally spreading the forces thin just to test what the real end of their reach is.  

Graph Magic

I wanted to step up the quality of the graphs this time around.  Thanks to JMP, I was able to play with some best-fit tools.  

Using the default "best fit" method (or Smooth) isn't a great fit.  Though it is better than nothing, the fact that graph is so stark between periods, the smooth fit just isn't as useful.  Thankfully to a presentation at work, I looked up Kernel Smoother, which gives a bunch of knobs for how to best fit the data.  By changing the fit function to cosine and lowering the weight to better match the data, I'm able to put much better fit lines.  The best part is I can export the results from my best fit fiddling and move those to graph in any way/shape/form.  You'll see the numbers in the raw dumps.

This was exceptionally useful in my favorite graph of the piece, the Tags4Sec graph.  as a reference, this is kinda how it started:

Volume data is particularly spiky.  Being able to line up the wider trend of peaks and valleys I think gave a much more useful picture of the data rather than overlaying that noisy base data.  When I add the default smooth line, it's pretty clear that line tells the "macro" story, but isn't really useful for this project's analysis.  Also, the "repair rate" was an attempt to combine 3 volumes together in appropriate parts.  In this case, it was 1 part Trainer, 1.5 parts Recruiter, and 1.5 parts Transporter.  In retrospect, I could have been more creative with the combination, since I seriously doubt people were going all the way to -10 in ganking, so equal parts Recruiter/Transporter with no Trainer would have been a better picture.

What's Next?

Economic topics are running pretty lean lately.  There is the noise about PLEX prices skirting 600M and then dropping like a stone, thanks to an Amazon sale.  Also, I've been requested to analyze the "special edition" assets, to try and make a ruling on whether it's best to hold onto it as a collectors item, or dump in the initial release frenzy.  Beyond that, I'm out of topics until Rubicon shows up.

Also, though I am loving how much JMP is enabling me to get these sexy graphs out for people to enjoy, there's a licence problem: I have access to JMP at work, but players aren't going to buy keys.  I'd like to ween myself onto tools like R, but there's a lot of code between where I am today and being able to slice-n-dice as easily in a new set of tools.

Cache Scraping SNAFU in Odyssey 1.1

This article was a pretty quick turn around for how many external resources I needed to reference:

• Out of Game Market Feeds Broken in Odyssey 1.1

First off, I'd like to thank the sources on this one.  Fuzzysteve was paramount in breaking the story and bringing the initial coverage.  Also, I have to thank him and K162space for fact checking me on some of the softer parts of the article in draft.

Also, I'd like to thank the other sources who were so good about getting back to me in a timely manner and sharing the kind of insights and links that really brought the piece together:

• Yann Ramin of EVE-Central
• pmchem of GARPA
• Tazuki Falorn of GARPA
• Entity of Reverence

Specifically Tazuki Falorn sending me the best response:

Not-an-official-GARPA-response: 

It's a ridiculous situation and one that should have been fixed years ago. Apart from CCP's general incompetence (see: anything to do with the current API server), there's no reason at all that this data shouldn't be available via some sort of API. Update it every x minutes with data from y minutes ago to limit the use in market bots. Cache data heavily. Spit out data in multiple formats, JSON would be nice since it's no longer 2005 but XML works if we really must. History data is even easier since it only changes once a goddamn day, ugh. 

Mood: mad because people bug me about goonmetrics not working and I'm lazy.

I didn't want to share that directly in the TMDC piece, but it's pretty on-target with most of the feedback I ever receive on this topic.  I didn't want to completely rehash my previous article, The Machines Are Taking Over, but ended up giving a TL;DR review of that piece to build this one

Furthermore, as I was pitching the article to Ali Aras, she was keenly interested in the topic as a CSM member.  Kind of highlighted how narrow I've been with my personal lobbying scope focusing almost exclusively on Mynnna.  We had a pretty decent discussion of the state of things as they are today and I expect to be in more regular contact with him as this topic continues to evolve.  I anticipate another pretty serious round of threats from CCP and counter lobbying from the community around the next expansion, especially since Stillman has hinted at changes to the EULA in the works since last spring.  I sure am glad the CSM is there to help promote some of the less-sexy, but generally important, topics.

Eat Your Heart Out Ejyo - Odyssey Moon Resource Report

This is meant as behind-the-scenes for my latest TheMittani.com article: Odyssey Moon Resource Report

As always, I've made most of my data available for private review:

• Material Price Review

I was lucky enough to get an easy to parse database of moon data directly from Blake Armatage at K162space.com.  This made the map section of the article EXTREMELY easy to make, as opposed to a bunch of pavement pounding and guessing.  Unfortunately for you guys, I've been sworn to secrecy on the data source... so I will leave it up to him to post any more of that data.

This turned into a monumental undertaking.  Trying to slice-n-dice data into presentable formats took a ton of time.  Also, trying to understand and then present in a cogent manner the entire scope of these changes, from moon mining to T2 product, was a ton of work.

Map Data

I am still trying to get over my irrational fear of SQL and the datadump.  Though it took some crowbarring, I was able to force the data into a usable format.

By stacking the raw data K162space furnished into one database, I was able to merge that data set with the existing map data to get a table with counts, per system, of each moon resource.  I have included some basic map SQL fu in the code snippet gallery.

I had prodded Mynnna several times to try and check/improve the database I have, since GSF boasts the most complete database of money moons.  Unfortunately, beyond some jibes about certain groups being over reported (Thulium), I wasn't able to get better results.  As such, if you look at the R8 map, there are some pretty wide swaths missing in the data set in the south and the east.  This was an effort of "do the best you can", and having K162space's dataset was a godsend, even if it may not be 100% complete or accurate.

Once it was merged into the form above, it was onto JMP to try and build the charts.  Turns out mapping values to custom coordinates is a massive pain in the ass.  The solution?  Crunch the original table down further.  By expanding the table to include boolean values of if a system included a particular resource type, then using that boolean to trigger writing the X, Y, Z coordinates to each material.  Then with data columns like Cadmium (x), Cadmium (y), Cadmium (z), of either valid coordinates or NULL, I was able to overlay the data into very pretty maps.

Also, pro tip: 2d map coordinates are (X,Z), not (X,Y).  Credit to @AideronRobotics for finding the issue and pointing me at the correct pairing
This is wrong: X,Y R8 report:

Meanwhile, these are right:

Click to embiggen

I had to get somewhat creative with the data points on R16/R64 because they were completely overlaid.  So by making the "background" points large and the "foreground" points small, this allowed for a decent two-level effect that was better than when all the dots were the same size.  Also this allowed for multiple overlaid points in a single system to show up a little better.  In the future, it might be worth adding small offsets for each data point so they don't overlay as badly.

The REAL treat I wanted to do was recreate the hex-grid for regions/constellations or at least add borders using an ESRI map... but I was extremely out of my element on trying to make that.  Perhaps a project for another day.  If I could build an ESRI map, I could build heatmaps by rarity in each location, which would be sweet to have.

Also, those interested in data dump fu, I have included my SQL queries in the site's Code Snippet Gallery.

Graphs, Graphs EVERYWHERE

I learned my lesson last time with historical data.  This time I picked a reasonably arbitrary date (CCP Rise's HAC announcement) and locked in all the data I would need for that period.  Thankfully, that was a good 120d swath to get a picture before any changes were announced and out until all of Odyssey's messyness had subsided.

The topic I tried to include, but could not do so graphically, was Alchemy.  I added alchemy math to my advanced-material data feeds, but trying to include them in the final graphs was extremely cluttered.  Also, since the demand for T2 materials has been waning in the face of T1 hull preferences, no material was properly bottlenecked to really incentivize alchemy as a side path.  So rather than clutter and add another 2 pages to my report with essentially little value, I skirted the topic.

Trying to merge the above with all the other racials was messy and the added data didn't really add a lot of value.  So, though it's a little more convoluted, I think the below was a better use of that image space.

 Also, I found Google Drawings to be my savior in this endeavor.  Though replacing the graphs when that layer needed a change was more trouble than I would have liked, it took minutes to build really solid pictures and combine images, where real image processing like Photoshop or GIMP would be nearly impossible for me.  

Another big problem was the post-patch spike on new resources that didn't exist on patch day went absolutely nuts until supplies had been secured.  As such, I had to do a little graph finagling.

First, on metamaterials, I added a "production cost" dotted line over the first 10d

I did some data masking to try and accurately project the pre-patch cost, so that when it lined up with Odyssey release, it would show a reasonably flat transition.  I think the picture does reasonable justice, even if the intended line up missed my original goal.

It was more problematic on component build graphs.  In this case, though the instantaneous cost on patch-day was insane, the truth is probably no one was building them at that price.  So, I instead made a sliding conversion rate over the first 10d.  

Without conversion:

With conversion:

I know the shapes look pretty similar, but the spike is greatly reduced.  I contemplating cutting the 10d spike period completely, or forcing the axis to chop that data, but I think this is a reasonably fair look at the spike-and-decay how it probably played out among manufacturers.  The other option was to chart market prices on those items in the first 10d, but I didn't think that was a better picture than the manufacturing cost.

In total, I generated something on the order of 20 graphs while working on this project, and used maybe half of them.  Though my spreadsheet fu is strong, it was a lot of work to collate all that data.

Also, I have to give props to the admins over at TheMittani.com, because the recent contributor-side changes made this article 10x easier to transfer over compared to the Burn Jita article I wrote.  There was a bunch of streamlining, and it's almost as easy as writing here in Blogger.

Egg on my Face

Despite quadruple checking my work, seems I let a misprint through on one of my images.

What was printed:

What should have been printed:

And I sparked a real CSM conspiracy within minutes of my reddit posting.  Thankfully, CCP Fozzie was quick to note the error was mine, and I was able to push correct graphs back to the site.

I feel particularly terrible about this flub because I'm already trying to overcome the mass opinion that TheMittani.com is only a propaganda mouthpiece for CFC.  It's frankly the highest/best place I can publish this data and get it into the hands of the players.  The issue should be corrected now, and I will be more diligent about getting second reviews of my data in the future.

What's Next?

I still want to write up something about mining post-Odyssey, but with the recent announcement of CFC Interdiction of Caldari Ice, I think I will need to hold off another 30-40d.  I will see about gathering data so my turn around post event is a little faster than the ~3wks this article took.  At least mineral thoroughfares are much more direct and less convoluted than the entire T2 production sphere.  Also, I am open to more ideas about industry topics that require a deeper review.

I am also hoping to jump on a few smaller topics.  As much as writing these big articles is fun and all, it's a ton of work and I would like to get more articles out.  

TMC Followup: Burn Jita

Burn Jita: An Industrial Perspective

This article took longer to write than I originally expected.  I should have anticipated that a factual article is going to take longer to gather together and present.  Also, continued trouble with Google Drive processing power keeps putting a damper on my progress.

Why Freighters?

The simple answer is freighters have been at the front of my mind recently.  With the JF Project, my own BPOs, and my historical charting work, freighters have proven to be the most directly useful and recognizable to the widest player base.

Also, by combining the data into freighters as a group, rather than a specific ship, I can accomplish a whole list of things:

1. Avoid graph overload.  Having to reference 4 charts is worse than referencing 1 chart
2. Provide a higher level to talk at, rather than get tangled in the minutia of each item's behavior
3. Not share "secret sauce" to the global audience that reads TMC while still making useful content

I did want to also show T2 mirrored the post-event dumps, but it ended up on the cutting room floor.  I instead used the semi-interview with Warr Atini (head of Miniluv), staying on-topic.  I was afraid of running off the rails with 5 pages of analysis for each sub group and speculation therein.

Pre vs Post Event

This was another topic that fell out pretty fluidly.  The TL;DR, which matches my intuition, is that as a producer, pre-event is nearly useless (except for volume spikes, if you're wanting to cash out), but the post-event is an enormous opportunity to save a lot of money.  I don't usually stock raw materials, but come the next big gank-event, I make efforts to buy up some percentage of materials to suit me for 30 days.

The interesting thing I learned was the speculator side of the equation.  I've never had success participating in the margin/speculating trading, and I'm starting to see why.  Burn Jita was a margin trader's paradise, peaking prices on gank fittings, but that price bubble evaporated once the event was complete.

Missing Part of the Picture

The big missing slice of my article is other hub analysis.  I did start fiddling with feeds to try and generate that data, but found the results too messy and hard to display in one or two graphs.  I also skipped T2 production as a topic because it added another page without adding a lot of value.

I expect to take both the T2 and "everywhere but Jita" topics into their own articles.  The not-Jita article will probably be next, since I still would like 2 weeks for the T2 market to shake out. 

The To-Do List looks something like this:

1. Write Decryptor Tutorial with new decryptors
• Working on refreshed 2.0 tools
2. Draft Mining Report artcile for TMC
3. Draft "Anywhere But Jita" article
4. Write T2 Post-Odyssey analysis
• Working on new charts to group data intelligently 

Data Sources

One of the themes I'd really like to drive home here is "this data can be mined".  These graphs were generated purely using EVE-marketdata's history API.  To increase quality, I had to download their private data pusher to get the history graphs to reflect reality.  

As I wrote before, it took writing some custom code to process it, but I now have a moderately easy to use tool, except when Drive is under heavy load.  In an effort for transparency, so I don't get accused of making it up, I hope to publish all my data sources for review.

• Freighter Data
• Gank Fitting Data
• Mineral Data

DISCLAIMER: history feeds tend to break 9a-5p M-F US TZ.  This is a limit of Drive, please be patient.

None of this data was collected with any special privilege.  The gank fittings came from goon's killboard, the price data from EVE-Marketdata.com.  A theme I would like to push among my articles is "the data is already out there", though you may need to process it.

Error Bars

There's one glaring assumption I made that still makes me uneasy: summing sales volume where prices were averaged.  On every graph that shows volume, I just summed together the volumes to get a general picture.  I feel reasonably okay on the freighters (everything is similar order-of-magnitude), but had a lot of trouble justifying it on the ship fits and mineral report.  I played around with a few different weight modifiers, but never found a combination that I felt was truly honest.  

Hitting the Big Times

My first staff article just went up at themittani.com.  This is my second article for the site, but I only wrote as a guest-author before.

• The Machines are Taking Over: about cache scraping realities in 3rd party dev sphere
• Last Minute Odyssey Speculation: about predicted market impacts for Odyssey expansion

It has been an amazing experience being in the newsroom at TMC.  I find the staff is incredibly open to my ideas, despite my misgivings about how many hits they will actually generate.  I am finding it a really refreshing experience to work collaboratively over at TMC, and it's breathing new fire into my inspiration to write/create elsewhere.

My (few) readers may believe that now that I've "hit the big times" I will neglect my blog here.  This is absolutely false.  Where TMC content is "for the masses", I believe that there are still a lot of personal and hyper-focused topics to talk about here.  As I make new tools, I will debut them here.  As I find mass trends worth reporting, I will use my TMC soapbox.  I hope to leverage both sources here for maximum scope.

With Odyssey just around the corner, there is a torrent of industrial topics to write about, so I hope to be busy on both sites.  I look forward to talking more about industry all summer!