Author Topic: VGC_Analyze - an unofficial analyzer for your VGCollect collection  (Read 957 times)

 Hello everyone,

 
most of you probably haven’t seen me around. I have been using VGCollect to manage my collection for a number of years, but only recently started using the forum.

 
I’ve always appreciated the way VGCollect handles collection management but over the years I felt that the analysis side was a little lacking. To determine specific properties of my collection, I often download the CSV-file from my user profile and dig through the data by hand, using a spreadsheet application. I imagine many of you have done the same before.

 
A while ago I figured I could make my life a little easier by automating the process. So I put together a little script that would download my collection CSV from the website and apply various criteria to analyze it. This turned out to be a great hobby project which provided lots of entertainment in its own right so over the last few months I added a lot of features that I found useful and were fun to implement.

 
At some point I figured that other users might also be interested in some of the functionality my application provides so I got in touch with the staff to see whether it would be alright to make my project public, some of you might have seen my previous post. Turns out it was alright and so I’d like to introduce you to what I’ve been working on.

 
VGC_Analyze

 


 
The creatively titled VGC_Analyze is an analyzer for your VGCollect game collection.

 
VGC_Analyze is an unofficial hobby project, not affiliated with VGCollect.com.

 
For a general overview of the application and installation instructions you can visit the projects GitHub page

 
In this post I’d like to demonstrate the general functionality and provide some use cases for the application.

 
Available data / limitations
As stated before, the application uses the CSV backup of your collection as its data source. The CSV backup was initially intended as a way to restore a VGCollect collection on the website and thus only includes data that is necessary for the given task. VGC_Analyze does a lot with the data that is there but some attributes like publisher, developer, genre or release date can’t be evaluated since they are not included in the file.

 
General mode of operation
The core of VGC_Analyze is the filter sidebar on the left side of the screen. The various filters allow you to pick out specific parts of your collection.  All evaluations are always based on the filtered data.

 
Let’s say I want to compare the number of items of my sixth generation home consoles. I set the platform filter to the respective platforms and all other metrics automatically adjust to the filter result.

 


 
Every filter can be combined with every other filter. If you want to see all your japanese import games with missing manuals, purchased between new years and last week, grouped by the month they were added to your collection, you can.

 
Graphs
Currently VGC_Analyze offers four different graph types. Each graph type can plot multiple attributes like platforms, regions and dates. Some represent total amounts at a specific moment in time, others show growth over a given period.

 


 
Templates
Filters can be saved as templates for quick access to often used configurations. The template manager can be accessed via the application menu or via the key combination Ctrl+B.

 


 
On the left side the previously saved templates are shown. The entry “[current configuration]” represents the filter attributes that are currently set in the main window and can be saved as a new template. The right side shows the individual attributes of the selected template. Templates can be loaded or deleted.

 
Collection download
VGC_Analyze can automatically download the CSV backup of your collection from your VGCollect user profile. The download manager can be accessed via the application menu or via the key combination Ctrl+D. The download manager automatically opens on first run when no collection data is present.

 
Your VGCollect login credentials must be provided to be download your collection. The login credentials are currently not saved and must be re-entered every time you want to update the local copy of your collection. Maybe some time in the future I’ll find the motivation to deal with secure password storage, until then VGC_Analyze forgets credentials immediately.

 
If you don’t feel like entering your login data into some unknown piece of software (which I totally understand), you can instead download your collection backup manually and copy it into the data folder.

 
You can access the data folder from the application menu (File -> “Open data folder”).

 
Restart VGC_Analyze after a new file has been added to the folder. The active file can be selected from the drop down list on the top right of the screen.

 

 
Item search
Search through the active list of items with the quick search function, accessible from the application menu or via the key combination Ctrl+F.

 

 
Export
VGC_analyze can export your collection as a printable HTML file. The HTML consists of a simple table and reflects the display in the main application one to one. All filters, groupings and sort orders are applied exactly as they are set in the application itself.

 


 
The export file is generated in the data folder (File -> “Open data folder”). A simple CSS file is generated alongside the HTML and can be modified if required.

 
The columns of the HTML table reflect the columns in the main application. Currently there are no separate column configurations for the main table and the export table.

 

 
Platform holders
The underlying data for the “platform holder” filter is not provided by the backup CSV file. Instead I chose to implement a list of platform holders that is applied when the data is loaded into the application and can be modified by the user.

 
To access the platform holders configuration open the application settings (File -> Settings) and select the corresponding settings tab.

 


 
VGC_Analyze comes with a default list of platform holders and platform keywords but the list can be modified and expanded by the user.

 
When a CSV file is read the application checks the platform attributes of each collection item and compares them to the platform keywords. So, as displayed in the above example, any platforms that match “gamecube”, “amiibo” or “wii” are associated with Nintendo while “playstation”, “psone” and “psp” are associated with Sony.

 
All platforms that can’t be matched to a platform holder are grouped under “[Other]”.

 
Platform overwrites
To achieve a more consistent data foundation VGC_Analyze provides a function to overwrite platform names. The specific overwrites can be configured in the application settings (File -> Settings -> Platforms).

 


 
The above screenshot represents the default list of platform overwrites, which can be modified or expanded by the user.

 
The reason for platform overwrites is that platforms on VGCollect sometimes have different names for different regions. In general VGC_Analyze tries to extract the region tag of the platform name (e.g. “Nintendo Switch [EU]” is split into platform “Nintendo Switch” and region “EU”) and allows each to be filtered separately.

 
In the case of platforms that have different names for different regions (e.g. “Nintendo Entertainment System” and “Famicom”) VGC_Analyze can’t make a connection between the two and would list each as its own platform. By overwriting both with “NES/Famicom” the data is combined and can be filtered by region if needed.

 
Local data extensions
Currently there are two ways to add additional data to your downloaded collection. Items can be marked as bookmarked and/or finished. Both flags are stored locally in the data folder and can either be set via the buttons on the top right of the screen or via the right click context menu.

 
Regular expressions
The “Title” and “Notes” filters allow the use of regular expressions. To activate regular expressions press the button next to the corresponding input.

 
VGC_Analyze uses the default Python notation for regular expressions.

 
For example: “^Legend.*Time$” would match all items that start with “Legend” and end with “Time”.

 

 
This concludes the basic overview of VGC_Analyze. Please keep in mind that this is purely a hobby project so some ugly code and a bug or two are to be expected.

 
If anyone of you is interested in trying the app yourself check out the projects GitHub page. If you have any feedback please post it here as a response.

 
I’m not a native English speaker so let me know if anything is unclear and I’ll try to describe it in greater detail.

zappman

PRO Supporter

Re: VGC_Analyze - an unofficial analyzer for your VGCollect collection
« Reply #1 on: July 03, 2021, 07:14:26 pm »
Hello x211321,

Thank you for creating "VGC_Analyze". :)
I have downloaded and installed the application.
It seems to be fully functional.
I am still learning what "VGC_Analyze" does.
I hope to provide more feedback and suggestions in the future.

Thanks again,
zappman

retromangia

PRO Supporter

Re: VGC_Analyze - an unofficial analyzer for your VGCollect collection
« Reply #2 on: July 03, 2021, 08:30:49 pm »
Hey X211,

Thank you for creating this awesome data tool, and thank you for taking the time to share with us how to use it.  The VGC Team should hire you! :)


Re: VGC_Analyze - an unofficial analyzer for your VGCollect collection
« Reply #3 on: July 04, 2021, 01:25:56 am »
I downloaded the program and it installed no problem. I went ahead and downloaded my collection data from the site and put into the data folder and restarted the program but not anytime I try to launch it I get a window saying fatal error detected: Failed to execute script VGC_Analyze. Im sure taking the file out from folder will help but for the life of me I cant locate the data folder. Any help would be appreciated.

Re: VGC_Analyze - an unofficial analyzer for your VGCollect collection
« Reply #4 on: July 04, 2021, 05:56:29 am »
I get a window saying fatal error detected: Failed to execute script VGC_Analyze


Sorry to hear you ran into trouble. I’m pretty sure that problem occurs when the app tries to parse the CSV file. There must be some difference between your file and the files I’ve tested with so far. I’ll add some better error handling in the loading routine.

The location of the data folder depends on your operating system.

On Windows the data folder is at:
%APPDATA%\VGC_Analyze_data

This usually maps to:
C:\Users\<USERNAME>\AppData\Roaming\VGC_Analyze_data

On Linux its at:
~/.VGC_Analyze_data

On macOS:
/Users/<USERNAME>/Library/Application Support/VGC_Analyze_data


Thanks to everyone that provided feedback so far. Some of you (especially retromangia) seem to have rather large collections. I’m quite interested whether the application feels sluggish when dealing with so much data.

Edit: Typos

Edit2: I put out a new version with improved error handling. This will not correct the core issue of the incompatible CSV file but at least the program should no longer crash when it tries to read the file. To fix the incompatibility I'd need to figure out the difference that's causing it.


« Last Edit: July 04, 2021, 08:00:51 am by x211321 »

Re: VGC_Analyze - an unofficial analyzer for your VGCollect collection
« Reply #5 on: July 04, 2021, 04:23:41 pm »
Just now got the standalone executable, placed an export I just got into the data folder and encountered this error:

"Error parsing collection data
could not convert string to float: '' "

Note that those are 2 single quotes, not a quotation mark.  The quotes don't appear in the csv file when I searched for it, so that's not the issue. It looks like it's encountering an empty string somewhere.

I haven't looked into the code yet, but I might do that and see if I can get to the bottom of what's breaking. I actually started my software engineering career last year, but have never contributed any code on Github. Maybe this could be my first contribution.

Don't know the forum rules. Can I attach my csv file, and can others that don't have issues attach theirs? It might be good for comparison.

Re: VGC_Analyze - an unofficial analyzer for your VGCollect collection
« Reply #6 on: July 04, 2021, 04:35:31 pm »
Thanks for your feedback.

The only value that is converted to float when the file is read is the purchase price.

I never noticed that VGCollect allows non-numeric values in the price input. All CSV files I tested with always had valid prices so I never thought about validating myself.
A simple oversight that is easy to fix.

I'll publish an update asap.
Edit: update is online
Hope that fixes it.
« Last Edit: July 04, 2021, 05:00:59 pm by x211321 »

Re: VGC_Analyze - an unofficial analyzer for your VGCollect collection
« Reply #7 on: July 04, 2021, 05:05:01 pm »
Ah that makes sense. Is it the field "Purchased Price" that's causing a problem? I think that is something I would have to manually add when adding a game to the collection. I don't do that; just add the game and move on.

Will a lack of purchase price cause some graphs to break?

Re: VGC_Analyze - an unofficial analyzer for your VGCollect collection
« Reply #8 on: July 04, 2021, 05:10:50 pm »
A lack of purchase prices is no problem (you'll simply not be able to evaluate them).
Normally the CSV file VGCollect generates includes a "0" in the price field when no price has been entered but it seems like your file somehow included a white space in one of the price fields instead.
Did the new version help with the problem?

Re: VGC_Analyze - an unofficial analyzer for your VGCollect collection
« Reply #9 on: July 04, 2021, 05:15:47 pm »
Yep it sure did.

Re: VGC_Analyze - an unofficial analyzer for your VGCollect collection
« Reply #10 on: July 04, 2021, 05:22:29 pm »
Great. Again thank you for your feedback.
Missing purchase prices are not much of a problem but if you also don't enter purchase dates then the area and line graphs won't be of much use to you.

In the future I plan to provide an option to base those graphs on the "added date" instead of the purchase date.

pzeke

Re: VGC_Analyze - an unofficial analyzer for your VGCollect collection
« Reply #11 on: July 04, 2021, 08:42:33 pm »
I love coming across people that say they're not "native English speakers", yet, not only do they end up having a good grasp of the language, but better grammar than most actual native speakers. Your post structure could use some work, though...but I'm not going to hold it against you.

Anyway, props for doing this. I'll admit it isn't something I'd use, but I commend you for your labor and thank you at the same time; I'm sure others will find this useful. Quite frankly, this website could use more people like you to inject some much needed vigor to it.

Thumbs up!

I know your every move behind this face; I have control over expendable slaves.
When confrontation comes down to the wire, I'll use my cyclotrode to commence the fire.
You're never gonna get me!

Re: VGC_Analyze - an unofficial analyzer for your VGCollect collection
« Reply #12 on: July 04, 2021, 10:53:05 pm »
I get a window saying fatal error detected: Failed to execute script VGC_Analyze


Sorry to hear you ran into trouble. I’m pretty sure that problem occurs when the app tries to parse the CSV file. There must be some difference between your file and the files I’ve tested with so far. I’ll add some better error handling in the loading routine.

The location of the data folder depends on your operating system.

On Windows the data folder is at:
%APPDATA%\VGC_Analyze_data

This usually maps to:
C:\Users\<USERNAME>\AppData\Roaming\VGC_Analyze_data

On Linux its at:
~/.VGC_Analyze_data

On macOS:
/Users/<USERNAME>/Library/Application Support/VGC_Analyze_data


Thanks to everyone that provided feedback so far. Some of you (especially retromangia) seem to have rather large collections. I’m quite interested whether the application feels sluggish when dealing with so much data.

Edit: Typos

Edit2: I put out a new version with improved error handling. This will not correct the core issue of the incompatible CSV file but at least the program should no longer crash when it tries to read the file. To fix the incompatibility I'd need to figure out the difference that's causing it.

After downloading the second revision after this one, it is up and running on my end. Im figuring stuff out but I am liking what I am seeing so far. You did a great job my friend.

Re: VGC_Analyze - an unofficial analyzer for your VGCollect collection
« Reply #13 on: July 06, 2021, 11:35:45 am »
Oh, this is neat.  I downloaded it and played with it.  I don't leverage the purchase feature of VGC and just go strictly by the date added, so some of the basic stuff doesn't appeal to me, but good stuff.


leonefamily

PRO Supporter

Re: VGC_Analyze - an unofficial analyzer for your VGCollect collection
« Reply #14 on: July 06, 2021, 02:14:06 pm »
Didn't try the program but it looks neat! Congrats!
Freeze iou imperialist pig! Zat is ze propertay of ze Gouvernement Français. Hand eet over!