General request: Accessing website data from an external application

[x211321]

Hello everyone,

I hope this is the right place for this. If not please move this thread somewhere else, thanks.

I've been using VGCollect to manage my collection for a number of years but haven't been active in the forum until now.

I think VGCollect is a great place to manage game collections. I'm especially fond of how community driven and down to earth everything is. At this point I'd like to thank all of the admins and mods for their efforts in creating and maintaining the site.

As much as I like the website, I sometimes find myself in situations where I'd like to view my collection from different angles. I'm really into lists and stats and often wonder about certain aspects of my collection. So whenever I want to discern something like my total spending or the number of purchases per year I'd download the backup CSV-File from my user profile and then use a spreadsheet application like Excel to find the answers I'm looking for.

A while ago I figured that doing this manually every time is a bit of a pain so I started automating the process. Since I was already looking for an excuse to look into Python programming I figured I'd write a simple Python script to download the CSV-File from the website and to process the data according to my needs. At first it was a simple command line application that could filter, group and sum the data by different aspects and output the result in the terminal. Over time I expanded the concept by adding a GUI, charts, graphs and a lot of other features that I found useful.

A couple month ago the application was at a point where I'd start wondering whether others might also be interested in its features. Until then I didn't plan on sharing it since it's a simple hobby project and I only worked on it for my own convenience and entertainment. The application is still nothing fancy but I figured that it had enough useful features that there might be some others that'd also like to use it.

By this point the application had some features that directly interacted with the VGCollect.com website and I figured that I should not make it public without having someone from the staff approve of those interactions. I tried getting in contact with some staff members via forum messages (hope I didn't bother any of you) but so far no one was able to help me out. That's why I'm posting here now, in the hopes of finding someone who can answer my questions.

Currently there are three ways in which the application interacts with the website.


1) Download game covers from the website

Whenever the user selects an entry in the view table the application downloads the corresponding game covers from the website. The covers are stored locally so that they will be downloaded only the first time an entry is selected or when a manual refresh is triggered by the user.

The downloads are made by requesting

Code: [Select]

https://vgcollect.com/images/<front-box/back-box/cart>-art/<ID>.jpg



2) Download the backup CSV-File from the user profile

The user has the option to input their VGCollect login information to automatically download the backup CSV-File from their user profile. The application logs into the website and downloads

Code: [Select]

https://vgcollect.com/settings/export/collection
The CSV-File can also be provided manually so using the download function is not necessary.

The provided login information is only used for the immediate download and gets discarded after that.


3) Extracting item information from individual item pages

The user has the option to open a detailed item view where all available information of the selected item is displayed. Here the user can manually download additional information directly from the website. In general the application only works with the data provided by the CSV-File. Since the CSV-File doesn't provide a whole lot of information for individual items I provided a way to get more directly from the site.

For this feature the application downloads the HTML from

Code: [Select]

https://vgcollect.com/item/<ID>

and extracts the info-table. This data is cached locally and will only be downloaded when the user explicitly requests so.





These are the three ways in which the application interacts with the VGCollect.com website. In general I don't believe that any of those will have any measurable effect on the site as whole (especially since I doubt many will use it in the first place) but it's clear that it still generates some external traffic while not generating any potential ad revenue. So on principal I completely understand if any of those interactions are undesirable. I can also imagine that the collection download feature can be seen as a security risk since the user has to provide their login data. Of course the application would be open source so everyone could check how it works in that regard.

Depending on the wishes of the owner and staff I'd either remove any offending features or refrain from releasing it at all.

In the end to me this is a simple hobby project, so I have no greater ambitions and am perfectly fine to keep it private.

So in case anyone read this far, I'd be glad if someone could tell me whether it'd be alright to make the application publicly available.

Thanks

EDIT: Typos

[jason]

Thanks for reaching out! That looks pretty slick! Sending PM for further discussion.

P.S. Python is also a fun language to work with.