Skip navigation
Welcome, Guest! Please Login or Join

Loading...

Hunting rare games with artificial intelligence We propose an automated way of finding NFR games

Apr 15 at 7:33:26 AM
notforresale (0)
avatar
< Cherub >
Posts: 15 - Joined: 04/15/2019
Profile
Edit: Since people have recognized this can be used for all sorts of other things within the hobby (variant detection, special releases, etc.) I've made a Discord where we can collaborate in gathering training data for the NN.
Click here to join: 
https://discord.gg/sFAsG5j...

Dear NFR collectors.

For many months now I've been pondering whether or not it is possible to distinguish a regular cartridge from a Not For Resale edition.
My hypothesis was that a lot of "regular people" (non-collectors), list their games for sale on eBay, without realizing they have a Not For Resale edition. I, myself has made (at current market price) roughly 5400 USD in the span of 4 weeks (if I were to resale these games), and the people that I have shared this with have also had great luck using this.

But until now there's been no way to find those rare gems without browsing through 15.000+ listings on eBay, Craigslist, Kajiji or whatever site you prefer.
Today we propose a solution for this problem. I assembled a training set of 300 different pictures of different NFR titles for Nintendo 64. Different games have different NFR boxes (e.g. Legend of Zelda: OT has a tiny square) while Perfect Dark has a rectangle. Therefore we needed a diverse dataset that included all variants of the different types.

Before going on with this thread, I'd like to acknowledge the work of aero for helping make this project possible, he's an expert in general adversarial networks, and coming up with unique ideas on data formatting. Without him this project would not be possible.

Perfect Dark NFR Label:
Perfect Dark - Not For Resale

Legend of Zelda: Ocarina of Time
Ocarina of Time - Not for Resale

Before training a neural network model on the data, we had to augment the data, this essentially means transforming the image in various ways including: flipping it, rotating it, resizing it (typically to a lesser resolution), adding noise, etc. to make sure that the network is agnostic of the surrounding conditions, and properties of the photos (you never know what you'll encounter on eBay). Based on the length, shape and position of the label, we can also make educated guesses about what game it has found.

The dataset was on a server-grade (meant to be run in a GPU rack, so it has no cooling and sounds like a jetplane) Nvidia Tesla K80 graphics card that aero bought (I firmly believe he has an intimate relationship with the thing  ). Enough with the explanations, let's get to the visuals. These were some of the early pictures I took during training at epoch 12/100 (meaning it wasn't even close to the fully trained network).

The way the network learns is by manually drawing a mask (polygon) around a cartridge to make it learn what a cartridge looks like, aswell as a the various Not For Research labels. After many hours of training it recognize these features even on cartridges it has never seen before, in any lighting conditions/angles/etc. The image below explains what is fed to the network. "Ground truth" means the mask that was manually drawn by a person, the output is the networks guess as to where the features are located (the network does not know what the image to the right looks like, it's purely rendering the mask based on feature recognition that it sees on the image to the left.)



I do not have images of the later stages of the training process, but as you can see it can detect non-standard NFR labels at weird angles. The fully trained network can perfectly mask out the cart + label.



Heres DK64 - a pretty rare cart.



Here's another image where it recognizes the NFR logo on a Super Mario 64 label. Keep in mind, it has to determine that the bedsheets (?) are noise, aswell as ignore the very similar looking Nintendo logo.


Here is an example of a regular 007 cartridge, that has no NFR label. As you can see, it correctly determines that this is in fact not a NFR cart.



This is how the network did at epoch 20/100.
If there's any interest in this I'll elaborate and perhaps me & aero can hook it up to a web interface and shre it with some avid demo game collectors.
The goal of this project was to get rare games out of the hands of people who may not take care of them as well as collectors do and ruin history. Oh, and also the financial aspect. It's essentially a nice little bonus ontop of my actual job.

This algorithm can be made to work with every "Not For Resale" game including DS, 3DS, Gameboy & SNES games aswell.

Hope this was an interesting read.

- nfr


Edited: 04/16/2019 at 05:11 PM by notforresale

Apr 15 at 8:38:37 AM
SNESNESCUBE64 (38)
avatar
(Orange Yoshi) < El Ripper >
Posts: 1356 - Joined: 01/16/2016
Michigan
Profile
Really cool read. I got to learn about how neural networks for a class I had to take last semester, really interesting stuff. It's neat to see the captured outputs. I guess my biggest question is how do you pull data from the listings themselves? Or is that something you have to do manually?

Also, welcome to the forums!

-------------------------
Y-akuza
O-perations
S-ecure
H-is
I-ncome


Edited: 04/15/2019 at 08:39 AM by SNESNESCUBE64

Apr 15 at 9:05:12 AM
notforresale (0)
avatar
< Cherub >
Posts: 15 - Joined: 04/15/2019
Profile
Originally posted by: SNESNESCUBE64

Really cool read. I got to learn about how neural networks for a class I had to take last semester, really interesting stuff. It's neat to see the captured outputs. I guess my biggest question is how do you pull data from the listings themselves? Or is that something you have to do manually?

Also, welcome to the forums!

Hey there. Thanks for the positive comment.  

Just some extra info - the training set is comprised of two parts. Here's what they contain:
The first dataset has roughly 300 images of Not For Resale carts taken from various angles, with various resolutions, these are the types of images you get when you fetch them off eBay, and that's exactly the type of data we're looking for. Real world photos as they would appear in an eBay listing.

The second dataset has a few more images ~650 consisting of regular NTSC carts (they're simply easier to find images of).
To verify that the network is able to distinguish between NFR and non-NFR carts we mix the datasets and measure the amount of correct guesses. I'm happy to say that we're currently sitting on a 100% success rate after the network was done training.

Now, to answer your question: I've written a crawler that has a list of all known NFR titles for the Nintendo 64. It fetches all images, from all listings, on eBay and alerts me if a listing pops up that the neural network with some certainty has determined to be an NFR cartridge.  It also sorts by "newest listings" and alerts me via SMS if I'm not at the computer at the moment (I have this running on a Raspberry Pi with a GSM module + SIM card installed).

- nfr
 


Edited: 04/15/2019 at 12:36 PM by notforresale

Apr 15 at 11:42:56 AM
SUBSCRIBER
coffeewithmrsaturn (355)
avatar
(Chris ) < Bowser >
Posts: 5800 - Joined: 08/16/2012
California
Profile
It's fun to see something that feels almost futuristic (a la Altered Carbon or Black Mirror) applied to something as niche as finding NFR games, haha  

-------------------------
Have any GBA NFR games with back stickers for sale or trade?  See my want list below :)
My WTB: http://nintendoage.com/forum/mess...

"It is difficult to get a man to understand something when his salary depends upon his not understanding it."  --Upton Sinclair
 

Apr 15 at 12:14:34 PM
notforresale (0)
avatar
< Cherub >
Posts: 15 - Joined: 04/15/2019
Profile
Originally posted by: coffeewithmrsaturn

It's fun to see something that feels almost futuristic (a la Altered Carbon or Black Mirror) applied to something as niche as finding NFR games, haha  

Haha, yeah - but I couldn't think of a better way to solve this moderately hard problem.
Retro Nintendo games (amongst other items I know are valueable) is my fun little arbitrage Wall Street-type game. I have a piece of software monitoring the market and general price trends.

If there's interest I could share it with a select few collectors that have proven to be in it for the actual collection part and not the money - ironic, I realize. :-)

- nfr


Edited: 04/15/2019 at 12:36 PM by notforresale

Apr 15 at 12:19:14 PM
16BitBricks (1)
avatar
(Josh ) < Crack Trooper >
Posts: 119 - Joined: 03/27/2016
Michigan
Profile
Brilliant idea, that's awesome! I'm a little worried that you may have just started Skynet. It's only a matter of time before it becomes self aware.

Apr 15 at 12:26:06 PM
notforresale (0)
avatar
< Cherub >
Posts: 15 - Joined: 04/15/2019
Profile
Originally posted by: 16BitBricks

Brilliant idea, that's awesome! I'm a little worried that you may have just started Skynet. It's only a matter of time before it becomes self aware.


Every morning I wake up, I fear I may accidentally invent Skynet.  
I wish I could share more details about my personal life and the computer science work I do there, this was done over 2.5 days with 2 people working on it - but then again, I doubt any of you are excited about data as I am.

This technology can be expanded to any game, and using some formulas, you can automatically determine if an item has been mispriced. If you have an item you'd see tracked, give shoot me a PM and make me aware in this thread (I tend to miss the red alert telling me I have a message).

- nfr


Edited: 04/15/2019 at 12:36 PM by notforresale

Apr 15 at 12:46:56 PM
GPX (1)

< Eggplant Wizard >
Posts: 382 - Joined: 05/17/2017
Profile
Pretty interesting read. I know nothing about software programming and don't collect for NFR carts, but I can see some interesting elements that can be useful for other fields of collecting. Is the software ready for use yet?

Apr 15 at 12:52:44 PM
notforresale (0)
avatar
< Cherub >
Posts: 15 - Joined: 04/15/2019
Profile
Originally posted by: GPX

Pretty interesting read. I know nothing about software programming and don't collect for NFR carts, but I can see some interesting elements that can be useful for other fields of collecting. Is the software ready for use yet?


Hello.

Yep. It can indeed - it can be used to spot misprints, rare editions, and any other feature that may be valueable, without the seller knowing.
If you have any suggestions, I'm currently in the process of making a web portal that is hooked up to this software, so a small sample of collectors can use it to get deals.


- nfr


Edited: 04/15/2019 at 08:09 PM by notforresale

Apr 15 at 12:59:42 PM
PowerPlayers (87)
avatar
(The User Formerly Known As Joseph Leo) < Bowser >
Posts: 7108 - Joined: 11/06/2011
New Jersey
Profile
This technology is *REALLY* nice and I can think of a few applications for it beyond NFR hunting.

Variant hunting would be nice.

Also, it would be nice to apply it to price chart applications to determine specific variants, specific condition, specific completion (CIB/Loose/missing pieces)

It would take the inhumane task of trying to document every possible variant of every game and automate it to the point of just having to verify things outside of the spec.

Really cool tech, I'd love to see this idea grow.

-------------------------

Got any of these for sale? Sell them to me. I also buy other NES Publisher inserts, and even GB/GBC, and SNES inserts too.

Apr 15 at 1:14:43 PM
SUBSCRIBER
Krunch (145)
avatar
(My name's Krunch) < King Solomon >
Posts: 3860 - Joined: 11/08/2010
Alberta
Profile
This is crazy! but not like randy quaid crazy, more like da Vinci crazy. So many other uses too. So cool

-------------------------
eBay - http://www.ebay.ca/sch/i.html?_nk...

#YKNOTK

Apr 15 at 1:20:21 PM
notforresale (0)
avatar
< Cherub >
Posts: 15 - Joined: 04/15/2019
Profile
Originally posted by: Krunch

This is crazy! but not like randy quaid crazy, more like da Vinci crazy. So many other uses too. So cool


"So many other uses too", that's what people has been telling me. I decided to create a Discord. The admins of this forum is welcome to join and gain administrator privileges I have no desire to lead this thing, I just wanna write code.
If you want to join here's a link: https://discord.gg/Ncs33mQ...


We can find variants all sorts of other rarities.

Apr 15 at 1:23:44 PM
SUBSCRIBER
Krunch (145)
avatar
(My name's Krunch) < King Solomon >
Posts: 3860 - Joined: 11/08/2010
Alberta
Profile
Originally posted by: notforresale
 
Originally posted by: Krunch

This is crazy! but not like randy quaid crazy, more like da Vinci crazy. So many other uses too. So cool


"So many other uses too", that's what people has been telling me. I decided to create a Discord. The admins of this forum is welcome to join and gain administrator privileges I have no desire to lead this thing, I just wanna write code.
If you want to join here's a link: https://discord.gg/Ncs33mQ


We can find variants all sorts of other rarities.

Yeah when I’m on the computer later I’ll jump in just cause I’m curious.  Right away my first thought was N64 box variants such as Mischief Makers, MM non-CE, Fighting Force, Toy Story 2, Gaunlet Legends.... list goes on and on and on
 

-------------------------
eBay - http://www.ebay.ca/sch/i.html?_nk...

#YKNOTK

Apr 15 at 1:25:44 PM
Abelardo (76)
avatar
(Abelardo González) < El Ripper >
Posts: 1447 - Joined: 10/26/2011
Mexico
Profile
This is very neat! Makes me want to learn about neural networks and find something cool to apply it.

Apr 15 at 1:30:05 PM
notforresale (0)
avatar
< Cherub >
Posts: 15 - Joined: 04/15/2019
Profile
Originally posted by: PowerPlayers

This technology is *REALLY* nice and I can think of a few applications for it beyond NFR hunting.

Variant hunting would be nice.

Also, it would be nice to apply it to price chart applications to determine specific variants, specific condition, specific completion (CIB/Loose/missing pieces)

It would take the inhumane task of trying to document every possible variant of every game and automate it to the point of just having to verify things outside of the spec.

Really cool tech, I'd love to see this idea grow.

This is a really good idea. However, it needs to be a collaborative effort, because in order to train something like this to recognize variants there's a few things I can think of that needs to be done:
  1. Gather images of every Nintendo 64 cartridge (or other console) in existance.
  2. Avoid images that has clear defects, such as sharpies, ripped labels, etc.
  3. Label each image NTSC or PAL
There's a ton of possibilities. Feel free to come up with suggestions.
 

Apr 15 at 1:56:44 PM
hoisinberg (0)

< Crack Trooper >
Posts: 103 - Joined: 04/01/2018
Profile
great for collectors trying to find unlisted variants

bad for collectors when the resellers with access to ebay APIs and a jump on buyitnow listings before everything else are able to insta-buy flippable items before anyone sees them hit the live server.

Apr 15 at 2:43:27 PM
SUBSCRIBER
Every1whocountsluvsNedFlanders (69)
avatar
< Wiz's Mom >
Posts: 11214 - Joined: 10/20/2014
Ontario
Profile
That's some next level shit man.

-------------------------
Get that hitter.

Apr 15 at 2:44:40 PM
notforresale (0)
avatar
< Cherub >
Posts: 15 - Joined: 04/15/2019
Profile
Originally posted by: hoisinberg

great for collectors trying to find unlisted variants

bad for collectors when the resellers with access to ebay APIs and a jump on buyitnow listings before everything else are able to insta-buy flippable items before anyone sees them hit the live server.

Oh don't worry. This technology is far out of their reach. If they want to comb through 15.000 listings for "nintendo 64 game" then be my guest. This tool was intended for collectors only.  

- nfr
 

Apr 15 at 3:04:57 PM
Abelardo (76)
avatar
(Abelardo González) < El Ripper >
Posts: 1447 - Joined: 10/26/2011
Mexico
Profile
Originally posted by: notforresale
 
Originally posted by: hoisinberg

great for collectors trying to find unlisted variants

bad for collectors when the resellers with access to ebay APIs and a jump on buyitnow listings before everything else are able to insta-buy flippable items before anyone sees them hit the live server.

Oh don't worry. This technology is far out of their reach. If they want to comb through 15.000 listings for "nintendo 64 game" then be my guest. This tool was intended for collectors only.  

- nfr
 


I feel that now that you've done it the cat is out of the bag, my guess is that there must be resellers capable of implemeting something like this, I would guess that a few already have some sort of IA to scout for deals, next step would be to implement something like this on top of that.  Of course I don't really know what it takes to implement something like this in an efficient enough manner, but resellers that do this for a living will surely try to find a way to do it.

Apr 15 at 3:31:22 PM
notforresale (0)
avatar
< Cherub >
Posts: 15 - Joined: 04/15/2019
Profile
Originally posted by: Abelardo
I feel that now that you've done it the cat is out of the bag, my guess is that there must be resellers capable of implemeting something like this, I would guess that a few already have some sort of IA to scout for deals, next step would be to implement something like this on top of that.  Of course I don't really know what it takes to implement something like this in an efficient enough manner, but resellers that do this for a living will surely try to find a way to do it.
If they knew how to invent this tech, they would've done it long ago. Think about how much the Nintendo logo looks like the NFR logo. Now think about how similar these 2 look if we're dealing with a 600x600 image of a cartridge that's far away.

 

Apr 15 at 3:52:01 PM
Abelardo (76)
avatar
(Abelardo González) < El Ripper >
Posts: 1447 - Joined: 10/26/2011
Mexico
Profile
Originally posted by: notforresale

If they knew how to invent this tech, they would've done it long ago. Think about how much the Nintendo logo looks like the NFR logo. Now think about how similar these 2 look if we're dealing with a 600x600 image of a cartridge that's far away.
 
Looks like it's no easy task for sure, what are the chances of someone implementing a watered-down version of this? Use some pre-existing deep learning framework, have it look at full resolution images and looking only at new listings as they become live, is this somenthing that could be achieved? Of course if this can be achieved it won't be as good as yours but still it would be profitable.

 

Apr 15 at 4:13:17 PM
notforresale (0)
avatar
< Cherub >
Posts: 15 - Joined: 04/15/2019
Profile
Originally posted by: Abelardo
 
Originally posted by: notforresale

If they knew how to invent this tech, they would've done it long ago. Think about how much the Nintendo logo looks like the NFR logo. Now think about how similar these 2 look if we're dealing with a 600x600 image of a cartridge that's far away.
 
Looks like it's no easy task for sure, what are the chances of someone implementing a watered-down version of this? Use some pre-existing deep learning framework, have it look at full resolution images and looking only at new listings as they become live, is this somenthing that could be achieved? Of course if this can be achieved it won't be as good as yours but still it would be profitable.

 

No I attempted a standard CNN that used the fullsize listing images, and it didn't detect (at least the way I modeled the data) the box accurately. It had a false positive for almost every image. Imagine scanning a bundled lot - that's gonna contain a lot of noise. Unless you have a large dataset, and a good way to structure the data, this isn't possible. You have however convinced me not to release the source code.  

Apr 15 at 4:52:40 PM
Abelardo (76)
avatar
(Abelardo González) < El Ripper >
Posts: 1447 - Joined: 10/26/2011
Mexico
Profile
Originally posted by: notforresale

No I attempted a standard CNN that used the fullsize listing images, and it didn't detect (at least the way I modeled the data) the box accurately. It had a false positive for almost every image. Imagine scanning a bundled lot - that's gonna contain a lot of noise. Unless you have a large dataset, and a good way to structure the data, this isn't possible. You have however convinced me not to release the source code.  

Sorry about that  

Ok so it's not really easily achievable by the tools available at the moment, maybe later it would be, who knows. I love game changing (or game breaking) software like this, personally I've developed a couple of tools over the years to give me an advantage, nothing fancy at all, and certainly not AI.

Since I am not really familiar with the technology I'm just wondering about it, is looking for the NFR legend the simplest approach? How about instead of looking for the NFR legend, how about you first identify which game it is, once that is determined then compare with a NFR label of said game, maybe just compare the region where the NFR legend is expected, I wonder if that approach would be possible.

Apr 15 at 4:53:49 PM
SUBSCRIBER
Gloves (104)
avatar
(Douglas Glover) < Master Higgins >
Posts: 8757 - Joined: 01/21/2017
Ontario
Profile
Really really cool stuff, very impressive!

-------------------------


Apr 15 at 5:40:21 PM
notforresale (0)
avatar
< Cherub >
Posts: 15 - Joined: 04/15/2019
Profile
Originally posted by: Abelardo
 
Originally posted by: notforresale

No I attempted a standard CNN that used the fullsize listing images, and it didn't detect (at least the way I modeled the data) the box accurately. It had a false positive for almost every image. Imagine scanning a bundled lot - that's gonna contain a lot of noise. Unless you have a large dataset, and a good way to structure the data, this isn't possible. You have however convinced me not to release the source code.  

Sorry about that  

Ok so it's not really easily achievable by the tools available at the moment, maybe later it would be, who knows. I love game changing (or game breaking) software like this, personally I've developed a couple of tools over the years to give me an advantage, nothing fancy at all, and certainly not AI.

Since I am not really familiar with the technology I'm just wondering about it, is looking for the NFR legend the simplest approach? How about instead of looking for the NFR legend, how about you first identify which game it is, once that is determined then compare with a NFR label of said game, maybe just compare the region where the NFR legend is expected, I wonder if that approach would be possible.


The thing is, there's only roughly ~25 (maybe a little more) N64 NFR carts, so there's no need to identify the game. You simply search for the title on eBay and they pop up.
The approach you suggested does not account for weird angles, torn labels, different ligting, etc. and like it would produce a ton of false-positives. This network is designed to look for features in the image, such as the shape of a cartridge (from multiple angles), aswell as the NFR label.

You could tilt the cartrige so the label is insanely stretched, but because it looks for features, and knows what a cartridge looks like, aswell as an NFR label from different angles, it'll have no trouble labeling it. This is a general solution to the problem of different camera quirks. However the suggestion you proposed has been tried before, and does (to some extent) work somtimes if implemented correctly.  

- nfr