Custom Vision game help
Match the tiles in our version of the classic Rummy game. You will have to identify what the content of each tile’s image is.
Custom Vision (Rummy) is our extended version of the classic card game, Rummy. In our version the cards have pictures on them. To be able to play the game, you will have to be able to analyse the image to identify what it is an image of. You can do this using the A.I. power of Azure Cognitive Services.
The objective of the game is to score more points than your opponent. However the game ends, the person with the most points at the end of the game is the winner.
In this turn based game, each card in the deck may have one of four planets on it. The whole deck contains 40 cards: 10 cards of each landmark (the landmarks change every game). At the start of the game, each player is dealt 10 cards and the starting player is chosen randomly.
On each turn the player will have one new card in their hand from the deck until all cards in the deck have been used.
The player can look at their own cards and their opponents cards. The player makes a move by doing one of the following:
- 1.Lay down a set of cards. This may be 3, 4 or 5 cards of the same landmark. It may also be 3 or 4 cards of all different landmarks. This will score the player points as shown in the table below.
- 2.Take a card from the opponent's hand.
- 3.Give a card from your hand to the opponent.
Note: When Laying cards, if a player lays an invalid set of cards, the cards are returned to their hand and the play continues with the opponent's turn. In effect the player has wasted a turn.
The game ends when one of the players runs out of cards, or when 20 turns are taken and the deck runs out.
The players lose 3 points for every card they still have in their hand by the end of the game and the player with the highest score wins. If the players score the same number of points then the player who started second wins.
These are the valid hands that you can play with the points they earn:
Action | Reward |
Take a card from your opponent's hand | 0 points |
Give a card to your opponent | 0 points |
Set of three different landmarks | 5 points |
Set of four different landmarks | 10 points |
Set of three of the same landmark | 20 points |
Set of four of the same landmark | 40 points |
Set of five of the same landmark | 80 points |
Penalty for cards held at the end of the game | -3 points per card |
In order to analyse an image and identify what the subject of the image is, you will need to create an AI image recogniser or image classifier. You can use Microsoft's Custom Vision service to create this.
Every card in the deck is unique. Your hand might contain two cards showing Jupiter, but they will be different images fo Jupiter. They might be taken from a different angle. For this reason you can't compare images pixel by pixel for a match. We have to understand what each one is showing.
In order to sign in to and create a Custom Vision service, you need to have a Microsoft Azure account. If you are a student, you can sign up for a student account on Azure that gives you $100 of credit.
Once you have a Microsoft Azure Account, you will need to sign in to Microsoft's Custom Vision site to create and train a planets image classifier.
Follow these steps, to create your own planets image classifier
Navigate to the Custom Vision Portal and sign in with your Microsoft Azure account details, the same account details you use to login to the Azure portal.
Select the 'New Project', which will open the following dialogue:
Fill in the dialogue as follows:
- Name - enter your project name e.g. 'Planets'.
- Description - add a sentence to summarise the project.
- Resource - click on 'create new' in order to create a resource.
- In the Create New Resoource dialog that appears
- Give it a descriptive name
- You will have subscriptions like 'Free Trial' or 'Azure For Student' if you have signed up for a student account
- Select one of you existing resource groups or create a new one. Resource groups just group resources together for easy reporting or management.
- Leave the Kind as CognitiveServices (which sets it to create both training and predictions resources for you)
- For location, make sure that you set its location to West Europe. This is important as the AI Gamin servers are located in West Europe.
- Select S0 as the pricing tier. S0 allows more transaction per second needed to play the game.
- Make sure that if you are creating a new group you set its location to West Europe. Do the same for the location of the resource. Make sure to leave the 'Kind' as 'CognitiveServices', since this will allow you to use this resource for both training and prediction:
After creating the resource you will return to the Create Project dialog
- Project Types - choose 'Classification', since we want to categorise whole images in this project.
- Classification Types - choose 'Multiclass', since our pictures will contain only one subject.
- Domain - you may leave this as 'General'.
Clicking 'Create project' button sets the project up for you and takes you to the project page. This is where you can upload training images and label them to train your image classifier to recognise planets.
To add training images click on 'Add images' and select the first batch of images of the same type. Here is an example doing this with Jupiter:
Type in the tag for your images and press 'Enter' to label them. Then click on 'Upload' to add them to the training library. Repeat this process for as many images and as many categories as necessary.
Once you are done uploading and labelling all images, you can click the 'Train' button to train your model. There are two training options Quick and Advanced. Select the 'Quick training' option and then the 'Train' button. It will take some time to train your model before taking you to the performance page and showing you the results for this new training iteration. You may go back, add more images and train your model again to produce new iterations until you are happy with the result.
To use this model in in your code, you will need to select the iteration that you would like to use and click 'Publish' in the top left corner. You will be presented with the following dialogue:
Give your model a suitable name and select the same resource you have created when creating this model. Click 'Publish'.
Now you will be able to click on the 'Prediction URL' button to get all the necessary information in order to use this model in your code:
Since in this game the card pictures will be presented as URLs, you will need to look at the top section: 'If you have an image URL'. Copy the prediction URL in the grey box and the prediction key.
When you first select the Rummy Vision game type in the online code editor, the code will be a template that plays a very simple game by submitting random moves. This version of the code does not require any Custom Vision prediction urls or keys.
To play the Custom Vision version of Rummy Vision, you should select a Game Style that includes the Planets card type.
Then, from the New button dropdown, you can load template code that can use a Custom Vision model to play the game.
The calculate_move() function is the equivalent of your main function. This is where you need to make your changes and from where you will control the game.
- Called each time you need to submit a move to the game.
- Receives information about the game in a Python dictionary that describes the current game state.
- Must return a game move.
Your calculate_move() function will be passed the current state of the game, the 'gamestate'.
The gamestate is where all of the game information is held. It is a Python dictionary. The following shows an example of the gamestate information that you will receive for each move of the game and some examples of how to access data within it.
For Rummy Vision, the most important fields are MyHand, and OppHand.
Fields that you are unlikely to need in this game are ResponseDeadline, GameStatus, IsMover, GameId and OpponentId.
{
'MyHand': [
[0, 'https://matchgameimages.blob.core.windows.net/rummy-vision/f3ce6bfa-b73e-4de7-9a81-d56806e96b7a/a2b9232f-402f-460a-8cca-d4898aed3baa.jpg'],
[2, 'https://matchgameimages.blob.core.windows.net/rummy-vision/f3ce6bfa-b73e-4de7-9a81-d56806e96b7a/435e4c3b-080a-4259-a19a-1c2ca6364320.jpg'],
[5, 'https://matchgameimages.blob.core.windows.net/rummy-vision/f3ce6bfa-b73e-4de7-9a81-d56806e96b7a/f15d2a20-daed-47fd-a9c2-f4e4dcc846f7.jpg'],
[6, 'https://matchgameimages.blob.core.windows.net/rummy-vision/f3ce6bfa-b73e-4de7-9a81-d56806e96b7a/c23e00c3-09d2-488a-ab10-584be90b77d9.jpg'],
[3, 'https://matchgameimages.blob.core.windows.net/rummy-vision/f3ce6bfa-b73e-4de7-9a81-d56806e96b7a/0fd0fb03-312c-4739-9f15-305e657ff369.jpg'],
[11, 'https://matchgameimages.blob.core.windows.net/rummy-vision/f3ce6bfa-b73e-4de7-9a81-d56806e96b7a/7e57ddc8-2736-4544-8d64-afe6e4e70d2d.jpg'],
[9, 'https://matchgameimages.blob.core.windows.net/rummy-vision/f3ce6bfa-b73e-4de7-9a81-d56806e96b7a/641953c5-c7ca-4d85-b676-6b3c9ce420b2.jpg'],
[1, 'https://matchgameimages.blob.core.windows.net/rummy-vision/f3ce6bfa-b73e-4de7-9a81-d56806e96b7a/a75eefed-1bb1-458c-bd5f-80c3f5b888c8.jpg'],
[13, 'https://matchgameimages.blob.core.windows.net/rummy-vision/f3ce6bfa-b73e-4de7-9a81-d56806e96b7a/c50fc797-9a4d-46be-8d78-32cff23996de.jpg'],
[12, 'https://matchgameimages.blob.core.windows.net/rummy-vision/f3ce6bfa-b73e-4de7-9a81-d56806e96b7a/d777e840-c9ea-44cb-ad92-e85597f2f423.jpg']
],
'OppHand': [
[14, 'https://matchgameimages.blob.core.windows.net/rummy-vision/f3ce6bfa-b73e-4de7-9a81-d56806e96b7a/91b905b9-3554-4ef9-881b-5ddb52d47386.jpg'],
[16, 'https://matchgameimages.blob.core.windows.net/rummy-vision/f3ce6bfa-b73e-4de7-9a81-d56806e96b7a/77756f5a-ad9e-4ada-8293-a68616a67713.jpg'],
[10, 'https://matchgameimages.blob.core.windows.net/rummy-vision/f3ce6bfa-b73e-4de7-9a81-d56806e96b7a/f4569040-ddd6-4af7-8317-4d150ff590c6.jpg'],
[8, 'https://matchgameimages.blob.core.windows.net/rummy-vision/f3ce6bfa-b73e-4de7-9a81-d56806e96b7a/28e9431a-2b0f-42b7-99ec-405a978d94cb.jpg'],
[7, 'https://matchgameimages.blob.core.windows.net/rummy-vision/f3ce6bfa-b73e-4de7-9a81-d56806e96b7a/bc0d5996-6c80-4acf-9659-9adbd1f707a8.jpg'],
[15, 'https://matchgameimages.blob.core.windows.net/rummy-vision/f3ce6bfa-b73e-4de7-9a81-d56806e96b7a/67287c45-4965-43c3-8b8a-f32baf8a13ad.jpg'],
[24, 'https://matchgameimages.blob.core.windows.net/rummy-vision/f3ce6bfa-b73e-4de7-9a81-d56806e96b7a/fbf01213-13db-4d87-8321-7754f718693d.jpg'],
[23, 'https://matchgameimages.blob.core.windows.net/rummy-vision/f3ce6bfa-b73e-4de7-9a81-d56806e96b7a/b74889e0-e1ba-4012-8179-b46944ea769c.jpg'],
[20, 'https://matchgameimages.blob.core.windows.net/rummy-vision/f3ce6bfa-b73e-4de7-9a81-d56806e96b7a/c6295bf6-768e-4397-aa11-d323b8a271b0.jpg'],
[18, 'https://matchgameimages.blob.core.windows.net/rummy-vision/f3ce6bfa-b73e-4de7-9a81-d56806e96b7a/224252d1-025b-40bd-9946-df3460334d9a.jpg']
],
'MyPoints': 0,
'OppPoints': 0,
'MyScore': 0,
'OppScore': 0,
'ScoreLimit': 1,
'RemainingMoves': 20,
'IsMover': True,
'ResponseDeadline': 1612537323056,
'CardsInDeck': 20,
'GameStatus': 'RUNNING',
'AllCardTypes': ['jupiter', 'mercury', 'pluto', 'venus'],
'InvalidMove': None,
'OpponentId': 'housebot-practise'
}
Examples of accessing data in the gamestate would be:
gamestate["MyHand"][0]
gamestate["OppScore"]
The following list gives a description of what each element in the gameState represents:
- AllCardTypes - A list of all the possible card types this game style might present you with.
- You can use this list to make sure your image recognition is able to deal with all possible cards.
- Keep in mind that some game styles may not use all of these types.
- MyHand - A list representing the cards you hold in your hand. Each card is represented by a two element list consisting of its ID and image URL.
- To determine how many cards you are holding you could use
len(gamestate["MyHand"]). - An example of accessing the ID of your first card is
gamestate["MyHand"][0][0]. - An example of accessing the image URL of your first card is
gamestate["MyHand"][0][1].
- OppHand - A list representing the cards in your opponent's hand. Each card is represented by a two element list consisting of its ID and image URL.
- To determine how many cards your opponent is holding you could
len(gamestate["OppHand"]). - An example of determining the first card of your opponent is
gamestate["OppHand"][0].
- MyPoints - You current points score for this game.
- OppPoints - Your opponent's current points score for this game.
- RemainingRounds - The number of rounds of Rummy you have remaining with your opponent.
- RemainingMoves - The number of moves you have left until the end of the game
- You can use this value to determine how many turns you have left for your plays.
- CardsInDeck - The number of cards not yet drawn from the deck.
- ResponseDeadline - The epoch time, in milliseconds, that a successful move has to be sent and received by to prevent you from timing out.
- There is a time limit to how long you have to calculate your move. If you exceed this time limit your game will be terminated and your opponent will be awarded as the winner.
- It is unlikely that you will need to check this time as timeouts are set generously to allow you time to calculate your move, however, if you see yourself timing out a lot, you may need to limit yourself using this value.
- GameStatus - A string that will have value "RUNNING" if the game is in progress or a reason the game has ended otherwise.
- You are unlikely to need the GameStatus for this game type.
- IsMover - In this turn based game, this will always be true.
- GameId - An integer representing the unique game id for the current game.
- You are unlikely to need the GameId for this game type.
- OpponentId - A string containing the name of your opponent.
- You are unlikely to need the OpponentId for this game type.
The whole point of the calculate_move() function is for you to return the move you want to make in the game. In Rummy Vision, there are three types of moves.
- 1.You may give your opponent a card in your hand by returning a dictionary with the key "Give" whose value is the ID of the card you want to give.
- 2.You may take a card from your opponent's hand by returning a dictionary with the key "Take" whose value is the ID of the card you want to take.
- 3.You may lay a set of cards on the board to gain points by returning a dictionary with the key "Lay" whose value is the list of card IDs you want to lay down. This list may can contain 3, 4 or 5 cards of the same type. It may also contain 3 or 4 cards of unique types.
An example of each move:
return {"Give": 2}
return {"Take": 10}
return {"Lay": [5, 6, 14, 1]}
If in the Editor, you select the Game Type of Python Editor, there are some code templates. One of these gives a very simple code example of calling a Custom Vision model's prediction URL so that you can see a simple example of how it works without having to work your way through lots of other code that plays our Rummy Vision game.
After selecting the Python Editor Game Type, create a new file and select the Custom Vision Prediction API call Example.py template.
Once you load the template, you can copy the Prediction URL and the Prediction KEY from you Custom Vision model and paste them into the variables at the top of code. Once you've done this you'll be able to run the code and it will use your Custom Vision model to try to identify planets in some images from the web.
If you look at the template code, you will see there is a very simple loop that reads some image URLs from a list and then sends them to the Custom Vision prediction URL to analyse them using the requests.post function.
response = requests.post(prediction_url, {}, headers=headers, json=data).json()
The response is then read to see what the Custom Vision Model though was in the image.
This simple piece of code is designed to let you see a very basic and minimal example of how easy it is to use your Custom Vision model in Python Code.