Vision AI step by step

By the end of this page you will have trained a first model that recognises an object of your choice on your phone screen, and you will have seen it working live. ⏱ 30 minutes of hands-on work, plus training time (a few minutes to over an hour depending on the machine).

What is a detection model?

A detection model is a small program that looks at an image and answers: "there is a play button here, a monster there". Nobody codes it by hand: it learns by example.

You show it dozens of images where you drew a rectangle and a name (the class) around the object yourself. Training means letting the computer find what those examples have in common. Then, given a new image, the model draws a box around what it recognises and gives a confidence percentage.

AppsMobs uses models from the YOLO family. You do not need to know them: the app walks you through 5 steps, shown at the top of the Vision AI screen.

How many images do you need?

Everyone asks this first. The honest answer is: it depends, and the numbers below are only indicative.

• The app suggests aiming for 50 to 100 examples per class for a first attempt.
• Variety matters more than quantity: different maps, times, zoom levels. Ten near-identical images are worth about one.
• Keep a few images with nothing to detect, so the model learns not to see the object everywhere.
• If the test (step 5) misses cases, add images of those exact cases and train again. You iterate; you do not guess the right number on the first go.

The 5 steps

1. 1

   Open Vision AI and pick your device

   Click Vision AI in the sidebar. You are on step 1 Capture, Live screen view.

   In the right-hand panel, open the System section and choose your phone from the device list. If the list is empty, click the refresh button next to it.

   123456

   You are done when your phone appears in the list.

2. 2

   Name your project and start the preview

   In Current Project, type a project name (for example my-game). One project = one game or one goal. Everything is stored in Documents\AppsMobs\VisionProjects.

   Click Start the preview (mark 3 above). The phone screen appears on the left, live. You can also press Alt + Shift + S to come back to this Live screen view.

3. 3

   Capture varied images

   Play normally on the phone, or set the screen to the situation you want. For each situation, click the 📷 button in the toolbar (it captures the screen to annotate later). The ⭐ button next to it starts automatic captures: you set the frequency and the duration.

   Change situation between captures: another map, another moment, another zoom. The counter on the 3 Images tab goes up with each image you keep.

4. 4

   Annotate: draw around the object (step 2)

   Click 2 Annotate. Create a class with + New Class and give it a short, clear name (play_button, monster). Select the rectangle tool (the first icon of the image toolbar), then draw a tight rectangle around the object.

   • A half-hidden object still gets a box, on its visible part.
   • Several objects in one image: one rectangle each.
   • A box that is too wide teaches the model that the background is part of the object.

   Save with Ctrl + S and move to the next image. The number next to 2 Annotate counts your annotations.

5. 5

   Check and sort your images (step 3)

   Click 3 Images (or See my images). Filter by class, or show only images without boxes. Open an image to fix its boxes, and move between images with ← →.

   Delete blurry images, duplicates and bad captures: a clean dataset beats a big one.

6. 6

   Prepare the dataset and train (step 4)

   Click 4 Train. The panel starts with Generate Dataset:

   1. Source Images: keep All project, or pick only the validated images.
   2. Split: three boxes Train / Valid / Test. Keep 70 / 20 / 10 (the total must be 100 %).
   3. Optional: the Pre-processing and Augmentation options (flip, rotation, noise, blur) create variants of your images.
   4. Click Generate Version. The app shows an Estimated number of resulting images.

   Then choose where to train (next section) and click Train. A progress bar follows the run. At the end, the model is a .pt file.

   

7. 7

   Test your model live (step 5)

   Click 5 Test. In Trained Models, choose your model (or Browse for a .pt file on your disk, for instance the one you brought back from Colab), then Load.

   Set the Confidence Threshold: the higher it is, the fewer false detections, but the more objects the model lets through. Start the preview: boxes with a percentage appear on the live screen.

   You are done when your object is boxed on screen. If not, go back to steps 1 and 2 to add images of the missed cases, then train again.

Colab or on your PC?

In step 4 Train, the Train YOLO section offers two routes.

And reinforcement?

Next to the 5 steps, the Advanced area holds Reinforcement and Agent settings. This is reinforcement learning: an agent learns by trying actions. It is slow (often hours) and reserved for the Ultimate plan. Always start with a detection model.

What is limited

It does not work?

All error messages are gathered in Troubleshooting.

What next

• Write a Python script to use what you taught the app to detect.
• Scheduling with Automation to run your script by itself.
• Troubleshooting if something blocks.