The tool takes a screenshot of a Magic Sort puzzle and attempts to convert the image into a structured game state. The current pipeline uses YOLO-based detection to identify bottles, board positions and special objects. After detection, the system needs to identify the color of each puzzle segment and arrange everything in the correct order before the solving algorithm can calculate the moves.
The main architectural question I’m exploring is how much responsibility YOLO should have.
One approach is to train YOLO with separate classes for every object and color combination. However, this could create many classes and may become unreliable when screenshots have different themes, brightness levels, compression artifacts or screen resolutions.
Another approach is to use YOLO only for locating bottles and segments, then pass each detected crop to a separate color-classification process. That second stage could use HSV or LAB color values, clustering or a lightweight image-classification model.
A simplified pipeline would be:
Detect bottles, segments and special objects with YOLO.
Crop each detected puzzle segment.
Classify its color separately.
Reconstruct the complete board layout.
Send the structured board to the puzzle-solving algorithm.
Which approach would be more reliable for this type of project?
One YOLO model with object and color classes
YOLO detection plus a separate color classifier
YOLO detection plus traditional HSV/LAB color matching
A hybrid approach with confidence thresholds and manual correction
I’m also considering how to evaluate the complete system. Standard detection metrics such as precision, recall and mAP may not be enough because one incorrect color can produce an invalid puzzle state. Would exact board reconstruction accuracy be a better end-to-end metric?
I’d appreciate suggestions from anyone who has worked with YOLO, screenshot analysis or multi-stage computer-vision pipelines.
I’d lean toward the hybrid approach. YOLO is very good at localization, but making it responsible for every color/object combination can quickly increase the number of classes and make the model more sensitive to visual variations.
Using YOLO to detect and crop the segments, followed by HSV/LAB-based color classification, would probably be a good starting point. You could then fall back to a lightweight classifier when the color confidence is low and allow manual correction for edge cases.
For evaluation, I agree that exact board reconstruction accuracy is probably the most meaningful end-to-end metric, while mAP/precision/recall can still help identify where individual stages are failing.
I’d approach this as a two-stage pipeline rather than trying to make YOLO handle the color classification itself. First use YOLO to detect and localize the puzzle objects—such as bottles/tubes and their positions—and then crop each detected tube and run a separate color/segment classifier on the cropped region. This keeps the responsibilities clear: YOLO answers “where is the object?”, while the color model answers “what colors are inside it?”
For a puzzle solver, I’d also avoid classifying the entire tube as a single color because a tube can contain multiple stacked segments. After detecting the tube, divide its interior into expected segment regions based on the tube geometry and sample/classify each segment separately. You can use HSV or LAB color space for relatively clean screenshots, but a small CNN or pretrained vision model would be more robust when there are gradients, shadows, transparency, or special effects.
The important part is converting the visual result into a reliable game-state representation. For example, instead of storing only tube_1 = red, store something like tube_1 = [red, blue, blue, empty], along with tube position, capacity, and special-object information. Then pass that structured state to the puzzle-solving algorithm.
I’d also include a confidence score at each stage. If YOLO is confident about the tube but the color classifier is uncertain, don’t immediately generate a solution. Let the user manually correct that particular segment. This human-in-the-loop approach can be much more reliable than trying to make the entire pipeline 100% automatic.
For a screenshot-based solver, the overall architecture could be:
That separation also makes the system much easier to improve later because you can replace the color-classification component without rebuilding the object-detection or solving logic.
I’d keep the detection and color recognition as separate stages. YOLO can identify and locate each tube, while a lightweight color classifier can analyze the individual segments inside each detected tube. This makes the pipeline easier to debug and improve, especially when screenshots contain multiple colors or special objects.
For a puzzle solver, I’d also add a confidence check and allow manual correction when the model is uncertain. The final goal isn’t just detecting objects accurately, but converting the screenshot into a reliable game state that the solving algorithm can work with.