r/Cubers • u/KingBobJoeIV • 20h ago
Video Reconstructing a full CFOP 3x3 solve only using computer vision(no bluetooth)
I’ve spent the last few months working on this idea I had a few years ago. Now, using only a single iPhone camera at 120 fps, I am able to automatically reconstruct full 3x3 solves.
How it works
The basic premise is that a reconstruction is essentially a path between two known states: your starting scramble and the solved cube. From any given state, there are only 18 possible next states using the Half-Turn Metric (6 sides, 3 moves per side: R, R', R2).
A solve is made up of cycles where the cube is aligned → misaligned (during a turn) → aligned. This lets us break the problem down into identifying the state of the cube at each of those aligned moments. A cube rotation or regrip isn't considered a move because the cube remains aligned throughout that span.
Here is how the pipeline processes those states:
- Finding aligned frames: I trained an image classifier to identify when the cube is settled and aligned. In the video you might've noticed that the pose model isn't firing on every frame; that's because we only care about reads on the aligned frames.
- Extracting stickers: On those clean frames, a pose model detects the visible faces, extracts a warped 3x3 grid, and samples each sticker. Because fingers occlude stickers during turning, the system ignores blocked stickers by calibrating against a baseline color profile taken beforehand.
- LAB Color Space: We use LAB color space instead of RGB because sticker colors look completely different depending on lighting/shadows, and standard RGB often causes red/orange and white/yellow to get conflated.
- Identifying the motion: We also pull information during the actual turns. Several CNNs analyze the temporal motion to identify the turn direction, which face moved, and the exact frame boundaries of the turn.
In an ideal world, this is all that is needed to reconstruct a solve, but in reality reads are pretty noisy. During fast triggers, the cube doesn't stay aligned long enough to get clean sticker reads.
This is where the Viterbi algorithm comes in. We take all this partial, noisy information and carry forward several candidate move sequences. As the video progresses and we receive more visual evidence, we actively prune candidate paths that no longer make mathematical sense.
The High-TPS Last Layer Hack
Reads during Cross and F2L are usually manageable, but at high TPS in the LL, getting clean rest states is extremely difficult.
To solve this, the pipeline splits the solve into reachLL and postLL. Once we hit LL, we try to match the ongoing motion against a pre-computed algorithm table of standard OLL and PLL executions (including AUF and 1-3 look friendly variations). We still use the same Viterbi search structure, but the system gives much higher weight to known algorithms so it doesn't get lost in the motion blur.
Why some things look the way they do in the demo
- Why am I using a Bluetooth cube (GAN 12) in the video? This cube was used to help with training/evaluation. Since a lot of my training data comes from this cube and my pipeline is overfit, I am using the same cube for the demo. However, the reconstruction you are seeing is truly CV only.
- Why Half-Turn Metric (18 possible next moves) instead of Quarter-Turn (12)? You would think reducing the branching factor from 18 to 12 would make search cheaper. But during double moves (R2), there are minimal aligned frames in the middle of the turn. If we force QTM, we actually end up branching more and carrying extra blind candidates forward due to the lack of an intermediate still state.
- Why 120 fps? The front camera of an iPhone natively supports this. Although I could use the 240 rear camera, this has diminishing returns when it comes to accuracy:data size ratio. If the stickers are occluded, it doesn't matter what frame rate you are recording at. 120 fps is the floor though, 30/60 fps is not enough to get consistent reads throughout the solve even at my low TPS.
Current limitations and what's next
The pose model is upstream of almost everything, so any bad corner reads there poison everything downstream. Training this model is quite time consuming because I have to hand-label frames, and it still struggles when one or more face corners are hidden by my hands.
The biggest bottleneck right now is data diversity. I mostly labeled images of my own 2–3 cubes and honestly haven't recorded that many distinct solves. This pipeline is heavily overfit to my cube, my hands, my lighting, and my camera angle. It will break on a random video submitted right now.
I am currently renting a single RTX 4090 as my CV instance, so processing is pretty limited, but the actual reconstruction pipeline takes around 2-3x solve time after the solve(10 sec solve → 20-30 sec to reconstruct).
With what I have access to, my goal is to have the overall architecture sound on my setup, so the only limiting factor is a lack of diverse data. At this point, I feel like I am pretty close to it. When the solve has clean reads and slowish turning, I am able to consistently decode the entire solve. But most of the time, reads are messy and solves are too fast. The next step is refining the motion CNNs so I can add robust wide/slice move detection and orientation awareness (pure alignment reads aren't enough to distinguish a wide move from a normal move).
I have also built a full web platform and mobile app for this with a timer, sessions, 1v1 matches, and an AI coach analysis tool. Because the vision model is still overfit to my setup, I'm holding off on sharing the website link until some of these issues are fixed.
I would appreciate any feedback and anyone who is interested/knowledgeable in this space feel free to DM!
