Last December I wanted to try and get two birds stoned at the same time: take a stab at an ML-based project using Claude Code, and experiment toward my longtime dream of creating an automatic transcription method for guitarists.
The goal was to see if an ML model could tell me not just which note I played, but where on the neck I played it. That turned into fretListener: a tool to record and label my own dataset, tooling to build training mixes and train models, and a live fretboard that lights up with whatever I just played. I had a ton of fun with this one!

Full disclosure: this was my first foray into neural nets and ML, and I’m not an expert in ML, AI or data science. Claude Code was used heavily for the ML side, and I learned a lot along the way!
Background
- Note vs. position
- The same pitch can be played in up to three places on frets 0–12. E4 is the open high E string, the B string at fret 5 and the G string at fret 9.
- Neural network (CNN)
- A model that learns patterns from labeled examples instead of hand-written rules. A convolutional neural network (CNN) is a kind that’s good at finding patterns in images, including pictures of sound.
- Spectrogram (log-mel)
- A picture of sound: time across, frequency up, loudness as brightness. The mel scale squishes the frequency axis to roughly match how we hear.
- Training vs. validation
- The model learns from the training clips and gets graded on validation clips it has never seen.
- Test set
- Clips held back until the very end and used only once, for a final performance grade.
- Data leakage
- When the validation clips are too similar to the training clips (here, the same guitar), so the grade looks better than real life.
- Epoch
- One full pass through all the training clips. A training run is usually a few dozen of them.
- Learning rate
- How big a step the model takes each time it adjusts. Big steps learn fast but bounce around, small steps settle in.
Pictures of sound
Right from the start, each note’s first quarter second (the pick attack) gets turned into a spectrogram, so the CNNs are really looking at pictures of sound.
Why pictures? A raw .wav file is just a long list of numbers (12,000 of them for a quarter second at 48 kHz), and a CNN would have to figure out pitch from scratch, meaning more data and more training. A spectrogram does the frequency math up front, so the harmonics show up as stripes in a picture. Also, CNNs are great at finding patterns in images!
Here’s D3 played in three different spots on the neck:

Above shows the same pitch, but the spectrogram of the open string is noticeably brighter up top (lacking the same darker bands that the other two images show). Those small differences in timbre are exactly what the string models learn to pick up on later.
Data first
My first attempt was one CNN to rule them all: note, string and fret prediction all at once. It got the note, string or fret right about 77% of the time, but all three at once only 55% of the time. The code was also (as I stated in one commit) “kind of a mess”, so I started over, this time beginning with the data.
The first two datasets were gathered manually using Logic Pro X, then processed with a script. At this point, results were sufficiently performant (97.5% on the first note model) to warrant hope that more data could yield a usable end product. To reduce capture error and human error in trimming/bouncing the data for each iteration (and to reduce my time in data collection), I wanted to automate this process moving forward.

data_collection.pyis that very automation tool. It walks me through every string and fret, records continuous takes, finds each pick attack, and slices the takes into labeled clips (string, fret, pitch, take and source recording) that are ready for training. It also organizes the clips and builds a manifest that the training mix tool reads directly. More on that later.- In total: I (painstakingly) collected 12,192 labeled single-note clips. This amounts to about 2 hours of audio, across six datasets on Stratocasters and an Ibanez RG550.
Recording all of that took several hours a day over the course of a week. I was doing data collection runs in the morning, evening and any time I could find in between. The hardest part was focusing on my dynamics and trying to play the same way for each capture type.

At one point, training results got worse after a refactor of the data collection tool, so I had to dig into each step of the sample preparation across several captures to find the issue. Captures were getting chopped because I was doing the processing inside the audio callback. The fix will look familiar to an embedded person: the callback only copies the buffer into a queue, and a worker thread does the rest. Treat audio callbacks like interrupt handlers!
Training tooling
build_training_mix.py: a user-configurable tool that runs before training and turns datasets into training mixes. The mixes live as CSV lists of clips and labels, so every run can be rebuilt from a file like a recipe.train_gui.py: builds the mixes and runs the CNN training from one window, showing each training epoch and generating plots upon training completion.- Every run writes its command, settings, training curves and per-class accuracy to its own folder, so I have a record and can go back and compare runs at any time.
Some real numbers
With a random train/validation split, the note model scores 99.8% (hooray, we have a computationally expensive guitar tuner!). Here’s what training looked like for the latest version using six datasets, 37 pitches, and 30 epochs. Validation accuracy was all over the place until the learning rate stepped down, and then everything smoothed out.

But that split puts the same guitars on both sides, so it doesn’t say much about a guitar the model has never heard (classic data leakage). Training on just the Strats and validating on the RG550 dropped it to 94.7%, which is the number I actually trust (and still not bad).

Note, then string
Pitch alone doesn’t tell you where a note is on the neck. My first try at the string part was a single model choosing between all six strings. It did 97.6% on four datasets, then dropped to 93.2% when I added two more Strat datasets (more tone variety for the same label made its job harder).
So I split the job into tiers. First, the note CNN figures out the pitch (37 pitches, E2 to E5). If that pitch only lives in one spot on the neck, then great. Job done. If not, a small disambiguator CNN that’s only trained on that one decision picks the string by its timbre. The two-string disambiguators (like A string fret 5 vs. open D) hit 98–100% and stayed there across reruns.

The three-string disambiguators only had 54–72 validation clips each, and identical reruns swung by up to 20 points, so I won’t trust those numbers until I’ve recorded a lot more clips.
The fun part
Once the models were good enough to actually use, it was time to build some apps around them:
- fretListener itself: play a note and the fretboard on screen lights up where it thinks you played it (string and fret). Live!
- fretTrainer: a practice tool for scales, arpeggios and intervals that checks what you play in real time and gives near-instant feedback.
- A side effect I didn’t plan for: it also works with my guitar going into a mic instead of plugged in DI!
The first time fretListener lit up the right fret, I was so pumped to see it work in nearly real time! I got my first glimpse at a fret-locating capture platform that could be guitar/interface agnostic.


Next
- A proper test set, plus testing every model on a guitar it’s never heard. Right now the numbers above are validation accuracy from the best epoch, and validation also picks which checkpoint to keep, so they’re probably a bit optimistic.
- About 10× more clips per region for the three-string disambiguators. There’s a reason I’ve punted this: it’s incredibly time consuming and tedious.
- Chords, bends and distortion. So far it’s clean tone, single notes and frets 0–12 only.
- Integration with tab software like Guitar Pro. This is the ultimate end goal.
This project really scratched the guitar itch and the building itch at the same time. I’m looking forward to getting back into this one when I have more time… it’s going to be a while before I willingly sign up for more data collection though.