Last December I wanted to try and get two birds stoned at the same time: take a stab at an ML-based project using Claude Code, and experiment toward my longtime dream of creating an automatic transcription method for guitarists.

The goal was to see if an ML model could tell me not just which note I played, but where on the neck I played it. That turned into fretListener: a tool to record and label my own dataset, tooling to build training mixes and train models, and a live fretboard that lights up with whatever I just played. I had a ton of fun with this one!

fretListener after hearing a D3: the note model is 98.8% sure of the pitch, the E/A/D disambiguator puts it on the low E string at fret 10, and a red dot marks that spot on the fretboard

Full disclosure: this was my first foray into neural nets and ML, and I’m not an expert in ML, AI or data science. Claude Code was used heavily for the ML side, and I learned a lot along the way!

Background

Note vs. position
The same pitch can be played in up to three places on frets 0–12. E4 is the open high E string, the B string at fret 5 and the G string at fret 9.
Neural network (CNN)
A model that learns patterns from labeled examples instead of hand-written rules. A convolutional neural network (CNN) is a kind that’s good at finding patterns in images, including pictures of sound.
Spectrogram (log-mel)
A picture of sound: time across, frequency up, loudness as brightness. The mel scale squishes the frequency axis to roughly match how we hear.
Training vs. validation
The model learns from the training clips and gets graded on validation clips it has never seen.
Test set
Clips held back until the very end and used only once, for a final performance grade.
Data leakage
When the validation clips are too similar to the training clips (here, the same guitar), so the grade looks better than real life.
Epoch
One full pass through all the training clips. A training run is usually a few dozen of them.
Learning rate
How big a step the model takes each time it adjusts. Big steps learn fast but bounce around, small steps settle in.

Pictures of sound

Right from the start, each note’s first quarter second (the pick attack) gets turned into a spectrogram, so the CNNs are really looking at pictures of sound.

Why pictures? A raw .wav file is just a long list of numbers (12,000 of them for a quarter second at 48 kHz), and a CNN would have to figure out pitch from scratch, meaning more data and more training. A spectrogram does the frequency math up front, so the harmonics show up as stripes in a picture. Also, CNNs are great at finding patterns in images!

Here’s D3 played in three different spots on the neck:

Spectrograms of D3 played three ways (low E string fret 10, A string fret 5, open D string), each averaged over 20 takes, next to a spectral profile showing the open string is brightest above 1 kHz Spectrograms of D3 played three ways (low E string fret 10, A string fret 5, open D string), each averaged over 20 takes, next to a spectral profile showing the open string is brightest above 1 kHz

Above shows the same pitch, but the spectrogram of the open string is noticeably brighter up top (lacking the same darker bands that the other two images show). Those small differences in timbre are exactly what the string models learn to pick up on later.

Data first

My first attempt was one CNN to rule them all: note, string and fret prediction all at once. It got the note, string or fret right about 77% of the time, but all three at once only 55% of the time. The code was also (as I stated in one commit) “kind of a mess”, so I started over, this time beginning with the data.

The first two datasets were gathered manually using Logic Pro X, then processed with a script. At this point, results were sufficiently performant (97.5% on the first note model) to warrant hope that more data could yield a usable end product. To reduce capture error and human error in trimming/bouncing the data for each iteration (and to reduce my time in data collection), I wanted to automate this process moving forward.

The data collection tool ready to record the open low E string: guitar, pickup and takes-per-note settings on the left, progress bars for the current note, string and session on the right

  • data_collection.py is that very automation tool. It walks me through every string and fret, records continuous takes, finds each pick attack, and slices the takes into labeled clips (string, fret, pitch, take and source recording) that are ready for training. It also organizes the clips and builds a manifest that the training mix tool reads directly. More on that later.
  • In total: I (painstakingly) collected 12,192 labeled single-note clips. This amounts to about 2 hours of audio, across six datasets on Stratocasters and an Ibanez RG550.

Recording all of that took several hours a day over the course of a week. I was doing data collection runs in the morning, evening and any time I could find in between. The hardest part was focusing on my dynamics and trying to play the same way for each capture type.

Bar chart of clips per dataset: 12,192 hand-recorded clips, about 2 hours of audio, the largest being 5,712 clips from a Strat neck pickup Bar chart of clips per dataset: 12,192 hand-recorded clips, about 2 hours of audio, the largest being 5,712 clips from a Strat neck pickup

At one point, training results got worse after a refactor of the data collection tool, so I had to dig into each step of the sample preparation across several captures to find the issue. Captures were getting chopped because I was doing the processing inside the audio callback. The fix will look familiar to an embedded person: the callback only copies the buffer into a queue, and a worker thread does the rest. Treat audio callbacks like interrupt handlers!

Training tooling

  • build_training_mix.py: a user-configurable tool that runs before training and turns datasets into training mixes. The mixes live as CSV lists of clips and labels, so every run can be rebuilt from a file like a recipe.
  • train_gui.py: builds the mixes and runs the CNN training from one window, showing each training epoch and generating plots upon training completion.
  • Every run writes its command, settings, training curves and per-class accuracy to its own folder, so I have a record and can go back and compare runs at any time.

Some real numbers

With a random train/validation split, the note model scores 99.8% (hooray, we have a computationally expensive guitar tuner!). Here’s what training looked like for the latest version using six datasets, 37 pitches, and 30 epochs. Validation accuracy was all over the place until the learning rate stepped down, and then everything smoothed out.

Note model training curves over 30 epochs: training accuracy climbs smoothly, validation accuracy swings as low as 41% at epoch 8, then both settle near 99.8% once the learning rate drops 10x after epoch 21 Note model training curves over 30 epochs: training accuracy climbs smoothly, validation accuracy swings as low as 41% at epoch 8, then both settle near 99.8% once the learning rate drops 10x after epoch 21

But that split puts the same guitars on both sides, so it doesn’t say much about a guitar the model has never heard (classic data leakage). Training on just the Strats and validating on the RG550 dropped it to 94.7%, which is the number I actually trust (and still not bad).

Validation accuracy per training run: about 99.8% for the note model on random splits, 94.7% on a guitar it never heard, and 97.6% to 93.2% for the six-string model as datasets were added Validation accuracy per training run: about 99.8% for the note model on random splits, 94.7% on a guitar it never heard, and 97.6% to 93.2% for the six-string model as datasets were added

Note, then string

Pitch alone doesn’t tell you where a note is on the neck. My first try at the string part was a single model choosing between all six strings. It did 97.6% on four datasets, then dropped to 93.2% when I added two more Strat datasets (more tone variety for the same label made its job harder).

So I split the job into tiers. First, the note CNN figures out the pitch (37 pitches, E2 to E5). If that pitch only lives in one spot on the neck, then great. Job done. If not, a small disambiguator CNN that’s only trained on that one decision picks the string by its timbre. The two-string disambiguators (like A string fret 5 vs. open D) hit 98–100% and stayed there across reruns.

Disambiguator accuracy across reruns: two-string models stay within 2 points near 100%, three-string models swing by up to 20 points Disambiguator accuracy across reruns: two-string models stay within 2 points near 100%, three-string models swing by up to 20 points

The three-string disambiguators only had 54–72 validation clips each, and identical reruns swung by up to 20 points, so I won’t trust those numbers until I’ve recorded a lot more clips.

The fun part

Once the models were good enough to actually use, it was time to build some apps around them:

  • fretListener itself: play a note and the fretboard on screen lights up where it thinks you played it (string and fret). Live!
  • fretTrainer: a practice tool for scales, arpeggios and intervals that checks what you play in real time and gives near-instant feedback.
  • A side effect I didn’t plan for: it also works with my guitar going into a mic instead of plugged in DI!

The first time fretListener lit up the right fret, I was so pumped to see it work in nearly real time! I got my first glimpse at a fret-locating capture platform that could be guitar/interface agnostic.

The full fretListener window: inference mode, model, audio device and trigger settings on the left, and the detected D3, the fretboard and a log of predictions on the right

fretTrainer running a C major scale: the target note C3 in big yellow letters, the scale’s notes marked on the fretboard, and correct and wrong counters for the session

Next

  • A proper test set, plus testing every model on a guitar it’s never heard. Right now the numbers above are validation accuracy from the best epoch, and validation also picks which checkpoint to keep, so they’re probably a bit optimistic.
  • About 10× more clips per region for the three-string disambiguators. There’s a reason I’ve punted this: it’s incredibly time consuming and tedious.
  • Chords, bends and distortion. So far it’s clean tone, single notes and frets 0–12 only.
  • Integration with tab software like Guitar Pro. This is the ultimate end goal.

This project really scratched the guitar itch and the building itch at the same time. I’m looking forward to getting back into this one when I have more time… it’s going to be a while before I willingly sign up for more data collection though.