A real-time sound detection app is a short pipeline: capture mono audio at 16 kHz, slice it into overlapping windows of roughly one second, convert each window to a log-mel spectrogram, run it through a pretrained audio embedding model (YAMNet, PANNs or an Audio Spectrogram Transformer) fine-tuned on your own classes, then smooth the per-window scores before you fire an alert. Run inference on the device, not in the cloud, using Core ML on iOS, LiteRT or MediaPipe on Android, or ONNX Runtime elsewhere. The model is the easy part. Most of the engineering effort goes into collecting audio from the microphone you will actually ship on, labeling it correctly, and tuning thresholds so the app does not wake people up at 3 a.m. because the furnace kicked on.

What does the pipeline look like end to end?

Every working sound detector we have seen or built follows the same five stages. The specifics below are the defaults that work for most projects, with the reason behind each one.

• Capture. 16 kHz, mono, 16-bit PCM. Almost every pretrained audio model expects this rate, and anything above it is wasted battery for non-music tasks. On Android use AudioRecord with the unprocessed source where available; on the web use the Web Audio API with an AudioWorklet so processing stays off the main thread.
• Windowing. Buffer 0.96 s of audio and advance by 0.48 s. That gives one prediction every half second with enough context to recognize a bark, a cough or glass breaking. Shorter windows cut latency but accuracy falls sharply below about 0.5 s.
• Features. A 64-band log-mel spectrogram with a 25 ms frame and 10 ms hop. Compute it with torchaudio or librosa during training, and reimplement the exact same transform on-device. Mismatched mel filterbanks between training and deployment are the most common silent bug in these apps.
• Model. A pretrained embedding with a small classifier head trained on your data. See the next section for which one.
• Decision logic. Median-filter the scores over the last three to five windows, use separate on and off thresholds (hysteresis), and require two consecutive positive windows before alerting. This step alone typically removes most false positives.

Which model should you use?

Four years ago the debate was CNN versus RNN. That debate is over. Today you start from a model pretrained on Google’s AudioSet, which contains over two million 10-second YouTube clips labeled with a 527-class ontology, and you fine-tune it. Training a sound classifier from scratch is only worth it for narrow signals with no overlap with everyday sound, such as a specific machine bearing.

For an on-device app, start with YAMNet embeddings and a logistic-regression or two-layer head. You will get a working prototype in an afternoon. Move to AST or BEATs only if your evaluation shows the embedding is the bottleneck, which is rarer than people expect. The bottleneck is usually the data.

How much data do you need, and where does it come from?

For a fine-tuned head on top of a pretrained embedding, 50 to 200 clean clips per class is enough for a demo. A production detector that must hold up across rooms, devices and seasons needs thousands of positive examples per class and far more negatives. The negatives matter more than the positives: a dog-bark detector fails in the field because nobody recorded the dishwasher, the ice maker or the neighbor’s leaf blower.

Public datasets get you started. ESC-50 has 2,000 five-second clips in 50 classes and is the standard quick benchmark. FSD50K has about 51,000 human-labeled clips across 200 classes under Creative Commons licences, which makes it usable in commercial products where AudioSet’s YouTube-sourced clips are not. The annual DCASE challenge publishes task-specific sets for sound event detection and anomaly detection in machines, along with baseline code and leaderboards.

Then record your own, on the deployment hardware. We spend most of our days recording audio on film sets, and the single most important thing we have learned from running classifiers against our own footage library is that the microphone changes the signal more than the room does. A phone with automatic gain control and noise suppression enabled reshapes the spectrum of every event. A model trained on clean 48 kHz lavalier recordings and tested on a phone in a kitchen drops noticeably, and the drop is worst for quiet events like a cough or a distant knock. Record training data with the same mic, the same OS audio settings and the same app in the foreground that you will ship.

How do you label audio without ruining the model?

Label at the event level, with start and end times, not at the clip level. Weak clip-level labels are why AudioSet-trained models are confident about what is in a 10-second clip but vague about when it happened. For real-time detection, timing is the product.

Budget for it honestly. A trained annotator can mark a few hundred short, unambiguous events per hour, and far fewer when events overlap or classes are easily confused. Write a labeling guide with audio examples of each class and of the hard negatives, and have two people label a 5 percent overlap set so you can measure agreement. If two humans disagree on 15 percent of events, your model cannot be expected to do better, and your evaluation numbers are noise below that floor.

Check labels mechanically before training. Clip-length histograms per class, a listen to the top-loss examples after a first training pass, and a confusion matrix between classes that should never be confused will each surface mislabeled files. Our standard failure is an exported clip that was trimmed to silence or room tone and still carries a positive label.

How do you make it run in real time on a phone?

Export the model to the platform runtime and run it on a background audio thread. On iOS, Apple’s Sound Analysis framework handles buffering, windowing and inference for a Core ML model, and ships a built-in classifier covering roughly 300 everyday sounds that you can use before training anything. Apple’s Create ML sound classifier trains a custom model from a folder of labeled clips and exports it directly for that framework.

On Android, TensorFlow Lite was renamed LiteRT in 2024, and the MediaPipe audio classifier task wraps YAMNet with the windowing and scoring already implemented. For cross-platform or desktop targets, ONNX Runtime runs the same exported graph on CPU, GPU and NPU backends. On any platform, quantize to 8-bit integers once accuracy is stable; for YAMNet-sized models the accuracy cost is small and the battery saving is not.

Latency is dominated by the window, not by inference. With a 0.96 s window and 0.48 s hop plus a two-window confirmation rule, an event is reported about 1 to 1.5 s after it starts. If the application needs faster response, such as gunshot or glass-break detection, use a shorter window with a lighter model and accept more false positives, or add a cheap energy-based trigger that only runs the full model when the level jumps. A voice activity detector such as Silero VAD is the standard gate for speech-adjacent tasks.

How should you evaluate it?

Do not report accuracy. For an always-on detector the metrics that predict whether users keep the app installed are per-class precision and recall at the deployed threshold, and false alarms per hour of ordinary life. Measure that last one on multi-hour recordings of real environments with no target events in them. A detector that is 95 percent accurate on a balanced test set but fires three times a night in a quiet house will be uninstalled within a week.

Evaluate on a held-out set recorded in different rooms, by different people, on different days from the training data. Random splits of a single recording session leak acoustic fingerprints and overstate results. For event timing, follow DCASE’s convention of event-based F-score with a collar of around 200 ms.

What goes wrong in practice?

• Domain shift. Training on public YouTube-derived data and deploying on a phone mic. Fix with on-device recordings and augmentation that simulates the target mic: band-limiting, added room reverb, gain changes and compression.
• Steady-state noise. HVAC, refrigerators and traffic shift the noise floor and push quiet events below the threshold. Per-channel energy normalization or simple spectral subtraction before the mel transform helps, but recording training data in those conditions helps more.
• OS audio processing. Both mobile platforms apply echo cancellation and noise suppression by default on some audio sources. Turn them off explicitly, or train with them on. Mixing the two is the worst option.
• Battery. Continuous capture and inference is a real cost. Use the platform’s low-power audio paths, batch windows where latency allows, and gate the model behind a cheap energy detector.
• Privacy. Keep raw audio on the device and send only class labels and timestamps if you need telemetry. This is also what makes the app shippable under the microphone-permission rules of both app stores.

What is being built with this now?

Accessibility alerts for deaf and hard-of-hearing users are built into both mobile operating systems, detecting doorbells, alarms, crying and running water locally. Cornell’s BirdNET identifies thousands of bird species from phone recordings and has become a standard tool in field ecology. Factory and HVAC maintenance teams run anomaly detectors on vibration and acoustic sensors to catch bearing wear weeks before failure, a task that DCASE has benchmarked since 2020. Respiratory health research uses cough and breathing classifiers, though clinical deployment there still depends on regulatory clearance rather than model accuracy.

Start with YAMNet embeddings and a tiny classifier, record a few hundred clips on the device you will ship, measure false alarms per hour on real recordings, then decide whether a larger model or more data is the better next investment. In our experience it is almost always more data.