I Built a License Plate Detector That Runs on a Raspberry Pi (No Ultralytics License Required!)

Ever looked at a license plate detector demo and thought, “Cool, but I can’t afford an enterprise license”? Same. So I built my own — from scratch, using open-source tools, optimized to run on edge hardware like a Raspberry Pi or a smartphone.

In this post I’ll walk you through the full pipeline: from wrangling raw Pascal VOC annotations all the way to exporting a quantized TFLite model AND a CoreML package for iPhones. By the end, you’ll have a detector that’s lean, fast, and completely license-free. Let’s go! 🚀 (This Blog is an example project of my previous blog go read here).

Full code + trained models are on my GitHub — link at the bottom. You’re welcome. 😄

The Dataset: Where It All Begins

I used a license plate dataset sourced from Roboflow, downloaded in Pascal VOC format (XML annotations paired with JPEG images). If you want to follow along, grab a similar dataset from Roboflow Universe.

Step 1: Teaching Python to Read Pascal VOC Annotations

Before any training can happen, we need to translate raw image files and their XML annotations into a format KerasCV actually understands. Think of this step as building the bridge between “files sitting on disk” and “tensors ready for a GPU.”

Here’s what the code does, broken down simply:

Parsing the XML files — Each Pascal VOC annotation is an XML file containing the bounding box coordinates (xmin, ymin, xmax, ymax) of every license plate in that image. We read those coordinates and map the class name LicensePlate to a numeric index, because neural networks speak numbers, not words.

Handling variable plate counts with Ragged Tensors — Some images have one plate, some have three. A regular tensor can’t handle that variability without crashing. TensorFlow’s RaggedTensor is specifically designed for this — it's like a flexible list that doesn't force every row to be the same length.

Packaging into KerasCV’s required format — KerasCV’s object detection API is strict about its input format. It expects a dictionary with exactly two keys: "images" and "bounding_boxes". Get this wrong and you'll spend an hour debugging a cryptic error. The code below handles this automatically.

import tensorflow as tf
import keras_cv
import os
import xml.etree.ElementTree as ET

CLASSES = ["LicensePlate"]
CLASS_MAPPING = {name: idx for idx, name in enumerate(CLASSES)}
BATCH_SIZE = 8

def parse_voc_annotation(xml_file):
    """Parses a Pascal VOC XML file and returns bounding boxes and class IDs."""
    tree = ET.parse(xml_file)
    root = tree.getroot()
    boxes, classes = [], []

    for obj in root.findall('object'):
        class_name = obj.find('name').text
        if class_name not in CLASS_MAPPING:
            continue
        classes.append(CLASS_MAPPING[class_name])
        bndbox = obj.find('bndbox')
        boxes.append([
            float(bndbox.find('xmin').text),
            float(bndbox.find('ymin').text),
            float(bndbox.find('xmax').text),
            float(bndbox.find('ymax').text)
        ])
    return boxes, classes

def load_data(image_dir):
    """Walks through a directory and collects image paths, boxes, and classes."""
    image_paths, all_boxes, all_classes = [], [], []

    for img_name in os.listdir(image_dir):
        if not img_name.endswith('.jpg'):
            continue
        img_path = os.path.join(image_dir, img_name)
        xml_path = os.path.join(image_dir, img_name.replace('.jpg', '.xml'))
        if not os.path.exists(xml_path):
            continue
        boxes, classes = parse_voc_annotation(xml_path)
        if boxes:
            image_paths.append(img_path)
            all_boxes.append(boxes)
            all_classes.append(classes)

    return image_paths, all_boxes, all_classes

# Update these paths to wherever you unzipped the dataset
train_img_paths, train_boxes, train_classes = load_data("/content/license-plate-detection-2/train")
valid_img_paths, valid_boxes, valid_classes = load_data("/content/license-plate-detection-2/valid")

def create_dataset(image_paths, boxes, classes):
    # Ragged tensors handle the variable number of boxes per image
    box_tensor   = tf.ragged.constant(boxes, dtype=tf.float32)
    class_tensor = tf.ragged.constant(classes, dtype=tf.float32)
    path_tensor  = tf.constant(image_paths)

    dataset = tf.data.Dataset.from_tensor_slices((path_tensor, class_tensor, box_tensor))

    def load_image_and_format(path, cls, box):
        img = tf.io.read_file(path)
        img = tf.image.decode_jpeg(img, channels=3)
        img = tf.cast(img, tf.float32)
        # This exact dictionary structure is what KerasCV requires — don't change the key names!
        return {
            "images": img,
            "bounding_boxes": {"classes": cls, "boxes": box}
        }

    return dataset.map(load_image_and_format, num_parallel_calls=tf.data.AUTOTUNE)

train_ds = create_dataset(train_img_paths, train_boxes, train_classes)
valid_ds = create_dataset(valid_img_paths, valid_boxes, valid_classes)

Step 2: Resizing Without Accidentally Deleting License Plates

Now that data is loaded, every image needs to be a uniform 416×416 before the GPU can batch them together. But here's a sneaky trap: standard resizing can crop out license plates sitting near the edges of an image, leaving you with zero bounding boxes in a batch — and a training loop that promptly crashes.

The fix is pad_to_aspect_ratio=True, which adds black "letterbox" padding around the image instead of cropping anything off. Every plate stays in the frame, guaranteed.

There’s a second problem to solve. We’ve been using flexible Ragged Tensors, but GPUs need perfectly uniform static-sized matrices to run efficiently with XLA compilation. The solution is to convert Ragged Tensors into regular Dense Tensors by padding any missing bounding box slots with the value -1.0. This isn't arbitrary — KerasCV's loss functions are hardcoded to recognize -1.0 as "nothing here, skip this slot." It's the agreed-upon ignore signal between the data pipeline and the trainer.

# This layer resizes images AND automatically adjusts bounding box coordinates to match
resizing_layer = keras_cv.layers.Resizing(
    height=416,
    width=416,
    pad_to_aspect_ratio=True,  # Letterbox padding — never crops out a plate
    bounding_box_format="xyxy"
)

def prepare_dataset(dataset, batch_size, is_training=True):
    dataset = dataset.ragged_batch(batch_size, drop_remainder=is_training)
    dataset = dataset.map(resizing_layer, num_parallel_calls=tf.data.AUTOTUNE)

    def dict_to_dense_tuple(inputs):
        # Pad empty box slots with -1.0 so KerasCV's loss function knows to ignore them
        dense_boxes   = inputs["bounding_boxes"]["boxes"].to_tensor(default_value=-1.0)
        dense_classes = inputs["bounding_boxes"]["classes"].to_tensor(default_value=-1.0)
        return inputs["images"], {"boxes": dense_boxes, "classes": dense_classes}

    dataset = dataset.map(dict_to_dense_tuple, num_parallel_calls=tf.data.AUTOTUNE)
    return dataset.prefetch(tf.data.AUTOTUNE)

train_dataset = prepare_dataset(train_ds, BATCH_SIZE, is_training=True)
valid_dataset = prepare_dataset(valid_ds, BATCH_SIZE, is_training=False)

Step 3: Assembling a YOLOv8 Model Built for Edge Hardware

Here’s where the fun begins. Since the end goal is edge deployment, we deliberately choose the YOLOv8 Extra-Small (xs) backbone. This variant trades a tiny amount of accuracy for dramatically faster inference — exactly the right trade-off when your compute budget is a Raspberry Pi or a mobile phone NPU.

We also set fpn_depth=1 to keep the Feature Pyramid Network shallow. The FPN is the part of the model that detects objects at multiple scales. A depth of 1 means it's fast and lightweight, which is perfect for a single-class detector like ours.

The most important line in this whole block is the dummy tensor pass. After building the model but before compiling it, we run a single fake image of zeros through it. This forces TensorFlow to trace the computational graph with exact static shapes locked in. Skip this step and your TFLite or CoreML conversion will either fail outright or produce a model with mysterious shape errors at runtime.

Finally, we compile with Binary Cross-Entropy for classification (correct for single-class detection) and CIoU for bounding boxes. CIoU is a smarter loss than plain IoU — it penalizes the model not just for getting the box size wrong, but also for getting the aspect ratio and center point wrong.

import keras
import keras_cv

NUM_CLASSES = 1
INPUT_SHAPE = (416, 416, 3)

print("Loading YOLOv8-XS backbone...")
backbone = keras_cv.models.YOLOV8Backbone.from_preset("yolo_v8_xs_backbone")

print("Building detector head...")
model = keras_cv.models.YOLOV8Detector(
    num_classes=NUM_CLASSES,
    bounding_box_format="xyxy",  # Matches our data pipeline format exactly
    backbone=backbone,
    fpn_depth=1,                 # Shallow FPN = fast inference on edge devices
)

# THE EDGE TRICK: Run a dummy input through the model to lock in static tensor shapes.
# This is required for reliable TFLite and CoreML conversion later — don't skip it!
print("Tracing static graph for edge compatibility...")
dummy_input = keras.ops.zeros((1, *INPUT_SHAPE))
_ = model(dummy_input)

optimizer = keras.optimizers.Adam(learning_rate=0.001)
model.compile(
    optimizer=optimizer,
    classification_loss="binary_crossentropy",  # BCE is correct for single-class detection
    box_loss="ciou"                              # CIoU penalizes size, aspect ratio, and center offset
)

model.summary()

Step 4: Training Smarter With Callbacks

We train for up to 100 epochs — but we’re not going to babysit a loss curve for hours. Two callbacks handle everything automatically.

ModelCheckpoint watches the validation loss after every epoch. The moment it sees a new personal best, it saves the model to disk. You always have the best version safely stored, no matter what happens next.

EarlyStopping is your insurance policy. If validation loss hasn’t improved for 20 consecutive epochs, it stops training and restores the best weights it ever saw. This prevents wasting hours on a model that has already plateaued, and it stops overfitting before it gets bad.

We also drop the learning rate from 1e-3 to 1e-4. Smaller steps mean the optimizer makes more precise adjustments — especially helpful on smaller datasets where it's easy to overshoot the optimal weights.

optimizer = keras.optimizers.Adam(learning_rate=1e-4)  # Smaller steps = more precise learning
model.compile(
    optimizer=optimizer,
    classification_loss="binary_crossentropy",
    box_loss="ciou"
)

callbacks = [
    keras.callbacks.ModelCheckpoint(
        filepath="best_edge_detector.keras",
        monitor="val_loss",
        save_best_only=True,   # Only overwrites the file when we beat our previous best
        mode="min",
        verbose=1
    ),
    keras.callbacks.EarlyStopping(
        monitor="val_loss",
        patience=20,           # Give the model 20 epochs to improve before calling it done
        restore_best_weights=True,
        verbose=1
    )
]

EPOCHS = 100
print("Starting training — go grab a coffee ☕")
history = model.fit(
    train_dataset,
    validation_data=valid_dataset,
    epochs=EPOCHS,
    callbacks=callbacks
)

Step 5: Running Inference and Drawing the Results

Training is done — time to see what we built! 🎉

Two things in the post-processing step are worth understanding properly.

Injecting NMS (Non-Maximum Suppression) — Without NMS, the model outputs a messy cluster of overlapping boxes around every plate. This happens because many anchor points in the detection grid independently “think” they spotted a plate. NMS merges those overlapping guesses into a single clean box by keeping the one with the highest confidence score and discarding any other box that overlaps it by more than 30%.

Scaling coordinates back to original image size — The model was trained on 416×416 images, so its output coordinates are relative to that small square. Your original photo might be 1920×1080. We calculate scale_x and scale_y ratios and multiply every coordinate by them to project the predictions back onto the full-resolution image. Without this step, all boxes would appear clustered in the top-left corner.

import keras, keras_cv, tensorflow as tf, cv2
import matplotlib.pyplot as plt
import numpy as np

MODEL_PATH           = "best_edge_detector.keras"
IMAGE_PATH           = "/content/images.jpg"
INPUT_SHAPE          = (416, 416)
CONFIDENCE_THRESHOLD = 0.15
CLASSES              = {0: "LicensePlate"}
COLORS               = {0: (0, 255, 0)}  # Green boxes

model = keras.models.load_model(MODEL_PATH, compile=False)

# Inject NMS so we get one clean box per plate instead of a cluster of overlapping guesses
model.prediction_decoder = keras_cv.layers.MultiClassNonMaxSuppression(
    bounding_box_format="xyxy",
    from_logits=False,
    max_detections=50,
    iou_threshold=0.30,        # Merge boxes that overlap by more than 30%
    confidence_threshold=0.50  # Discard any prediction below 50% confidence
)

def load_and_preprocess(image_path):
    img = cv2.imread(image_path)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    orig_h, orig_w = img.shape[:2]
    img_resized = cv2.resize(img, INPUT_SHAPE)
    img_tensor  = tf.cast(img_resized, tf.float32)
    img_tensor  = tf.expand_dims(img_tensor, axis=0)
    return img, img_tensor, orig_w, orig_h

original_img, input_tensor, orig_w, orig_h = load_and_preprocess(IMAGE_PATH)

# Disable XLA JIT for inference to avoid shape errors
tf.config.optimizer.set_jit(False)
tf.config.run_functions_eagerly(True)
predictions = model.predict(input_tensor, verbose=0)
tf.config.run_functions_eagerly(False)

boxes       = predictions['boxes'][0]
classes     = predictions['classes'][0]
confidences = predictions['confidence'][0]

# Scale factors: project model's 416x416 coordinates back to the original image resolution
scale_x  = orig_w / INPUT_SHAPE[0]
scale_y  = orig_h / INPUT_SHAPE[1]
draw_img = original_img.copy()
detections_found = 0

for i in range(len(boxes)):
    confidence = confidences[i]
    class_id   = int(classes[i])
    if class_id == -1 or confidence < CONFIDENCE_THRESHOLD:
        continue  # Skip padding slots (-1) and low-confidence predictions
    detections_found += 1
    xmin, ymin, xmax, ymax = boxes[i]
    xmin = int(xmin * scale_x)
    ymin = int(ymin * scale_y)
    xmax = int(xmax * scale_x)
    ymax = int(ymax * scale_y)
    color = COLORS.get(class_id, (255, 255, 255))
    cv2.rectangle(draw_img, (xmin, ymin), (xmax, ymax), color, 3)
    label = f"{CLASSES[class_id]}: {confidence:.2f}"
    (text_w, text_h), _ = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.6, 2)
    cv2.rectangle(draw_img, (xmin, ymin - text_h - 10), (xmin + text_w, ymin), color, -1)
    cv2.putText(draw_img, label, (xmin, ymin - 5), cv2.FONT_HERSHEY_SIMPLEX, 0.6, (255, 255, 255), 2)

print(f"Found {detections_found} license plate(s).")
plt.figure(figsize=(10, 10))
plt.imshow(draw_img)
plt.axis('off')
plt.title("Edge Detection Results")
plt.show()

Step 6: Shrinking the Model 4× With INT8 Quantization (TFLite)

A standard Keras model uses 32-bit floating-point math everywhere. That’s great for training accuracy, but it’s far too heavy for edge devices — the model is too large, too slow, and burns too much battery.

INT8 Post-Training Quantization compresses every weight from a 32-bit float down to an 8-bit integer. The result is a model that’s roughly 4× smaller and significantly faster on hardware with dedicated integer math units, which includes most modern microcontrollers, Edge TPUs, and Android phones.

The catch is that you can’t blindly compress the weights — the converter needs to know what range of values the model sees during real use. That’s the job of representative_data_gen. It feeds 100 real validation images through the network so the converter can measure activation ranges and calibrate integer scaling factors accurately. Skip this and quantization can noticeably hurt accuracy.

We keep the input and output as float32 for easy integration with mobile app code, while all the internal math runs at INT8.

import tensorflow as tf, cv2, os, numpy as np

MODEL_PATH  = "best_edge_detector.keras"
IMAGE_DIR   = "/content/license-plate-detection-2/valid"
INPUT_SHAPE = (416, 416)

model = tf.keras.models.load_model(MODEL_PATH, compile=False)

def representative_data_gen():
    """Feeds real images to the converter so it can calibrate INT8 scaling ranges."""
    image_files = [f for f in os.listdir(IMAGE_DIR) if f.endswith('.jpg')][:100]
    for img_name in image_files:
        img = cv2.imread(os.path.join(IMAGE_DIR, img_name))
        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
        img = cv2.resize(img, INPUT_SHAPE)
        img_tensor = tf.cast(img, tf.float32)
        img_tensor = tf.expand_dims(img_tensor, axis=0)
        yield [img_tensor]

converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations          = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_data_gen

# Force INT8 for internal ops — required for Edge TPU and Android NPU acceleration
converter.target_spec.supported_ops = [
    tf.lite.OpsSet.TFLITE_BUILTINS_INT8,
    tf.lite.OpsSet.SELECT_TF_OPS
]
# Float32 interface makes it easy to drop into Android or iOS app code
converter.inference_input_type  = tf.float32
converter.inference_output_type = tf.float32

print("Converting to INT8 TFLite (this takes a few minutes)...")
tflite_model = converter.convert()

with open("edge_detector_quantized.tflite", "wb") as f:
    f.write(tflite_model)

print("Saved: edge_detector_quantized.tflite ✅")

Step 7: Exporting to CoreML for the Apple Neural Engine

Want this running natively on an iPhone, using the dedicated Apple Neural Engine chip? This final step handles that.

coremltools translates our TensorFlow graph into Apple's proprietary .mlpackage format, but there are a few iOS-specific things to get right.

Dynamic input name extraction — Keras sometimes auto-assigns cryptic names like keras_tensor_12 to input nodes. We read the actual name directly from the model instead of hardcoding it, which prevents mismatch errors during conversion.

Declaring the input as an ImageType — Passing the input as a generic tensor means Xcode treats it as a raw math array, requiring extra glue code in Swift. Declaring it as ct.ImageType tells Xcode it's a standard camera image, which makes app integration significantly cleaner.

Setting compute_units=ALL — This is the key flag. It grants iOS permission to route the model's computation to the Apple Neural Engine (ANE). On an iPhone 13 or newer, this means near-instant inference at minimal battery cost.

import coremltools as ct
import tensorflow as tf

MODEL_PATH = "best_edge_detector.keras"
model = tf.keras.models.load_model(MODEL_PATH, compile=False)

# Read the actual input node name — never hardcode this for KerasCV models
model_input_name = model.inputs[0].name.split(':')[0]
print(f"Detected input name: {model_input_name}")

# Declare the input as an image so Xcode handles it natively in Swift
image_input = ct.ImageType(
    name=model_input_name,
    shape=(1, 416, 416, 3),
    color_layout=ct.colorlayout.RGB,
)

print("Converting to CoreML...")
mlmodel = ct.convert(
    model,
    inputs=[image_input],
    convert_to="mlprogram",           # Modern .mlpackage format for Xcode 14+
    compute_units=ct.ComputeUnit.ALL  # Let iOS route to the Apple Neural Engine
)

mlmodel.save("EdgeDetector.mlpackage")
print("Saved: EdgeDetector.mlpackage ✅")

What We Built: A Quick Summary

The whole pipeline runs on a completely open-source stack with zero proprietary licenses. The INT8 quantized model is roughly 4× smaller and meaningfully faster on any device with dedicated integer hardware. The CoreML export unlocks the Apple Neural Engine for near-instant on-device inference.

The single most important takeaway: the dummy tensor pass in Step 3 is what makes Steps 6 and 7 work reliably. Don’t skip it.

Get the Full Code

The complete Jupyter notebook, pre-trained models, and the ready-to-use quantized TFLite file are all on my GitHub. Clone it and run — no setup headaches.

👉 Github

If this post helped you out, a clap or two goes a long way in encouraging me to keep writing. Happy building! 🙌