Chat about this codebase

AI-powered code exploration

Online

Project Overview

Doddle_Detection is a Unity-based framework for rapid in-editor experimentation with image-classification models. It streamlines the entire workflow—from loading standard datasets and generating training data via GPU shaders to running neural-network inference and visualizing training metrics directly inside Unity scenes.

Problems Addressed

• Slow iteration on model input pipelines and visual feedback in traditional ML environments
• Manual dataset handling and preprocessing outside the game engine
• Lack of real-time, in-context training and debugging tools for computer-vision workflows

Supported Datasets

• MNIST (handwritten digits)
• Fashion-MNIST (apparel images)
• CIFAR-10 (small object classes)
• Google Quick-Draw Doodles (sketch data)

Major Feature Pillars

Data Loaders

Prebuilt loaders parse, preprocess (normalize, resize) and batch dataset samples for both training and inference.

GPU-Driven Drawing

Compute-shader pipelines render procedural input patterns or augment dataset samples in real time, offloading CPU work.

Neural-Network Runtime

Integrates Unity Barracuda (or ONNX backends) to run forward and backward passes directly in-editor.

In-Scene Training & Visualization

Live training controls and overlay graphs show loss, accuracy and confusion matrices on Unity canvases or 3D planes.

Setup Requirements

Unity Version

Check ProjectSettings/ProjectVersion.txt for the required Unity editor. For example: 

m_EditorVersion: 2021.3.14f1
m_EditorVersionWithRevision: 2021.3.14f1 (abcd1234)

Use Unity Hub to install the matching version for full compatibility.

Package Dependencies

Open Packages/manifest.json to verify or add required modules: 

{
  "dependencies": {
    "com.unity.animation": "1.2.0",
    "com.unity.physics": "0.7.0-preview",
    "com.unity.ui": "1.0.0",
    "com.unity.barracuda": "1.5.0",
    "com.unity.test-framework": "1.1.29",
    "com.unity.xr.management": "4.2.1"
  }
}

Save the file and Unity will auto-resolve and install these packages.

Why You Should Care

In just seconds you can load a standard dataset, tweak a neural-network architecture, train it live within your scene, and immediately see how adjustments affect model performance—all without leaving the Unity editor. This tight feedback loop accelerates prototyping, teaching, and deploying vision-based features in games, simulations, AR/VR experiences or interactive installations.

2. Getting Started

Follow these steps to clone the repo, open in Unity, and run your first demos in under 5 minutes.

Prerequisites

• Unity 2021.3.2f1 (LTS)
• Git
• Optional: Visual Studio or Rider for script editing

2.1 Clone the Repository

Open your terminal or PowerShell and run:

git clone https://github.com/Krishna-Gowami/Doddle_Detection.git
cd Doddle_Detection

2.2 Open in Unity

  1. Launch Unity Hub.
  2. Click Add, select the cloned Doddle_Detection folder.
  3. Open the project with Unity 2021.3.2f1.
  4. Wait for Package Manager to install dependencies defined in Packages/manifest.json.

2.3 Run the MNIST Training Demo

  1. In the Project window navigate to
    Assets/Scenes/Mnist/Train Mnist.unity
  2. Double-click the scene to open it.
  3. Press Play in the Editor toolbar.
  4. Watch the training graph update in real time.

Inspector tips:

  • Select the Trainer object to tweak batchSize, learningRate, epochs.
  • Observe loss/accuracy curves on the Training Graph prefab.

2.4 Try the Doodle Drawing Demo

  1. Open
    Assets/Scenes/Doodles/Draw Doodles.unity
  2. Press Play.
  3. Draw on the canvas with mouse or touch.
  4. See live confidence and label display.

2.5 Run the CIFAR-10 Training Demo

  1. Open
    Assets/Scenes/Cifar10/Train Cifar.unity
  2. Select the Trainer object in Hierarchy.
  3. Adjust hyperparameters (batchSize, learningRate, epochs).
  4. Press Play to begin CIFAR-10 training and view the graph.

2.6 Include All Demos in Your Build

By default the build includes only the MNIST scene. To add the other scenes:

  1. Go to File > Build Settings.
  2. Click Add Open Scenes for each demo scene.

Automate via editor script (place under Editor/ folder):

using System.Linq;
using UnityEditor;
using UnityEditor.SceneManagement;

public class DemoBuildSetup
{
    [MenuItem("Build/Include All Demo Scenes")]
    static void IncludeAllScenes()
    {
        string[] demoScenes = {
            "Assets/Scenes/Mnist/Train Mnist.unity",
            "Assets/Scenes/Cifar10/Train Cifar.unity",
            "Assets/Scenes/Doodles/Draw Doodles.unity"
        };
        EditorBuildSettings.scenes = demoScenes
            .Select(path => new EditorBuildSettingsScene(path, true))
            .ToArray();
        UnityEngine.Debug.Log("All demo scenes added to Build Settings.");
    }
}

You’re now ready to explore training workflows and real-time inference demos. Enjoy!

3. Data Pipeline

Transforms raw binary datasets into ready-to-train tensors. You’ll load DataPoint arrays via your chosen loader, then split, shuffle, batch and optionally augment on-the-fly using DataSetHelper utilities.

3.1 Splitting, Shuffling and Batching

3.1.1 SplitData

Divide inputs/labels into training and validation sets.

using Assets.Scripts.Training;

// Assume you’ve loaded all inputs and outputs:
double[][] inputs      = loader.GetAllData().Select(dp => dp.inputs).ToArray();
double[][] outputs     = loader.GetAllData().Select(dp => dp.expectedOutputs).ToArray();
double validationSplit = 0.2; // reserve 20% for validation

// Overload returning tuples (C# 7+)
(var trainX, var trainY, var valX, var valY) =
    DataSetHelper.SplitData(inputs, outputs, validationSplit);

// Or older‐style out parameters:
// DataSetHelper.SplitData(inputs, outputs, validationSplit,
//     out trainX, out trainY, out valX, out valY);

Console.WriteLine($"Train: {trainX.Length}, Val: {valX.Length}");

3.1.2 ShuffleBatches

Randomize order in-place before batching:

// Shuffle training set before creating mini‐batches
DataSetHelper.ShuffleBatches(trainX, trainY);

3.1.3 CreateMiniBatches

Group data into batches of fixed size:

int batchSize = 32;
// Returns List<(double[][] inputs, double[][] outputs)>
var trainBatches =
    DataSetHelper.CreateMiniBatches(trainX, trainY, batchSize);

Iterate through batches in your training loop:

foreach (var (batchInputs, batchOutputs) in trainBatches)
{
    // Convert to tensors or feed directly into your model
    model.TrainOnBatch(batchInputs, batchOutputs);
}

3.2 On-the-Fly Augmentation

Inject data augmentation between batching and training without materializing a second dataset.

using System;
using System.Linq;

// Example augmentation: add uniform noise to each pixel
Func<double[], double[]> AddNoise = pixels =>
{
    var rnd = new Random();
    return pixels.Select(p => p + (rnd.NextDouble() - 0.5) * 0.1).ToArray();
};

// Generate augmented mini‐batches
var augmentedBatches = trainBatches
    .Select(batch =>
    {
        var augInputs = batch.inputs
            .Select(AddNoise)            // apply to each sample
            .ToArray();
        return (augInputs, batch.outputs);
    })
    .ToList();

// Training with augmentation
foreach (var (augX, augY) in augmentedBatches)
{
    model.TrainOnBatch(augX, augY);
}

Practical Tips

  • Shuffle before each epoch: call ShuffleBatches on trainX/trainY at start of loop.
  • For multiple augmentations, compose Func<double[], double[]> delegates.
  • Adjust batchSize to balance GPU utilization and memory.

Neural Network Core

This section covers the internal C# implementation of the neural network used in Doddle_Detection. You’ll learn how layers propagate activations, how activations and cost functions are selected dynamically, how hyperparameters configure training, and how networks serialize to JSON for persistence.

Layer Forward Propagation

Purpose
Compute a layer’s outputs (activations) from its inputs, with optional intermediate storage for backpropagation.

How It Works

  • Each output node j calculates a weighted sum plus bias:
    weightedInputs[j] = biases[j] + Σ_i inputs[i] * weights[i,j]
  • Apply activation function per node.
  • Overloads:
    • CalculateOutputs(inputs) returns activations only.
    • CalculateOutputs(inputs, learnData) also records inputs, weightedInputs and activations in learnData.

Signatures 

public double[] CalculateOutputs(double[] inputs)
public double[] CalculateOutputs(double[] inputs, LayerLearnData learnData)

Key Steps

  1. Validate inputs.Length == numNodesIn.
  2. For each output j:
    • Compute weightedInputs[j]
    • Compute activations[j] = activation.Activate(weightedInputs, j)
  3. Return activations.

Code Examples

Direct inference (no intermediates):

// layer: 784 inputs → 128 outputs
double[] inputs = LoadImageVector();
double[] hidden = layer.CalculateOutputs(inputs);
// hidden: 128 values in [0,1]

Training pass (store intermediates):

// Pre-allocate once per layer
var learnData = new LayerLearnData(numNodesOut);

double[] activations = layer.CalculateOutputs(previousActivations, learnData);

// Later: use learnData.weightedInputs and learnData.activations in backprop

Practical Guidance

  • Reuse one LayerLearnData per layer per sample/batch to reduce GC.
  • Ensure activation functions implement both Activate(...) and Derivative(...).
  • In batched training, accumulate gradient updates across calls before ApplyGradients.
  • Keep inputs/outputs in contiguous arrays—avoid per-node objects.
  • To switch activation, call layer.SetActivationFunction(new Activation.ReLU()) before forward passes.

Activation Factory (GetActivationFromType)

Purpose
Dynamically retrieve an IActivation implementation from an ActivationType enum. Layers can swap activations without direct class references.

Factory Method

public static IActivation GetActivationFromType(ActivationType type)
{
  switch (type)
  {
    case ActivationType.Sigmoid: return new Sigmoid();
    case ActivationType.TanH:    return new TanH();
    case ActivationType.ReLU:    return new ReLU();
    case ActivationType.SiLU:    return new SiLU();
    case ActivationType.Softmax: return new Softmax();
    default:
      UnityEngine.Debug.LogError("Unhandled activation type: " + type);
      return new Sigmoid();
  }
}

Practical Usage

public class DenseLayer
{
  private readonly int neuronCount;
  private readonly IActivation activation;
  private double[] outputs;

  public DenseLayer(int count, ActivationType type)
  {
    neuronCount = count;
    activation  = Activation.GetActivationFromType(type);
    outputs     = new double[count];
  }

  public double[] Forward(double[] inputs, double[,] weights, double[] biases)
  {
    for (int i = 0; i < neuronCount; i++)
    {
      double sum = biases[i];
      for (int j = 0; j < inputs.Length; j++)
        sum += inputs[j] * weights[j, i];
      outputs[i] = activation.Activate(new[] { sum }, 0);
    }
    return outputs;
  }

  public double[] Backward(double[] zValues, double[] upstreamGradient)
  {
    var grad = new double[neuronCount];
    for (int i = 0; i < neuronCount; i++)
    {
      double dA = activation.Derivative(zValues, i);
      grad[i]   = upstreamGradient[i] * dA;
    }
    return grad;
  }
}

Best Practices

  • For Softmax, pass the full pre-activation array so Activate/Derivative compute across classes.
  • Cache IActivation per layer—avoid multiple GetActivationFromType calls in loops.
  • To add a new activation: extend ActivationType, implement IActivation, update the switch.

Retrieving Cost Function Instances

Purpose
Obtain and use ICost implementations (MSE or CrossEntropy) via Cost.GetCostFromType for loss computation and its derivatives.

Usage

public void TrainStep(double[][] inputs, double[][] expected, Cost.CostType costType)
{
    ICost cost = Cost.GetCostFromType(costType);

    // Forward pass
    double[][] predictions = network.Forward(inputs);

    // Compute batch loss
    double totalCost = 0;
    for (int i = 0; i < predictions.Length; i++)
        totalCost += cost.CostFunction(predictions[i], expected[i]);
    Debug.Log($"Batch loss ({cost.CostFunctionType()}): {totalCost}");

    // Backward: output layer gradients
    for (int s = 0; s < predictions.Length; s++)
    {
        var yPred = predictions[s];
        var yTrue = expected[s];
        for (int n = 0; n < yPred.Length; n++)
        {
            double delta = cost.CostDerivative(yPred[n], yTrue[n]);
            network.OutputLayer.Neurons[n].Delta = delta;
        }
    }
    // Continue weight updates…
}

Guidance

  • Pair CrossEntropy with sigmoid/softmax for stable gradients.
  • Use MSE for regression or symmetric penalization.
  • GetCostFromType logs and falls back to MSE on unknown types—ensure correct config.
  • Combine CostDerivative with activation derivatives during backprop.

HyperParameters Class

Purpose
Encapsulate network architecture and training settings in a serializable object.

Key Fields

  • layerSizes (int[]): including input and output sizes.
  • activationType (ActivationType): hidden layers.
  • outputActivationType (ActivationType): output layer.
  • costType (CostType): loss function.
  • initialLearningRate (double): starting η.
  • learnRateDecay (double): decay per epoch: newRate = initialLearningRate / (1 + learnRateDecay * epoch).
  • minibatchSize (int), momentum (double), regularization (double).

Default Constructor 

activationType       = ReLU  
outputActivationType = Softmax  
costType             = CrossEntropy  
initialLearningRate  = 0.05  
learnRateDecay       = 0.075  
minibatchSize        = 32  
momentum             = 0.9  
regularization       = 0.1

Code Example

var hp = new HyperParameters {
  layerSizes           = new[] { 784, 128, 64, 10 },
  activationType       = ActivationType.LeakyReLU,
  outputActivationType = ActivationType.Softmax,
  costType             = Cost.CostType.CrossEntropy,
  initialLearningRate  = 0.01,
  learnRateDecay       = 0.05,
  minibatchSize        = 64,
  momentum             = 0.8,
  regularization       = 0.001
};

var network = new NeuralNetwork(hp);
network.Train(trainingData, epochs: 30);

Guidance

  • First layerSizes element = feature length; last = classes/targets.
  • Typical minibatchSize: 32–128.
  • Momentum: 0.8–0.99 accelerates convergence.
  • Regularization: 0.0001–0.1 to control overfitting.
  • Use ReLU family + CrossEntropy/Softmax for classification.

NetworkSaveData: Persisting NeuralNetwork Instances

Purpose
Serialize and deserialize NeuralNetwork objects (architecture, weights, biases, activations, cost) via JSON.

Key Methods

  • string SerializeNetwork(NeuralNetwork network)
  • void SaveToFile(NeuralNetwork network, string path)
  • NeuralNetwork LoadNetworkFromFile(string path)
  • NeuralNetwork LoadNetworkFromData(string json)
  • Instance method NeuralNetwork LoadNetwork() builds from in-memory fields.

Usage

Saving:

NeuralNetwork trained = trainer.GetNetwork();
string path = Application.persistentDataPath + "/network.json";
NetworkSaveData.SaveToFile(trained, path);
Debug.Log("Saved at: " + path);

Loading:

void Awake()
{
  string path = Application.persistentDataPath + "/network.json";
  if (File.Exists(path))
    network = NetworkSaveData.LoadNetworkFromFile(path);
  else
    network = new NeuralNetwork(new[] {784, 64, 10});
}

In-Memory JSON:

string json = NetworkSaveData.SerializeNetwork(myNetwork);
// send or display json
NeuralNetwork net2 = NetworkSaveData.LoadNetworkFromData(json);

Implementation Notes

  • Uses UnityEngine.JsonUtility—no external libs.
  • ConnectionSaveData stores flat double[] for weights/biases plus ActivationType.
  • Re-links activations and cost via GetActivationFromType/GetCostFromType.
  • Wraps file I/O in using blocks for safe streams.

Tips

  • Verify write permissions for Application.persistentDataPath.
  • Update NetworkSaveData when adding new fields to NeuralNetwork.

5. Training & Evaluation Workflow

This section outlines the end-to-end process for training a neural network: preparing data, visualizing metrics in real time, and performing post-training evaluation.

5.1 Preparing Data with DataSetHelper

Split your raw dataset into training and validation sets, then create and shuffle mini-batches for iterative training.

Core Methods

  • SplitData(DataPoint[] data, float splitRatio, bool shuffleBeforeSplit)
  • CreateMiniBatches(DataPoint[] data, int batchSize, bool shuffleOnCreate)
  • ShuffleBatches(Batch[] batches)

Example: Splitting & Batching

// 1. Load all data
var allData = FindObjectOfType<ImageLoader>().GetAllData();

// 2. Split into train/validation
float splitRatio = 0.8f;
var (trainingData, validationData) = 
    DataSetHelper.SplitData(allData, splitRatio, true);

// 3. Create mini-batches
int minibatchSize = 32;
var trainingBatches = DataSetHelper.CreateMiniBatches(
    trainingData, minibatchSize, true
);

// 4. At end of each epoch, shuffle batches
DataSetHelper.ShuffleBatches(trainingBatches);

Tips:

  • Set splitRatio based on dataset size and desired validation coverage.
  • Choose minibatchSize to balance convergence stability and throughput.
  • Always shuffle before splitting or between epochs for unbiased training.

5.2 Configuring & Using the TrainingGraph Prefab

Visualize training and validation accuracy curves in your Unity scene.

Prefab Hierarchy

  • Training Graph (root, has TrainingGraph script)
    • Info (TMP_Text) – live accuracy & epoch progress
    • Train Graph (LineRenderer) – training accuracy line
    • Validation Graph (LineRenderer) – validation accuracy line
    • Axis Marker Text (TMP_Text) – template for tick labels
    • Axis X / Axis Y – static mesh lines

Inspector Fields (TrainingGraph.cs)

  • TMP_Text text – tick‐label template
  • TMP_Text evalText – info panel reference
  • int percentageIncrement = 5 – vertical tick step (%)
  • float epochSpacing = 0.25f – horizontal spacing per epoch
  • LineRenderer trainAccuracyGraph, validationAccuracyGraph

Runtime Behavior

  1. Awake()
    • Subscribes to NetworkTrainer.onTrainingStarted and onEpochComplete.
    • Generates axis tick labels based on percentageIncrement and epochSpacing.
  2. onEpochComplete(int epoch, float trainAcc, float valAcc)
    • Calls UpdateGraphs(epoch), adds new points.
    • Calls UpdateInfo(trainAcc, valAcc, percent) to refresh evalText.
  3. ClearGraphs() resets line renderers for a new training run.

Example: Instantiating & Customizing

public class GraphSpawner : MonoBehaviour
{
    public GameObject trainingGraphPrefab;
    public Material trainLineMat, valLineMat;

    void Start()
    {
        var graphGO = Instantiate(trainingGraphPrefab);
        var graph = graphGO.GetComponent<TrainingGraph>();

        // Override colors
        graph.trainAccuracyGraph.sharedMaterial = trainLineMat;
        graph.validationAccuracyGraph.sharedMaterial = valLineMat;

        // Tighter spacing for many epochs
        graph.epochSpacing = 0.1f;
        graph.percentageIncrement = 2;
    }
}

Checklist:

  • Drag Assets/Prefabs/Visualizers/Training Graph.prefab into your scene
  • Assign text, evalText, trainAccuracyGraph, validationAccuracyGraph in Inspector
  • Use an orthographic main camera for correct Y-mapping
  • Ensure a NetworkTrainer instance is present to fire events

5.3 Orchestrating the Training Loop

Tie together data helpers, network trainer, and graph visualizer.

public class NetworkTrainer : MonoBehaviour
{
    public float trainingSplit = 0.8f;
    public int minibatchSize = 32;
    public int epochs = 50;

    TrainingGraph graph;
    Batch[] trainingBatches;
    DataPoint[] validationData;
    NeuralNetwork network;

    void Start()
    {
        // Prepare data
        var allData = FindObjectOfType<ImageLoader>().GetAllData();
        (var trainData, validationData) = 
            DataSetHelper.SplitData(allData, trainingSplit, true);
        trainingBatches = DataSetHelper.CreateMiniBatches(trainData, minibatchSize, true);

        // Initialize network
        network = new NeuralNetwork(...);

        // Hook up graph
        graph = FindObjectOfType<TrainingGraph>();
        onTrainingStarted += graph.ClearGraphs;
        onEpochComplete += graph.UpdateGraphs;

        // Begin training
        StartCoroutine(TrainRoutine());
    }

    IEnumerator TrainRoutine()
    {
        onTrainingStarted?.Invoke();
        for (int e = 0; e < epochs; e++)
        {
            foreach (var batch in trainingBatches)
                network.Train(batch.inputs, batch.expectedOutputs);

            // Compute accuracies
            float trainAcc = network.EvaluateAccuracyOnBatch(trainingBatches);
            float valAcc   = network.EvaluateAccuracyOnData(validationData);

            onEpochComplete?.Invoke(e, trainAcc, valAcc);

            DataSetHelper.ShuffleBatches(trainingBatches);
            yield return null;
        }
    }

    public event Action onTrainingStarted;
    public event Action<int, float, float> onEpochComplete;
}

5.4 Post-Training Evaluation with NetworkEvaluator

Compute overall and per-class accuracy on any labeled dataset.

Signature

public static EvaluationData Evaluate(
    NeuralNetwork network, DataPoint[] data
)

Returns

EvaluationData containing:

  • total, numCorrect
  • totalPerClass[], numCorrectPerClass[]
  • wronglyPredictedAs[]

Example Usage

// After training completes
var testSet = validationData;
var results = NetworkEvaluator.Evaluate(network, testSet);

// Overall accuracy
Debug.Log(results.GetAccuracyString());

// Per-class breakdown
for (int i = 0; i < results.totalPerClass.Length; i++)
{
    int correct = results.numCorrectPerClass[i];
    int total   = results.totalPerClass[i];
    Debug.Log($"Class {i}: {correct}/{total} ({(correct/(double)total)*100:F2}%)");
}

// Common misclassifications
for (int pred = 0; pred < results.wronglyPredictedAs.Length; pred++)
    Debug.Log($"Misclassified as {pred}: {results.wronglyPredictedAs[pred]} times");

Practical tips:

  • Call Evaluate after key epochs to monitor learning curves.
  • Use per-class metrics to uncover data imbalance.
  • Combine with confusion-matrix visualizers for deeper insights.

6. Interactive UI & Tools

This section covers the user-facing interaction layer: GPU-accelerated brush drawing, confidence overlays, and the Image Viewer prefab for browsing datasets.

GPU Brush Stroke Compute Shader

Render and erase brush strokes on a 2D canvas entirely on the GPU, tracking the damaged region via a bounds buffer.

Shader Inputs and Workflow

  • Canvas (RWTexture2D): target draw surface
  • bounds (RWStructuredBuffer[4]): xMin, xMax, yMin, yMax
  • resolution (uint): canvas width/height
  • brushCentre, brushCentreOld (int2): current/previous brush positions
  • brushRadius (float): radius in texels
  • smoothing (float 0–1): falloff control
  • mode (int): 0=draw, 1=erase

Workflow per thread (one pixel at id.xy):

  1. Discard if id outside [0,resolution).
  2. Compute distance to segment brushCentreOld→brushCentre.
  3. If dst < brushRadius:
    • val = 1 – smoothstep(brushRadius*(1−smoothing), brushRadius, dst)
    • mode 0: Canvas[id.xy] = max(val, Canvas[id.xy])
    • mode 1: Canvas[id.xy] = max(0, Canvas[id.xy] – val)
  4. If Canvas[id.xy].r ≠ 0, atomically update bounds via InterlockedMin/Max.

Key Shader Functions

float smoothstep(float minVal, float maxVal, float t) {
    t = saturate((t - minVal) / (maxVal - minVal));
    return t * t * (3 - 2 * t);
}

float dstToLineSegment(float2 a, float2 b, float2 p) {
    float2 ab = b - a, ap = p - a;
    float len2 = dot(ab, ab);
    if (len2 == 0) return length(ap);
    float t = clamp(dot(ap, ab) / len2, 0, 1);
    return length(p - (a + ab * t));
}

C# Setup and Dispatch

In DrawingController.Update():

// Convert mouse to canvas texels
Vector2 mouseWorld = cam.ScreenToWorldPoint(Input.mousePosition);
Bounds cb = canvasCollider.bounds;
float u = Mathf.InverseLerp(cb.min.x, cb.max.x, mouseWorld.x);
float v = Mathf.InverseLerp(cb.min.y, cb.max.y, mouseWorld.y);
Vector2Int brushCentre = new Vector2Int(
    (int)(u * drawResolution),
    (int)(v * drawResolution)
);

// Set compute parameters
drawCompute.SetInts("brushCentre", brushCentre.x, brushCentre.y);
drawCompute.SetInts("brushCentreOld", brushCentreOld.x, brushCentreOld.y);
drawCompute.SetFloat("brushRadius", brushRadius);
drawCompute.SetFloat("smoothing", smoothing);
drawCompute.SetInt("resolution", drawResolution);
drawCompute.SetInt("mode", Input.GetMouseButton(0) ? 0 : 1);

// Dispatch on mouse down/hold
if (Input.GetMouseButton(0) || Input.GetMouseButton(1))
    ComputeHelper.Dispatch(drawCompute, drawResolution, drawResolution);

brushCentreOld = brushCentre;

Practical Usage Guidance

  • Use drawResolution divisible by 8 to match numthreads(8,8,1).
  • smoothing = 0 yields a hard brush; near 1 produces soft falloff.
  • Read back damaged rectangle: 
    uint[] rect = new uint[4];
boundsBuffer.GetData(rect); // [xMin, xMax, yMin, yMax]
  • To reset canvas and bounds:
    boundsBuffer.SetData(new uint[]{ drawResolution-1, 0, drawResolution-1, 0 });
ComputeHelper.ClearRenderTexture(canvas);
  • Batch multiple strokes per frame for multi-touch or multi-brush setups.

Displaying Network Prediction and Confidence Scores

Process neural network outputs each frame, sort by confidence, and update both the 3D preview and on-screen UI.

Implementation Details

  • In Update(), capture the user’s drawing as a RenderTexture, convert to Image, then call network.Classify(), returning (predictionIndex, outputProbabilities[]).
  • UpdateDisplay():
    1. Map each probability to its label via ImageLoader.LabelNames.
    2. Wrap in RankedLabel structs (formats as “XX.XX%”).
    3. Sort descending by score.
    4. Build two TMP_Text fields (labelsUI, confidenceUI), highlighting the top result.
    5. Assign the processed drawing to display.material.mainTexture.

Code Examples

UpdateDisplay core loop:

void UpdateDisplay(Image image, double[] outputs, int prediction) {
    var ranked = new List<RankedLabel>();
    for (int i = 0; i < outputs.Length; i++)
        ranked.Add(new RankedLabel {
            name  = loader.LabelNames[i],
            score = (float)outputs[i]
        });

    ranked.Sort((a, b) => b.score.CompareTo(a.score));

    labelsUI.text     = "<color=#ffffff>";
    confidenceUI.text = "<color=#ffffff>";
    for (int i = 0; i < ranked.Count; i++) {
        bool top = (i == 0);
        labelsUI.text     += ranked[i].name + "\n" + (top ? "</color>" : "");
        confidenceUI.text += ranked[i].Text + "\n" + (top ? "</color>" : "");
    }

    display.material.mainTexture = image.ConvertToTexture2D();
}

RankedLabel struct:

public struct RankedLabel {
    public string name;
    public float  score;  // raw [0..1]

    public string Text => $"{score * 100:0.00}%";
}

Usage Tips

  • Attach to a GameObject with a MeshRenderer for the 3D preview.
  • In the Inspector, assign: • networkFile: your .txt model asset
    • display: the preview mesh’s MeshRenderer
    • labelsUI, confidenceUI: two TMP_Text components
  • Ensure an ImageLoader exists with correct LabelNames.
  • Provide a 28×28 (or expected size) RenderTexture from DrawingController.RenderOutputTexture().
  • The top prediction appears in bright white; others fade automatically.

Image Viewer Prefab Setup

Quickly configure and instantiate the Image Viewer UI for browsing images and running predictions.

Prefab Structure

Assets/Prefabs/Visualizers/Image Viewer.prefab contains:

  • Root Image Viewer (with ImageViewer MonoBehaviour)
  • Child bindings in Inspector: • RawImage → display
    • TMP_Text → label
    • Button → prevButton
    • Button → nextButton
    • Button → randomizeButton

Inspector fields in ImageViewer:

public int displayIndex;         // start index
public TextAsset networkFile;    // optional model

[Header("UI")]
public RawImage display;
public TMPro.TMP_Text label;
public Button prevButton;
public Button nextButton;
public Button randomizeButton;

Scene Setup

  1. Drag Image Viewer prefab into your scene.
  2. Ensure one ImageLoader exists to supply images and labels.
  3. (Optional) Assign your trained network .txt to networkFile.
  4. Set displayIndex for the initial image.

Runtime Behavior

  • Prev/Next/Randomize buttons update displayIndex.
  • On change, ImageViewer calls ImageLoader.GetImage(displayIndex), converts to a Texture2D, and sets display.texture.
  • If networkFile is set, it classifies the image and appends the prediction to label.text.

Instantiation & Customization in Code

var prefab   = Resources.Load<GameObject>("Visualizers/Image Viewer");
var viewerGO = Instantiate(prefab);
var viewer   = viewerGO.GetComponent<ImageViewer>();

viewer.displayIndex = 5;
viewer.networkFile  = myTrainedNetworkTextAsset;

// Wire an external button to re-run classification
myEvaluateBtn.onClick.AddListener(viewer.EvaluateNetwork);

Tips

  • displayIndex auto-clamps between 0 and ImageLoader.NumImages-1.
  • Theme prediction text using <color=#…> tags; ImageViewer uses a private helper SetTextColour.
  • Call EvaluateNetwork() manually to log overall accuracy across the dataset.

7. Extending & Contributing

This section guides you through extending Doddle_Detection’s core architecture, adding custom visualizers or data loaders, and submitting changes.

7.1 Contribution Workflow

  1. Fork the repository and create a feature branch:
    git clone git@github.com:your-user/Doddle_Detection.git
cd Doddle_Detection
git checkout -b feature/your-feature
  2. Write code, tests, and documentation for your feature.
  3. Run existing tests and linters:
    pytest --maxfail=1 --disable-warnings -q
flake8 doddle
  4. Commit with a descriptive message, push your branch, and open a pull request against main.

7.2 Coding Standards & Testing

  • Follow PEP8 conventions. Doddle_Detection uses black for formatting.
    pip install black flake8
black .
flake8 doddle tests
  • Write unit tests under tests/. Use pytest fixtures where appropriate.
  • Aim for clear docstrings and type hints on all public classes/functions.

7.3 Extending the Detection Architecture

Doddle_Detection splits logic into data/, models/, trainer/, and visualizers/. To add a new detector:

  1. Create your model in doddle/models/custom_detector.py:
    # doddle/models/custom_detector.py
from doddle.models.base import BaseDetector, register_model
import torch.nn as nn

@register_model("custom_cnn")
class CustomCNN(BaseDetector):
    """
    Example custom CNN detector.
    """

    def __init__(self, num_classes: int = 2):
        super().__init__()
        self.feature_extractor = nn.Sequential(
            nn.Conv2d(3, 16, 3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2),
        )
        self.classifier = nn.Linear(16 * 128 * 128, num_classes)

    def forward(self, x):
        features = self.feature_extractor(x)
        flat = features.view(features.size(0), -1)
        return self.classifier(flat)
  2. Update your config to use "model": "custom_cnn".
  3. Write a unit test in tests/test_custom_detector.py:
    import torch
from doddle.models import get_model

def test_custom_cnn_forward():
    model = get_model("custom_cnn", num_classes=3)
    dummy = torch.randn(2, 3, 256, 256)
    out = model(dummy)
    assert out.shape == (2, 3)

7.4 Integrating New Visualizers

Visualizers implement BaseVisualizer under doddle/visualizers/. To add one:

  1. Create doddle/visualizers/score_heatmap.py:
    # doddle/visualizers/score_heatmap.py
import matplotlib.pyplot as plt
from doddle.visualizers.base import BaseVisualizer

class ScoreHeatmap(BaseVisualizer):
    """
    Overlay a heatmap of prediction scores on the input image.
    """

    def visualize(self, image, predictions, save_path=None):
        """
        image: HxWx3 ndarray
        predictions: HxW float scores
        """
        fig, ax = plt.subplots()
        ax.imshow(image)
        ax.imshow(predictions, cmap='hot', alpha=0.5)
        if save_path:
            plt.savefig(save_path)
        return fig
  2. Register it in your config:
    visualizers:
  - type: score_heatmap
    params:
      alpha: 0.6
  3. Test manually by calling in script:
    from doddle.visualizers import build_visualizer

vis = build_visualizer("score_heatmap")
fig = vis.visualize(image_array, score_array, save_path="out.png")

7.5 Adding New Data Loaders

  1. Implement in doddle/data/my_loader.py:
    from torch.utils.data import Dataset
from doddle.data.base import register_loader

@register_loader("my_dataset")
class MyDataset(Dataset):
    def __init__(self, root_dir, transform=None):
        self.paths = list_paths(root_dir)
        self.transform = transform

    def __len__(self):
        return len(self.paths)

    def __getitem__(self, idx):
        image, label = load_pair(self.paths[idx])
        if self.transform:
            image = self.transform(image)
        return image, label
  2. Reference in config:
    dataset:
  type: my_dataset
  root_dir: /path/to/data
  transforms:
    - Resize: {size: [256, 256]}
    - ToTensor: {}
  3. Add a unit test in tests/test_my_loader.py to ensure batching and transforms.

7.6 Submitting Your Pull Request

  • Ensure your branch is rebased onto the latest main.
  • Include:
    • A clear PR title and description.
    • Updated documentation in docs/ if adding user-facing features.
    • Passing CI (tests + lint).
  • Address review feedback promptly.

Thank you for contributing to Doddle_Detection!