Two ways to write a letter: DCGAN versus LSTM-MDN
Handwriting can be modelled as a picture or as a movement. I built both on the same EMNIST letters and measured what each one actually captures — because they fail in completely different ways.
Two ways to frame the same thing
A handwritten letter is a picture. It is also a movement — a pen going down, travelling, lifting. Those two descriptions suggest completely different models, and which one you pick decides what the model can and cannot get wrong.
Framed as an image, generation is a texture problem: produce a 28×28 grid of pixels that looks like a letter. Framed as a movement, it is a sequence problem: predict where the pen goes next, and when it lifts. A model that has never represented a pen cannot produce an impossible stroke, because it is not producing strokes at all — it is producing pixels that resemble the result of strokes.
What I built
Both models train on EMNIST letters, so the comparison is on shared ground. The image side is a DCGAN over 28×28 grayscale letters. The sequence side is an LSTM with a mixture density network head, run over stroke trajectories derived from the same EMNIST images — a mixture density output because the next pen position is genuinely multi-modal, and a single predicted point would average incompatible options into mush.
A separate small CNN is trained as a realism classifier, which gives both generators a common yardstick instead of judging samples by eye. Stroke sequences are capped at 160 points with 20 mixture components.
What the metrics show
| Metric | DCGAN | LSTM-MDN |
|---|---|---|
| Classifier confidence (mean) | 0.706 | 0.603 |
| Class entropy | 3.174 | 2.607 |
| Mean pen lifts per sequence | — | 30.4 |
| Mean stroke length | — | 8.76 |
| Stroke smoothness (mean abs turn, rad) | — | 0.868 |
The DCGAN leads on both shared metrics — its samples read as more letter-like to the classifier, and it covers the class space more evenly. That is the expected result: it is optimising directly for the thing being measured, which is how the image looks.
The stroke metrics have no DCGAN column at all, and that absence is the interesting part. Pen lifts, stroke length and smoothness are only defined for a model that represents a pen. The LSTM-MDN scores lower on looking right while being the only one of the two that has any notion of how the letter was made. Which model is better depends entirely on which of those you needed.