You can create a validation base; i.e. a set of character images from your dataset with the label (like image of an "A" with the label A). Labelizing 100 images won't take long.