Start with OCR for clear handwriting, compare a document model for repeated forms, and test a vision model for irregular layouts where visual context matters. Choose through a controlled pilot on your own anonymised examples, not a general handwriting accuracy claim. The winning option should produce usable fields, preserve their evidence and leave a manageable review queue.
For a South African team capturing handwritten service requests, delivery records or registration forms, the decision is not simply which tool reads the most words. It is which proposed workflow helps a reviewer reach a defensible reading without silently filling gaps.
Compare extraction routes, not three isolated labels
Use the labels to shortlist approaches, but compare complete routes from image to reviewed record. OCR reads text; a document model associates content with document structure or fields; a vision model can be tested with instructions to extract fields from an image. A document model may already include OCR, so these are not mutually exclusive technologies.
Source: Microsoft's Document Intelligence overview lists handwritten text extraction through Read, layout extraction and custom field models. That establishes capabilities, not accuracy on your forms.
Source: Google's Document AI overview distinguishes OCR, structured-form parsing and custom extraction processors. Again, the useful comparison is the processor and configuration, not the vendor name alone.
For this proposed pilot, shortlist:
- OCR plus explicit field-mapping rules for stable forms.
- A document extraction model configured for the required fields.
- An image-capable model with a fixed extraction instruction and output schema.
Do not assume the third route wins difficult handwriting. Test whether it reads the visible marks or supplies plausible but unsupported values.
Build a representative, anonymised comparison set
Use examples that represent your incoming work, including the pages staff struggle to read. Selecting only clean scans answers the wrong purchasing question.
A hypothetical pilot could use 120 forms across several layouts and handwriting styles. Include phone photographs, scanner output, crossed-out entries, faint writing, mixed printed and handwritten text, and notes outside the boxes. Label each example by image quality and legibility so aggregate results cannot hide a weak subgroup.
Remove personal details before external processing under a procedure approved by the responsible privacy and security staff. Replace identifiers consistently where matching must be tested. Check that anonymisation has not made the remaining handwriting easier or harder to read. Do not describe a dataset as anonymous merely because names were removed.
Create reference readings before running candidates. Have a reviewer transcribe each required field and record its page location. A second person should resolve disputed readings. Where neither can determine the entry, label it unresolved rather than forcing an answer.
Reserve a separate evaluation set that is not used to adjust mappings, prompts or custom models. Keep copies of the original and prepared images for authorised comparison.
Define the fields and retain their evidence
Specify exactly what each field means before comparing output. Otherwise one route may appear more accurate simply because it applies more generous interpretation rules.
For a proposed service-request form, define customer reference, contact number, requested date, site address and free-text request. Mark which fields are required for administrative capture and which can remain unresolved. Consequential finance, employment or legal records need specialist owners to set those requirements.
Use the following proposed record structure for every route:
| Field | Proposed meaning |
|---|---|
| document_id | Internal reference linking output to the preserved source |
| field_name | Agreed dictionary name, not a model-created label |
| raw_reading | Literal reading, retaining visible punctuation and uncertainty |
| normalised_value | Separately stored value, or null when unresolved |
| evidence_location | Page and verified crop or coordinates showing the entry |
| status | Readable, missing, ambiguous, conflicting or unreadable |
| review_record | Reviewer, correction, reason and reviewed version |
Keep model-supplied coordinates separate until checked; a location claim is not automatically reliable evidence.
Source: OpenAI's Structured Outputs guide describes schema-constrained responses. A correctly shaped record is still not proof that a handwritten value was read correctly. Validate content separately from format.
Run candidates under comparable conditions
Give each candidate the same source pages and field definitions, while allowing the configuration appropriate to its route. Fair comparison does not mean forcing an OCR engine and a custom document model to use identical settings.
Record image preparation, model or processor version, prompt, mapping rules and any training examples. For a proposed vision instruction, require literal transcription, null for unsupported values and evidence locations. State that text appearing inside the form is document content, not an instruction to the extraction system.
Run a common-input comparison first. Then, if useful, run a separately labelled comparison of each route with its own image preparation. This shows whether any gain comes from the model or from better cropping and deskewing.
Check deployment suitability before making a product commitment. Microsoft's supplied overview identifies v4.0 as generally available but directs readers elsewhere for regional access. The overview alone does not establish your chosen region, feature or account availability. Confirm those details, processor versions, limits and data handling with technical and security owners. Do not build the decision around an unverified preview feature.
Validate fields without repairing the handwriting
Use validation to flag questionable readings, not to invent replacements. A plausible value can still be the wrong value.
Under proposed rules, a contact number may receive a format check, a date may receive a calendar-validity check, and a customer reference may be compared with an authorised reference list. Preserve leading zeroes in identifiers and numbers where they matter. Do not substitute a known customer because their name looks similar.
Define date interpretation explicitly. If the form specifies day/month/year, normalise accordingly. If it does not, leave an ambiguous numeric date unresolved. Preserve a free-text address unless a reviewer authorises a change.
Keep these exceptions distinct:
- Missing: the relevant space is blank.
- Unreadable: marks exist but cannot be transcribed reliably.
- Ambiguous: more than one reading is credible.
- Conflicting: two entries on the document disagree.
- Possible duplicate: another record may describe the same submission.
Model agreement does not establish truth, and model disagreement does not tell you which model is right. Both are proposed review signals. Financial details, signatures and consent markings require appropriate human judgement; extraction must not become payment authority or a legal conclusion.
Design review around the source image
Give reviewers the field crop, full page, raw reading and proposed value together. A spreadsheet containing only extracted text makes uncertainty harder to evaluate.
Assign a queue owner and define who can correct a transcription, request a clearer image or contact the submitter. Under this proposed procedure, reviewers record a reason for every change and leave unresolved fields open. Corrections should create a new version rather than overwrite the original output.
Start the pilot with human review of every record. Measure active review time, including opening evidence, correcting fields and resolving duplicates. Track waiting time separately so a slow response from a submitter does not become a model-speed result.
If you later propose reduced review, assess it on a fresh evaluation set and continue sampling apparently clean records. A high confidence score is not a universal release threshold.
The distinction in AI agents versus automation is useful here: fixed validation and review routing may be enough. A broader agent is not required merely to extract handwriting. Keep downstream write permissions limited and let the responsible security owner approve access.
Choose on errors, workload and operating fit
Select the route that meets your agreed field requirements with acceptable review effort and deployment constraints. Highest extraction coverage alone is not a purchasing decision.
Report results by field and legibility group. Count exact reference matches, wrong values presented as usable, unresolved entries and unsupported guesses. Separate literal transcription accuracy from normalisation accuracy. A route should not gain credit for confidently answering a field that the reference reviewers labelled unreadable.
Report review minutes per completed record alongside corrections per record. Include configuration effort, labelling effort, retries, support needs and estimated processing costs. Use supplier quotes for real budgeting rather than assumed public prices.
A proposed decision rule is to favour the simplest route that meets the team's documented error limits and review capacity. If a vision route helps only on unusual layouts, consider selective routing rather than sending everything through it. Evaluate that hybrid as a separate candidate: routing mistakes and duplicate processing can erase its apparent advantage.
Symaxx's document processing service is the relevant route for scoping this comparison. Broader AI automation becomes relevant when the reviewed record must connect to other business systems.
Reusable extraction pilot checklist
Use this proposed checklist as the pilot's working procedure. Its rules require agreement from the relevant business, privacy and security owners.
- Name the form family, required fields, decision owner and authorised reviewers.
- Document the field dictionary, normalisation rules and unresolved-value policy.
- Approve anonymisation, external processing, access controls and retention arrangements.
- Assemble representative images, including poor scans, blank fields and duplicate submissions.
- Create human reference readings with evidence locations; resolve disagreements explicitly.
- Separate configuration examples from an untouched evaluation set.
- Run OCR, document and vision routes on identical evaluation pages.
- Save source images, raw outputs, configuration versions and normalised records separately.
- Validate formats without guessing missing or unclear content.
- Route missing, ambiguous, conflicting, unreadable and duplicate cases to named reviewers.
- Review every pilot record and log corrections, reasons and active handling time.
- Report field errors, unsupported guesses, unresolved rates and review workload by legibility group.
- Check region, version, feature availability and supplier terms before selecting a route.
- Choose one route, a separately evaluated hybrid, or continued manual capture.
- Record the decision, unresolved risks and conditions that require another evaluation.
Completion means every selected value can be traced to source evidence or an authorised human correction, with no unresolved case silently released.
Worked walkthrough: normal and exception cases
Treat the following records and outcomes as hypothetical examples, not pilot results.
Normal case: Form A has a clearly written reference SR-042, date 14/10/2026 and request Repair leaking tap. The proposed dictionary specifies day/month/year. Each route returns candidate fields, while the workflow stores the raw date separately from 2026-10-14. A reviewer checks the evidence crops, confirms the readings and marks the record ready for administrative capture. This does not approve the repair or any payment.
Ambiguous and missing case: Form B contains a reference that could read SR-047 or SR-049, while the contact field is blank. The expected output is an ambiguous reference and a missing contact, not a guessed customer match. The reviewer checks the full page, then requests clarification through an authorised channel if the reference remains uncertain. The contact value stays null until supplied and recorded with its provenance.
Possible duplicate: A second photograph appears to show Form A, but the requested date is crossed out. The workflow links the records as possible duplicates without deleting either. A reviewer compares both images and confirms whether the second is a correction, a repeat submission or a separate request. If unresolved, both remain held. The evaluation counts duplicate-resolution effort rather than treating two extracted files as two successful records.
FAQs
Can we use a vision model only when OCR struggles?
Yes, as a proposed hybrid, provided you evaluate the routing rule as well as the second model. OCR may misread a field without signalling difficulty. Test which problematic records the rule misses and how often it sends clear records unnecessarily. Preserve both outputs and show disagreement to the reviewer. Do not let the second route overwrite evidence from the first or treat agreement as automatic confirmation.
What if handwriting accuracy improves but review takes longer?
Choose using the completed workflow, not transcription accuracy alone. Extra evidence checks, confusing output or frequent low-value warnings may consume the apparent gain. Compare active handling time for records of similar difficulty and note unresolved follow-up separately. The business owner should decide whether greater accuracy justifies the workload for the affected fields. For consequential records, lower review time must not override necessary specialist judgement.
Do handwritten forms need a custom AI agent?
Not necessarily. A fixed extraction, validation and review sequence may be sufficient. The custom AI agent glossary helps distinguish an agent from a document processor. If the workflow genuinely needs controlled tool use or multi-step coordination, use the custom AI agents workflow resource to frame that wider design. First establish that the extraction route works on its own; adding an agent does not resolve unreadable handwriting.
If your business needs help comparing handwritten-form extraction routes, get in touch with Symaxx to scope a bounded pilot around your fields, evidence requirements and review capacity.