Prompt Engineering for Data Extraction and Classification

The Extraction Prompt Pattern

Every extraction prompt follows the same core pattern: define the fields you want, specify the format for each field, provide rules for ambiguous or missing cases, and structure the output as JSON (or another machine-parseable format). The quality difference between an amateur extraction prompt and a production one is in the specificity of field definitions and edge case handling.

Basic pattern that works for most extraction tasks:

"Read the following text and extract these fields:
- customer_name: Full name of the person writing. If only first name given, use that. If no name present, return null.
- company: Company or organization name. If not mentioned, return null.
- issue_type: One of [billing, technical, account, shipping, product_quality, other].
- order_number: Any reference number mentioned (formats: #12345, ORDER-12345, or just a number they identify as an order). Return null if not mentioned.
- urgency: 'high' if they mention a deadline, revenue impact, or production downtime. 'medium' if they express frustration or have been waiting. 'low' for general inquiries with no time pressure.
- summary: One sentence describing their core request in 20 words or fewer.

Return ONLY valid JSON matching this structure. No additional text."

This pattern works because every field has: a clear name, a defined data type or value set, rules for common ambiguities, and explicit null handling for missing information.

Field Definition Best Practices

Use Enums Over Free Text

Wherever possible, constrain fields to a defined set of values rather than allowing free text. "issue_type: one of [billing, technical, account, shipping]" produces consistent, parseable output. "issue_type: describe the type of issue" produces unpredictable strings that break downstream systems. Free text fields should only be used for genuinely open-ended information like summaries or descriptions.

Define Format Normalization Rules

Real-world data comes in inconsistent formats. Phone numbers might be "(555) 123-4567" or "5551234567" or "+1-555-123-4567." Dates might be "January 5" or "1/5/26" or "5th Jan." Your prompt should specify the normalized format: "phone: digits only, 10 digits for US (5551234567), include country code for international. date: ISO 8601 format YYYY-MM-DD. If year not stated, assume current year."

Handle Ambiguity Explicitly

What happens when a customer mentions two different issues in one email? What if they reference "my account" without giving an account number? Your prompt must address these cases: "If multiple issues are mentioned, classify by the PRIMARY issue (the one they seem most concerned about). If a reference number is ambiguous (could be order or ticket number), include it in the order_number field and add a flag field: ambiguous_reference: true."

Confidence Scoring

Production extraction systems need to know when they are uncertain. A field extracted with high confidence can be processed automatically. A field with low confidence should be flagged for human review. Your prompt should include confidence scoring instructions:

"For each extracted field, provide a confidence score (1-10):
- 10: The information is explicitly stated verbatim in the text
- 7-9: The information is clearly implied or stated in slightly different words
- 4-6: The information required inference from context
- 1-3: The information is a best guess based on limited evidence

Flag any field with confidence below 6 for human review."

This scoring lets your system route high-confidence extractions directly to processing while queuing low-confidence ones for human verification. The result is full automation for clear-cut cases (typically 70-85% of inputs) with human oversight only where the AI is genuinely uncertain.

Classification Hierarchies

When extracting categories from text, flat category lists work for simple tasks (5-8 categories). For complex categorization (20+ categories), use hierarchical classification: first classify into a broad category, then sub-classify within that category.

Flat approach (simple): "Classify as one of: billing, technical, account, shipping, other."

Hierarchical approach (complex): "First, classify the primary department: billing, technical, customer_success, operations. Then, classify the sub-category within that department. Billing sub-categories: charge_dispute, refund_request, plan_change, invoice_question. Technical sub-categories: bug_report, performance_issue, integration_failure, feature_question. Return both: {department: string, sub_category: string}."

Hierarchical classification improves accuracy because each decision is simpler. Choosing between 4 departments is easier than choosing between 20 specific categories. Once the department is decided, choosing between 4-5 sub-categories within that department is also straightforward. The combined accuracy of two simple decisions is typically higher than one complex decision.

Multi-Entity Extraction

Some documents contain multiple entities to extract: all line items from an invoice, all participants mentioned in a meeting transcript, all action items from an email thread. These require array-based extraction patterns.

"Extract all line items from the following invoice. For each line item, extract: description (string), quantity (integer), unit_price (number with 2 decimal places), total (number with 2 decimal places). Return as a JSON array of objects. Also extract the invoice-level fields: vendor_name, invoice_number, invoice_date (YYYY-MM-DD), total_amount, tax_amount, due_date."

For multi-entity extraction, add validation rules: "The sum of all line item totals plus tax_amount should equal total_amount. If they do not match, flag discrepancy: true and include both the calculated total and the stated total." This self-consistency check catches extraction errors before they propagate downstream.

Handling Noisy and Malformed Input

Real-world text is messy: typos, informal abbreviations, multiple languages, copy-paste artifacts, email signatures mixed into message body, quoted reply text from previous messages. Your prompt must handle this gracefully.

Noise handling instructions: "Ignore email signatures, disclaimers, and quoted reply text (lines starting with > or prefixed with 'On [date], [name] wrote:'). Extract information only from the actual message content. If the text contains obvious typos (e.g., 'accont' for 'account'), interpret the intended meaning. If abbreviations are used (pls, thx, bc), interpret them normally."

For OCR text (scanned documents), add: "This text was extracted via OCR and may contain recognition errors. Common OCR errors: 0 confused with O, 1 confused with l, $ confused with S. Use context to resolve ambiguous characters, especially in numbers and reference codes."

Extraction From Long Documents

When extracting from documents longer than a few paragraphs, you face context window limitations and attention degradation. Two approaches handle this:

Chunked extraction: Split the document into sections, run extraction on each section independently, then merge results. This works for documents with repeated structures (invoices with many line items, meeting notes with multiple discussion topics). Your prompt processes one chunk at a time with focused attention.

Targeted extraction with location hints: For documents where information is scattered (contracts, legal documents), tell the model where to look: "The parties to this agreement are typically stated in the first paragraph. The effective date is usually near the signature section. The payment terms are typically in sections titled 'Payment,' 'Compensation,' or 'Fees.' Look in these locations first." This focuses the model's attention on the most likely locations, improving accuracy on long documents.

Testing Extraction Accuracy

Extraction prompts are the easiest to test because you can measure field-level accuracy objectively. Build a test set of 30-50 real documents with human-verified correct extractions. For each test run, measure:

• Field-level accuracy: What percentage of fields are extracted correctly? Measure per field, not just overall, because some fields are harder than others.
• Null handling: When information is genuinely absent, does the model return null (correct) or fabricate a value (incorrect)?
• Format compliance: Are dates in the right format? Are enums within the allowed values? Is the JSON valid?
• Confidence calibration: When the model reports high confidence, is it actually correct? When it reports low confidence, is it actually uncertain?

Target accuracy thresholds: 95%+ for fields that are explicitly stated (names, numbers, dates clearly present in text), 85%+ for fields requiring inference (urgency assessment, intent classification), 99%+ for format compliance (valid JSON, correct enum values). See How to Test and Iterate on AI Prompts for the full testing methodology.

Combining Extraction With Classification

Most real-world applications need both: extract the structured data AND classify the input into a category. Run these as a single prompt to save API calls and latency. The extraction context actually helps classification (seeing the order number confirms it is an order-related inquiry), and the classification context helps extraction (knowing it is a billing issue makes the model look harder for invoice references).

Combined pattern: "Extract fields AND classify. The extraction and classification should inform each other. If you extract an order number, this increases confidence that issue_type is 'shipping' or 'product_quality' rather than 'technical.' If the language indicates frustration, increase the urgency score. Return everything in a single JSON response."

This integrated approach matches how human agents process incoming requests: they read the message once and simultaneously understand what it is about (classification) and what specific details are relevant (extraction). See Prompt Engineering for Business Applications for more combined patterns.