Building LLM-Powered Internal Tools: Natural Language to SQL, Document Q&A, and Workflow Automation

Most LLM demos look the same: a clean UI, a clever prompt, a screenshot. What you rarely see is what happens after the demo: the SQL query that drops a table, the retrieval that silently returns the wrong document, the workflow automation that fires twice because no one thought about idempotency.

This is a guide for building LLM-powered internal tools that hold up beyond the demo. Three use cases: natural language to SQL, document Q&A over internal knowledge bases, and workflow automation with LLM routing and execution. For each, here is what actually works, what fails quietly, and how to manage cost at scale.

---

Natural Language to SQL

The Setup

Users ask questions in plain English, the LLM generates a SQL query, you execute it, and return results. The workflow sounds simple. The failure surface is not.

The core risk is not hallucination in the philosophical sense. It is the LLM generating syntactically valid SQL that does the wrong thing: an unbounded SELECT *, a missing WHERE clause, a DELETE instead of a SELECT, a cross-join on two large tables that takes your analytics database down for ten minutes.

Schema Injection

The LLM needs schema context to generate accurate queries. Inject it as a system prompt. Do not inject your entire schema if you have 200 tables. Inject the relevant subset.

interface SchemaContext {
  tables: TableDefinition[];
  relationships: ForeignKeyHint[];
  examples: QueryExample[];
}

interface TableDefinition {
  name: string;
  description: string;
  columns: ColumnDefinition[];
  rowCount?: number; // helps the model reason about join costs
}

interface ColumnDefinition {
  name: string;
  type: string;
  nullable: boolean;
  description: string;
  sampleValues?: string[]; // critical for enum-like columns
}

function buildSchemaPrompt(context: SchemaContext): string {
  const tablesDDL = context.tables
    .map((t) => {
      const cols = t.columns
        .map(
          (c) =>
            `  ${c.name} ${c.type}${c.nullable ? "" : " NOT NULL"} -- ${c.description}${
              c.sampleValues ? ` (e.g. ${c.sampleValues.slice(0, 3).join(", ")})` : ""
            }`
        )
        .join("\n");
      return `-- ${t.description}\nCREATE TABLE ${t.name} (\n${cols}\n);`;
    })
    .join("\n\n");

  const examplesBlock = context.examples
    .map((e) => `-- Q: ${e.question}\n${e.sql}`)
    .join("\n\n");

  return `You are a SQL query generator for a PostgreSQL database.

SCHEMA:
${tablesDDL}

RELATIONSHIPS:
${context.relationships.map((r) => `${r.from} -> ${r.to} via ${r.column}`).join("\n")}

EXAMPLES:
${examplesBlock}

Rules:
- Generate SELECT queries only. Never INSERT, UPDATE, DELETE, DROP, or TRUNCATE.
- Always include a LIMIT clause, maximum 10000 rows.
- Use explicit column names, never SELECT *.
- If you cannot answer the question safely, explain why instead of guessing.`;
}

The sampleValues field is worth the extra work. Column names like status or type are ambiguous without knowing what values are valid. Injecting three representative values eliminates a large category of errors.

Query Validation Before Execution

Never execute LLM-generated SQL directly. Parse and validate it first.

import { Parser } from "node-sql-parser";

type ValidationResult =
  | { valid: true; normalizedSql: string }
  | { valid: false; reason: string };

function validateGeneratedSql(sql: string): ValidationResult {
  const parser = new Parser();

  let ast;
  try {
    ast = parser.astify(sql, { database: "PostgreSQL" });
  } catch (e) {
    return { valid: false, reason: `Parse error: ${(e as Error).message}` };
  }

  // Reject non-SELECT statements
  const stmts = Array.isArray(ast) ? ast : [ast];
  for (const stmt of stmts) {
    if (stmt.type !== "select") {
      return {
        valid: false,
        reason: `Only SELECT statements allowed, got: ${stmt.type}`,
      };
    }
  }

  // Reject missing LIMIT
  const hasLimit = stmts.every((s) => s.limit !== null && s.limit !== undefined);
  if (!hasLimit) {
    return { valid: false, reason: "Query must include a LIMIT clause" };
  }

  // Reject wildcard SELECT
  const hasWildcard = stmts.some((s) =>
    (s.columns ?? []).some(
      (c: { expr: { type: string } }) => c.expr?.type === "star"
    )
  );
  if (hasWildcard) {
    return { valid: false, reason: "SELECT * is not allowed, use explicit columns" };
  }

  return { valid: true, normalizedSql: sql.trim() };
}

This catches most dangerous patterns at parse time, before you touch the database. When validation fails, feed the error back to the LLM and ask it to correct the query. One retry resolves the majority of cases.

Cost Management for NL-to-SQL

Schema injection is expensive. A schema prompt with 20 tables and examples can run 3,000-5,000 tokens per request. At $3/M input tokens, that is manageable for low-volume internal tools, but it adds up fast if you have 50 analysts running dozens of queries per day.

Mitigations that work in practice:

• Table selection pre-step: Run a fast, cheap call (or a keyword classifier) to identify the 3-5 relevant tables before injecting the full schema.
• Prompt caching: Anthropic and OpenAI both support prompt caching for static prefixes. Cache the schema block and pay for it once per cache TTL, not per request.
• Query result caching: Cache identical natural language queries with TTLs matched to your data freshness requirements. Most internal analytics questions repeat.

---

Document Q&A with RAG and Access Control

The Architecture

Retrieval-Augmented Generation (RAG) over internal documents is the second most common LLM internal tool pattern. The basic shape: embed documents, store vectors, retrieve relevant chunks on query, inject chunks into LLM context, generate an answer.

The part that gets skipped in demos: access control. In an internal knowledge base, not every user should see every document. Engineering specs, HR policies, financial projections, and customer contracts all have different access levels. If you retrieve from a shared vector store without filtering, a junior contractor can ask “what was our Q3 revenue?” and get a factual answer from a board document they were never supposed to see.

Access-Controlled Retrieval

Solve this at the retrieval layer, not the generation layer. The LLM cannot enforce access control reliably. The vector store can.

interface DocumentChunk {
  id: string;
  documentId: string;
  content: string;
  embedding: number[];
  metadata: {
    title: string;
    source: string;
    accessGroups: string[]; // e.g. ["engineering", "all-staff"]
    createdAt: string;
  };
}

interface RetrievalOptions {
  query: string;
  userGroups: string[];
  topK?: number;
  similarityThreshold?: number;
}

async function retrieveWithAccessControl(
  options: RetrievalOptions,
  vectorStore: VectorStore
): Promise<DocumentChunk[]> {
  const { query, userGroups, topK = 5, similarityThreshold = 0.72 } = options;

  const queryEmbedding = await embedText(query);

  // Pass access filter to the vector store query.
  // Most vector stores (Pinecone, Qdrant, pgvector) support metadata filtering.
  const results = await vectorStore.query({
    vector: queryEmbedding,
    topK: topK * 3, // fetch more, then filter -- avoids ranking artifacts
    filter: {
      accessGroups: { $in: userGroups },
    },
  });

  return results
    .filter((r) => r.score >= similarityThreshold)
    .slice(0, topK);
}

The topK * 3 over-fetch with post-filter is worth noting. If you apply access filtering in the vector database’s pre-filter (before ranking), you may get fewer results than expected because the filter reduces the candidate pool before similarity scoring. Over-fetching and filtering after ranking gives you better top-K quality at the cost of slightly more vector compute, which is cheap.

Grounding and Hallucination Reduction

RAG does not eliminate hallucination. If the retrieved chunks do not contain the answer, the LLM will often confabulate one. Mitigate this with explicit no-answer handling.

const SYSTEM_PROMPT = `You are a knowledge base assistant. Answer questions using only the provided context.

If the context does not contain enough information to answer the question:
- Say "I don't have enough information in the knowledge base to answer this."
- Do not speculate or use knowledge outside the provided context.
- Suggest the user contact a specific team or document owner if you can identify one from the context.

Always cite the document title and section when you answer.`;

async function generateAnswer(
  question: string,
  chunks: DocumentChunk[],
  llm: LLMClient
): Promise<{ answer: string; citations: string[] }> {
  const context = chunks
    .map((c, i) => `[${i + 1}] Source: ${c.metadata.title}\n${c.content}`)
    .join("\n\n---\n\n");

  const response = await llm.complete({
    system: SYSTEM_PROMPT,
    messages: [
      {
        role: "user",
        content: `Context:\n${context}\n\nQuestion: ${question}`,
      },
    ],
  });

  const citations = chunks
    .map((c) => `${c.metadata.title} (${c.metadata.source})`)
    .filter((_, i) => response.content.includes(`[${i + 1}]`));

  return { answer: response.content, citations };
}

Forcing citations does two things: it helps users verify answers, and it creates a feedback loop. When users flag wrong answers, you have enough context to trace whether the problem was retrieval (wrong chunks) or generation (right chunks, wrong answer).

Common Failure Modes

Chunk size mismatch: Chunks too small lose context; chunks too large dilute relevance scores. 400-600 tokens with 100-token overlap works for most prose documents. Code and tables need different strategies.

Embedding model drift: If you re-embed documents with a different model version, old embeddings become incompatible. Version your embeddings alongside the model that generated them.

Stale documents: RAG answers are only as fresh as your index. If a document is updated but not re-indexed, users get outdated answers confidently stated. Set up a pipeline that re-indexes on document mutation, not just on a nightly schedule.

---

Workflow Automation with LLM Routing

What This Actually Means

The third pattern is less discussed but often the highest-leverage: using an LLM as a routing and classification layer in a multi-step business workflow. Examples: classifying inbound support tickets and routing to the right team, extracting structured data from unstructured emails and writing to a CRM, triaging pull request reviews based on risk level.

The LLM is not doing the work here. It is deciding what work needs to be done and invoking the right tools to do it.

Typed Action Schema

The core abstraction is a typed action schema. Define what actions the LLM can take, give it those definitions in the prompt, and require structured output.

import { z } from "zod";

const TicketRoutingAction = z.discriminatedUnion("type", [
  z.object({
    type: z.literal("route"),
    team: z.enum(["engineering", "billing", "support", "security"]),
    priority: z.enum(["low", "medium", "high", "critical"]),
    reasoning: z.string(),
    extractedFields: z.object({
      customerTier: z.enum(["free", "pro", "enterprise"]).optional(),
      affectedFeature: z.string().optional(),
      errorCode: z.string().optional(),
    }),
  }),
  z.object({
    type: z.literal("clarify"),
    missingInformation: z.string(),
    suggestedQuestion: z.string(),
  }),
  z.object({
    type: z.literal("reject"),
    reason: z.string(),
  }),
]);

type TicketAction = z.infer<typeof TicketRoutingAction>;

async function classifyTicket(
  ticketContent: string,
  llm: LLMClient
): Promise<TicketAction> {
  const response = await llm.complete({
    system: `You are a support ticket classifier. Analyze the ticket and return a JSON action.

Actions available:
- route: assign to the correct team with priority and extracted metadata
- clarify: request more information from the customer
- reject: mark as spam or out-of-scope

Return only valid JSON matching the action schema. No explanation outside the JSON.`,
    messages: [{ role: "user", content: ticketContent }],
    responseFormat: { type: "json_object" },
  });

  const parsed = JSON.parse(response.content);
  return TicketRoutingAction.parse(parsed); // throws on invalid schema
}

Zod validation on the LLM output is not optional. Structured output modes (JSON mode, function calling) reduce but do not eliminate malformed responses. Parse and validate, and handle failures with a fallback path.

Multi-Step Execution with Idempotency

When the LLM’s decision triggers downstream actions, idempotency is critical. The LLM might be called twice due to a retry; the external action should not fire twice.

interface WorkflowExecution {
  id: string; // idempotency key
  ticketId: string;
  action: TicketAction;
  status: "pending" | "completed" | "failed";
  executedAt?: string;
}

async function executeRouting(
  ticketId: string,
  action: TicketAction,
  db: Database
): Promise<void> {
  const executionId = `ticket-route-${ticketId}`;

  // Check for existing execution before acting
  const existing = await db.query<WorkflowExecution>(
    "SELECT * FROM workflow_executions WHERE id = $1",
    [executionId]
  );

  if (existing.rows[0]?.status === "completed") {
    return; // already executed, skip
  }

  await db.transaction(async (tx) => {
    // Upsert execution record first
    await tx.query(
      `INSERT INTO workflow_executions (id, ticket_id, action, status)
       VALUES ($1, $2, $3, 'pending')
       ON CONFLICT (id) DO NOTHING`,
      [executionId, ticketId, JSON.stringify(action)]
    );

    if (action.type === "route") {
      await assignTicketToTeam(ticketId, action.team, action.priority, tx);
      await writeToSlack(action.team, ticketId, action.reasoning);
    }

    await tx.query(
      "UPDATE workflow_executions SET status = 'completed', executed_at = NOW() WHERE id = $1",
      [executionId]
    );
  });
}

The ON CONFLICT DO NOTHING pattern ensures that even if the execution is triggered twice concurrently, the downstream action fires exactly once.

---

Production Considerations

Tradeoffs by Use Case

Dimension	NL to SQL	Document Q&A (RAG)	Workflow Automation
Latency	1-3s with caching	2-5s (embed + retrieve + generate)	3-10s (multi-step)
Cost per request	High (schema injection)	Medium (chunked context)	Low-medium (short prompts)
Failure mode	Wrong/dangerous query	Stale or hallucinated answer	Missed classification, double execution
Safety mechanism	SQL AST validation	Access control at retrieval layer	Idempotency keys, schema validation
Observability need	Query audit log	Citation tracking, user feedback	Execution log per action

Observability That Actually Helps

Log four things for every LLM call in an internal tool:

1. The full prompt (or a hash of the schema prefix plus the variable part)
2. The raw model output before any parsing
3. The parsed/validated result or the validation error
4. The downstream action taken (query executed, chunks retrieved, workflow step fired)

When something goes wrong in production, you need all four. The raw output tells you whether the model failed or your validation was wrong. The downstream action log tells you what actually happened to data.

Cost at Scale

Internal tools have an advantage over customer-facing products: usage is predictable and bounded by headcount. A 100-person company with 20 analysts running 50 NL-to-SQL queries per day is 1,000 calls per day. At the token volumes above with prompt caching, this is $50-150/month. Not a line item worth engineering around.

Where cost becomes a problem: document Q&A over large context windows, or workflow automation with long-running chains. Keep a hard per-request token budget and monitor p95 token usage per endpoint. Alerts at 2x the baseline catch runaway prompts before they become invoice surprises.

The Production Deployment Checklist

Before shipping any of these patterns internally:

• All LLM calls have a hard timeout (30s is a reasonable ceiling for interactive tools)
• Every LLM output is validated against a schema before use
• Dangerous operations (writes, deletes) require a secondary confirmation step
• Access control is enforced at the retrieval layer, not in the prompt
• All LLM calls are logged with full context for debugging
• Idempotency keys are set on any action that triggers external side effects
• Token budgets are set per endpoint with alerting on p95 overrun
• Model fallback is configured (if primary model times out, degrade gracefully)
• Users can flag wrong answers, and that feedback routes to a review queue

---

Closing

The gap between “LLM demo” and “LLM production tool” is mostly not about the model. It is about the scaffolding: schema validation, access control, idempotency, observability. The three patterns here, natural language to SQL, document Q&A, and workflow automation, each have a different failure surface, but they share the same underlying discipline: treat LLM output as untrusted input until you have validated it, and make every downstream action reversible or idempotent.

Build the scaffolding first. The model improves on its own schedule. Your safeguards do not.