A laborious Policy Library before glamorous AI work
We were building an AI system that writes clinical appeals against denials. It compares a patient’s record with the payer’s medical policy and finds where they diverge. That comparison is only as good as the policy it reads. An appeal argued against last quarter’s version of a policy is a weak appeal. An appeal argued against the wrong policy, or half of one, is worse. So before any model work, we needed a library of medical policies and clinical guidelines across many payers: complete, current, and stored in a standard structure an agent could search. VectorDB was NOT the answer We deliberately did not use a vector database. An appeal needs the policy for a given payer and service, not the five passages that sound most similar. A structured store answers that question directly. If you work on appeals, denials or prior authorization, you need this library too. And if you have started building it, you already know it is harder than it looks.Why this is harder than it looks
Three things multiply against each other:- Many sources. National carriers, regional Blues plans, state Medicaid programs, Medicare, military health plans, specialty benefit managers, and clinical sources such as society guidelines and drug labeling.
- Many formats. Each site hosts and serves its policies differently: PDF listings, search portals, category trees, A–Z indexes, XML files, PDFs that link to other PDFs.
- Constant change. Policies are revised all the time, so a one-time download goes stale. You have to re-scrape on a schedule and capture every change.
Five major painpoints
No two payer sites broke the same way, but every breakage we hit fell into one of five patterns. If you are building this yourself, you will meet all of them.1. Getting in the door
Before you can read a policy, the site has to let you in. Some payer sites only respond to traffic from US IP addresses. Some sit behind bot detection that asks for human verification even when a real, visible browser is doing the clicking. One regional plan sends you to a terms page and, once you accept, drops you on a general information page instead of the policy list. Another puts a zip code prompt in front of all its content. A federal site shows a license dialog, but only some of the time. A person gets past each of these in seconds. An unattended scraper has to get past all of them, on every run, with nobody watching.2. Finding every policy
Most payers do not publish one clean list. You have to reconstruct it. One Blues plan exposes its policies only through an A–Z search, with every letter paginated. Another splits its list into alphabetical sections, plus a “#” section for policies that start with a number, which is very easy to miss. Some have no listing at all, only category-by-category search. One national carrier puts its list behind a state selection, so the same crawl runs once per state. A specialty benefit manager makes you go through a category, then a search by health plan, before any policy list appears. Drug labeling was a puzzle of its own. Some documents sat on a web page, others existed only as individual XML files. The complete set turned out to be in a bulk download that is easy to overlook. Each of these is a different crawl. Each one has to run to the last page before you can trust the count.3. Listings you can’t trust
The listings cannot be trusted blindly.You need to look-out for unorderd/missing/stale/duplicated data One large carrier’s search results repeated records across pages, so a single pass looked complete and quietly missed policies. Another payer published several files for the same policy, and the list did not order the versions by effective date. On one site, policy URLs went stale and kept returning the same PDF, whichever policy you asked for. On another, broken links served error pages that looked exactly like successful downloads. So a list, a URL or a successful download proves nothing about what you actually got.4. Documents that aren’t one document
A “policy” is sometimes a web of files. On one plan’s site, a policy page linked to further PDFs, and the policy was only complete once every one of them was collected and merged. Another source put links to more content inside the PDF itself, so you had to read the document to know what else to fetch. If you store only the first file, your appeal is arguing against part of a policy.5. Browser mechanics
Small things stop an unattended run cold: a “Save As” dialog on every download, or policies that open in a new window instead of the current one. None of these is hard on its own but each one is enough to stall a run nobody is watching.Then do it all again next month
A policy library is never finished. Payers revise policies continuously, and the library has to keep up. That changes what “working” means. A workaround a person can do by hand once, like clicking past a terms page or typing a zip code, has to run unattended every cycle. Every fix for every pattern above had to become code that survives the next run. The checks matter as much as the scraping. Deduplication, reconciling counts against the payer’s own total, and ordering versions by effective date are what tell you a re-scrape captured every change and nothing else.
How we finally automated it
We got here in three phases. The longer story is in The groundwork behind an AI workflow; here is the short version.- Manual. We started manually. People navigated each site, paged through results, copied URLs into spreadsheets and downloaded files one by one. Apart from being brain numbing this process was totally flawed, It missed policies, created duplicates and did not scale, and per-payer scripts would not have scaled either.
- AI where judgment is needed. We rebuilt the scraper around an AI agent, but used AI only where a decision had to be made: reading page structure, picking out relevant documents, navigating dynamic content and checking whether a file was really a policy. Deterministic code did the downloads, HTTP requests, browser automation and PDF conversion. Scraping became about 5x faster.
- A tougher browser layer. Access controls then became the bottleneck: verification loops, CAPTCHAs, IP blocks, rate limits and “Access Denied” pages. No prompt fixes that. We replaced the browser automation layer with one built to handle verification challenges reliably. Because that layer sat behind an interface, the switch barely touched anything else.A central orchestrator ties it together. It coordinates search, navigates each site, deduplicates results and aggregates everything into the standard policy structure.
If you’re building your own policy library
- Budget for the groundwork. Consolidating payers, understanding each one’s policy structure, scraping and normalizing is most of the work, and none of it is AI.
- Never trust a single pass. Paginate to the end, deduplicate, and reconcile against the payer’s own total.
- Never trust a URL or a list order. Verify what you downloaded and version by effective date.
- Design for the re-scrape, not the first scrape. Every manual workaround must run unattended.
- Use AI for decisions, code for execution. Where you use AI matters more than how much.
- Keep the store as simple as the question. If you look policies up by payer and service, a structured store beats a vector database.
Cennest builds the groundwork that AI in revenue cycle depends on: payer policy libraries, claims and remittance data pipelines, and the AI workflows that run on top of them. We have hit every wall described here and built a system that keeps a multi-payer library current, run after run. If you are building appeals, denials or prior-auth automation and policy data is holding you back, let's talk.Team Cennest
