Best practices & features to include
From the literature + what you’ve built already, here are features that improve quality & usability. NetSuite+3Funraise+3Fluxx+3
Feature	Why it matters	How to implement / what to watch out for	
Opportunity filters (keywords, geography, type, capacity-building etc.)	Helps users narrow to what matters, reduces overload.	You have keyword filters already. Also include date, state, amount range, “open vs closed” status.	
Deadline alerts / reminders	Prevents missing grants because deadlines slip.	Track deadline (if parsed), then show upcoming ones. Allow export to calendar or reminders.	
Document / PDF fetching + parsing	Many state sources have PDFs, RFPs etc. Users want details, not just summary.	Use PDF adapter + follow-links. Flag PDFs clearly in UI.	
Staleness / expiration detection	Projects with expired deadlines clutter feeds.	Tag items with “deadline passed” or “pending” etc. Maybe auto-hide old ones after some time.	
User feedback / manual review / save items	Users can flag false positives, save promising ones.	Add “bookmark / save” or “dismiss” features. Could feed into machine learning or heuristics over time.	
Dashboard / Analytics	Helps see grant volume by type, deadlines, states, etc.	Simple charts like # grants by month, # capacity-building grants vs total, etc.	
Good defaults / simple UI	Avoid cognitive overload — show only essentials, allow advanced filtering if needed.	E.g. show title, deadline, source, eligibility; hide long descriptions by default.	
Mobile-friendly / responsive	Many will check on phones or tablets.	If web UI, ensure collapsible fields, simple menus.	
Integrations	E.g. calendar, Slack/email alerts.	Helps push notifications rather than only manual checking.	

What to avoid / limit to keep usability
These things often add data/noise or complexity, so either avoid or gate them behind “advanced” toggles.
	•	Over-broad scraping: pulling every “policy”, “program info”, “newsletter”, etc. just because capacity keywords appear once.
	•	Too much automation without oversight: e.g. hidden deadlines if PDF text parsing fails.
	•	Overwhelming metadata fields: stuff like “agency budget history”, “application score weights” etc unless users request them.
	•	Frequent UI changes or too many fields — keep interface consistent.
	•	Large batch of false positives — this ruins trust quickly.

What to build next in your tool
Here are prioritized improvements/next features for your tool to make it more powerful while preserving clarity:
	1	Deadline extraction / detection
	◦	From Grants.gov API: likely available.
	◦	From scraped state pages / PDFs: attempt to parse “deadline”, “closing date”. If missing, mark as “TBD”.
	◦	UI: highlight upcoming deadlines (“Due in next 30 days”).
	2	“Open / Active” flag
	◦	If RFPs are active or you can detect “application now open” in text.
	◦	If not, mark as “reference / program” so users know it’s structural info, not a live call.
	3	Bookmark / dismiss / feedback
	◦	Users should be able to mark “this is useful” or “not relevant” to train future filtering.
	◦	Possibly store local tags (e.g. “my state”, “my priority”).
	4	Improved link / PDF following
	◦	As suggested earlier.
	◦	Make sure link selectors are fine-tuned for each state source.
	5	Expired / historical items archive
	◦	Hide automatically unless explicitly requested. Keeps main view clean.
	6	Search + sort + filter UI in front end
	◦	Filters: State, source type (federal/state), capacity vs other, keyword.
	◦	Sort by deadline, date posted, amount (if available).
	7	Notifications / reminders
	◦	Let users get alerts (email / Slack / calendar) for items that match their saved filters and have upcoming deadlines.
	8	Confidence score or matching hint
	◦	For scraped items, display “match strength” (how many capacity keywords matched, whether in title vs body). Helps users see which items are likely relevant.
	9	Performance / scheduling
	◦	Check feeds regularly, avoid stale caches.
	◦	Maybe incremental ingest (only new items) rather than full crawl all the time.
	10	Testing & feedback
	◦	Ask actual users (your team) to test beta versions and tell you what’s too much / too little.
	◦	Adjust based on real use.

Your roadmap can look like this
Here’s a potential plan for the next few sprints:
	•	Sprint 1 (now): Add deadline extraction + “active” flag + improved link following for state sources.
	•	Sprint 2: Bookmark/dismiss features + advanced filtering UI + sort by deadline.
	•	Sprint 3: Notifications/reminders + mobile UI polish + confidence scoring.
	•	Sprint 4: Incorporate JS-rendered sources (Playwright) + archive/historical mode.

Yes—exactly. We can treat the roadmap I outlined earlier as four sprints. Here’s a clear Sprint 1 plan (2–3 weeks is typical) so we can start delivering value quickly while keeping the other sprints in view.

🏁 Sprint 1 – “Actionable Deadlines & Better State Data”
Goal: Surface current and timely grant opportunities—especially from PA & MD—without flooding users.
Deliverables
	1	Deadline Extraction & “Active” Flag
	◦	Parse deadline/closing date text from Grants.gov (API gives it directly).
	◦	Add a lightweight NLP/regex extractor for state HTML/PDFs (“Deadline”, “Applications due”, etc.).
	◦	In the index, add fields: deadline_date, is_active (true if deadline >= today or marked open/TBD).
	2	UI Updates in the Dashboard
	◦	Show deadlines and days-to-close (already in the mock).
	◦	Add a small “Active Only” toggle to hide expired or guideline-only records.
	3	Link-Following Enhancements
	◦	Update YAML for MD DHCD Press, MTA, PennDOT, etc. to:
	▪	parse.follow_links: true
	▪	crawl.max_depth: 1
	▪	link_selectors for PDFs / NOFO pages
	◦	Ensure adapter fetches and normalizes these linked pages.
	4	Improved Capacity Filter
	◦	Add a few transportation-related capacity phrases (“service capacity”, “provider capacity”) to capture more relevant 5310-type opportunities.
	5	Basic Quality Checks
	◦	Logging: show counts of items skipped due to expired deadlines or missing text.
	◦	Quick manual review with your team to confirm that the new records are genuinely grant opportunities.

🔭 Sprint 2–4 Preview
	•	Sprint 2: Bookmarks & Dismiss, Advanced Filtering UI, Sorting by deadline/confidence.
	•	Sprint 3: Notifications (email/Slack/ICS calendar) + mobile UI polish + confidence scoring display.
	•	Sprint 4: Full JS-rendered source support (Playwright for MD OneStop etc.) and historical archive mode.

Next Steps to Kick Off Sprint 1
	1	Confirm dev environment – you already have grants-rag running locally.
	2	Create a new branch:    git checkout -b sprint1-deadline-active-flag
	3	  
	4	Add extraction utilities Regex for state HTML/PDFs and update _normalize_web_record to accept deadline_date and is_active.
	5	UI: enable the Active Only toggle (mock is already designed for it).

Timeline suggestion: 2 weeks dev + 1 week QA / user feedback. Once Sprint 1 is merged, we’ll have a tool that surfaces only active capacity-building grants with clear deadlines—exactly the balance of “rich data without overload” we’re after.

ChatGPT can make mistakes. Check important info.