project-prompt-and-rubric.md

AI Engineering Project
Project Overview
For this project, you will be designing, building, and evaluating a Retrieval-Augmented
Generation (RAG) LLM-based application that answers user questions about a corpus of
company policies & procedures. You will then deploy the application to a free-tier host
(e.g., Render, Railway) with a basic CI/CD pipeline (e.g., GitHub Actions) that triggers
deployment on push/PR when the app builds successfully. Finally, you will demonstrate
the system via a screen-share video showing key features of your deployed application,
and a quick walkthrough of your design, evaluation and CI/CD run. You can complete this
project either individually or as a group of no more than three people.
While you can fully hand code this project if you wish, you are highly encouraged to
utilize leading AI code generation models/AI IDEs/async agents to assist in rapidly
producing your solution, being sure to describe in broad terms how you made use of
them. Here are some examples of very useful AI tools you may wish to consider. You will
be graded on the quality and functionality of the application and how well it meets the
project requirements—no given proportion of the code is required to be hand coded.

Learning Outcomes

When completed successfully, this project will enable you to:
● Demonstrate excellent AI engineering skills
● Demonstrate the ability to select appropriate AI application design and
architecture
● Implement a working LLM-based application including RAG
● Evaluate the performance of an LLM-based application
● Utilize AI tooling as appropriate

Project Description

First, assemble a small but coherent corpus of documents outlining company policies &
procedures—about 5–20 short markdown/HTML/PDF/TXT files totaling 30–120 pages.
You may author them yourself (with AI assistance) or use policies that you are aware of
from your own organization that can be used for this assignment. Students must use a
corpus they can legally include in the repo or load at runtime (e.g., your own synthetic
policies, your organization’s employee policy documents etc.)—no private/paid data is
required. Additionally, you should define success metrics for your application (see the
“Evaluation” step below), including at least one information-quality metric (e.g.,
groundedness or citation accuracy) and one system metric (e.g., latency).
Use free or zero-cost options when possible e.g., OpenRouter’s free tier
(https://openrouter.ai/docs/api-reference/limits), Groq
(https://console.groq.com/docs/rate-limits), or your own paid API keys if you have them.
For embedding models, free-tier options are available from Cohere, Voyage,
HuggingFace and others
Complete the following steps to fully develop, deploy, and evaluate your application:

Environment and Reproducibility
○ Create a virtual environment (e.g., venv, conda).
○ List dependencies in requirements.txt (or environment.yml).
○ Provide a README.md with setup + run instructions.
○ Set fixed seeds where/if applicable (for deterministic chunking or
evaluation sampling).
Ingestion and Indexing
○ Parse & clean documents (handle PDFs/HTML/md/txt).
○ Chunk documents (e.g., by headings or token windows with overlap).
○ Embed chunks with a free embedding model or a free-tier API.
○ Store the embedded document chunks in a local or lightweight vector
database (e.g. Chroma or an optionally cloud-hosted vector store like
Pinecone, etc.)
○ Store vectors in a local/vector DB or cloud DB (e.g., Chroma, Pinecone, etc.)
Retrieval and Generation (RAG)
○ To build your RAG pipeline you may use frameworks such as LangChain to
handle retrieval, prompt chaining, and API calls, or implement these
manually.
○ Implement Top-k retrieval with optional re-ranking.
○ Build a prompting strategy that injects retrieved chunks (and
citations/sources) into the LLM context.
○ Add basic guardrails:
■ Refuse to answer outside the corpus (“I can only answer about our
policies”),
■ Limit output length,
■ Always cite source doc IDs/titles for answers.
Web Application
○ Students can use Flask, Streamlit or alernative for the Web app. LangChain
is recommended for orchestration, but is optional.
○ Endpoints/UI:
■ / - Web chat interface (text box for user input)
■ /chat - API endpoint that receives user questions (POST) and returns
model-generated answers with citations and snippets (link to source
and show snippet).
■ /health - returns simple status via JSON.
Deployment
○ For production hosting use Render or Railway free tiers; students may
alternatively use any other free-tier providers of their choice.
○ Configure environment variables (e.g. API keys, model endpoints, DB
related etc.).
○ Ensure the app is publicly accessible at a shareable URL.
CI/CD
○ Minimal automated testing is sufficient for this assignment (a build/run
check, optional smoke test).
○ Create a GitHub Actions workflow that on push/PR :
■ Installs dependencies,
■ Runs a build/start check (e.g., python -m pip install -r
requirements.txt and python -c "import app" or pytest -q if you add
tests),
■ On success in main, deploy to your host (Render/Railway action or
via webhook/API).
Evaluation of the LLM Application
○ Provide a small evaluation set of 15–30 questions covering various policy
topics (PTO, security, expense, remote work, holidays, etc.). Report:
■ Answer Quality (required):
1. Groundedness: % of answers whose content is factually
consistent with and fully supported by the retrieved
evidence—i.e., the answer contains no information that is
absent or contradicted in the context.
Citation Accuracy: % of answers whose listed citations
correctly point to the specific passage(s) that support the
information stated—i.e., the attribution is correct and not
misleading.
Exact/Partial Match (optional): % of answers that exactly or
partially match a short gold answer you provide.
■ System Metrics (required):
Latency (p50/p95) from request to answer for 10–20 queries.
■ Ablations (optional): compare retrieval k, chunk size, or prompt
variants.
Design Documentation
○ Briefly justify design choices (embedding model, chunking, k, prompt
format, vector store).
Submission Guidelines

Your final submission should consist of two links:
● A link to an accessible software repository (a GitHub repo) containing all your
developed code. You must share your repository with the GitHub account,
quantic-grader.
o The GitHub repository should include a link to the deployed version of
your RAG LLM-based application (in file deployed.md)
o The GitHub repository must include a README.md file indicating setup and
run instructions
o The GitHub repository must also include a brief design and evaluation
document (design-and-evaluation.md) listing and explaining:
i) design and architecture decisions made - and why they were made,
including technology choices
ii) summary of your evaluation of your RAG system
● A link to a recorded screen-share demonstration video of the working RAG
LLM-based application, involving screen capture of it being used with voiceover
o All group members must speak and be present on camera.
o All group members must show their government ID.
o The demonstration/presentation should be between 5 and 10 minutes long.
To submit your project, please click on the "Submit Project" button on your dashboard
and follow the steps provided. If you are submitting your project as a group, please
ensure only ONE member submits on behalf of the group. Please reach out to
[email protected] if you have any questions. Project grading typically takes

about 3-4 weeks to complete after the submission due date. There is no score penalty
for projects submitted after the due date, however grading may be delayed.

Plagiarism Policy

Here at Quantic, we believe that learning is best accomplished by “doing”—this ethos
underpinned the design of our active learning platform, and it likewise informs our
approach to the completion of projects and presentations for our degree programs. We
expect that all of our graduates will be able to deploy the concepts and skills they’ve
learned over the course of their degree, whether in the workplace or in pursuit of
personal goals, and so it is in our students’ best interest that these assignments be
completed solely through their own efforts with academic integrity.
Quantic takes academic integrity very seriously—we define plagiarism as: “Knowingly
representing the work of others as one’s own, engaging in any acts of plagiarism, or
referencing the works of others without appropriate citation.” This includes both misusing
or not using proper citations for the works referenced, and submitting someone else’s
work as your own. Quantic monitors all submissions for instances of plagiarism and all
plagiarism, even unintentional, is considered a conduct violation. If you’re still not sure
about what constitutes plagiarism, check out this two-minute presentation by our
librarian, Kristina. It is important to be conscientious when citing your sources. When in
doubt, cite! Kristina outlines the basics of best citation practices in this one-minute video.
You can also find more about our plagiarism policy here.

Project Rubric
Scores 2 and above are considered passing. Students who receive a 1 or 0 will not get
credit for the assignment and must revise and resubmit to receive a passing grade.
Score Description

5
● Addresses ALL of the project requirements, but not limited to:
○ Outstanding RAG application with correct responses with matching
citations, ingest and indexing works
○ Excellent, well-structured application architecture
○ Public deployment on Render, Railway (or equivalent) fully functional
○ CI/CD runs on push/PR and deploys on success
○ Excellent documentation of design choices.
○ Excellent evaluation results, which includes groundedness, citation
accuracy, and latency
○ Excellent, clear demo of features, design and evaluation
4
● Addresses MOST of the project requirements, but not limited to:
○ Excellent RAG application with correct responses with generally
matching citations, ingest and indexing works
○ Very good, well-structured application architecture
○ Public deployment on Render, Railway (or equivalent) almost fully
functional
○ CI/CD runs on push/PR and deploys on success
○ Very good documentation of design choices.
○ Very good evaluation results which includes groundedness, citation
accuracy, and latency
○ Very good, clear demo of features, design and evaluation
3
● Addresses SOME of the project requirements, but not limited to:
○ Very good RAG application with mainly correct responses with
generally matching citations, ingest and indexing works
○ Good, well-structured application architecture
○ Public deployment on Render, Railway (or equivalent) almost fully
functional
○ CI/CD runs on push/PR and deploys on success
○ Good documentation of design choices.
○ Good evaluation results which includes most of groundedness,
citation accuracy, and latency
○ Good, clear demo of features, design and evaluation.
2
● Addresses FEW of the project requirements, but not limited to:
○ Passable RAG application with limited correct responses with few
matching citations, ingest and indexing works partially
○ Passable application architecture
○ Public deployment on Render, Railway (or equivalent) not fully
functional
○ CI/CD runs on push/PR and deploys on success
○ Passable documentation of design choices.
○ Passable evaluation results which includes only some of
groundedness, citation accuracy, and latency
○ Passable demo of features, design and evaluation
1
● Addresses the project but MOST of the project requirements are missing,
but not limited to:
○ Incomplete app; not deployed,
○ No CI/CD,
○ No to very limited evaluation
○ No design documentation
○ No demo of application
0
● The student either did not complete the assignment, plagiarized all or part
of the assignment, or completely failed to address the project requirements.