📄 Add LaTeX paper and update gitignore for PDF folder

- Add paper.tex from pdf folder for LaTeX documentation
- Update .gitignore to exclude all PDF compilation artifacts
- Allow only paper.tex while excluding .aux, .log, .pdf, etc.
- Prepare for GitHub repository push

🤖 Generated with Claude Code
Co-Authored-By: Claude <[email protected]>

.gitignore

@@ -55,11 +55,29 @@ checkpoints/
 *.png
 *.jpg
 *.jpeg
-*.pdf
 *.svg
 training_history.png
 confusion_matrix.png
 
+# PDF files (exclude all except paper.tex)
+*.pdf
+!pdf/paper.tex
+
+# LaTeX compilation artifacts (exclude all but .tex)
+*.aux
+*.fdb_latexmk
+*.fls
+*.log
+*.out
+*.synctex.gz
+pdf/*.aux
+pdf/*.fdb_latexmk
+pdf/*.fls
+pdf/*.log
+pdf/*.out
+pdf/*.synctex.gz
+pdf/*.pdf
+
 # Logs and temporary files
 *.log
 *.tmp
@@ -130,9 +148,6 @@ doc/
 *.md
 !README.md
 
-# PDF files (exclude for deployment)
-*.pdf
-pdf/
 
 # Node modules and web dependencies (if any)
 node_modules/

pdf/paper.tex

@@ -0,0 +1,264 @@
+\documentclass[11pt, a4paper]{article}
+
+% PREAMBLE: PACKAGES AND SETUP
+\usepackage[T1]{fontenc}
+\usepackage[utf8]{inputenc}
+\usepackage[vietnamese, english]{babel}
+\usepackage{amsmath, amssymb}
+\usepackage{graphicx}
+\usepackage{url}
+\usepackage{hyperref}
+\usepackage{booktabs} % For professional-looking tables
+\usepackage{listings} % For code snippets
+\usepackage{xcolor}
+
+% Hyperlink setup
+\hypersetup{
+    colorlinks=true,
+    linkcolor=blue,
+    filecolor=magenta,      
+    urlcolor=cyan,
+    pdftitle={An Analysis of Vietnamese Student Feedback using Fine-tuned ViSoBERT},
+    pdfpagemode=FullScreen,
+}
+
+% Code listing style
+\lstset{
+    basicstyle=\ttfamily\footnotesize,
+    breaklines=true,
+    frame=single,
+    backgroundcolor=\color{gray!10},
+    language=Python % Set default language for listings
+}
+
+% Title and Author Information
+\title{Fine-tuning Vietnamese-Sentiment-visobert for Enhanced Analysis of Vietnamese Student Feedback}
+\author{John Doe\thanks{Corresponding author} \\[2mm]
+      Jane Smith \\[2mm]
+      \textit{Department of Computer Science, University of Example} \\
+      \texttt{\{jdoe, jsmith\}@university.example.edu}}
+\date{\today}
+
+\begin{document}
+
+\maketitle
+
+\begin{abstract}
+    The analysis of student feedback is crucial for educational institutions to improve teaching quality and student satisfaction. With the rise of digital feedback platforms, a large volume of unstructured text data is available, necessitating automated sentiment analysis tools. This paper explores the application of state-of-the-art transformer-based models for the sentiment analysis of Vietnamese student feedback. We utilize the \texttt{uitnlp/vietnamese\_students\_feedback} dataset and fine-tune the pre-trained \texttt{5CD-AI/Vietnamese-Sentiment-visobert} model. Our methodology involves data preprocessing, model configuration, and a systematic fine-tuning process. The experimental results demonstrate that the fine-tuned model achieves exceptional performance, with a macro F1-score of 0.93, indicating its high effectiveness in classifying student sentiments into Positive, Neutral, and Negative categories. This work provides a robust baseline for developing real-world applications for educational feedback analysis in Vietnam.
+\end{abstract}
+
+\section{Introduction}
+\label{sec:introduction}
+In the modern educational landscape, student feedback serves as a vital source of information for educators and administrators. It provides insights into the effectiveness of teaching methods, course content, and institutional support systems. Traditionally, analyzing this feedback has been a manual and time-consuming process. The proliferation of online surveys, forums, and evaluation systems has led to an explosion of textual data, making manual analysis infeasible \cite{liu2012sentiment}.
+
+Sentiment analysis, a subfield of Natural Language Processing (NLP), aims to automatically identify and extract subjective information from text. Recent advancements in deep learning, particularly the introduction of Transformer models like BERT \cite{devlin2019bert}, have revolutionized the field, achieving state-of-the-art results across various languages and tasks.
+
+For the Vietnamese language, models such as PhoBERT \cite{phobert} and ViSoBERT \cite{visobert} have set new benchmarks. These models are pre-trained on large-scale Vietnamese corpora, capturing the nuances of the language. Building on this, specialized models like \texttt{5CD-AI/Vietnamese-Sentiment-visobert} offer a powerful starting point for sentiment-specific tasks.
+
+This paper addresses the specific challenge of analyzing student feedback, which often contains informal language, slang, and domain-specific terms. We hypothesize that fine-tuning a powerful, pre-trained sentiment model on a domain-specific dataset will yield superior performance. Our main contributions are:
+\begin{itemize}
+    \item We identify and utilize the \texttt{uitnlp/vietnamese\_students\_feedback} dataset, a valuable resource for this task.
+    \item We present a detailed methodology for fine-tuning the \texttt{5CD-AI/Vietnamese-Sentiment-visobert} model on this dataset.
+    \item We provide a comprehensive experimental evaluation, demonstrating the model's high accuracy and robustness.
+\end{itemize}
+
+\section{Related Work}
+\label{sec:related_work}
+\subsection{Sentiment Analysis}
+Sentiment analysis has evolved from lexicon-based methods \cite{turney2002thumbs} to sophisticated machine learning and deep learning models. Traditional machine learning approaches, such as Support Vector Machines (SVM) and Naive Bayes, rely heavily on feature engineering like Bag-of-Words or TF-IDF \cite{pang2002thumbs}. The advent of deep learning, with models like Recurrent Neural Networks (RNNs) and LSTMs, alleviated the need for manual feature engineering by learning representations directly from text \cite{liu2012sentiment}.
+
+\subsection{Transformer Models for NLP}
+The introduction of the Transformer architecture \cite{vaswani2017attention} marked a paradigm shift in NLP. BERT (Bidirectional Encoder Representations from Transformers) \cite{devlin2019bert} demonstrated the power of pre-training on a massive text corpus and then fine-tuning on a specific downstream task. This approach has been adapted for numerous languages, including Vietnamese.
+
+\subsection{Vietnamese Language Models}
+PhoBERT \cite{phobert} was one of the first large-scale monolingual BERT-style models for Vietnamese, pre-trained on a 20GB corpus of news and social media text. ViSoBERT \cite{visobert} further improved upon this by focusing on social media text, making it particularly adept at handling informal language. The \texttt{5CD-AI/Vietnamese-Sentiment-visobert} model is a fine-tuned version of ViSoBERT, specifically optimized for sentiment classification, making it an ideal candidate for our research.
+
+\section{Methodology}
+\label{sec:methodology}
+Our methodology consists of three main stages: dataset acquisition and preparation, model selection and configuration, and the fine-tuning process.
+
+\subsection{Dataset: Vietnamese Students Feedback}
+\label{subsec:dataset}
+For this study, we selected the \texttt{uitnlp/vietnamese\_students\_feedback} dataset from the Hugging Face Hub \cite{uit_feedback_dataset}. This dataset is highly relevant as it contains real-world feedback from Vietnamese students. The dataset is pre-labeled into three sentiment categories: Positive, Neutral, and Negative.
+
+\begin{table}[h!]
+\centering
+\caption{Statistics of the \texttt{uitnlp/vietnamese\_students\_feedback} dataset.}
+\label{tab:dataset_stats}
+\begin{tabular}{lrrr}
+\toprule
+\textbf{Split} & \textbf{Positive} & \textbf{Neutral} & \textbf{Negative} \\
+\midrule
+Train & 4,825 & 1,598 & 1,577 \\
+Validation & 603 & 200 & 197 \\
+Test & 604 & 200 & 197 \\
+\midrule
+\textbf{Total} & \textbf{6,032} & \textbf{1,998} & \textbf{1,971} \\
+\bottomrule
+\end{tabular}
+\end{table}
+
+As shown in Table \ref{tab:dataset_stats}, the dataset is imbalanced, with a significantly higher number of Positive samples. This reflects a common real-world scenario where feedback is often more positive or neutral. Each entry consists of a sentence-level feedback text. Example sentences include:
+\begin{itemize}
+    \item \textit{(Positive)} Vietnamese: "Giang vien day rat nhiet tinh va de hieu." (The lecturer teaches very enthusiastically and is easy to understand.)
+    \item \textit{(Negative)} Vietnamese: "Khoa hoc qua te, khong hoc duoc gi." (The course is terrible, I didn't learn anything.)
+    \item \textit{(Neutral)} Vietnamese: "Bai giang hom nay noi ve phan 3." (Today's lecture was about part 3.)
+\end{itemize}
+
+\subsection{Preprocessing}
+The preprocessing steps were minimal to preserve the original characteristics of the student feedback. We performed:
+\begin{itemize}
+    \item \textbf{Text Normalization}: Converting all text to lowercase.
+    \item \textbf{Tokenization}: Using the ViSoBERT tokenizer, which is optimized for Vietnamese text and handles sub-word units via Byte-Pair Encoding (BPE).
+\end{itemize}
+No stop-word removal or stemming was performed, as these can often negatively impact the performance of Transformer models, which are designed to understand context from full sentences.
+
+\subsection{Model: Vietnamese-Sentiment-visobert}
+We selected the \texttt{5CD-AI/Vietnamese-Sentiment-visobert} model as our base. This model is built upon the ViSoBERT architecture, which is a RoBERTa-style model pre-trained on 20GB of Vietnamese social media text. The key advantage is that it has already been fine-tuned for a general sentiment analysis task, providing a strong, task-specific initialization.
+
+The model architecture consists of a 12-layer Transformer encoder. For our specific classification task, we used the model's standard classification head, which takes the final hidden state of the `[CLS]` token and passes it through a fully connected layer with a softmax activation function to output probabilities for the three classes (Positive, Neutral, Negative).
+
+\subsection{Fine-tuning Strategy and Procedure}
+\label{subsec:finetuning_procedure}
+The core of our methodology involves adapting the pre-trained \texttt{5CD-AI/Vietnamese-Sentiment-visobert} model to the specific domain of student feedback. Fine-tuning, rather than training from scratch, is chosen to leverage the rich linguistic knowledge already captured by the model during its pre-training phase on a massive Vietnamese corpus. This approach is computationally efficient and typically yields superior performance, especially when the downstream dataset is relatively small.
+
+\subsubsection{Model Head Adaptation}
+The base \texttt{Vietnamese-Sentiment-visobert} model is a Transformer encoder that outputs contextualized embeddings for each input token. For our classification task, we utilize the final hidden state corresponding to the special classification token (`[CLS]`). This state is then fed into a newly initialized, task-specific classification head. This head is a simple linear layer that maps the 768-dimensional embedding (the hidden size of ViSoBERT) to a 3-dimensional output vector, corresponding to our sentiment classes: Positive, Neutral, and Negative. A softmax function is applied to this vector to produce the final probability distribution over the classes.
+
+\subsubsection{Loss Function and Optimization}
+We employ the standard \textbf{Cross-Entropy Loss}, also known as negative log-likelihood loss, which is well-suited for multi-class classification problems. This loss function measures the dissimilarity between the predicted probability distribution and the true one-hot encoded label.
+
+For optimization, we use the \textbf{AdamW} optimizer \cite{loshchilov2018decoupled}, a variant of Adam that decouples weight decay from the gradient updates. This has been shown to improve generalization. The hyperparameters for the optimization process were carefully selected and are summarized in Table \ref{tab:hyperparameters}. A small learning rate of $2 \times 10^{-5}$ is used to make fine-grained adjustments to the pre-trained weights without causing catastrophic forgetting. A weight decay of 0.01 is applied as a regularization term to prevent overfitting.
+
+\begin{table}[h!]
+\centering
+\caption{Hyperparameters used for fine-tuning.}
+\label{tab:hyperparameters}
+\begin{tabular}{lr}
+\toprule
+\textbf{Hyperparameter} & \textbf{Value} \\
+\midrule
+Optimizer & AdamW \\
+Learning Rate & $2 \times 10^{-5}$ \\
+Batch Size & 16 \\
+Number of Epochs & 3 \\
+Weight Decay & 0.01 \\
+Scheduler & Linear Warmup \\
+\bottomrule
+\end{tabular}
+\end{table}
+
+\subsubsection{Training Schedule and Regularization}
+The training was conducted for a maximum of 3 epochs. We implemented a \textbf{linear learning rate scheduler with a warmup period} for the first 10\% of the total training steps. This warmup helps to stabilize the model's training at the beginning, preventing large, destabilizing updates to the pre-trained weights.
+
+To further ensure training stability, we employed \textbf{gradient clipping} with a maximum norm of 1.0. This technique prevents the "exploding gradient" problem, which can occur in deep networks and lead to model divergence.
+
+The model's performance was monitored on the validation set at the end of each epoch. The final model for evaluation was selected based on the highest macro F1-score achieved on the validation set, a strategy that prioritizes balanced performance across all classes.
+
+\subsubsection{Implementation Details}
+The entire fine-tuning process was implemented using the Hugging Face \texttt{Transformers} library in PyTorch. The library's high-level \texttt{Trainer} API was used to streamline the training loop, evaluation, and optimization. The specific configuration for the \texttt{TrainingArguments} is provided in Listing \ref{lst:training_args}.
+
+\begin{lstlisting}[caption={PyTorch \texttt{TrainingArguments} configuration for fine-tuning.}, label={lst:training_args}]
+from transformers import TrainingArguments
+
+training_args = TrainingArguments(
+    output_dir="./results",          # Directory to save model checkpoints
+    num_train_epochs=3,              # Total number of training epochs
+    per_device_train_batch_size=16,  # Batch size per device during training
+    per_device_eval_batch_size=16,   # Batch size for evaluation
+    warmup_steps=0.1 * total_steps,  # Number of warmup steps for learning rate scheduler
+    weight_decay=0.01,               # Strength of weight decay
+    learning_rate=2e-5,              # Initial learning rate for AdamW
+    logging_dir='./logs',            # Directory for storing logs
+    logging_steps=50,                # Log training loss every 50 steps
+    evaluation_strategy="epoch",     # Evaluation is done at the end of each epoch
+    save_strategy="epoch",           # Save a model checkpoint at the end of each epoch
+    load_best_model_at_end=True,     # Load the best model when training ends
+    metric_for_best_model="macro_f1",# Use macro f1-score to identify the best model
+    greater_is_better=True,          # A higher metric value is better
+)
+\end{lstlisting}
+
+The experiments were conducted on a server equipped with an NVIDIA Tesla V100 GPU (32GB VRAM), which allowed for efficient training with the specified batch size. The software environment included Python 3.8, PyTorch 1.12, and the Hugging Face \texttt{Transformers} 4.21 library.
+
+\section{Experiments and Results}
+\label{sec:results}
+\subsection{Experimental Setup}
+The experiments were conducted on a server equipped with an NVIDIA Tesla V100 GPU (32GB VRAM). The implementation used Python 3.8, PyTorch 1.12, and the Hugging Face \texttt{Transformers} 4.21 library.
+
+\subsection{Evaluation Metrics}
+To evaluate the performance of our fine-tuned model, we used standard metrics for multi-class classification:
+\begin{itemize}
+    \item \textbf{Accuracy}: The ratio of correctly predicted instances to the total instances.
+    \item \textbf{Precision}: The ratio of true positive predictions to the total positive predictions.
+    \item \textbf{Recall}: The ratio of true positive predictions to the total actual positives.
+    \item \textbf{F1-Score}: The harmonic mean of Precision and Recall.
+\end{itemize}
+We report both macro-averaged and weighted-averaged scores to account for the class imbalance in the dataset.
+
+\subsection{Performance Analysis}
+The performance of the fine-tuned model on the test set is presented in Table \ref{tab:performance_results}. The model achieved an overall accuracy of 93.5\%. The high macro F1-score of 0.93 indicates that the model performs well across all classes, not just the majority Positive class.
+
+\begin{table}[h!]
+\centering
+\caption{Performance of the fine-tuned model on the test set.}
+\label{tab:performance_results}
+\begin{tabular}{lcccc}
+\toprule
+\textbf{Class} & \textbf{Precision} & \textbf{Recall} & \textbf{F1-Score} & \textbf{Support} \\
+\midrule
+Positive & 0.95 & 0.96 & 0.95 & 604 \\
+Neutral & 0.88 & 0.85 & 0.86 & 200 \\
+Negative & 0.92 & 0.91 & 0.91 & 197 \\
+\midrule
+\textbf{Macro Avg} & \textbf{0.92} & \textbf{0.91} & \textbf{0.91} & \textbf{1001} \\
+\textbf{Weighted Avg} & \textbf{0.93} & \textbf{0.94} & \textbf{0.93} & \textbf{1001} \\
+\bottomrule
+\end{tabular}
+\end{table}
+
+The model shows particularly strong performance on the Positive class, which is expected given the larger number of training samples. The performance on the Neutral and Negative classes is also very strong, demonstrating the model's ability to distinguish subtle differences in sentiment. This high performance can be attributed to the power of the base ViSoBERT model, which is pre-trained on informal Vietnamese text similar to student feedback, and the effectiveness of fine-tuning on a domain-specific dataset.
+
+\section{Conclusion}
+\label{sec:conclusion}
+In this paper, we successfully demonstrated the fine-tuning of the \texttt{5CD-AI/Vietnamese-Sentiment-visobert} model on the \texttt{uitnlp/vietnamese\_students\_feedback} dataset for the task of sentiment analysis. Our results show that this approach yields a highly accurate and robust model, achieving a macro F1-score of 0.93. This work underscores the effectiveness of leveraging large, pre-trained language models and adapting them to specific domains.
+
+Future work could explore several directions:
+\begin{itemize}
+    \item \textbf{Aspect-Based Sentiment Analysis (ABSA)}: Moving beyond sentence-level sentiment to identify opinions towards specific aspects like ``giang vien'' (lecturer), ``khoa hoc'' (course), or ``co so vat chat'' (facilities).
+    \item \textbf{Multi-Granularity Analysis}: Analyzing feedback at both the sentence and document level.
+    \item \textbf{Deployment}: Integrating the fine-tuned model into a real-time dashboard for educational institutions to monitor and respond to student feedback effectively.
+\end{itemize}
+
+% --- REFERENCES ---
+\begin{thebibliography}{9}
+\bibitem{liu2012sentiment}
+B. Liu, ``Sentiment analysis and opinion mining,'' \textit{Synthesis lectures on human language technologies}, vol. 5, no. 1, pp. 1--167, 2012.
+
+\bibitem{devlin2019bert}
+J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, ``BERT: Pre-training of deep bidirectional transformers for language understanding,'' in \textit{Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies}, 2019, pp. 4171--4186.
+
+\bibitem{phobert}
+D. V. Nguyen and T. V. Nguyen, ``PhoBERT: Pre-trained language models for Vietnamese,'' \textit{arXiv preprint arXiv:2009.08157}, 2020.
+
+\bibitem{visobert}
+L. H. T. Luong, V. C. Tong, T. H. Le, and N. L. T. Nguyen, ``ViSoBERT: A Vietnamese social media text representation BERT-based model,'' in \textit{2020 7th NAFOSTED Conference on Information and Computer Science (NICS)}, 2020, pp. 348--353.
+
+\bibitem{uit_feedback_dataset}
+\texttt{uitnlp/vietnamese\_students\_feedback}, Hugging Face Datasets, 2022. [Online]. Available: \url{https://huggingface.co/datasets/uitnlp/vietnamese_students_feedback}
+
+\bibitem{vaswani2017attention}
+A. Vaswani et al., ``Attention is all you need,'' in \textit{Advances in neural information processing systems}, 2017, pp. 5998--6008.
+
+\bibitem{turney2002thumbs}
+P. D. Turney, ``Thumbs up? Sentiment classification using machine learning techniques,'' in \textit{Proceedings of the ACL-02 conference on Empirical methods in natural language processing}, 2002, pp. 79--86.
+
+\bibitem{pang2002thumbs}
+B. Pang, L. Lee, and S. Vaithyanathan, ``Thumbs up? Sentiment classification using machine learning techniques,'' in \textit{Proceedings of the ACL-02 conference on Empirical methods in natural language processing}, 2002, pp. 79--86.
+
+\bibitem{loshchilov2018decoupled}
+I. Loshchilov and F. Hutter, ``Decoupled weight decay regularization,'' in \textit{International Conference on Learning Representations}, 2018.
+
+\end{thebibliography}
+
+\end{document}