\section{Introduction} Recently, Large Language Models (LLMs) such as GPT-4~\citep{openai2023gpt4}, PaLM 2~\citep{anil2023palm}, and LLaMA~\citep{touvron2023llama} have shown exceptional performance on a wide range of NLP tasks, including common sense reasoning~\citep{wei2022chain,zhou2022least} and mathematical problem-solving~\citep{lewkowycz2022solving,chen2022program}. Despite their ability to learn from huge amounts of data, LLMs still have limitations in their capacity to retain and understand information. To ensure responsible usage, it is crucial for LLMs to have the capability of recognizing their limitations and conveying uncertainty when responding to unanswerable or unknowable questions. This acknowledgment of limitations, also known as ``\textit{knowing what you don't know},'' is a crucial aspect in determining their practical applicability. In this work, we refer to this ability as model self-knowledge. The Know-Unknow quadrant in Figure~\<PIC> illustrates the relationship between the model's knowledge and comprehension. The ratio of ``Known Knows'' to ``Unknown Knows'' demonstrates the model's proficiency in understanding and applying existing knowledge. Techniques such as Chain-of-Thought~\cite{wei2022chain}, Self-Consistency~\citep{wang2022self}, and Complex CoT~\citep{fu2022complexity} can be utilized to increase this ratio, resulting in improved performance on NLP tasks. We focus on the ratio of ``Known Unknows'' to ``Unknown Unknows'', which indicates the model's self-knowledge level, specifically understanding its own limitations and deficiencies in the unknows. Existing datasets such as SQuAD2.0~\citep{rajpurkar2018know} and NewsQA~\citep{trischler2017newsqa}, widely used in question answering (QA), have been utilized to test the self-knowledge of models with unanswerable questions. However, these questions are context-specific and could become answerable when supplemented with additional information. \citet{srivastava2022beyond} attempted to address this by evaluating LLMs' competence in delineating their knowledge boundaries, employing a set of 23 pairs of answerable and unanswerable multiple-choice questions. They discovered that these models' performance barely surpassed that of random guessing. \citet{kadavath2022language} suggested probing the self-knowledge of LLMs through the implementation of a distinct "Value Head". Yet, this approach may encounter difficulties when applied across varied domains or tasks due to task-specific training. Consequently, we redirect our focus to the inherent abilities of LLMs, and pose the pivotal question: ``\textit{Do large language models know what they don’t know?}''. \paragraph{Input Forms.} The incorporation of instructions and examples serves to boost the self-knowledge of both the GPT-3 and InstructGPT series. Specifically, ICL input form, providing richer contextual information, contributes to a significant enhancement in models' self-knowledge. This impact is particularly noticeable in the davinci model, where ICL facilitates a 27.96\% improvement over the direct. Moreover, a comparison between Figure~\<PIC> reveals that the inclusion of instructions and examples successfully minimizes the performance disparity between the davinci and text-davinci models, suggesting an acquisition of self-knowledge from the instructions and provided examples. \vspace{-.1em}
\paragraph{Compared with Human.} Figure~\<PIC> reveals that, without supplementary samples, GPT-4 currently performs best among the tested models, achieving an impressive F1 score of 75.47\%. However, a noticeable gap becomes evident when comparing this performance to the human benchmark of 84.93\%. This underscores the considerable potential that remains for enhancing the self-knowledge level of LLMs. This study investigates the self-knowledge of LLMs by evaluating their ability to identify unanswerable questions. Through the introduction of a novel dataset and an automated method for detecting uncertainty in the models' responses, we are able to accurately measure the self-knowledge of LLMs such as GPT-3, InstructGPT and LLaMA. Our results reveal that while these models possess a certain degree of self-knowledge, there is still an apparent disparity in comparison to human self-knowledge. This highlights the need for further research in this area to enhance the ability of LLMs to understand their own limitations on the unknows. Such efforts will lead to more accurate and reliable responses from LLMs, which will have a positive impact on their applications in diverse fields.


With the increases in pre-training corpora and the number of parameters \citep{radford2018improving,radford2019language,brown2020language}, large language models (LLMs) have shown remarkable performance across various natural language processing (NLP) tasks, even generating expert-level responses to user queries.
The extraordinary capabilities of LLMs hold potential for further real-world applications \citep{wang2023chatcad,owl,driess2023palm}, which necessitate higher requirements for model trustworthiness \citep{wang2023decodingtrust,li2023trustworthy,sun2024trustllm} and consistency \citep{consistency-analysis,elazar-etal-2021-measuring}.  However, LLMs still suffer from inconsistency issues: semantically equivalent queries \citep{elazar-etal-2021-measuring,raj2023semantic} and insignificant order changes of inputted contents \citep{wang2023fair-evaluator} can lead to divergent outcomes; LLMs can also behave differently in the generation versus validation of the same content \citep{li2023generate-validate}; moreover, logical transformations like negation and symmetry can also induce inconsistent behaviors \citep{jang-etal-2022-consistency-bench}.
In addition to previous work, we also find LLMs able to solve hard problems surprisingly fail to solve easier ones (as shown in Figure \<PIC>), suffering from the hard-to-easy inconsistency.
Unlike LLMs, humans are naturally consistent reasoners, and it is undisputed that an individual proficient in calculus can easily address simpler arithmetic problems.
However, why this difference exists is still unknown and relevant research to explore hard-to-easy consistency of LLMs is still lacking. To systematically evaluate the hard-to-easy consistency of LLMs, we develop ConsisEval with data from code, mathematics, and instruction-following domains, which are widely considered to be difficult but of significant importance for LLMs \citep{wei2021finetuned,gsm8k,humaneval,ifeval}.
Different from traditional benchmarks in which data are usually individual, there are only pairwise data in ConsisEval: one datum is comprised of two questions (an easy question and a harder one)
with a strict order of difficulty, and we present some example data from ConsisEval in Table \ref{tab:example_data}.
To construct ConsisEval, we collect easy data from some established public datasets (\S \ref{subsec:easy_data}); then we acquire hard data through automatic generation by GPT-4 and human annotation (\S \ref{subsec:hard_data}), and this process is shown in Figure \<PIC>.  \subsection{Hard Data Collection}
\label{subsec:hard_data}
To build our pairwise dataset in which a strict order of difficulty is guaranteed for each pair of easy and hard problems, all of the hard data are modified from easy data.
We employ a semi-automatic pipeline that integrates the automatic generation of GPT-4 with human annotation to acquire hard data, and the whole workflow is illustrated in Figure \<PIC>. Compared to traditional methods that rely solely on manual annotation, our semi-automatic approach can significantly alleviate the workload of human annotators.


\section{Cross-lingual Transfer for Automatic Question Generation} The zero-shot XLT approach--leveraging multilingual pretrained language models (mPLMs) fine-tuned exclusively on English data for target languages--has shown promising performance across various classification tasks \cite{liu2019zero,conneau2019cross,gritta2021xeroalign,wu2022zero,li2023does}.
However, when applied to natural language generation (NLG) tasks, this approach often results in catastrophic forgetting of the target language.
To mitigate this issue,\citet{maurya2021zmbart} proposed fine-tuning only the encoder layers of mPLMs while keeping the word embeddings and all decoder layer parameters frozen. In our preliminary investigation, we found that this technique did not completely prevent code-switching in XLT-QG, as shown in Figure \<PIC>.
Specifically, the models struggled to fully grasp interrogative structures in the target language, a phenomenon we refer to as ``interrogative code-switching.''
In this study, we propose a method that enables small mPLMs to learn interrogative structures without relying on target language data during training. In our preliminary investigation, we found that this technique did not completely prevent code-switching in XLT-QG, as shown in Figure \<PIC>.
Specifically, the models struggled to fully grasp interrogative structures in the target language, a phenomenon we refer to as ``interrogative code-switching.''
In this study, we propose a method that enables small mPLMs to learn interrogative structures without relying on target language data during training. As illustrated in Figure \<PIC>, we divide the task into two stages.
In the QTC stage, a classification model identifies the type of question to be generated.
We focus on Wh-questions, categorizing them into eight types based on English interrogative words.
While the type of question is primarily influenced by the type of answer, the model considers both the answer and the context.
This is crucial, as the same answer can result in different types of questions depending on the context.
For example, the number ``911'' could refer to a quantity, year, or proper noun. The set of question exemplars corresponding to the question type identified by the QTC model is used in the QG stage.
These exemplars are pre-created for each question type and language, as detailed in Section \ref{sec:experiments/data}.
By leveraging the shared interrogative structures among the exemplars, the QG model generates questions using the provided answer and context.
Both the QTC and QG models are trained exclusively on English QA data and can be deployed to new languages without the need for additional training with target language data.


\section{Introduction}
Solving Mathematical reasoning problems is a significant challenge for current LLMs~\cite{madaan-etal-2022-language,openai2023gpt4}. This task requires interpreting information, identifying relevant mathematical concepts, and formulating equations to solve the problems~\cite{ahn2024large}. Due to computational errors in LLMs~\cite{wei2023chainofthought,gao2023pal}, using CoT~\cite{wang-etal-2023-plan,wei2022chain,chen2024diahaludialoguelevelhallucinationevaluation} solely implemented in natural language can lead to calculation mistakes~\cite{lewkowycz2022solving,wei2023chainofthought,gao2023pal}. The most common practice currently is to use PoT~\cite{chen2023program} for handling mathematical reasoning tasks, by guiding the large model to write the code that is then computed using tool calls.
However, we made a surprising discovery recently: when a problem is phrased in a manner closer to verbal scenarios (for example, the question is ``One apple costs three dollars, how much for three apples?'' instead of ``3$\times$3=?''), PoT tends to make more reasoning errors or text comprehension mistakes, but this phenomenon is almost non-existent in CoT. For such problems, CoT can correctly reason out the answer, whereas PoT makes mistakes. We refer to this type of error as~\textbf{Code Translation Error (CTE)}. We report the percentage of CTE on five datasets with multiple types of models, the results illustrated in Figure~\<PIC>. This error is due to the amount of training data for natural language far exceeding that for code. In the scope of CodeLlama’s pretraining data, which includes 500 billion code tokens, this represents a small fraction compared to the 2 trillion natural language tokens used in the Llama-2 model~\cite{rozière2023code,touvron2023Llama}. Natural language is more suitable for semantic analysis, planning, and abstract reasoning than code~\cite{gou2023tora}.
Existing work also finds the advantage of Natural language, but they have not effectively utilized the reasoning capabilities of natural language. Current research focuses on the following approaches to integrate natural language to enhance the precision of code: (1) Using natural language prompts to guide the model in writing code~\cite{gao2023pal,toshniwal2024openmathinstruct,wang2023mathcoder}: write a brief step in natural language before generating code. (2) Employing methods like self-correction and hybrid approaches to generate answers in multiple stages~\cite{gou2023tora,yue2023MAmmoTH,gou2023critic}. (3) Utilizing prompts like ``rethink question''~\cite{deng2023rephrase} to have the model first paraphrase the question, thereby avoiding comprehension errors.
However, existing methods fall short in two main aspects: First, using few natural language steps or simple paraphrasing methods alone is insufficient for effectively controlling code generation; a more comprehensive natural language reasoning process is necessary to generate more reliable code. Secondly, reasoning within LLMs is not always faithful~\cite{lanham2023measuring,bao2024llms,turpin2023language}. Frequently, the final answers seem to be derived directly from the questions themselves rather than aligning with the reasoning process. Consequently, even correct reasoning can lead to incorrect answers.  To more effectively utilize natural language reasoning to enhance PoT, we propose~\textbf{Human-Think Language (HTL)}: A novel information-control-based approach to utilize complete CoT reasoning steps to control PoT generation. HTL is inspired by the way humans write code. Humans consider the entire reasoning process using natural language, and the code can fully rely on natural language reasoning. On the right side of Figure~\<PIC>, we highlight the parallels between our integrated model and the human approach to solving mathematical problems. Compared to previous works, our framework offers a strong capacity for aligning calculation with reasoning by integrating CoT and PoT. We design Focus Attention mechanism that, during code generation, concentrates solely on information from CoT to promote the chain reasoning better, thereby biasing the answer to be more faithful to CoT. On the other hand, using complete CoT reasoning tends to lead LLMs to use mathematical induction to enumerate reasoning steps verbosely, which results in repetitive generation. We incorporate the error assessment function based on PPO~\cite{ppo}, leveraging reinforcement learning to penalize repetitive generation. We conduct experiments based on CodeLlama-7B and Mistral-7B and achieve outstanding results on eight datasets using only self-distillation data. In summary, our contributions are as follows: (1) We are the first to conduct a detailed evaluation of current closed-source models, open-source base models, and specialized models. We highlight the shortcomings of PoT and propose that using full natural language reasoning to enhance PoT performance is essential. (2) We propose an advanced model named HTL, which utilizes the complete reasoning process of CoT to enhance PoT. HTL incorporates a novel Focus Attention that approximates chain reasoning, complemented by an error assessment function designed to prevent repetitive generation. (3) We evaluate our work on eight mathematical reasoning datasets, and our experimental results demonstrate that our method achieves outstanding results. HTL shows significant effectiveness in in-domain datasets, out-of-domain datasets, and natural language inference task, demonstrating strong usability and potential. The design of HTL is divided into three parts: reasoning format, Focus Attention, and error assessment function based PPO.


We define a structured entity as a named entity with associated properties and relationships with other named-entities.
Extracting a structured entity from unstructured text therefore \new{implicitly} combines named-entity recognition, entity-property extraction, relationship extraction, and coreference resolution.
Fig.~\<PIC> shows an illustration of the structured entity extraction task.
Given a text description, we aim to first identify the two entities mentioned \textit{``Bill Gates''} and \textit{``Microsoft''}.
During this phase, we need to also address the \new{coreference resolution} and identify that \textit{``Bill Gates''} and \textit{``Gates''} are the same entity.
Then, given some predefined schema on the all possible entity types and property keys, the exact types, property keys, property values on all identified entities in the text are expected to be predicted, as well as the relations between these two entities (i.e., \textit{Bill Gates} co-founded \textit{Microsoft}).
Such extracted structured entities may be further linked and merged to automatically construct KBs from text corpora.
We present our formalization of the \textit{structured entity extraction} task which lays the foundation for evaluation for this information extraction task.
Along with this, we propose a new evaluation metric with numerous variants for measuring the similarity between the predicted set of entities and the ground truth set, which is appropriate for our task (see Sec.~\ref{sec:prelim}). Among the three stages depicted, \textit{\textcolor{blue}{pred\_ent\_names}}, \textit{\textcolor{blue}{pred\_type\_and\_property}}, and \textit{\textcolor{blue}{pred\_val}} are special tokens to indicate the task.
For each model prediction behavior, the first ``$\Rightarrow$'' indicates inputting the text into the encoder of MuSEE, while the second ``$\Rightarrow$'' means inputting the encoded outputs into the decoder.
All tokens in \textcolor{blue}{blue} are the prompt tokens input into the decoder which do not need to be predicted, while all tokens in \textbf{bold} are the model predictions.
For the stage 1, we emphasize that MuSEE outputs a unique identifier for each entity in the given text. Taking the example in Fig.~\<PIC>, the first stage outputs ``\textit{Bill Gates}'' only, rather than both ``\textit{Bill Gates}'' and ``\textit{Gates}''. This requires the model implicitly learn how to do coreference resolution, namely learning that ``\textit{Bill Gates}'' and ``\textit{Gates}'' are referring to the same entity. Therefore, our approach uses neither surface forms, as the outputs of the first stage are unique identifiers, nor the entity titles followed by entity linkings.
For stage 2, the MuSEE model predicts the entity types and property keys, which are all represented by special tokens. Hence, the prediction can be made by sampling the token with highest probability over the special tokens for entity types and property keys only, rather than all tokens.
Notice that we do not need to predict the value for ``\textit{type}'' and ``\textit{name}'' in stage 3, since the type can be directly derived from the ``\textbf{ent\_type\_}'' special key itself, and the name is obtained during stage 1.
The tokens in the bracket ``\{..\}'' are also part of the prompt tokens and are obtained in different ways during training and inference.
During training, these inputs are obtained from the ground truth due to the teacher forcing technique~\citep{raffel2023exploring}.
During inference, they are obtained from the output predictions from the previous stages.
The full training loss is a sum of three cross-entropy losses, one for each stage.
An illustration of our model's pipeline is shown in Fig.~\<PIC>.
More implementation details are elaborated in Appendix~\ref{appendix:details}.


{INTRODUCTION}
The visual servoing problem~\cite{538972} involves the challenge of controlling the motion of a robot by utilizing feedback from visual sensors, typically cameras, to adjust its actions in real-time. This process entails the robot's ability to interpret visual data to determine its relative position and orientation with respect to target objects or locations within its environment. The core objective is to enable the robot to perform precise movements or reach specific goals by continuously comparing the current visual scene against a desired configuration or outcome. This approach can been seen in Fig. \<PIC>. This approach requires sophisticated algorithms for image processing and control theory to bridge the gap between visual perception and mechanical action, thereby allowing the robot to adapt its movements based on the visual feedback it receives.
Recently, there has been a lot of progress in optical flow-based visual servoing methods~\cite{rtvs, katara2021deepmpcvs, argus2020flowcontrol, harish2020dfvs, 9550239}. These methods are shown to be highly precise in reaching their goals with some guarantees of convergence. However, the utility of visual servoing has remained limited due to major limitations common to all servoing algorithms: 1) They necessarily require a goal image during test time. This makes it quite tough for visual servoing algorithms to be applied in real-world navigation or manipulation since if we already have a map of the environment, there are better ways to reach the goal pose than through a target image. 2) Visual Servoing cannot work if there is not much overlap between the initial and target image. 3) Visual Servoing can only accommodate feedback from a single camera.
Solving each of these problems can greatly enhance the utility of visual servoing methods. For example, solving the problem of final image generation based on the skill the robot is executing can make servoing quite useful for real-world tasks. Imagine a drone mounted with a monocular camera trying to cross a door. The robot will first have to visualise the approximate position of the door just before it crosses it; then, it will have to visually servo to the imagined image and then apply a simple hardcoded skill to cross the door. This pattern of imagining a goal, servoing to the goal and applying a hardcoded skill can be repeated for many skills, both in navigation and manipulation. Another good example of a skill we can solve using servoing is the `reaching' skill, which is a part of several manipulation tasks. For instance, take the example of `unplugging the charger', where the robot has to `reach' a particular grasping pose before applying a hardcoded policy.
In this paper, we leverage the recent advancements in diffusion-based image editing to provide a much-needed and major update to servoing algorithms. Our contribution can be stated as follows:
{METHOD}
{Problem Formulation}
At any given moment, our system receives an RGB image $I_t$ from the robot's camera sensor, along with a textual prompt $P$ that specifies the task to be accomplished. Our objective is to accurately predict and implement the necessary control commands, denoted as [$v_t, \omega_t$], to fulfil the task described in the prompt.  Tasks vary, ranging from navigating through a doorway to disconnecting a charger. Note that in traditional servoing algorithms, the final image $I_g$ is usually provided to perform the task. This greatly limits the utility of the visual servoing models. To address this challenge, we divide our approach into two distinct phases. Initially, our framework is designed to conceptualize a subgoal $I_g$ based on the task and current camera feedback $I_t$. Next, we aim to attain this subgoal image $I_g$ by employing a servoing algorithm. In the forthcoming sections, we will elaborate on the components of our Imagine2Servo framework. We begin by introducing our foresight model for sub-goal generation in Section \ref{foresight} followed by the mechanics of our servoing framework \ref{servoing}, which facilitates reaching the subgoal. We then describe our overall framework in \ref{overall} and training details in \ref{training}. Our overall pipeline is summarised using Fig. \<PIC>.


The in-band radiative load emitted by the optical elements is the product of their absorptive loss and physical temperature.The absorption loss of a material is directly related to the complex permittivity ($\varepsilon = \varepsilon' - i \varepsilon''$) of a material. Absorption is discussed numerically throughout this paper as a material's loss tangent, which is the ratio of the real part over the complex part of the permittivity, denoted as $\tan\delta$. The permittivity of a material varies with temperature, and the $\tan\delta$ at millimeter wavelengths of many polymers decreases with cryogenic cooling, reducing the absorption and emission of a polymer optic in-band \citep{Frank1977,Schnabel2014}. However, the reduction in optical emission at lower temperature is driven primarily by the lower physical temperature rather than the lower loss \citep{Lamb1996}.
The entire instrument is placed within a vacuum cryostat to cool both the optics and the sub-kelvin detectors. However, the cryostat obviously requires a transmissive window to observe external signals, which must remain in contact with the ambient environment. Vacuum windows must have low transmission loss in band and be strong enough to withstand the force of atmospheric pressure. Previously, millimeter-wave vacuum windows have been made of ceramics [such as fused-silica \citep{piper2021}], plastic foams [including various formulations of Zotefoam \citep{Keck2015B}], or bulk plastic materials [primarily high density polyethylene (HDPE) or ultra high molecular weight polyethylene (UHMWPE) \citep{bicep3,delessandro2018}]. A more detailed history of the development of millimeter vacuum windows can be found in \cite{Barkats2018}.
The larger aperture sizes of modern CMB receivers, however, limit the materials available for use. BICEP3 and BICEP Array have nominal clear apertures at the window of 730 mm, while the full outer diameter of the window is 900 mm. Current plans for the CMB-Stage 4 Small Aperture Telescopes (SATs) and Large Aperture Telescope (LATs) include similarly sized windows \citep{abazajian2019cmbs4}.
In addition to the initial elastic deflection, plastics will undergo a visco-elastic creep deformation under continuous load. The creep deformation is logarithmic with time; thus, most of the deflection occurs very quickly, and deflection out to longer time scales is simple to predict with a model. We need to determine that the window will not creep into contact with the filters below (see Figure \<PIC>). The initial elastic deflection is described reasonably well by flat plate predictions (previously described in Eq. \ref{eq:delta}) from \emph{Theory of Plates and Shells} \citep{TimoshenkoStephen1959Topa}.Ceramics become prohibitively expensive at those scales. Plastic foams would become impracticably thick and lossy: the primary advantage of the foam windows are their low index of refraction and low transmission loss, but they have the disadvantage of being very weak. A suitably thick foam window would include many laminated layers, which would make their loss properties significantly worse. For example, the 150 GHz \emph{Keck} receiver windows were 120 mm thick and estimated to cause $\lesssim$2\% transmission loss \citep{Keck2015B}; the doubled aperture size for new BICEP receivers would require a foam window four times thicker. A window with such high transmission loss ($\lesssim$8\%) would add approximately 3 pW optical load to a 150 GHz detector, swamping out all other sources of optical power at that frequency.
Bulk plastics, being relatively weak though still stronger than plastic foams, must also become significantly thicker to maintain a suitable safety factor for a vacuum vessel (discussed further in Section \ref{sec:strength}). For BICEP3, a 95 GHz receiver, we previously estimated that over half of the instrument loading originated from the 31.8 mm thick HDPE slab window \citep{bicep3}. That window has since been replaced by the thin window presented in this paper.
We also show the relative change in per-detector white noise and survey time for different window thicknesses. For the lowest frequency band, at 30--40 GHz, decreasing the window thickness only yields marginal improvements in white noise and mapping speed. For bands above 40 GHz, however, a substantial reduction in window thickness from the nominal could potentially decrease noise by tens of percent. At the high frequencies in the 270 GHz band, such a window thickness reduction could potentially decrease the survey time by as much as 50\%. The window is the only element in the optical chain where improvement would produce such a significant return, primarily due to the window's relatively high temperature.
For safety, we performed this test hydrostatically. As water is nearly incompressible, minimal energy will be stored within the fluid as pressure is increased; energy is only stored within the deflected window and surrounding enclosure. Given the safety factors of 3 calculated in Table \ref{tab:strength}, we used standard utility water pressure (roughly 60 psi or 4 atm)  as that was expected to exceed the ultimate tensile strength of these windows and produce a failure.
The BICEP Array frame and clamp were designed specifically to hold thin windows with a knurled pattern on the outer radius, with bolts running through the knurling to maximize concentration of force over relatively small points \citep{Barkats2018}. This knurled pattern on the frame and clamp works well but results in an unique window clamping configuration. We therefore used an actual frame and clamp for the hydrostatic test. Preexisting flange holes for circulating gas between the window and BOPP membrane were used to push the water into the chamber above the window. We sealed the water chamber with a 1/2" aluminum blanking plate and two 3/16" thick blended EPDM gaskets. The chamber was clamped with two sets of correctly torqued 1/4" bolts and C-clamps for additional clamping force, which are visible in a ring around the window in Figure \<PIC>.
We reached 68 psi in the hydrostatic test using only the standard utility water supply. At that point we took a side-on picture of the deflected window, and estimated the deflection (and therefore radial strain) of the window (see Section \ref{sec:creep} for a description of the calculations). The window had deflected approximately 75 mm. which corresponds to a radial strain of approximately 3\%, well below the typical failure strain of 10\%. In an attempt to produce a failure, we added compressed air to the system, increasing the pressure to 85 psi. At this pressure water began to leak around the gaskets, and the test was stopped. The end state of the window is shown in Figure \<PIC>, with the water pressure valve showing the final pressure on the right and the water leaking out around the gaskets on the left. We note that 85 psi is twice the predicted failure pressure for a safety factor of 3, a degree of uncertainty that emphasizes the importance of empirical testing of this novel structure.
Given these results, we find that the safety factor of the thin windows is at least approximately 5.7 at sea


The structure of this article is as follows: a summary of the data to be analyzed is presented in Sec. \ref{sec:data}, with the implemented methodology being described in Sec. \ref{sec:method}, including  an overview of the analysis sequence in Section \ref{sec:proc}, details on the adopted stellar component analysis and how the stellar templates are created in Sec. \ref{sec:rsp}, and a description of the delivered data products in Sec. \ref{sec:DP};  the reliability of the methodology is tested with extensive simulations described in Sec. \ref{sec:code}, including both physically motivated (Sec. \ref{sec:sim_idea}) and purely empirical (Sec. \ref{sec:sim_real}) simulations; the accuracy and prevision in the recovery of the emission lines and the stellar population properties are described in Sec.  \ref{sec:sim_el}, \ref{sec:sim_MaStar}, and \ref{sec:sim_el}; an example of the use of the DAP is presented in Sec. \ref{sec:show}, in which we analyze a deep exposure centred in the Huygens region of the Orion nebula, including a description of this particular dataset in Sec. \ref{sec:O_dataset}, with a summary of the performed analysis results in Sec. \ref{sec:O_results}, a detailed exploration of the generated deep integrated spectrum in Sec. \ref{sec:O_int}, the list of detected emission lines in Sec. \ref{sec:O_el}, the physical properties derived from those emission lines in Sec. \ref{sec:ori_fprop}, and the spatial distribution of the emission line fluxes presented in Sec. \ref{sec:ori_spa}; how to download the code is presented in Sec. \ref{sec:dist}, and finally a summary of the main results of this study is presented in Sec. \ref{sec:conc}.
Observational data for the Local Volume Mapper (LVM) project are obtained using a newly constructed facility at Las Campanas Observatory. As indicated before, the instrumental setup comprises four telescopes: one devoted to the acquisition of the science exposures (using the ultra-wide IFU comprising 1801 fibers described before), two monitoring the night-sky spectrum towards the east and the west (using 59 and 60 fibers, respectively), and one dedicated to the acquisition of spectrophotometric calibration stars (using 24 fibers). This setup feeds into three DESI-like spectrographs covering the wavelength range from 3600 to 9800 Angstroms with a resolution of R $\sim$ 4000 at H$\alpha$. Each spectrograph features a dichroic system, which divides the light received from each fiber into three wavelength ranges: (b) blue, from 3600 to 5800 \AA, (r) red, from 5750 to 7570 \AA, and (z) infrared, from 7520 to 9800 \AA. The size of the LVM's ultra-wide field IFU projected in the sky ensures detailed spectral and spatial coverage across the survey's targets, including the Milky Way, Magellanic Clouds, and a selection of local volume galaxies.
The LVM survey's data reduction pipeline (DRP, Mejía-Narváez in prep.) follows the procedures described by \citet{sanchez06a}. The actual code is based on the {\sc py3D} reduction package, initially developed for the CALIFA survey \citep[][]{dr1}. Originally written in Python 2.7, it has been updated to Python 3.11, incorporating specialized procedures to cater to the unique features and requirements of the LVM dataset.
Figure \<PIC> shows the analysis flow of the DAP for each individual spectrum within each LVM exposure { (i.e., no spatial binning is performed in general within the DAP)}. It mimics the one implemented in \pyp, as described in \citet{pypipe3d}, Fig.~1, slightly modified to adapt it to the peculiarities of the LVM data regarding the modeling of the stellar component (discussed in Sec \ref{sec:rsp}). We include here a brief summary of the main steps that comprise the analysis to avoid unnecessary repetition. The key steps performed by the LVM-DRP include: (i) initial raw data preprocessing to merge readings from different amplifiers into a single frame for each channel (b, r, and z) of each spectrograph (producing 9 different frames), followed by bias removal, gain correction, and cosmic ray identification and masking; (ii) identification and tracing of fiber spectra in each CCD of each spectrograph, including assessment of the FWHM along both dispersion and cross-dispersion axes; (iii) extraction of spectra using the established trace and width parameters, assuming a Gaussian shape for each fiber's spectrum projected along the cross-dispersion axis, and performing a concurrent stray-light correction; (iv) wavelength calibration and linear resampling of the extracted spectra; (v) differential correction for fiber-to-fiber transmission discrepancies; (vi) flux calibration based on the stars acquired by the spectrophotometric telescope simultaneously with the science observations; (vii) combination into a single spectrum of the spectra acquired in each channel (b, r, and z) for each science fiber, weighting by the inverse of the errors in the overlapping regions between arms; (viii) estimation and subtraction of the sky spectrum using the spectra obtained by the west and east telescopes devoted to observing the sky; (ix) implementation of an astrometric solution for each observation, using the information provided by the guiding cameras. Errors are propagated during each step of the data reduction.
The result of this analysis shows the spectra of a subset of the RSP library generated by selecting 108 clusters, together with the spectra of the original RSPs included in each cluster. By construction, the shape of all spectra grouped in the same cluster is rather similar. However, as already indicated, this does not mean that each of those spectra corresponds to the same or similar physical parameters. This is evident when exploring the projected PDFs of each cluster in the space of observed parameters. Figure~\<PIC> shows three examples of PDFs corresponding to three different clusters whose spectra (\#18, \#34, and \#47), together with the overall distribution for the full RSP library comprising 108 spectra. A simple exploration of the distribution of each PDF in the space of physical parameters shows that for some parameters the RSP corresponds to almost a single value (e.g., $T_{\mathrm{eff}}$), while for other parameters they present a bi-valuated (or multi-valuated) distribution (e.g., log($g$)). Besides this multi-valuated distribution, the errors and degeneracies are considerably different for each parameter and pair of parameters, also varying from RSP to RSP. For instance, $T_{\mathrm{eff}}$ and log($g$) do not present any clear degeneracy, while [Fe/H] and [$\alpha$/Fe] present a degeneracy that is described by an anticorrelation of the PDFs of both parameters. We should stress that these multi-valuated distributions, degeneracies, and errors can be mitigated, for instance, by the selection of a larger number of clusters ($n_{cl}$). Obviously, the selection of a very low $n_{cl}$ number, e.g., 4-12, increases them. However, increasing this number beyond 108 does not produce any significant improvement in the results, as we will translate the degeneracy to the fitting procedure: the DAP algorithm would choose a different RSP among the degenerated ones in each MC iteration described in Sec. \ref{sec:proc}. This will only slow down the process without improving the quality of the modeling.  Using external information, like the distance provided by GAIA \citep{gaia2} and the absolute magnitude may break some of the degenerancies. . { We may explore how to implement them in future versions of the DAP, or as a part of the post-processing of the data required to interpret the results. Nevertheless this is out of the scope of the goals of the current manuscript, aimed to describe the overall procedure, without focusing on the physical interpretation of the results, which would require a much more detailed exploration.}
Structure of extensions included in the each RSP file, where: (i) \#WAVE is the number of spectral pixels; (ii) \#RSP is the number of templates in the library; (iii) \#PAR is the number of parameters associated to each template, as described in the text; (iv) \#RSP\_ORG is the number of RSP templates or stellar spectra from the library adopted to build the described RSP template; and (v) (NX,NY) are the dimension of the projection of the PDF in the considered space of parameters.
We store the result of the clustering procedure described previously in a single FITS file with multiple extensions that comprises: (i) the RSP spectra themselves stored in a raw-stacked spectra (RSS) format (SPECTRA); (ii) the uncertainty spectra that retain the dependencies among the spectra coadded to create each RSP template, in the same format (ERROR); (iii) the LW average physical parameters that correspond to each RSP template (PARAMETERS), together with the same values for the original stellar library or RSP template from which the current template was built (ORG\_PARAM); and (iv) the PDFs projected in the space of physical parameters associated to each template, as shown in Fig. \<PIC>. These PDFs are stored as cubes in which each slice along the z-axis corresponds to the individual PDF projection for the considered pair of parameters (e.g., PDF\_TEFF\_LOGG corresponds to T$_{eff}$ vs. log(g)) for each RSP template in the library. In this way,  the PDF corresponding to a particular stellar decomposition can be generated by multiplying each slice of the cube by the corresponding weight/coefficient of the decomposition and averaging all of them. A set of RSP-template files are distributed (\url{https://ifs.astroscu.unam.mx/sfsanchez/lvmdap/}) together with some python notebooks illustrating how to handle them (included in the DAP distribution).


A significant portion of chemical information remains locked in unstructured formats within printed or digital documents, such as patents and scientific papers. In many of these documents, especially in the field of chemistry, molecular structures are often depicted as images. These chemical structures figures are critically important for drug discovery, patent examination, and the retrieval  of chemical data. But they are difficult to extract into machine-readable text. Automatically parsing molecular structures from these document images, a process known as Optical Chemical Structure Recognition (OCSR), becomes particularly important. With the growing popularity of Large Language Models (LLMs), many efforts are now focusing on applying them to the understanding scientific literature. Parsing molecular structure images into structured, understandable text also unlocks the potential of large models in processing patents and literature in chemistry-related fields.
OCSR aims to automatically convert chemical structure diagrams from scientific literature, patents, and other scanned documents into machine-readable string such as SMILES ~\cite{weininger1988smiles} representation. In order to better address the challenges of the OCSR task in real-world literature. We introduce a new end-to-end framework named MolParser for Optical Chemical Structure Recognition in the wild, illustrated in Figure \<PIC> The main contributions of this paper. SMILES, though widely used for molecular representation, has notable limitations in handling complex chemical entities. It struggles with representing Markush structures, which are used in patents to describe a broad class of molecules by allowing variability at certain positions, enabling the protection of entire families of compounds. Additionally, SMILES cannot effectively handle connection points, abstract rings, ring attachments, duplicated groups or polymers, all of which require a level of flexibility that its linear format does not support. Furthermore, SMILES is not well-suited for tasks involving large models, such as Markush-molecule matching, as its structure lacks the clarity and hierarchical organization needed for efficient interpretation by machine learning models. These limitations hinder its utility in advanced cheminformatics applications.
We etend the SMILES representation method, to more effectively represent the Markush structures commonly found in patents, as well as complex compositions such as connection points, abstract rings, ring attachments, duplicated groups, and polymer structures. Additionally, we ensure that this approach is compatible with RDKit and LLM-friendly, facilitating subsequent analysis and processing tasks. The extended SMILES format will be denoted as:
\texttt{SMILES<sep>EXTENSION}. Where \texttt{SMILES} refers to the RDKit-compatible SMILES representation. Special token \texttt{<sep>} serves as the separator, and optional \texttt{EXTENSION} represents supplementary descriptive used to handle complex cases such as Markush structures.
In \texttt{EXTENSION}. We use some XML-like special tokens to represent certain special functional groups. For Markush R-groups and abbreviation groups, we add special token \texttt{<a>} and \texttt{</a>} to encapsulate descriptions of these special substituents. Similarly, we use special token \texttt{<r>} and \texttt{</r>} for ring attachments; \texttt{<c>} and \texttt{</c>} for abstract rings. Additionally, there is a special token \texttt{<dum>} representing a connection point. For the specific description of each functional group, we use the following format: \texttt{[INDEX]:[GROUP\_NAME]}. Figure \<PIC> shows a example of our extended SMILES. Although there are numerous complex Markush structures in actual patents, our extended SMILES rules can still adequately address these cases. For more details, please refer to the supplementary materials.


In this paper, we propose a time series anomaly detection method based on INR reconstruction (TSINR for short).
Specifically, we introduce a transformer-based architecture to predict the INR parameters of the given time series data.  To better learn and reconstruct time series data, we design a novel form of continuous function to decompose time series \cite{cleveland1990stl,fons2022hypertime}. The designed function mainly comprises three components and individually learns the trend, seasonal, and residual information of time series. In addition, to further enhance the capability of INR to capture inter- and intra-channel information, we propose a group-based architecture to specifically learn the complex residual information. Simultaneously, we leverage a pre-trained large language model (LLM) to encode the original data to the feature domain, amplifying the fluctuations of anomalies from both time and
channel dimensions and enabling INR to subsequently better differentiate normal and abnormal points. The major contributions of this paper are summarized as follows:    \item We utilize the spectral bias property of INR to prioritize fitting low-frequency signals and enhance sensitivity to discontinuous anomalies, thereby improving anomaly detection performance. A transformer-based architecture is employed to generate the parameters for INR, requiring only a single forward step in the inference phase.
    \item We design a novel form of INR continuous function, which mainly consists of three components to implicitly learn the unique trend, seasonal, and residual information of time series. Furthermore, a group-based strategy is proposed to further learn intricate residual information.
    \item We leverage a pre-trained LLM to encode the original time series to the feature domain, enabling amplification of the fluctuations of anomalies in both time and channel domains that facilitate INR to be further sensitive for noncontinuous anomaly areas. Ablation studies and visual analysis validate the aforementioned capacity to better distinguish anomaly points via our proposed framework.
    \item Extensive experiments demonstrate the overall effectiveness of TSINR compared with other state-of-the-art methods on seven multivariate and one univariate time series anomaly detection benchmark datasets.
Time series anomaly detection methods primarily include statistical, classic machine learning, and deep learning methods. As shown in Figure \<PIC>, we innovatively propose a INR continuous function to better learn and reconstruct time series data. Inspired by classical time series decomposition methods \cite{cleveland1990stl,fons2022hypertime}, the proposed INR continuous function $f$ consists of three components, including trend $f_{tr}$, seasonal $f_{s}$, and residual $f_{r}$. Statistical methods rely on analyzing the statistical properties of the data to identify patterns that deviate from the expected behavior. They are valuable for their simplicity and interpretability, but have limitations in capturing complex patterns \cite{cleveland1990stl, coluccia2013distribution}.
Classic machine learning methods rely on manual feature extraction and various algorithms like clustering \cite{ruff2018deepsvdd,shin2020itad}, density estimation \cite{breunig2000lof,yairi2017data-MPPCACD,zong2018deep-DAGMM}, and isolation forests \cite{liu2008isolation-iForest} to identify anomalies in structured data. However, because they require manual feature extraction and selection, they can be labor-intensive and less effective at capturing complex patterns in data.
In this section, we analyze the effectiveness of the proposed decomposition components and group-based architecture. The decomposition components indicate the three components (i.e., trend, seasonal, and residual) designed in our paper. And the group-based architecture is proposed for the residual block.
The main purpose of the decomposition components is to extract the unique trend and seasonal information of the time series data. The results in Table \ref{ablation_group} indicate that capturing these distinctive features significantly enhances the capability for anomaly detection. In addition, the group-based architecture is designed to enhance the representational capacity of INR for multivariate data. Experimental results indicate an improvement in the capability for anomaly detection when employing the proposed group-based architecture. This is because modeling multiple variables and capturing both inter- and intra-channel information with a simple continuous function, which only consists of fully-connected layers, is challenging. Our approach addresses this by dividing the variables into several groups and applying independently fully-connected layers in different groups, thereby reducing the number of variables each group needs to model and improving the representational capacity. The global layers extract the inter-channel information, while the group layers selectively focus on detailed information for specific channels. This enhances the representational capability for each variable without losing any knowledge. Detailed ablation studies on the number of groups are left in Appendix \ref{app_group_num}.
Further, we prove the validity of the pre-trained LLM encoder, which is utilized to encode the data into the feature domain to amplify the fluctuations of anomalies and thereby enhance the capability of TSINR in identifying anomalies. Table \ref{ablation_llm} displays the ablation studies of the pre-trained LLM encoder. For the multivariate datasets, it can be observed that applying this encoder enhances the performance of anomaly detection. To further demonstrate the effectiveness, we compare the raw data with the features extracted through the encoder. The figures in the first row illustrate that during the time interval when anomalies occur, the extracted features exhibit more pronounced fluctuations compared to the original data. Furthermore, we validate the robustness of TSINR with the synthetic data generated for time series anomaly detection. It has univariate time series and involves different types of anomalies \cite{yang2023dcdetector,lai2021revisiting}, including the point-wise anomaly (global point and contextual point anomalies) and pattern-wise anomalies (shapelet, seasonal, and trend anomalies). As shown in Figure \<PIC>, the
red points are anomaly points and the red areas are anomaly areas. This implies that the discontinuity in anomalies is increased in time domain. Also, these extracted features incorporate inter-channel information, providing a manifestation of anomalies among all variables. As shown in the second line, the features exhibit anomalous fluctuations in the same time interval as other channels, whereas the original data only shows a brief peak. This verifies that the anomalies are shared in channel domain. Based on these results, we indicate that utilizing the pre-trained LLM encoder can effectively enhance abnormal information both intra- and inter-channel. This aligns with the spectral bias of INR, making our model more sensitive to anomalous data.


RL techniques are generally used to learn optimal policies and actions leading to best rewards. These techniques can be classified into value-based (critic-centric), policy-gradient (actor-centric), or hybrid approaches.
Actor-critic algorithms merge the benefits of both by employing two key components: the actor, responsible for policy optimization, and the critic, which evaluates the policy \citep{sutton2018reinforcement}. This hybrid approach has demonstrated success in various domains, including continuous control problems \cite{pmlr-v48-mniha16, haarnoja2018soft, haarnoja2018softapp} and with generative adversarial networks (GANs) \citep{goodfellow2014generative}, where a generator and a critic network collaboratively optimize performance.
Inspired by the success of actor-critic approaches, we introduce a (stateless) actor-critic algorithm, as provided in Algorithm~\ref{alg:ssac}, tailored to our infinite continuum bandit setting, enabling autonomous learning of effective prompts with a constrained evaluation budget and outperforming previous state-of-the-art black-box prompt learning approaches. Both the actor and the critic are parameterized by neural networks. In typical RL problems, the agent interacts with an environment with varying states, so its policy depends on the current state. That is, the policy network generates actions based on the current state $s$, $\pi(.| s; \theta)$, where $\theta$ denotes the parameters.
However, in the problem we consider, the setting is stateless. Therefore, we define a policy network $\pi(.; \theta)$ with a constant input that produces a $d^{'}$-dimensional vector as output $\mathbf{a}$, representing the continuous action. For the critic, we use a neural network with a $d^{'}$-dimensional input (the action) and a single output that estimates the action's quality, thereby assessing the quality of the policy that proposed it. The critic network's parameters are trained to minimize the following:  Furthermore, to avoid fine-tuning, and to achieve adaptive exploration, the entropy coefficient $\alpha$ can be adjusted dynamically to maintain a target entropy level \cite{haarnoja2018softapp}, $H_{\text{target}}$, ensuring sufficient exploration, as follows:
We use a stochastic approximation of gradients, specifically the Adam algorithm \cite{kingma2014adam, reddi2019convergence}, to minimize the losses via stochastic gradient descent. The learning rates $\lambda$, $\beta$, and $\gamma$ are used for the Q-network, the policy networks, and the temperature, respectively.
The methodology of our proposed algorithm, ACING, is illustrated in Fig.~\<PIC>, showing the actor-critic interaction with the environment. In Fig.~\<PIC>, the environment is zoomed in, using the \emph{larger\_animal} dataset as an example. In the following, we provide a detailed explanation of the full methodology.
{Overview.} In each iteration \( t \leq T\), the actor-critic agent generates a continuous vector ``action'' \(\mathbf{a}\) (step 1). This action is then played on the environment, which projects \(\mathbf{a}\) into the appropriate space using a fixed projection matrix $P$ to obtain \(\mathbf{z}\). The environment then concatenates the projected vector \(\mathbf{z}\) with a set of exemplars' embeddings from \(\mathcal{E}\) and feeds it into a white-box LLM $h$ (step 2). The white-box LLM produces a discrete prompt, \(\tau\), which is evaluated using the validation dataset \(\mathcal{V}\) based on the responses from the black-box LLM $f$ (step 3). The black-box LLM generates a prediction, which is then contrasted to the true labels of the validation examples, and a score function provides a final reward to the critic. This reward is used to compute the critic's loss and update both the critic and actor networks accordingly. \\
\smartparagraph{Step  \textcircled{{\scriptsize 1}}.}
The policy-network (actor) outputs a mean and variance of a distribution from which the action is sampled. Specifically, the action, represented as a soft prompt vector, $\mathbf{a}$ $\in$ $\mathbb{R}^{d^{'}}$, is obtained by sampling from a Gaussian distribution with the outputted parameters. The network also computes the associated log probability, which is crucial for policy optimization, as it guides the learning process by balancing exploration and exploitation, by controlling the policy loss, as shown in Eq.~(\ref{policy_loss}).\\
\smartparagraph{Step  \textcircled{{\scriptsize 2}}.}
As depicted in the left side of Fig.~\<PIC>, the examples describing the task from the set of exemplars $\mathcal{E}$, along with additional text such as ``The instruction was to,'' are input into the embedding layer of the white-box LLM to generate continuous vectors. These continuous vectors are then concatenated with the soft prompt $\mathbf{z}$, projected from the action $\mathbf{a}$. The layers of the white-box LLM subsequently process the resulting concatenated vector to produce the discrete prompt $\tau$ (using the instruction generation template depicted in Fig.~\<PIC> top right). This transformation is essential for converting the continuous prompt into a discrete format suitable for input into the black-box LLM. \\
\smartparagraph{Step  \textcircled{{\scriptsize 3}}.}
As depicted in the right side of Fig.~\<PIC>, for every input $x_i$ in the validation set $\mathcal{V} = \{(x_j, y_j)\}_{j=1}^{m}$, the generated prompt $\tau$ is concatenated to the input sentence $x_i$ and fed to the black-box LLM, which generates an output sentence $\hat{y}_i = f(\tau(\mathbf{z}) \oplus x_i)$.  The output of the black-box LLM is fed into a scoring function $q(\cdot, \cdot)$, which computes the score between the predicted output $\hat{y}_i$ and the true label $y_i$. The overall score is calculated by averaging the scores across all samples, representing the reward: $r = \frac{1}{m} \sum_{i=1}^m q(\hat{y}_i, y_i)$,
where $m$ represents the number of samples and $r$ serves as feedback for the actor-critic algorithm.\\
\smartparagraph{Step \textcircled{{\scriptsize 4}}.}
The critic is responsible for evaluating the quality of actions taken by the actor using the network $Q_{\textbf{w}}$. This network estimates the expected reward for a given action $\mathbf{a}$, which is generated by the policy network $\pi$. Observing the reward $r(\mathbf{a})$, the critic computes the loss Eq.~(\ref{critic_loss}) and updates its network using Eq.~(\ref{critic_gradient}). The critic provides feedback to the actor by estimating the Q-values of the actions sampled by the policy, and the actor uses this feedback to improve its policy by maximizing Eq.~(\ref{policy_loss}). This continuous interaction between the actor and critic ensures the actor learns to take actions that maximize the expected reward while balancing exploration and exploitation.  By repeating steps 1 to 4 until the allowed budget $T$ is exhausted, the agent returns the best instruction prompt, denoted as $\tau^\star$. This prompt is then evaluated using the test set $\mathcal{T}$ on the black-box LLM, yielding the final score $\mathcal{S}^\star = \frac{1}{m^{\prime}} \sum_{i=1}^{m^{\prime}} q(f(\tau^\star, x^\prime_i), y^\prime_i)$ (using the evaluation template depicted in Fig.~\<PIC> bottom right).


Leadsee-Precip has an encoder-decoder structure as illustrated in Fig. \<PIC>. It consists of three parts: feature extraction, hidden translator, and precipitation upsampling. The feature extraction part primarily serves to derive features from circulation variables and reduce spatial resolution. To facilitate the model coupling, the input circulation variables consist of 69 channels, including 5 upper-air variables, each with 13 levels, and 4 surface variables, as the popular deep learning global atmospheric circulation prediction models. The upper-air variables are processed using 3D ConvNets for feature extraction and downsampling, allowing the model to capture the correlations across different variables and altitude levels. The surface variables use 2D ConvNets only. We set zonal circular padding in downsampling processes to ensure proper handling of the boundary conditions. After downsampling, the spatial resolution of upper-air and surface variables reduces to one-fourth. The downsampled upper-air and surface variables then concatenate together with an additional static layer, which includes three channels and has weights automatically learned by the model. A shortcut connection with 64 channels is established between the initial variables and the precipitation upsampling part to improve the accuracy of predictions. The hidden translator part uses an encoder-decoder structure as well. The spatial resolution is reduced by half after the Enc module to save GPU memory usage while the number of channels remains unchanged. The MogaNet Hidden module contains 16 MogaNet blocks, which outperforms many of the currently leading Transformer-based network structures. The Dec module doubles the spatial resolution and adds the shortcut connection from the feature extraction part.
The precipitation upsampling part increases the spatial resolution of forecast results back to 0.25 degrees. A shortcut connection from the initial variables is added to this process to better reconstruct precipitation prediction. The long-tail distribution of precipitation data presents a challenge for DL models when directly trained using RMSE, as these models rarely sample the infrequent extreme precipitation events. This leads to biased forecasts, especially for heavy rainfall. The Google team introduced a method using the logit function to adjust the loss function (called logit adjustment loss), specifically aimed at addressing the class imbalance issue \cite{menon2020long}. Inspired by this approach, we develop an IB scheme specifically designed to address the regression challenges posed by long-tail data in precipitation forecasting. The IB scheme operates on the principle that no precipitation or light precipitation, due to their high probability of occurrence (low uncertainty), actually provides less information content. Therefore, the information content demonstrated using a logit form $-\log P(y_i)$ corresponding to the $i$-th precipitation sample can naturally serve as the weight for precipitation error, balancing the differences in sample sizes across different precipitation bins. The loss function of the model aims to minimize the total "information content-weighted error" across all precipitation samples. Specifically, The IB scheme uses the following formula to calculate the weight (\textit{$W_i$}):
We use weather station precipitation data from about 10470 weather stations in the China \cite{feng2024enhancing} to conduct extra evaluations. The original data is stored in hourly precipitations and is processed to 6-hour cumulative ones for the test. When evaluating with weather station data, only the precipitation results at the station location are used.  It is important to note that we did not use the weather station dataset to train the model. Therefore, such evaluation based on “third-part” weather stations dataset is more objective but challenging.
Here, we present the evaluation results of Leadsee-Precip on the global test set, followed by verification results using weather station data, and finally, the evaluation results with weather station data of the fine-tuning model.
 Fig. \<PIC> shows the diagnosing skill of our model over the global range of $60^\circ S$ to $60^\circ N$. The precipitation in the upper panel is diagnosed by the model based on 69 variables from upper-air layers and surface data at a single time step while the lower panel illustrates the ground truth of the 6-hour accumulated precipitation of NOAA CMORPH. The diagnosed large rainfall (e.g., above the 25 $mm$ $6h^{-1}$ threshold) shows good consistency with the ground truth in both intensity and location. For instance, the model accurately reproduces several events of 6-hour accumulated rainfall exceeding 50 $mm$ in the eastern and western Pacific regions. For precipitation below 1 $mm$ $6h^{-1}$, the diagnostic results show a tendency for overestimation, with finer details appearing comparatively smoothed.
Table \ref{tab:TS_FSS_25km} presents the evaluation metrics TS and FSS of the model on the test dataset. We selected seven 6-hour accumulated precipitation thresholds: 0.1, 1, 5, 10, 25, and 50 $mm$ $6h^{-1}$. The metrics generally show a decline with increasing precipitation levels while both TS and FSS slightly increase at the 0.1 and 1 $mm$ $6h^{-1}$ thresholds, suggesting the model tends to overestimate small rainfalls. However, the FSS of 25 $mm$ $6h^{-1}$ still exceeds 0.5, highlighting the model's ability to capture heavy precipitation events.


In recent years, the Transformer architecture and foundation models, leveraging self-attention mechanisms to capture complex dependencies in text, have transformed Natural Language Processing (NLP) benchmarks \citep{vaswani2017attention, touvron2023llama2openfoundation, albert_q_jiang_mistral_2023, gemmateam2024gemmaopenmodelsbased}. Similarly, Vision Transformers (ViTs) \citep{dosovitskiy2020image} have been adapted in Computer Vision (CV) and now serve as the backbone for various tasks such as segmentation and object detection \citep{thisanke2023semantic, liu2021swin}. Despite their success, understanding the interpretability of Transformers remains a challenge due to the complexity of their latent token representations.
{Hierarchical clustering tree showing the grouping of token relevance maps for all tokens in a latent layer of the Vision Transformer, not limited to the \texttt{CLS} token. Each leaf node represents a single token relevance map, while higher-level nodes show aggregated clusters based on a clustering threshold $( \zeta)$, which controls the level of detail. Lower $\zeta$ values reveal finer details, while higher values create broader, more general clusters. This approach demonstrates how pre-trained Vision Transformers can perform unsupervised semantic segmentation, identifying meaningful patterns within token representations without requiring additional training or fine-tuning.}
everal methods have been developed to enhance the interpretability of CNN-based models \citep{simonyan2014deep, zeiler2014visualizing, selvaraju2017grad}. While some of these can be extended to Transformer architectures, they do not fully leverage the unique attention mechanisms inherent to Transformers. Recent research has introduced interpretability methods specifically designed for Transformers \citep{chefer2021transformer, abnar2020quantifying, vig2019analyzing}. However, these approaches primarily focus on explaining final model outputs, providing limited insight into the intermediate processes that lead to predictions. For instance, \citep{cheninvite} maps latent tokens into CLIP's \citep{radford2021learning} multi-modal space to find corresponding text descriptions, relying on an external text encoder for interpretability. In contrast, our approach directly interprets the latent space of ViTs, elucidating the role and function of each token within the high-dimensional space without relying on external models.
This paper introduces a framework to interpret latent tokens, offering a deeper understanding of the internal workings of Transformers. This understanding enables users to perform image semantic segmentation using pre-trained Transformer-based vision models in an unsupervised, zero-shot manner, without any additional training. We demonstrate that applying semantic segmentation based on our interpretability framework achieves state-of-the-art performance on benchmark image segmentation datasets.
Drawing inspiration from \citep{chefer2021transformer}, our method analyzes the semantic information retained by latent tokens, enabling tasks such as object selection and semantic segmentation without additional training. We demonstrate that Transformers inherently understand the semantic structure of their input, viewing it as a collection of distinct concepts. Each latent token identifies a specific concept with semantic significance, thereby shedding light on the decision-making process of these models.
As shown in Section \ref{sec: Experiment}, our framework proves effective in a range of tasks, including semantic segmentation and model interpretation. Most recent unsupervised semantic segmentation methods involve an additional training phase to learn feature representations \citep{sick2024unsupervised, hamilton2022unsupervised, li2023acseg}. Our approach, however, utilizes the understanding embedded in pre-trained models to achieve zero-shot segmentation, leveraging their inherent knowledge of images. The stronger a model's comprehension of image content, the more accurately it performs segmentation. We further demonstrate that our method is capable of interpreting large language models (LLMs) at the token level, validating its application in tasks such as text summarization. The main contributions of this paper are as follows:
Due to the skip connections in the transformer, most of the contribution of \( S_i^{(l)} \) is concentrated on \( S_i^{(l)}[i-1] \) which makes it hard to analyze other tokens' contribution. To address this issue, we replace this element with the maximum value of other elements, thereby capturing the contributions of additional tokens to the selected token.  For vision tasks, we first reshape the relevance map and then upsample it using bilinear or cubic interpolation to match the resolution of the model's input. The resulting higher-dimensional matrix is denoted as \( \Tilde{S} \). This upsampling step is essential for enabling accurate object selection and semantic segmentation tasks.
\subsection{ULTra in Unsupervised Tasks}
In this section, we aim to examine ULTra's capability to adapt to various tasks involving semantic knowledge. Importantly, it requires no additional training, leveraging the inherent understanding within transformers rather than relying on loss functions objective, final layer outputs, or fine-tuning.
As previously discussed, a relevance map can be defined for each latent token at a fixed layer, with the total number of relevance maps equal to the number of latent tokens. In the context of segmentation, the goal is to assign a class label to each pixel within an image. To achieve this, we employ clustering techniques, such as hierarchical clustering, that do not require a predefined number of classes. These techniques group the relevance maps into \( k \) distinct clusters, where \( k \) is unknown. Ideally, we aim for \( k \) to approximate the actual number of classes present in the image.
Our approach provides flexibility in adjusting the value of \( k \) by modifying the cutoff distance threshold $\zeta$ within the clustering algorithm. Increasing $\zeta$ produces fewer, broader clusters that capture general categories, such as background and foreground. Conversely, reducing $\zeta$ allows for finer segmentation, distinguishing more specific features, such as an object's head or hands. To prevent the method from disproportionately favoring larger objects, given that the number of elements in each cluster may vary, we apply min-max scaling to each cluster independently. After clustering, we define \( k \) distinct concepts by aggregating the relevance maps within each cluster. The aggregated relevance map \( \tilde{S}^{(l)}_c[x, y] \) for a cluster \( c \) is computed as:
Some examples illustrating our segmentation method is presented in Figure \<PIC>. Additionally, the hierarchy tree and the effect of the threshold are demonstrated in Figure \<PIC>.


UGAD has attracted increasing attention in recent years, many researchers focus on how to use self-supervised signals to improve the model's ability to detect anomalies in unsupervised scenarios. Early traditional UGAD methods, such as Radar~ \cite{li2017radar} and ResGCN~\cite{pei2022resgcn}, which capture anomalous features by describing the residuals of attribute information and their consistency with network information, are concise and effective. With the rapid development of GNNs, it has also been naturally introduced into the field of anomaly detection, and recent researchers have been trying to develop UGAD models based on GNNs. We broadly categorize GNN-based models into three types: MPI methods, CL-based methods, and GAE-based methods. In this section, we propose a novel unsupervised multiplex graph anomaly detection method named \model depicted in Fig.~\<PIC>, which mainly includes three key components:  \textit{(1) Original-view Graph Reconstruction}, \textit{(2) Augmented-view Graph Reconstruction} and \textit{(3) Dual-view Contrastive Learning}, to capture the anomalous information on both original and augmented graphs, and fuse the attribute and structural information of the nodes learned from the two views by contrastive learning to improve the model anomaly detection performance. It is well known that most of the GNN-based methods exploit the message passing mechanism~\cite{wang2021modeling} of GNN to learn node attributes by aggregating the neighborhood information of nodes. However, due to the feature inconsistency between normal and anomalous nodes, the original message passing mechanism may impair the model's ability to extract the anomalous information. Luo et al.~\cite{luo2022comga} improve the message passing mechanism from the perspectives of community segmentation and community structure learning and obtain node attributes by introducing the community structure. Qiao et al.~\cite{qiao2024truncated} propose a truncated affinity maximization method named TAM, which maximizes the local affinity of a node with its neighbor nodes and performs node characterization on the truncated graph instead of the original graph. by maximizing the local affinity between nodes and neighboring nodes, and performing node attribute learning on a truncated graph instead of the original graph. As a common method for mining self-supervised signals~\cite{liu2024towards,kong2024federated,liu2024bourne}, contrastive learning is also introduced to the UGAD task~\cite{wang2023cross} to capture feature inconsistencies between normal and anomalous nodes. CoLA~\cite{liu2021anomaly} is a novel contrastive instance pair sampling method that leverages local information in network data to learn information embedding from high-dimensional attributes and local structures. Zhang et al.~\cite{zhang2022reconstruction} propose a self-supervised learning framework called Sub-CR that jointly optimizes two modules, multi-view contrastive-based learning, and attribute-based reconstruction, to more accurately detect anomalies on attribute networks. In addition to focusing on node attribute learning and attribute reconstruction, contrastive learning can also be extended to the subgraph level to capture complex structural anomalies. GRADATE~\cite{duan2023graph}, a multi-view and multi-scale contrastive learning framework including node-subgraph contrastive learning and subgraph-subgraph contrastive learning, can better capture different features between normal and abnormal substructures, which helps to identify complex structural anomalies. Huang et al.~\cite{huang2023unsupervised} combine variance-based models and attribute reconstruction models to detect outliers in a balanced way.GAE-based methods:} GAE-based methods~\cite{li2019specae} are one of the mainstay methods to detect anomalies by reconstructing the node attributes and structural features in the graphs. Ding et al.~\cite{ding2019deep} use GCN to model graph topology and node attributes to learn node embeddings. Roy et al.~\cite{roy2024gad} incorporate neighborhood reconstruction in their proposed GAD-NR, aiming to reconstruct the entire neighborhood of a node based on the corresponding node representation (including local structure, self-properties, and neighborhood properties). ADA-GAD~\cite{he2024ada} introduces a two-stage anomaly denoising autoencoder framework, where the first stage trains the GAE on the generated denoised graphs, and the second stage retains only the encoder and retrains the encoder on the original graphs while introducing regularization of the distribution of node anomalies to prevent anomalous overfitting.The results of ranked anomaly scores on two datasets with injected anomalies are shown in Fig.~\<PIC> and Fig.~\<PIC>. Compared with the other four best-performing baselines, the curve of our UMGAD can converge quickly (become stable) to a position closer to the actual number of anomalies in the Retail and Alibaba datasets. In this section, we evaluate the performance of our proposed method through extensive experiments and answer the following questions: {Dataset.} We have conducted experiments on two datasets injected with synthetic anomalies: Retail Rocket~\cite{ren2024sslrec} (Retail for short) and Alibaba~\cite{fu2023multiplex}, and two real-world publicly available datasets with anomalies: Amazon~\cite{dou2020enhancing} and YelpChi~\cite{kumar2019predicting}. \textit{The detailed description of these datasets is shown in the supplementary material.} {Tables/Datasets}We compare with four categories of methods on the unsupervised GAD task.


This work focuses on complementary-label learning (CLL), a WSL problem where each label indicates only a class to which a data instance \textit{does not belong}~\cite{ishida2017learning}. CLL aims to train models with these complementary labels while still enabling accurate predictions of the ordinary labels during testing. CLL makes machine learning more practical in scenarios where obtaining ordinary labels is difficult or costly~\cite{ishida2017learning}. Additionally, CLL broadens our understanding of machine learning's practical potential under limited supervision.
Current research on CLL has introduced numerous learning algorithms~\cite{scl2020, fwd2018, gao2021discriminative, cpe2023} that have been evaluated using a diverse range of datasets, from synthetic datasets based on varied complementary-label generation assumptions to real-world datasets~\cite{clcifar2023}.
However, the performance of these algorithms often varies significantly across studies due to differences in underlying label-generation assumptions, the absence of a standardized evaluation platform, and the use of diverse network architectures~\cite{scl2020, fwd2018, ishida2017learning, cpe2023}. Establishing a fair, reproducible, and stable evaluation environment is therefore essential for advancing CLL research. For instance, variations in network architectures, such as the use of ResNet18~\cite{xu2019generativediscriminative, clcifar2023} versus ResNet34~\cite{fwd2018, scl2020}, contribute to inconsistencies in performance and hinder fair comparisons across studies. Furthermore, most CLL research has not publicly released implementations~\cite{mcl2020, cpe2023, scl2020, ComCo2023}, particularly regarding details like loss calculation and data pre-processing. This lack of accessibility presents a challenge for researchers seeking to validate and build upon existing work in CLL.
To enable meaningful comparisons among CLL algorithms and create a user-friendly environment for implementation and innovation, we introduce \texttt{libcll}, a complementary-label learning toolkit built with PyTorch-Lightning. This toolkit standardizes the evaluation process while offering extensive customization options, making it easier for researchers to develop, test, reproduce, and refine algorithms. By performing comprehensive benchmark experiments across established CLL datasets, various algorithms, and a range of complementary-label distributions, as illustrated in Figure \<PIC>,
t{libcll} provides a robust and reproducible evaluation framework. Our goal is for {libcll} to accelerate progress in CLL research and foster a collaborative research community. Additionally, there is a growing body of research focused on leveraging multiple complementary labels for supervision \cite{mcl2020, reg2021, ComCo2023, conu2023}, where each instance is assigned multiple complementary labels generated from a transition matrix without replacement. In fact, the problem of learning from multiple complementary labels can be connected to partial-label learning or negative-unlabeled learning \cite{conu2023}. Building on these concepts, \cite{clcifar2023} curated a human-labeled CIFAR~\cite{krizhevsky2009learning} dataset with complementary labels to better understand real-world CLL distributions, where the transition matrices are both biased and noisy, and each instance has three complementary labels. There remain several open problems in CLL. First, because the transition matrix $T$ is predefined, there is no universal generation process for biased complementary labels, and a general framework is needed for fair comparison. Second, in the absence of ordinary labels, the transition matrix $T$ is often assumed to be given. If a small portion of true labels is available, the transition matrix can be estimated using the anchor point method proposed in \cite{fwd2018}. These variations can lead to inconsistent experimental outcomes. Finally, without ordinary labels, the reliability of validation using only complementary labels is uncertain.
To address these challenges, we introduce \texttt{libcll}, the first CLL toolkit, to support future CLL research and advance understanding in the weakly-supervised learning community.
Previous methods on CLL}\label{sec:previous-methods}
In this section, we present a timeline of key developments in CLL, as illustrated in Figure~\<PIC>. We implement three primary categories of CLL methods in tt{libcll}: URE (unbiased risk estimator), CPE (complementary probability estimation), and MCL (multiple complementary label) methods. Additionally, we include several bridging works that connect CLL with other learning frameworks.
{URE Series of Works} The concept of complementary-label learning was initially proposed by \cite{ishida2017learning}, who introduced a risk estimator using Pairwise Comparison (PC) and One-vs-All (OvA) strategies for restricted loss functions. With biased complementary labels, \cite{fwd2018} employed forward correction to reconstruct classification risk using cross-entropy loss on complementary labels and the transition matrix. Furthermore, \cite{ishida2019complementarylabel} removed these restrictions, extending the unbiased risk estimator to support arbitrary loss functions and models.
{CPE Framework} \cite{cpe2023} offered a new perspective by approaching prediction as a decoding process with the transition matrix. This proposed decoding framework can unify various risk estimator methods \cite{fwd2018, scl2020, gao2021discriminative}.


Data Collection
Figure~\<PIC> presents the methodology of this study.
In this section we explain how we spend significant effort to construct a comprehensive dataset that is critical to this study and the like, as shown on the left side of Figure~\<PIC>.
For the convenience of operation, we define project deprecation in Section~\ref{ss:difinitiondeprecation}.
We introduce how we select projects and construct dataset in Section~\ref{ss:projectselection} and ~\ref{ss:dataprocessing} respectively.
Definition of Repository Deprecation
There is no \textit{de facto} definition of repository deprecation.
Rather than defining the concept on all repositories in the open source world, we first limit the scope of our research to GitHub.
GitHub is a code hosting platform of global prominence and extensive use, with a vast array of projects and contributors, ensuring our dataset's scale and diversity.
Besides, GitHub's rich development history and ``archive repository'' feature highlight repository deprecations, ensuring the number and correctness of positive samples.
How does HITS work in practice? How is its capability of forecasting deprecation compared with other metrics?
To answer the questions, firstly we choose the once-popular code editor Brackets, which was deprecated in favor of the more feature-complete VSCode in 2021, as a showcase.
Secondly we explore the prediction power of HITS's delta $\Delta$HITS ($\Delta HITS = HITS_{t} - HITS_{t-1month}$)
within three randomly sampled projects: Project~\cite{<PIC>}, ~\cite{<PIC>} and~\cite{<PIC>}.
{<PIC>} displays the activity statistics of the Brackets project since 2015. The figure shows the number of new events created each month for stars, issues, PRs, commits, comments, and tags. It is evident that the development of Brackets has been gradually stagnating since 2015. Yet, the project continued to receive a high and stable number of stars each month, with a notable surge in 2021. The HITS weight however, as a reliable indicator of the project's impact, has been on a steady decline since 2015.
This case demonstrates the HITS weight to be a more promising representation of projects' deprecation trends, and less prone to noise compared with other indicators.
Figure~\ref{delta_hits} illustrates the relationship between $\Delta$HITS and time for the three randomly selected projects. It is clear that for Project~\cite{<PIC>}, represented in Figure~\<PIC>, there was no negative $\Delta$HITS during the observation period. Indeed, this project has not been deprecated and is still under active maintenance. However, for Project~\cite{<PIC>} and~\cite{<PIC>}, represented in Figures~\<PIC> and~\<PIC> respectively, deprecation occurred during the observation period. In each case, a negative peak in $\Delta$HITS in the month preceding deprecation serves as a harbinger of this event.
Therefore, it is evident that the HITS weight, as a predictor of repository deprecation, exhibits a high degree of sensitivity and have the potential to accurately detect a trend towards deprecation.


Introduction
Diffusion models have shown remarkable capabilities in generating high-quality images~\citep{ho2020denoising}, with recent advances further enhancing user control over the generation process. Trained on vast data, these models can create stunning images from simple text prompts, unlocking diverse image editing and synthesis applications~\citep{meng2022sdedit,ruiz2023dreambooth,zhang2023adding}.
To pursue higher image quality and more precise text-to-image alignment, researchers are increasingly scaling up diffusion models. As shown in \<PIC>, Stable Diffusion~(SD)~\citep{rombach2022high} 1.4 only has 800M parameters, while SDXL~\citep{podell2023sdxl} scales this up to 2.6B parameters. AuraFlow v0.1~\citep{auraflow0.1} extends this further to 6B parameters, with the latest model, FLUX.1~\citep{flux1}, pushing the boundary to 12B parameters. Compared to large language models (LLMs), diffusion models are significantly more computationally intensive. Their computational costs\footnote{Computational cost is measured by number of Multiply-Accumulate operations (MACs). 1 MAC=2 FLOPs.} increase more rapidly with model size, posing a prohibitive memory and latency barrier for real-world model deployment, particularly for interactive use cases that demand low latency.
As Moore's law slows down, hardware vendors are turning to low-precision inference to sustain performance improvements. For instance, NVIDIA's Blackwell Tensor Cores introduce a new 4-bit floating point (FP4) precision, doubling the performance compared to FP8~\citep{blackwell}. Therefore, using 4-bit inference to accelerate diffusion models is appealing. In the realm of LLMs, researchers have leveraged quantization to compress model sizes and boost inference speed~\citep{dettmers2022gpt3,xiao2023smoothquant}. However, unlike LLMs--where latency is primarily constrained by loading model weights, especially with small batch sizes--diffusion models are heavily computationally bound, even with a single batch. As a result, weight-only quantization cannot accelerate diffusion models. To achieve speedup, both weights and activations must be quantized to the same bit width; otherwise, the lower-precision side will be upcast during computation, negating potential performance enhancements.
From low-rank-compute, the quantization error can be expressed as
Quantization error bound
For any tensor $\mR$ and quantization method described in \eqn{quantization_def} as $Q(\mR) = s_\mR \cdot \mQ_\mR $. Assuming the element of $\mR$ follows a normal distribution, we have
\mathbb{E} \sbr{\norm{\mR - Q(\mR)}_F} \le \frac{\sqrt{\log \rbr{\text{size}(\mR)}\pi}}{q_{\max}}\mathbb{E}\sbr{\norm{\mR}_F},
where $\text{size}(\mR)$ denotes the number of elements in $\mR$.
See \app{proof2} for the proof. From this proposition, we obtain the intuition that the quantization error $\norm{\mR-Q(\mR)}_F$ is bounded by the magnitude of the residual $\norm{\mR}_F$. Thus, our goal is to find the optimal $\mL_1\mL_2$ that minimizes $\norm{\mR}_F = \norm{\hat{\mW} - \mL_1\mL_2}_F$, which can be solved by simple Singular Value Decomposition (SVD). Given the SVD of $\hat{\mW}=\mU\mSigma\mV$, the optimal solution is $\mL_1 = \mU\mSigma_{:,:r}$ and $\mL_2 = \mV_{:r,:}$. \<PIC> illustrates the singular value distribution of the original weight $\mW$, transformed weight $\hat{\mW}$ and residual $\mR$. The singular values of the original weight $\mW$ are highly imbalanced. After smoothing, the singular value distribution of $\hat{\mW}$ becomes even sharper, with only the first several values being significantly larger. By removing these dominant values, Eckart–Young–Mirsky theorem\footnote{\url{https://en.wikipedia.org/wiki/Low-rank_approximation}} suggests that the magnitude of the residual
$\mR$ is dramatically reduced, as $\norm{\mR}_F = \sqrt{\sum_{i=r+1}^{\min(m,n)} \sigma_i^2}$, compared to the original magnitude $\norm{\hat{\mW}}_F = \sqrt{\sum_{i=1}^{\min(m,n)} \sigma_i^2}$, where $\sigma_i$ is the $i$-th singular value of $\hat{\mW}$. Furthermore, empirical observations reveal that $\mR$ exhibits fewer outliers with a substantially compressed value range compared to $\hat{\mW}$, as shown in \<PIC>(d)(e). In practice, we can further reduce quantization errors by iteratively updating the low-rank branch through decomposing $\mW - Q(\mR)$ and adjusting $\mR$ accordingly for several iterations, and then picking the result with the smallest error.


Introduction
Transformer-based models have not just advanced, but fundamentally reshaped how we approach both vision and language processing, merging these domains in shared sequential representation spaces. Indeed, most recent multi-modal models including DALL-E \citep{ramesh2022hierarchical},  LLaVA \citep{liu2024visual} and Chameleon \citep{team2024chameleon} operate over joint tokenized representations of images and language, where models decompose images into ``visual languages'': linearized discrete patches or tokens analogous to words in a sentence. This process, shown in \<PIC>, enables seamless integration of images into transformer architectures and allows models to solve multimodal tasks, ranging from image generation and image captioning to visual question answering and translation.
Despite the success of such shared-structure models, current research lacks an in-depth understanding of whether the internal structure of visual tokens mirrors the principles governing natural languages. Specifically, the question arises: do languages formed of visual tokens follow the same statistical patterns, such as frequency distributions, grammatical rules, or semantic dependencies, that human languages exhibit? Investigating such statistical behavior of discrete visual tokens extends beyond theoretical curiosity; it has broad implications for practical machine learning applications. While in linguistic theory, phenomena like Zipf’s law and entropy shape natural languages’ structure and shape the design of machine learning algorithms, no such rules exist for visual languages. Such rules, if they exist, have the potential to motivate creating modality-specific models and procedures to capture the unique statistical properties of the underlying visual data.
In pursuit of such rules, in this paper we inspect the equivalence of visual and natural languages through an empirical analysis of token distributions, segmentation granularity, and syntactic and semantic structures. We start by investigating the frequency statistics of visual words and compare them to natural languages. Our analysis reveals that although visual languages can follow power-law (Zipfian) distributions, they use more tokens more uniformly. This leads to languages with greater per-token entropy and lower compression ratios, and implies that vision models may require more attention heads, larger embeddings, and longer training times with more diverse data compared to natural language models. Noting in these experiments that visual languages have coarser granularity than patches, we demonstrate through correlation analysis that visual tokens operate at an intermediate level of granularity, and typically represent object parts rather than whole objects or sub-parts in images. Correspondingly, we show visual tokens are less effective at representing fine-grained details or whole-object structures. Following this line of reasoning, we explore if tokens have composable structure, and using parse trees generated by Compound Probabilistic Context-Free Grammars (C-PCFG), we show visual languages have grammatical structures that are more fragmented, with grammars trained on them exhibiting higher perplexity compared to natural languages. We confirm these observations by building a co-occurrence based embedding space, and evaluating the topological alignment between natural and visual languages. In this, we find visual languages align more with natural languages than with other visual languages, but less so than natural languages align with each other.
Together, we aim to show through these experiments show that while visual languages have striking similarities to natural languages, there are also notable and fundamental differences, motivating a unique modality-specific approaches to vision-language learning.
re Visual Languages Structured Like Natural Languages?
In \autoref{sec:entropy} we showed that visual languages are not very compressible using Huffman encodings, suggesting that visual languages may not have hierarchical structures similar to those of natural languages.
To inquire further into this question, we test whether Context-free Grammars~\citep{chomsky1959algebraic} can approximate the structure of visual languages as well as they can natural languages by fitting grammars to each modality using unsupervised grammar induction techniques.
Particularly, we use Compound Probabilistic Context-Free Grammars (C-PCFG)~\citep{kim-etal-2019-compound} as the grammar formalism for our experiments.
C-PCFGs are a type of neural PCFG, where grammar production rules are modeled as compound probability distributions~\citep{robbins1956empirical} -- every production depends on both the set of symbols in the grammar as well as a global latent variable $z$.
This formulation, trained with variational methods, allows for global sentence information to flow through all parsing decisions in a sentence while remaining compatible with efficient inference methods which standard PCFGs enjoy \citep{baker1979trainable}.
For more details on C-PCFGs see Appendix \ref{app:cpcfgs}.
C-PCFG memory costs are cubic on sentence length, leading us to use the \verb|compvis-vq-f8-64| tokenizer for visual grammars, which provides a tractable 32 tokens per image.
For each dataset, we train grammars over five seeds for 15 epochs and select the seed with the lowest test set perplexity for analysis.
We test our pipeline by evaluating parsers learned on English COCO captions (COCO-EN) against silver-label parse trees extracted with Benepar~\citep{kitaev-klein-2018-constituency}, attaining an F1 score of 49 on the best seed, which is comparable to prior work~\citep{zhao-titov-2020-visually}.
We report test set statistics over learned grammars, such as final parse tree perplexity (PPL) and percentage reduction in perplexity (PPL-R) from random initialization to convergence.
The mean branching factor~\citep{li-etal-2024-evaluating} (MBF) measures on average whether generated parse trees tend to branch right or left.
This is achieved by averaging the proportion of leaves between the right and left branches of nodes $n$ across parse trees $t$ in the dataset:
Here, CR and CL represent the counts of leaves in the right and left branches of a node, respectively.
To get a better sense of parse tree topology, we also measure the ratio between tree height (the length of the longest path in the tree) and the minimum possible height for the tree:
Where H$(t)$ and L$(t)$ are the height and number of tokens in the input sequence, respectively.
Codebook utilization (CU) measures the percentage of non-terminal labels utilized within generated parse trees.
We present these statistics in \autoref{tab:grammar_stats}, as well as normalized non-terminal node frequencies for parse trees generated by each grammar in \auto<PIC>, with some example parse trees in \auto<PIC>.
Although both modalities experience a great reduction in perplexity compared to random initialization, textual grammars (COCO-EN and COCO-DE) generally exhibit greater reductions in perplexities than visual grammars, corroborating findings from \autoref{sec:entropy} which suggest that visual tokens are not as compressible as textual tokens.
Although visual grammars converge to PPL values an order of magnitude greater than the textual grammars, we observed that their PPL values at the start of training are proportionally higher, likely due to the generally longer visual sentence length (32 tokens in these experiments).
All other measures are generally comparable across modalities -- both modalities show similar proclivities towards right-branching trees (MBF), although visual grammars are somewhat more balanced.
Both modalities present similar tree heights (FR), with the non-terminal label codebooks being largely utilized.
The notable exception to these trends is the grammar trained on XM3600 tokens.
XM3600 contains a significantly lower number of training examples (one order of magnitude less than SPIN, and two orders less than all other datasets), which may have resulted in a degenerate grammar being learned.
These results suggest that the structure of visual languages may not be as well approximated by context-free grammars as natural languages are.
This raises the question of whether they may be better fit by other grammatical formalisms, such as mildly context-sensitive grammars~\citep{yang-etal-2023-unsupervised} which allow for dependencies to cross between token spans.