+
+ + Computation and Language 86 +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ + ☆ VisionZip: Longer is Better but Not Necessary in Vision Language Models +
+
+
+
+
+
+
+
+ Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, Jiaya Jia
+
+
+ Recent advancements in vision-language models have enhanced performance by
+increasing the length of visual tokens, making them much longer than text
+tokens and significantly raising computational costs. However, we observe that
+the visual tokens generated by popular vision encoders, such as CLIP and
+SigLIP, contain significant redundancy. To address this, we introduce
+VisionZip, a simple yet effective method that selects a set of informative
+tokens for input to the language model, reducing visual token redundancy and
+improving efficiency while maintaining model performance. The proposed
+VisionZip can be widely applied to image and video understanding tasks and is
+well-suited for multi-turn dialogues in real-world scenarios, where previous
+methods tend to underperform. Experimental results show that VisionZip
+outperforms the previous state-of-the-art method by at least 5% performance
+gains across nearly all settings. Moreover, our method significantly enhances
+model inference speed, improving the prefilling time by 8x and enabling the
+LLaVA-Next 13B model to infer faster than the LLaVA-Next 7B model while
+achieving better results. Furthermore, we analyze the causes of this redundancy
+and encourage the community to focus on extracting better visual features
+rather than merely increasing token length. Our code is available at
+https://github.com/dvlab-research/VisionZip .
+
+
+
+ comment: 2 columns, 28 pages, 15 figures, 18 tables
+
+
+
+ + ☆ Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction +
+
+
+
+
+
+
+
+ Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, Caiming Xiong
+
+
+ Graphical User Interfaces (GUIs) are critical to human-computer interaction,
+yet automating GUI tasks remains challenging due to the complexity and
+variability of visual environments. Existing approaches often rely on textual
+representations of GUIs, which introduce limitations in generalization,
+efficiency, and scalability. In this paper, we introduce Aguvis, a unified pure
+vision-based framework for autonomous GUI agents that operates across various
+platforms. Our approach leverages image-based observations, and grounding
+instructions in natural language to visual elements, and employs a consistent
+action space to ensure cross-platform generalization. To address the
+limitations of previous work, we integrate explicit planning and reasoning
+within the model, enhancing its ability to autonomously navigate and interact
+with complex digital environments. We construct a large-scale dataset of GUI
+agent trajectories, incorporating multimodal reasoning and grounding, and
+employ a two-stage training pipeline that first focuses on general GUI
+grounding, followed by planning and reasoning. Through comprehensive
+experiments, we demonstrate that Aguvis surpasses previous state-of-the-art
+methods in both offline and real-world online scenarios, achieving, to our
+knowledge, the first fully autonomous pure vision GUI agent capable of
+performing tasks independently without collaboration with external
+closed-source models. We open-sourced all datasets, models, and training
+recipes to facilitate future research at https://aguvis-project.github.io/.
+
+
+
+ comment: https://aguvis-project.github.io/
+
+
+
+ + ☆ p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decay +
+ +
+ Despite the remarkable performance of multimodal large language models
+(MLLMs) across diverse tasks, the substantial training and inference costs
+impede their advancement. The majority of computation stems from the
+overwhelming volume of vision tokens processed by the transformer decoder. In
+this paper, we propose to build efficient MLLMs by leveraging the
+Mixture-of-Depths (MoD) mechanism, where each transformer decoder layer selects
+essential vision tokens to process while skipping redundant ones. However,
+integrating MoD into MLLMs is non-trivial. To address the challenges of
+training and inference stability as well as limited training data, we adapt the
+MoD module with two novel designs: tanh-gated weight normalization (TanhNorm)
+and symmetric token reweighting (STRing). Moreover, we observe that vision
+tokens exhibit higher redundancy in deeper layer and thus design a progressive
+ratio decay (PRD) strategy, which gradually reduces the token retention ratio
+layer by layer, employing a shifted cosine schedule. This crucial design fully
+unleashes the potential of MoD, significantly boosting the efficiency and
+performance of our models. To validate the effectiveness of our approach, we
+conduct extensive experiments with two baseline models across 14 benchmarks.
+Our model, p-MoD, matches or even surpasses the performance of the baseline
+models, with only 55.6% TFLOPs and 53.8% KV cache storage during inference, and
+77.7% GPU hours during training.
+
+
+
+ comment: Technical Report; Code released at https://github.com/MCG-NJU/p-MoD
+
+
+
+ + ☆ Moto: Latent Motion Token as the Bridging Language for Robot + Manipulation +
+ +
+ Recent developments in Large Language Models pre-trained on extensive corpora
+have shown significant success in various natural language processing tasks
+with minimal fine-tuning. This success offers new promise for robotics, which
+has long been constrained by the high cost of action-labeled data. We ask:
+given the abundant video data containing interaction-related knowledge
+available as a rich "corpus", can a similar generative pre-training approach be
+effectively applied to enhance robot learning? The key challenge is to identify
+an effective representation for autoregressive pre-training that benefits robot
+manipulation tasks. Inspired by the way humans learn new skills through
+observing dynamic environments, we propose that effective robotic learning
+should emphasize motion-related knowledge, which is closely tied to low-level
+actions and is hardware-agnostic, facilitating the transfer of learned motions
+to actual robot actions. To this end, we introduce Moto, which converts video
+content into latent Motion Token sequences by a Latent Motion Tokenizer,
+learning a bridging "language" of motion from videos in an unsupervised manner.
+We pre-train Moto-GPT through motion token autoregression, enabling it to
+capture diverse visual motion knowledge. After pre-training, Moto-GPT
+demonstrates the promising ability to produce semantically interpretable motion
+tokens, predict plausible motion trajectories, and assess trajectory
+rationality through output likelihood. To transfer learned motion priors to
+real robot actions, we implement a co-fine-tuning strategy that seamlessly
+bridges latent motion token prediction and real robot control. Extensive
+experiments show that the fine-tuned Moto-GPT exhibits superior robustness and
+efficiency on robot manipulation benchmarks, underscoring its effectiveness in
+transferring knowledge from video data to downstream visual manipulation tasks.
+
+
+
+ comment: Project released at: https://chenyi99.github.io/moto/
+
+
+
+ + ☆ CA-SSLR: Condition-Aware Self-Supervised Learning Representation for + Generalized Speech Processing NeurIPS + 2024 +
+
+
+
+
+
+
+
+ Yen-Ju Lu, Jing Liu, Thomas Thebaud, Laureano Moro-Velazquez, Ariya Rastrow, Najim Dehak, Jesus Villalba
+
+
+ We introduce Condition-Aware Self-Supervised Learning Representation
+(CA-SSLR), a generalist conditioning model broadly applicable to various
+speech-processing tasks. Compared to standard fine-tuning methods that optimize
+for downstream models, CA-SSLR integrates language and speaker embeddings from
+earlier layers, making the SSL model aware of the current language and speaker
+context. This approach reduces the reliance on input audio features while
+preserving the integrity of the base SSLR. CA-SSLR improves the model's
+capabilities and demonstrates its generality on unseen tasks with minimal
+task-specific tuning. Our method employs linear modulation to dynamically
+adjust internal representations, enabling fine-grained adaptability without
+significantly altering the original model behavior. Experiments show that
+CA-SSLR reduces the number of trainable parameters, mitigates overfitting, and
+excels in under-resourced and unseen tasks. Specifically, CA-SSLR achieves a
+10% relative reduction in LID errors, a 37% improvement in ASR CER on the
+ML-SUPERB benchmark, and a 27% decrease in SV EER on VoxCeleb-1, demonstrating
+its effectiveness.
+
+
+
+ comment: 38th Conference on Neural Information Processing Systems (NeurIPS
+ 2024)
+
+
+
+ + ☆ Establishing Task Scaling Laws via Compute-Efficient Model Ladders +
+
+
+
+
+
+
+
+ Akshita Bhagia, Jiacheng Liu, Alexander Wettig, David Heineman, Oyvind Tafjord, Ananya Harsh Jha, Luca Soldaini, Noah A. Smith, Dirk Groeneveld, Pang Wei Koh, Jesse Dodge, Hannaneh Hajishirzi
+
+
+ We develop task scaling laws and model ladders to predict the individual task
+performance of pretrained language models (LMs) in the overtrained setting.
+Standard power laws for language modeling loss cannot accurately model task
+performance. Therefore, we leverage a two-step prediction approach: first use
+model and data size to predict a task-specific loss, and then use this task
+loss to predict task performance. We train a set of small-scale "ladder"
+models, collect data points to fit the parameterized functions of the two
+prediction steps, and make predictions for two target models: a 7B model
+trained to 4T tokens and a 13B model trained to 5T tokens. Training the ladder
+models only costs 1% of the compute used for the target models. On four
+multiple-choice tasks written in ranked classification format, we can predict
+the accuracy of both target models within 2 points of absolute error. We have
+higher prediction error on four other tasks (average absolute error 6.9) and
+find that these are often tasks with higher variance in task metrics. We also
+find that using less compute to train fewer ladder models tends to deteriorate
+predictions. Finally, we empirically show that our design choices and the
+two-step approach lead to superior performance in establishing scaling laws.
+
+
+
+
+ + ☆ BhashaVerse : Translation Ecosystem for Indian Subcontinent Languages +
+ +
+ This paper focuses on developing translation models and related applications
+for 36 Indian languages, including Assamese, Awadhi, Bengali, Bhojpuri, Braj,
+Bodo, Dogri, English, Konkani, Gondi, Gujarati, Hindi, Hinglish, Ho, Kannada,
+Kangri, Kashmiri (Arabic and Devanagari), Khasi, Mizo, Magahi, Maithili,
+Malayalam, Marathi, Manipuri (Bengali and Meitei), Nepali, Oriya, Punjabi,
+Sanskrit, Santali, Sinhala, Sindhi (Arabic and Devanagari), Tamil, Tulu,
+Telugu, and Urdu. Achieving this requires parallel and other types of corpora
+for all 36 * 36 language pairs, addressing challenges like script variations,
+phonetic differences, and syntactic diversity. For instance, languages like
+Kashmiri and Sindhi, which use multiple scripts, demand script normalization
+for alignment, while low-resource languages such as Khasi and Santali require
+synthetic data augmentation to ensure sufficient coverage and quality.
+ To address these challenges, this work proposes strategies for corpus
+creation by leveraging existing resources, developing parallel datasets,
+generating domain-specific corpora, and utilizing synthetic data techniques.
+Additionally, it evaluates machine translation across various dimensions,
+including standard and discourse-level translation, domain-specific
+translation, reference-based and reference-free evaluation, error analysis, and
+automatic post-editing. By integrating these elements, the study establishes a
+comprehensive framework to improve machine translation quality and enable
+better cross-lingual communication in India's linguistically diverse ecosystem.
+
+
+
+
+ + ☆ Retrieval-Augmented Machine Translation with Unstructured Knowledge +
+ +
+ Retrieval-augmented generation (RAG) introduces additional information to
+enhance large language models (LLMs). In machine translation (MT), previous
+work typically retrieves in-context examples from paired MT corpora, or
+domain-specific knowledge from knowledge graphs, to enhance models' MT ability.
+However, a large amount of world knowledge is organized in unstructured
+documents, and might not be fully paired across different languages. In this
+paper, we study retrieval-augmented MT using unstructured documents.
+Specifically, we build RAGtrans, the first benchmark to train and evaluate
+LLMs' retrieval-augmented MT ability. RAGtrans contains 79K MT samples
+collected via GPT-4o and human translators. Besides, documents from different
+languages are also provided to supply the knowledge to these samples. Based on
+RAGtrans, we further propose a multi-task training method to teach LLMs how to
+use information from multilingual documents during their translation. The
+method uses existing multilingual corpora to create auxiliary training
+objectives without additional labeling requirements. Extensive experiments show
+that the method improves LLMs by 1.58-3.09 BLEU and 1.00-2.03 COMET scores.
+
+
+
+
+ + ☆ The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for + Open-Ended Text Generation ICLR +
+ +
+ This paper introduces the counter-intuitive generalization results of
+overfitting pre-trained large language models (LLMs) on very small datasets. In
+the setting of open-ended text generation, it is well-documented that LLMs tend
+to generate repetitive and dull sequences, a phenomenon that is especially
+apparent when generating using greedy decoding. This issue persists even with
+state-of-the-art LLMs containing billions of parameters, trained via next-token
+prediction on large datasets. We find that by further fine-tuning these models
+to achieve a near-zero training loss on a small set of samples -- a process we
+refer to as hyperfitting -- the long-sequence generative capabilities are
+greatly enhanced. Greedy decoding with these Hyperfitted models even outperform
+Top-P sampling over long-sequences, both in terms of diversity and human
+preferences. This phenomenon extends to LLMs of various sizes, different
+domains, and even autoregressive image generation. We further find this
+phenomena to be distinctly different from that of Grokking and double descent.
+Surprisingly, our experiments indicate that hyperfitted models rarely fall into
+repeating sequences they were trained on, and even explicitly blocking these
+sequences results in high-quality output. All hyperfitted models produce
+extremely low-entropy predictions, often allocating nearly all probability to a
+single token.
+
+
+
+ comment: Under review at ICLR
+
+
+
+ + ☆ Densing Law of LLMs +
+ +
+ Large Language Models (LLMs) have emerged as a milestone in artificial
+intelligence, and their performance can improve as the model size increases.
+However, this scaling brings great challenges to training and inference
+efficiency, particularly for deploying LLMs in resource-constrained
+environments, and the scaling trend is becoming increasingly unsustainable.
+This paper introduces the concept of ``\textit{capacity density}'' as a new
+metric to evaluate the quality of the LLMs across different scales and
+describes the trend of LLMs in terms of both effectiveness and efficiency. To
+calculate the capacity density of a given target LLM, we first introduce a set
+of reference models and develop a scaling law to predict the downstream
+performance of these reference models based on their parameter sizes. We then
+define the \textit{effective parameter size} of the target LLM as the parameter
+size required by a reference model to achieve equivalent performance, and
+formalize the capacity density as the ratio of the effective parameter size to
+the actual parameter size of the target LLM. Capacity density provides a
+unified framework for assessing both model effectiveness and efficiency. Our
+further analysis of recent open-source base LLMs reveals an empirical law (the
+densing law)that the capacity density of LLMs grows exponentially over time.
+More specifically, using some widely used benchmarks for evaluation, the
+capacity density of LLMs doubles approximately every three months. The law
+provides new perspectives to guide future LLM development, emphasizing the
+importance of improving capacity density to achieve optimal results with
+minimal computational overhead.
+
+
+
+
+ + ☆ ALMA: Alignment with Minimal Annotation +
+
+
+
+
+
+
+
+ Michihiro Yasunaga, Leonid Shamis, Chunting Zhou, Andrew Cohen, Jason Weston, Luke Zettlemoyer, Marjan Ghazvininejad
+
+
+ Recent approaches to large language model (LLM) alignment typically require
+millions of human annotations or rely on external aligned models for synthetic
+data generation. This paper introduces ALMA: Alignment with Minimal Annotation,
+demonstrating that effective alignment can be achieved using only 9,000 labeled
+examples -- less than 1% of conventional approaches. ALMA generates large
+amounts of high-quality synthetic alignment data through new techniques:
+diverse prompt synthesis via few-shot learning, diverse response generation
+with multiple model checkpoints, and judge (reward model) enhancement through
+score aggregation and self-distillation. Using only a pretrained Llama3 base
+model, 5,000 SFT examples, and 4,000 judge annotations, ALMA achieves
+performance close to Llama3-Instruct across diverse alignment benchmarks (e.g.,
+0.1% difference on AlpacaEval 2.0 score). These results are achieved with a
+multi-round, self-bootstrapped data synthesis and training recipe that
+continues to improve for 10 rounds, surpassing the typical 3-round ceiling of
+previous methods. These results suggest that base models already possess
+sufficient knowledge for effective alignment, and that synthetic data
+generation methods can expose it.
+
+
+
+
+ + ☆ Evolutionary Pre-Prompt Optimization for Mathematical Reasoning +
+ +
+ Recent advancements have highlighted that large language models (LLMs), when
+given a small set of task-specific examples, demonstrate remarkable
+proficiency, a capability that extends to complex reasoning tasks. In
+particular, the combination of few-shot learning with the chain-of-thought
+(CoT) approach has been pivotal in steering models towards more logically
+consistent conclusions. This paper explores the optimization of example
+selection for designing effective CoT pre-prompts and shows that the choice of
+the optimization algorithm, typically in favor of comparison-based methods such
+as evolutionary computation, significantly enhances efficacy and feasibility.
+Specifically, thanks to a limited exploitative and overfitted optimization,
+Evolutionary Pre-Prompt Optimization (EPPO) brings an improvement over the
+naive few-shot approach exceeding 10 absolute points in exact match scores on
+benchmark datasets such as GSM8k and MathQA. These gains are consistent across
+various contexts and are further amplified when integrated with
+self-consistency (SC)
+
+
+
+
+ + ☆ Arabic Stable LM: Adapting Stable LM 2 1.6B to Arabic +
+
+
+
+
+
+
+
+ Zaid Alyafeai, Michael Pieler, Hannah Teufel, Jonathan Tow, Marco Bellagente, Duy Phung, Nikhil Pinnaparaju, Reshinth Adithyan, Paulo Rocha, Maksym Zhuravinskyi, Carlos Riquelme
+
+
+ Large Language Models (LLMs) have shown impressive results in multiple
+domains of natural language processing (NLP) but are mainly focused on the
+English language. Recently, more LLMs have incorporated a larger proportion of
+multilingual text to represent low-resource languages. In Arabic NLP, several
+Arabic-centric LLMs have shown remarkable results on multiple benchmarks in the
+past two years. However, most Arabic LLMs have more than 7 billion parameters,
+which increases their hardware requirements and inference latency, when
+compared to smaller LLMs. This paper introduces Arabic Stable LM 1.6B in a base
+and chat version as a small but powerful Arabic-centric LLM. Our Arabic Stable
+LM 1.6B chat model achieves impressive results on several benchmarks beating
+multiple models with up to 8x the parameters. In addition, we show the benefit
+of mixing in synthetic instruction tuning data by augmenting our fine-tuning
+data with a large synthetic dialogue dataset.
+
+
+
+
+ + ☆ Representation Purification for End-to-End Speech Translation COLING 2025 +
+ +
+ Speech-to-text translation (ST) is a cross-modal task that involves
+converting spoken language into text in a different language. Previous research
+primarily focused on enhancing speech translation by facilitating knowledge
+transfer from machine translation, exploring various methods to bridge the gap
+between speech and text modalities. Despite substantial progress made, factors
+in speech that are not relevant to translation content, such as timbre and
+rhythm, often limit the efficiency of knowledge transfer. In this paper, we
+conceptualize speech representation as a combination of content-agnostic and
+content-relevant factors. We examine the impact of content-agnostic factors on
+translation performance through preliminary experiments and observe a
+significant performance deterioration when content-agnostic perturbations are
+introduced to speech signals. To address this issue, we propose a
+\textbf{S}peech \textbf{R}epresentation \textbf{P}urification with
+\textbf{S}upervision \textbf{E}nhancement (SRPSE) framework, which excludes the
+content-agnostic components within speech representations to mitigate their
+negative impact on ST. Experiments on MuST-C and CoVoST-2 datasets demonstrate
+that SRPSE significantly improves translation performance across all
+translation directions in three settings and achieves preeminent performance
+under a \textit{transcript-free} setting.
+
+
+
+ comment: Accepted by COLING 2025
+
+
+
+ + ☆ Aya Expanse: Combining Research Breakthroughs for a New Multilingual + Frontier +
+
+
+
+
+
+
+
+ John Dang, Shivalika Singh, Daniel D'souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, Acyr Locatelli, Hangyu Lin, Dwarak Talupuru, Bharat Venkitesh, David Cairuz, Bowen Yang, Tim Chung, Wei-Yin Ko, Sylvie Shang Shi, Amir Shukayev, Sammie Bae, Aleksandra Piktus, Roman Castagné, Felipe Cruz-Salinas, Eddie Kim, Lucas Crawhall-Stein, Adrien Morisot, Sudip Roy, Phil Blunsom, Ivan Zhang, Aidan Gomez, Nick Frosst, Marzieh Fadaee, Beyza Ermis, Ahmet Üstün, Sara Hooker
+
+
+ We introduce the Aya Expanse model family, a new generation of 8B and 32B
+parameter multilingual language models, aiming to address the critical
+challenge of developing highly performant multilingual models that match or
+surpass the capabilities of monolingual models. By leveraging several years of
+research at Cohere For AI and Cohere, including advancements in data arbitrage,
+multilingual preference training, and model merging, Aya Expanse sets a new
+state-of-the-art in multilingual performance. Our evaluations on the
+Arena-Hard-Auto dataset, translated into 23 languages, demonstrate that Aya
+Expanse 8B and 32B outperform leading open-weight models in their respective
+parameter classes, including Gemma 2, Qwen 2.5, and Llama 3.1, achieving up to
+a 76.6% win-rate. Notably, Aya Expanse 32B outperforms Llama 3.1 70B, a model
+with twice as many parameters, achieving a 54.0% win-rate. In this short
+technical report, we present extended evaluation results for the Aya Expanse
+model family and release their open-weights, together with a new multilingual
+evaluation dataset m-ArenaHard.
+
+
+
+
+ + ☆ CLINICSUM: Utilizing Language Models for Generating Clinical Summaries + from Patient-Doctor Conversations +
+
+
+
+
+
+
+
+ Subash Neupane, Himanshu Tripathi, Shaswata Mitra, Sean Bozorgzad, Sudip Mittal, Shahram Rahimi, Amin Amirlatifi
+
+
+ This paper presents ClinicSum, a novel framework designed to automatically
+generate clinical summaries from patient-doctor conversations. It utilizes a
+two-module architecture: a retrieval-based filtering module that extracts
+Subjective, Objective, Assessment, and Plan (SOAP) information from
+conversation transcripts, and an inference module powered by fine-tuned
+Pre-trained Language Models (PLMs), which leverage the extracted SOAP data to
+generate abstracted clinical summaries. To fine-tune the PLM, we created a
+training dataset of consisting 1,473 conversations-summaries pair by
+consolidating two publicly available datasets, FigShare and MTS-Dialog, with
+ground truth summaries validated by Subject Matter Experts (SMEs). ClinicSum's
+effectiveness is evaluated through both automatic metrics (e.g., ROUGE,
+BERTScore) and expert human assessments. Results show that ClinicSum
+outperforms state-of-the-art PLMs, demonstrating superior precision, recall,
+and F-1 scores in automatic evaluations and receiving high preference from SMEs
+in human assessment, making it a robust solution for automated clinical
+summarization.
+
+
+
+ comment: accepted at the the 2024 IEEE International Conference on Big Data
+ workshop Workshop on Big Data and AI for Healthcare
+
+
+
+ + ☆ A History of Philosophy in Colombia through Topic Modelling +
+ +
+ Data-driven approaches to philosophy have emerged as a valuable tool for
+studying the history of the discipline. However, most studies in this area have
+focused on a limited number of journals from specific regions and subfields. We
+expand the scope of this research by applying dynamic topic modelling
+techniques to explore the history of philosophy in Colombia and Latin America.
+Our study examines the Colombian philosophy journal Ideas y Valores, founded in
+1951 and currently one of the most influential academic philosophy journals in
+the region. By analyzing the evolution of topics across the journal's history,
+we identify various trends and specific dynamics in philosophical discourse
+within the Colombian and Latin American context. Our findings reveal that the
+most prominent topics are value theory (including ethics, political philosophy,
+and aesthetics), epistemology, and the philosophy of science. We also trace the
+evolution of articles focusing on the historical and interpretive aspects of
+philosophical texts, and we note a notable emphasis on German philosophers such
+as Kant, Husserl, and Hegel on various topics throughout the journal's
+lifetime. Additionally, we investigate whether articles with a historical focus
+have decreased over time due to editorial pressures. Our analysis suggests no
+significant decline in such articles. Finally, we propose ideas for extending
+this research to other Latin American journals and suggest improvements for
+natural language processing workflows in non-English languages.
+
+
+
+
+ + ☆ Addressing Hallucinations with RAG and NMISS in Italian Healthcare LLM + Chatbots +
+ +
+ I combine detection and mitigation techniques to addresses hallucinations in
+Large Language Models (LLMs). Mitigation is achieved in a question-answering
+Retrieval-Augmented Generation (RAG) framework while detection is obtained by
+introducing the Negative Missing Information Scoring System (NMISS), which
+accounts for contextual relevance in responses. While RAG mitigates
+hallucinations by grounding answers in external data, NMISS refines the
+evaluation by identifying cases where traditional metrics incorrectly flag
+contextually accurate responses as hallucinations. I use Italian health news
+articles as context to evaluate LLM performance. Results show that Gemma2 and
+GPT-4 outperform the other models, with GPT-4 producing answers closely aligned
+with reference responses. Mid-tier models, such as Llama2, Llama3, and Mistral
+benefit significantly from NMISS, highlighting their ability to provide richer
+contextual information. This combined approach offers new insights into the
+reduction and more accurate assessment of hallucinations in LLMs, with
+applications in real-world healthcare tasks and other domains.
+
+
+
+
+ + ☆ A Context-aware Framework for Translation-mediated Conversations +
+
+
+
+
+
+
+
+ José Pombal, Sweta Agrawal, Patrick Fernandes, Emmanouil Zaranis, André F. T. Martins
+
+
+ Effective communication is fundamental to any interaction, yet challenges
+arise when participants do not share a common language. Automatic translation
+systems offer a powerful solution to bridge language barriers in such
+scenarios, but they introduce errors that can lead to misunderstandings and
+conversation breakdown. A key issue is that current systems fail to incorporate
+the rich contextual information necessary to resolve ambiguities and omitted
+details, resulting in literal, inappropriate, or misaligned translations. In
+this work, we present a framework to improve large language model-based
+translation systems by incorporating contextual information in bilingual
+conversational settings. During training, we leverage context-augmented
+parallel data, which allows the model to generate translations sensitive to
+conversational history. During inference, we perform quality-aware decoding
+with context-aware metrics to select the optimal translation from a pool of
+candidates. We validate both components of our framework on two task-oriented
+domains: customer chat and user-assistant interaction. Across both settings,
+our framework consistently results in better translations than state-of-the-art
+systems like GPT-4o and TowerInstruct, as measured by multiple automatic
+translation quality metrics on several language pairs. We also show that the
+resulting model leverages context in an intended and interpretable way,
+improving consistency between the conveyed message and the generated
+translations.
+
+
+
+
+ + ☆ AL-QASIDA: Analyzing LLM Quality and Accuracy Systematically in + Dialectal Arabic +
+ +
+ Dialectal Arabic (DA) varieties are under-served by language technologies,
+particularly large language models (LLMs). This trend threatens to exacerbate
+existing social inequalities and limits language modeling applications, yet the
+research community lacks operationalized LLM performance measurements in DA. We
+present a method that comprehensively evaluates LLM fidelity, understanding,
+quality, and diglossia in modeling DA. We evaluate nine LLMs in eight DA
+varieties across these four dimensions and provide best practice
+recommendations. Our evaluation suggests that LLMs do not produce DA as well as
+they understand it, but does not suggest deterioration in quality when they do.
+Further analysis suggests that current post-training can degrade DA
+capabilities, that few-shot examples can overcome this and other LLM
+deficiencies, and that otherwise no measurable features of input text correlate
+well with LLM DA performance.
+
+
+
+ comment: Pre-print
+
+
+
+ + ☆ If You Can't Use Them, Recycle Them: Optimizing Merging at Scale + Mitigates Performance Tradeoffs +
+
+
+
+
+
+
+
+ Muhammad Khalifa, Yi-Chern Tan, Arash Ahmadian, Tom Hosking, Honglak Lee, Lu Wang, Ahmet Üstün, Tom Sherborne, Matthias Gallé
+
+
+ Model merging has shown great promise at combining expert models, but the
+benefit of merging is unclear when merging ``generalist'' models trained on
+many tasks. We explore merging in the context of large ($\sim100$B) models, by
+\textit{recycling} checkpoints that exhibit tradeoffs among different tasks.
+Such checkpoints are often created in the process of developing a frontier
+model, and many suboptimal ones are usually discarded. Given a pool of model
+checkpoints obtained from different training runs (e.g., different stages,
+objectives, hyperparameters, and data mixtures), which naturally show tradeoffs
+across different language capabilities (e.g., instruction following vs. code
+generation), we investigate whether merging can recycle such suboptimal models
+into a Pareto-optimal one. Our optimization algorithm tunes the weight of each
+checkpoint in a linear combination, resulting in a Pareto-optimal models that
+outperforms both individual models and merge-based baselines. Further analysis
+shows that good merges tend to include almost all checkpoints with with
+non-zero weights, indicating that even seemingly bad initial checkpoints can
+contribute to good final merges.
+
+
+
+ comment: 13 pages, 9 figures
+
+
+
+ + ☆ Reducing Tool Hallucination via Reliability Alignment +
+
+
+
+
+
+
+
+ Hongshen Xu, Su Zhu, Zihan Wang, Hang Zheng, Da Ma, Ruisheng Cao, Shuai Fan, Lu Chen, Kai Yu
+
+
+ Large Language Models (LLMs) have extended their capabilities beyond language
+generation to interact with external systems through tool calling, offering
+powerful potential for real-world applications. However, the phenomenon of tool
+hallucinations, which occur when models improperly select or misuse tools,
+presents critical challenges that can lead to flawed task execution and
+increased operational costs. This paper investigates the concept of reliable
+tool calling and highlights the necessity of addressing tool hallucinations. We
+systematically categorize tool hallucinations into two main types: tool
+selection hallucination and tool usage hallucination. To mitigate these issues,
+we propose a reliability-focused alignment framework that enhances the model's
+ability to accurately assess tool relevance and usage. By proposing a suite of
+evaluation metrics and evaluating on StableToolBench, we further demonstrate
+the effectiveness of our framework in mitigating tool hallucination and
+improving the overall system reliability of LLM tool calling.
+
+
+
+
+ + ☆ Text Change Detection in Multilingual Documents Using Image Comparison +
+ +
+ Document comparison typically relies on optical character recognition (OCR)
+as its core technology. However, OCR requires the selection of appropriate
+language models for each document and the performance of multilingual or hybrid
+models remains limited. To overcome these challenges, we propose text change
+detection (TCD) using an image comparison model tailored for multilingual
+documents. Unlike OCR-based approaches, our method employs word-level text
+image-to-image comparison to detect changes. Our model generates bidirectional
+change segmentation maps between the source and target documents. To enhance
+performance without requiring explicit text alignment or scaling preprocessing,
+we employ correlations among multi-scale attention features. We also construct
+a benchmark dataset comprising actual printed and scanned word pairs in various
+languages to evaluate our model. We validate our approach using our benchmark
+dataset and public benchmarks Distorted Document Images and the LRDE Document
+Binarization Dataset. We compare our model against state-of-the-art semantic
+segmentation and change detection models, as well as to conventional OCR-based
+models.
+
+
+
+ comment: 15pages, 11figures 6tables, wacv2025 accepted
+
+
+
+ + ☆ GRAF: Graph Retrieval Augmented by Facts for Legal Question Answering +
+
+
+
+
+
+
+
+ Cristian-George Crăciun, Răzvan-Alexandru Smădu, Dumitru-Clementin Cercel, Mihaela-Claudia Cercel
+
+
+ Pre-trained Language Models (PLMs) have shown remarkable performances in
+recent years, setting a new paradigm for NLP research and industry. The legal
+domain has received some attention from the NLP community partly due to its
+textual nature. Some tasks from this domain are represented by
+question-answering (QA) tasks. This work explores the legal domain
+Multiple-Choice QA (MCQA) for a low-resource language. The contribution of this
+work is multi-fold. We first introduce JuRO, the first openly available
+Romanian legal MCQA dataset, comprising three different examinations and a
+number of 10,836 total questions. Along with this dataset, we introduce CROL,
+an organized corpus of laws that has a total of 93 distinct documents with
+their modifications from 763 time spans, that we leveraged in this work for
+Information Retrieval (IR) techniques. Moreover, we are the first to propose
+Law-RoG, a Knowledge Graph (KG) for the Romanian language, and this KG is
+derived from the aforementioned corpus. Lastly, we propose a novel approach for
+MCQA, Graph Retrieval Augmented by Facts (GRAF), which achieves competitive
+results with generally accepted SOTA methods and even exceeds them in most
+settings.
+
+
+
+
+ + ☆ Missing Melodies: AI Music Generation and its "Nearly" Complete Omission + of the Global South +
+ +
+ Recent advances in generative AI have sparked renewed interest and expanded
+possibilities for music generation. However, the performance and versatility of
+these systems across musical genres are heavily influenced by the availability
+of training data. We conducted an extensive analysis of over one million hours
+of audio datasets used in AI music generation research and manually reviewed
+more than 200 papers from eleven prominent AI and music conferences and
+organizations (AAAI, ACM, EUSIPCO, EURASIP, ICASSP, ICML, IJCAI, ISMIR,
+NeurIPS, NIME, SMC) to identify a critical gap in the fair representation and
+inclusion of the musical genres of the Global South in AI research. Our
+findings reveal a stark imbalance: approximately 86% of the total dataset hours
+and over 93% of researchers focus primarily on music from the Global North.
+However, around 40% of these datasets include some form of non-Western music,
+genres from the Global South account for only 14.6% of the data. Furthermore,
+approximately 51% of the papers surveyed concentrate on symbolic music
+generation, a method that often fails to capture the cultural nuances inherent
+in music from regions such as South Asia, the Middle East, and Africa. As AI
+increasingly shapes the creation and dissemination of music, the significant
+underrepresentation of music genres in datasets and research presents a serious
+threat to global musical diversity. We also propose some important steps to
+mitigate these risks and foster a more inclusive future for AI-driven music
+generation.
+
+
+
+ comment: Submitted to CACM, 12 pages, 2 figures
+
+
+
+ + ☆ GEITje 7B Ultra: A Conversational Model for Dutch +
+ +
+ Language models have rapidly evolved, predominantly focusing on English while
+often neglecting extensive pretraining in other languages. This approach has
+required initiatives to adapt powerful, English-centric models to other
+linguistic contexts through finetuning. For Dutch, such a recent endeavour is
+``GEITje'' a model originally derived from the English-based Mistral 7B.
+Building on this fundamental work, the current research extends the
+capabilities of GEITje by supervised finetuning on newly created high-quality
+synthetic conversational datasets, along with an additional preference
+alignment procedure on a synthetic feedback dataset. Both the developed models
+and the created datasets are openly available.
+
+
+
+
+ + ☆ Automated Medical Report Generation for ECG Data: Bridging Medical Text + and Signal Processing with Deep Learning +
+ +
+ Recent advances in deep learning and natural language generation have
+significantly improved image captioning, enabling automated, human-like
+descriptions for visual content. In this work, we apply these captioning
+techniques to generate clinician-like interpretations of ECG data. This study
+leverages existing ECG datasets accompanied by free-text reports authored by
+healthcare professionals (HCPs) as training data. These reports, while often
+inconsistent, provide a valuable foundation for automated learning. We
+introduce an encoder-decoder-based method that uses these reports to train
+models to generate detailed descriptions of ECG episodes. This represents a
+significant advancement in ECG analysis automation, with potential applications
+in zero-shot classification and automated clinical decision support.
+ The model is tested on various datasets, including both 1- and 12-lead ECGs.
+It significantly outperforms the state-of-the-art reference model by Qiu et
+al., achieving a METEOR score of 55.53% compared to 24.51% achieved by the
+reference model. Furthermore, several key design choices are discussed,
+providing a comprehensive overview of current challenges and innovations in
+this domain.
+ The source codes for this research are publicly available in our Git
+repository https://git.zib.de/ableich/ecg-comment-generation-public
+
+
+
+
+ + ☆ Hostility Detection in UK Politics: A Dataset on Online Abuse Targeting + MPs +
+ +
+ Numerous politicians use social media platforms, particularly X, to engage
+with their constituents. This interaction allows constituents to pose questions
+and offer feedback but also exposes politicians to a barrage of hostile
+responses, especially given the anonymity afforded by social media. They are
+typically targeted in relation to their governmental role, but the comments
+also tend to attack their personal identity. This can discredit politicians and
+reduce public trust in the government. It can also incite anger and disrespect,
+leading to offline harm and violence. While numerous models exist for detecting
+hostility in general, they lack the specificity required for political
+contexts. Furthermore, addressing hostility towards politicians demands
+tailored approaches due to the distinct language and issues inherent to each
+country (e.g., Brexit for the UK). To bridge this gap, we construct a dataset
+of 3,320 English tweets spanning a two-year period manually annotated for
+hostility towards UK MPs. Our dataset also captures the targeted identity
+characteristics (race, gender, religion, none) in hostile tweets. We perform
+linguistic and topical analyses to delve into the unique content of the UK
+political data. Finally, we evaluate the performance of pre-trained language
+models and large language models on binary hostility detection and multi-class
+targeted identity type classification tasks. Our study offers valuable data and
+insights for future research on the prevalence and nature of politics-related
+hostility specific to the UK.
+
+
+
+
+ + ☆ M$^{3}$D: A Multimodal, Multilingual and Multitask Dataset for Grounded + Document-level Information Extraction +
+
+
+
+
+
+
+
+ Jiang Liu, Bobo Li, Xinran Yang, Na Yang, Hao Fei, Mingyao Zhang, Fei Li, Donghong Ji
+
+
+ Multimodal information extraction (IE) tasks have attracted increasing
+attention because many studies have shown that multimodal information benefits
+text information extraction. However, existing multimodal IE datasets mainly
+focus on sentence-level image-facilitated IE in English text, and pay little
+attention to video-based multimodal IE and fine-grained visual grounding.
+Therefore, in order to promote the development of multimodal IE, we constructed
+a multimodal multilingual multitask dataset, named M$^{3}$D, which has the
+following features: (1) It contains paired document-level text and video to
+enrich multimodal information; (2) It supports two widely-used languages,
+namely English and Chinese; (3) It includes more multimodal IE tasks such as
+entity recognition, entity chain extraction, relation extraction and visual
+grounding. In addition, our dataset introduces an unexplored theme, i.e.,
+biography, enriching the domains of multimodal IE resources. To establish a
+benchmark for our dataset, we propose an innovative hierarchical multimodal IE
+model. This model effectively leverages and integrates multimodal information
+through a Denoised Feature Fusion Module (DFFM). Furthermore, in non-ideal
+scenarios, modal information is often incomplete. Thus, we designed a Missing
+Modality Construction Module (MMCM) to alleviate the issues caused by missing
+modalities. Our model achieved an average performance of 53.80% and 53.77% on
+four tasks in English and Chinese datasets, respectively, which set a
+reasonable standard for subsequent research. In addition, we conducted more
+analytical experiments to verify the effectiveness of our proposed module. We
+believe that our work can promote the development of the field of multimodal
+IE.
+
+
+
+ comment: 14 pages, 9 figures, 6 tables
+
+
+
+ + ☆ Exploring the Influence of Label Aggregation on Minority Voices: + Implications for Dataset Bias and Model Training +
+ +
+ Resolving disagreement in manual annotation typically consists of removing
+unreliable annotators and using a label aggregation strategy such as majority
+vote or expert opinion to resolve disagreement. These may have the side-effect
+of silencing or under-representing minority but equally valid opinions. In this
+paper, we study the impact of standard label aggregation strategies on minority
+opinion representation in sexism detection. We investigate the quality and
+value of minority annotations, and then examine their effect on the class
+distributions in gold labels, as well as how this affects the behaviour of
+models trained on the resulting datasets. Finally, we discuss the potential
+biases introduced by each method and how they can be amplified by the models.
+
+
+
+
+ + ☆ Marco-LLM: Bridging Languages via Massive Multilingual Training for + Cross-Lingual Enhancement +
+
+
+
+
+
+
+
+ Lingfeng Ming, Bo Zeng, Chenyang Lyu, Tianqi Shi, Yu Zhao, Xue Yang, Yefeng Liu, Yiyu Wang, Linlong Xu, Yangyang Liu, Xiaohu Zhao, Hao Wang, Heng Liu, Hao Zhou, Huifeng Yin, Zifu Shang, Haijun Li, Longyue Wang, Weihua Luo, Kaifu Zhang
+
+
+ Large Language Models (LLMs) have achieved remarkable progress in recent
+years; however, their excellent performance is still largely limited to major
+world languages, primarily English. Many LLMs continue to face challenges with
+multilingual tasks, especially when it comes to low-resource languages. To
+address this issue, we introduced Marco-LLM: Massive multilingual training for
+cross-lingual enhancement LLM. We have collected a substantial amount of
+multilingual data for several low-resource languages and conducted extensive
+continual pre-training using the Qwen2 models. This effort has resulted in a
+multilingual LLM named Marco-LLM. Through comprehensive evaluations on various
+multilingual benchmarks, including MMMLU, AGIEval, Belebele, Flores-200, XCOPA
+and many others, Marco-LLM has demonstrated substantial improvements over
+state-of-the-art LLMs. Furthermore, Marco-LLM achieved substantial enhancements
+in any-to-any machine translation tasks, showing the effectiveness of our
+multilingual LLM. Marco-LLM is a pioneering multilingual LLM designed to not
+only perform exceptionally well in multilingual tasks, including low-resource
+languages, but also maintain strong performance in English and other major
+languages, closing the performance gap between high- and low-resource language
+capabilities. By bridging languages, this effort demonstrates our dedication to
+ensuring LLMs work accurately across various languages.
+
+
+
+
+ + ☆ MTMT: Consolidating Multiple Thinking Modes to Form a Thought Tree for + Strengthening LLM +
+ +
+ Large language models (LLMs) have shown limitations in tasks requiring
+complex logical reasoning and multi-step problem-solving. To address these
+challenges, researchers have employed carefully designed prompts and
+flowcharts, simulating human cognitive processes to enhance LLM performance,
+such as the Chain of Thought approach. In this paper, we introduce MTMT
+(Multi-thinking Modes Tree), a novel method that interacts with LLMs to
+construct a thought tree, simulating various advanced cognitive processes,
+including but not limited to association, counterfactual thinking, task
+decomposition, and comparison. By breaking down the original complex task into
+simpler sub-questions, MTMT facilitates easier problem-solving for LLMs,
+enabling more effective utilization of the latent knowledge within LLMs. We
+evaluate the performance of MTMT under different parameter configurations,
+using GPT-4o mini as the base model. Our results demonstrate that integrating
+multiple modes of thinking significantly enhances the ability of LLMs to handle
+complex tasks.
+
+
+
+
+ + ☆ Demonstration Selection for In-Context Learning via Reinforcement + Learning +
+ +
+ Diversity in demonstration selection is crucial for enhancing model
+generalization, as it enables a broader coverage of structures and concepts.
+However, constructing an appropriate set of demonstrations has remained a focal
+point of research. This paper presents the Relevance-Diversity Enhanced
+Selection (RDES), an innovative approach that leverages reinforcement learning
+to optimize the selection of diverse reference demonstrations for text
+classification tasks using Large Language Models (LLMs), especially in few-shot
+prompting scenarios. RDES employs a Q-learning framework to dynamically
+identify demonstrations that maximize both diversity and relevance to the
+classification objective by calculating a diversity score based on label
+distribution among selected demonstrations. This method ensures a balanced
+representation of reference data, leading to improved classification accuracy.
+Through extensive experiments on four benchmark datasets and involving 12
+closed-source and open-source LLMs, we demonstrate that RDES significantly
+enhances classification accuracy compared to ten established baselines.
+Furthermore, we investigate the incorporation of Chain-of-Thought (CoT)
+reasoning in the reasoning process, which further enhances the model's
+predictive performance. The results underscore the potential of reinforcement
+learning to facilitate adaptive demonstration selection and deepen the
+understanding of classification challenges.
+
+
+
+
+ + ☆ MIND: Effective Incorrect Assignment Detection through a Multi-Modal + Structure-Enhanced Language Model +
+ +
+ The rapid growth of academic publications has exacerbated the issue of author
+name ambiguity in online digital libraries. Despite advances in name
+disambiguation algorithms, cumulative errors continue to undermine the
+reliability of academic systems. It is estimated that over 10% paper-author
+assignments are rectified when constructing the million-scale WhoIsWho
+benchmark. Existing endeavors to detect incorrect assignments are either
+semantic-based or graph-based approaches, which fall short of making full use
+of the rich text attributes of papers and implicit structural features defined
+via the co-occurrence of paper attributes. To this end, this paper introduces a
+structure-enhanced language model that combines key structural features from
+graph-based methods with fine-grained semantic features from rich paper
+attributes to detect incorrect assignments. The proposed model is trained with
+a highly effective multi-modal multi-turn instruction tuning framework, which
+incorporates task-guided instruction tuning, text-attribute modality, and
+structural modality. Experimental results demonstrate that our model
+outperforms previous approaches, achieving top performance on the leaderboard
+of KDD Cup 2024. Our code has been publicly available.
+
+
+
+
+ + ☆ A Survey on Large Language Model-Based Social Agents in Game-Theoretic + Scenarios +
+
+
+
+
+
+
+
+ Xiachong Feng, Longxu Dou, Ella Li, Qinghao Wang, Haochuan Wang, Yu Guo, Chang Ma, Lingpeng Kong
+
+
+ Game-theoretic scenarios have become pivotal in evaluating the social
+intelligence of Large Language Model (LLM)-based social agents. While numerous
+studies have explored these agents in such settings, there is a lack of a
+comprehensive survey summarizing the current progress. To address this gap, we
+systematically review existing research on LLM-based social agents within
+game-theoretic scenarios. Our survey organizes the findings into three core
+components: Game Framework, Social Agent, and Evaluation Protocol. The game
+framework encompasses diverse game scenarios, ranging from choice-focusing to
+communication-focusing games. The social agent part explores agents'
+preferences, beliefs, and reasoning abilities. The evaluation protocol covers
+both game-agnostic and game-specific metrics for assessing agent performance.
+By reflecting on the current research and identifying future research
+directions, this survey provides insights to advance the development and
+evaluation of social agents in game-theoretic scenarios.
+
+
+
+
+ + ☆ MISR: Measuring Instrumental Self-Reasoning in Frontier Models +
+ +
+ We propose a suite of tasks to evaluate the instrumental self-reasoning
+ability of large language model (LLM) agents. Instrumental self-reasoning
+ability could improve adaptability and enable self-modification, but it could
+also pose significant risks, such as enabling deceptive alignment. Prior work
+has only evaluated self-reasoning in non-agentic settings or in limited
+domains. In this paper, we propose evaluations for instrumental self-reasoning
+ability in agentic tasks in a wide range of scenarios, including
+self-modification, knowledge seeking, and opaque self-reasoning. We evaluate
+agents built using state-of-the-art LLMs, including commercial and open source
+systems. We find that instrumental self-reasoning ability emerges only in the
+most capable frontier models and that it is highly context-dependent. No model
+passes the the most difficult versions of our evaluations, hence our evaluation
+can be used to measure increases in instrumental self-reasoning ability in
+future models. We open-source our evaluations at
+https://github.com/kaifronsdal/Self-Reasoning-Evals.
+
+
+
+ comment: 10 pages, 65 page appendix, 5 figures
+
+
+
+ + ☆ Uniform Discretized Integrated Gradients: An effective attribution based + method for explaining large language models +
+ +
+ Integrated Gradients is a well-known technique for explaining deep learning
+models. It calculates feature importance scores by employing a gradient based
+approach computing gradients of the model output with respect to input features
+and accumulating them along a linear path. While this works well for continuous
+features spaces, it may not be the most optimal way to deal with discrete
+spaces like word embeddings. For interpreting LLMs (Large Language Models),
+there exists a need for a non-linear path where intermediate points, whose
+gradients are to be computed, lie close to actual words in the embedding space.
+In this paper, we propose a method called Uniform Discretized Integrated
+Gradients (UDIG) based on a new interpolation strategy where we choose a
+favorable nonlinear path for computing attribution scores suitable for
+predictive language models. We evaluate our method on two types of NLP tasks-
+Sentiment Classification and Question Answering against three metrics viz Log
+odds, Comprehensiveness and Sufficiency. For sentiment classification, we have
+used the SST2, IMDb and Rotten Tomatoes datasets for benchmarking and for
+Question Answering, we have used the fine-tuned BERT model on SQuAD dataset.
+Our approach outperforms the existing methods in almost all the metrics.
+
+
+
+
+ + ☆ AyutthayaAlpha: A Thai-Latin Script Transliteration Transformer +
+ +
+ This study introduces AyutthayaAlpha, an advanced transformer-based machine
+learning model designed for the transliteration of Thai proper names into Latin
+script. Our system achieves state-of-the-art performance with 82.32%
+first-token accuracy and 95.24% first-three-token accuracy, while maintaining a
+low character error rate of 0.0047. The complexity of Thai phonology, including
+tonal features and vowel length distinctions, presents significant challenges
+for accurate transliteration, which we address through a novel two-model
+approach: AyutthayaAlpha-Small, based on the ByT5 architecture, and
+AyutthayaAlpha-VerySmall, a computationally efficient variant that unexpectedly
+outperforms its larger counterpart. Our research combines linguistic rules with
+deep learning, training on a carefully curated dataset of 1.2 million
+Thai-Latin name pairs, augmented through strategic upsampling to 2.7 million
+examples. Extensive evaluations against existing transliteration methods and
+human expert benchmarks demonstrate that AyutthayaAlpha not only achieves
+superior accuracy but also effectively captures personal and cultural
+preferences in name romanization. The system's practical applications extend to
+cross-lingual information retrieval, international data standardization, and
+identity verification systems, with particular relevance for government
+databases, academic institutions, and global business operations. This work
+represents a significant advance in bridging linguistic gaps between Thai and
+Latin scripts, while respecting the cultural and personal dimensions of name
+transliteration.
+
+
+
+
+ + ☆ Automated LaTeX Code Generation from Handwritten Math Expressions Using + Vision Transformer +
+ +
+ Converting mathematical expressions into LaTeX is challenging. In this paper,
+we explore using newer transformer based architectures for addressing the
+problem of converting handwritten/digital mathematical expression images into
+equivalent LaTeX code. We use the current state of the art CNN encoder and RNN
+decoder as a baseline for our experiments. We also investigate improvements to
+CNN-RNN architecture by replacing the CNN encoder with the ResNet50 model. Our
+experiments show that transformer architectures achieve a higher overall
+accuracy and BLEU scores along with lower Levenschtein scores compared to the
+baseline CNN/RNN architecture with room to achieve even better results with
+appropriate fine-tuning of model parameters.
+
+
+
+ comment: 7 pages; 3 figures
+
+
+
+ + ☆ Educational-Psychological Dialogue Robot Based on Multi-Agent + Collaboration +
+ +
+ Intelligent dialogue systems are increasingly used in modern education and
+psychological counseling fields, but most existing systems are limited to a
+single domain, cannot deal with both educational and psychological issues, and
+often lack accuracy and professionalism when dealing with complex issues. To
+address these problems, this paper proposes an intelligent dialog system that
+combines educational and psychological counseling functions. The system
+consists of multiple AI agent, including security detection agent, intent
+identification agent, educational LLM agent, and psychological LLM agent, which
+work in concert to ensure the provision of accurate educational knowledge Q\&A
+and psychological support services. Specifically, the system recognizes
+user-input intentions through an intention classification model and invokes a
+retrieval-enhanced educational grand model and a psychological grand model
+fine-tuned with psychological data in order to provide professional educational
+advice and psychological support.
+
+
+
+
+ + ☆ Beyond the Binary: Capturing Diverse Preferences With Reward + Regularization +
+ +
+ Large language models (LLMs) are increasingly deployed via public-facing
+interfaces to interact with millions of users, each with diverse preferences.
+Despite this, preference tuning of LLMs predominantly relies on reward models
+trained using binary judgments where annotators select the preferred choice out
+of pairs of model outputs. In this work, we argue that this reliance on binary
+choices does not capture the broader, aggregate preferences of the target user
+in real-world tasks. We propose a taxonomy that identifies two dimensions of
+subjectivity where different users disagree on the preferred output-namely, the
+Plurality of Responses to Prompts, where prompts allow for multiple correct
+answers, and the Indistinguishability of Responses, where candidate outputs are
+paraphrases of each other. We show that reward models correlate weakly with
+user preferences in these cases. As a first step to address this issue, we
+introduce a simple yet effective method that augments existing binary
+preference datasets with synthetic preference judgments to estimate potential
+user disagreement. Incorporating these via a margin term as a form of
+regularization during model training yields predictions that better align with
+the aggregate user preferences.
+
+
+
+
+ + ☆ Detecting Redundant Health Survey Questions Using Language-agnostic BERT + Sentence Embedding (LaBSE) +
+ +
+ The goal of this work was to compute the semantic similarity among publicly
+available health survey questions in order to facilitate the standardization of
+survey-based Person-Generated Health Data (PGHD). We compiled various health
+survey questions authored in both English and Korean from the NIH CDE
+Repository, PROMIS, Korean public health agencies, and academic publications.
+Questions were drawn from various health lifelog domains. A randomized question
+pairing scheme was used to generate a Semantic Text Similarity (STS) dataset
+consisting of 1758 question pairs. Similarity scores between each question pair
+were assigned by two human experts. The tagged dataset was then used to build
+three classifiers featuring: Bag-of-Words, SBERT with BERT-based embeddings,
+and SBRET with LaBSE embeddings. The algorithms were evaluated using
+traditional contingency statistics. Among the three algorithms, SBERT-LaBSE
+demonstrated the highest performance in assessing question similarity across
+both languages, achieving an Area Under the Receiver Operating Characteristic
+(ROC) and Precision-Recall Curves of over 0.99. Additionally, it proved
+effective in identifying cross-lingual semantic similarities.The SBERT-LaBSE
+algorithm excelled at aligning semantically equivalent sentences across both
+languages but encountered challenges in capturing subtle nuances and
+maintaining computational efficiency. Future research should focus on testing
+with larger multilingual datasets and on calibrating and normalizing scores
+across the health lifelog domains to improve consistency. This study introduces
+the SBERT-LaBSE algorithm for calculating semantic similarity across two
+languages, showing it outperforms BERT-based models and the Bag of Words
+approach, highlighting its potential to improve semantic interoperability of
+survey-based PGHD across language barriers.
+
+
+
+
+ + ☆ Synergizing LLMs and Knowledge Graphs: A Novel Approach to Software + Repository-Related Question Answering +
+ +
+ Software repositories contain valuable information for gaining insights into
+their development process. However, extracting insights from these repository
+data is time-consuming and requires technical expertise. While software
+engineering chatbots have been developed to facilitate natural language
+interactions with repositories, they struggle with understanding natural
+language and accurately retrieving relevant data. This study aims to improve
+the accuracy of LLM-based chatbots in answering repository-related questions by
+augmenting them with knowledge graphs. We achieve this in a two-step approach;
+(1) constructing a knowledge graph from the repository data and (2) synergizing
+the knowledge graph with LLM to allow for the natural language questions and
+answers. We curated a set of 20 questions with different complexities and
+evaluated our approach on five popular open-source projects. Our approach
+achieved an accuracy of 65%. We further investigated the limitations and
+identified six key issues, with the majority relating to the reasoning
+capability of the LLM. We experimented with a few-shot chain-of-thought
+prompting to determine if it could enhance our approach. This technique
+improved the overall accuracy to 84%. Our findings demonstrate the synergy
+between LLMs and knowledge graphs as a viable solution for making repository
+data accessible to both technical and non-technical stakeholders.
+
+
+
+ comment: Submitted to ACM Transactions on Software Engineering and Methodology
+ for review
+
+
+
+ + ☆ Agent AI with LangGraph: A Modular Framework for Enhancing Machine + Translation Using Large Language Models +
+ +
+ This paper explores the transformative role of Agent AI and LangGraph in
+advancing the automation and effectiveness of machine translation (MT). Agents
+are modular components designed to perform specific tasks, such as translating
+between particular languages, with specializations like TranslateEnAgent,
+TranslateFrenchAgent, and TranslateJpAgent for English, French, and Japanese
+translations, respectively. These agents leverage the powerful semantic
+capabilities of large language models (LLMs), such as GPT-4o, to ensure
+accurate, contextually relevant translations while maintaining modularity,
+scalability, and context retention.
+ LangGraph, a graph-based framework built on LangChain, simplifies the
+creation and management of these agents and their workflows. It supports
+dynamic state management, enabling agents to maintain dialogue context and
+automates complex workflows by linking agents and facilitating their
+collaboration. With flexibility, open-source community support, and seamless
+integration with LLMs, LangGraph empowers agents to deliver high-quality
+translations.
+ Together, Agent AI and LangGraph create a cohesive system where LangGraph
+orchestrates agent interactions, ensuring that user inputs are analyzed,
+routed, and processed efficiently. Experimental results demonstrate the
+potential of this system to enhance multilingual translation accuracy and
+scalability. By highlighting modular design and automated workflows, this paper
+sets the stage for further innovations in intelligent machine translation
+services.
+
+
+
+
+ + ☆ The broader spectrum of in-context learning +
+ +
+ The ability of language models to learn a task from a few examples in context
+has generated substantial interest. Here, we provide a perspective that
+situates this type of supervised few-shot learning within a much broader
+spectrum of meta-learned in-context learning. Indeed, we suggest that any
+distribution of sequences in which context non-trivially decreases loss on
+subsequent predictions can be interpreted as eliciting a kind of in-context
+learning. We suggest that this perspective helps to unify the broad set of
+in-context abilities that language models exhibit $\unicode{x2014}$ such as
+adapting to tasks from instructions or role play, or extrapolating time series.
+This perspective also sheds light on potential roots of in-context learning in
+lower-level processing of linguistic dependencies (e.g. coreference or parallel
+structures). Finally, taking this perspective highlights the importance of
+generalization, which we suggest can be studied along several dimensions: not
+only the ability to learn something novel, but also flexibility in learning
+from different presentations, and in applying what is learned. We discuss
+broader connections to past literature in meta-learning and goal-conditioned
+agents, and other perspectives on learning and adaptation. We close by
+suggesting that research on in-context learning should consider this broader
+spectrum of in-context capabilities and types of generalization.
+
+
+
+
+ + ♻ ☆ The Semantic Hub Hypothesis: Language Models Share Semantic + Representations Across Languages and Modalities +
+ +
+ Modern language models can process inputs across diverse languages and
+modalities. We hypothesize that models acquire this capability through learning
+a shared representation space across heterogeneous data types (e.g., different
+languages and modalities), which places semantically similar inputs near one
+another, even if they are from different modalities/languages. We term this the
+semantic hub hypothesis, following the hub-and-spoke model from neuroscience
+(Patterson et al., 2007) which posits that semantic knowledge in the human
+brain is organized through a transmodal semantic "hub" which integrates
+information from various modality-specific "spokes" regions. We first show that
+model representations for semantically equivalent inputs in different languages
+are similar in the intermediate layers, and that this space can be interpreted
+using the model's dominant pretraining language via the logit lens. This
+tendency extends to other data types, including arithmetic expressions, code,
+and visual/audio inputs. Interventions in the shared representation space in
+one data type also predictably affect model outputs in other data types,
+suggesting that this shared representations space is not simply a vestigial
+byproduct of large-scale training on broad data, but something that is actively
+utilized by the model during input processing.
+
+
+
+
+ + ♻ ☆ SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large + Language Models by Summarizing Training Trajectories of Small Models +
+ +
+ Despite the effectiveness of data selection for large language models (LLMs)
+during pretraining and instruction fine-tuning phases, improving data
+efficiency in supervised fine-tuning (SFT) for specialized domains poses
+significant challenges due to the complexity of fine-tuning data. To bridge
+this gap, we introduce an effective and scalable data selection method for SFT,
+SmallToLarge (S2L), which leverages training trajectories from small models to
+guide the data selection for larger models. We demonstrate through extensive
+experiments that S2L significantly improves data efficiency in SFT for
+mathematical problem-solving, reducing the training data to just 11% of the
+original MathInstruct dataset (Yue et al., 2023) to match full dataset
+performance while outperforming state-of-the-art data selection algorithms by
+an average of 4.7% across 6 in- and out-domain evaluation datasets. Remarkably,
+selecting only 50K data for SFT, S2L achieves a 32.7% accuracy on the most
+challenging MATH (Hendrycks et al., 2021) benchmark, improving Phi-2 (Li et
+al., 2023b) by 16.6%. In clinical text summarization on the MIMIC-III dataset
+(Johnson et al., 2016), S2L again outperforms training on the full dataset
+using only 50% of the data. Notably, S2L can perform data selection using a
+reference model 40x smaller than the target model, proportionally reducing the
+cost of data selection.
+
+
+
+
+ + ♻ ☆ WaveletGPT: Wavelets Meet Large Language Models +
+ +
+ Large Language Models (LLMs) have ushered in a new wave of artificial
+intelligence advancements impacting every scientific field and discipline. They
+are trained on a simple objective: to predict the next token given the previous
+context. We live in a world where most of the data around us, e.g., text,
+audio, and music, has a multi-scale structure associated with it. This paper
+infuses LLMs with traditional signal processing ideas, namely wavelets, during
+pre-training to take advantage of the structure. Without adding \textbf{any
+extra parameters} to a GPT-style LLM architecture, we achieve the same
+pre-training performance almost twice as fast in text, raw audio, and symbolic
+music. This is achieved by imposing a structure on intermediate embeddings.
+When trained for the same number of training steps, we achieve significant
+gains in performance, which is comparable to pre-training a larger neural
+architecture. Our architecture allows every next token prediction access to
+intermediate embeddings at different temporal resolutions in every Transformer
+decoder block. This work will hopefully pave the way for incorporating
+multi-rate signal processing ideas into traditional LLM pre-training. Further,
+we showcase pushing model performance by improving internal structure instead
+of just going after scale.
+
+
+
+ comment: 16 pages, 4 figures
+
+
+
+ + ♻ ☆ CNNSum: Exploring Long-Conext Summarization with Large Language Models + in Chinese Novels +
+ +
+ Large Language Models (LLMs) have been well-researched in many long-context
+tasks. However, due to high annotation costs, high-quality long-context summary
+datasets for training or evaluation are scarce, limiting further research. In
+this work, we introduce CNNSum, a new multi-scale Chinese long-context novel
+summarization benchmark, including four subsets, length covering
+16k\textasciitilde128k, 695 samples in total, the annotations are human-driven.
+We evaluate commercial and open-source models on CNNSum and conduct a detailed
+analysis. Based on the observations, we further conduct fine-tuning exploration
+with short-context summary data. In our study: (1) GPT-4o underperformed, due
+to excessive subjective commentary. (2) Currently, long-context summarization
+mainly relies on memory ability, small LLMs with stable longer context lengths
+are the most cost-effective. Using long data concatenated from short-context
+summaries makes a significant improvement. (3) Prompt templates may cause a
+large performance gap but can be mitigated through fine-tuning. (4) Fine-tuned
+Chat or Instruction versions may harm the Base model and further fine-tuning
+cannot bridge performance gap. (5) while models with RoPE base scaling exhibit
+strong extrapolation potential, their performance may vary significantly when
+combined with other interpolation methods and need careful selection. (6)
+CNNSum provides more reliable and insightful evaluation results than other
+benchmarks. We release CNNSum to advance research in this field.
+
+
+
+
+ + ♻ ☆ Context-Informed Machine Translation of Manga using Multimodal Large + Language Models COLING 2025 +
+ +
+ Due to the significant time and effort required for handcrafting
+translations, most manga never leave the domestic Japanese market. Automatic
+manga translation is a promising potential solution. However, it is a budding
+and underdeveloped field and presents complexities even greater than those
+found in standard translation due to the need to effectively incorporate visual
+elements into the translation process to resolve ambiguities. In this work, we
+investigate to what extent multimodal large language models (LLMs) can provide
+effective manga translation, thereby assisting manga authors and publishers in
+reaching wider audiences. Specifically, we propose a methodology that leverages
+the vision component of multimodal LLMs to improve translation quality and
+evaluate the impact of translation unit size, context length, and propose a
+token efficient approach for manga translation. Moreover, we introduce a new
+evaluation dataset -- the first parallel Japanese-Polish manga translation
+dataset -- as part of a benchmark to be used in future research. Finally, we
+contribute an open-source software suite, enabling others to benchmark LLMs for
+manga translation. Our findings demonstrate that our proposed methods achieve
+state-of-the-art results for Japanese-English translation and set a new
+standard for Japanese-Polish.
+
+
+
+ comment: COLING 2025
+
+
+
+ + ♻ ☆ Unveiling Entity-Level Unlearning for Large Language Models: A + Comprehensive Analysis COLING 2025 +
+
+
+
+
+
+
+
+ Weitao Ma, Xiaocheng Feng, Weihong Zhong, Lei Huang, Yangfan Ye, Xiachong Feng, Bing Qin
+
+
+ Large language model unlearning has garnered increasing attention due to its
+potential to address security and privacy concerns, leading to extensive
+research in the field. However, much of this research has concentrated on
+instance-level unlearning, specifically targeting the removal of predefined
+instances containing sensitive content. This focus has left a significant gap
+in the exploration of full entity-level unlearning, which is critical in
+real-world scenarios such as copyright protection. To this end, we propose a
+novel task of Entity-level unlearning, which aims to erase entity-related
+knowledge from the target model completely. To thoroughly investigate this
+task, we systematically evaluate trending unlearning algorithms, revealing that
+current methods struggle to achieve effective entity-level unlearning. Then, we
+further explore the factors that influence the performance of the unlearning
+algorithms, identifying that knowledge coverage and the size of the forget set
+play pivotal roles. Notably, our analysis also uncovers that entities
+introduced through fine-tuning are more vulnerable to unlearning than
+pre-trained entities. These findings collectively offer valuable insights for
+advancing entity-level unlearning for LLMs.
+
+
+
+ comment: Accepted by COLING 2025
+
+
+
+ + ♻ ☆ CoSy: Evaluating Textual Explanations of Neurons +
+
+
+
+
+
+
+
+ Laura Kopf, Philine Lou Bommer, Anna Hedström, Sebastian Lapuschkin, Marina M. -C. Höhne, Kirill Bykov
+
+
+ A crucial aspect of understanding the complex nature of Deep Neural Networks
+(DNNs) is the ability to explain learned concepts within their latent
+representations. While methods exist to connect neurons to human-understandable
+textual descriptions, evaluating the quality of these explanations is
+challenging due to the lack of a unified quantitative approach. We introduce
+CoSy (Concept Synthesis), a novel, architecture-agnostic framework for
+evaluating textual explanations of latent neurons. Given textual explanations,
+our proposed framework uses a generative model conditioned on textual input to
+create data points representing the explanations. By comparing the neuron's
+response to these generated data points and control data points, we can
+estimate the quality of the explanation. We validate our framework through
+sanity checks and benchmark various neuron description methods for Computer
+Vision tasks, revealing significant differences in quality.
+
+
+
+ comment: 10 pages, 5 figures
+
+
+
+ + ♻ ☆ Finding Transformer Circuits with Edge Pruning NeurIPS 2024 +
+ +
+ The path to interpreting a language model often proceeds via analysis of
+circuits -- sparse computational subgraphs of the model that capture specific
+aspects of its behavior. Recent work has automated the task of discovering
+circuits. Yet, these methods have practical limitations, as they rely either on
+inefficient search algorithms or inaccurate approximations. In this paper, we
+frame automated circuit discovery as an optimization problem and propose *Edge
+Pruning* as an effective and scalable solution. Edge Pruning leverages
+gradient-based pruning techniques, but instead of removing neurons or
+components, it prunes the \emph{edges} between components. Our method finds
+circuits in GPT-2 that use less than half the number of edges compared to
+circuits found by previous methods while being equally faithful to the full
+model predictions on standard circuit-finding tasks. Edge Pruning is efficient
+even with as many as 100K examples, outperforming previous methods in speed and
+producing substantially better circuits. It also perfectly recovers the
+ground-truth circuits in two models compiled with Tracr. Thanks to its
+efficiency, we scale Edge Pruning to CodeLlama-13B, a model over 100x the scale
+that prior methods operate on. We use this setting for a case study comparing
+the mechanisms behind instruction prompting and in-context learning. We find
+two circuits with more than 99.96% sparsity that match the performance of the
+full model and reveal that the mechanisms in the two settings overlap
+substantially. Our case study shows that Edge Pruning is a practical and
+scalable tool for interpretability and sheds light on behaviors that only
+emerge in large models.
+
+
+
+ comment: NeurIPS 2024 (Spotlight)
+
+
+
+ + ♻ ☆ A Complexity-Based Theory of Compositionality +
+ +
+ Compositionality is believed to be fundamental to intelligence. In humans, it
+underlies the structure of thought, language, and higher-level reasoning. In
+AI, compositional representations can enable a powerful form of
+out-of-distribution generalization, in which a model systematically adapts to
+novel combinations of known concepts. However, while we have strong intuitions
+about what compositionality is, there currently exists no formal definition for
+it that is measurable and mathematical. Here, we propose such a definition,
+which we call representational compositionality, that accounts for and extends
+our intuitions about compositionality. The definition is conceptually simple,
+quantitative, grounded in algorithmic information theory, and applicable to any
+representation. Intuitively, representational compositionality states that a
+compositional representation satisfies three properties. First, it must be
+expressive. Second, it must be possible to re-describe the representation as a
+function of discrete symbolic sequences with re-combinable parts, analogous to
+sentences in natural language. Third, the function that relates these symbolic
+sequences to the representation, analogous to semantics in natural language,
+must be simple. Through experiments on both synthetic and real world data, we
+validate our definition of compositionality and show how it unifies disparate
+intuitions from across the literature in both AI and cognitive science. We also
+show that representational compositionality, while theoretically intractable,
+can be readily estimated using standard deep learning tools. Our definition has
+the potential to inspire the design of novel, theoretically-driven models that
+better capture the mechanisms of compositional thought.
+
+
+
+
+ + ♻ ★ Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild NeurIPS 2024 +
+
+
+
+
+
+
+
+ Xinyu Zhao, Guoheng Sun, Ruisi Cai, Yukun Zhou, Pingzhi Li, Peihao Wang, Bowen Tan, Yexiao He, Li Chen, Yi Liang, Beidi Chen, Binhang Yuan, Hongyi Wang, Ang Li, Zhangyang Wang, Tianlong Chen
+
+
+ As Large Language Models (LLMs) excel across tasks and specialized domains,
+scaling LLMs based on existing models has garnered significant attention, which
+faces the challenge of decreasing performance when combining disparate models.
+Various techniques have been proposed for the aggregation of pre-trained LLMs,
+including model merging, Mixture-of-Experts, and stacking. Despite their
+merits, a comprehensive comparison and synergistic application of them to a
+diverse model zoo is yet to be adequately addressed. In light of this research
+gap, this paper introduces Model-GLUE, a holistic LLM scaling guideline. First,
+our work starts with a benchmarking of existing LLM scaling techniques,
+especially selective merging, and variants of mixture. Utilizing the insights
+from the benchmark results, we formulate an optimal strategy for the selection
+and aggregation of a heterogeneous model zoo characterizing different
+architectures and initialization.Our methodology involves the clustering of
+mergeable models and optimal merging strategy selection, and the integration of
+clusters through a model mixture. Finally, evidenced by our experiments on a
+diverse Llama-2-based model zoo, Model-GLUE shows an average performance
+enhancement of 5.61%, achieved without additional training. Codes are available
+at: https://github.com/Model-GLUE/Model-GLUE.
+
+
+
+ comment: 24 pages, 4 figures, accepted to NeurIPS 2024 Datasets and Benchmarks
+ Track
+
+
+
+ + ♻ ☆ SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving + Model Transformation +
+ +
+ LLM inference for popular enterprise use cases, such as summarization, RAG,
+and code-generation, typically observes orders of magnitude longer prompt
+lengths than generation lengths. This characteristic leads to high cost of
+prefill and increased response latency. In this paper, we present SwiftKV, a
+novel model transformation and distillation procedure specifically designed to
+reduce the time and cost of processing prompt tokens while preserving high
+quality of generated tokens. SwiftKV combines three key mechanisms: i)
+SingleInputKV, which prefills later layers' KV cache using a much earlier
+layer's output, allowing prompt tokens to skip much of the model computation,
+ii) AcrossKV, which merges the KV caches of neighboring layers to reduce the
+memory footprint and support larger batch size for higher throughput, and iii)
+a knowledge-preserving distillation procedure that can adapt existing LLMs for
+SwiftKV with minimal accuracy impact and low compute and data requirement. For
+Llama-3.1-8B and 70B, SwiftKV reduces the compute requirement of prefill by 50%
+and the memory requirement of the KV cache by 62.5% while incurring minimum
+quality degradation across a wide range of tasks. In the end-to-end inference
+serving using an optimized vLLM implementation, SwiftKV realizes up to 2x
+higher aggregate throughput and 60% lower time per output token. It can achieve
+a staggering 560 TFlops/GPU of normalized inference throughput, which
+translates to 16K tokens/s for Llama-3.1-70B in 16-bit precision on 4x H100
+GPUs. Our training, inference, and model implementations are open-sourced and
+can be found through
+https://huggingface.co/collections/Snowflake/swiftkv-models-674f7d7474eb789e185d31cb.
+
+
+
+
+ + ♻ ☆ RARE: Retrieval-Augmented Reasoning Enhancement for Large Language + Models +
+ +
+ This work introduces RARE (Retrieval-Augmented Reasoning Enhancement), a
+versatile extension to the mutual reasoning framework (rStar), aimed at
+enhancing reasoning accuracy and factual integrity across large language models
+(LLMs) for complex, knowledge-intensive tasks such as commonsense and medical
+reasoning. RARE incorporates two innovative actions within the Monte Carlo Tree
+Search (MCTS) framework: A6, which generates search queries based on the
+initial problem statement, performs information retrieval using those queries,
+and augments reasoning with the retrieved data to formulate the final answer;
+and A7, which leverages information retrieval specifically for generated
+sub-questions and re-answers these sub-questions with the relevant contextual
+information. Additionally, a Retrieval-Augmented Factuality Scorer is proposed
+to replace the original discriminator, prioritizing reasoning paths that meet
+high standards of factuality. Experimental results with LLaMA 3.1 show that
+RARE enables open-source LLMs to achieve competitive performance with top
+open-source models like GPT-4 and GPT-4o. This research establishes RARE as a
+scalable solution for improving LLMs in domains where logical coherence and
+factual integrity are critical.
+
+
+
+ comment: 24 pages
+
+
+
+ + ♻ ☆ Agent-OM: Leveraging LLM Agents for Ontology Matching +
+ +
+ Ontology matching (OM) enables semantic interoperability between different
+ontologies and resolves their conceptual heterogeneity by aligning related
+entities. OM systems currently have two prevailing design paradigms:
+conventional knowledge-based expert systems and newer machine learning-based
+predictive systems. While large language models (LLMs) and LLM agents have
+revolutionised data engineering and have been applied creatively in many
+domains, their potential for OM remains underexplored. This study introduces a
+novel agent-powered LLM-based design paradigm for OM systems. With
+consideration of several specific challenges in leveraging LLM agents for OM,
+we propose a generic framework, namely Agent-OM (Agent for Ontology Matching),
+consisting of two Siamese agents for retrieval and matching, with a set of
+simple OM tools. Our framework is implemented in a proof-of-concept system.
+Evaluations of three Ontology Alignment Evaluation Initiative (OAEI) tracks
+over state-of-the-art OM systems show that our system can achieve results very
+close to the long-standing best performance on simple OM tasks and can
+significantly improve the performance on complex and few-shot OM tasks.
+
+
+
+ comment: 14 pages, 13 figures, 4 tables
+
+
+
+ + ♻ ☆ Molmo and PixMo: Open Weights and Open Data for State-of-the-Art + Vision-Language Models +
+
+
+
+
+
+
+
+ Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, Aniruddha Kembhavi
+
+
+ Today's most advanced vision-language models (VLMs) remain proprietary. The
+strongest open-weight models rely heavily on synthetic data from proprietary
+VLMs to achieve good performance, effectively distilling these closed VLMs into
+open ones. As a result, the community has been missing foundational knowledge
+about how to build performant VLMs from scratch. We present Molmo, a new family
+of VLMs that are state-of-the-art in their class of openness. Our key
+contribution is a collection of new datasets called PixMo, including a dataset
+of highly detailed image captions for pre-training, a free-form image Q&A
+dataset for fine-tuning, and an innovative 2D pointing dataset, all collected
+without the use of external VLMs. The success of our approach relies on careful
+modeling choices, a well-tuned training pipeline, and, most critically, the
+quality of our newly collected datasets. Our best-in-class 72B model not only
+outperforms others in the class of open weight and data models, but also
+outperforms larger proprietary models including Claude 3.5 Sonnet, and Gemini
+1.5 Pro and Flash, second only to GPT-4o based on both academic benchmarks and
+on a large human evaluation. Our model weights, new datasets, and source code
+are available at https://molmo.allenai.org/blog.
+
+
+
+ comment: Updated with ablations and more technical details
+
+
+
+ + ♻ ☆ Adaptive Circuit Behavior and Generalization in Mechanistic + Interpretability +
+ +
+ Mechanistic interpretability aims to understand the inner workings of large
+neural networks by identifying circuits, or minimal subgraphs within the model
+that implement algorithms responsible for performing specific tasks. These
+circuits are typically discovered and analyzed using a narrowly defined prompt
+format. However, given the abilities of large language models (LLMs) to
+generalize across various prompt formats for the same task, it remains unclear
+how well these circuits generalize. For instance, it is unclear whether the
+models generalization results from reusing the same circuit components, the
+components behaving differently, or the use of entirely different components.
+In this paper, we investigate the generality of the indirect object
+identification (IOI) circuit in GPT-2 small, which is well-studied and believed
+to implement a simple, interpretable algorithm. We evaluate its performance on
+prompt variants that challenge the assumptions of this algorithm. Our findings
+reveal that the circuit generalizes surprisingly well, reusing all of its
+components and mechanisms while only adding additional input edges. Notably,
+the circuit generalizes even to prompt variants where the original algorithm
+should fail; we discover a mechanism that explains this which we term S2
+Hacking. Our findings indicate that circuits within LLMs may be more flexible
+and general than previously recognized, underscoring the importance of studying
+circuit generalization to better understand the broader capabilities of these
+models.
+
+
+
+ comment: 10 pages, 8 figures
+
+
+
+ + ♻ ☆ Lexicalization Is All You Need: Examining the Impact of Lexical + Knowledge in a Compositional QALD System +
+ +
+ In this paper, we examine the impact of lexicalization on Question Answering
+over Linked Data (QALD). It is well known that one of the key challenges in
+interpreting natural language questions with respect to SPARQL lies in bridging
+the lexical gap, that is mapping the words in the query to the correct
+vocabulary elements. We argue in this paper that lexicalization, that is
+explicit knowledge about the potential interpretations of a word with respect
+to the given vocabulary, significantly eases the task and increases the
+performance of QA systems. Towards this goal, we present a compositional QA
+system that can leverage explicit lexical knowledge in a compositional manner
+to infer the meaning of a question in terms of a SPARQL query. We show that
+such a system, given lexical knowledge, has a performance well beyond current
+QA systems, achieving up to a $35.8\%$ increase in the micro $F_1$ score
+compared to the best QA system on QALD-9. This shows the importance and
+potential of including explicit lexical knowledge. In contrast, we show that
+LLMs have limited abilities to exploit lexical knowledge, with only marginal
+improvements compared to a version without lexical knowledge. This shows that
+LLMs have no ability to compositionally interpret a question on the basis of
+the meaning of its parts, a key feature of compositional approaches. Taken
+together, our work shows new avenues for QALD research, emphasizing the
+importance of lexicalization and compositionality.
+
+
+
+ comment: 24th International Conference on Knowledge Engineering and Knowledge
+ Management (EKAW 2024), November 26-28, 2024, Amsterdam, The Netherlands
+
+
+
+ + ♻ ☆ KV Shifting Attention Enhances Language Modeling +
+ +
+ The current large language models are mainly based on decode-only structure
+transformers, which have great in-context learning (ICL) capabilities. It is
+generally believed that the important foundation of its ICL capability is the
+induction heads mechanism, which requires at least two layers attention. In
+order to more efficiently implement the ability of the model's induction, we
+revisit the induction heads mechanism and proposed a KV shifting attention. We
+theoretically prove that the KV shifting attention reducing the model's
+requirements for the depth and width of the induction heads mechanism. Our
+experimental results demonstrate that KV shifting attention is beneficial to
+learning induction heads and language modeling, which lead to better
+performance or faster convergence from toy models to the pre-training models
+with more than 10 B parameters.
+
+
+
+ comment: 22 pages
+
+
+
+ + ♻ ☆ Words in Motion: Extracting Interpretable Control Vectors for Motion + Transformers +
+ +
+ Transformer-based models generate hidden states that are difficult to
+interpret. In this work, we aim to interpret these hidden states and control
+them at inference, with a focus on motion forecasting. We use linear probes to
+measure neural collapse towards interpretable motion features in hidden states.
+High probing accuracy implies meaningful directions and distances between
+hidden states of opposing features, which we use to fit interpretable control
+vectors for activation steering at inference. To optimize our control vectors,
+we use sparse autoencoders with fully-connected, convolutional, MLPMixer layers
+and various activation functions. Notably, we show that enforcing sparsity in
+hidden states leads to a more linear relationship between control vector
+temperatures and forecasts. Our approach enables mechanistic interpretability
+and zero-shot generalization to unseen dataset characteristics with negligible
+computational overhead. Our implementation is available at
+https://github.com/kit-mrt/future-motion
+
+
+
+ comment: Add autoencoders with convolutional, MLPMixer layers, and JumpReLU
+ activations
+
+
+
+ + ♻ ☆ SCAR: Sparse Conditioned Autoencoders for Concept Detection and Steering + in LLMs NeurIPS 2024 +
+
+
+
+
+
+
+
+ Ruben Härle, Felix Friedrich, Manuel Brack, Björn Deiseroth, Patrick Schramowski, Kristian Kersting
+
+
+ Large Language Models (LLMs) have demonstrated remarkable capabilities in
+generating human-like text, but their output may not be aligned with the user
+or even produce harmful content. This paper presents a novel approach to detect
+and steer concepts such as toxicity before generation. We introduce the Sparse
+Conditioned Autoencoder (SCAR), a single trained module that extends the
+otherwise untouched LLM. SCAR ensures full steerability, towards and away from
+concepts (e.g., toxic content), without compromising the quality of the model's
+text generation on standard evaluation benchmarks. We demonstrate the effective
+application of our approach through a variety of concepts, including toxicity,
+safety, and writing style alignment. As such, this work establishes a robust
+framework for controlling LLM generations, ensuring their ethical and safe
+deployment in real-world applications.
+
+
+
+ comment: Accepted at Socially Responsible Language Modelling Research (SoLaR)
+ Workshop at NeurIPS 2024
+
+
+
+ + ♻ ☆ Hybrid-SQuAD: Hybrid Scholarly Question Answering Dataset +
+ +
+ Existing Scholarly Question Answering (QA) methods typically target
+homogeneous data sources, relying solely on either text or Knowledge Graphs
+(KGs). However, scholarly information often spans heterogeneous sources,
+necessitating the development of QA systems that integrate information from
+multiple heterogeneous data sources. To address this challenge, we introduce
+Hybrid-SQuAD (Hybrid Scholarly Question Answering Dataset), a novel large-scale
+QA dataset designed to facilitate answering questions incorporating both text
+and KG facts. The dataset consists of 10.5K question-answer pairs generated by
+a large language model, leveraging the KGs DBLP and SemOpenAlex alongside
+corresponding text from Wikipedia. In addition, we propose a RAG-based baseline
+hybrid QA model, achieving an exact match score of 69.65 on the Hybrid-SQuAD
+test set.
+
+
+
+
+ + ♻ ☆ Quest: Query-centric Data Synthesis Approach for Long-context Scaling of + Large Language Model +
+ +
+ Recent advancements in large language models (LLMs) have highlighted the
+importance of extending context lengths for handling complex tasks. While
+traditional methods for training on long contexts often use filtered long
+documents, these approaches lead to domain imbalances, limiting model
+performance. To address this, techniques like random document concatenation
+(Standard) and similarity-based methods (KNN, ICLM) have been developed.
+However, they either sacrifice semantic coherence or diversity. To balance both
+aspects, we introduce Quest, a query-centric data synthesis method aggregating
+semantically relevant yet diverse documents. Quest uses a generative model to
+predict potential queries for each document, grouping documents with similar
+queries and keywords. Extensive experiments demonstrate Quest's superior
+performance on long-context tasks, achieving remarkable results with context
+lengths of up to 1M tokens and confirming its scalability across various model
+sizes.
+
+
+
+
+ + ♻ ☆ HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced + Context Awareness and Extrapolation +
+ +
+ Many positional encodings (PEs) are designed to exhibit long-term decay,
+based on an entrenched and long-standing inductive opinion: tokens farther away
+from the current position carry less relevant information. We argue that
+long-term decay is outdated in the era of LLMs, as LLMs are now applied to
+tasks demanding precise retrieval of in-context information from arbitrary
+positions. Firstly, we present empirical analyses on various PEs, demonstrating
+that models inherently learn attention with only a local-decay pattern while
+forming a U-shape pattern globally, contradicting the principle of long-term
+decay. Furthermore, we conduct a detailed analysis of rotary position encoding
+(RoPE, a prevalent relative positional encoding in LLMs), and found that the
+U-shape attention is caused by some learned components, which are also the key
+factor limiting RoPE's expressiveness and extrapolation.Inspired by these
+insights, we propose High-frequency rotary Position Encoding (HoPE). HoPE
+replaces the specific components in RoPE with position-independent ones,
+retaining only high-frequency signals, which also breaks the principle of
+long-term decay in theory. HoPE achieves two major advantages: (1) Without
+constraints imposed by long-term decay, contradictory factors that limit
+spontaneous attention optimization and model extrapolation performance are
+removed. (2) Components representing positions and semantics are are optimized.
+These enhances model's context awareness and extrapolation, as validated by
+extensive experiments.
+
+
+
+
+ + ♻ ☆ ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of + Large Language Models in Real-world Scenarios COLING 2025 +
+
+
+
+
+
+
+
+ Junjie Ye, Guanyu Li, Songyang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Tao Ji, Qi Zhang, Tao Gui, Xuanjing Huang
+
+
+ Existing evaluations of tool learning primarily focus on validating the
+alignment of selected tools for large language models (LLMs) with expected
+outcomes. However, these approaches rely on a limited set of scenarios where
+answers can be pre-determined, diverging from genuine needs. Furthermore, a
+sole emphasis on outcomes disregards the complex capabilities required for LLMs
+to effectively use tools. To tackle this issue, we propose ToolEyes, a
+fine-grained system tailored for the evaluation of the LLMs' tool learning
+capabilities in authentic scenarios. The system meticulously examines seven
+real-world scenarios, analyzing five dimensions crucial to LLMs in tool
+learning: format alignment, intent comprehension, behavior planning, tool
+selection, and answer organization. Additionally, ToolEyes incorporates a tool
+library boasting approximately 600 tools, serving as an intermediary between
+LLMs and the physical world. Evaluations involving ten LLMs across three
+categories reveal a preference for specific scenarios and limited cognitive
+abilities in tool learning. Intriguingly, expanding the model size even
+exacerbates the hindrance to tool learning. The code and data are available at
+https://github.com/Junjie-Ye/ToolEyes.
+
+
+
+ comment: Accepted by COLING 2025 conference
+
+
+
+ + ♻ ☆ LuxEmbedder: A Cross-Lingual Approach to Enhanced Luxembourgish Sentence + Embeddings COLING 2025 +
+ +
+ Sentence embedding models play a key role in various Natural Language
+Processing tasks, such as in Topic Modeling, Document Clustering and
+Recommendation Systems. However, these models rely heavily on parallel data,
+which can be scarce for many low-resource languages, including Luxembourgish.
+This scarcity results in suboptimal performance of monolingual and
+cross-lingual sentence embedding models for these languages. To address this
+issue, we compile a relatively small but high-quality human-generated
+cross-lingual parallel dataset to train LuxEmbedder, an enhanced sentence
+embedding model for Luxembourgish with strong cross-lingual capabilities.
+Additionally, we present evidence suggesting that including low-resource
+languages in parallel training datasets can be more advantageous for other
+low-resource languages than relying solely on high-resource language pairs.
+Furthermore, recognizing the lack of sentence embedding benchmarks for
+low-resource languages, we create a paraphrase detection benchmark specifically
+for Luxembourgish, aiming to partially fill this gap and promote further
+research.
+
+
+
+ comment: Accepted at COLING 2025
+
+
+
+ + ♻ ☆ DRS: Deep Question Reformulation With Structured Output +
+ +
+ Question answering represents a core capability of large language models
+(LLMs). However, when individuals encounter unfamiliar knowledge in texts, they
+often formulate questions that the text itself cannot answer due to
+insufficient understanding of the underlying information. Recent studies reveal
+that while LLMs can detect unanswerable questions, they struggle to assist
+users in reformulating these questions. Even advanced models like GPT-3.5
+demonstrate limited effectiveness in this regard. To address this limitation,
+we propose DRS: Deep Question Reformulation with Structured Output, a novel
+zero-shot method aimed at enhancing LLMs ability to assist users in
+reformulating questions to extract relevant information from new documents. DRS
+combines the strengths of LLMs with a DFS-based algorithm to iteratively
+explore potential entity combinations and constrain outputs using predefined
+entities. This structured approach significantly enhances the reformulation
+capabilities of LLMs. Comprehensive experimental evaluations demonstrate that
+DRS improves the reformulation accuracy of GPT-3.5 from 23.03% to 70.42%, while
+also enhancing the performance of open-source models, such as Gemma2-9B, from
+26.35% to 56.75%.
+
+
+
+
+ + ♻ ☆ A Little Goes a Long Way: Efficient Long Context Training and Inference + with Partial Contexts +
+ +
+ Training and serving long-context large language models (LLMs) incurs
+substantial overhead. To address this, two critical steps are often required: a
+pretrained LLM typically undergoes a separate stage for context length
+extension by training on long-context data, followed by architectural
+modifications to reduce the overhead of KV cache during serving. This paper
+argues that integrating length extension with a GPU-friendly KV cache reduction
+architecture not only reduces training overhead during length extension, but
+also achieves better long-context performance. This leads to our proposed
+LongGen, which finetunes a pretrained LLM into an efficient architecture during
+length extension. LongGen builds on three key insights: (1) Sparse attention
+patterns, such as window attention (attending to recent tokens), attention sink
+(initial ones), and blockwise sparse attention (strided token blocks) are
+well-suited for building efficient long-context models, primarily due to their
+GPU-friendly memory access patterns, enabling efficiency gains not just
+theoretically but in practice as well. (2) It is essential for the model to
+have direct access to all tokens. A hybrid architecture with 1/3 full attention
+layers and 2/3 efficient ones achieves a balanced trade-off between efficiency
+and long-context performance. (3) Lightweight training on 5B long-context data
+is sufficient to extend the hybrid model's context length from 4K to 128K.
+ We evaluate LongGen on both Llama-2 7B and Llama-2 70B, demonstrating its
+effectiveness across different scales. During training with 128K-long contexts,
+LongGen achieves 1.55x training speedup and reduces wall-clock time by 36%,
+compared to a full-attention baseline. During inference, LongGen reduces KV
+cache memory by 62%, achieving 1.67x prefilling speedup and 1.41x decoding
+speedup.
+
+
+
+
+ + ♻ ☆ Concept Based Continuous Prompts for Interpretable Text Classification +
+ +
+ Continuous prompts have become widely adopted for augmenting performance
+across a wide range of natural language tasks. However, the underlying
+mechanism of this enhancement remains obscure. Previous studies rely on
+individual words for interpreting continuous prompts, which lacks comprehensive
+semantic understanding. Drawing inspiration from Concept Bottleneck Models, we
+propose a framework for interpreting continuous prompts by decomposing them
+into human-readable concepts. Specifically, to ensure the feasibility of the
+decomposition, we demonstrate that a corresponding concept embedding matrix and
+a coefficient matrix can always be found to replace the prompt embedding
+matrix. Then, we employ GPT-4o to generate a concept pool and choose potential
+candidate concepts that are discriminative and representative using a novel
+submodular optimization algorithm. Experiments demonstrate that our framework
+can achieve similar results as the original P-tuning and word-based approaches
+using only a few concepts while providing more plausible results. Our code is
+available at https://github.com/qq31415926/CD.
+
+
+
+
+ + ♻ ☆ Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language + Models NeurIPS 2024 +
+
+
+
+
+
+
+
+ Jiacheng Ye, Shansan Gong, Liheng Chen, Lin Zheng, Jiahui Gao, Han Shi, Chuan Wu, Xin Jiang, Zhenguo Li, Wei Bi, Lingpeng Kong
+
+
+ Recently, diffusion models have garnered significant interest in the field of
+text processing due to their many potential advantages compared to conventional
+autoregressive models. In this work, we propose Diffusion-of-Thought (DoT), a
+novel approach that integrates diffusion models with Chain-of-Thought, a
+well-established technique for improving the reasoning ability of
+autoregressive language models. In contrast to autoregressive language models
+that make decisions in a left-to-right, token-by-token manner, DoT allows
+reasoning steps to diffuse over time through a diffusion language model and
+offers greater flexibility in trading-off computation for reasoning
+performance. Our experimental results demonstrate the effectiveness of DoT in
+multi-digit multiplication, boolean logic, and grade school math problems, with
+a small diffusion model outperforming a much larger autoregressive model in
+both efficiency and accuracy. In addition to that, DoT showcases promising
+self-correction abilities and benefits from existing reasoning-enhancing
+techniques like self-consistency decoding. Our findings contribute to the
+understanding and development of reasoning with diffusion language models.
+
+
+
+ comment: NeurIPS 2024
+
+
+
+ + ♻ ☆ Text-Tuple-Table: Towards Information Integration in Text-to-Table + Generation via Global Tuple Extraction EMNLP 2024 +
+
+
+
+
+
+
+
+ Zheye Deng, Chunkit Chan, Weiqi Wang, Yuxi Sun, Wei Fan, Tianshi Zheng, Yauwai Yim, Yangqiu Song
+
+
+ The task of condensing large chunks of textual information into concise and
+structured tables has gained attention recently due to the emergence of Large
+Language Models (LLMs) and their potential benefit for downstream tasks, such
+as text summarization and text mining. Previous approaches often generate
+tables that directly replicate information from the text, limiting their
+applicability in broader contexts, as text-to-table generation in real-life
+scenarios necessitates information extraction, reasoning, and integration.
+However, there is a lack of both datasets and methodologies towards this task.
+In this paper, we introduce LiveSum, a new benchmark dataset created for
+generating summary tables of competitions based on real-time commentary texts.
+We evaluate the performances of state-of-the-art LLMs on this task in both
+fine-tuning and zero-shot settings, and additionally propose a novel pipeline
+called $T^3$(Text-Tuple-Table) to improve their performances. Extensive
+experimental results demonstrate that LLMs still struggle with this task even
+after fine-tuning, while our approach can offer substantial performance gains
+without explicit training. Further analyses demonstrate that our method
+exhibits strong generalization abilities, surpassing previous approaches on
+several other text-to-table datasets. Our code and data can be found at
+https://github.com/HKUST-KnowComp/LiveSum.
+
+
+
+ comment: Accepted to EMNLP 2024
+
+
+
+ + ♻ ☆ PreAct: Prediction Enhances Agent's Planning Ability +
+
+
+
+
+
+
+
+ Dayuan Fu, Jianzhao Huang, Siyuan Lu, Guanting Dong, Yejie Wang, Keqing He, Weiran Xu
+
+
+ Addressing the disparity between forecasts and actual results can enable
+individuals to expand their thought processes and stimulate self-reflection,
+thus promoting accurate planning. In this research, we present **PreAct**, an
+agent framework that integrates **pre**diction, **rea**soning, and **act**ion.
+By utilizing the information derived from predictions, the large language model
+(LLM) agent can provide a wider range and more strategically focused reasoning.
+This leads to more efficient actions that aid the agent in accomplishing
+intricate tasks. Our experimental results show that PreAct surpasses the ReAct
+method in completing complex tasks and that PreAct's performance can be further
+improved when paired with other memory or selection strategy techniques. We
+presented the model with varying quantities of historical predictions and
+discovered that these predictions consistently enhance LLM planning.The
+variances in single-step reasoning between PreAct and ReAct indicate that
+PreAct indeed has benefits in terms of diversity and strategic orientation over
+ReAct.
+
+
+
+ comment: Coling 2025
+
+
+
+ + ♻ ☆ Network Formation and Dynamics Among Multi-LLMs +
+ +
+ Social networks fundamentally shape human opinions, behaviors, and the
+dissemination of information. As large language models (LLMs) like GPT, Claude,
+and Llama increasingly integrate into social and professional settings,
+understanding their behavior in the context of social interactions and network
+formation becomes essential. This study develops a framework to systematically
+examine whether the network formation behaviors of multiple LLMs approximate
+certain aspects of human network dynamics. By simulating interactions among LLM
+agents across various model families, we observe that these models consistently
+exhibit key patterns associated with social network principles including
+preferential attachment, triadic closure, homophily, community structure, and
+the small-world phenomenon when forming networks. Moreover, LLMs adapt their
+network formation strategies based on each network's characteristics,
+reflecting the context-dependent nature of human behavior: in Facebook
+networks, they prioritize triadic closure and homophily, mirroring close-knit
+friendships; in phone networks, homophily and preferential attachment dominate,
+capturing personal and professional connections, while in employment networks,
+LLMs favor heterophily and high-degree connections, aligning with career
+advancement dynamics. These results open new avenues for using LLMs in network
+science research, with potential applications in agent-based modeling and
+synthetic network generation.
+
+
+
+
+ + ♻ ☆ Yi-Lightning Technical Report +
+
+
+
+
+
+
+
+ 01. AI, :, Alan Wake, Albert Wang, Bei Chen, C. X. Lv, Chao Li, Chengen Huang, Chenglin Cai, Chujie Zheng, Daniel Cooper, Ethan Dai, Fan Zhou, Feng Hu, Heng Ji, Howard Qiu, Jiangcheng Zhu, Jun Tian, Katherine Su, Lihuan Zhang, Liying Li, Ming Song, Mou Li, Peng Liu, Qicheng Hu, Shawn Wang, Shijun Zhou, Shiyong Li, Tianhang Zhu, Wen Xie, Xiang He, Xiaobo Chen, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Yanpeng Li, Yongke Zhao, Yongzhen Luo, Yuchi Xu, Yuxuan Sha, Zhaodong Yan, Zhiyuan Liu, Zirui Zhang
+
+
+ This technical report presents Yi-Lightning, our latest flagship large
+language model (LLM). It achieves exceptional performance, ranking 6th overall
+on Chatbot Arena, with particularly strong results (2nd to 4th place) in
+specialized categories including Chinese, Math, Coding, and Hard Prompts.
+Yi-Lightning leverages an enhanced Mixture-of-Experts (MoE) architecture,
+featuring advanced expert segmentation and routing mechanisms coupled with
+optimized KV-caching techniques. Our development process encompasses
+comprehensive pre-training, supervised fine-tuning (SFT), and reinforcement
+learning from human feedback (RLHF), where we devise deliberate strategies for
+multi-stage training, synthetic data construction, and reward modeling.
+Furthermore, we implement RAISE (Responsible AI Safety Engine), a
+four-component framework to address safety issues across pre-training,
+post-training, and serving phases. Empowered by our scalable super-computing
+infrastructure, all these innovations substantially reduce training, deployment
+and inference costs while maintaining high-performance standards. With further
+evaluations on public academic benchmarks, Yi-Lightning demonstrates
+competitive performance against top-tier LLMs, while we observe a notable
+disparity between traditional, static benchmark results and real-world, dynamic
+human preferences. This observation prompts a critical reassessment of
+conventional benchmarks' utility in guiding the development of more intelligent
+and powerful AI systems for practical applications. Yi-Lightning is now
+available through our developer platform at https://platform.lingyiwanwu.com.
+
+
+
+
+ + ♻ ☆ Social Life Simulation for Non-Cognitive Skills Learning +
+ +
+ Non-cognitive skills are crucial for personal and social life well-being, and
+such skill development can be supported by narrative-based (e.g., storytelling)
+technologies. While generative AI enables interactive and role-playing
+storytelling, little is known about how users engage with and perceive the use
+of AI in social life simulation for non-cognitive skills learning.
+Additionally, the benefits of AI mentorship on self-reflection awareness and
+ability in this context remain largely underexplored. To this end, we
+introduced Simulife++, an interactive platform enabled by a large language
+model (LLM). The system allows users to act as protagonists, creating stories
+with one or multiple AI-based characters in diverse social scenarios. In
+particular, we expanded the Human-AI interaction to a Human-AI-AI collaboration
+by including a Sage Agent, who acts as a bystander, providing users with some
+perspectives and guidance on their choices and conversations in terms of
+non-cognitive skills to promote reflection. In a within-subject user study, our
+quantitative results reveal that, when accompanied by Sage Agent, users exhibit
+significantly higher levels of reflection on motivation, self-perceptions, and
+resilience & coping, along with an enhanced experience of narrative
+transportation. Additionally, our qualitative findings suggest that Sage Agent
+plays a crucial role in promoting reflection on non-cognitive skills, enhancing
+social communication and decision-making performance, and improving overall
+user experience within Simulife++. Multiple supportive relationships between
+Sage Agent and users were also reported. We offer design implications for the
+application of generative AI in narrative solutions and the future potential of
+Sage Agent for non-cognitive skill development in broader social contexts.
+
+
+
+
+ + ♻ ☆ CREMA: Generalizable and Efficient Video-Language Reasoning via + Multimodal Modular Fusion +
+ +
+ Despite impressive advancements in recent multimodal reasoning approaches,
+they are still limited in flexibility and efficiency, as these models typically
+process only a few fixed modality inputs and require updates to numerous
+parameters. This paper tackles these critical challenges and proposes CREMA, a
+generalizable, highly efficient, and modular modality-fusion framework that can
+incorporate any new modality to enhance video reasoning. We first augment
+multiple informative modalities (such as optical flow, 3D point cloud, audio,
+thermal heatmap, and touch map) from given videos without extra human
+annotation by leveraging sensors or existing pre-trained models. Next, we
+introduce a query transformer with multiple parameter-efficient modules
+associated with each accessible modality. It projects diverse modality features
+to the LLM token embedding space, allowing the model to integrate different
+data types for response generation. Furthermore, we propose a novel progressive
+multimodal fusion design supported by a lightweight fusion module and
+modality-sequential training strategy. It helps compress information across
+various assisting modalities, maintaining computational efficiency in the LLM
+while improving performance. We validate our method on 7 video-language
+reasoning tasks assisted by diverse modalities, including conventional VideoQA
+and Video-Audio/3D/Touch/Thermal QA, and achieve better/equivalent performance
+against strong multimodal LLMs, including OneLLM, BLIP-2, and SeViLA while
+reducing over 90% trainable parameters. We provide extensive analyses of CREMA,
+including the impact of each modality on reasoning domains, the design of the
+fusion module, and example visualizations.
+
+
+
+ comment: first two authors contributed equally. Project page:
+ https://CREMA-VideoLLM.github.io/
+
+
+
+ + ♻ ☆ Calibrating Reasoning in Language Models with Internal Consistency NeurIPS 2024 +
+ +
+ Large language models (LLMs) have demonstrated impressive capabilities in
+various reasoning tasks, aided by techniques like chain-of-thought prompting
+that elicits verbalized reasoning. However, LLMs often generate text with
+obvious mistakes and contradictions, raising doubts about their ability to
+robustly process and utilize generated rationales. In this work, we investigate
+reasoning in LLMs through the lens of internal representations, focusing on how
+these representations are influenced by generated rationales. Our preliminary
+analysis reveals that while generated rationales improve answer accuracy,
+inconsistencies emerge between the model's internal representations in middle
+layers and those in final layers, potentially undermining the reliability of
+their reasoning processes. To address this, we propose internal consistency as
+a measure of the model's confidence by examining the agreement of latent
+predictions decoded from intermediate layers. Extensive empirical studies
+across different models and datasets demonstrate that internal consistency
+effectively distinguishes between correct and incorrect reasoning paths.
+Motivated by this, we propose a new approach to calibrate reasoning by
+up-weighting reasoning paths with high internal consistency, resulting in a
+significant boost in reasoning performance. Further analysis uncovers distinct
+patterns in attention and feed-forward modules across layers, providing
+insights into the emergence of internal inconsistency. In summary, our results
+demonstrate the potential of using internal representations for self-evaluation
+of LLMs. Our code is available at github.com/zhxieml/internal-consistency.
+
+
+
+ comment: NeurIPS 2024 camera ready
+
+
+
+ + ♻ ☆ From Pixels to Insights: A Survey on Automatic Chart Understanding in + the Era of Large Foundation Models +
+
+
+
+
+
+
+
+ Kung-Hsiang Huang, Hou Pong Chan, Yi R. Fung, Haoyi Qiu, Mingyang Zhou, Shafiq Joty, Shih-Fu Chang, Heng Ji
+
+
+ Data visualization in the form of charts plays a pivotal role in data
+analysis, offering critical insights and aiding in informed decision-making.
+Automatic chart understanding has witnessed significant advancements with the
+rise of large foundation models in recent years. Foundation models, such as
+large language models, have revolutionized various natural language processing
+tasks and are increasingly being applied to chart understanding tasks. This
+survey paper provides a comprehensive overview of the recent developments,
+challenges, and future directions in chart understanding within the context of
+these foundation models. We review fundamental building blocks crucial for
+studying chart understanding tasks. Additionally, we explore various tasks and
+their evaluation metrics and sources of both charts and textual inputs. Various
+modeling strategies are then examined, encompassing both classification-based
+and generation-based approaches, along with tool augmentation techniques that
+enhance chart understanding performance. Furthermore, we discuss the
+state-of-the-art performance of each task and discuss how we can improve the
+performance. Challenges and future directions are addressed, highlighting the
+importance of several topics, such as domain-specific charts, lack of efforts
+in developing evaluation metrics, and agent-oriented settings. This survey
+paper serves as a comprehensive resource for researchers and practitioners in
+the fields of natural language processing, computer vision, and data analysis,
+providing valuable insights and directions for future research in chart
+understanding leveraging large foundation models. The studies mentioned in this
+paper, along with emerging new research, will be continually updated at:
+https://github.com/khuangaf/Awesome-Chart-Understanding.
+
+
+
+ comment: IEEE Transactions on Knowledge and Data Engineering (TKDE)
+
+
+
+ + ♻ ☆ Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via + Mechanistic Localization +
+ +
+ Methods for knowledge editing and unlearning in large language models seek to
+edit or remove undesirable knowledge or capabilities without compromising
+general language modeling performance. This work investigates how mechanistic
+interpretability -- which, in part, aims to identify model components
+(circuits) associated to specific interpretable mechanisms that make up a model
+capability -- can improve the precision and effectiveness of editing and
+unlearning. We find a stark difference in unlearning and edit robustness when
+training components localized by different methods. We highlight an important
+distinction between methods that localize components based primarily on
+preserving outputs, and those finding high level mechanisms with predictable
+intermediate states. In particular, localizing edits/unlearning to components
+associated with the lookup-table mechanism for factual recall 1) leads to more
+robust edits/unlearning across different input/output formats, and 2) resists
+attempts to relearn the unwanted information, while also reducing unintended
+side effects compared to baselines, on both a sports facts dataset and the
+CounterFact dataset across multiple models. We also find that certain localized
+edits disrupt the latent knowledge in the model more than any other baselines,
+making unlearning more robust to various attacks.
+
+
+
+ comment: 31 pages, 45 figures, 7 tables
+
+
+
+ + ♻ ☆ VersaTune: An Efficient Data Composition Framework for Training + Multi-Capability LLMs +
+
+
+
+
+
+
+
+ Keer Lu, Keshi Zhao, Zheng Liang, Da Pan, Shusen Zhang, Xin Wu, Weipeng Chen, Zenan Zhou, Guosheng Dong, Bin Cui, Wentao Zhang
+
+
+ Large-scale pretrained models, particularly Large Language Models (LLMs),
+have exhibited remarkable capabilities in handling multiple tasks across
+domains due to their emergent properties. These capabilities are further
+augmented during the Supervised Fine-Tuning (SFT) phase. Despite their
+potential, existing work mainly focuses on domain-specific enhancements during
+fine-tuning, the challenge of which lies in catastrophic forgetting of
+knowledge across other domains. In this study, we introduce VersaTune, a novel
+data composition framework designed for enhancing LLMs' overall multi-ability
+performances during training. We categorize knowledge into distinct domains
+including law, medicine, finance, science, code, etc. We begin with detecting
+the distribution of domain-specific knowledge within the base model, followed
+by the training data composition that aligns with the model's existing
+knowledge distribution. During the training process, domain weights are
+dynamically adjusted based on their learnable potential and forgetting degree.
+Experimental results demonstrate that VersaTune achieves significant
+improvements in multi-domain performance, with an 35.21% enhancement in
+comprehensive multi-domain tasks. Additionally, in scenarios where specific
+domain optimization is required, VersaTune reduces the degradation of
+performance in other domains by 38.77%, without compromising the target
+domain's training efficacy.
+
+
+
+
+ + ♻ ☆ UniPoll: A Unified Social Media Poll Generation Framework via + Multi-Objective Optimization +
+ +
+ Social media platforms are vital for expressing opinions and understanding
+public sentiment, yet many analytical tools overlook passive users who mainly
+consume content without engaging actively. To address this, we introduce
+UniPoll, an advanced framework designed to automatically generate polls from
+social media posts using sophisticated natural language generation (NLG)
+techniques. Unlike traditional methods that struggle with social media's
+informal and context-sensitive nature, UniPoll leverages enriched contexts from
+user comments and employs multi-objective optimization to enhance poll
+relevance and engagement. To tackle the inherently noisy nature of social media
+data, UniPoll incorporates Retrieval-Augmented Generation (RAG) and synthetic
+data generation, ensuring robust performance across real-world scenarios. The
+framework surpasses existing models, including T5, ChatGLM3, and GPT-3.5, in
+generating coherent and contextually appropriate question-answer pairs.
+Evaluated on the Chinese WeiboPolls dataset and the newly introduced English
+RedditPolls dataset, UniPoll demonstrates superior cross-lingual and
+cross-platform capabilities, making it a potent tool to boost user engagement
+and create a more inclusive environment for interaction.
+
+
+
+ comment: Accepted by IEEE Transactions on Neural Networks and Learning
+ Systems. Project page is live at https://uni-poll.github.io . Code are
+ available at https://github.com/X1AOX1A/UniPoll
+
+
+
+ + ♻ ☆ ScribeAgent: Towards Specialized Web Agents Using Production-Scale + Workflow Data +
+
+
+
+
+
+
+
+ Junhong Shen, Atishay Jain, Zedian Xiao, Ishan Amlekar, Mouad Hadji, Aaron Podolny, Ameet Talwalkar
+
+
+ Large Language Model (LLM) agents are rapidly improving to handle
+increasingly complex web-based tasks. Most of these agents rely on
+general-purpose, proprietary models like GPT-4 and focus on designing better
+prompts to improve their planning abilities. However, general-purpose LLMs are
+not specifically trained to understand specialized web contexts such as HTML,
+and they often struggle with long-horizon planning. We explore an alternative
+approach that fine-tunes open-source LLMs using production-scale workflow data
+collected from over 250 domains corresponding to 6 billion tokens. This simple
+yet effective approach shows substantial gains over prompting-based agents on
+existing benchmarks -- ScribeAgent achieves state-of-the-art direct generation
+performance on Mind2Web and improves the task success rate by 7.3% over the
+previous best text-only web agents on WebArena. We further perform detailed
+ablation studies on various fine-tuning design choices and provide insights
+into LLM selection, training recipes, context window optimization, and effect
+of dataset sizes.
+
+
+
+
+ + ♻ ☆ Designing LLM Chains by Adapting Techniques from Crowdsourcing Workflows +
+
+
+
+
+
+
+
+ Madeleine Grunde-McLaughlin, Michelle S. Lam, Ranjay Krishna, Daniel S. Weld, Jeffrey Heer
+
+
+ LLM chains enable complex tasks by decomposing work into a sequence of
+subtasks. Similarly, the more established techniques of crowdsourcing workflows
+decompose complex tasks into smaller tasks for human crowdworkers. Chains
+address LLM errors analogously to the way crowdsourcing workflows address human
+error. To characterize opportunities for LLM chaining, we survey 107 papers
+across the crowdsourcing and chaining literature to construct a design space
+for chain development. The design space covers a designer's objectives and the
+tactics used to build workflows. We then surface strategies that mediate how
+workflows use tactics to achieve objectives. To explore how techniques from
+crowdsourcing may apply to chaining, we adapt crowdsourcing workflows to
+implement LLM chains across three case studies: creating a taxonomy, shortening
+text, and writing a short story. From the design space and our case studies, we
+identify takeaways for effective chain design and raise implications for future
+research and development.
+
+
+