Jongwon Lim

Stable RL with verifiable rewards requires both reliable baselines and diverse training prompts. Group-based methods improve baseline estimates by sampling more responses per prompt, but this leaves fewer distinct prompts under a fixed rollout budget. POISE estimates baselines with a lightweight probe of the actor's hidden states and entropy, without a separate LLM-scale critic. Predicting each rollout's baseline from an independent rollout keeps the policy gradient unbiased with two rollouts per prompt. This allows greater prompt diversity and improves average performance and training stability over the evaluated baselines in multi-domain training.

Prompting an LLM to express a value can produce different behavior from the model expressing that value without instructions. We examine whether the two rely on the same internal mechanisms. We identify shared neurons that causally contribute to both forms of value expression, alongside components with distinct roles. Components specific to intrinsic expression promote response diversity, whereas those specific to prompted expression strengthen instruction compliance, even on tasks unrelated to values.

RL improves vision-language-action (VLA) models, but which parameter changes drive these gains remains unclear. Across the flow-based VLAs we study, low-rank RL updates concentrate in timestep modules. These modules depend only on denoising time, not observations or instructions, yet module-replacement experiments show that their updates capture a large share of RL gains. We link the low-rank structure to training on discrete denoising timesteps. Targeting these modules with LoRA outperforms standard configurations, which typically omit them. Steering along their learned shift directions further improves RL-trained policies without additional RL training.

Negating a question should exclude the original answer, yet LLMs often repeat it, especially when they were confident in that answer. In the models we analyze, negation suppresses original-answer retrieval and promotes favored candidates within the answer category. However, these preferences do not reliably exclude the original answer, and the correction can be too weak to overturn it. We therefore propose ALiT, which requires larger corrections for more confident original predictions. ALiT reduces repetition while preserving the answer category and avoiding the sharp diversity loss observed with supervised fine-tuning.