LLMs for Coding: Training & Evaluation

The initial stage of developing powerful LLMs for code involves pre-training on vast amounts of data. This phase equips the model with a fundamental understanding of programming language syntax, semantics, and common coding patterns. Effective pre-training is crucial as it forms the foundation upon which more specialized capabilities are built through fine-tuning.

Foundational Pre-training Objectives:

• Masked Language Modeling (MLM): Predicts masked tokens using bidirectional context. Good for code understanding.
• Next Token Prediction (NTP): Predicts the next token autoregressively. Suited for code generation.
• Fill-in-the-Middle (FIM): Predicts missing code spans given prefix and suffix. Crucial for infilling and editing. Includes Horizon-Length Prediction (HLP) for better planning.
• Contrastive Learning: Learns semantic similarity between code/text pairs. Useful for search and clone detection.

Data-Centric Pre-training Strategies:

The quality, diversity, and volume of data are paramount. Strategies include:

• Code Corpora Selection: Using large-scale repositories (The Stack), specialized datasets (CodeParrot), web data, and technical documentation.
• Data Curation and Refinement: Cleaning, deduplication, quality filtering (Ask-LLM, ProX), syntax validation.
• Data-Efficient Pre-training: Coverage/diversity sampling, focusing on high-quality samples.

Fine-tuning adapts pre-trained LLMs to specific coding tasks or domains, significantly enhancing performance. This involves training on smaller, curated datasets.

Instruction Tuning & Domain-Specific Fine-tuning:

Models are trained on (instruction, output) pairs to follow natural language commands (e.g., "generate a Python function..."). Frameworks like LLaMoCo provide comprehensive instruction sets. Domain-specific datasets help models learn nuances of particular languages or domains.

Parameter-Efficient Fine-Tuning (PEFT):

PEFT techniques update only a small subset of parameters, reducing computational cost. Common methods include:

• LoRA (Low-Rank Adaptation): Injects trainable low-rank matrices.
• IA3 (Infused Adapter by Inhibiting and Amplifying Inner Activations): Modifies activations.
• Prompt Tuning: Learns a "soft prompt" prepended to input.
• Prefix Tuning: Adds trainable prefix vectors to each layer.
• QLoRA: Combines LoRA with quantization for further memory reduction.

PEFT offers reduced cost, faster training, comparable performance to full fine-tuning, and mitigation of catastrophic forgetting.

Reinforcement Learning from Human Feedback (RLHF):

RLHF aligns LLMs with human preferences for code quality (correctness, readability, efficiency). The process involves:

Benefits: Improved alignment, handling subjectivity, personalization.
Challenges: Complexity, cost, reward hacking, reward model design.
Innovations: cRLHF, Direct Preference Optimization (DPO), RLAIF.

Data augmentation increases training data diversity and volume without new raw data, improving robustness and generalization.

Prompt-Based and LLM-Driven Data Generation:

• Prompting Techniques: Zero-shot, one-shot, few-shot, topic/controlled generation, Chain-of-Thought (CoT).
• LLM-Driven Pipelines: Self-Instruct (LLM generates new instruction-data), Inverse-Instruct (generates NL instructions for existing code), generating NL-code pairs.

Semantic-Preserving Transformations:

Modify code syntax without altering functionality.

• Lexical/Syntactic: Variable renaming, reformatting, comment changes.
• AST-Based: Loop conversion, statement reordering, boolean expression modification.
• Refactoring Operations: Extract method, inline variable.

Applications include robustness training, bug detection/repair, code clone detection, and contrastive pre-training.

Back-Translation and Cross-Lingual Techniques:

• Back-Translation for NL: NL prompt -> Other Language -> Original NL (lexical diversity).
• Code-to-NL-to-Code: Code -> NL summary -> New Code (syntactic diversity).
• Programming Language Translation: Python -> Java -> Python (functional equivalents in different syntaxes).

Continual Learning (CL) enables LLMs to learn new information and skills over time without forgetting previous knowledge, crucial for staying updated with evolving software, APIs, and practices.

Multi-Stage Continual Learning:

CL can be applied at different LLM development stages:

• Continual Pre-training (CPT): Expands fundamental understanding (new APIs, evolving practices, domain adaptation).
• Continual Instruction Tuning (CIT): Improves ability to follow new commands/tasks (new coding tasks, tool usage).
• Continual Alignment (CA): Ensures outputs remain aligned with human expectations and ethics (new coding standards, value alignment).

Addressing Catastrophic Forgetting:

A major CL challenge where models forget past knowledge. Mitigation strategies include:

• Experience Replay
• Regularization-Based Methods
• Parameter Isolation/Expansion
• Knowledge Distillation
• Data Pruning/Selection

Automated metrics quantify LLM performance on code tasks. Key metrics include:

• pass@k: Probability that at least one of $k$ generated samples passes unit tests. Measures functional correctness.
• CodeBLEU: BLEU adaptation for code, considering n-grams, AST matching, and data-flow. Measures syntactic/semantic similarity.
• Exact Match (EM): Percentage of exact matches to a reference solution. Often too strict.
• BLEU: N-gram overlap. Less suitable for code's functional aspects.
• pass-ratio@n: Average pass rate of test cases across $n$ solutions. Granular correctness.
• ICE-Score: LLM-based assessment of usefulness and functional correctness.

Standardized benchmarks provide common ground for comparing Code LLMs. Prominent examples:

• HumanEval: Hand-crafted Python problems for function-level generation.
• MBPP (Mostly Basic Python Problems): Crowd-sourced entry-level Python problems.
• CodeXGLUE: Comprehensive suite for 10 diverse code intelligence tasks.
• Web-Bench: Project-level web development tasks simulating real-world workflows.
• SWE-Bench & RepoBench: Repository-level bug fixes/feature implementations.
• SAFIM: Syntax-aware Fill-in-the-Middle tasks.

Many early benchmarks are becoming saturated, necessitating more complex and realistic challenges.

Automated metrics often miss nuanced code quality aspects.

Human Evaluation:

Indispensable for assessing readability, maintainability, style, algorithmic elegance, and subtle correctness. Methods include Likert scales, A/B testing, expert reviews, ranking. Criteria: correctness, efficiency, readability, maintainability, security. Challenges: slow, expensive, subjective.

LLM-as-a-Judge:

Uses a powerful LLM (e.g., GPT-4) to evaluate outputs of other LLMs. Can achieve high correlation with human scores for tasks like code translation/generation. Benefits: scalable, cost-effective. Challenges: judge LLM capability/bias, consistency.

Integrating established software engineering tools provides objective assessments.

Static Analysis Tools:

Analyze source code without execution. Examples:

• Pylint: Python standards, errors, code smells.
• Bandit: Python security issues.
• Radon: Python code complexity metrics.
• Infer: Memory safety (C, Java, Obj-C).
• SonarQube, PMD, FindBugs, Checkstyle: General bug/vulnerability/style checkers.

Provide objective metrics and structured feedback for LLM self-improvement.

Dynamic Analysis / Testing:

Executing generated code with test inputs to verify correctness and observe runtime behavior (basis for pass@k). Can also gather performance metrics and provide feedback for debugging.

As tasks become complex, evaluation incorporates more sophisticated, execution-driven assessments.

• Execution-Driven Evaluation for Correctness/Performance: Iterative refinement based on execution feedback (e.g., PerfCodeGen). Using runtime info for debugging.
• Assessing Complex Code Edits and Agentic Workflows: LLM-based critics for execution-free evaluation of repo-level changes.
• Frameworks Combining Multi-Agent Collaboration and Execution-Based Debugging: Simulating development teams with specialized LLM agents, followed by execution and debugging.