raihan-js/orch-nextjs-3b

ORCH Next.js 3B - Full-Stack Code Generation Model

A 3 billion parameter model specialized for generating complete, production-ready Next.js applications

Model Description

ORCH Next.js 3B is a decoder-only transformer model trained specifically for generating full-stack Next.js applications. The model can generate complete project structures including:

• Frontend: React components, pages, layouts with TypeScript
• Backend: API routes, server actions, middleware
• Database: Prisma schemas, database utilities
• Styling: Tailwind CSS, component libraries
• Configuration: Package.json, tsconfig, next.config

Model Architecture

Parameter	Value
Total Parameters	~3.0 Billion
Architecture	LLaMA-style Transformer
Hidden Size	2,560
Layers	32
Attention Heads	32
KV Heads (GQA)	8
Vocabulary Size	32,000
Max Context Length	16,384 tokens
Activation	SwiGLU
Normalization	RMSNorm
Position Encoding	RoPE (Rotary Position Embeddings)

Training Details

• Training Data: Curated Next.js repositories from GitHub
• Training Duration: 3 epochs (~2 hours on A40 48GB)
• Training Steps: ~29,000 steps
• Batch Size: 1 (with gradient accumulation 16)
• Sequence Length: 512 tokens
• Precision: BFloat16
• Hardware: NVIDIA A40 48GB (RunPod)

Usage

Quick Start

import torch
from orch.model.config import OrchConfig
from orch.model.transformer import OrchForCausalLM
from tokenizers import Tokenizer

# Load model
config = OrchConfig.load("config.json")
tokenizer = Tokenizer.from_file("tokenizer.json")

model = OrchForCausalLM(config)
model.load_state_dict(torch.load("model.pt", map_location="cpu"))
model = model.to(dtype=torch.bfloat16).cuda()
model.eval()

# Generate code
prompt = "// Next.js dashboard page\nexport default function Dashboard() {"

tokens = tokenizer.encode(prompt)
input_ids = torch.tensor([tokens.ids]).cuda()

with torch.no_grad():
    for _ in range(200):
        outputs = model(input_ids)
        logits = outputs["logits"][:, -1, :]
        next_token = torch.argmax(logits, dim=-1)
        input_ids = torch.cat([input_ids, next_token.unsqueeze(0)], dim=1)

output = tokenizer.decode(input_ids[0].tolist())
print(output)

Using with ORCH Framework

# Clone ORCH
git clone https://github.com/raihanmehran/orch.git
cd orch

# Install dependencies
pip install -e .

# Download model
python -c "from huggingface_hub import snapshot_download; snapshot_download('raihan-js/orch-nextjs-3b', local_dir='models/orch-3b-nextjs')"

# Run web interface
python web/server.py --model models/orch-3b-nextjs --port 8080

Model Outputs

The model generates Next.js/React code patterns including:

// Example output for dashboard prompt
export default function Dashboard() {
  return (
    <div className="flex flex-col min-h-[100dvh]">
      <main className="flex-1">
        <section className="w-full py-12 md:py-24 lg:py-32">
          <div className="container px-4 md:px-6">
            {/* Dashboard content */}
          </div>
        </section>
      </main>
    </div>
  );
}

Limitations

1. Early Training: Model trained for only 3 epochs - quality improves with more training
2. Instruction Following: Currently generates code patterns, not fully instruction-following
3. Context Length: While architecture supports 16K, training used 512 token sequences
4. Validation: Generated code should be reviewed before production use

Intended Use

• Code Completion: Complete partial Next.js code
• Boilerplate Generation: Generate project scaffolding
• Learning: Understand Next.js patterns and best practices
• Prototyping: Quickly generate prototype code

Hardware Requirements

Hardware	Memory Usage	Notes
RTX 3060 12GB	~6.6 GB	Works with bfloat16
RTX 4090 24GB	~6.6 GB	Fast inference
A100 80GB	~6.6 GB	Training capable

Files

File	Size	Description
model.pt	6.41 GB	Model weights (bfloat16)
config.json	610 B	Model configuration
tokenizer.json	2.2 MB	BPE tokenizer

Citation

@misc{orch-nextjs-3b,
  author = {Raihan, Akteruzzaman},
  title = {ORCH Next.js 3B: Full-Stack Code Generation Model},
  year = {2024},
  publisher = {HuggingFace},
  url = {https://huggingface.co/raihan-js/orch-nextjs-3b}
}

License

Apache 2.0

Links

• GitHub: ORCH Repository
• Author: Akteruzzaman Raihan