# distributed-llm-pretraining-torchtitan

> Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP).

Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.

- Type: Skill
- Made by: Orchestra-Research
- Category: AI & agents
- Formats: Skills
- Version: 7b08ed5
- Source: https://github.com/orchestra-research/ai-research-skills/tree/HEAD/01-model-architecture/torchtitan
- Page: https://pluginboard.ai/skills/distributed-llm-pretraining-torchtitan

## Works with

- ChatGPT
- Claude
- Gemini
- GitHub Copilot
- Cursor
- Claude Code
- Codex
- Antigravity
- Gemini CLI
- Replit
- OpenClaw
- OpenCode
- Hermes Agent

## Install

### ChatGPT

In ChatGPT: Download the distributed-llm-pretraining-torchtitan folder and zip it, then Plugins → Skills → Create → Upload from your computer. Business and Enterprise plans.

View the folder on GitHub: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/HEAD/01-model-architecture/torchtitan

### Claude

In Claude (web or desktop): Download the distributed-llm-pretraining-torchtitan folder and zip the folder itself, then upload it in Customize → Skills.

View the folder on GitHub: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/HEAD/01-model-architecture/torchtitan

### Gemini

In Gemini: Download the distributed-llm-pretraining-torchtitan folder, then at gemini.google.com click Customize → Skills → Create → Upload and pick the folder (or a .zip of it).

View the folder on GitHub: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/HEAD/01-model-architecture/torchtitan

### GitHub Copilot

Run in your terminal:

```shell
npx -y degit Orchestra-Research/AI-Research-SKILLs/01-model-architecture/torchtitan ~/.copilot/skills/distributed-llm-pretraining-torchtitan
```

View the folder on GitHub: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/HEAD/01-model-architecture/torchtitan

### Cursor

Run in your terminal:

```shell
npx -y degit Orchestra-Research/AI-Research-SKILLs/01-model-architecture/torchtitan ~/.cursor/skills/distributed-llm-pretraining-torchtitan
```

View the folder on GitHub: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/HEAD/01-model-architecture/torchtitan

### Claude Code

Run in your terminal:

```shell
npx -y degit Orchestra-Research/AI-Research-SKILLs/01-model-architecture/torchtitan ~/.claude/skills/distributed-llm-pretraining-torchtitan
```

View the folder on GitHub: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/HEAD/01-model-architecture/torchtitan

Or all of AI Research Skills (Orchestra), in Claude Code:

```text
/plugin marketplace add Orchestra-Research/AI-Research-SKILLs
/plugin install rag@ai-research-skills
```

### Codex

Run in your terminal:

```shell
npx -y degit Orchestra-Research/AI-Research-SKILLs/01-model-architecture/torchtitan ~/.agents/skills/distributed-llm-pretraining-torchtitan
```

View the folder on GitHub: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/HEAD/01-model-architecture/torchtitan

### Antigravity

In Antigravity: Copy the distributed-llm-pretraining-torchtitan folder into ~/.gemini/config/skills (Antigravity app and IDE) or ~/.gemini/antigravity-cli/skills (Antigravity CLI).

View the folder on GitHub: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/HEAD/01-model-architecture/torchtitan

### Gemini CLI

Run in your terminal:

```shell
npx -y degit Orchestra-Research/AI-Research-SKILLs/01-model-architecture/torchtitan ~/.gemini/skills/distributed-llm-pretraining-torchtitan
```

View the folder on GitHub: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/HEAD/01-model-architecture/torchtitan

### Replit

Run in your terminal:

```shell
npx -y degit Orchestra-Research/AI-Research-SKILLs/01-model-architecture/torchtitan .agents/skills/distributed-llm-pretraining-torchtitan
```

View the folder on GitHub: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/HEAD/01-model-architecture/torchtitan

### OpenClaw

Run in your terminal:

```shell
npx -y degit Orchestra-Research/AI-Research-SKILLs/01-model-architecture/torchtitan ~/.openclaw/skills/distributed-llm-pretraining-torchtitan
```

View the folder on GitHub: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/HEAD/01-model-architecture/torchtitan

### OpenCode

Run in your terminal:

```shell
npx -y degit Orchestra-Research/AI-Research-SKILLs/01-model-architecture/torchtitan ~/.config/opencode/skills/distributed-llm-pretraining-torchtitan
```

View the folder on GitHub: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/HEAD/01-model-architecture/torchtitan

### Hermes Agent

Run in your terminal:

```shell
npx -y degit Orchestra-Research/AI-Research-SKILLs/01-model-architecture/torchtitan ~/.hermes/skills/distributed-llm-pretraining-torchtitan
```

View the folder on GitHub: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/HEAD/01-model-architecture/torchtitan

---
From Pluginboard (https://pluginboard.ai).