Llama 3.1

Latest

Meta

•

Open Source

•

405B

# 65

Released

Jul 23, 2024

# 11

Knowledge Cutoff

Dec 23

# 9

Context Length

128K

Benchmarks

# 85

Code RankedAGI

32.1%

# 29

SWEBench Verified

24.5%

# 34

WebDev Arena

811.91

# 35

LiveCodeBench v5

28.4%

# 34

Code LMArena

1260

# 41

Code LiveBench (old)

42.6%

# 52

GPQA Diamond

51.1%

# 27

Reason LiveBench (old)

53.3%

# 39

ELO LMArena

1266

# 38

Math LiveBench (old)

40.5%

# 24

MATH

73.8%

# 12

Human Eval

89.0%

# 21

NYT Connections

16.2%

# 24

MMLU Pro

73.3%

# 14

MMLU

87.0%

# 21

Halluc. Hughes

3.9%

# 19

Aidan Bench

778

# 22

IF LiveBench (old)

75.9%

# 24

Avg LiveBench (old)

58.6%

# 7

IF Evaluation

88.6%

Pricing

# 29

Input Cost /M

$5

# 33

Output Cost /M

$15