Tiny LLM Benchmark: Jetson Orin Nano Super 8GB — Four Power Modes × Eight Models

Four Power Modes × Eight Models: llama.cpp / CUDA

Platform: NVIDIA Jetson Orin Nano Super 8GB
CPU: 6-core Arm Cortex-A78AE · GPU: NVIDIA Ampere (1024 CUDA cores, 32 Tensor cores)
Memory: 8 GB LPDDR5 shared CPU+GPU · JetPack: R36.4.7 (L4T 36.4)
Backend: llama.cpp CUDA, -ngl 99 (all layers on GPU), --no-cache-prompt
Runs: Four full sweeps: 7W, 15W, 25W, MAXN_SUPER
Sweep: prompt ∈ {128, 512, 1024, 2048} tok × gen ∈ {64, 128, 256} tok × 20 reqs/combo
Concurrency: 1 (single-user) · Key metric: output tok/J = OSL ÷ (avg_power_W × RL_p50_s)

Raw data on Hugging Face — complete per-cell JSON exports (all 33 metrics, 12 prompt×gen combos × 20 requests per cell, profile_export_aiperf.json + tegrastats.log + server logs):

Executive Summary

Eight tiny non-thinking LLMs were benchmarked across all four Jetson Orin Nano Super power modes: 7W, 15W, 25W, and MAXN_SUPER. Each model ran 12 combinations of prompt × generation length (20 requests per combo) at every power mode where it could load.

Key finding: 25W (nvpmodel -m 1) is the energy-efficiency sweet spot for every model tested. It delivers 36–47 % more output tok/s than 15W while pushing output tok/J 3–26 % higher than 15W and 8–35 % higher than MAXN_SUPER across every model (ctx=2048, gen=256).

Throughput winner at each mode (ctx=2048, gen=256, highest sweep point):

Table 1: Throughput and efficiency winner at each power mode

gemma3-4b (Q4_K_M, 2.4 GB) fails at every power mode: too large for 8 GB unified memory when combined with KV cache and CUDA overhead.

1. Test Setup

1.1 Hardware

Table 2: Hardware configuration

1.2 Software Stack

Table 3: Software stack

1.3 Models Under Test

Table 4: Models under test

Quantization note: SmolLM2-360M-Instruct and Qwen3-0.6B use Q8_0 (8-bit, near-lossless); all other models use Q4_K_M (4-bit K-quant medium). These match the quantizations that ollama ships for Qwen2.5-0.5B, Gemma3-1B, and Gemma3-4B, but differ for SmolLM2-135M (Ollama: F16), SmolLM2-360M (Ollama: F16), Qwen3-0.6B (Ollama: Q4_K_M), and Llama-3.2-1B (Ollama: Q8_0). Results should not be assumed comparable to ollama run defaults without accounting for these quantization differences.

1.4 Power Modes

Table 5: Power mode configurations

1.5 Benchmark Methodology

• For each model × prompt × gen combo, aiperf sends 20 single-concurrency requests with synthetic prompts at the exact target token count.
• Power is computed from tegrastats VDD_CPU_GPU_CV (mW → W) averaged over each run’s start_time/end_time window. output_tok_J = OSL ÷ (avg_power_W × RL_p50_s).
• Clocks were locked with jetson_clocks at all modes. CMA was compacted (/proc/sys/vm/compact_memory) between model loads.
• Each run’s power and clock speed was capped at x W through nvpmodel and monitored for thermal stability (no sustained throttling; junction temp ≤ 73 °C).
• Latency percentile used throughout: all TTFT, ITL, and request latency (RL) values reported in charts, tables, and energy calculations use the p50 (median) over the 20 requests per combo. The mean is not used for latency because occasional slow requests (GC pause, memory compaction, OS scheduling) inflate it without reflecting typical behaviour. p90 and p99 are available in the raw per-mode Hugging Face datasets (see raw data table at the top of this post) for tail-latency analysis.

2. Results: Charts

All charts use data from all four power modes.

2.1 Throughput vs Prompt Length

Output tok/s vs prompt length at gen=256 across all models and modes; 25W (orange) consistently leads:

Figure 1: Output tok/s vs prompt length (gen=256, all models and modes)

Canonical cell (ctx=2048, gen=256), side-by-side output tok/s and output tok/J bars for all 4 modes:

Figure 2: Canonical cell: output tok/s and tok/J side by side (ctx=2048, gen=256)

2.2 Energy Efficiency

• Output Tok/J vs prompt length at gen=256; 25W leads for every model at every prompt length:

Figure 3: Output tok/J vs prompt length (gen=256, all models and modes)

• Output Tok/J heatmap (gen x prompt) for small models (available at all 4 modes):

Figure 4: Output tok/J heatmap: small models at all 4 power modes (gen x prompt)

• Output Tok/J heatmap for larger models (all 4 modes):

Figure 5: Output tok/J heatmap: larger models at 15W / 25W / MAXN (gen x prompt)

• SmolLM2-135M spotlight: output tok/J at all 4 modes across gen sizes:

Figure 6: SmolLM2-135M output tok/J at all 4 power modes across gen lengths

• Prefill tok/J (input tokens per joule of prefill energy) vs prompt length at gen=256, how efficiently each mode processes the prompt; higher is better:

Figure 7a: Prefill tok/J (input tok / J) vs prompt length (gen=256, all models and modes)

⚠ Prefill tok/J is approximate for 63 % of cells (TTFT < 500 ms → no tegrastats sample in prefill window). Decode tok/J and total tok/J are not affected.

• Decode tok/J (output tokens per joule of decode energy) vs prompt length at gen=256, output generation efficiency; decreases with increase in prompt length since decode cost depends on output length not input; 25W leads:

Figure 7b: Decode tok/J (output tok / J) vs prompt length (gen=256, all models and modes)

• Total tok/J ((input + output) tokens per joule of total request energy) vs prompt length at gen=256, overall request efficiency; 25W wins at every model and prompt length:

Figure 7c: Total tok/J (input+output tok / J) vs prompt length (gen=256, all models and modes)

Full tok/J charts for all ctx/gen combinations: F.1 Prefill · F.2 Decode · F.3 Total.

2.3 Latency

TTFT p50 at ctx=2048, gen=256; 25W and MAXN reduce TTFT by 30–38 % vs 15W:

Figure 8: TTFT p50 by power mode (ctx=2048, gen=256)

ITL (inter-token latency) p50 at ctx=2048, gen=256; lower is better:

Figure 9: ITL p50 by power mode (ctx=2048, gen=256)

Request latency (E2E) p50 at ctx=2048, gen=256; total time from request start to last token received:

Figure 10: Request latency (E2E) p50 by power mode (ctx=2048, gen=256)

2.4 Prefill Throughput

25W and MAXN provide ~35-40 % faster prefill than 15W:

Figure 11: Prefill throughput by power mode (gen=256, avg over all prompt lengths)

2.5 Power Draw

Average VDD_CPU_GPU_CV per model at each mode:

Figure 12: Average VDD_CPU_GPU_CV power draw per model at each power mode

Table 6: Average power draw per model at each power mode (W, VDD_CPU_GPU_CV)

Formulae used - tok/s / tok/J. Bold = highest power draw per model.

3. Analysis

3.1 Higher tok/sec != efficient model (tok/J)

Tok/s (left half) and tok/J (right half) are intentionally both shown, a faster mode does not always mean a more efficient one.

• MAXN beats 25W on raw tok/s for some models but loses on tok/J because its power increase outpaces the throughput gain for ctx = 2048, gen = 256.

output_tok_J = tok_s / VDD_CPU_GPU_CV (W), averaged over each aiperf run window.

Table 7: Canonical cell comparison (ctx=2048, gen=256)

3.2 The 25W Sweet Spot

25W is unambiguously the best mode for output tok/J and tok/sec across every model. The reason is arithmetic:

• Going from 15W → 25W: output tok/s rises 36–47 % (GPU clock 612 → 820 MHz), while power rises ~36 %. Net output tok/J gain: +3 to +26 % (range wider than tok/s because 25W also cuts TTFT, shrinking the RL denominator).
• Going from 25W → MAXN: output tok/s changes −3 % to +8 % depending on model (decode is memory-bandwidth bound, not compute bound), while power rises ~17 %. Net output tok/J loss: −8 to −35 %.

The GPU clock ceiling at 15W (612 MHz) leaves significant decode throughput on the table. Raising it to 820 MHz at 25W captures most of the available throughput improvement with modest additional power. The final jump to 1020 MHz at MAXN costs disproportionate power for marginal gains.

Practical recommendation: Run at 25W for the best balance of speed and efficiency. Use MAXN only when minimising latency (TTFT) matters more than energy (e.g. interactive chat with long prompts).

3.3 Best use cases for each power mode

Table 8: Recommended power mode by use case

3.4 Throughput Speedup Summary

All figures are mean(p50) across the full prompt × gen sweep (12 combos per model); throughput uses mean(avg tok/s) since aiperf does not report a p50 for tok/s.

Table 9: Output throughput speedup ratios - all pairwise mode comparisons

• 25W delivers a consistent ~1.43-1.47x speedup vs 15W across all models.
• 15W gives about 2.1-2.7x boost vs 7W and even about 1.2 times on top of it for 25W.
• MAXN/25W < 1 for the smallest models (MAXN gains no throughput) but > 1 for larger models (compute-bound, benefit from clock ceiling). MAXN/7W reaches 4.37x for Llama3.2-1B - the largest speedup in the sweep.
• Speedups involving MAXN are higher for models in the range of 0.5B - 1B parameters, where the GPU clock increase from 820 MHz to 1020 MHz has the most impact before memory bandwidth becomes the bottleneck.

3.5 Latency Characteristics

TTFT scales near-linearly with prompt across all modes. At ctx=128 a model like LFM2.5-350M prefills in ~80 ms (25W); at ctx=2048 that grows to ~820 ms. The 25W / MAXN modes reduce TTFT proportionally to their clock ratio vs 15W.

Inter-token latency (ITL) p50 is the median per-token decode cost. ITL heatmaps per power mode (all 8 models, all 12 prompt×gen combos) are in Appendix H.2 — see Figures H.2a–H.2d. At the canonical ctx=2048, gen=256:

7W	15W
25W	MAXN

Figure 10a: ITL p50 heatmaps — all 4 power modes (rows = gen length, cols = prompt length)

• ITL depends on gen-length (64, 128, 256) and to some extent reflects the memory-bandwidth bound.
• In our case the gen-lengths tested are not enough to cause differences across model × mode combinations beyond the general trend: models <1B have lower ITL than ~1B models, possibly because the KV-cache stays small enough to avoid refills.

Decode time (s) p50 is the time spent generating output tokens: decode_time = request_latency − TTFT. At ctx=2048, gen=256 (computed as RL_s − TTFT_s where RL_s = OSL / tok_s):

Table 10a: Decode time speedup ratios - all pairwise mode comparisons (ctx=2048, gen=256)

Speedup = mean(decode_time_baseline) / mean(decode_time_mode) where decode_time = RL p50 − TTFT p50, averaged over all 12 prompt × gen combos. decode_J = avg_power_W × decode_time_s.

• Decode speedups closely mirror throughput speedups (Table 9) since decode time ≈ 1 / tok_s once TTFT is subtracted.
• MAXN vs 25W > 1.0 for 0.5B–1B models; < 1.0 for SmolLM2 (memory-bandwidth bound, extra clock gives no decode gain).

TTFT speedup - TTFT_baseline / TTFT_mode; values > 1.0 mean the mode has lower TTFT (faster prefill). Speedup = mean(TTFT p50 at baseline) / mean(TTFT p50 at mode), averaged over all 12 prompt × gen combos:

Table 11: TTFT speedup ratios - all pairwise mode comparisons

• MAXN has the highest speedup ratios across all modes, with the largest gains for the bigger models (Qwen3-0.6B, Llama3.2-1B, Gemma3-1B) where the GPU clock increase has more impact.
• MAXN/25W ratios cluster near 1.0x (~0.95–1.11x). Prefill is compute-bound (parallel GEMMs over all input tokens), so a naive expectation would be that higher clocks help proportionally - but this was not the case. Why? maybe it becomes memory-bandwidth bound?(let me know in the comments!). For the two smallest models (SmolLM2) the prefill completes so quickly (<300 ms at 25W) that kernel-launch overhead dominates, making higher clocks irrelevant (0.95–0.97x).

Request latency (E2E) speedup - Speedup = mean(RL p50 at baseline) / mean(RL p50 at mode), averaged over all 12 prompt × gen combos:

Table 12: Request latency (E2E) speedup ratios - all pairwise mode comparisons

• Mirrors the TTFT speedup trends since prefill dominates the request latency at these gen lengths.

3.6 Model Size vs Efficiency

The relationship is clear: smaller quantized models always win on total tok/J, not just tok/s.

Table 13: Best total tok/J ranked by model size

Total tok/J = (ISL + OSL) / (avg_power_W × RL_p50_s) — see Appendix J.6 for the full formula. Peaks at ctx=2048, gen=64 for every model because the long prompt dominates the numerator while 25W minimises energy per token. All 48 mode × ctx × gen combinations were searched.

SmolLM2-135M at 25W achieves 487 total tok/J, nearly 4.5× more efficient than Llama3.2-1B across the full request.

3.7 Energy Efficiency: Decode tok/J and Total tok/J

Two complementary tok/J lenses on energy efficiency — see J.6 for formulas:

• Decode tok/J = OSL / decode_J — output tokens generated per joule of decode energy only (TTFT excluded). Measures how efficiently the GPU runs the autoregressive generation loop.
• Total tok/J = (ISL + OSL) / total_J — all tokens processed per joule of the full request. Accounts for both prompt processing and generation; favours models that handle long prompts cheaply.

See Figure 7b (decode tok/J vs prompt length) and Figure 7c (total tok/J vs prompt length) in section 2.2 — 25W leads at every model and prompt length. Full combinations: F.2 Decode · F.3 Total.

Key findings:

1. 25W wins on both metrics for almost every model. The exception is SmolLM2-360M, where 7W edges ahead on both decode and total tok/J — decode is memory-bandwidth bound for this model and the lower clock still delivers competitive throughput at much lower power.

2. ~1B around models tops at ~5-8 tok/J (decode) whereas the <1B models can reach 15-35 tok/J. Thus these are more energy efficient (decode) than ~1B models we have tested.

3. Charts in F.2 show that the ~1B models are roughly flat, that is, prompt length becomes independent of tok/J in decode tok/J as going from 64 to 256 gen length.

4. Total tok/J grows with prompt length because ISL dominates (ISL+OSL) as ctx increases while total_J grows more slowly (decode time is constant), see F.3.

Figure 15: Total energy per request vs output length at 25W, ctx=2048

Figure 16: Decode energy per output token in mJ (ctx=2048, gen=256)

5. Conclusion

What These Numbers Mean for Edge Inference

Tiny LLM inference on a $250 Jetson Orin Nano Super 8GB is genuinely practical. At SmolLM2-135M Q4_K_M:

• 187 tok/s at 25W : real-time fluent generation
• 101 MB on disk : trivially portable
• 5.4 W under load : runs on a USB-C power bank
• 22.6 output tok/J : the best energy efficiency in this suite

The LFM2.5 models (Liquid AI) are a notable new entrant: LFM2.5-350M achieves 120 tok/s at 25W (competitive with SmolLM2-360M) in 219 MB. LFM2.5-1.2B at 25W (ctx=2048, gen=256):

• Throughput: 54.1 tok/s — 13 % faster than Llama3.2-1B (47.0) and 33 % faster than Gemma3-1B (40.8)
• Output tok/J: 5.26 vs Llama 4.67 (+13 %) vs Gemma 5.14 (+2 %) — clear lead over Llama, narrow over Gemma
• Total tok/J: Gemma edges ahead here (118.5 vs 116.2 vs Llama 108.9) — its lower power draw (6.87 W vs 8.46 W) compensates for slower decode
• Disk footprint: 698 MB vs 771 MB (Llama) / 769 MB (Gemma) — smallest in the class, making it the best tok/s-per-byte overall

The 25W Mode:

25W is the Pareto-optimal power mode for edge LLM inference on the Jetson Orin Nano Super. It is the right answer for virtually every deployment:

• 43 % more throughput than 15W; within −3 % to +8 % of MAXN (MAXN gains only marginally on larger models; 25W wins or ties on sub-500M models)
• Only 36 % more power than 15W; ~17 % less power than MAXN
• 3–26 % better output tok/J than 15W; 8–35 % better output tok/J than MAXN
• Low enough peak power (≤ 10 W for sub-1B models) to stay comfortable for sustained operation

NOTE: CMA fragmentation caveat

• After three sequential model loads in the same OS session, the CUDA IOVA address space accumulates fragmentation that blocks cudaMalloc calls requiring > 300 MB contiguous buffers. Qwen3-0.6B, Llama3.2-1B, Gemma3-1B, and Gemma3-4B all hit NvMapMemAllocInternalTagged: error 12 (ENOMEM) when loaded after other models without a reboot. A reboot + --resume run recovered all three smaller models (Gemma3-4B is OOM at every mode regardless). All 8 non-gemma3-4b models produced valid 7W data after this workaround; the full 96-cell 7W dataset is now complete.

Appendix

A. Full 4-Mode Comparison (ctx=2048, gen=256)

Raw numbers from the canonical benchmark cell. All latencies in milliseconds. Power = VDD_CPU_GPU_CV averaged over each run window.

Table 15: Full 4-mode comparison, ctx=2048, gen=256

B. Thermal Summary - All Power Modes

Power and temperature averaged over each model’s full benchmark window (all 12 prompt×gen combos). No model triggered thermal throttling at any power mode (threshold ≈ 95 °C).

Junction temperature (TJ) is the hottest internal die temperature on the Jetson SoC, reported by tegrastats as tj@. It is the primary metric for thermal safety: if TJ reaches ~95 °C, the hardware automatically throttles clocks to prevent damage. Peak TJ < 70 °C across all runs means thermal headroom is ample.

Table 16: Thermal summary - all power modes

E. Full 12-Combination Heatmaps (All Power Modes)

Each heatmap is a 2×4 grid (8 models) showing all 12 prompt×gen combinations for one power mode and one metric. Rows = gen length (64, 128, 256 tok), columns = prompt length (128, 512, 1024, 2048 tok). Brighter colour = higher value.

E.1 Output Tok/s heatmaps

Figure E.1a: All 12 combos at 7W

Figure E.1b: All 12 combos at 15W

Figure E.1c: All 12 combos at 25W

Figure E.1d: All 12 combos at MAXN

E.2 Output Tok/J heatmaps

Figure E.2a: All 12 combos at 7W

Figure E.2b: All 12 combos at 15W

Figure E.2c: All 12 combos at 25W

Figure E.2d: All 12 combos at MAXN

F. Prefill / Decode / Total tok/J: All Combinations

All charts are 2×4 faceted line plots with a fixed y-scale across all subplots. The canonical combination (ctx=2048, gen=256) is also shown in §2.2.

F.1 Prefill tok/J (input tok / J) vs prompt length

Figure F.1a: Prefill tok/J vs prompt length: gen=64

⚠ Prefill tok/J is approximate for 63 % of cells.

Figure F.1b: Prefill tok/J vs prompt length: gen=128

⚠ Prefill tok/J is approximate for 63 % of cells.

Figure F.1c: Prefill tok/J vs prompt length: gen=256 (canonical, also in § 2.2)

⚠ Prefill tok/J is approximate for 63 % of cells.

F.2 Decode tok/J (output tok / J) - independent of prompt length

Decode tok/J depends on the number of output tokens (gen length), not input prompt length, since decode happens after prefill completes. These charts show decode tok/J as a function of gen length for each prompt context length.

Figure F.2a: Decode tok/J vs gen length: ctx=128

Figure F.2b: Decode tok/J vs gen length: ctx=512

Figure F.2c: Decode tok/J vs gen length: ctx=1024

Figure F.2d: Decode tok/J vs gen length: ctx=2048

F.3 Total tok/J ((input+output) tok / J) vs prompt length

Figure F.3a: Total tok/J vs prompt length: gen=64

Figure F.3b: Total tok/J vs prompt length: gen=128

Figure F.3c: Total tok/J vs prompt length: gen=256 (canonical, also in § 2.2)

G. Request Latency (E2E): All Combinations

Request latency (E2E) p50 - total time from request start to last token received. Line charts show variation with prompt length (2×4 facet, fixed y-scale). Grouped bar charts show per-model × per-mode breakdown.

G.1 Request latency vs prompt length (by gen length)

Figure G.1a: Request latency vs prompt length: gen=64

Figure G.1b: Request latency vs prompt length: gen=128

Figure G.1c: Request latency vs prompt length: gen=256 (canonical, also in §2.3)

G. TTFT: All Prompt x Gen Combinations

TTFT p50 (median time to first token, ms) is driven almost entirely by prompt length, it is the prefill cost. These charts show how it varies across all 12 prompt x gen combinations and across all 4 power modes.

G.1 TTFT vs prompt length (by gen length)

Figure G.1a: TTFT vs prompt length: gen=64

Figure G.1b: TTFT vs prompt length: gen=256 (canonical, also in section 2.3)

TTFT is independent of gen length, so only gen=64 and gen=256 are shown.

G.2 TTFT heatmaps (gen x prompt) per power mode

Each cell is TTFT in ms. Rows = gen length, columns = prompt length. Independent of gen length hence the same across rows.

Figure G.2a: TTFT heatmap: 7W	Figure G.2b: TTFT heatmap: 15W
Figure G.2c: TTFT heatmap: 25W	Figure G.2d: TTFT heatmap: MAXN

H. ITL: All Combinations

Inter-token latency (ms) = time between consecutive output tokens. It measures decode cost and is driven by model size and GPU clock, not prompt length.

H.1 ITL vs prompt length (by gen length)

Figure H.1a: ITL vs prompt length: gen=64

Figure H.1b: ITL vs prompt length: gen=128

Figure H.1c: ITL vs prompt length: gen=256 (canonical, also in section 2.3)

H.2 ITL heatmaps (gen x prompt) per power mode

Figure H.2a: ITL heatmap: 7W	Figure H.2b: ITL heatmap: 15W
Figure H.2c: ITL heatmap: 25W	Figure H.2d: ITL heatmap: MAXN

I. Prefill Throughput: All Combinations

Prefill throughput (tok/s) measures how fast the model processes input tokens. It scales with prompt length (longer prompts hit peak GPU utilisation) and GPU clock speed.

I.1 Prefill throughput vs prompt length (by gen length)

Figure I.1a: Prefill throughput vs prompt length: gen=64

Figure I.1b: Prefill throughput vs prompt length: gen=256 (canonical, also in section 2.4)

Prefill throughput is independent of gen length, so only gen=64 and gen=256 are shown.

I.2 Prefill throughput heatmaps (gen x prompt) per power mode

Figure I.2a: Prefill throughput heatmap: 7W	Figure I.2b: Prefill throughput heatmap: 15W
Figure I.2c: Prefill throughput heatmap: 25W	Figure I.2d: Prefill throughput heatmap: MAXN

J. All Metrics, Formulas, and Calculation Methods

This appendix documents every metric reported in this benchmark, its formula, its source, and any caveats.

J.1 Raw inputs from aiperf and tegrastats

All aiperf metrics are averages over 20 requests per combo. Percentile variants (p50, p90, p99) are also available in the raw JSON but not reproduced here.

J.2 Power

avg_power_W = mean(mW_i for all tegrastats samples where t0 <= sample_time <= t1) / 1000

• VDD_CPU_GPU_CV covers the CPU, GPU, and Computer Vision engine rail
• Does NOT include board overhead (fan, storage, USB) which is on VDD_IN
• VDD_IN is ~1.5-3 W higher than VDD_CPU_GPU_CV during inference
• Tegrastats interval: 500 ms

J.3 Output tok/J (main efficiency metric)

output_tok_J = OSL / (avg_power_W * RL_p50_s)

Where RL_s = RL / 1000 (request latency in seconds).

Higher is better. This measures how many output tokens are generated per joule of compute energy. It is the primary metric of the benchmark.

Not affected by the prefill/decode split approximation (see section J.7).

J.4 Request latency energy

total_J = avg_power_W * (RL / 1000)

Energy consumed by one average request from first byte sent to last token received. Accurate for all cells regardless of TTFT.

J.5 Prefill and decode energy

prefill_J  = avg_power_W * (TTFT / 1000)
decode_J   = avg_power_W * ((RL - TTFT) / 1000)
           = total_J - prefill_J

prefill_%  = prefill_J / total_J * 100

CAUTION: See energy measurement caveat.

J.6 Phase tok/J metrics

prefill_tok_J = ISL / prefill_J
              = ISL / (avg_power_W * TTFT_s)

decode_tok_J  = OSL / decode_J
              = OSL / (avg_power_W * (RL_s - TTFT_s))

total_tok_J   = (ISL + OSL) / total_J
              = (ISL + OSL) / (avg_power_W * RL_s)

Where TTFT_s = TTFT / 1000, RL_s = RL / 1000.

• prefill_tok_J: input tokens processed per joule of prefill energy. Affected by the approximation in J.5.
• decode_tok_J: output tokens generated per joule of decode energy. Reasonably accurate.
• total_tok_J: all tokens (in + out) per joule of total request energy. Accurate.

J.7 mJ per output token

mJ_per_output_tok = (decode_J / OSL) * 1000
                  = 1000 / decode_tok_J

Millijoules per generated output token (decode_J is in joules, ×1000 converts to mJ for readability). Carries the same caveat as J.5 for cells where TTFT < 500 ms.

J.8 Prefill throughput

prefill_tput (tok/s) = aiperf JSON prefill_throughput_per_user.avg

Directly from aiperf. Measures how fast input tokens are processed during the prefill phase. Scales with prompt length (longer prompts hit peak GPU utilisation) and GPU clock.

J.9 Throughput speedup ratios (Table 9)

speedup_25W_vs_15W  = mean(tok_s_25W  over all 12 combos) / mean(tok_s_15W  over all 12 combos)
speedup_MAXN_vs_15W = mean(tok_s_MAXN over all 12 combos) / mean(tok_s_15W  over all 12 combos)
speedup_15W_vs_7W   = mean(tok_s_15W  over all 12 combos) / mean(tok_s_7W   over all 12 combos)

Averages are over all 4 prompt lengths × 3 gen lengths = 12 combos. tok_s = output_token_throughput_per_user.avg (aiperf); no p50 is available for throughput. Latency speedup ratios (Tables 10a, 11, 12) use mean of p50 values instead.

J.10 Best total tok/J per model (Table 13)

best_total_tok_J(model) = max(total_tok_J(mode, model, gen, ctx))
                          over all modes in {7W, 15W, 25W, MAXN}
                          and all gen in {64, 128, 256}
                          and all ctx in {128, 512, 1024, 2048}

total_tok_J = (ISL + OSL) / (avg_power_W * RL_p50_s)

The single highest total tok/J value observed for that model across all 48 combinations. Peaks at ctx=2048, gen=64 for every model because the long prompt dominates the (ISL + OSL) numerator.

J.11 TTFT, ITL, RL percentiles

All percentile variants come directly from aiperf JSON without further computation:

TTFT       = time_to_first_token.p50   (canonical; p50 used everywhere)
TTFT_p90   = time_to_first_token.p90
TTFT_p99   = time_to_first_token.p99
ITL        = inter_token_latency.p50    (canonical; p50 used everywhere)
ITL_p99    = inter_token_latency.p99
RL         = request_latency.p50        (canonical; p50 used everywhere)
RL_p99     = request_latency.p99

J.12 Energy caveat: which metrics are accurate vs approximate

J.13 Power and temperature

avg_power_W = mean(tegrastats.VDD_CPU_GPU_CV[mW] / 1000
              for all samples where aiperf_t0 <= sample_time <= aiperf_t1)

Power is the mean VDD_CPU_GPU_CV (CPU+GPU+CV rail) from tegrastats sampled at 500 ms intervals, averaged over each model’s active inference windows only (idle/cool-down between models excluded).

Junction temperature (TJ) is the hottest internal die temperature on the Jetson SoC, reported by tegrastats as tj@. The hardware automatically throttles GPU/CPU clocks when TJ reaches ~95 °C to prevent damage. Peak TJ < 70 °C across all runs confirms ample thermal headroom at every power mode.

Throttling is flagged when peak_tj_C > 85 °C (leaving a 10 °C safety margin below the hardware limit).