Scaling laws and a first look inside a model
This is a read-only rendering. To run it: open in Google Colab, or download the .ipynb and upload it yourself.
The cells have no saved outputs: you run them and keep your own. Back to the lab page.
African Technical AI Safety · Week 1, Session 2.5
This notebook works through the three parts of the lab:
Everything runs on free Colab; no GPU is needed, though a free GPU runtime makes part 3 quicker.
The code is written for you. Your work is to run it, poke at it, and say what you found: each part ends with an Explore cell giving you two or three things to change and re-run, and one short question to answer. Changing a number and watching the answer move is the point of the lab.
What you submit: this notebook, run end to end, with your answers typed into the three Explore cells.
Nothing here is fragile. If you break a cell, Runtime → Restart session and run from the top.
# Run once per Colab session (~1 min). Skip the install if running locally with the packages already present.
import importlib.util, sys
IN_COLAB = 'google.colab' in sys.modules
if importlib.util.find_spec('transformer_lens') is None:
%pip install -q transformer_lens
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (7, 4.5)
plt.rcParams['axes.grid'] = True
plt.rcParams['grid.alpha'] = 0.3
print('Colab:', IN_COLAB)
Kaplan et al. (2020) report that test loss falls as a power law in compute:
\[L(C_{\min}) = \left(\frac{C_c^{\min}}{C_{\min}}\right)^{\alpha_C^{\min}}, \qquad \alpha_C^{\min} \approx 0.050, \quad C_c^{\min} \approx 3.1 \times 10^8 \ \text{PF-days}\]
Taking logs turns that into a straight line, which is why the fit below is ordinary least squares:
\[\log L = a - \alpha \log C\]
Kaplan states \(C_c\) in PF-days and the data below is in FLOP (1 PF-day = \(8.64 \times 10^{19}\) FLOP). That changes the intercept \(a\) but not the slope, so \(\alpha\) is comparable across the two and the constant is not.
Your job is to recover \(\alpha\) from data and see how small it is.
# Real runs: Cerebras-GPT, five compute-optimal models trained on the Pile.
# Parameters and tokens from Table 1, Pile test loss from Table 3 of arXiv:2304.03208.
COURSE = ('https://raw.githubusercontent.com/shocklab/'
'African-Technical-AI-Safety-Course/main/docs/labs/data/')
CSV_PATH = None # set this to a filename to use a CSV of your own instead
ALPHA_KAPLAN = 0.050 # Kaplan et al. (2020), eq. 1.3, their alpha_C^min
try:
df = pd.read_csv(CSV_PATH or COURSE + 'cerebras-gpt-scaling.csv', comment='#')
# Compute is not in the file on purpose. The standard estimate for a transformer
# is C = 6ND FLOP: roughly 2 FLOP per parameter per token forward, 4 back.
# float, not int: 6ND overflows a 64-bit integer above ~9.2e18 and wraps negative
compute = 6.0 * df['parameters'].to_numpy(float) * df['training_tokens'].to_numpy(float)
loss = df['pile_test_loss'].to_numpy()
source = CSV_PATH or 'Cerebras-GPT, measured runs (arXiv:2304.03208)'
except Exception as e:
# Offline fallback: generate points FROM Kaplan's law. Recovering alpha from
# these tests your fitting code and nothing else, since the law made the data.
print('Could not load the CSV, falling back to synthetic points:', e)
rng = np.random.default_rng(0)
compute = np.logspace(18, 21, 18) # FLOP
pf_days = compute / 8.64e19 # 1 PF-day = 8.64e19 FLOP
loss = (3.1e8 / pf_days) ** ALPHA_KAPLAN * np.exp(rng.normal(0, 0.015, compute.size))
source = 'synthetic, generated from Kaplan et al. eq. 1.3'
print(f'{len(compute)} points from {source}')
print(f'compute spans {compute.min():.2e} to {compute.max():.2e} FLOP')
print(f'loss falls from {loss.max():.3f} to {loss.min():.3f}')
# Plot on log-log axes: a power law is a straight line here, and nowhere else.
fig, ax = plt.subplots()
ax.loglog(compute, loss, 'o', color='#2a5298')
ax.set_xlabel('Training compute $C$ (FLOP)')
ax.set_ylabel('Test loss $L$')
ax.set_title('Loss against compute, log–log')
plt.show()
# Least-squares fit of log10(L) = a - alpha * log10(C).
slope, intercept = np.polyfit(np.log10(compute), np.log10(loss), 1)
alpha = -slope
print(f'fitted exponent alpha = {alpha:.4f}')
print(f'Kaplan et al. = {ALPHA_KAPLAN:.4f} (their alpha_C^min)')
print(f'for reference, their other two exponents: alpha_N ~ 0.076, alpha_D ~ 0.095')
print()
print('Five real runs from one lab, spanning under three orders of magnitude, landing')
print('near Kaplan\'s exponent measured on different models and different data. Treat that')
print('agreement carefully: both fits are on raw loss over a similar compute range, and raw')
print('loss carries a floor it cannot fall below. The Explore cell separates the two.')
print()
print(f'A 10x increase in compute multiplies the loss by 10^-alpha = {10 ** -alpha:.3f},')
print(f'i.e. it buys a {100 * (1 - 10 ** -alpha):.1f}% reduction. The exponent is small: this is')
print('why the curve is so punishing, and why frontier runs cost what they do.')
fig, ax = plt.subplots()
ax.loglog(compute, loss, 'o', color='#2a5298', label='data')
grid = np.logspace(np.log10(compute.min()), np.log10(compute.max()), 100)
ax.loglog(grid, 10 ** (intercept + slope * np.log10(grid)), '-', color='#003A70',
label=f'fit: slope $-{alpha:.3f}$')
ax.set_xlabel('Training compute $C$ (FLOP)'); ax.set_ylabel('Test loss $L$')
ax.legend(); ax.set_title('Fitted power law')
plt.show()
# The same authors fitted their own frontier and published it (eq. 1 of the paper):
# L(f) = (f / 5.984e22)^-0.0737 + 0.5066
# Their exponent is 0.0737, not the ~0.054 you just fitted. Same runs, different answer.
their = (compute / 5.984e22) ** -0.0737 + 0.5066
print('compute measured their law difference')
for c, l, t in zip(compute, loss, their):
print(f'{c:.2e} {l:.3f} {t:.3f} {t - l:+.3f}')
print()
print('The gap between 0.054 and 0.0737 is not a disagreement about the data. It is a')
print('disagreement about the shape: they fit L = E + (C/Cc)^-alpha, with a floor E = 0.507,')
print('while you fit a pure power law with no floor. Fitting a floor needs more points than')
print('the five here: try it and the three parameters trade off against each other freely.')
print()
print('So \'the\' scaling exponent is not a property of the data alone. It depends on the')
print('functional form you assumed before you started, which is worth remembering the next')
print('time you read a headline exponent anywhere, including in this course.')
# Extrapolate one order of magnitude past the largest run in the data.
biggest = np.argmax(compute) # not [-1]: a CSV need not arrive sorted by compute
C_far = compute[biggest] * 10
L_far = 10 ** (intercept + slope * np.log10(C_far))
print(f'largest run in the data: C = {compute[biggest]:.2e} FLOP, L = {loss[biggest]:.3f}')
print(f'extrapolated one OOM out: C = {C_far:.2e} FLOP, L = {L_far:.3f}')
print(f'predicted improvement: {100 * (1 - L_far / loss[biggest]):.1f}%')
Five points from one lab is a thin basis for a law. The cell below loads a much larger set: 245 (compute, loss) points from the real training runs behind Chinchilla (Hoffmann et al., 2022), extracted from that paper's Figure 4 by Epoch AI. Two things change when you look at a cloud rather than a handful of points.
First, the law is the frontier of the cloud, not a line through the middle of it. Each compute budget was spent several ways, on a bigger model with less data or a smaller model with more, and only the best of those is on the curve the law describes.
Second, the loss cannot fall to zero. Language has entropy the model can never predict away, so a real curve is \(L(C) = E + (C_c/C)^{\alpha}\) with an irreducible \(E\). Chinchilla estimates \(E = 1.69\).
Things to try:
SUBTRACT_E = True. The exponent jumps from about 0.05 to about 0.16. Which is the honest description of how fast the reducible part of the loss falls?E to 1.5 or 1.9 and watch the fitted exponent move. How confident can you be in a number that depends this much on a constant someone else estimated?BINS to 8 or 25. Does the envelope hold up when you slice the compute axis differently?Question. In one sentence: why should you distrust your own extrapolation? Session 2.4 is where to look for what a smooth loss curve fails to tell you about the capabilities that appear along it, and 2.3 for the cautionary case: what happened to Kaplan's compute-optimal advice when Chinchilla re-ran the experiment more carefully.
Your answer:
CHINCHILLA = ('https://raw.githubusercontent.com/epoch-research/'
'analyzing-chinchilla/main/data/svg_extracted_data.csv')
SUBTRACT_E = False # True: fit the reducible part of the loss, L - E
E = 1.69 # Chinchilla's irreducible loss (Hoffmann et al., 2022)
BINS = 18 # compute bins used to trace the frontier
try:
ch = pd.read_csv(CHINCHILLA)
except Exception as err:
ch = None
print('Could not load the Chinchilla points:', err)
if ch is not None:
cC, cL = ch['Training FLOP'].to_numpy(), ch['loss'].to_numpy()
print(f'{len(cC)} runs spanning {np.log10(cC.max() / cC.min()):.1f} orders of magnitude')
# the frontier: the lowest loss anyone achieved at each compute budget
edges = np.logspace(np.log10(cC.min()), np.log10(cC.max()), BINS)
which = np.digitize(cC, edges)
envC = np.array([np.median(cC[which == k]) for k in range(1, BINS + 1) if (which == k).sum() >= 3])
envL = np.array([cL[which == k].min() for k in range(1, BINS + 1) if (which == k).sum() >= 3])
target = envL - E if SUBTRACT_E else envL
ok = target > 0
b_cloud, _ = np.polyfit(np.log10(cC), np.log10(cL), 1)
b_env, a_env = np.polyfit(np.log10(envC[ok]), np.log10(target[ok]), 1)
print(f'fit through the whole cloud: alpha = {-b_cloud:.4f}')
print(f'fit along the frontier: alpha = {-b_env:.4f}'
f'{" (of L - E)" if SUBTRACT_E else ""}')
print(f'your Cerebras fit, for comparison: alpha = {alpha:.4f}')
print(f'Kaplan et al.: alpha = {ALPHA_KAPLAN:.4f}')
print()
print(f'lowest loss anyone reached here: {cL.min():.3f}, against an irreducible {E}')
fig, ax = plt.subplots(figsize=(7.5, 5))
ax.scatter(cC, cL, s=10, alpha=0.35, color='#7f9ec4', label='individual runs')
ax.scatter(envC, envL, s=34, color='#003A70', label='frontier (best per budget)')
gridc = np.logspace(np.log10(envC.min()), np.log10(envC.max()), 100)
fitted = 10 ** (a_env + b_env * np.log10(gridc)) + (E if SUBTRACT_E else 0)
ax.plot(gridc, fitted, color='#c0392b', label=f'frontier fit, alpha = {-b_env:.3f}')
ax.axhline(E, ls=':', color='#555', label=f'irreducible loss E = {E}')
ax.set_xscale('log'); ax.set_yscale('log')
ax.set_xlabel('Training compute $C$ (FLOP)'); ax.set_ylabel('Test loss $L$')
ax.set_title('Chinchilla runs: the law is the frontier, not the cloud')
ax.legend(); plt.show()
Epoch AI maintain a public dataset of notable AI models with, among much else, a publication date and an estimate of training compute. We plot compute against date on a log axis and read off a doubling time.
Note what this measurement is and is not. Training compute is an input, not a capability, and the estimates are reconstructed from papers and reports of varying candour.
EPOCH_CSV = 'https://epoch.ai/data/notable_ai_models.csv'
try:
models = pd.read_csv(EPOCH_CSV, low_memory=False)
print(f'loaded {len(models)} rows from epoch.ai')
except Exception as e:
print('Download failed:', e)
print('Fetch the CSV by hand from https://epoch.ai/data/notable-ai-models')
print('("Download the data in CSV"), upload it to this session, and read it here.')
models = None
if models is not None:
print('columns we need:', [c for c in models.columns if c in
('Model', 'Publication date', 'Training compute (FLOP)')])
# Keep the rows that have both a date and a compute estimate.
d = models[['Model', 'Publication date', 'Training compute (FLOP)']].dropna().copy()
d['Publication date'] = pd.to_datetime(d['Publication date'], errors='coerce')
d = d.dropna().sort_values('Publication date')
START = '2010-01-01' # the deep-learning era; try 2018 or 2020 and watch the answer move
era = d[d['Publication date'] >= START]
print(f'{len(era)} models from {START} onwards, out of {len(d)} with usable data')
print(f'earliest: {era.iloc[0]["Model"]} ({era.iloc[0]["Publication date"].date()})')
print(f'latest: {era.iloc[-1]["Model"]} ({era.iloc[-1]["Publication date"].date()})')
# Fit log10(compute) against time in years, then convert the slope to a doubling time.
years = (era['Publication date'] - pd.Timestamp(START)).dt.days / 365.25
logC = np.log10(era['Training compute (FLOP)'])
slope_yr, icept_yr = np.polyfit(years, logC, 1)
doubling_months = np.log10(2) / slope_yr * 12
print(f'slope: {slope_yr:.3f} orders of magnitude per year')
print(f'doubling time: {doubling_months:.1f} months')
print(f'that is {10 ** slope_yr:.1f}x per year')
fig, ax = plt.subplots(figsize=(8, 5))
ax.scatter(era['Publication date'], era['Training compute (FLOP)'], s=14, alpha=0.55,
color='#2a5298', label='notable models')
line_x = pd.to_datetime([era['Publication date'].min(), era['Publication date'].max()])
line_years = (line_x - pd.Timestamp(START)).days / 365.25
ax.plot(line_x, 10 ** (icept_yr + slope_yr * line_years), color='#003A70',
label=f'fit: doubling every {doubling_months:.1f} months')
ax.set_yscale('log'); ax.set_xlabel('Publication date'); ax.set_ylabel('Training compute (FLOP)')
ax.set_title(f'Training compute of notable AI models, {START[:4]} onwards')
ax.legend(); plt.show()
# The fit is not the whole story: label the models at the top of the range and see
# how much of the trend rests on a handful of frontier runs.
top = era.nlargest(8, 'Training compute (FLOP)')[['Model', 'Publication date', 'Training compute (FLOP)']]
top['Publication date'] = top['Publication date'].dt.date
print(top.to_string(index=False))
The cell below runs the same fit over several starting years, so you can see the answer move without editing anything. Run it, then read the spread.
Things to try:
2023 to WINDOWS. Does a shorter, more recent window give a more trustworthy number or just a noisier one?TOP_ONLY = True to keep only the largest run in each calendar quarter, i.e. the frontier rather than the field. Which trend is the one people quote?Question. Two sentences. First, which quantity you actually measured, precisely: not "AI progress" but the thing on the y-axis and how it was estimated. Second, over what window, and what would change your answer. This is the Session 1.4 checklist turned on a plot you made yourself, which is harder than turning it on someone else's.
Your answers:
WINDOWS = [2010, 2015, 2018, 2020]
TOP_ONLY = False # True: keep only the largest run per quarter (the frontier, not the field)
source = d.copy()
if TOP_ONLY:
q = source['Publication date'].dt.to_period('Q')
source = source.loc[source.groupby(q)['Training compute (FLOP)'].idxmax()]
for start in WINDOWS:
w = source[source['Publication date'] >= f'{start}-01-01']
yrs = (w['Publication date'] - pd.Timestamp(f'{start}-01-01')).dt.days / 365.25
s, _ = np.polyfit(yrs, np.log10(w['Training compute (FLOP)']), 1)
print(f'from {start}: {len(w):>4} models | {s:.2f} OOM/year | doubling every {np.log10(2) / s * 12:.1f} months')
First contact with the residual-stream picture from 2.2, on a real model. GPT-2 small: 12 layers, 12 heads, \(d_{\text{model}} = 768\).
The first cell downloads the weights (about 500 MB) and takes a minute or two. CPU is fine.
You will probably see a deprecation warning about from_pretrained and a note about
unauthenticated Hugging Face requests. Both are harmless: the model loads either way.
import torch
from transformer_lens import HookedTransformer
model = HookedTransformer.from_pretrained('gpt2')
print(f'layers: {model.cfg.n_layers} | heads/layer: {model.cfg.n_heads} | '
f'd_model: {model.cfg.d_model} | vocab: {model.cfg.d_vocab}')
# Feed it some prompts and read off the top predicted next tokens.
prompts = [
'The Eiffel Tower is in the city of',
'The capital city of South Africa is',
'The largest city in Africa is',
'The answer to 17 plus 25 is',
]
for p in prompts:
logits = model(p) # [batch, position, vocab]
probs = logits[0, -1].softmax(dim=-1) # distribution over the NEXT token only
top_p, top_i = probs.topk(5)
print(f'\n{p!r}')
for prob, idx in zip(top_p, top_i):
print(f' {model.to_string(idx.item())!r:>16} {prob.item():.3f}')
print(f' entropy: {-(probs * probs.log()).sum().item():.2f} nats')
# Before believing any of that: TransformerLens prepends a beginning-of-text token to your
# prompt by default. It is the '<|endoftext|>' you saw in the token list. Take it away.
text = 'The Eiffel Tower is in the city of'
context = 'I visited France last summer. ' + text
for label, prompt, prepend in [('bare, token prepended', text, True),
('bare, prompt alone ', text, False),
('with context ', context, True)]:
toks = model.to_tokens(prompt, prepend_bos=prepend)
probs = model(toks)[0, -1].softmax(dim=-1)
paris = probs[model.to_single_token(' Paris')].item()
london = probs[model.to_single_token(' London')].item()
print(f'{label}: Paris {paris:.4f} London {london:.4f} -> '
f'{chr(39) + "London" + chr(39) if london > paris else chr(39) + "Paris" + chr(39)}')
print()
print()
print('The bare prompt flips on a token you never typed. One ordinary sentence of context')
print('settles it: the model is not ignorant about the Eiffel Tower, the bare prompt was.')
print('A claim about what a model knows turned on setup nobody chose deliberately, which')
print('is the evaluation problem of Session 11 arriving early.')
Where is it confident and where is it not? Entropy is the number to watch: a peaked distribution (low entropy) means the model has effectively decided. Note that confidence and correctness are different things, which is the whole reason Session 1.4 has a checklist.
# run_with_cache captures every intermediate activation.
text = 'The Eiffel Tower is in the city of'
logits, cache = model.run_with_cache(text)
resid = cache['resid_post', 0] # residual stream after block 0
print(f'residual stream shape: {tuple(resid.shape)} # [batch, position, d_model]')
print(f'tokens: {model.to_str_tokens(text)}')
print()
print('This is the T x d matrix from 2.2: one row per token position, one column per')
print('residual-stream dimension. Every block reads from it and writes back into it.')
print()
print('a few other things the cache holds:')
for key in ['blocks.0.attn.hook_pattern', 'blocks.0.hook_mlp_out', 'ln_final.hook_normalized']:
if key in cache:
print(f' {key:<34} {tuple(cache[key].shape)}')
else:
print(f' {key:<34} (not in this build; try list(cache.keys())[:20])')
print()
print('attention pattern is [batch, head, query_pos, key_pos]: for each head, how much')
print('each token attends to each earlier token.')
# How the residual stream grows as it passes through the blocks.
norms = [cache['resid_post', l][0].norm(dim=-1).mean().item() for l in range(model.cfg.n_layers)]
fig, ax = plt.subplots()
ax.plot(range(model.cfg.n_layers), norms, 'o-', color='#2a5298')
ax.set_xlabel('block'); ax.set_ylabel('mean residual-stream norm')
ax.set_title('The residual stream accumulates as it goes')
plt.show()
GPT-2's tokenizer is a byte-pair-encoding vocabulary of 50,257 tokens fitted mostly to English text. The same sentence in a low-resource language is therefore chopped into more pieces.
To measure this honestly we need parallel text: the same meaning in both languages. We use MAFAND-MT, the Masakhane news translation dataset (Adelani et al., 2022; dataset CC BY-NC 4.0), downloaded from the source rather than reproduced here.
MAFAND = 'https://raw.githubusercontent.com/masakhane-io/lafand-mt/main/data/tsv_files/en-{lang}/dev.tsv'
try:
pairs = pd.read_csv(MAFAND.format(lang='zul'), sep='\t').dropna()
print(f'{len(pairs)} parallel English–isiZulu sentence pairs')
example = pairs.iloc[1]
except Exception as e:
print('Download failed:', e)
print('Substitute your own sentence pair below: any language you know, same meaning in both.')
pairs, example = None, {'en': 'Type an English sentence here.',
'zul': 'Type the same sentence in your language here.'}
for lang, sentence in [('English', example['en']), ('isiZulu', example['zul'])]:
toks = model.to_str_tokens(sentence)
print(f'\n{lang}: {sentence}')
print(f' {len(toks)} tokens, {len(sentence)} characters')
print(f' {toks}')
# Across many sentence pairs, not one: the ratio is the number that matters.
if pairs is not None:
sample = pairs.head(200)
en_tok = sample['en'].apply(lambda s: len(model.to_str_tokens(s)))
zu_tok = sample['zul'].apply(lambda s: len(model.to_str_tokens(s)))
en_ch, zu_ch = sample['en'].str.len(), sample['zul'].str.len()
print(f'over {len(sample)} parallel sentences:')
print(f' mean tokens, English: {en_tok.mean():.1f}')
print(f' mean tokens, isiZulu: {zu_tok.mean():.1f}')
print(f' token ratio (isiZulu / English): {zu_tok.mean() / en_tok.mean():.2f}x')
print()
print('Characters, to check the ratio is not just longer words:')
print(f' tokens per 100 chars, English: {100 * en_tok.sum() / en_ch.sum():.1f}')
print(f' tokens per 100 chars, isiZulu: {100 * zu_tok.sum() / zu_ch.sum():.1f}')
fig, ax = plt.subplots()
ax.hist(zu_tok / en_tok, bins=30, color='#2a5298', alpha=0.85)
ax.axvline(1.0, color='#555', ls='--', label='parity')
ax.set_xlabel('tokens in isiZulu ÷ tokens in English, per sentence pair')
ax.set_ylabel('sentence pairs'); ax.legend()
ax.set_title('The same meaning costs more tokens in isiZulu')
plt.show()
# Optional: the same measurement across several African languages.
# MAFAND covers amh, hau, ibo, kin, lug, luo, nya, pcm, sna, swa, tsn, twi, xho, yor, zul.
LANGS = ['zul', 'xho', 'yor', 'swa', 'hau']
rows = []
for lang in LANGS:
try:
p = pd.read_csv(MAFAND.format(lang=lang), sep='\t').dropna().head(150)
col = [c for c in p.columns if c != 'en'][0]
e = p['en'].apply(lambda s: len(model.to_str_tokens(s))).mean()
o = p[col].apply(lambda s: len(model.to_str_tokens(s))).mean()
rows.append({'language': lang, 'en tokens': round(e, 1),
'lang tokens': round(o, 1), 'ratio': round(o / e, 2)})
except Exception as err:
print(f'{lang}: skipped ({err})')
if rows:
print(pd.DataFrame(rows).sort_values('ratio', ascending=False).to_string(index=False))
Put your own text through the model in the cell below: prompts it should find easy or hard, and a sentence pair in a language you speak.
Things to try:
'amh' (Amharic, a non-Latin script) to LANGS and compare its ratio with the Latin-script languages.Question. A short paragraph on what your ratio implies, covering cost (APIs bill per token, and context windows are counted in tokens), quality (the model sees the language in smaller, less meaningful pieces), and safety (if a language is under-represented enough to tokenise badly, what does that predict about how much safety training and red-teaming it received?). Sessions 9 and 18 return to this.
Your answer:
MY_PROMPTS = [
'The capital city of Lesotho is',
'Ubuntu is a philosophy that says',
]
# Defaults are a real pair from the dataset above; replace them with your own.
MY_ENGLISH = pairs.iloc[7]['en'] if pairs is not None else 'Type an English sentence here.'
MY_OTHER = pairs.iloc[7]['zul'] if pairs is not None else 'Type the same sentence in your language.'
for p in MY_PROMPTS:
probs = model(p)[0, -1].softmax(dim=-1)
top_p, top_i = probs.topk(3)
tops = ', '.join(f'{model.to_string(i.item())!r} {v:.3f}' for v, i in zip(top_p, top_i))
print(f'{p!r}\n {tops}\n')
for label, s in [('English', MY_ENGLISH), ('yours', MY_OTHER)]:
t = model.to_str_tokens(s)
print(f'{label:>8}: {len(t):>3} tokens for {len(s):>3} characters {t}')
Save the notebook with its outputs intact (Colab: File → Download → .ipynb) and submit it. Graded on completion and correctness; resubmission is allowed, because the point is mastery rather than one-shot performance.
Sources used here. Kaplan et al. (2020), arXiv:2001.08361, for the power law and its exponents. Epoch AI for the compute trend data. TransformerLens (MIT) for model access. MAFAND-MT (Adelani et al., 2022; CC BY-NC 4.0) for the parallel sentences.