hckrnws

Understanding GPT tokenizers

by simonw

mike_hearn
4m
s-macke
4m
josecyc
3m
sillysaurusx
4m
thyselius
3m
hammeiam
3m
og_kalu
4m
mike_hearn
3m
og_kalu
3m
kordlessagain
3m
base698
4m
kordlessagain
3m
belladoreai
3m
mike_hearn
3m
belladoreai
3m
mike_hearn
3m
belladoreai
3m
hammeiam
3m
belladoreai
3m
belladoreai
3m

Comment was deleted :(

weinzierl
4m
resters
3m
mike_hearn
3m
anotherpaulg
4m
Der_Einzige
4m
simonw
4m
weinzierl
4m
jcims
4m
PeterisP
4m
emmelaich
4m
f137
3m
ftxbro
4m
Der_Einzige
4m
nsinreal
4m
spywaregorilla
4m
ftxbro
4m
spywaregorilla
3m
whimsicalism
4m
yorwba
4m
sp332
4m
jcims
4m
zecg
3m
Der_Einzige
4m
chaxor
4m
bpiche
4m
justanotheratom
4m
LudwigNagasena
4m
ajuc
3m
gwern
4m
pmoriarty
4m
gwern
4m
zarzavat
4m
telotortium
3m
gwern
3m
burnished
4m
Der_Einzige
4m
TechBro8615
4m
jamesk_au
4m
wolfgang42
4m
belladoreai
3m
Majromax
3m
simonw
4m
kgeist
4m
Ambix
3m
szopa
4m
Ambix
3m
jameshart
3m
api
4m
PeterisP
4m
sebzim4500
4m
sanxiyn
3m
belladoreai
3m
sandinmyjoints
4m
seanhunter
3m
simonw
3m
seanhunter
3m
seanhunter
3m
ftxbro
4m
simonw
4m
Ambix
3m
throwaway2016a
4m
nighthawk454
4m
CuriousSkeptic
4m
quickthrower2
4m
nighthawk454
3m
nighthawk454
4m
binarymax
4m
RC_ITR
4m
z3c0
4m
Ambix
3m

Comment was deleted :(

samstave
4m
plewd
4m
washadjeffmad
3m
samstave
4m
hsjqllzlfkf
4m
ftxbro
4m
kenjackson
4m
GeneralMayhem
4m
SkyPuncher
4m
thaumasiotes
4m
wolfgang42
4m
doormatt
4m
hsjqllzlfkf
4m
mike_hearn
4m
hsjqllzlfkf
4m
SkyPuncher
4m
wolfgang42
4m
washadjeffmad
3m
whimsicalism
4m
SkyPuncher
4m
simonw
4m
llm_nerd
3m

Comment was deleted :(

Der_Einzige
4m
isoprophlex
4m
h_mirin
3m
YetAnotherNick
4m
bluepoint
4m
simonw
4m
jiggawatts
4m
sebzim4500
4m
jiggawatts
4m
PeterisP
4m
PeterisP
4m
minimaxir
4m
ywmario
4m
Ambix
3m
pietroppeter
3m
tabtab
4m
jjtheblunt
4m
lee101
3m
buggythebug
4m

Crafted by Rajat

Source Code