• keskarnitish's avatar
    Adding CTRL (squashed commit) · dbed1c5d
    keskarnitish authored
    adding conversion script
    
    adding first draft of modeling & tokenization
    
    adding placeholder for test files
    
    bunch of changes
    
    registering the tokenizer/model/etc
    
    tests
    
    change link; something is very VERY wrong here
    
    weird end-of-word thingy going on
    
    i think the tokenization works now ; wrote the unit tests
    
    overall structure works;load w next
    
    the monster is alive!
    
    works after some cleanup as well
    
    adding emacs autosave to gitignore
    
    currently only supporting the 48 layer one; seems to infer fine on my macbook
    
    cleanup
    
    fixing some documentation
    
    fixing some documentation
    
    tests passing?
    
    now works on CUDA also
    
    adding greedy?
    
    adding greedy sampling
    
    works well
    dbed1c5d