Run TinyLlama (AI Chat) off-line on any computer

Simple guide for running very small, open-source LLaMa model for chat.

It's small and fast. Runs on CPU and has very low memory footprint.

Download and Install

• Download GPT4ALL
• Download
  • Official: ggml-model-q4_0.gguf (607MB), or
  • TheBloke: tinyllama-1.1b-chat-v1.0.Q5_K_M.gguf (746MB) Recommend!
  • Install GPT4ALL
  • Run GPT4ALL and close the program (for files initialization)
  • Move model file (.gguf) to your home directory

    C:\Users\USERNAME\AppData\Local\nomic.ai\GPT4ALL\

  Setup

  Run GPT4ALL, go to setting and adjust System Prompt and Template as follows.

  System Prompt

  <|system|>
  
  You are a friendly chatbot who is helpful and help as much as you can to the user request.
  
  <|user|>
  <|assistant|>
          

  Template

  <|user|>
  %1
  <|assistant|>
          

  

  Have Fun!

  

  ---

  Back to home