As the use of GenAI models becomes increasingly prevalent, it’s crucial for organizations to ensure the security and ownership of their intellectual property. One way to achieve this is by running open-source GenAI models locally on your own infrastructure.
Why Run Open-Source GenAI Models Locally?
Running open-source GenAI models locally provides several benefits:
- Data Protection: Keep sensitive information within your organization’s control. By processing data locally, you can keep sensitive information from leaving your organization’s premises, reducing the risk of data breaches and intellectual property theft.
- Increased Control: With full control over the deployment environment, you can tailor the setup to meet your specific requirements, ensuring that your models are used as intended.
- Improved Performance: Running GenAI models locally allows for faster processing times and reduced latency, making them more suitable for real-time applications.
- Enhance Security: Reduce the risk of data breaches and intellectual property theft by processing data locally.
Open Source License Requirements
Please check the licensing requirements for the open source model you are going to use as it may be quite different for each model. The license may also be different depending on the type of use (personal, educational, commercial, etc.)
Step-by-Step Process
Follow these three simple steps to install and run multiple LLMs locally.
Step 1: Set up Ollama
Ollama is an open source project that enables you to run large language models (LLMs) locally without going through too much hassle. It is available at GItHub (see references below)
Open terminal window on Macbook and use the following commands:
mkdir llmcd llmcurl -L https://ollama.com/download/ollama-darwin-arm64 -o ./ollamachmod u+x ollama ./ollamaStep 2: Download LLM model you want to run
Go to a new terminal windows and pull llama3
./ollama pull llama3Step 3: Run the Model
Once the model is downloaded, you can run it and use prompts on the command line. Following is a typical session with one prompt and its response:
./ollama run llama3>>> who wrote harry potter?The Harry Potter series was written by J.K. Rowling (Joanne Kathleen Rowling). She is a British author, screenwriter, and
philanthropist best known for creating the beloved Harry Potter fantasy series.
Rowling began writing the first book in the series, “Harry Potter and the Philosopher’s Stone” (published as “Harry Potter
and the Sorcerer’s Stone” in the United States), while she was a student at the University of Exeter. The book was published
in 1997 and became a worldwide bestseller.
The subsequent books in the series were:
1. “Harry Potter and the Chamber of Secrets” (1998)
2. “Harry Potter and the Prisoner of Azkaban” (1999)
3. “Harry Potter and the Goblet of Fire” (2000)
4. “Harry Potter and the Order of the Phoenix” (2003)
5. “Harry Potter and the Half-Blood Prince” (2005)
6. “Harry Potter and the Deathly Hallows” (2007)
The Harry Potter series has since been adapted into a successful film franchise, as well as stage plays, video games, and
other media. J.K. Rowling’s creation has become a cultural phenomenon, inspiring countless fans around the world.
Rowling has also written several companion books to the series, including “Fantastic Beasts and Where to Find Them” (2001)
and “The Tales of Beedle the Bard” (2008). She has also published several novels for adults, including “The Casual Vacancy”
(2012) and “The Cuckoo’s Calling” (2013), under the pseudonym Robert Galbraith.
>>>If you have completed the above three steps, you are on your way to start using LLMs locally on your Macbook. No need to have Internet connectivity or send data to a third party chatbot. Ollama supports many LLMs and the list is available on its Github page. You can “pull” any of these models and test them locally.
Running open-source GenAI models locally provides a secure and controlled environment for developing and deploying AI-powered applications. By maintaining ownership and control over your data, you can protect your intellectual property while still leveraging the benefits of GenAI technology.
In a next blog post, I will discuss using a web interface with Ollama.
References
https://github.com/ollama
https://github.com/ollama/ollama
https://ollama.com/library
