Associate
- Joined
- 18 Dec 2008
- Posts
- 568
I have been using AI constantly over the past year helping me learn Linux and it is truly amazing. I mainly use the MS Coplot Linux app or Duck Duck Go's AI features between them and can usually point me in the right direction whenever I hit a wall.
I wanted to move away from online AI as my questions and answers are probably being data mined by Microsoft as they do with everything these days.
Since I have a RTX4090 with tensor cores doing nothing for years I dipped my toes into running a LLM locally on my PC.
It turns out that it is really easy to do. Download the programme LM Studio choose the model you what to use from a huge list, and you are done, just ask away. I chose Meta Llama 3.1 70B Instruct as it fits perfectly within the VRAM of my 4090
While it’s not as fast as the online variants it has so far been able to answer all my questions with a high level of technical accuracy and excellent at suggesting possible fixes for issues I encountered.
Best of all its completely free!
I wanted to move away from online AI as my questions and answers are probably being data mined by Microsoft as they do with everything these days.
Since I have a RTX4090 with tensor cores doing nothing for years I dipped my toes into running a LLM locally on my PC.
It turns out that it is really easy to do. Download the programme LM Studio choose the model you what to use from a huge list, and you are done, just ask away. I chose Meta Llama 3.1 70B Instruct as it fits perfectly within the VRAM of my 4090
While it’s not as fast as the online variants it has so far been able to answer all my questions with a high level of technical accuracy and excellent at suggesting possible fixes for issues I encountered.
Best of all its completely free!