Hey folks, I'm an "old-school" ML programmer from the time of Scikit-Learn / Tensorflow, back when LSTMs roamed the Earth, and all this newfangled talk of agentic AI and self-hosted LLMs has finally caught my attention. I'd like to set up a coding self-host LLM on my older gaming PC, but I'm on a fairly tight budget (can't buy much right now, but might drop a couple hundred on an upgrade if it'd be worth it) and don't know how well things could run on my PC or if it's even worth it right now.
Currently running:
AMD Ryzen 5 5600X 6-core
64GB DDR4 RAM
RTX 3070 TI 8GB
I also have an old nVidia 1070 8GB I could plug into the second PCIe slot, but I don't know how helpful that would be.
The cost of upgrading RAM / GPU seems absolutely bonkers right now.
The reason I'm looking to set this up is I'd be starting up a weekend-project type side business and would need to keep my data confidential from the big-model companies. Even if it takes a minute to load responses while I grab a cup of coffee, that's fine so long as the quality of the output is good.
Advice would be appreciated.