I have avoided AI (LLMS) because it didn’t seem nearly useful enough to warrant supporting these tech assholes. Is there anyway to use it locally on a normal device (16 Gb ram, 8 year old Geforce graphics, etc?). Can I just slow it down and get it to work locally?
The LLMs would be of minor use without the ingestion of the Internet to update its weights. It’s a clanker so it has no ideas; you’ve got to feed it whatever you’re purpose is with it.
Use open weight models either small enough to run yourself or via your diverse online host of choice.
without supporting the data centers
online host of choice
I believe there are options to host open models on hardware you don’t own without having to give money to the “evil” data centers. Getting away from data centers entirely is impossible as that is how the internet has worked for a very long time now.
Not to mention that it’s training and not inference where most of the compute is going anyway.
Download Ollama. Download one of the smaller Qwen models. Have at it.
Try qwen3.5:9b (a 7.6Gb model) to start.
Friends don’t let friends use Ollama.
Somethin’ happening with them?
I think from a performance standpoint llamacpp is better, and requires approximately the same amount of technical knowledge to operate.
I donate my compute to https://aihorde.net/ once in a while. got to play around with some cool AI bullshit there. Its fun to play around with.
Its all volunteer run. I got my steam deck pushing small updates to peoples like dnd rollplay kinda stuff. koboldcpp makes it easy to set up and you can use hugging face models or your own.
Theres some really excellent open models out there if you care to look! Most are hyper specialized better than say GPT and such.
I use tinfoil.sh with locally hosted models, you are giving tinfoil money for the service but they aren’t paying the model trainers for open source models nor are they training models with your data
I like deepseek v4flash, I use opencode go and commandcode, somehow feels not as bad since its cheaper and technically can be ran at home if you can afford it. I don’t care for regular ai chat, feels useless after using agents. Agents are what you expect when you think of ai at the computer, hermes is a solid free one. Those two subs rotating as one runs out should last you a while at 20$ a month. These smaller companies/datacenters aren’t the issue imo, the larger ones like Anthropic, google, openai, etc. are.
It just functions so much better to use low cost api subs over trying to run anything locally. Let them subsidize this shit for you, don’t abuse your own hardware while it’s expensive af to replace, can spend 100s a year on subs and it’ll be less than the hardware costs for that amount of time.
Lumo and some on duck.ai seem fine for the free usage I do. Here’s a fun one that appears accurate, enjoy! duck.ai/share/01cee404-a240-4c14-9a83-584d406c5a68#Ch22BaAoEAejJEeRNZYh-MfEYLFoDCsVZsPd23IGU5w
Depends what you’re wanting to use that AI for. Is it figuring out protein chains that might help make a vaccine? Sure. Is it making media at the expense of creators? No thanks. It’s hard to justify the costs of large AI rollout and honestly, from what I’ve read and heard, it’s hamstrung from the power grid.
Death to clankers.
To be honest, I’m not sure how it all works exactly, but I downloaded some models that run locally and i can choose an agent that can run models that will work with my (and your) limited graphics memory. It’s all local and you don’t have any limits or anything and it’s private. No tokens. However, it will run very slowly depending on what you are doing.
I haven’t used it. I tried a couple of times but I was trying to generate an image and apparently it was too much for it and I never tried it again. But maybe that somehow supports rich assholes. I don’t know. I didn’t pay anything and I was under the impression it was open source.
If you want to try, you can go on YouTube and search under something like “running AI locally.” And you should find some tutorials. Good luck.
Personally I have a love/hate relationship with AI. It’s simultaneously amazing and horrifying.
Use it all you want. Don’t listen to people here. They’re fucking stupid. These are the same people who told everyone not to get into bitcoin. Meanwhile all the assholes are millionaires and these people here gave up every space the had to the investors and are still trying to moral police others.
It’s here and isn’t going away. You’re not a hero for not using it. Learn it like any other tool. Get good with it. Use it to better your life unlike people here who will continue to lose power, territory, success and try to drag you down with them.
Simply: don’t be a dick and don’t use AI. Using AI you’re commiting theft.
Such a drama queen. You don’t commit theft.
Don’t use the camera on your phone, it steals your soul!
If you are not using AI, you are getting used by AI.
Good. Nobody should have we’ve capitalized on the internet. It wasn’t a space to sell onlyfans and subscribe to comics about cats. Those people opened the door for every single capitalist who ruined every other space the ever existed. Turned it into a privacy hellscape that wants a picture of your asshole every time to want to sign up to your favorite artist weekly chance to sell you ads.
Fuck them
Burn it all to the ground. Steal everything. Share everything until people trying to capitalize online fuck off.
No, and why do you want to rot your brain and/or produce more slop anyway?
Because LLMs now work better for searches than commercial search engines.
At producing misinformation.
Why would you want misinformation? It’s going to lower your productivity, because you then have to fact check everything it tells you
Ha! No they don’t! It gives you a hallucinated answer that’s not even on the webpage that it claims it got the answer from.
Man you can absolutely run something on ollama yourself. Right now I have ollama running qwen 3:4b-instruct on 6vCPU and no GPU and for very simple conversations it’s quick enough. TTFT is slow when prompt is uncached but otherwise TTFT is like 2s and I get maybe 5 words/s out.
If I gave it my 1660 I could probably get ~100tps but my card won’t pass through to my VM properly…
I’ll just leave this here. https://sleepingrobots.com/dreams/stop-using-ollama/
So… I’m with you/them about the vendor lock in but I think they’re slightly underestimating what ollama brings to the table. For example, ollama bundles CUDA libraries so I don’t need to worry about that as I wrangle with my GPU. It’s also well supported by other ecosystems like home assistant in a way that raw openai compatible API server isn’t.
Overall I don’t know if I find these arguments compelling at this time.
Reading through it I see some good points but… What’s the alternative? Isn’t Ollama still the least of evils (by far)?
Just use llama-server. I quickly found that all wrappers just make life harder and they made some great progress to improve usability with stuff like router-mode
ollama
Shittiest local inference engine. Use llama.cpp instead.
qwen 3:4b-instruct
Old version 3 model from Alibaba. 4 billion parameters total. It will struggle with anything other than natural language generation.
on 6vCPU and no GPU
Running on a 6 vcore VM with no GPU pass through.
TTFT
Time.To.First.Token. How long it takes the model to start responding after you hit enter. Dependent on not only processing speed but also context length.
prompt is uncached
Cold start = no cache = slow. But faster when mid conversation and the key-value cache is hot.
100tps
100 tokens per second. 1 token is roughly 1/4 word.
