Dont give up you got this! I had luck with using vulkan in kobold.cpp as a substitute for rocm with my amd rx 580 card.

Your primary gaming desktop gpu will be best bet for running models. First check your card for exact information more vram the better. Nvidia is preferred but AMD cards work.

First you can play with llamafiles to just get started no fuss no muss download them and follow the quickstart to run as app.

Once you get it running learn the ropes a little and want some more like better performance or latest models then you can spend some time installing and running kobold.cpp with cublas for nvidia or vulcan for amd to offload layers onto the GPU.

If you have linux you can boot into CLI environment to save some vram.

Connect with program using your phone pi or other PC through local IP and open port.

In theory you can use all your devices in distributed interfacing like exo.

First you need to get a program that reads and runs the models. If you are an absolute newbie who doesn’t understand anything technical your best bet is llamafiles. They are extremely simple to run just download and follow the quickstart guide to start it like a application They recommend llava model you can choose from several prepackaged ones. I like mistral models.

Then once you get into it and start wanting to run things more optimized and offloaded on a GPU you can spend a day trying to setup kobold.cpp.

They both start a local server you can point your phone or other computer on WiFi network to it with local ip address and port forward for access on phone data.

Local LLM gang represent! ✌️

Local LLMs aren’t perfect but are getting more usable. There’s abliterated models and uncensored fine tunes to choose from if you don’t like your LLM rejecting your questions.

To see if it can do it and how accurate its general knowledge is compared to the real data. A locally hosted LLM doesnt leak private data to the internet.

Most webpages and reddit post in search results are themselves full of LLM generated slop now. At this stage of the internet if your gonna consume slop one way or the other it might as well be on your own terms by self hosting an open weights open license LLM that can directly retrieve information from fact databases like wolframalpha, Wikipedia, world factbook, ect through RAG. Its never going to be perfect but its getting better every year.

Redditor chud behavior: gets bothered by a single downvote, edits comment to ask why they were down voted turning into a rant about the NPCs and sheeples who disagree with them. Subconsciously worried about karma points and awards.

Lemming chad behavior: Gets 20 upvotes and 10 down votes, happy 30 people reads their thing and glad to have put a controversial opinion into the world that might make someone think a new way.

When will you learn? When will you learn that your choices in process management software have consequences?

thanks for sharing it was a great watch! Here’s one of my favorite gen alpha branrot speech parody videos

https://m.youtube.com/watch?v=o-n5qVwn-Xc&pp=ygUOY29sdW1ibyBnaWFubmk%3D

I see your five paragraph long greentext post about forming pacts with deities from pagan occultism to charge your J.O crystals with long forgotten rituals requiring blessed lube on /x/, and raise the ante by sharing six paragraphs of my personal tales using heroic doses of magic mushrooms to experience ego death, astral project, and telepathically communicate with divine cosmic consciousness using trees as a medium!

F in chat for the invidious/newpipe bot that spammed every comment that dared to link a YouTube video directly. Still think of you sometimes, you were a real artificial one.

F

Very weird fun fact about arrows/darts and statistics, theres 0% chance of hitting an exact bullseye. You can hit it its possible to throw a perfect bullseye. It just has a probability of zero when mathematically analyzed due to being an infinitesimally small point. Sound like I’m making shit up? Here’s the sauce

How can an outcome both be entirely possible and have 0% probability?

Q.E.D

Qwen is super powerful but the CCP endorsed lobodomy to censor it make it useless for my needs. Mistral 22B > qwen 32B all day any day just because it won’t shriek at me in rejection when I ask the wrong question.