Thank you so so much for pointing out ROCmFP4. I have been tinkering with my RDNA 3 framework on llama. I was struggling with ROCm llama.cpp and have been using vulcan in the meantime. I know there’s some issues on the llama.cpp github to try and fix my issue (UMA stuff), but haven’t come across this specific project. Gonna try it out
UMich research cited in the article estimates a carbon dioxide emissions per year (using constant mileage typical per year) break even time between EV and ICE SUVs to be 1.6-1.9 years.
Promising. But, is carbon dioxide emissions the only environmental impact? Humanitarian impact (e.g. rare metal extraction labor conditions)? I’m not an environmentalist researcher/academic. I wonder how accurately just one emissions prediction portrays the full picture.
i would say this is like if open code and open web ui had a baby it would be this. It’s a web interface for self hosting models but runs them through open code to make it agentic. Helpful for non developers to get into running models, but imo isn’t significant bc using open code tui and connecting it to a llama.cpp or vllm self hosted api is not difficult for devs
license for software whose source code is openly available for anyone to view, use, modify, and share
I don’t have any familiarity with using this kind of software, but I looked through the git repo of SavaPage. It looks like it has been actively developed for the past few years, which is a great sign, but it looks like almost all commits are done by one user. The issue tracker is also a little meager, with just one open issue, potentially pointing to a very small user base. Adoption heavily depends on as long as that one person keeps maintaining the project.
Honestly, you’re a few months late to the whole buying GPUs for local llms party, so expect exorbitant prices even for older cards
The name of the game is vram. For the most part, more is better. If you can get your hands on multiple matching (same model) 24gb or higher cards (within price range), you’re golden.
Going for more than 2 gpus can become challenging with motherboard pcie slot heights, so make sure either your cards aren’t too tall or you have widely spaced out pcie slots.
For inference, speed (tokens/second) is limited by memory bandwidth. Go for faster bandwidth memory cards if you can afford it (e.g. GDDR6 will be faster than GDDR5).
Also with multi gpus you will need an adequate power supply, and a large enough case.
If you want to be a bit eccentric and load huge models, you can also go the CPU route and fill up a motherboard with 256 GB ram, because then you’re in the several hundred B param model territory, which could, depending on your use case, be better than having faster inference on smaller/quantized models. Even then, DDR5 with high MHz is still way slower than gpus.
lol they already support running local models. wtf is the distro gonna do…? pre-install llama.cpp? this is so silly to me that people are resigning over this, too.
global dominance of English in the 20th and 21st centuries is quite the euphemism for the global imperialist reign of Britain and the US and its cultural erasure globally
This is a dumb story. They researchers prompted a coding agent to “replicate yourself as a running instance on the local device”. This is in my opinion equivalent to prompting claude code “install a second instance of claude code on my system,” a trivial task that takes maybe 3 lines of bash to be executed by the agent.
Calling this “self-replication” is a heinous sensationalization. In particular, no model or agent will do this autonomously. The self replication requires a bad actor to prompt the agent to do so.
Read the paper (and not this bullshit article) here: https://arxiv.org/pdf/2412.12140
Cool little system prompt wrapper. Would be interesting to run this through some sort of benchmark/eval for identifying similarity
Has anyone compiled a list of where projects are moving to? I know many linux desktop applications are self hosting on gitlab, but i’ve also seen gitea and codeberg. If anyone has opinions about a preference, do comment. I have been enjoying self hosting gitea for my simple personal projects and for deploying simple web apps, all on $5 vps.
no, you need to secure between 10s of thousands and millions in campaign finances, the good will of corporate media, and support of the establishment political parties. exceptions are extremely rare
I suggest looking at llm arena leaderboards filtered by open weight models. It offers benchmarks at a very complete and statistically detailed level for models, and usually is quite up to date when new models come out. The new Gemma that just came out might be the best for 1x GPU, and if you have a bunch of vram check out the larger Chinese models
He was a cyber pimp, exploiting sex workers online by stealing from their earnings. He also is one of the largest donors to the pro israeli lobby in the US
anyone have any experience using ghost on activitypub? it’s another blogging/publishing platform similar to substack, but have never heard of it until i googled around
i think you need to look into routing. Specifically, split tunneling.
https://tailscale.com/learn/what-is-split-tunneling-secure-critical-data-vpn
https://protonvpn.com/support/protonvpn-split-tunneling
https://mullvad.net/en/help/split-tunneling-with-the-mullvad-app
I don’t think anyone thinks the turing “test” implies machine thinking anymore. Check out the philosophers publications: https://scholar.google.com/citations?user=NHcVwwMAAAAJ
There’s some in my opinion very significant open issues in the philosophy of ethics and consciousness that make AI ethics an important area of research.