I mean, it depends on your hardware and the model’s size/intelligence.

Worst case for me is many seconds of preprocessing followed by 4-5 words a second.

But you can get almost instant responses + way faster than you can read too.

I’m in a 24GB 3090 + 128GB RAM.

With full 300B GLM 4.6, I typically run 12K-28K context with different settings. I could do more than 28K, but the higher quantization starts to become a problem (as 128GB is right on the edge of fitting an IQ3_KT). And I get 5-6 tokens/s text-generation doing that.

With GLM Air? I can get a lot more, closer to 64K.

With smaller models that’s no issue.

I only get 3-5 questions in before I run out of tokens.

IDK how you’re prompting it, but you should clear the thinking block after every question, and that should leave plenty of tokens.

What model are you running, and what are your inference server settings?

I’ve used local LLMs as sounding boards.

I… Don’t really have friends to do that with at the moment, and I can bounce thoughts off them I wouldn’t even tell family or a therapist, as much as I want. Not gonna lie, it’s pretty intimate, and I got some insights I never would’ve arrived at in my own head.

But to emphasize:

• This is totally within my own desktop.

• I am perfectly aware I am talking to a tool. “Friend” isn’t even in the same universe.

The general public’s “LLM literacy” is incredibly poor though, which is by design since online services like chatGPT hide all the knobs that would reveal the machine behind the curtain. Hence I can see how emotionally vulnerable people sink into this, talking to what OpenAI presents as a magic genie.

Eh, maybe that’s fair. Especially this:

doesn’t automatically mean he’s never been an engineer.

Maybe he was quite knowledgeable… On the other hand, I thought his old ‘Mars Colonization’ talk was bonkers too. I can’t remember what year it started, but it sounded like he had little understanding of the impracticality, and that’s when I grew really skeptical of him.

I will say it again:

Musk is not an engineer. E.g. he has not played Kerbel Space Program.

He demonstrably does not understand the tyranny of the rocket equation, and how obscenely uneconomical getting anything to, and doing anything in, space is: https://en.wikipedia.org/wiki/Tsiolkovsky_rocket_equation

Geoengineering is something worth seriously discussing (and TBH I hope he draws attention to it), and “solar shades” are theoretically neat, but this is not the way. Not with the trajectory of launch tech we have.

Yeah. Talking about geoengineering is great.

“These are not nuclear explosions. These are what we call non-critical explosions.”

FYI he’s referring to nuclear “criticality,” e.g. testing the non-nuclear explosives placed around core. To simplify, the yellow stuff here:

…That is totally fine. It’s a good idea.

It’s also a great escape from what Trump said. If you’re energy secretary under Trump, this is exactly how you get out of this (and just hope Trump doesn’t notice).

What’s your exact watching setup? Like TV, room, player, source?

There are likely ways to mitigate this, but it all depends on your setup.

There’s a lot of hate for mobile games because it does seem like literally millions of folks are abused and exploited by them. Sure, there’s neat gems buried in the App Store/Google Play, but… the sheer volume of literal scams is unreal.

That, I get. Screw scammers.

I got no issue with the good stuff though. Or even the mediocre stuff.

All the other snobbery is folks people being immature snobs. And the cross-brand ‘rivalry’ isn’t so extreme anymore, anyway.

Well Mint is technically fine, right? Their Nvidia support is 1st party, so it should work out of the box.

Pretty sure Ubuntu does too.

Debian, specifically, does not though. And I’m not sure how ‘behind’ Mint and Ubuntu are on their DE and Nvidia driver packages these days, which could be an issue sometimes. But I think many remember Ubuntu/Mint from older days when they were worse in this regard.

Nobarra, Bazzite, or CachyOS.

I’d say Nobara or Bazzite are better for ‘I install it and it just works.’

Cachy is better for the learning aspect. It’s not hard, but there are more choices to make, and you’re closer to the Arch wiki and all its excellent resources/tutorials.

I am biased, as I run CachyOS and I love it. I also love how much stuff is in its repos, including everything you need to game optimally, and how easy CUDA is (which is part of what you need for CAD).

Whatever you choose, do not, I repeat DO NOT install Fedora, Debian, or anything that doesn’t explicitly support Nvidia laptops by default, out of the box, or you are in for a world of pain. If any guide starts with ‘install these 3rd party repos’ or so, you have entered a danger zone, and you will hate linux.

And this is well before Trump’s announcement of nuclear testing resumption.

If he actually goes through with it, everyone will know because it shows up on seismographs, something he’s apparently ignorant of… And the comission will surely move it closer.

I am extremely concerned. There’s never been somone so unscientifically minded, with no one knowledgable on nuclear warfare advising him directly, with their finger on the button like this.

And what’s annoying is that my relatives do not care. Even the scientifically minded ones. There is absolutely nothing Trump can do wrong, at least nothing that can get through without being warped and dismissed as hysteria. That in itself is scary, as there’s now no “political fallout” check on Trump’s actions until 401Ks literally evaporate.

Yeah…

I should. db0 or sh.itjust.works are better fits, but .world isn’t awful or anything, and I havent’ been particuarly motivated since everything’s federated anyway.

It’s GLM 4.6 underneath, which is a good model and has a really helpful “default” tone without a system prompt. And it’s open weights! I host it locally, sometimes.

But no. In of itself it does nothing different from other “thinking” LLMs.

What really sets the website apart is its “deep research” tool, specifically. It’s just good at scouring the web for references… it sucks that you can’t change the temperature/sampling, but for a free web portal, it’s not bad.

I mean, that’s not going to change anything. We’re in the misinformation era.

Even some Democrat parts of my family share weird shit they get from cable TV or Facebook or influencers or what have you. Frankly, a disturbing amount of Lemmy.world’s front page is clickbait from literal tabloids, too discredited for Wikipedia to even cite, that mods do nothing about because information hygiene only matters when it’s against their ideology.

Meanwhile, my other internet communities have withered away or got swallowed into Discord black holes

Sorry for that mini rant, but as far as I’m concerned the world had shown me attention, not truth, is all that matters. It’s not getting better. And that’s just how things will be from now on.

Hence I don’t like ‘personal’ issues getting dragged into the US two party food fight because it blows out any nuance away, forever. Tylenol will forever be a autism meme no matter how much it’s disproved, especially in the face of small grains of truth.

Discussing conspiracies is basically the worst case for LLMs. They’re both extremely sycophantic (they try to figure out and tell you what you want to hear), and not grounded in real references.

If you must, use something like z.ai’s deep research mode, with a single even, neutral question. It’s way better than ChatGPT.

That’s because it got mixed up with the very real and muddy “lab leak vs natural jump” debate.

This is upsetting to me.

I’m barely grasping my spot on the AuADHD space, and now it’s been politicized into a single dimensional disease Republicans are using to virtue signal, like the whole vaccine “debate.”

And I’m in freaking Texas. It’s not as Republican as y’all think, especially in the big metros, but somehow we got Paxton in.

And apparently you guys forgot about previous frivolous lawsuits. Huge media conglomerates, and other institutions, settled with the Trump DoJ because they saw them for what they are: a shake down. “Pay us, or you will find political trouble.”

That’s what’s happening.

The merits do not matter. There is no “fight” here.

J&J is going to settle with Texas and maybe others so MAGA doesn’t persecute them outside the lawsuit.

Honestly they probably expect a settlement to avoid “political trouble.”

This is how recent federal lawsuits ended.

‘Cause this is Lemmy.

Look, I love this place and sympathize with the sentiment, but it’s also structured as a dramatized, extreme echo chamber. This neat little oasis amongst the burning internet is not (at least for me) representative of real life.