Let's make the whole working/colaborating environment a training input for the model, we have to replace each and every link of human everywhere, think about the stock prices... yes it will have cathastropic consequences to succeed but then think again when you vest in 40 years you will be rich!
and then the first reply is someone saying that if not someone else will...
and then another saying that the whole idea of business is to make money...
and then another that the guy from valve is a jerk...
get your eyesight check, at some point in life I was having difficulties staying focus and even started having headaches... and I needed prescription glasses...
Also very important, try to sleep the best you can and do exercise... I remember reading here about exercise in some cases exceeding the benefits of ADHD medications...
If you are overdosing things... don't... caffeine can be overdosed and the effects are nasty and lingering... caffeine is meant to be a zero sum game, where it gets you speed in the morning and when it wears off then you go to sleep... if you over dose it then you are making your body over exert and at some point the body will communicate the results via extreme tiredness, bad humor, etc...
Don't be hard on yourself, from everyone we are the most hard on ourselves when we should be the most kind of them all.
> Also very important, try to sleep the best you can and do exercise... I remember reading here about exercise in some cases exceeding the benefits of ADHD medications...
Exercise is very true. I used to take walks but it didn't help; pushing myself at the gym (inc using a trainer) is a lot more effective.
Also, ADHD meds interact with sleep, both making it better and worse. I was on Intuniv 2mg and it absolutely ruined my sleep, but 1mg plus a sleep medication (25mg trazodone) are working very well.
Definitely agree on exercise. I'm a temperamentally neurotic person and had ADHD diagnosed as a child. I no longer take ADHD medication, but I run 3-4 days a week and it makes my baseline tension and neuroticism much lower, and improves focus.
I would like to argue that trying to provide a free service is non achievable, most of the time it will drill down to ads, people are already paying electricity and time in ads.
If we pay say 3 secs of compute time of monero, and everyone pay the same... you remove the ads from the internet, people will start gettind paid without gate keepers for content they generate, and you can charge the AI machine for ingesting your content.
We were providing free services decades ago. Hosting a website, or a Minecraft server, or a VOIP server, or IRC, or a forum simply doesn't cost that much. Well within "some guy's hobby budget" type expenses.
Yet absolute price per GB isn't the full story when the popular tools accessible for the general public require more RAM for the same task today as compared to the past. You won't get far using the latest version of MS Word on a desktop with specs from 1999, or running MS Word 97 to edit academic essays turned this year.
Even for background services, rather than the general public, I would not attempt running an email server with the same RAM as I used in the 90s, 00s, or 10s.
Yeah, it went from ~$10 / month to ~$30 a month to rent a relatively beefy VPS capable of doing the above (game servers being the most resource demanding of the lot). Still well within hobby money.
X402... I was not aware, I had this idea of making HTTP connection depend on a monero transaction, the monero transaction should take around 3 secs of the average computer/cellphone... once you have paid that you can access the resource.
You wanna crawl the whole internet non stop, you pay non stop, 3 secs is probably the same as we pay in ads for those without adblockers and then content generators can start getting paid for the resources they generate.
because we need quadratic energy increase to increase speed linearly, that's why a 200hp car is not twice as fast as a 100hp one.
Gee let me elaborate a bit more... if the 100hp car has a top speed of 100mph, a 200hp car will have maybe 130mph of top speed (assuming all other things the same) because you are fighting friction of an inmense amount of things that want to stop that movement.
Anecdote... this famous US plane made with this titanium allow, I remember reading that at the speeds it was able to reach, the pressure of the air hitting the surfaces is so much that it will cause other metals to fail.
Imagine the amount of energy you have to spend to heat the whole surface of the plane while traveling through cool air!
I want to rush to git clone, but as things are, the odds are extremely high that this kind of things that are too good to be real are honeypots and something there will compromise your machine or make your llm start working for someone else...
Another perspective, if you compare it to two years ago, how much more expensive is it and how much better? we are paying the sAIm Taxltman.
Just see, you could buy the steam deck for 250 refurbished 2 years ago, now it's what 700$?
Try to buy 2 64GB dims of ram.
I created a patch for llama.cpp to store on disk instead of deleting the kv cache as well as the checkpoints... there is this bug on llama.cpp if you have more than one instance going on of chats... and that causes the kv cache to be lost between changes of chat...
And I can tell you, using Qwen3.627B after one day of use you can have 120-200Gb of chats on disk.
And yes it's way way faster, even if you get it from a spinning disk it's still faster than re-computing the whole thing...
I guess for a 300B parameter or more and couple million users with the price of storage increasing as part of ramagedon this is also not viable...
Qwen 27B maxes out at a 16GB context. A nice thing about DeepSeek V4, especially Flash, is that its context size stays tiny even at 1M tokens! Which in turn opens up wide batching on common consumer platforms.
You need wider batches to get effective reuse of experts in any given layer, but you absolutely can. DeepSeek V4 has tiny KV caches that make this quite feasible. When targeting consumer platforms that only have a limited amount of compute headroom to begin with, the approach is quite reasonable.
What do you mean by doing the math? If you repeatedly sample n_active experts out of n_total, why wouldn't you expect to get some meaningful probability of reuse/overlap once your batch grows past size 5 or so (for the sparsest MoE models in common use)? And you only need enough reuse to fill the compute headroom which is quite small on consumer platforms (we won't have huge TOPS numbers for the typical integrated GPU in Strix Halo or even the upcoming RTX Spark). Plus if you're a single user running multiple streams in parallel the choice of experts will be highly biased leading to more reuse.
There's been some very rough experiments with batching on Apple Silicon (and that's not a highly suitable platform since the compute/thermals bottleneck hits sooner than elsewhere) that seem to be broadly consistent with what I argued, showing as much as 2x total decode throughput with an 8-wide batch. That's substantial in this context.
Assuming you magically use all 128GiB of xRAM you need to read ~32GiB per token in batched mode. On a good SSD that would be 1/3 tokens per second. Cool, 2x that you can do 2/3 tokens per second. Let's assume you are lucky and can actually do 6/7 tokens per second. That's still an extremely far cry from 20+ tokens per second of 27B before any batching.
Before batching. The slowdown is because the model does not fit the xRAM so experts will have to be read from SSD on every forward pass. That's why it is impractically slow.
Batching could allow you to generate 10 tokens for 10 different conversations at the time, but it also means that you need to load different experts for different tokens, so it does not help as much as it does for dense models.
But IIUC the point is that each expert gets used for more than just the one token. So yes, the tps of a given thread takes a hit because now you're sometimes going to schedule in unrelated experts and it will have to pause. But overall you're utilizing the hardware much more efficiently and so in aggregate there's a speedup.
On top of that (as previously pointed out by zoz) for a single user running a single overarching task the choice of experts is expected to be highly biased.
> Why do you think that's the case? Part of the training is balancing load between experts.
That is a fair point. That expectation may have been misplaced on my part. I'm not sufficiently familiar with the details of MoE training.
> The slowdown is because the model does not fit the xRAM so experts will have to be read from SSD on every forward pass.
> 20+ tokens per second of 27B before any batching.
Does the model fit in RAM or not? What is your justification for your stated expectation that the unbatched model will perform 20x faster than the aggregate tps (note, not the single stream tps) of the batched model?
My expectation is that if the unbatched model is 20 tps and batching provides a 2x speedup then each individual stream will be slower but the aggregate throughput should rise to 40 tps. What do you believe me to be missing here?
It has been quantized to 80GB (2-bit quantization for experts) with limited degradation. Certainly competitive with a 27B model, and especially useful in a size range where few "native" models exist.
> Why? Why do you think that's the case? Part of the training is balancing load between experts.
The training balances expert choice across the entire scope of the model. Experiments have consistently shown that within a given session or topic (taken in a broad sense) expert choice is biased in a way that's likely to make caching useful and reuse across a user-specific batch realistic.
I hope someone at jetbrains with enough power read what we are saying in this thread.... changes for the sake of changes are bad... when they did the change to the "new UI" I kind of stomach it.. because the option to go to the other one was there... but I don't know... how much benefit did everyone derive from it?
But when the AI crazyness started it was f downhill... Jetbrains need to do a one thing... expose a connector so the ai can connect to the thing and do what it needs to do... so the IDE amplifies the model... go to definition... give me back all the errors... without having to play with bash, make the thing go fast... and please don't put the stuff we want to do behind a gated thing that only you can give us and charge us for...
I stopped paying and stayed in 2024.2, every once in a while I see if there is anything meaningful worth it there... nop
>expose a connector so the ai can connect to the thing and do what it needs to do...
They have done this now, they've implemented ACP so you can use any agent that supports that from any IntelliJ editor. They (optionally) provide an MCP server which the agents can also use.
Ohhhh geee!!! I just applied the patch to my local git copy.
You need to use the model on the PR that he submitted, the model is particular because it has extra information that allows the MTP to happen.
I have two amd gpus, and qwen3.6 27B qk6 does around 20t/s generation... If I run it only on one I get like 35t/s.
But with this patch I saw 46t/s with qwen3.6 27B q8... this is insane, it's 250% faster than the original speed, there was no gpu I could upgrade to get that kind of boost, amazing!
and then the first reply is someone saying that if not someone else will... and then another saying that the whole idea of business is to make money... and then another that the guy from valve is a jerk...
reply