Hacker News new | past | comments | ask | show | jobs | submit
jermaustin1 | on: Muse Glimmer: 30B-parameter model optimized for always-on local agent workflows
I've been using Qwen3.6 35B A3B, and with reasoning turned on, I'd say 2/3 (give or take) of the tokens for a response are thinking tokens. Which at 70+ tps locally, that isn't that awful. I run an 80k context across 4-10 "agents" for my solo TTRPG, where Qwen is the GM, each NPC at a location, the director, and the narrator.

Each turn is about 45-60 seconds to generate all of the various responses. The GM and director have reasoning on, and the NPCs/Location/Narrator do not.

It's a fairly good "engine" for that. I'm not sure how a denser Qwen would do here regarding speed.

jakswa | parent | next
I like the tabletop RPG use case, and wanted to say: If your hardware likes it you should check out Gemma 4 for creative DMing use case. I found it to be much better at holding the plotlines and being creative on gaming turns. My experimental case was an audio-only Zork and Gemma 12B and even E4B were pretty good!
crorella | parent | next
This sound very interesting, do you have any resource I could look at? Me and my son did a very rudimentary (compared to yours) setup to play Paranoia, but this is at another level.
jermaustin1 | root | parent
I'll go over my repo, and see if it is hiding any API keys and maybe make it public. The issue I have is it relies on a nuget package that also isn't live (its in my local nuget feed).

I'm not sure what all is needed to make that work for people.

makr17 | parent | next
I'm working on something similar. My biggest annoyance is that the overly-helpful LLM was making every die roll succeed. I ended up building some tooling around rolling dice. Also some tooling around character stats and inventory management, so those don't get lost in context compression.
toyg | parent | next
Is there some sort of dedicated tool for this type of setup, or did you hand-craft it ?
jermaustin1 | root | parent | next
Somewhat hand rolled, somewhat claude coded.

Back in 2023 I started my own C# LLM library for doing tool calls and structured output, and over the years it has morphed bigger and bigger, and that is the backbone of almost all of my LLM-based projects.

I've never released it, but its easy to understand, and simple to add your own tools:

  [AIDescription("Get current weather for a location")]
  static string GetWeather(
    [AIDescription("The city name")] string city,
    [AIDescription("The country name")] string country,
    [AIDescription("Temperature unit", ["C", "F"])] string unit = "C")
  {
    // make some API call to a weather API and return a string to the LLM
    return $"The weather in {city}, {country} is 22°{unit} and sunny";
  }
  
  var chat = client.StartConversation("You are a helpful assistant with access to weather data.");
  var response = await chat.SendAsync<string>("What's the weather in London?", GetWeather);
I'm sure plenty of better libraries exist for this now, but in 2023, I don't think any existed in the dotnet ecosystem. I've never released it though, because I've never "finished" it.
rhplus | root | parent
It looks like you’re describing something like an MCP server and a client model.

If you’re in the C# ecosystem you could consider converting your APIs to MCP format tools using the MCP SDK.

https://devblogs.microsoft.com/dotnet/build-a-model-context-...

And then leveraging Microsoft Agent Framework for the client and orchestration side of things:

https://learn.microsoft.com/en-us/agent-framework/

seanmcdirmid | root | parent
Not parent, but I use Goose for my non-handcrafted Qwen use cases, I’m also working on handcrafting as well. Goose was the only harness that didnt bloat context too much with system prompts (like openclaw) and I could get reasonable web search working with Qwen.
lostmsu | parent
Are you running inference in parallel? 70 tps seems low for parallel execution.
jermaustin1 | root | parent
It is on a single 3090, and that seems to be where it averages out. I'll get 85tps on turn 0, but then it settles down to low 70s within a few turns, but holds steady at that.

My issue currently is KV Cache, because I can't keep enough parallel caches running (4 is where I'm at), so TTFT (is that the initialism?) can be long when I have a particularly large scene (basically more than 2 NPCs).

But my harness does let me offload to any OpenAI compatible endpoint, I just prefer local cuz free.