Been working professionally for ~15 years or so, and it's never been this bad. Other than "knowing someone" (i.e. a referral, and even that falls through routinely), not sure what the best way to snag a position is these days.

  “US business continues to boom,” S&P Global Market Intelligence chief business
  economist Chris Williamson said in a statement. “To put the growth surge in
  context, barring the spike in demand following the opening up of the economy
  after the COVID-19 lockdowns, the latest improvement in business activity
  is the greatest recorded since early 2015. Business is clearly booming now in
  both manufacturing and services.”

Is it booming for everyone except software developers? I keep hearing about developers who have been looking for a job for 6 months or longer and don't get responses to applications.

Or is it some process failure (which you hint at) that is making it difficult to capitalize on that boom?

Business doing good doesn't automatically mean populations of humans are doing good themselves.

Fractal tool discovery: tool taxonomy where an agent can "drill deeper" to find what specific tool it's looking for. Helps if/when polluting context with a zillion (mostly unnecessary) tools.

Leveraging splay trees: this is my favorite data structure and I think relatively unused in the context of agents/harnesses. A lot of times, recently-used workflows/tool-chains will be used again, so having those at the top of the search hierarchy is an awesome optimization.

Virtual containerized notebooks: models working in sandboxed (WASI) Python notebooks is incredible. Even local models (if given enough time) will usually converge on a good solution. Being able to mount tools/resources/fs is again, imo quite untapped. Some problems here are running native things (thing numpy/pandas) in containers is a nightmare (or impossible).

Anyway, happy to see other folks seriously doing stuff in this space. If anyone wants to collaborate on anything don't hesitate to reach out :) I'm also actively looking for a job or some contract gigs.

Fun times ahead.

Ugh, I can't help but respond to this one point. The fact that this is even an issue in the current year just tells us how screwed the software field is in a lot of ways. I don't mean that in existential terms, but of how divided we've become in terms of what's happened to human reasoning. On the one hand, you have people who apply deep thinking to develop the sort of approaches you described, and there's the exponentially growing segment of not-even-programmers who seem to never ask themselves whether any of their ideas have any sort of consequences.

Take MCPs for instance. Sure, I guess it can sometimes make sense to have a stateful API that is optimized for agents. Yet, more often than not, these MCPs frontload a ton of context where it's not needed, and solve problems where none existed. Merely sticking an API (MCP) in front of an API (CLI, REST, GraphQL) without a benefit that can be explained in a single sentence is lunacy and demonstrates a real lack of complex thinking.

However, MCPs are a godsend to anyone trying to manage access to systems (especially SaaS systems) in a large organization that includes a ton of non-technical employees. We use at least a couple of SaaS products where RBAC is severely limited in what it can govern or where API keys do not have any kind of governance so handing one out is basically handing out read/write on the entire system. Being able to configure an MCP to only have read endpoints enabled it a very useful piece of duct tape. Obviously it would be great if the underlying API was just designed with proper governance but that's out of the user's control.

But, you just very clearly described why human software engineers are in higher demand than ever before.

It's always been "screwed" in the sense that everyone else sucks at wielding the power.

What we're actually witnessing is a watershed moment where a lot of technologically illiterate people are getting left behind. Those of us on the literate side are left to fight amongst ourselves and the powers that be for control over the future. We have actually been doing an alright job all things considered (else this conversation wouldn't be happening). Politicians are aging out and we're the adults in the room now.

mcp with 2 tools: execute(code) and search(query)

all other tools, mcps, apis, whatever are encapsulated by the one interface. new tools don't bloat the agent's context, and it can write its own code to perform more advanced and batch operations against the available tools (executed within a sandbox).

executor is a great implementation of this - https://executor.sh

opencode v2 also provides its own native implementation

In this way, I tend to think of the context environment on an agent is the "agent nav" -- it presents context, allows progressive disclosure, and if poorly designed, makes the agent flounder as a poorly designed UI/UX does.

I wrote a bot to scrape the feeds of local event organizers and extract their events for me so I wouldn't have to look at that damned site just to find out what's going on. My account was banned for bot activity.

Never posted, commented, liked, or interacted in any way. Just read feeds and exported data.

Abandoned the project and blocked instagram at the DNS level for all my devices.

Fuck instagram.

Then try again.

I think this would be likely comparable to a scheduled backup, so I think it would be an acceptable maintenance window. However, deterministic algorithms would likely beat re-training (or re-fine-tuning) the model. For example, one could analyze actual distributions or whatever (instead of assuming uniform), and then some plans would automatically be eliminated.

Imo a good thought experiment is to look at places that are hyper-optimized, like compilers. Would LLMs bring anything to the table (architecturally or performance-wise) to a piece of software that has been carefully crafted for decades? (Methinks no.)

As an example, register allocation is graph colouring, and thus NP complete; a model for producing an allocation plan is learning heuristics that might look at more features in combination than the ones hand-crafted into the compiler. An LLM for the job might do better than a more focused model like a GNN, due to sheer size, the effectiveness of transformers, or magic. But it probably won't do an overall better job than the handcrafted heuristics, because those handcrafted heuristics also tend to compile very, very fast with a small memory footprint, and can be debugged (more) easily when they go wrong.

Postgres keeps histograms (including N most common values) for all columns; it does not blindly assume uniform distributions. (Presumably an LLM would have access to the same histograms.)

I know this all sounds abstract. I've been mulling over it for the past year and it's very hard; and LLMs are super janky and inconsistent so it's 100% not trivial. So in some sense I understand why a lazy bottom-of-the-barrel "chat interface" has become the de facto standard.

The protocol will of course be `text -> text` for the typical LLM (though some new models are structured, as we saw yesterday with Jev). Even so, the user doesn't need to be exposed to the protocol (almost ever). It's not like I'm crafting a POST request to send this form.

There's a lot of room for window dressing, and I look at AI like I look at the touch-screen. It was a fun technical toy until people (mostly Apple) poured in a lot of serious effort into "ok, how do we make this pleasant to use, intuitive, and genuinely useful?"

Dictation? Neuralink thought-to-text?

This isn't necessarily true. I'm working on a local harness that doesn't do this and instead coerces everything to YAML (including tool calls) for better bucketing. Some models are indeed trained on the `<|tool_call>...<tool_call|>` token schema (or something similar—e.g. jinja), but it's vendor-specific and often times inconsistent (so you're constantly fixing calls or going back to the LLM).

Was trying to combine AI with generative storytelling with a card game. It was a fun experiment. To play, you'll have to get a friend to queue up at the same time.

It's also clear that, as Tan indicates, open-weight models will be (and basically already are) just as good as frontier models. It's all about the harness, baby. We will have two main forks in the road, and two new industries created:

    - AI hardware (NVidia/Cerebras/etc.), the equivalent of Intel/AMD
    - AI software (harnesses, assistants, etc.) the equivalent of Microsoft/Apple

Inference is not being subsidized and in fact has pretty high margins.

Similar-sized open weight models on openrouter are 15x cheaper per token than the big labs. This should reflect the isolated cost of inference, since 3rd party hosts have no reason to subsidize and no training costs to amortize.

Only datacenter buildout costs are being subsidized.

If it were true that Anthropic and OpenAI were profitable on all inference they wouldn’t need to constantly raise so much money. Anthropic regularly announce huge investments in infrastructure but it is all smoke and mirrors, data center build out costs aren’t being paid by OpenAI and Anthropic, they’re financed externally. Google, for example, are backstopping tens of billions of datacenter build outs that are being financed based on commitments but not investment from Anthropic.

You are underestimating the insanity of subscription subsidization. Being profitable on API inference is meaningless when it is such a small proportion of usage and is only going to fall off a cliff as cheap open weight models become more capable.

https://hraness.com/writing/my-girlfriend-asked-me-why-i-hav...

The absolute majority of tokens are being subsidized and as soon as the subsidies end usage will fall off a cliff, rendering all the data center buildout a terrible waste of money.

This is untrue.

You are way underestimating enterprise usage here.

You can't get the $200 subscription on Teams plans at all, and Enterprise plans don't have any subsidized plans.

Anthropic has 80% margins on inference: https://archive.is/BtEeN#selection-1575.0-1575.75

If $7.5bn is their enterprise revenue and it costs just $1.5bn to generate, that leaves $3.5bn in compute costs to account for. Dario previously said that training costs less than inference so training can’t explain it.

If subscriptions aren’t the majority of usage and aren’t subsidized, where is the money going? Anthropic don’t spend money on data centre build out so that can’t be it either.

I don't think you can reverse this out like this because the 80% rate is before payments to "distribution partners, including Amazon". I think that payment includes the hosting cost for that those model but it's unclear.

> Dario previously said that training costs less than inference

Do you have a source for that?

Are you sure you aren't conflating the statements Dario has made that training costs less than they make on inference (over the life cycle of a model)?

The "hosting cost" is paid for by Anthropic and is the largest cost. The money Anthropic pay to Amazon for delivering Anthropic models via Bedrock is separate, independent of compute costs, best thought of as commission.

The forecasted / guessed / estimated 80% number is based what customers pay per token minus the projected compute costs, i.e: the people who believe that Anthropic has 80% margins on tokens believe that Anthropic spend $0.20 on inference compute for every $1 of per-token billed-via-the-api revenue.

We know that there are hundreds of thousands of fixed-price subscriptions being used to their absolute maximum, with many people bragging about how many subscriptions they run in parallel. These tokens are not included in the 80% margins, they are acknowledged to be "subsidized". People like @theo on Twitter post almost daily about how much they're milking Anthropic and OpenAI with leaderboards.

Both Anthropic and OpenAI (more so OpenAI) do "resets" where they increase the limits available to people on their fixed price plans. We know that there are people paying $1,000 per month for multiple subscriptions to generate tokens that would cost $50,000 via the API. Even if Anthropic's margins are 80% on compute for per-token billing, that's still $10,000 of cost to Anthropic generating just $1,000 in revenue. Multiply that by tens of thousands or maybe even hundreds of thousands of subscriptions.

Anthropic and OpenAI have raised over $100 billion each and continue to raise. If they're making 80% or even 50% margins on $10 billion in revenue per month they would not need to raise, they would be shouting for the roof tops about how profitable they are, they wouldn't be delaying their IPOs, yet they're only profitable by non-GAAP metrics like WeWork's classic "Community-adjusted EBITDA" or in this case "per-token-adjusted EBITDA" or whatever they will call it in their IPOs.

Yes, they're selling tokens via the API for more than they cost, they are profitable on per-token billed inference, it has positive margins, but those profits are obliterated when you account for all the inference they're paying for out of pocket on fixed price subscriptions, upon which they keep increasing limits because they desperately need to show growth further harming their profitability (consuming all of the money they make from their API).

If Anthropic and OpenAI needed to be profitable tomorrow, they could be, they could kill off all their fixed price subscription plans and charge only for usage via the API, they'd print money, but they'd lose mindshare because nobody except for enterprises can afford to pay the true cost, all the regular people would switch to cost effective good-enough models, and then within months, the enterprises would start to switch too because no longer would their employees be claude-pilled.

Anthropic and OpenAI cannot turn off subsidization, thus, their margins on per-token API billing are not important in any discussion about their long term financial wellbeing. Just look at the large scale customers like Harvey (~15 trillion tokens per month, ~$50m+ in spend) who are, sensibly, investing in building their own specialized models that are cheap to run so they can cut their spend by 90%. That's profitable revenue for Anthropic / OpenAI today, but completely gone soon.

> Do you have a source for that?

https://www.youtube.com/watch?v=7xij6SoCClI

"This week, Noah Smith and Erik Torenberg are joined by Dario Amodei, CEO and Co-founder of Anthropic. Dario talks about the economics of AI development, the comparative advantage of AI companies like Anthropic, AI safety, and his stance on California's SB 1047 bill. They also discuss the impacts of AI on global power dynamics, competition between the US and China, and inequality in an AI-powered world."

At around 12 minutes in:

"I think actually even if such a model is released one thing you know that's a this analogy to to open- Source software is that these big models they're actually very expensive to run on inference the majority of the cost is is inference not necessarily the training of the model so if you have only you know I don't know 10 20% 30% better way to do inference that can kind of negate the effect so the economics are kind of strange yes there's this giant fixed cost that you have to amortise but then there's also the per unit cost of inference and small differences in that can actually again assuming the thing is deployed widely enough make a very big difference so I don't know quite how that's going to play out"

The scales have changed since then with inference costs falling and more being spent on training but the fundamentals are the same. Inference is expensive, in part, because peak usage dictates capacity whereas capacity can dictate training. Anthropic must pay billions of dollars per month to be able to handle peak inference, hence their efforts to try and shape usage by offering discounts / flexible limits at different times of the day. They can train when capacity permits.

My point is that you can't reverse out the maths like you did without knowing how much this is.

> These tokens are not included in the 80% margins, they are acknowledged to be "subsidized".

No, this isn't correct. Even including these they are claiming 80% margins.

There's nothing at all that indicates subscriptions aren't included in this - it's a simple statement of their running margin.

> these big models they're actually very expensive to run on inference the majority of the cost is is inference not necessarily the training of the model

I don't think you can take this statement to claim that currently they spend more on inference than on training. I think he's saying over the lifetime of a model maybe inference ends up costing more unless they keep finding "better way to do inference that can kind of negate the effect".

> If Anthropic and OpenAI needed to be profitable tomorrow, they could be, they could kill off all their fixed price subscription plans and charge only for usage via the API, they'd print money

My point is that their subscription costs are a lot less than you think because of this statement by Anthropic that they have 80% margins including these subscriptions.

If their margins are 80%, that means on $10 billion in revenue they're spending just $2 billion. Anthropic's own announcements put their spending at much, much higher, such as the $1.25 billion per month they are paying to SpaceX for compute, and the ~$3.5 billion they're paying to Google each month, and the billions to Amazon each month too.

https://www.anthropic.com/news/higher-limits-spacex

> We’ve signed an agreement with SpaceX to use all of the compute capacity at their Colossus 1 data center. This gives us access to more than 300 megawatts of new capacity (over 220,000 NVIDIA GPUs) within the month. This additional capacity will directly improve capacity for Claude Pro and Claude Max subscribers.

There's no world in which Anthropic has 80% margins. At their current expenditure on compute it would require at least $20 billion in revenue to be mathematically possible. The 80% figure on compute margins that is widely discussed is based on analysis by SemiAnalysis and refers only to their per-token compute margin (which is calculated comparing hardware costs + electricity costs to what they charge via the API).

The estimated training costs for models like Opus and Astra are ~$1 billion and they're not training multiple frontier models in parallel every month. Training costs cannot explain where billions of dollars per month are disappearing if they have 80% margins. And that's before even considering all the money they're raising and spending. Anthropic raised tens of billions just a few months ago, OpenAI even more.

Where is the money going?

No.

To quote from my link above:

> Anthropic has told its backers it will be profitable this quarter, as it moves to allay investor concerns about the aggressive cash burn of frontier AI companies ahead of its blockbuster initial public offering.

> The company has told a small group of shareholders that its adjusted operating income will be positive for the second consecutive quarter,

[snip]

> Anthropic’s gross margins are above 80 per cent before accounting for revenue shared with distribution partners, including Amazon, and the cost of training its models, according to two of the people.

So this is a direct claim by Anthropic people, not analysis by outsiders.

> If their margins are 80%, that means on $10 billion in revenue they're spending just $2 billion.

Agree - but only spending $2B on inference.

> Anthropic's own announcements put their spending at much, much higher, such as the $1.25 billion per month they are paying to SpaceX for compute, and the ~$3.5 billion they're paying to Google each month, and the billions to Amazon each month too.

I don't believe this compute is sorely dedicated to inference. Only the part dedicated to inference is included in this profit rate calculation.

Do we know that? As I understand it, enterprise customers pay more. Do we know the usage breakdown between monthly subscribers vs enterprise accounts? I agree that it's inevitable that subsidized subscriptions are unlikely to last forever, but that's not the only assumption in your argument.

Edit: I think "enterprise customers pay more" was poorly phrased. I mean that enterprise customers are charged per token, presumably with a profit margin, and thus are not subsidized. While personal accounts are (thought to be) highly subsidized if you consistently max out the quotas. We also don't know what proportion of personal accounts do that though, which is another big question mark.

I know people that have the most expensive plan on all the platforms... because

The other side to that is, what is 'cost'? Is cost just inference or are expenses also being taken into account? Because the expenses of these companies are huge to build the models.

1. Companies are trying to decrease costs, not increase it, and are looking at alternatives

2. Competitors are catching up, and even if the frontier labs are "better" at some things (like writing plans or complicated analysis), the competitors can take a lot of the inference on routine tasks like implementing a well-defined plan

3. The frontier labs don't just need to have high margins right now. They have to pay back their massive liabilities.

I was referring to the "AI labs" here. Sam Altman himself conceded that OpenAI is losing money on the $200 subscription. Using open-weight/open-source models is indeed cheaper (and no reason for inference to be subsidized).

>Sam Altman himself conceded that OpenAI is losing money on the $200 subscription.

They have since stopped offering the $200 subscription, probably for this reason.

Subscription margins are harder to judge because it depends on usage; token costs are a better comparison.

The US can no longer keep global trade secure on the high seas. What if the supply chains for GPUs get disrupted for months, a year? Then what?

I fear Google will win in the longer run.

    > inference is already being subsidized

the only place where actual subsidization (below cost) might be happening are the subscriptions. even that is unlikely because to be truly below cost you either need to offer below cost of electricity which isn't happening, have potential API users using multiple subscriptions or have opportunity cost loss due to saturation.

Maybe it is “sunked cost” or maybe it is “I will do it myself dammit”.