For help please visit help.ft.com. We apol­o­gise for any in­con­ve­nience.

The fol­low­ing in­for­ma­tion can help our sup­port team to re­solve this is­sue.

V1.0 · Recurring · July 2026

A Letter From Our CTO, Raffi Krikorian “

In New Zealand’s far north, a Māori broad­caster trains speech mod­els for te reo — a lan­guage too small for any mar­ket — un­der a li­cense that keeps the data with its peo­ple. PwC, one of the largest ac­count­ing firms in the world, fine-tuned an open model on the lan­guage of fi­nance and runs it to­day for hun­dreds of clients, on its own hard­ware, with no per-to­ken me­ter run­ning. Researchers in Lausanne built an open med­ical model with the Red Cross, tuned to its hu­man­i­tar­ian guide­lines, and are prepar­ing clin­i­cal tri­als at home and in Tanzania. In East Africa, farm­ers di­ag­nose cas­sava dis­ease with a model that runs on the phone it­self, of­fline, in fields the cloud has never reached. In Switzerland, a pub­lic con­sor­tium trained a na­tional model on pub­lic su­per­com­put­ers and re­leased all of it: weights, data, train­ing code. None of them asked per­mis­sion, and none of them could have rented this. They own it — that is the whole idea.

We have been here be­fore. Mozilla ex­ists be­cause one com­pany tried to own the front door to the web, and an open com­mu­nity rose up to make sure it never could. Twenty-five years later, some­one is run­ning the same play. We bet on open the first time. Open won. Together, we can do it again.

Our be­lief is sim­ple: the path for­ward is com­pe­ti­tion and in­ter­op­er­abil­ity. We be­lieve in a world of many mod­els, stan­dard ways to plug them to­gether, and the free­dom to walk away from any ven­dor at any time. Open has a record here. It grew the pie and let more peo­ple own a slice of it.

Read what fol­lows as a map: where open AI is win­ning — some num­bers sur­prised even us — and where it is ex­posed. A case that hides its weak points is an ad­ver­tise­ment.”

Open weights closed the ca­pa­bil­ity gap while the price of in­tel­li­gence col­lapsed.

0%

Capability gap to the top closed mod­els — at par­ity on cod­ing, be­hind on rea­son­ing

0×

Fall in GPT-4-class in­fer­ence cost in 36 months: $20 → $0.40 per 1M to­kens

01The cur­rent state of open-source AI

Parity reached. The con­test is one layer up.

Open weights are no longer a com­pro­mise. They are where the work hap­pens: a ma­jor­ity of pro­duc­tion to­kens now route through them, and the five high­est-vol­ume mod­els on OpenRouter are all open. Closed mod­els still lead at the fron­tier, on rea­son­ing and mul­ti­modal­ity, but the fron­tier is not what most work­loads need. Commodity in­puts do not hold pric­ing power. Value moves up, to the agen­tic har­ness.

The ca­pa­bil­ity gap: 8.04% → 0.5% → 3.3%

Open-vs-closed gap on Chatbot Arena over 24 months. By August 2024, the gap had col­lapsed to 0.5%, and in February 2025 DeepSeek-R1 briefly matched the top US model. By March 2026 it had re­opened to 3.3% as closed rea­son­ing mod­els pulled ahead. But 3.3% is an av­er­age over a jagged fron­tier: open is at or near par­ity on cod­ing, in­struc­tion-fol­low­ing and gen­eral knowl­edge, while the gap con­cen­trates in rea­son­ing, long-con­text re­trieval and agen­tic tasks. The ques­tion is no longer whether open mod­els are good enough. It’s what you need for your work­load. Hover the points.

Source: Chatbot Arena, Jan 2024 — Mar 2026.

Inference fell 50× in 36 months

GPT-4-equivalent price per 1M to­kens — faster than dot­com-era band­width or PC-compute price curves. Log scale.

Sources: Stanford HAI AI Index 2025 (280× GPT-3.5-class drop over 18 months); Epoch AI (9 – 900× an­nual de­cay); Nov 2025 MIT study (5 – 10×/yr at the fron­tier, hard­ware-ad­justed).

Open weights win the to­kens

The share of to­kens routed on OpenRouter through open-weight mod­els grew from a neg­li­gi­ble base to a third by late 2025 to a ma­jor­ity by mid-2026.

Source: OpenRouter 100T-token study (Nov 2024–Nov 2025) and live leader­board; in­ter­me­di­ate points in­ter­po­lated. By re­quest count, closed US providers still lead — the open lead is a to­ken-vol­ume lead, con­cen­trated in cod­ing and agen­tic work­loads.

OpenRouter live leader­board — trail­ing month, to­kens routed

The five high­est-vol­ume mod­els are all open weights. Anthropic’s closed Claude mod­els are the next US-built en­trants.

Open weightsClosed

By mid-2026 the top nine mod­els route roughly 18T weekly to­kens for Chinese-built mod­els against ~5.5T for US-built ones — more than 3:1 (FT analy­sis). Where de­vel­op­ers route by cost, they route to open weights.

Open ships easy.Open de­ploys hard.

Data from the Mozilla / SlashData 2026 de­vel­oper sur­vey. Open mod­els lead in adop­tion: 79% of de­vel­op­ers adding AI func­tion­al­ity use them, against 71% for closed, and the two are largely com­ple­men­tary, with half of de­vel­op­ers us­ing both. But pro­duc­tion is where teams stall: only 51% of open-model teams reach pro­duc­tion ver­sus 63% for closed. The gap is op­er­a­tional tool­ing and trust, not model ca­pa­bil­ity.

Open mod­els lead in adop­tion, and mostly co­ex­ist with closed

Share of de­vel­op­ers adding AI func­tion­al­ity to their ap­pli­ca­tions who cur­rently use each model type, and how the two over­lap.

Open mod­els

79%

Closed mod­els

71%

How they com­bine

29%OS only

50%Both

21%CS only

Source: Mozilla / SlashData 2026 de­vel­oper sur­vey. Open and closed aren’t sub­sti­tutes for most teams: 50% run both, 29% open only, 21% closed only.

Where open adop­tion peaks, and where closed still edges it

Open-model adop­tion by re­gion. Greater China and East Asia lead at 89%; South America and Western Europe are the only two re­gions where closed adop­tion ex­ceeds open.

Same sur­vey, by de­vel­oper re­gion. In South America and Western Europe, and only there, closed-model adop­tion runs ahead of open.

Production rate by com­pany size

If the gap were about re­sources, scale would close it, and it does­n’t. Closed climbs 54% → 73% with scale. Open barely moves: 53% → 57%.

Closed mod­el­sOpen mod­els

Enterprises can buy their way through closed de­ploy­ment. Open de­ploy­ment waits on tool­ing no­body has fin­ished. Source: Mozilla / SlashData 2026 de­vel­oper sur­vey.

Why teams churn: chal­lenges with open mod­els

Δ = churned − still us­ing, in per­cent­age points. The biggest gaps (performance, in­te­gra­tion, main­te­nance) are op­er­a­tional, not ca­pa­bil­ity. Hover the bars.

Still us­ing openChurned away

Mozilla sur­vey, n=1,410. “What are the main chal­lenges you face when work­ing with open or open-source AI mod­els?”

The same chal­lenges, every­where: what blocks open by re­gion

Share of cur­rent and churned open-model de­vel­op­ers nam­ing each chal­lenge, by re­gion. Warmer cells mean more de­vel­op­ers blocked. The top rows are op­er­a­tional in every re­gion: in­fra­struc­ture cost, se­cu­rity and com­pli­ance, main­te­nance, de­ploy­ment com­plex­ity. South Asia leans hard­est on se­cu­rity and sup­port; only North America and Greater China have more than 15% re­port­ing no ma­jor chal­lenges.

Source: Mozilla / SlashData 2026 de­vel­oper sur­vey (MZCS1). n=1,410 cur­rent or churned open-model de­vel­op­ers; the Oceania col­umn (n=39) and Eastern Europe & CIS (n=98) fall be­low re­li­able thresh­olds.

02The open-source AI stack

The open stack scores high on ca­pa­bil­ity,low on op­er­a­tions.

Nine lay­ers and 48 com­po­nents of the stack scored across 10 cri­te­ria (1 – 5). Click a layer to open its com­po­nents: each car­ries its own cri­te­rion scores, ma­tu­rity grade, open-vs-closed par­ity ver­dict, and sur­faces some of its most-starred open-source pro­jects.

Hover any cell for de­tail.

StrongViable, but frag­ment­edEarly stage

Strong (≥4.0) 3.5 – 3.9 3.0 – 3.4 2.5 – 2.9 Weak (<2.5) the op­er­a­tional gap = stan­dard­iza­tion + en­ter­prise readi­ness

Cells are scores per ma­tu­rity cri­te­rion (1 – 5), or­dered strongest to weak­est left to right; layer rows are the means of their com­po­nents. The two cold­est columns, stan­dard­iza­tion and en­ter­prise readi­ness, re­peat down every layer and every com­po­nent: that re­peat­ing cold edge is the op­er­a­tional gap. Source: Mozilla stack map, June 2026 (48 com­po­nents, 1,361 pro­jects).

03Who’s bet­ting on it

Open source is a busi­ness model.

Open-weight AI is a com­mer­cial mar­ket at multi-hun­dred-bil­lion-dol­lar scale, built by funded com­pa­nies and run in pro­duc­tion by global en­ter­prises. Databricks crossed a $5.4B run-rate; Mistral scaled 20× to ~$400M ARR in twelve months; DeepSeek reached ~$220M ARR and re­cently raised $7.4B at a val­u­a­tion over $50B. Five rev­enue mod­els are proven at scale: hosted in­fer­ence, en­ter­prise plat­forms, on-prem li­cens­ing, fine-tun­ing ser­vices, and har­ness tool­ing.

The ven­ture-funded open-source ecosys­tem: to­tal dis­closed fund­ing, USD M

Bars grow as you scroll. Color by re­gion of the com­pany.

North AmericaChinaEurope & rest of world

Selected com­pa­nies; Zhipu AI and MiniMax went pub­lic (HK IPO 2026) with undis­closed to­tals. Corporate strate­gics (Nvidia, Salesforce, AMD, Google, IBM, ASML, Tencent, CATL, Schwarz Group) back the same ecosys­tem across model, in­fer­ence, and tool­ing lay­ers.

Financial ma­tu­rity of the open ecosys­tem

Funding, val­u­a­tion and rev­enue trac­tion for the com­pa­nies car­ry­ing the open stack. The ecosys­tem has moved from grants to ven­ture scale to pub­lic mar­kets.

Five rev­enue mod­els are proven at scale: hosted in­fer­ence, en­ter­prise plat­forms, on-prem li­cens­ing, fine-tun­ing ser­vices, and har­ness tool­ing. “—” = not pub­licly dis­closed.

The me­tered model breaks at scale

Closed fron­tier mod­els are sold by the to­ken — and at pro­duc­tion scale the me­ter be­comes the prob­lem.

A fifth of the us­age, 4% of the rev­enue

On OpenRouter (May–Sep 2025), closed mod­els held ~80% of us­age and ~96% of rev­enue. Price dri­ves it: at ~90% par­ity, closed costs ~6× more per call.

~$24.8B

in un­re­al­ized an­nual sav­ings — the Nagle–Yue study for the Linux Foundation’s es­ti­mate of the open-vs-closed price asym­me­try, at ~6× the cost per call for com­pa­ra­ble ca­pa­bil­ity

Where de­vel­op­ers route by cost, they route to open weights.

04Why it’s hap­pen­ing every­where

Open is­n’t a ven­dor choice.It’s a sov­er­eignty choice.

More than 70 na­tional AI strate­gies are live. The strate­gic ques­tion has shifted from whether to have a na­tional AI pol­icy to which layer of the stack a coun­try can own.

Click a marker or a coun­try be­low.

The case for open is op­tion­al­ity

Optionality stopped be­ing ab­stract in June 2026, and it stopped be­ing a pro­cure­ment ques­tion. Three days af­ter Claude Fable 5 went on sale, a sin­gle gov­ern­men­t’s ex­port or­der forced Anthropic to cut ac­cess for every for­eign na­tional on earth. No other cap­i­tal was con­sulted. None could have been. Selective com­pli­ance was im­pos­si­ble, so the mod­els went dark for every­one at 5:21 p.m. on a Friday. Anyone who had built on that model in­her­ited a shut­down they had no warn­ing of and no part in. A provider can switch off a model. Nobody can switch off a copy al­ready run­ning on a ma­chine you hold, and that holds whether the ma­chine is a star­tup’s server or a na­tional su­per­com­puter. For a com­pany, weights on disk are a hedge. For a state, they are the dif­fer­ence be­tween a pol­icy and a per­mis­sion.

The strate­gic case for open is the abil­ity to leave, and the cloud era proved the cost of its ab­sence:

$90 – 120kto move one petabyte out of AWS S3

80%of en­ter­prises now repa­tri­at­ing work­loads

$3.2M → <$1M37signals’ cloud bill af­ter leav­ing

2.5×what GEICO’s cloud costs ran over plan

Closed model APIs re­pro­duce the same trap: build on a pro­pri­etary end­point and you in­herit the ven­dor’s pric­ing changes with no clean exit. Open weights are exit rights.

The largest source of open weights is China. By de­sign.

Cumulative Hugging Face down­loads, March 2026:

In February 2026 Qwen out-down­loaded the next eight or­ga­ni­za­tions com­bined. On OpenRouter, Chinese open-weight mod­els rose from un­der 2% of to­kens in late 2024 to more than 45% of weekly traf­fic by April 2026, and about 61% among the ten most-used mod­els. DeepSeek re­ports 26,000+ en­ter­prise ac­counts; 58% of new AI star­tups in 2025 in­cluded it in their stack, even as at least eight ju­ris­dic­tions re­stricted the hosted ser­vice. The res­o­lu­tion is ar­chi­tec­tural: en­ter­prises ban the hosted app and adopt the weights any­way, self-hosted or via Western end­points.

This is in­ten­tional pol­icy. The State Council’s “AI Plus” Initiative (Aug 2025) and the na­tional Five-Year Plan (Mar 2026) cod­ify open-source pro­lif­er­a­tion as a core di­rec­tive, and re­leas­ing pub­lic weights dou­bles as a macro hedge against semi­con­duc­tor ex­port con­trols, of­fload­ing global in­fer­ence onto end users’ lo­cal hard­ware. Across the Global South the draw is di­ver­si­fi­ca­tion away from US tech­nol­ogy mo­nop­o­lies; else­where it is purely fi­nan­cial. Even Microsoft is ex­plor­ing a se­cured, Azure-hosted DeepSeek V4 for its heav­i­est Copilot work­load.

Marker size ≈ scale of com­mit­ted pub­lic/​strate­gic cap­i­tal · Equirectangular pro­jec­tion

Source: Open Source AI ju­ris­dic­tions dataset, July 2026. Marker size scales with com­mit­ted pub­lic and strate­gic cap­i­tal.

05The har­ness is the new fron­tier

The agen­tic har­ness is an­other user agent.

The browser was the user agent of the open web: code on the user’s side, ne­go­ti­at­ing with servers on their be­half. That role is be­ing recre­ated one layer up. Above the model now sits the agen­tic har­ness — the or­ches­tra­tion loop, tools, mem­ory, sand­boxes, and per­mis­sion model. It is where pro­duc­tion dif­fi­culty con­cen­trates, and where the open-vs-closed, owner-vs-renter con­test restarts.

The user · other agents · the world­hu­mans · sys­tems · data · money

First at­mos­phere found on Earth-like planet in hab­it­able zone of dis­tant star

12 hours ago

Pallab GhoshScience Correspondent

Melissa Weiss/Center for Astrophysics |Harvard & Smithsonian

Researchers have found the first at­mos­phere sur­round­ing an Earth-like, rocky planet or­bit­ing within the hab­it­able zone of a dis­tant star.

The re­searchers say that their dis­cov­ery pro­vides the strongest ev­i­dence yet that worlds with con­di­tions sim­i­lar to Earth could ex­ist be­yond our so­lar sys­tem.

The gas de­tected in the at­mos­phere is he­lium, which would not be able to sup­port life, but other gasses may also be pre­sent.

The lead au­thor, Dr Collin Cherubim of Harvard University, de­scribed the dis­cov­ery as “a big deal”.

“This is the first time any­one has found an at­mos­phere on a rocky planet in the hab­it­able zone of an­other star.”

The planet, called LHS 1140 b, is 48 light-years from Earth or­bit­ing a red star much smaller and cooler than our Sun.

More than 6,000 worlds have been dis­cov­ered or­bit­ing dis­tant stars. But the new dis­cov­ery is sig­nif­i­cant be­cause it brings us a step closer to one of the biggest prizes in sci­ence: the dis­cov­ery of life on an­other world.

The re­searchers, writ­ing in the jour­nal Science, are clear — they have not done that, at least not yet. But for a planet to sup­port life it has to have wa­ter and for that it has to be the right dis­tance from its star: not too close be­cause it will be too hot and not too far, be­cause it will be too cold — but some­where in be­tween where it will be “just right”.

Planetary sci­en­tists call this the “Goldilocks zone”, af­ter the fairy tale girl who was fussy about the tem­per­a­ture of her por­ridge.

Hundreds of plan­ets have been found in the Goldilocks zones of their re­spec­tive stars — but only a few dozen are small and rocky — like our own Earth — which is an­other tick for a plan­et’s abil­ity to sup­port life.

But none of those have been found to have an at­mos­phere.

Until now.

But the only gas dis­cov­ered in the at­mos­phere so far is he­lium, prob­a­bly in the up­per at­mos­phere, which on its own would not sup­port life.

But there may be other, more life-sus­tain­ing gases, lower down. Dr David Charbonneau, also from Harvard, said that the im­por­tant thing was the dis­cov­ery of an Earth-like planet out­side of our so­lar sys­tem with an at­mos­phere.

“People are gen­er­ally in­ter­ested in the big ques­tions: Are we alone? Is there life be­yond the Earth or be­yond our so­lar sys­tem? To that end, this study re­veals the first at­mos­phere dis­cov­ered on a rocky planet in the hab­it­able zone of a star out­side of our so­lar sys­tem,” he said.

LHS 1140b is­n’t the only world un­der scrutiny in the search for life. K2 – 18b, a sub-Nep­tune with a pos­si­ble wa­ter-rich in­te­rior, made head­lines when sci­en­tists spot­ted signs of di­methyl sul­phide — a gas linked to ma­rine life on Earth.

But a Nasa-led re­analy­sis in 2025 found the sig­nal too weak to con­firm, and showed the gas can form with­out bi­ol­ogy.

The seven rocky worlds of TRAPPIST-1 re­main tan­ta­lis­ing, too: Nasa’s James Webb Space Telescope ruled out an Earth-like at­mos­phere on TRAPPIST-1d, while TRAPPIST-1e’s data stay frus­trat­ingly in­con­clu­sive.

Yet an­other thought piece about LLMs. I know. Bear with me.

This is an at­tempt to put words around some­thing I think most de­vel­op­ers are ex­pe­ri­enc­ing right now but haven’t had time to make sense of. Programming with LLMs is gen­uinely use­ful and gen­uinely desta­bi­liz­ing. These two things co­ex­ist. If we pre­tend the sec­ond one is­n’t hap­pen­ing, we will all burn out.

At Pydantic, we build tools that de­vel­op­ers use to val­i­date data, build AI agents, and ob­serve what their sys­tems are do­ing in pro­duc­tion. We are, quite lit­er­ally, in the busi­ness of mak­ing LLM-powered soft­ware more re­li­able. And we are also hav­ing a weird time.

This is­n’t a think­piece about whether AI will re­place pro­gram­mers. It’s not a doomer es­say and it’s not a hype piece. It’s an hon­est ac­count of what it feels like to be a de­vel­oper right now, from some­one in­side it, and some thoughts on what might ac­tu­ally help.

Hands in the fab­ric

When I was first learn­ing to code in my early twen­ties, I re­mem­ber hav­ing this dis­tinct sen­sa­tion that pro­gram­ming let me dip my hands into the fab­ric of the uni­verse and shape it to my will. This was, of course, be­fore I’d hit too many com­pile er­rors. But that feel­ing of touch­ing some deep fun­da­men­tal layer of ab­strac­tion, of be­ing able to make things from noth­ing but logic, has al­ways stuck with me.

I’m not a Computer Science grad­u­ate. I’m a de­signer and a pro­gram­mer — for­mally trained in the first, self-taught in the sec­ond. I came to the for­malisms of soft­ware en­gi­neer­ing through painful ex­pe­ri­ence rather than aca­d­e­mic in­struc­tion. If any­thing, that made me take those prin­ci­ples more se­ri­ously once I un­der­stood them. When you’ve earned your opin­ions about ar­chi­tec­ture and code qual­ity the hard way, they feel less like text­book rules and more like scar tis­sue.

That pri­mal feel­ing of cre­ation? It’s the same promise that the low-code and no-code tools of the 2010s kept mak­ing but never quite de­liv­ered on. I’m old enough to re­mem­ber build­ing web pages in Dreamweaver, watch­ing Adobe spruik zero-code de­sign tools that gen­er­ated ab­solute spaghetti un­der the hood. It was al­ways al­most there, just good enough to hint at a fu­ture that was just around the cor­ner (if only you were smart enough to grasp it).

If you’re cyn­i­cal about the cur­rent wave of AI tools, I get it. We’ve been promised this be­fore. But this time the gap be­tween promise and re­al­ity has ac­tu­ally, fi­nally, nar­rowed to some­thing mean­ing­ful. And that’s ex­actly what makes it so un­set­tling.

What “the code writes it­self” ac­tu­ally feels like

Yes the code (sorta) writes it­self, but the hu­man re­view­ing, di­rect­ing, and course-cor­rect­ing feels worse, not bet­ter.

I re­cently had a con­ver­sa­tion with my col­league Douwe, who main­tains the Pydantic AI frame­work and has been one of the most thought­ful peo­ple I know about in­te­grat­ing LLMs into open source work­flows. He de­scribed wak­ing up to thirty PRs every morn­ing, each one pulled overnight by some­one’s AI, and need­ing to make snap judg­ment calls on every sin­gle one. The temp­ta­tion to del­e­gate the re­view it­self to an AI was enor­mous. But, as he put it: “at that point, what am I still do­ing here?”.

The hon­est truth is that in the last few months, there have been days when I have spent close to two full days writ­ing a plan for an LLM to ex­e­cute: ob­ses­sively clar­i­fy­ing, spec­i­fy­ing, re-spec­i­fy­ing, only to have it still do some­thing in­ex­plic­a­bly stu­pid. Port a React hook into a Storybook story file. Read from the wrong plan. Invent com­po­nents that don’t ex­ist. And these aren’t er­rors of ca­pa­bil­ity; they’re er­rors of co­her­ence. The mod­els are smart enough to pro­duce plau­si­ble code, but not al­ways smart enough to main­tain a co­her­ent in­tent across a com­plex change.

This cre­ates a pe­cu­liar new kind of fa­tigue, the fa­tigue of su­per­vi­sion: of hold­ing the in­tent in your head while the ma­chine gen­er­ates vol­umes of mostly-cor­rect out­put that still needs your eyes, your judg­ment, and your taste. Douwe put it well: he used to get a dopamine hit from col­lab­o­rat­ing with a real per­son on a cool fea­ture in open source. Helping some­one be­come bet­ter at their craft. Now, he said, “everything I write goes into some AI black hole. There’s no per­son on the other side ac­tu­ally learn­ing any­thing.” That loss is real and it’s worth nam­ing.

The in­ten­sity trap

Simon Willison re­cently high­lighted a Berkeley Haas study which de­scribes how AI us­age in­creases the in­ten­sity of work. The con­stant pull of “one more prompt at the end of the day, one more fea­ture that could make this per­fect.” I felt that one in my bones. I was up un­til nearly 2am re­cently, prompt­ing, be­cause I was so close to get­ting a plan right. Or so I thought.

Marcelo, an­other Pydantic col­league, when asked about his Claude Code ses­sion freez­ing said: “just open 5 claude ses­sions. You’ll never no­tice be­cause you’re busy giv­ing feed­back to the oth­ers.” He was jok­ing. I think. But it cap­tures some­thing true about the cur­rent mo­ment. The par­al­lelism is ex­hil­a­rat­ing and kind of feral. The num­ber of things you can start has dra­mat­i­cally in­creased. The num­ber of things you can thought­fully fin­ish has­n’t changed at all, be­cause that part still re­quires the one re­source we can’t par­al­lelise: your brain.

Here’s a term for what I think is hap­pen­ing: the hu­man re­ward func­tion prob­lem. In ma­chine learn­ing, a re­ward func­tion tells an agent what good looks like. Writing code by hand was never easy, but it was full of small re­wards. Solving a prob­lem in your head. Understanding a gnarly bit of logic. Watching the code com­pile. The feel­ing of con­trol. LLM-assisted pro­gram­ming has au­to­mated much of the work that gen­er­ated those dopamine hits and re­placed it with the cog­ni­tive load of re­view and su­per­vi­sion. The sat­is­fy­ing part shrank. The ex­haust­ing part grew. And there are no new re­wards to fill the gap.

If you’re feel­ing like your work is si­mul­ta­ne­ously more pro­duc­tive and less sat­is­fy­ing, you’re not bro­ken. The feed­back loop is bro­ken. And I think we need to start treat­ing that as an en­gi­neer­ing prob­lem in its own right, not a per­sonal fail­ure.

It’s also, frankly, quite lonely. Programming with an LLM is an in­tensely soli­tary ac­tiv­ity.

You and the ma­chine, go­ing back and forth, re­fin­ing and prompt­ing and re­view­ing. The nat­ural mo­ments where you’d turn to a col­league to ask a ques­tion, to rub­ber-duck a prob­lem, to share the small vic­tory of some­thing fi­nally click­ing. Those mo­ments get qui­etly re­placed by an­other prompt. In a team with­out a strong ex­ist­ing cul­ture of col­lab­o­ra­tion, this has a ten­dency to fur­ther sep­a­rate peo­ple, to chill com­mu­ni­ca­tion at pre­cisely the mo­ment when you most need the re­as­sur­ance that other hu­mans are find­ing this hard too.

And it’s ad­dic­tive in a way that makes the iso­la­tion worse. Sometimes you get some­thing bril­liant, some­times garbage, and you never quite know which. Textbook Skinner Box. It can be gen­uinely hard to step back and re­mem­ber that you’re al­lowed to just… write code. But switch­ing be­tween LLM-assisted and man­ual work is jar­ring and un­com­fort­able, two very dif­fer­ent modes of think­ing, and it takes a kind of ma­tu­rity and con­fi­dence to give your­self per­mis­sion to switch.

Breakpoints

This mo­ment brings to mind the fear and angst caused by re­spon­sive de­sign. I was work­ing as a de­signer and fron­tend de­vel­oper at the time, fol­low­ing Ethan Marcotte and the Zeldman / A Book Apart crowd like every­one else, and I re­mem­ber how un­set­tling it felt to be told that the fixed-width lay­outs we’d all mas­tered were ba­si­cally over.

For the younger devs: there was a gen­uine cul­tural mo­ment around 2009 when web­sites moved from fixed, pixel-per­fect, mag­a­zine-style lay­outs to fluid, re­spon­sive ones. And de­sign­ers hated it. The loss of con­trol was ex­is­ten­tial for peo­ple whose en­tire iden­tity was built around pre­cise lay­outs and per­fect grids. You’re telling me the user might see my de­sign at any width? On any de­vice? That the lay­out I crafted would… flow?

Image de­sign by Jyotika Sofia Lindqvist

Image de­sign by Jyotika Sofia Lindqvist

The re­sis­tance was in­tense. And it was un­der­stand­able. People had built real ex­per­tise in a par­a­digm that was be­ing fun­da­men­tally dis­rupted. The de­sign­ers who thrived through that tran­si­tion were the ones who re­framed their skills. The eye for pro­por­tion still mat­tered. The un­der­stand­ing of hi­er­ar­chy still mat­tered. The craft did­n’t die, it evolved. What be­came less rel­e­vant was the ob­ses­sion with pixel-level con­trol. What be­came more rel­e­vant was un­der­stand­ing sys­tems, adapt­abil­ity, and de­sign­ing for un­cer­tainty.

I don’t want to over­sell this par­al­lel. Responsive de­sign played out over years. The cur­rent shift is mea­sured in months. Agencies lost clients and de­sign­ers lost gigs over the re­spon­sive tran­si­tion, but it did­n’t carry the same ex­is­ten­tial dread. The stakes are ma­te­ri­ally dif­fer­ent, and the pace is gen­uinely ex­haust­ing in a way that the re­spon­sive tran­si­tion never was. But the un­der­ly­ing pat­tern, of craft evolv­ing rather than dy­ing, of the core skills mat­ter­ing more not less, I think that holds.

Working with LLMs on code feels like a sim­i­lar in­flec­tion point. The skill is­n’t gone, it’s shift­ing. You’re not less of an en­gi­neer be­cause you did­n’t hand-write every line. But you do still need to know what good looks like, ar­guably more than ever, be­cause you’re now the qual­ity gate for a much higher vol­ume of out­put.

What sur­vives

In an era when any­one can pro­duce rea­son­able-look­ing UI and code that com­piles, the dis­tin­guish­ing mark­ers be­come: taste, nu­ance, ma­ture ar­chi­tec­tural opin­ions, and the con­trar­ian calls that come from gen­uine ex­per­tise rather than pat­tern-match­ing.

It’s no­tice­able to me that we are most suc­cess­ful guid­ing LLMs in the do­mains where we un­der­stand the code, the de­ci­sions, and the trade-offs most deeply. As we ven­ture into the shal­low ends of our skill sets, the out­puts be­come markedly more im­pres­sion­is­tic. Further from pro­duc­tion-ready. More plau­si­ble-look­ing, less ac­tu­ally cor­rect. The model does­n’t know what it does­n’t know, so it fills the gaps with con­fi­dence. Sound fa­mil­iar? It’s a very hu­man fail­ure mode, too.

But new skills are also emerg­ing. I’ve started run­ning what I call pre-mortems on com­plex plans: ask­ing a fresh LLM ses­sion to as­sume the plan has cat­a­stroph­i­cally failed and di­ag­nose why. It catches spec­i­fi­ca­tion gaps that I miss af­ter two days of be­ing too deep in the de­tails. One of our en­gi­neers built a tool that ex­tracts rules from thou­sands of his past code re­view com­ments to seed an AGENTS.md file, es­sen­tially en­cod­ing years of im­plicit en­gi­neer­ing judg­ment into in­struc­tions an LLM can fol­low. That’s not the death of ex­per­tise. That’s ex­per­tise be­ing dis­tilled.

The peo­ple who are find­ing their foot­ing right now seem to share a few traits: they have strong opin­ions earned through prac­tice, they can dis­tin­guish be­tween prin­ci­ples that still ap­ply and habits that were just band­width con­straints, and they’re will­ing to evolve their work­flow with­out aban­don­ing their stan­dards.

A view from in­side the loop

I don’t think the cur­rent wave of AI rep­re­sents the end of soft­ware en­gi­neer­ing as a pro­fes­sion. I do think it rep­re­sents a se­ri­ous con­trac­tion and a fun­da­men­tal re­shap­ing of what the work is. The fear of ob­so­les­cence is le­git­i­mate. The fear of skill rot is le­git­i­mate. And the fear that if you don’t go fast enough you’ll be left be­hind is — while of­ten over­stated — not en­tirely un­founded.

But the bot­tle­neck was never the code. It was al­ways the hu­man at­ten­tion, the en­gi­neer­ing judg­ment, the abil­ity to hold a co­her­ent vi­sion for a sys­tem. We just did­n’t no­tice be­cause writ­ing code felt like the hard part. Now that it’s be­ing au­to­mated, those hu­man ca­pac­i­ties are re­vealed as the ac­tual scarce re­source. And scarce re­sources are valu­able.

So if you’re feel­ing over­whelmed, desta­bi­lized, si­mul­ta­ne­ously more pro­duc­tive and less happy, know that you’re not alone. The team build­ing the tools you’re prob­a­bly us­ing to nav­i­gate this mo­ment is feel­ing it too. We’re de­bug­ging our re­ward func­tions in real time, same as you.

The code is chang­ing. What we do with it is chang­ing. How it feels is… a work in progress.

But the hu­mans are still in the loop. We’re just tired. And that’s worth talk­ing about.

We’re build­ing tools to make this less chaotic: Pydantic AI and Logfire. We’re also hir­ing.

16th July 2026

Chinese AI lab Moonshot AI an­nounced Kimi K3 this morn­ing, de­scrib­ing it as their “most ca­pa­ble model to date, with 2.8 tril­lion pa­ra­me­ters”. It’s cur­rently avail­able via their web­site and API, but an open weight re­lease is promised “by July 27, 2026”.

Moonshot are call­ing this the first “open 3T-class model” (I guess they’re round­ing 2.8 tril­lion up to 3 tril­lion), tak­ing the crown from DeepSeek’s 1.6T v4 Pro. Their self-re­ported bench­marks have K3 mostly beat­ing Claude Opus 4.8 max and GPT-5.5 high, while los­ing out to Claude Fable 5 and GPT-5.6 Sol.

A few high­lights from the Artificial Analysis re­port on the model:

“On our pri­vate long-hori­zon knowl­edge work eval­u­a­tion, Kimi K3 reaches an over­all Elo of 1547, +732 points from Kimi K2.6 and be­hind only Claude Fable 5.”

“Cost per task ($0.94) is sim­i­lar to GPT-5.6 Sol ($1.04), ~1/2 the price of Opus 4.8 ($1.80) and higher than open weights peers”

“Kimi K3’s to­ken us­age on the Artificial Analysis Intelligence Index de­creased sig­nif­i­cantly, us­ing 21% fewer out­put to­kens than K2.6.”

The model is also now the lead­ing model on Arena.ai’s Frontend Code arena, sur­pass­ing even Claude Fable 5.

The new model is no­table for the pric­ing: $3/million in­put to­kens and $15/million out­put to­kens, putting it at the same level as Anthropic’s Claude Sonnet se­ries and mak­ing it the most ex­pen­sive model re­leased by a Chinese AI lab to date. This is a sig­nif­i­cant in­crease on their ear­lier mod­els such as Kimi K2.6 at $0.95/$4. 2.8 tril­lion pa­ra­me­ters is also more than twice the size of that 1T model.

But how does it pel­i­can?

I used OpenRouter (to avoid sign­ing up for a Moonshot API key) with the llm-open­router plu­gin to gen­er­ate an SVG of a pel­i­can rid­ing a bi­cy­cle:

llm -m open­router/​moon­shotai/​kimi-k3 ‘Generate an SVG of a pel­i­can rid­ing a bi­cy­cle’

Here’s the tran­script. It looks like this:

That pel­i­can took 95 in­put to­kens and 16,658 out­put to­kens (13,241 were rea­son­ing to­kens), for a to­tal cost of 25 cents!

Since K3 ac­cepts im­age in­put I ran it against that ren­dered SVG above (with my alt text prompt) and got back (for 0.6 cents):

Cartoon il­lus­tra­tion of a white pel­i­can wear­ing a red scarf, rid­ing a red bi­cy­cle along a gray road with white dashed lines; the pel­i­can has a large or­ange beak and webbed or­ange feet ped­al­ing, with white mo­tion lines be­hind it; the back­ground shows a light blue sky with white clouds, a yel­low sun, two small black birds in flight, and green grass with tiny white flow­ers in the fore­ground

Cartoon il­lus­tra­tion of a white pel­i­can wear­ing a red scarf, rid­ing a red bi­cy­cle along a gray road with white dashed lines; the pel­i­can has a large or­ange beak and webbed or­ange feet ped­al­ing, with white mo­tion lines be­hind it; the back­ground shows a light blue sky with white clouds, a yel­low sun, two small black birds in flight, and green grass with tiny white flow­ers in the fore­ground

What can we learn from the pel­i­can?

My Generate an SVG of a pel­i­can rid­ing a bi­cy­cle test is 21 months old now. It was never a par­tic­u­larly great bench­mark. It started out as a joke on how ab­surdly dif­fi­cult it is to com­pare these mod­els, but then for the first year it turned out to have a sur­pris­ing cor­re­la­tion to how good the mod­els ac­tu­ally were.

That con­nec­tion has been mostly sev­ered now. The GPT-5.6 and Claude Fable 5 pel­i­cans are out­classed by GLM-5.2, and much as I love GLM I don’t think that’s a Fable-class model.

(I’m still not con­vinced that labs are train­ing for the bench­mark—if they were, I’d ex­pect much bet­ter re­sults. There’s a chance that Gemini has op­ti­mized for any com­bi­na­tion of an an­i­mal on a ve­hi­cle though!)

The biggest lim­i­ta­tion of the pel­i­can is that it does­n’t touch at all on the thing that mat­ters most for to­day’s model: agen­tic tool call­ing and the abil­ity to op­er­ate tools re­li­ably as con­ver­sa­tions grow in length.

So don’t go us­ing pel­i­cans to com­pare mod­els!

All of that said, I still get a de­cent amount of value out of run­ning the bench­mark my­self.

Firstly, it’s a forc­ing func­tion for ac­tu­ally try­ing the model. If I show you a pel­i­can, that means I’ve man­aged to run a prompt through it. If the model has an of­fi­cial API I’ll use that, if it’s open weight (and small enough to fit a 128GB M5 MacBook Pro) I’ll try run­ning it on my own ma­chine, usu­ally via llama.cpp or LM Studio or Ollama. I’ll fre­quently use OpenRouter since that usu­ally pro­vides a proxy to an of­fi­cial API with­out me need­ing a new API key.

Most of my pel­i­cans are gen­er­ated us­ing my LLM CLI tool, which helps en­cour­age me to en­sure the lat­est mod­els are sup­ported by that (via one of its plu­g­ins).

More im­por­tantly though, even the act of a sin­gle prompt to “Generate an SVG of a pel­i­can rid­ing a bi­cy­cle” can re­veal in­ter­est­ing model char­ac­ter­is­tics.

Consider the re­sult for Kimi K3 to­day. Running those sim­ple prompts helped em­pha­size sev­eral points about the model.

It only has one rea­son­ing ef­fort right now, “max”—and it shows. The model con­sumed 13,241 rea­son­ing to­kens to out­put 3,417 to­kens of re­sponse. This is ex­pen­sive—the pel­i­can cost 25 cents!

How does the prompt “Generate an SVG of a pel­i­can rid­ing a bi­cy­cle” add up to 95 in­put to­kens? OpenAI’s to­k­enizer counts 10, Anthropic’s counts 10 for Opus 4.6, 30 for Opus 4.7 and 25 for Sonnet 5/Fable 5. Prompting “hi” to Kimi K3 counted 86 to­kens, sug­gest­ing there may be an 85 to­ken hid­den sys­tem prompt. It re­fused to leak it though.

Vision works well: the alt text it gen­er­ated is very good.

K3 cur­rently only has one think­ing ef­fort level, but I’ve been de­riv­ing quite a bit of value re­cently from run­ning the same pel­i­can prompt through dif­fer­ent ef­fort lev­els to get a quick idea for what im­pact those have. Here’s my ma­trix for the GPT-5.6 model fam­ily, for ex­am­ple.

Really though the main things I gain from the pel­i­can test are:

It’s a “hello world” ex­er­cise for prompt­ing a model

A rough cost and rea­son­ing es­ti­mate for a sim­ple task

Confirmation that the model can out­put valid SVG and has a ba­sic idea of geom­e­try and spa­tial aware­ness. This is a much big­ger deal for the smaller mod­els that run on my lap­top.

It’s still in­ter­est­ing to com­pare pel­i­cans be­tween re­leases in the same model fam­ily. K3’s pel­i­can is a no­table im­prove­ment from Kimi 2.5.

It’s some­thing I can share that demon­strates I’ve tried it. Plus a com­ment with a pel­i­can in it is kind of a tra­di­tion on Hacker News at this point, any time I’m late I get com­ments ask­ing where it is!

TL:DR;

#Pebble Time 2 Shipping Status

Since we started mass pro­duc­tion in late March, we’ve built over 23,000 Pebble Time 2 watches. We’re over 80% of the way through ful­fill­ing all the pre-or­ders we’ve re­ceived! But that means there are still some ul­tra pa­tient folks who haven’t re­ceived their watches yet. If you’ve placed an pre-or­der for PT2 and haven’t re­ceived it yet (including Batch 6 - August), here’s when we ex­pect to ship your watch out:

Pebble Time 2 - Black → July 31

Pebble Time 2 - Red → July 31

Pebble Time 2 - Grey → July 28

Pebble Time 2 - Blue → July 28

Coincidentally, this means that we’ll be ‘in-stock’ with no wait very soon! If you’ve been hold­ing off plac­ing an or­der be­cause you did­n’t want to wait, now is the time to jump on it. This won’t last for­ever - first-come first serve. As soon as the cur­rent in­ven­tory is sold out, we’ll be back in pre-or­der mode wait­ing for the next ship­ment.

Order to­day on rePeb­ble.com/​watch.

Major props to our three per­son cus­tomer sup­port and lo­gis­tics team! Claudio, Trevor and Colin have an­swered thou­sands of your ques­tions and helped ship watches safely onto your wrist in 93 coun­tries. Have a ques­tion? Please check out our Help site first. If that does­n’t have an an­swer, please email us at [email protected].

Want an ex­tra Pebble charger? shop.repeb­ble.com now car­ries ac­ces­sories - full se­lec­tion of straps com­ing soon.

#Pebble Software - Progress and Roadmap

Over the last 6 months, the core four per­son Pebble soft­ware team built and shipped a met­ric ton of new Pebble open source soft­ware! Our im­prove­ments were cen­tered around these ar­eas:

Battery life

We’ve (well, mostly Gerard 🙂) worked ex­tra­or­di­nar­ily hard over the last few months, op­ti­miz­ing and re­duc­ing power con­sump­tion in PebbleOS. As pre­dicted, we boosted the me­dian bat­tery life of Pebble 2 Duo from 17 days (last sum­mer) to over 30 days. Pebble Time 2 me­dian is cur­rently around 21 days - more im­prove­ments in the works here too! The biggest con­sumers of power are back­light, watch­faces with a lot of an­i­ma­tions and health track­ing. If you want to ‘hypermile’ your Pebble, try switch­ing to a low-an­i­ma­tion watch­face and the new Battery Saver back­light mode (Settings → Display → Backlight).

Apps and SDK

Together with the Moddable team, we’ve pub­lished sev­eral Pebble SDK up­dates in­tro­duc­ing new fea­tures like:

Touch Screen API (Calculator on your wrist any­one?)

Speaker API (useful for tun­ing your gui­tar, or feed­ing your Tamagotchi)

RGB Backlight API (try it in this wild lit­tle app Chinese Toy Phone)

Apps can now de­ter­mine how they were quick launched (ie by sin­gle press, long press)

Alloy (native JS apps)

FFI - run C code within Alloy JS apps (similar to Android NDK) and js de­bug­ger A bunch of new JS APIs peb­ble build –debug now de­fines PBL_DEBUG and launches XSBUG, a pow­er­ful JS de­bug­ger

FFI - run C code within Alloy JS apps (similar to Android NDK) and js de­bug­ger

A bunch of new JS APIs

peb­ble build –debug now de­fines PBL_DEBUG and launches XSBUG, a pow­er­ful JS de­bug­ger

Developers in the Pebble com­mu­nity have cre­ated 2,120 apps and watch­faces for Pebble Time 2 and Pebble Round 2 al­ready!

Index 01

The first ver­sion of all Index 01 func­tion­al­ity is up and run­ning in­side the Pebble mo­bile app. Don’t have an Index 01 yet? You can check out how it works and try the soft­ware in­ter­face in the Pebble app, just go to Settings → General → Enable Index feed.

All the main fea­tures are in, in­clud­ing sync­ing to iOS Reminders, Obsidian, Google Tasks, Calendar, Android mu­sic con­trol, MCPs and send­ing record­ings or tran­scrip­tions to your own server or app via Webhook. Optional en­cryp­tion (you own the keys) pro­tects op­tional cloud backup. And of course, it’s all open source (github.com/​core­de­vices/​mo­bileapp). We even built a lit­tle we­bapp that you can use to ac­cess your Index in­for­ma­tion from any­where → in­dex.rePeb­ble.com. Watch the pod­cast or read the blog post to learn more.

Stability

Thanks to help­ful bug re­ports from y’all, we’ve made hun­dreds of small im­prove­ments to PebbleOS and the Pebble mo­bile app. Please keep it com­ing!

I’ll dive into one spe­cific (and ul­tra tech­ni­cal) topic - re­verse PPoGATT (Pebble Protocol over GATT). Quick his­tory: dur­ing the first Pebble era, we con­fig­ured the Pebble mo­bile app to ex­pose a PPoGATT ser­vice, as means to work around the lack of IPC be­tween iOS apps. This setup is the op­po­site of how Bluetooth ac­ces­sories nor­mally con­nect to phones and caused a num­ber of weird prob­lems! Also this setup blocks us from us­ing iOS AccessorySetupKit (ASK), which is a pre­req­ui­site for us to im­ple­ment the new Notification Forwarding fea­ture (EU only) that will fi­nally en­able you to re­ply to no­ti­fi­ca­tions. Enabling ASK is go­ing to be tough - our iOS app must ei­ther use ASK or not, mean­ing that we need to up­grade the re­cov­ery firmware on all Pebble watches in the field to re­verse PPoGATT be­fore we can switch ASK on. Anyways, we have the first piece of the puz­zle in place (Pebble Round 2’s re­cov­ery firmware al­ready has the up­grade). This saga will take a while.

Community Contributions

Thank you to the dozens of de­vel­op­ers from the broader Pebble com­mu­nity who have con­tributed huge im­prove­ments to PebbleOS and the mo­bile app, in­clud­ing Apple HealthKit and Google health sync, im­proved light sen­sor al­go­rithms, no­ti­fi­ca­tion fil­ter­ing, many new lan­guage packs, and so many bug fixes. It’s so fun and very en­er­giz­ing to see so many tal­ented hack­ers push PRs! See the full list and thank you devs! Some ex­cit­ing new com­mu­nity built fea­tures are on the hori­zon: HRV, SP02, ex­pos­ing HRM via BLE, mic API, mul­ti­ple BLE clients and more

Software Roadmap

Your browser does not sup­port the video tag.

We keep im­prov­ing Pebble soft­ware pri­mar­ily be­cause we are Pebble users. We love us­ing the prod­ucts we make and con­tin­u­ally want to make them bet­ter! Here’s some of the things we’re ex­cited to work on next:

Send text app (Android only)

Find my phone

Beautiful new weather app for PT2 and PR2 (created by grim, a win­ner of the Spring Developer Contest)

Tweaking PebbleOS UI for Round 2

Improving the Pebble mo­bile app UI

WYSIWYG watch­face ed­i­tor - spir­i­tual suc­ces­sor to Pebble Canvas

Continue tran­si­tion to fully re­verse PPoGATT role to en­able ASK and (eventually) replies to no­ti­fi­ca­tions for iOS users (in EU)

See be­low for in­dex roadmap

#Pebble Time 2 - Problems You’ve Reported

Thank you all for re­port­ing any bugs or is­sues you’ve spot­ted! We test each watch at the fac­tory be­fore it’s shipped out, and we test each soft­ware re­lease in­ter­nally and with a grow­ing team of beta testers (want to join? Sign up at rePeb­ble.com/​ac­count). But these tests are not in­fal­li­ble and we will make mis­takes. We ap­pre­ci­ate your re­ports as they help us get more in­for­ma­tion to help us fix prob­lems!

Software Issues

We’re track­ing three big soft­ware is­sues with PebbleOS, and a mul­ti­tude of smaller prob­lems. While we are ac­tively work­ing on fix­ing these with a fu­ture soft­ware , we don’t have an ETA on when these will be fixed.

Step and sleep track­ing met­rics are not ac­cu­rate for some peo­ple

Accelerometer some­times stops work­ing

Touch screen some­times stops work­ing or reg­is­ters touches in wrong lo­ca­tion

It would be tough to list here the long-tail of soft­ware is­sues we’ve had re­ported. But please note that while we don’t re­ply to every­one, we do read every sin­gle re­port and look for pat­terns and clues that help us fix many is­sues with each soft­ware up­date (see the changelog for PebbleOS and Pebble mo­bile app).

Hardware Issues

You’ve all demon­strated in­cred­i­ble pa­tience wait­ing for your PT2 to ship. You’re ex­cited to try the first brand new Pebble in the last 10 years. That’s why we un­der­stand how painful and dif­fi­cult it could be if you un­box your brand new watch and dis­cover man­u­fac­tur­ing flaw, or use it for a few weeks and find the bat­tery is dy­ing too quickly or ac­ci­den­tally crack the glass. It sucks!

We feel your pain, even more than you can pos­si­bly imag­ine. That’s why every­one who has re­ported a hard­ware is­sue to our sup­port team has re­ceived a free re­place­ment (with free world­wide ship­ping) re­gard­less of whether their de­vice is un­der war­ranty or not.

To date, we’ve re­placed 330 PT2s (out of 17.82 mil­lion hours of us­age from 19,000+ watches in the field).

Mass pro­duc­ing a con­sumer elec­tronic prod­uct is labour in­ten­sive. Making stuff is still a very hu­man-cen­tric process. We make mis­takes. A worker may not as­sem­ble a part cor­rectly. A test may be ac­ci­den­tally skipped. The test re­sult could be read in­cor­rectly. Procedures can be put in place to min­i­mize mis­takes, but the cost will rise. As with all of hard­ware prod­uct de­vel­op­ment - it’s a trade­off 🤷.

The most fre­quent hard­ware is­sue we’re see­ing is very high power con­sump­tion (less than ~3 day bat­tery life). We’ve taken apart some units and found a va­ri­ety of is­sues. To com­bat this, we’ve im­ple­mented more strin­gent power con­sump­tion test­ing on the as­sem­bly line. If you en­counter this is­sue (regardless of your war­ranty el­i­gi­bil­ity), please send us a bug re­port in the Pebble app and we can help you out!

Next most fre­quent are prob­lems with the touch panel. At first, we thought this could be a hard­ware prob­lem and re­placed around 70 watches. After re­view­ing the units with our fac­tory, we now be­lieve this could be a soft­ware bug. We’re work­ing to fix these is­sues with a soft­ware up­date - if we can’t, we’ll re­place the af­fected watches (regardless of your war­ranty el­i­gi­bil­ity).

Next up is the front glass crack­ing. We’ve had 51 re­ports so far, and we’ve sent a free re­place­ment to each per­son af­fected. If your glass has cracked, send us a video (preferably, pic­ture is ok) in a bug re­port in the Pebble app. During the lead up to mass pro­duc­tion, we per­formed ex­ten­sive en­vi­ron­men­tal test­ing - in­clud­ing drop test­ing, tum­ble test­ing, but­ton press, strap stretch and bend, ther­mal cy­cling and many other tests. All test re­sults showed nor­mal dura­bil­ity com­pared to sim­i­lar smart­watches. But if your watch glass cracks, do you care what the fac­tory test re­sults were? Or that this has hap­pened to just 0.25% of all PT2s - or once every 30+ years of us­age? Of course not - your watch just broke. That’s why we will con­tinue re­plac­ing rea­son­able re­ports of glass crack­ing for free as long as we can. At some point, we will shift to of­fer­ing a re­place­ment at a highly dis­counted amount. We are also look­ing into sourc­ing ex­tra LCM mod­ules (the en­tire front as­sem­bly - glass, touch panel, dis­play, metal top cover and back­light) and mak­ing them avail­able for folks who choose to fix their watch them­selves.

The fi­nal big cat­e­gory of hard­ware is­sue are re­ports of but­ton prob­lems (32 so far). In some cases, a small in­te­rior clip is im­prop­erly as­sem­bled, caus­ing the but­ton to pop off. We’ve ad­dressed this is­sue with changes to the pro­duc­tion line process and hope that it be­comes much less fre­quent as watches as­sem­bled af­ter the change start mak­ing their way out into the world. If you en­counter this is­sue (regardless of your war­ranty el­i­gi­bil­ity), please send us a bug re­port in the Pebble app and we can help you out!

Then we’ve had a long tail of smaller is­sues that I’m mod­er­ately em­bar­rassed by, like a re­port of the watch miss­ing screws on the bot­tom, or the front falling off. I guess these things do hap­pen!

#Pebble Round 2 - Production Update and Timeline

My cur­rent favourite watch­face - Chronology II by Nicholas Jitkoff

I posted a mini-up­date on Pebble Round 2 in June - we weren’t able to start mass pro­duc­tion in May be­cause of a cos­metic prob­lem with the stain­less steel bot­tom case (an ex­tra in­den­ta­tion made by the CNC milling ma­chine). Since then the fac­tory has re­ceived a new ver­sion of the bot­tom case and things are look­ing much bet­ter! In par­al­lel, we’ve been run­ning ex­ten­sive en­vi­ron­men­tal test­ing (including drop test­ing).

At the be­gin­ning of July, we shipped out more Pebble Round 2 watches to lucky folks who signed up for the beta test. Thanks for your help find­ing and test­ing fixes for bugs in PebbleOS!

Our plan (as of to­day July 14 - sub­ject to change) is to start mass pro­duc­ing Round 2 watches dur­ing last week of July. We’ll start ramp­ing up pro­duc­tion slowly and care­fully. Roughly 14,000 peo­ple have pre-or­dered Round 2. It will take us about 2 months to build all pre-or­dered watches. We ex­pect to fin­ish ship­ping out all pre-or­dered Round 2 watches by the end of September.

If you pre­ordered Round 2 on rePeb­ble.com/​watch, we’ll send you an email roughly 2 weeks be­fore your watch is ready to ship ask­ing you to con­firm your ad­dress, add op­tional ac­ces­sories to your or­der and pay any ad­di­tional taxes due. If you haven’t al­ready se­lected your watch colour, please do so on or­ders.rePeb­ble.com.

Each Round 2 pre-or­der in­cludes a sil­i­cone watch strap and charger. We’ve also cre­ated beau­ti­ful cus­tom leather straps for PR2 ($20 – 30), in­clud­ing brown or black soft leather straps that feel very sim­i­lar to the straps we made for the orig­i­nal Pebble Time Round.

#Index 01 - Production Update and Shipping Timeline

Since our last up­date, we ex­panded our beta test and learned a lot from the hun­dreds of will­ing test sub­jects. Thank you for your ser­vice and bug re­ports!

Index 01 is now of­fi­cially in mass pro­duc­tion! We’ve as­sem­bled sev­eral thou­sand rings so far, and have grad­u­ally be­gun ship­ping them out. Schedule has slipped slightly from our last es­ti­mate (early August), we’re now aim­ing to ship out nearly all pre-or­ders by the end of August, ex­cept for a few un­lucky size/​color vari­ants that will ship in September.

#⚠️ Important Note For Index 01 Pre-orderers ⚠️

We’ve re­ceived re­ports from testers that Index 01 may feel every so slightly smaller than the ring siz­ers. Please take the time now to recheck your ring size with the ring sizer kit. If the ring sizer feels tight on your fin­ger, is hard to get on/​off, or if you can­not eas­ily clench your hand with the sizer on, please change your size to the next larger size. When in doubt, or­der a larger size. You can al­ways ad­just a larger Index 01 to feel smaller with a foam ad­he­sive or clip but you can’t make it larger!

If you pre­ordered Index 01 on rePeb­ble.com/​in­dex, we’ll send you an email roughly 2 weeks be­fore your ring is ready to ship ask­ing you to con­firm your ad­dress and pay any ad­di­tional taxes due. If you haven’t al­ready se­lected your Index 01 size and colour, please do so on or­ders.rePeb­ble.com.

Index 01 has changed my life. There’s no way I could go back to a world with­out ex­ter­nal mem­ory for my brain. And this is just the be­gin­ning, Index 01 soft­ware is im­prov­ing every sin­gle day. I ex­cited to hear what you think of it!

Loading met­rics

Loading met­rics

Open Access

Peer-reviewed

Research Article

Emina Aličković,

Johannes Zaar,

Alejandro López Valdés  ,

Giovanni M. Di Liberto

Competing speech streams are si­mul­ta­ne­ously rep­re­sented in the hu­man cor­tex dur­ing at­ten­tion switch­ing

Sara Carta,

Emina Aličković,

Johannes Zaar,

Alejandro López Valdés,

Giovanni M. Di Liberto

x

Published: July 16, 2026

https://​doi.org/​10.1371/​jour­nal.pbio.3003876

Figures

Abstract

Successful speech com­mu­ni­ca­tion in multi-talker sce­nar­ios re­quires a skill­ful com­bi­na­tion of sus­tained at­ten­tion and rapid at­ten­tion switch­ing. While the neu­ro­phys­i­ol­ogy lit­er­a­ture of­fers de­tailed in­sights into the neural un­der­pin­nings of sus­tained at­ten­tion, there re­mains con­sid­er­able un­cer­tainty on how at­ten­tion switch­ing takes place. In this study, us­ing EEG record­ings from nor­mal-hear­ing adults in an im­mer­sive multi-talker en­vi­ron­ment, we mea­sured the neural en­cod­ing of two com­pet­ing speech streams amid back­ground bab­ble. Participants were cued to switch at­ten­tion be­tween streams every 15 – 30 s. Neural track­ing was as­sessed via Temporal Response Functions (TRF), con­firm­ing re­li­able de­cod­ing of at­ten­tional fo­cus. Our re­sults in­di­cate asym­met­ric dis­en­gage­ment and en­gage­ment processes dur­ing at­ten­tion switches, where the neural track­ing of the new tar­get stream emerges be­fore dis­en­gag­ing from the pre­vi­ous tar­get, re­veal­ing a tran­sient si­mul­ta­ne­ous en­cod­ing of two speech streams. That tran­si­tion was closely mir­rored by a re­duc­tion in EEG al­pha power, in­form­ing on the cog­ni­tive ef­fort dur­ing dif­fer­ent phases of the at­ten­tion switch. We then iso­lated cor­ti­cal ac­tiv­ity re­flect­ing lex­i­cal pre­dic­tion mech­a­nisms to de­ter­mine how lex­i­cal con­text is up­dated af­ter an at­ten­tion switch, com­par­ing four con­text-ac­cu­mu­la­tion strate­gies that were con­structed us­ing Large Language Models. Our find­ings elu­ci­date both the tem­po­ral and con­tex­tual mech­a­nisms un­der­ly­ing au­di­tory at­ten­tion shifts, point­ing to the pos­si­bil­ity that lis­ten­ers carry out a re­set in lex­i­cal con­text af­ter switch­ing at­ten­tion. By fo­cus­ing on dy­namic at­ten­tional re­al­lo­ca­tion, this study of­fers in­sights into the brain’s ca­pac­ity for flex­i­ble speech pro­cess­ing in com­plex lis­ten­ing en­vi­ron­ments.

Citation: Carta S, Aličković E, Zaar J, López Valdés A, Di Liberto GM (2026) Competing speech streams are si­mul­ta­ne­ously rep­re­sented in the hu­man cor­tex dur­ing at­ten­tion switch­ing. PLoS Biol 24(7): e3003876.

https://​doi.org/​10.1371/​jour­nal.pbio.3003876

Academic Editor: Manuel S. Malmierca, Universidad de Salamanca, SPAIN

Received: July 3, 2025; Accepted: June 12, 2026; Published: July 16, 2026

Copyright: © 2026 Carta et al. This is an open ac­cess ar­ti­cle dis­trib­uted un­der the terms of the Creative Commons Attribution License, which per­mits un­re­stricted use, dis­tri­b­u­tion, and re­pro­duc­tion in any medium, pro­vided the orig­i­nal au­thor and source are cred­ited.

Data Availability: All data sup­port­ing the find­ings re­ported in this man­u­script are freely ac­ces­si­ble with­out re­stric­tion. The EEG pre-processed dataset, the re­sult­ing analy­sis files, and the analy­sis code are pub­licly avail­able on the open repos­i­tory Zenodo (https://​zen­odo.org/​records/​20569817). The EEG record­ings are pro­vided fol­low­ing the Continuous-event Neural Data (CND) for­mat stan­dard. The as­so­ci­ated speech stim­uli can also be found in the same repos­i­tory, within the STIMULI folder.

Funding: S.C., A.L.V., and G.D.L. were sup­ported by the William Demant Fonden (https://​www.williamde­mant­fonden.dk/), un­der grants 21 – 0628 and 22 – 0552, and by Taighde Éireann — Research Ireland (https://​www.re­searchire­land.ie/) un­der grant No. 18/CRT/6223. G.D.L. ad­di­tion­ally con­ducted this re­search with the fi­nan­cial sup­port of Research Ireland at ADAPT, the Research Ireland Centre for AI-Driven Digital Content Technology (https://​www.adapt­cen­tre.ie/) at Trinity College Dublin [grant 13/RC/2106_P2]. The fun­ders had no role in study de­sign, data col­lec­tion and analy­sis, de­ci­sion to pub­lish, or prepa­ra­tion of the man­u­script.

Competing in­ter­ests: The au­thors have de­clared that no com­pet­ing in­ter­ests ex­ist.

Abbreviations: EEG, elec­troen­cephalog­ra­phy; EOG, elec­tro-ocu­log­ra­phy; EMG, elec­tro-myo­g­ra­phy; ERSP, event-re­lated spec­tral per­tur­ba­tion; iEEG, in­tra-cra­nial elec­troen­cephalog­ra­phy; FDR, false dis­cov­ery rate; fMRI, func­tional mag­netic res­o­nance imag­ing; ICA, Independent Component Analysis; IQR, in­terquar­tile range; LLM, large lan­guage model; MEG, mag­ne­toen­cephalog­ra­phy; PSD, power spec­tral den­sity; RMS, root-mean-squared; SE, stan­dard er­ror; SEM, stan­dard er­ror of the mean; SNR, sig­nal-to-noise ra­tio; SPL, sound pres­sure level; TRF, Temporal Response Functions

Introduction

To un­der­stand speech in multi-talker en­vi­ron­ments, lis­ten­ers sin­gle out the tar­get speaker from com­pet­ing sound streams [1 – 3]. The neu­ro­phys­i­ol­ogy of this se­lec­tive at­ten­tion process has been widely stud­ied with sim­u­lated cock­tail-party sce­nar­ios [4,5], shed­ding light on how our brains seg­re­gate a tar­get stream from com­pet­ing speech streams, and en­abling the trans­for­ma­tion of the tar­get speech into lin­guis­tic mean­ing. While the ex­tent to which masker speech streams are processed re­mains highly de­bated [6 – 8], there is no doubt that there are con­sid­er­able dif­fer­ences be­tween the pro­cess­ing of tar­get and masker speech, which have been mea­sured with var­i­ous tech­nolo­gies, such as non-in­va­sive elec­troen­cephalog­ra­phy (EEG) [1,9], in­tra-cra­nial elec­troen­cephalog­ra­phy (iEEG) [10], mag­ne­toen­cephalog­ra­phy (MEG) [3,11] and func­tional mag­netic res­o­nance imag­ing (fMRI) [12,13]. That work could pin­point pre­cise loci in the au­di­tory cor­ti­cal ar­eas where that seg­re­ga­tion emerges [14] as well as mea­sur­ing the sub­stan­tial (but not to­tal) sup­pres­sion of lin­guis­tic pro­cess­ing for the masker speech [1,15 – 17]. However, neu­ro­phys­i­ol­ogy lit­er­a­ture in this field has al­most en­tirely fo­cused on sus­tained at­ten­tion tasks [2,10], leav­ing con­sid­er­able un­cer­tainty on the neural un­der­pin­nings of at­ten­tion switch­ing.

Dynamic switch­ing par­a­digms have been widely used in the do­main of cog­ni­tive con­trol stud­ies to probe for cog­ni­tive flex­i­bil­ity and cog­ni­tive sta­bil­ity [18]. In those ex­per­i­ments, par­tic­i­pants are of­ten re­quired to flex­i­bly adapt their be­hav­ioral re­sponse de­pend­ing on new in­struc­tions, ini­ti­at­ing a task-switch [19 – 21]. For ex­am­ple, given a sin­gle digit, they are re­quired to clas­sify it ei­ther based on par­ity, i.e., whether it is even or odd, or based on rel­a­tive mag­ni­tude, i.e., whether the digit is greater than or less than 5 [22]. In these par­a­digms, the switch-cost is the in­crease in re­ac­tion time or er­ror rate when switch­ing from one task to the other. Similar be­hav­ioral par­a­digms have also in­volved sim­ple speech stim­uli in multi-talker set­tings [23 – 25]. However, the main in­ter­est of those tightly con­trolled ex­per­i­ments was to model the process of tar­get speech se­lec­tion as one par­tic­u­lar in­stance of a task-switch­ing prob­lem, i.e., tar­get stream se­lec­tion could ei­ther de­pend on spa­tial lo­ca­tion or voice iden­tity [23], rather than fo­cus­ing on the dy­namic as­pect of at­ten­tion re-al­lo­ca­tion per se in nat­u­ral­is­tic multi-talker sce­nar­ios. As such, very lit­tle is known on how a flex­i­ble re­ori­ent­ing of at­ten­tion might im­pact speech pro­cess­ing of con­tin­u­ous com­pet­ing streams.

In re­cent speech neu­ro­phys­i­ol­ogy re­search, ex­per­i­men­tal par­a­digms have started to in­clude switches of at­ten­tion as a tool to­wards tai­lored EEG/MEG method­olog­i­cal ad­vances in the do­main of at­ten­tion de­cod­ing [26,27], or to in­ves­ti­gate how sus­tained speech at­ten­tion un­folds for mov­ing au­di­tory ob­jects [28]. However, to the best of our knowl­edge, only one pre­vi­ous study has specif­i­cally fo­cused on the neu­ro­phys­i­ol­ogy of at­ten­tion switch­ing in multi-talker sce­nar­ios, re­lat­ing the neural en­cod­ing of speech dur­ing at­ten­tional re-ori­ent­ing with EEG al­pha ac­tiv­ity and pupil di­la­tion dy­nam­ics [29]. Those find­ings proved that the neu­ro­phys­i­ol­ogy of at­ten­tion switch­ing can be stud­ied non-in­va­sively. Building on that work, our study sheds light on the ex­act neural dy­nam­ics sup­port­ing the steer­ing of at­ten­tion be­tween two com­pet­ing speech streams, dis­en­gag­ing from the pre­vi­ous tar­get stream while en­gag­ing to the new one.

In this study, we mea­sure the neural en­cod­ing of speech us­ing a range of en­cod­ing win­dow lengths, as lis­ten­ers steer their at­ten­tion from one speaker to an­other. We test whether en­gage­ment with a new speech stream be­gins be­fore dis­en­gage­ment from the pre­vi­ous tar­get is com­plete, re­sult­ing in a brief pe­riod of si­mul­ta­ne­ous track­ing of both streams. Such an asym­me­try in the dis­en­gage­ment-en­gage­ment processes, even if tran­sient, could sup­port the abil­ity to ex­plore al­ter­na­tive au­di­tory streams while main­tain­ing at­ten­tion to a given stream [30].

The neural en­cod­ing of speech was mea­sured from nor­mal-hear­ing adult par­tic­i­pants us­ing EEG dur­ing an im­mer­sive multi-talker lis­ten­ing task. Participants were ex­posed to two com­pet­ing speech streams from TED talks, pre­sented via two front-fac­ing loud­speak­ers, while back­ground noise from a 16-talker speech bab­ble played from rear loud­speak­ers (Fig 1A). An on-screen ar­row cued par­tic­i­pants to at­tend to one of the two speech streams and to shift their at­ten­tion rapidly when­ever the ar­row changed di­rec­tion, ap­prox­i­mately every 10 – 30 s (Fig 1B). Neural track­ing of tar­get and masker speech was quan­ti­fied us­ing the Temporal Response Function (TRF), de­scrib­ing the lin­ear re­la­tion­ship be­tween each speech stream and the neural re­sponses. As an ini­tial val­i­da­tion, we con­firmed that the at­tended stream could be re­li­ably de­coded from the EEG, con­sis­tent with the ex­ten­sive lit­er­a­ture on sus­tained at­ten­tion [9,10,31]. This con­firms that the EEG re­sponses in this ex­per­i­ment re­flects dif­fer­en­tial en­cod­ing of tar­get ver­sus masker speech (Fig 1C).

Fig 1. Experiment overview and val­i­da­tion.

(A) Participants were pre­sented with speech from two loud­speak­ers placed in front of them with 60° of sep­a­ra­tion (30° left and 30° right), and with con­cur­rent 16-talker back­ground noise (B1–B4). In each trial, the screen pre­sented an ar­row point­ing to the tar­get speech stream. Participants were in­structed to switch at­ten­tion as soon as the vi­sual cue changes di­rec­tion. (B) Schematic di­a­gram of one ex­per­i­men­tal trial. The black area rep­re­sents blocks of at­ten­tion ei­ther to the left (L) or right (R) front streams. The red ar­rows in­di­cate the in­stants where the at­ten­tion cue switches side (six times per trial). Note that block du­ra­tion was ran­dom­ized and al­ways be­tween 15 and 30 s, with tri­als last­ing 3 min. (C) EEG data val­i­da­tion was car­ried out by run­ning an at­ten­tion de­cod­ing analy­sis. Progressively longer de­cod­ing win­dows were con­sid­ered (larger win­dows use more data, typ­i­cally lead­ing to more ac­cu­rate de­cod­ing scores). Binary clas­si­fi­ca­tion scores are re­ported ar­bi­trat­ing be­tween the tar­get and masker streams. The dashed line in­di­cates the 95th per­centile of a ran­dom dis­tri­b­u­tion cal­cu­lated by ran­dom­iz­ing the clas­si­fi­ca­tion la­bels. Statistically sig­nif­i­cant at­ten­tion de­cod­ing clas­si­fi­ca­tion scores were mea­sured for all the de­cod­ing win­dows con­sid­ered, with nu­mer­i­cal re­sults com­pa­ra­ble with pre­vi­ous stud­ies on se­lec­tive at­ten­tion [31,34,35]. Data sup­port­ing this fig­ure is avail­able at: https://​zen­odo.org/​records/​20569817.

https://​doi.org/​10.1371/​jour­nal.pbio.3003876.g001

We next ad­dressed two fun­da­men­tal ques­tions about the neural mech­a­nisms un­der­ly­ing at­ten­tion switch­ing in nat­u­ral­is­tic lis­ten­ing. First, we asked whether the processes of en­gag­ing with a new speech stream and dis­en­gag­ing from a pre­vi­ous one un­fold sym­met­ri­cally (Figs 2 and 3). To test this, we fit en­cod­ing TRF mod­els to EEG data, mea­sur­ing the neural track­ing of the two com­pet­ing speech streams over time. This al­lowed us to char­ac­ter­ize the av­er­age en­cod­ing dy­nam­ics sur­round­ing at­ten­tion switches, com­par­ing dis­en­gage­ment and en­gage­ment processes. The sec­ond ob­jec­tive was to un­der­stand how our brains up­date and use lex­i­cal con­text when switch­ing at­ten­tion (Fig 4). Building on pre­vi­ous work show­ing that speech com­pre­hen­sion is sup­ported by con­tex­tual pre­dic­tions [32,33], we for­mu­lated four com­pet­ing hy­pothe­ses re­flect­ing dif­fer­ent as­sump­tions about how lin­guis­tic con­text is pre­served, re­set, or se­lec­tively up­dated across an at­ten­tion switch. Using a state-of-the-art large lan­guage model (LLM), we de­rived quan­ti­ta­tive pre­dic­tions for each hy­poth­e­sis, re­sult­ing in four re­gres­sors for lex­i­cal sur­prisal and en­tropy, sep­a­rately, dif­fer­ing in their sen­si­tiv­ity to prior con­text and to the oc­cur­rence of the switch. Encoding TRF mod­els were then fit for each hy­poth­e­sis, al­low­ing us to com­pare al­ter­na­tive con­text-ac­cu­mu­la­tion strate­gies and iden­tify the model most con­sis­tent with the ob­served neural re­sponses. This study pro­vides sub­stan­tial new in­sights into the tem­po­ral un­fold­ing and con­tex­tual mech­a­nisms guid­ing at­ten­tion switch­ing, en­com­pass­ing both low and high lev­els of speech ab­strac­tion.

Fig 2. The at­ten­tion-switch­ing cue prompts a ro­bust dis­en­gage­ment from Speaker 1 and en­gage­ment to Speaker 2, and it is fol­lowed by a sig­nif­i­cant de­crease in the EEG al­pha ERSP.

Disengagement has longer tem­po­ral dy­nam­ics com­pared to en­gage­ment. (A) Left: Speech track­ing en­cod­ing for an at­ten­tion switch from Speaker 1 and 2. The tra­jec­tory in the panel rep­re­sents our null hy­poth­e­sis, where the dis­en­gage­ment and en­gage­ment processes progress in a sym­met­ric man­ner af­ter the switch-cue (vertical gray line). Right: Results for the neural track­ing of Speaker 1 and Speaker 2 across the switch­ing cue. EEG pre­dic­tion cor­re­la­tions (average across all chan­nels) ob­tained from a 4-s slid­ing-win­dow TRF model in­clud­ing Envelope (Env), Word Onset (WO) and Word Surprisal (WS) fea­tures. Coloured hor­i­zon­tal bars at the bot­tom of the plot in­di­cate the at­ten­tion in­struc­tion around the at­ten­tion switch­ing cue. The turquoise dot in­di­cates the en­cod­ing switch of EEG pre­dic­tion cor­re­la­tions based on Spk1- and Spk2- speech fea­tures. The piece­wise lin­ear model fit for dis­en­gage­ment and en­gage­ment is over­layed on the EEG pre­dic­tion cor­re­la­tion val­ues. Please note that the bro­ken-line-fit in this plot was per­formed on the grand-av­er­age cor­ti­cal track­ing curves here for il­lus­tra­tive pur­poses. Please find the es­ti­mates at the sin­gle-par­tic­i­pant level in Panel C. Hexagram shapes in­di­cate the start of the dis­en­gage­ment (blue) and en­gage­ment (yellow) processes, while di­a­monds rep­re­sent the end of the tran­si­tions. (B) Left: Diagram of ex­pected re­sults for al­pha-band ERSP (event-related spec­tral per­tur­ba­tion) across the switch­ing cue. Right: ERSP of the al­pha band (8 – 12 Hz) around the switch­ing cue (average of all chan­nels), com­puted with a 4-s slid­ing win­dow, as above. Scalp topogra­phies at se­lected time points re­veal a pat­tern of pos­te­rior neg­a­tiv­ity, which drops sig­nif­i­cantly fol­low­ing the in­struc­tion to switch (thick black lines in­di­cate a sta­tis­ti­cally sig­nif­i­cant change com­pared to pre-switch base­line). The red dot rep­re­sents the av­er­age of ERSP min­ima across par­tic­i­pants. The shaded area rep­re­sents the stan­dard er­ror of the mean (SEM) across par­tic­i­pants. (C) Left: Comparison of en­cod­ing switch of EEG pre­dic­tion cor­re­la­tions (turquoise bar) and al­pha ERSP min­i­mum (red bar) for a 4-s slid­ing win­dow. The al­pha ERSP reaches its min­i­mum sig­nif­i­cantly af­ter the Spk1-Spk2 en­cod­ing switch point. Right: Comparison of tem­po­ral dy­nam­ics for start and end points of dis­en­gage­ment and en­gage­ment processes, with start/​end tran­si­tion points es­ti­mated at the sin­gle-par­tic­i­pant level. Stars in­di­cate sig­nif­i­cant sta­tis­ti­cal ef­fects (paired sam­ple t-tests; *p ≤ 0.05; **p ≤ 0.01; **p ≤ 0.001). Data sup­port­ing this fig­ure is avail­able at: https://​zen­odo.org/​records/​20569817.

https://​doi.org/​10.1371/​jour­nal.pbio.3003876.g002

Fig 3. Comparing the start and end tran­si­tion points for the dis­en­gage­ment and en­gage­ment processes af­ter the at­ten­tions switch­ing cue.

The process of en­gag­ing to a new speaker be­gins and ends sig­nif­i­cantly ear­lier than dis­en­gag­ing from the pre­vi­ously at­tended speaker. (A, B) Start and end points of the tran­si­tion for the dis­en­gage­ment (blue) and en­gage­ment (yellow) processes over five TRF slid­ing win­dow lengths. Error bars rep­re­sent SEM across par­tic­i­pants. Stars in­di­cate sig­nif­i­cant ef­fects of process type (two-way re­peated mea­sures ANOVA; *p ≤ 0.05; **p ≤ 0.01; ***p ≤ 0.001). Data sup­port­ing this fig­ure is avail­able at: https://​zen­odo.org/​records/​20569817.

https://​doi.org/​10.1371/​jour­nal.pbio.3003876.g003

Fig 4. Investigating lex­i­cal pre­dic­tion mech­a­nisms dur­ing at­ten­tion switch­ing.

(A) Layout of the four con­text mod­els. Blocks coloured in black il­lus­trate sus­tained at­ten­tion ei­ther to the Left or Right stream, while or­ange ar­rows in­di­cate at­ten­tion switch­ing cues. The thick red ar­row in­di­cates the con­text used to guide word pre­dic­tions for the cur­rent block (B7, high­lighted in or­ange). (B) Average lex­i­cal en­tropy at words pre­ced­ing and fol­low­ing the at­ten­tion switch cue. Note that no value for en­tropy is dis­played in the Reset model for the first word af­ter the switch, due to the con­text be­ing fully re­in­stated. (C) EEG pre­dic­tion cor­re­la­tions for the four mul­ti­vari­ate TRF mod­els, only dif­fer­ing in their en­tropy fea­ture. Coloured dots in­di­cate the av­er­age across all elec­trodes and par­tic­i­pants. The gray area at the bot­tom rep­re­sents the av­er­age en­cod­ing ac­cu­racy of a mul­ti­vari­ate TRF with­out any se­man­tic in­for­ma­tion (Envelope + Word Onset). Stars rep­re­sent sta­tis­ti­cally sig­nif­i­cantly greater EEG pre­dic­tion cor­re­la­tions for the Reset model com­pared to the other mod­els (Significance lev­els: *p < 0.05, **p < 0.01, ***p < 0.001). Topographical pat­terns il­lus­trate the gain due to se­man­tic in­for­ma­tion (compared to the Envelope + Word Onset TRF) for the four mod­els. (D) (Left) TRF weights for the en­tropy fea­ture at time-lags be­tween −100 and 600 ms rel­a­tive to stim­u­lus on­set. Transparent shaded ar­eas rep­re­sent the stan­dard er­ror of the mean (SEM) across par­tic­i­pants. The hor­i­zon­tal black line in­di­cates the time win­dow em­ployed to com­pute the av­er­age TRF-N400 am­pli­tude. (Right) Boxplots rep­re­sent­ing the dis­tri­b­u­tion of the TRF-N400 am­pli­tude across par­tic­i­pants for the four con­text mod­els. The cen­tral line within each box rep­re­sents the me­dian, while the edges of the box in­di­cate the in­terquar­tile range (IQR). Whiskers ex­tend to the most ex­treme data points within 1.5 times the IQR from the quar­tiles. Outliers are plot­ted as in­di­vid­ual points be­yond the whiskers. Stars in­di­cate sta­tis­ti­cally sig­nif­i­cant dif­fer­ences (Significance lev­els: *p < 0.05, **p < 0.01, ***p < 0.001). Data sup­port­ing this fig­ure is avail­able at: https://​zen­odo.org/​records/​20569817.

https://​doi.org/​10.1371/​jour­nal.pbio.3003876.g004

Results

Behavioral per­for­mance

Following each trial, par­tic­i­pants were first pre­sented with a four-al­ter­na­tive forced-choice ques­tion about the con­tent of the at­tended speech stream to con­firm task en­gage­ment. Behavioral per­for­mance re­vealed that they were able to suc­cess­fully re­ply to con­tent-re­lated ques­tions, with an av­er­age ac­cu­racy of 86.3% (SEM 2.6%). Participants were also re­quired to in­di­cate their pref­er­ence be­tween left and right streams, which was found to be over­all bal­anced, with the left stream se­lected in 49.79% of the tri­als, on av­er­age (SEM 1.7%). Finally, per­ceived dif­fi­culty of the at­ten­tion switch for every trial was mea­sured by ask­ing par­tic­i­pants to rate it on a scale from 1 (very easy) to 5 (very hard). The av­er­age dif­fi­culty of the switch was judged to be 3.1 out of 5, with a SEM across par­tic­i­pants of 0.11 points. Due to tech­ni­cal is­sues, be­hav­ioral data for one of the 24 par­tic­i­pants was not avail­able, there­fore be­hav­ioral per­for­mance was com­puted based on the data from the re­main­ing 23 par­tic­i­pants.

Decoding of se­lec­tive au­di­tory at­ten­tion in a dy­namic switch­ing sce­nario

Participants’ at­ten­tion was de­coded with a back­ward TRF analy­sis, de­scrib­ing the re­la­tion­ship be­tween the EEG sig­nals and the en­ve­lope of the tar­get speech. For each left-out trial, the speech en­ve­lope re­con­structed from the tar­get de­cod­ing model was cor­re­lated with the en­velopes of both the left and right speech streams. Attention was clas­si­fied by de­ter­min­ing which speech stream’s en­ve­lope showed a higher cor­re­la­tion with the re­con­structed en­ve­lope. Since this was a dy­namic at­ten­tion-switch­ing sce­nario, the at­tended speech could al­ter­na­tively cor­re­spond to the left or the right stream. Classification was con­sid­ered cor­rect when the re­con­structed en­ve­lope cor­re­lated more strongly with the tar­get speech en­ve­lope than with the masker en­ve­lope. Classification ac­cu­racy was then com­puted as the pro­por­tion of in­stances where this cri­te­rion was met. To es­tab­lish chance per­for­mance, left and right la­bels were ran­domly shuf­fled 100 times for each de­cod­ing win­dow. As shown in Fig 1C, the longer de­cod­ing win­dows led to higher clas­si­fi­ca­tion per­for­mances. However, even with a 1-second win­dow, clas­si­fi­ca­tion ac­cu­racy was sig­nif­i­cantly above chance level, and all de­cod­ing win­dows yielded clas­si­fi­ca­tion rates sig­nif­i­cantly above the 95th per­centile of its chance dis­tri­b­u­tion (paired two-tailed t test, FDR-corrected for mul­ti­ple com­par­isons for win­dows of 1 s, 2 s, 4 s, 8 s, 16 s, 32 s, re­spec­tively: p = 0.47e−9; 0.53e−9; 0.27e−9; 0.24e−9; 0.24e−9; 0.24e−9). These find­ings align with pre­vi­ous work de­cod­ing sus­tained at­ten­tion or em­ploy­ing match-vs-mis­match clas­si­fi­ca­tion met­rics [9,31,36], and con­firm that a clas­si­fi­ca­tion based on the en­ve­lope re­con­struc­tion can re­li­ably track se­lec­tive at­ten­tion even dur­ing at­ten­tion switches.

Neural track­ing of com­pet­ing speech streams in a dy­namic switch­ing sce­nario re­flects the lis­ten­er’s fo­cus of at­ten­tion and is re­lated to changes in al­pha ERSP

A mul­ti­vari­ate TRF analy­sis was car­ried out to char­ac­ter­ize the neural track­ing of two com­pet­ing speech streams in a set­ting where par­tic­i­pants were in­structed to dy­nam­i­cally switch their at­ten­tion be­tween the two streams. Single-subject TRFs were trained on the tar­get stream and tested on both speak­ers (i.e., Spk1 and Spk2) us­ing a mul­ti­vari­ate speech rep­re­sen­ta­tion that in­cluded Envelope, Word Onset and Word Surprisal fea­tures (for more de­tails see Methods). EEG pre­dic­tion cor­re­la­tions were com­puted us­ing a slid­ing win­dow to cor­re­late true and pre­dicted EEG sig­nals over time, with a leave-one-out cross-val­i­da­tion pro­ce­dure and were av­er­aged across all EEG chan­nels. Importantly, be­cause these cor­re­la­tions are com­puted us­ing slid­ing win­dows, the re­sult­ing switch tim­ing de­pends on the slid­ing win­dow length. As such, the tem­po­ral dy­nam­ics de­riv­ing from our analy­ses do not re­flect the ex­act tim­ing of the un­der­ly­ing neural processes, and they should al­ways be in­ter­preted with the caveat of the slid­ing win­dow length.

In or­der to an­a­lyze ro­bust switch­ing dy­nam­ics, we se­lected 21 par­tic­i­pants dis­play­ing a re­li­able at­ten­tional bias over the course of the switch, based on an above-chance clas­si­fi­ca­tion ac­cu­racy cri­te­rion (>50%) over the course of the switch. In do­ing so, we re­moved par­tic­i­pants for whom the start and end points of the (dis)engagement could not be es­ti­mated (note that this ex­clu­sion is de­ter­mined be­fore iden­ti­fy­ing the start/​end es­ti­mates; in that sense, this is dif­fer­ent from an out­lier re­moval, which would ex­clude ex­treme start/​end val­ues in­stead).

Aligning with our ex­pec­ta­tion (Fig 2A), EEG pre­dic­tion cor­re­la­tions around the switch­ing cue re­flected track­ing of Spk1 and Spk2 streams con­sis­tent with the at­ten­tion in­struc­tions, such that Spk1 was sig­nif­i­cantly more tracked than Spk2 be­fore the switch, while the re­verse pat­tern was ob­served af­ter the switch (paired two-tailed t test of Spk1-Spk2 dif­fer­ence against zero, FDR-corrected for mul­ti­ple com­par­isons, p < 0.005).

As the at­ten­tion switch un­folds, also the grand-mean ERSP in the al­pha fre­quency band dis­played a sta­tis­ti­cally sig­nif­i­cant change com­pared to base­line (one-sample t test against zero, FDR-corrected for mul­ti­ple com­par­isons), re­veal­ing a pat­tern of oc­cip­ito-pari­etal neg­a­tiv­ity in the scalp topogra­phies (Fig 2B). This is con­sis­tent with our ex­pec­ta­tion of an im­pact of at­ten­tional re­ori­en­ta­tion on the EEG al­pha band, which has al­ready been shown to re­flect at­ten­tion switch­ing be­hav­ior in com­pet­ing speech lis­ten­ing sce­nar­ios [29].

EEG pre­dic­tion cor­re­la­tions for Spk1 and Spk2 con­verged, be­fore sig­nif­i­cantly sep­a­rat­ing again once the switch­ing process was con­cluded and, pre­sum­ably, the at­ten­tion was fully re­al­lo­cated. Here, we re­fer to the time point when EEG pre­dic­tion over­laps be­tween Spk1 and Spk2 as the en­cod­ing switch point. Given the ob­served sta­tis­ti­cally sig­nif­i­cant drop in al­pha ERSP, we asked how the tem­po­ral dy­nam­ics of this drop com­pared to those of the EEG pre­dic­tion cor­re­la­tions. To ad­dress this, for each par­tic­i­pant, we iden­ti­fied the time of the al­pha ERSP min­i­mum, and the en­cod­ing switch point, based on an en­cod­ing win­dow of 4s (Fig 2C). The choice of this par­tic­u­lar en­cod­ing win­dow for our main analy­sis is jus­ti­fied based on the clas­si­fi­ca­tion ac­cu­racy re­sults (Fig 1C), since it is a good com­pro­mise be­tween tem­po­ral res­o­lu­tion and clas­si­fi­ca­tion per­for­mance. However, the same pat­tern of re­sults holds when con­sid­er­ing mul­ti­ple en­cod­ing win­dows si­mul­ta­ne­ously (S1 Fig). A paired t test com­par­ing the tem­po­ral dy­nam­ics of the al­pha ERSP and the EEG pre­dic­tion cor­re­la­tions showed that the min­i­mum of the al­pha ERSP drop sig­nif­i­cantly fol­lows the en­cod­ing switch point (t(20) = 4.29, p = 3.59e-4, Cohen’s d = 0.94).

We then eval­u­ated mul­ti­ple en­cod­ing win­dow lengths, as­sess­ing the ef­fect of Metric (encoding switch ver­sus ERSP min­i­mum) and Window (1, 2, 4, 8 s) on the tim­ing of the en­cod­ing switch and the min­i­mum of the al­pha ERSP with a 2-way re­peated mea­sures ANOVA. The analy­sis re­vealed that the tem­po­ral dy­nam­ics of both en­cod­ing switch and al­pha ERSP min­i­mum be­came longer as the en­cod­ing win­dow length in­creased (F(1.68, 33.55) = 52.77, p = 2.3e−10, ηp2 = 0.72; a Greenhouse-Geisser’s cor­rec­tion ap­plied due to spheric­ity vi­o­la­tion), which is un­sur­pris­ing given the method­olog­i­cal con­straints we dis­cussed (see Methods). More in­ter­est­ingly for our ques­tion, a sta­tis­ti­cally sig­nif­i­cant ef­fect of Metric emerged (F(1,20) = 20.26, p = 2.18e−4, ηp2 = 0.5), with the al­pha ERSP min­i­mum oc­cur­ring sig­nif­i­cantly later than the en­cod­ing switch across a range of en­cod­ing win­dows (Holm-corrected post-hoc t test: t(20) = 4.5, p = 2.18e−4, Cohen’s d = 0.97).

Dissecting the tem­po­ral dy­nam­ics of at­ten­tional dis­en­gage­ment and en­gage­ment dur­ing at­ten­tion switch­ing

The at­ten­tion switch­ing cue prompts the lis­tener to re­al­lo­cate their at­ten­tion from the pre­vi­ously at­tended speaker, Spk1, to the newly at­tended speaker, Spk2. While this re-rout­ing of at­ten­tion ap­pears to be a sin­gle, uni­fied process, it is pos­si­ble to dis­tin­guish two sep­a­rate op­er­a­tions that are nec­es­sary for it to hap­pen: dis­en­gage­ment, which we de­fine as the de­crease in neural track­ing for the pre­vi­ously at­tended speech stream, and en­gage­ment, which we de­fine as the in­crease in neural track­ing for the pre­vi­ously un­at­tended speech stream. Our goal was to clar­ify the tem­po­ral dy­nam­ics of these two op­er­a­tions to un­der­stand whether they oc­cur fully in par­al­lel, se­ri­ally, or with a cer­tain de­gree of over­lap. It is worth not­ing that, due to the use of slid­ing en­cod­ing win­dows, the es­ti­mated tem­po­ral dy­nam­ics of en­gage­ment and dis­en­gage­ment do not re­flect the ex­act time course of the un­der­ly­ing neural processes and should be in­ter­preted as rel­a­tive, rather than ab­solute, tem­po­ral met­rics. As in the pre­vi­ous analy­sis, we first se­lected par­tic­i­pants dis­play­ing a re­li­able at­ten­tional bias over the course of the switch (see Methods). For this se­lec­tion of 21 par­tic­i­pants, we fit­ted a piece­wise lin­ear re­gres­sion on sin­gle-sub­ject EEG pre­dic­tion cor­re­la­tions, and found the op­ti­mal break­points, cor­re­spond­ing to the start and end time points of dis­en­gage­ment and en­gage­ment (Fig 2C). As above, we chose to fo­cus on an ex­am­ple win­dow of 4 s and later repli­cated our re­sults on a range of en­cod­ing win­dow lengths. To fur­ther char­ac­ter­ize the spa­tial pat­terns of en­gage­ment and dis­en­gage­ment processes, scalp topogra­phies of the EEG pre­dic­tion cor­re­la­tions at se­lected time points are shown in S2 Fig, in­di­cat­ing that the most pre­dic­tive chan­nels were pre­dom­i­nantly lo­cated over cen­tral-pari­etal re­gions. Disengagement and en­gage­ment processes were com­pared sep­a­rately based on their start times and end times, re­veal­ing con­sis­tently ear­lier tem­po­ral dy­nam­ics for the en­gage­ment com­pared to the dis­en­gage­ment. Engagement to the newly at­tended speaker started sig­nif­i­cantly ear­lier than the dis­en­gage­ment from the pre­vi­ously at­tended speaker (paired-sample t test: t(20) = 2.37, p = 0.03, Cohen’s d = 0.52), and fin­ished sig­nif­i­cantly ear­lier (paired-sample t test: t(20) = 2.35, p = 0.03, Cohen’s d = 0.39).

We then ex­tended our analy­sis to a range of slid­ing win­dow lengths and com­pared start times and end times for dis­en­gage­ment and en­gage­ment processes, in­clud­ing Window (1, 2, 4, 8, 16 s) and Process (disengagement ver­sus en­gage­ment) as main fac­tors in a re­peated mea­sures ANOVA. Regarding the start points (Fig 3A), our analy­ses re­vealed an ex­pected sta­tis­ti­cally sig­nif­i­cant ef­fect of Window (F(1.51,30.14) = 9.7, p = 0.001, ηp2 = 0.33; the as­sump­tion of spheric­ity was not met; hence, a Greenhouse–Geisser’s cor­rec­tion was ap­plied), with longer tem­po­ral dy­nam­ics cor­re­spond­ing to longer en­cod­ing win­dow lengths. More im­por­tantly, we also ob­served a sig­nif­i­cant main ef­fect of Process (F(1,20) = 5.48, p = 0.03, ηp2 = 0.21), with en­gage­ment to the newly at­tended stream start­ing sig­nif­i­cantly ear­lier than the dis­en­gage­ment to the pre­vi­ously at­tended stream (Holm-corrected post-hoc t test: t(20) = 2.34, p = 0.03, Cohen’s d = 0.54). The same sta­tis­ti­cal analy­sis was re­peated sep­a­rately on the end time points of dis­en­gage­ment and en­gage­ment processes (Fig 3B), re­veal­ing once again a main ef­fect of Window, whereby longer en­cod­ing win­dows yield longer tem­po­ral tran­si­tions (F(1.97,39.32) = 31.76, p = 7.2e−9, ηp2 = 0.61; the as­sump­tion of spheric­ity was not met; hence, a Greenhouse–Geisser’s cor­rec­tion was ap­plied). A sig­nif­i­cant main ef­fect of Process also emerged (F(1,20) = 4.46, p = 0.047, ηp2 = 0.18), re­veal­ing that the process of en­gage­ment to the newly at­tended speaker, not only starts, but also ends sig­nif­i­cantly ear­lier than the dis­en­gage­ment (Holm-corrected post-hoc t test: t(20) = 2.11, p = 0.047, Cohen’s d = 0.58).

A fol­low-up analy­sis in­clud­ing the three par­tic­i­pants with lower-than-chance clas­si­fi­ca­tion ac­cu­racy around the switch­ing cue con­firmed that these data points in­tro­duced noise to the es­ti­ma­tion of en­gage­ment and dis­en­gage­ment la­ten­cies. This was ex­pected, as the start and end tran­si­tion points can­not be de­ter­mined in those par­tic­i­pants. The pat­terns ob­served were qual­i­ta­tively sim­i­lar to the main re­sult re­ported above, with ear­lier tem­po­ral dy­nam­ics for the en­gage­ment com­pared to the dis­en­gage­ment, al­beit with weaker ef­fects be­low the sta­tis­ti­cal sig­nif­i­cance thresh­old (repeated-measures ANOVA; start point: F(1,23) = 2.69, p = 0.11, ηp2 = 0.1; end point: F(1,23) = 2.96, p = 0.1, ηp2 = 0.11).

Determining how lex­i­cal pre­dic­tions are built dur­ing at­ten­tion switch­ing

Reorienting at­ten­tion to a dif­fer­ent speech stream im­plies a change of con­text and, con­se­quently, dif­fer­ent se­man­tic pri­ors for lex­i­cal pre­dic­tions. We thus hy­poth­e­sized that in­cor­po­rat­ing this change of con­text into the struc­ture of our se­man­tic re­gres­sor in a mul­ti­vari­ate en­cod­ing TRF model would in­crease EEG pre­dic­tion cor­re­la­tions, as it would bet­ter re­flect the dy­nam­i­cally up­dat­ing neural track­ing of the com­pet­ing speech streams. We com­pared four al­ter­na­tive mod­els rep­re­sent­ing how con­text could be in­cre­men­tally ac­cu­mu­lated for per­form­ing lex­i­cal pre­dic­tions at one par­tic­u­lar at­ten­tion block (e.g., B7, in Fig 4A). A naïve Oracle model, which uses all avail­able con­text of pre­vi­ous blocks from the cur­rent stream, whether at­tended or un­at­tended, to pre­dict words from the cur­rent block, served as our base­line, since it was es­sen­tially a switch-un­aware con­tex­tual rep­re­sen­ta­tion. Speaker-Specific and Attention mod­els were in­stead switch-aware mod­els, as they only con­sid­ered pre­vi­ously at­tended blocks as part of the con­text for lex­i­cal pre­dic­tions. Speaker-Specific as­sumed a higher de­gree of stream seg­re­ga­tion, since its con­text only con­sisted of pre­vi­ously at­tended blocks from the same speech stream, while Attention in­cluded any pre­vi­ously at­tended block from both streams. The Reset model in­stead ig­nored all pre­vi­ously at­tended blocks from any of the streams and com­puted con­text only over the course of the cur­rent block of at­ten­tion, as if the pri­ors for lex­i­cal pre­dic­tions were re­set at each at­ten­tion switch (Fig 4A).

As lex­i­cal en­tropy is a proxy of un­cer­tainty for next-word pre­dic­tion, its val­ues should be im­pacted by a switch­ing cue, which de­ter­mines an abrupt change of con­text. Fig 4B shows the change of av­er­age lex­i­cal en­tropy val­ues in words pre­ced­ing and fol­low­ing the switch cue, which vary de­pend­ing on the con­text mod­els. It can be ob­served that the Reset model peaks with the high­est un­cer­tainty and slowly de­cays over the course of the next words, while the Attention and Speaker Specific mod­els have over­all sim­i­lar lex­i­cal en­tropy dy­nam­ics and more sta­ble val­ues. Consistently with its switch-un­aware na­ture, the Oracle model in­stead dis­plays en­tropy val­ues that are largely un­changed de­spite the switch. An ex­plicit com­par­i­son of the av­er­age en­tropy val­ues of the four con­text-ac­cu­mu­la­tion strate­gies re­vealed sta­tis­ti­cally sig­nif­i­cant dif­fer­ences (repeated-measures ANOVA, Greenhouse-Geisser cor­rec­tion due to spheric­ity vi­o­la­tion; F(1,19) = 39.57, p = 9.59e−10, ηp2 = 0.68). Post-hoc pair­wise tests (Holm-adjusted) in­di­cated that the Reset model showed an in­ter­me­di­ate av­er­age en­tropy, sig­nif­i­cantly higher than the Oracle model (t(19) = 5.75, p = 1.44e−6, Cohen’s d = 0.45), and sig­nif­i­cantly lower than the Attention (t(19) = 4.64, p = 6.28e − 5, Cohen’s d = 0.36) and Speaker-Specific (t(19) = 2.24, p = 0.04, Cohen’s d = 0.17) mod­els. As such, de­spite show­ing the high­est peak fol­low­ing the at­ten­tion switch­ing cue, the Reset model had over­all in­ter­me­di­ate en­tropy val­ues across the four lex­i­cal ex­pec­ta­tion mod­els con­sid­ered here.

Lexical sur­prisal and lex­i­cal en­tropy were used as se­man­tic in­for­ma­tion re­gres­sors for each con­text model and sep­a­rately in­cluded in a mul­ti­vari­ate stim­u­lus rep­re­sen­ta­tion to fit sin­gle-sub­ject en­cod­ing TRFs (Envelope-Word Onset-Word Surprisal and Envelope-Word Onset-Word Entropy). Resulting TRF weights and EEG pre­dic­tion cor­re­la­tions were then com­pared across con­text mod­els, with the hy­poth­e­sis that switch-aware and con­text-rich rep­re­sen­ta­tions (e.g., Speaker-Specific or Attention) would best de­scribe neural ac­tiv­ity in at­ten­tion-switch­ing sce­nar­ios.

Before com­par­ing the con­text mod­els, we first tested whether each of them yielded a sig­nif­i­cant en­cod­ing ac­cu­racy gain com­pared to the base­line model only con­sist­ing of acoustic fea­tures (Envelope and Word Onset). When us­ing en­tropy as a re­gres­sor for se­man­tics, all mod­els, with the ex­cep­tion of Oracle, showed a sta­tis­ti­cally sig­nif­i­cant gain, sug­gest­ing a ro­bust track­ing of se­man­tic in­for­ma­tion in ad­di­tion to the stim­u­lus acoustics (paired t-tests: Oracle ver­sus Acoustics: p = 0.2; Spk.Spec. ver­sus Acoustics: p = 0.04; Attention ver­sus Acoustics: p = 0.04; Reset ver­sus Acoustics: p = 0.002). Employing word sur­prisal as se­man­tic re­gres­sor yielded sim­i­lar re­sults, with all the mod­els show­ing a ro­bust en­cod­ing of se­man­tic in­for­ma­tion, apart from Oracle (paired t-tests: Oracle ver­sus Acoustics: p = 0.15; Spk.Spec. ver­sus Acoustics: p = 0.02; Attention ver­sus Acoustics: p = 0.02; Reset ver­sus Acoustics: p = 0.01). The non-sig­nif­i­cant gain of the Oracle model com­pared to the acoustic model was ex­pected, since Oracle was de­signed as a con­trol switch-un­aware model.

In con­trast to our ex­pec­ta­tion, the Reset con­text model was shown to yield higher EEG pre­dic­tion cor­re­la­tion val­ues when en­tropy was used as a re­gres­sor for se­man­tics (Fig 4C). A re­peated mea­sures ANOVA re­vealed a sta­tis­ti­cally sig­nif­i­cant ef­fect of the main fac­tor, Context Model (F(2.1,47.75) = 9, p = 4e−4, ηp2 = 0.28; with Greenhouse-Geisser’s cor­rec­tion). In the Holm-corrected post-hoc tests, the Reset model was shown to yield sig­nif­i­cantly higher en­cod­ing ac­cu­ra­cies than Oracle (t(23) = 4.99, p = 2.63e−5, Cohen’s d = 0.14), Speaker Specific (t(23) = 3.73, p = 0.002, Cohen’s d = 0.1), and Attention (t(23) = 3.28, p = 0.006, Cohen’s d = 0.09). We then as­sessed the dif­fer­ence of TRF weights for the en­tropy fea­ture across the four con­text mod­els (Fig 4D), av­er­ag­ing the weights’ am­pli­tude within a win­dow broadly cen­tered around the TRF-N400 la­tency (350 – 550 ms). A re­peated mea­sure ANOVA was run on the weights’ am­pli­tude val­ues, re­veal­ing a main ef­fect of Context Model (F(3,69) = 15.51, p = 8.2e−8, ηp2 = 0.4). Post-hoc tests (Holm-corrected) showed that weights for the Reset model had lower TRF-N400 am­pli­tude com­pared to Oracle (t(23) = −5.56, p = 2.4e−6, Cohen’s d = 0.45), Attention (t(23) = −5.84, p = 9.2e−7, Cohen’s d = 0.47), and Speaker Specific (t(23) = −5.24, p = 6.5e−6, Cohen’s d = 0.43).

When fit­ting a mul­ti­vari­ate TRF in­clud­ing lex­i­cal sur­prisal as a se­man­tic re­gres­sor, we ob­served a sta­tis­ti­cally sig­nif­i­cant dif­fer­ence in EEG pre­dic­tion cor­re­la­tions be­tween the four con­text mod­els (F(1.59,36.6) = 3.96, p = 0.04, ηp2 = 0.15, with Greenhouse–Geisser cor­rec­tion). Post-hoc analy­ses in­di­cated a sta­tis­ti­cally sig­nif­i­cant dif­fer­ence be­tween the Reset and Oracle mod­els (t(23) = 3.18, p = 0.013, Cohen’s d = 0.1), while all other post-hoc pair­wise com­par­isons did not reach the sig­nif­i­cance thresh­old (p < 0.05). Similarly, no sta­tis­ti­cally sig­nif­i­cant dif­fer­ence emerged when com­par­ing the TRF-N400 am­pli­tude of the mod­els’ TRF weights.

Discussion

Speech com­mu­ni­ca­tion in multi-talker en­vi­ron­ments re­quires a skill­ful com­bi­na­tion of sus­tained at­ten­tion and rapid at­ten­tion switch­ing abil­i­ties [5,30]. While the neu­ro­phys­i­ol­ogy of sus­tained speech at­ten­tion has been widely stud­ied [1,9,37,38], less is known about the neural mech­a­nisms of at­ten­tion switch­ing. Here, we fill this gap with a tai­lored EEG ex­per­i­ment ex­am­in­ing the neu­ro­phys­i­ol­ogy of at­ten­tion switch­ing across dif­fer­ent lev­els of speech ab­strac­tion. In do­ing so, we (1) demon­strated an ex­per­i­men­tal par­a­digm that can suc­cess­fully probe both sus­tained at­ten­tion and at­ten­tion switch­ing mech­a­nisms; (2) suc­cess­fully dis­sected dis­en­gage­ment and en­gage­ment processes with a high tem­po­ral res­o­lu­tion, iden­ti­fy­ing sub­stan­tial asym­me­tries in their tem­po­ral un­fold­ing and a tran­sient si­mul­ta­ne­ous en­cod­ing of two speech streams; and (3) pro­posed a neu­ro­phys­i­o­log­i­cally plau­si­ble ex­pla­na­tion of how our brains up­date and use lex­i­cal con­text when switch­ing at­ten­tion.

The find­ings in this study have sev­eral im­pli­ca­tions for our un­der­stand­ing of speech at­ten­tion switch­ing mech­a­nisms. The asym­me­try mea­sured be­tween dis­en­gage­ment and en­gage­ment processes high­lights the im­por­tance of study­ing the two processes sep­a­rately. That dis­tinc­tion was of­ten not con­sid­ered in pre­vi­ous stud­ies on sus­tained at­ten­tion, which of­ten fo­cused on mea­sures of at­ten­tion bias or clas­si­fi­ca­tion [10,31,34,39]. The ef­fec­tive­ness of such de­cod­ing met­rics has been a dri­ving force for re­search on brain-com­puter in­ter­faces such as cog­ni­tively-con­trolled hear­ing de­vices [40 – 43]. Our find­ing high­lights that en­cod­ing met­rics en­able a suf­fi­cient level of de­tail for dis­en­tan­gling how the en­cod­ing of dif­fer­ent streams evolves over time. Here, we mea­sured an asym­me­try be­tween dis­en­gage­ment and en­gage­ment processes dur­ing at­ten­tion switch­ing in a very spe­cific sce­nario. Indeed, it will be im­por­tant to de­ter­mine how that re­la­tion­ship is mod­u­lated by fac­tors such as cog­ni­tive load, ag­ing, cog­ni­tive abil­i­ties, hear­ing dif­fi­cul­ties, in­ter­est in the speech con­tent, fre­quency of at­ten­tion switches in a trial, among many oth­ers. Of course, fu­ture work should also scru­ti­nize how the spe­cific na­ture of the task might im­pact that phe­nom­e­non.

Sustained at­ten­tion tasks, where par­tic­i­pants fo­cus on a tar­get speech while ig­nor­ing the masker [2,44], in­volve a quite par­tic­u­lar sce­nario where lis­ten­ers have no in­cen­tive to mon­i­tor un­at­tended streams. In real-life sit­u­a­tions, how­ever, lis­ten­ers may have rea­sons to ex­plore al­ter­na­tive speech streams, for ex­am­ple, due to a lack of in­ter­est in the cur­rent speaker. Our ex­per­i­men­tal par­a­digm more closely mir­rors this sce­nario. Although the in­structed na­ture of the task makes it less re­al­is­tic, the par­a­digm in­cen­tivises mon­i­tor­ing the masker and be­ing ready to rapidly switch at­ten­tion, which con­trasts with sus­tained at­ten­tion tasks. While the asym­me­try ob­served may be spe­cific to this ex­per­i­men­tal par­a­digm, the re­sult in­di­cates that our brains can en­gage with a new tar­get even be­fore start­ing the dis­en­gage­ment from the pre­vi­ous one, lead­ing to a tran­sient si­mul­ta­ne­ous track­ing of the two streams, com­pat­i­ble with au­di­tory scene mon­i­tor­ing mech­a­nisms [45,46]. In other words, there is a brief pe­riod, fol­low­ing an at­ten­tion switch, where the track­ing of a new stream be­gins to emerge with­out al­ter­ing the track­ing of the pre­vi­ous stream. The en­gage­ment and dis­en­gage­ment la­ten­cies are win­dow-de­pen­dent and, as such, are not in­tended as ab­solute neural tim­ings. Nonetheless, the en­gage­ment-dis­en­gage­ment asym­me­try is ro­bust to the se­lec­tion of the slid­ing-win­dow length (Fig 3) and can­not be ex­plained by the tem­po­ral smooth­ing in­tro­duced by the win­dow or by trial- and par­tic­i­pant-av­er­ag­ing. The tem­po­ral smooth­ing could, in prin­ci­ple, tem­po­rally stretch the en­gage­ment-dis­en­gage­ment dy­nam­ics, but not gen­er­ate an asym­me­try. Future re­search could in­ves­ti­gate the vari­abil­ity across tri­als and par­tic­i­pants to bet­ter char­ac­ter­ize these tem­po­ral dy­nam­ics. Within the dor­sal–ven­tral at­ten­tion frame­work [47,48], at­ten­tion re­ori­ent­ing re­sults from the flex­i­ble in­ter­play of the goal-di­rected dor­sal net­work and the stim­u­lus-dri­ven ven­tral net­work. Our find­ing of an en­gage­ment-dis­en­gage­ment asym­me­try aligns with this view of an in­te­grated process of at­ten­tion re­lease and re­al­lo­ca­tion, with po­ten­tially over­lap­ping neural dy­nam­ics.

Intuitively, main­tain­ing a tran­sient par­al­lel rep­re­sen­ta­tion of mul­ti­ple speech sources dur­ing at­ten­tion switch­ing is an ef­fi­cient neural pro­cess­ing strat­egy. It al­lows the flex­i­bil­ity to switch back to the pre­vi­ous stream, if nec­es­sary, with­out fully com­mit­ting to the newly at­tended speech im­me­di­ately. This phe­nom­e­non sup­ports pre­vi­ous claims that our brains can process speech maskers be­yond the acoustic level, en­cod­ing lin­guis­tic prop­er­ties to some ex­tent [7,8,16]. Unattended speech streams are also rep­re­sented in hu­man cor­ti­cal ac­tiv­ity, with ev­i­dence for lower en­cod­ing strengths or longer time la­ten­cies than the tar­get stream [1,49,50], and with a gra­di­ent of at­ten­tional bias from pri­mary to non­pri­mary au­di­tory cor­tex [13,14], whereby the un­at­tended stream en­cod­ing tends to be sub­stan­tially re­duced or not mea­sur­able in higher-or­der cor­ti­cal ar­eas [51,52]. Prior re­search has shown that not only the speech en­ve­lope, but also other key fea­tures of the un­at­tended speech, such as acoustic on­sets, are neu­rally rep­re­sented [53,54] and, when not read­ily avail­able due to speech mask­ing, they are even re­stored at later tem­po­ral scales [55]. One in­ter­pre­ta­tion is that en­cod­ing a tem­plate struc­ture of the un­at­tended speech might be a use­ful strat­egy to sup­press it [53]. Other re­search found at­ten­tional fluc­tu­a­tions cor­re­spond­ing to the changes in Target-Masker rel­a­tive sound en­ergy, with ev­i­dence that our brains may en­code some pho­netic in­for­ma­tion of un­at­tended streams [56,57]. The en­cod­ing of such un­at­tended stream in­for­ma­tion may be one of the fac­tors fa­cil­i­tat­ing the rapid en­gage­ment dur­ing at­ten­tion switch­ing.

Using al­pha ERSP as in­di­ca­tor of lis­ten­ing ef­fort, this study re­lated per­cep­tual de­mands with at­ten­tion switch­ing dy­nam­ics. A large re­duc­tion in EEG al­pha-band power was mea­sured con­sis­tently about 4.5 s af­ter the at­ten­tion-switch­ing cue. The ERSP tra­jec­tory sug­gests that a strong lis­ten­ing ef­fort per­sists through­out the at­ten­tion switch, with a sub­stan­tial re­duc­tion near the end of the switch (Fig 2A). Interestingly, the trough of the al­pha ERSP ob­served in this study roughly cor­re­sponds to the mo­ment when the new tar­get stream be­comes fully tracked, i.e., when the en­gage­ment process is com­pleted, which is well be­fore com­ple­tion of the dis­en­gage­ment process. Since the at­ten­tion switch­ing process can be deemed com­pleted when cor­ti­cal track­ing mea­sure­ments re­turn to pre-switch lev­els for both streams, this re­sult points to a link be­tween al­pha power and the en­gage­ment process specif­i­cally. Another pos­si­bil­ity is that the al­pha ERSP dy­nam­ics re­flect a com­bi­na­tion of lis­ten­ing ef­fort while re­fo­cus­ing at­ten­tion on the new tar­get stream and ac­tive sup­pres­sion of the new masker stream. When the newly at­tended stream is tracked at pre-switch lev­els, a suf­fi­cient acoustic and lin­guis­tic con­text on that stream may have been ac­cu­mu­lated to fa­cil­i­tate the track­ing, re­leas­ing cog­ni­tive re­sources. Future stud­ies could ex­plore this pos­si­bil­ity by ex­am­in­ing how the switch dif­fi­culty in­flu­ences the cor­re­spon­dence be­tween cor­ti­cal track­ing asym­me­try and al­pha ERSP. This find­ing ex­tends prior re­search on the neural cor­re­lates of se­lec­tive at­ten­tion and lis­ten­ing ef­fort. Variations in al­pha-band ac­tiv­ity have been as­so­ci­ated not only with au­di­tory at­ten­tion ef­fort [58 – 62], but also with vari­a­tions of in­ter­nally- ver­sus ex­ter­nally-fo­cused brain states [63], and with the ac­tive sup­pres­sion of ir­rel­e­vant in­for­ma­tion in ac­cor­dance with be­hav­ioral goals [64 – 66]. Notably, cog­ni­tive load and the in­hi­bi­tion of ir­rel­e­vant stim­uli ap­pear to be even more strongly in­flu­enced by at­ten­tion re­ori­ent­ing than by main­tain­ing at­ten­tion on a sin­gle speaker [29,67], high­light­ing the in­creased cog­ni­tive de­mands of switch­ing at­ten­tion. While al­pha power has com­monly been linked to sub­jec­tive lis­ten­ing fa­tigue [58,60] or the sig­nal-to-noise ra­tio (SNR) be­tween at­tended and un­at­tended streams [61], it is less fre­quently as­so­ci­ated with neural mark­ers of speech track­ing [68,69]. In this study, we iden­ti­fied a link be­tween the tem­po­ral dy­nam­ics of neural speech track­ing and lis­ten­ing ef­fort, sug­gest­ing a po­ten­tially valu­able met­ric for fu­ture re­search into at­ten­tion-switch­ing chal­lenges.

While cor­ti­cal track­ing and al­pha ERSP met­rics can be used to in­ves­ti­gate how at­ten­tion is dy­nam­i­cally re­al­lo­cated be­tween com­pet­ing streams, they do not spec­ify how higher-level lin­guis­tic rep­re­sen­ta­tions are up­dated dur­ing the at­ten­tion switch. Considering the im­por­tance of con­text in speech com­pre­hen­sion, it is par­tic­u­larly rel­e­vant to in­ves­ti­gate how it is ad­justed while shift­ing at­ten­tion from one speech stream to an­other. Addressing this ques­tion re­quires fo­cus­ing on neural rep­re­sen­ta­tions with rel­a­tively ex­tended tem­po­ral dy­nam­ics, which can be mean­ing­fully com­pared across com­pet­ing mod­els, and with suf­fi­cient dis­tinc­tion from the speech en­ve­lope track­ing to en­able the iso­la­tion of re­lated neural sig­na­tures. Lexical en­tropy and sur­prisal, with their long-la­tency and widely stud­ied neural sig­na­tures, are an ideal choice for char­ac­ter­iz­ing the im­pact of con­text on at­ten­tion switch­ing. Other in­for­ma­tive speech prop­er­ties such as phonol­ogy or prosody would also be im­por­tant to ex­am­ine; how­ever, their fast tem­po­ral dy­nam­ics chal­lenge the iso­la­tion of their at­ten­tion-switch­ing dy­nam­ics from the speech en­ve­lope track­ing. We there­fore fo­cused on how se­man­tic pre­dic­tions are up­dated fol­low­ing an at­ten­tion switch, mod­el­ing whether lin­guis­tic con­text is main­tained, re­set, or up­dated dur­ing the process.

This study com­pared four con­text-ac­cu­mu­la­tion mod­els that var­ied in their sen­si­tiv­ity to at­ten­tion switches and ac­cess to prior con­text: an Oracle model (context-rich but switch-un­aware), Speaker Specific and Attention mod­els (both con­text-rich and switch-aware but dif­fer­ing in stream se­lec­tiv­ity), and a Reset model (aware of the switch but lim­ited to the cur­rent block’s con­text). For each model, a mul­ti­vari­ate TRF was fit us­ing a se­man­tic re­gres­sor aligned with the re­spec­tive con­text strat­egy. Our data in­di­cates that the Reset model best pre­dicted EEG data, out­per­form­ing mod­els that re­tained past con­text and chal­leng­ing the as­sump­tion that prior se­man­tic in­for­ma­tion aids com­pre­hen­sion dur­ing at­ten­tion switches. This find­ing was un­ex­pected but one of the pos­si­ble out­comes that we had hy­poth­e­sized, and it may sug­gest that lis­ten­ers re­set con­text and re­cal­i­brate their lex­i­cal pre­dic­tions dy­nam­i­cally when switch­ing at­ten­tion to a new stream, in line with find­ings in the episodic mem­ory and event-seg­men­ta­tion lit­er­a­ture [70 – 72].

Interestingly, the mul­ti­vari­ate TRF analy­sis re­vealed fine-grained dif­fer­ences among the four con­text mod­els when lex­i­cal en­tropy is used as the se­man­tic re­gres­sor, but not for lex­i­cal sur­prisal. This may re­flect the for­ward-look­ing na­ture of en­tropy, as op­posed to the re­ac­tive na­ture of lex­i­cal sur­prisal. Another con­sid­er­a­tion is that par­tic­i­pants ac­tively per­formed the at­ten­tion switch, there­fore ex­pect­ing to en­counter dif­fer­ent speech ma­te­r­ial, po­ten­tially damp­en­ing the sur­prisal. This was not the case for the LLM model, which did not re­ceive a switch cue. This pos­si­ble mis­match be­tween LLM and hu­man brain could im­pact sur­prisal but not en­tropy (as that re­flects the process of con­text-build­ing), pro­vid­ing a po­ten­tial ex­pla­na­tion of why the Reset model per­forms best only for the en­tropy re­gres­sor. For in­ter­pret­ing these re­sults, it is also im­por­tant to con­sider that LLMs like Mistral are op­ti­mized for next-word pre­dic­tion, with­out a re­quire­ment of be­ing neu­ro­phys­i­o­log­i­cally plau­si­ble. While one in­ter­pre­ta­tion is that our brain’s lex­i­cal pre­dic­tions are built af­ter re­set­ting the con­text, it is also pos­si­ble that Mistral LLM and our brains deal with these speech dis­con­ti­nu­ities in a dif­fer­ent way al­to­gether. With these caveats in mind, counter to con­sis­tent re­ports of the sim­i­lar­ity be­tween mod­ern LLMs with neu­ro­phys­i­o­log­i­cal ac­tiv­ity [73 – 75], the higher neu­ro­phys­i­o­log­i­cal plau­si­bil­ity of the Reset model sug­gests that prior con­text is not availed of by our brains in the way im­ple­mented by the com­pet­ing mod­els, Attention and Spk-Specific.

Other strate­gies mak­ing use of the prior con­text might also be in place. One pos­si­bil­ity is that switch­ing at­ten­tion prompts a dif­fer­ent use of con­text, for ex­am­ple sum­ma­riz­ing its ab­stract mean­ing, as the gist of the story [16]. As such, there could be value in ex­plor­ing dif­fer­ent strate­gies for con­text rep­re­sen­ta­tion, for ex­am­ple, by em­ploy­ing Large Concept Models [76], which are trained and op­ti­mized for sen­tence pre­dic­tion. This lat­ter pos­si­bil­ity is also sup­ported by re­cent work on the ac­cu­mu­la­tion of lin­guis­tic con­text in LLMs and the hu­man brain [77] while lis­ten­ing to con­tin­u­ous mono­logues, show­ing that LLMs with a lim­ited con­text win­dow (32 to­kens) and with ac­cess to a coarse sum­mary of the pre­vi­ous con­text pre­dict neural ac­tiv­ity bet­ter than LLMs with a higher to­ken-mem­ory.

In sum­mary, this study showed that the process of at­ten­tion switch­ing in a re­al­is­tic multi-talker sce­nario can be in­ves­ti­gated in terms of its en­gage­ment and dis­en­gage­ment com­po­nents, with a tran­sient par­al­lel rep­re­sen­ta­tion of the two streams. We high­light the im­por­tance of re­lat­ing met­rics of neural track­ing of speech with met­rics of lis­ten­ing ef­fort and demon­strate that the lis­ten­ing ef­fort starts de­creas­ing fol­low­ing suc­cess­ful dis­am­bigua­tion of the two streams dur­ing at­ten­tional re-al­lo­ca­tion. Finally, we in­tro­duce an ap­proach for mod­el­ing lex­i­cal con­text of dy­namic at­ten­tion sce­nar­ios, show­ing the sen­si­tiv­ity of trans­former-based lan­guage mod­els to sub­tle dif­fer­ences in con­text ac­cu­mu­la­tion strate­gies. These find­ings have im­pli­ca­tions for fu­ture in­ves­ti­ga­tions into the cor­ti­cal mech­a­nisms of at­ten­tion re-ori­ent­ing and can be em­ployed to high­light dif­fer­ences across di­verse pop­u­la­tions in terms of age and hear­ing lev­els.

Methods

Ethics state­ment

Written in­formed con­sent was ob­tained from the study par­tic­i­pants. The study was con­ducted in ac­cor­dance with the Declaration of Helsinki, and the pro­to­col was ap­proved by the School of Psychology Research Ethics Committee of Trinity College Dublin (ethics ap­proval num­ber: SPREC012023 – 08).

Participants and ex­per­i­men­tal pro­ce­dure

We re­cruited 24 young na­tive English speak­ers (between 18 and 39 years of age) to take part in the study. Participants had nor­mal hear­ing, as per a screen­ing pure tone au­dio­gram from 0.25 Hz to 8 kHz and re­ported no his­tory of neu­ro­log­i­cal or psy­chi­atric dis­or­ders and had nor­mal or cor­rected-to-nor­mal vi­sion.

The ex­per­i­ment sim­u­lated a multi-talker sce­nario (Fig 1A) with a cir­cu­lar ar­ray (1.50m ra­dius) of six loud­speak­ers sur­round­ing the lis­tener (at hor­i­zon­tal an­gles of ±30°, ±112.5°, and ±157.5° rel­a­tive to the par­tic­i­pant). Participants were in­structed to dy­nam­i­cally switch their at­ten­tion be­tween left and right speech streams in the fore­ground, fol­low­ing a vi­sual cue (left- or right-point­ing ar­row) in­di­cat­ing the to-be-at­tended side, which was dis­played at the cen­ter of a screen placed in front of them. While switch­ing their at­ten­tion be­tween the fore­ground streams, they were also asked to ig­nore a 16-talker noise played from the four loud­speak­ers in the back­ground (B1-B4, each of them de­liv­er­ing a 4-talker bab­ble). Frontal streams were pre­sented at 60 dB sound pres­sure level (SPL) each, while each of the noise bab­bles was de­liv­ered at a level of 54dB SPL, re­sult­ing in a 3dB SNR of the fore­ground rel­a­tive to the back­ground.

Participants were pre­sented with 20 tri­als (lasting 180 s each) and had to per­form 6 at­ten­tion switches per trial, oc­cur­ring at semi-ran­dom in­ter­vals (Fig 1B). For this rea­son, blocks of sus­tained at­ten­tion to one par­tic­u­lar speech stream var­ied con­sid­er­ably in du­ra­tion, span­ning from 10 to 30 s. For each trial, a dif­fer­ent male and fe­male speech stream was played from the left and right loud­speak­ers in the fore­ground, coun­ter­bal­anc­ing through­out the ex­per­i­ment for side of pre­sen­ta­tion and start of the at­ten­tion block (i.e., the ex­per­i­ment con­sisted of five sub-blocks with the fol­low­ing trial se­quence: Male Left — Attention Start: Left; Female Left — Attention Start: Left; Male Left — Attention Start: Right; Female Left — Attention Start: Right).

Each trial started with the vi­sual cue point­ing to­wards the to-be-at­tended side and back­ground noise only, fol­lowed by the two fore­ground speech streams start­ing si­mul­ta­ne­ously af­ter 5 s. At the end of each trial, par­tic­i­pants an­swered three mul­ti­ple-choice ques­tions. First, they were pre­sented with a four-al­ter­na­tive forced-choice ques­tion re­gard­ing the con­tent of the at­tended speech stream. As at­ten­tion al­ter­nated be­tween the left and right speech streams over the course of the trial, the ques­tion could be about ei­ther of the two com­pet­ing streams. The sec­ond ques­tion was a bi­nary choice as­sess­ing par­tic­i­pants’ pref­er­ence be­tween the two streams (left or right) based on per­sonal in­ter­est. Finally, a 5-point Likert-scale ques­tion (1: very easy; 5: very hard) was used to quan­tify the per­ceived dif­fi­culty of the at­ten­tion switch­ing task. The ex­per­i­ment flow was self-paced and, to min­i­mize fa­tigue, it in­cluded three manda­tory breaks, each last­ing no less than five min­utes, every fifth trial.

Speech streams were pre­sented at a sam­pling rate of 44.1 kHz, de­liv­ered through a Roland Octa-Capture 10 × 10 sound card (24-bit/192 kHz), and played through six PreSonus Eris 4.5BT loud­speak­ers. Participants’ EEG ac­tiv­ity was recorded us­ing a BioSemi ActiveTwo sys­tem at a sam­pling rate of 512 Hz, from 64 elec­trodes po­si­tioned on a stan­dard cap fol­low­ing the International 10/20 sys­tem. An ac­tive (CMS) and a pas­sive (DRL) elec­trode were used as ref­er­ence for all elec­trodes, and two ad­di­tional elec­trodes were placed on the mas­toids for of­fline ref­er­enc­ing. For 21 of our 24 par­tic­i­pants, we ad­di­tion­ally recorded elec­tro-ocu­log­ra­phy (EOG) and elec­tro-myo­g­ra­phy (EMG). Two elec­trodes were placed on the left and right tem­ples to cap­ture hor­i­zon­tal eye move­ments, and two elec­trodes were po­si­tioned above and be­low the left eye to record ver­ti­cal eye move­ments and blinks. To cap­ture EMG ac­tiv­ity re­lated to head ro­ta­tion, an elec­trode was placed on the left del­toid mus­cle. Please note that ac­tiv­ity from these ex­ter­nal elec­trodes has not been an­a­lyzed as part of this study.

Stimuli

The fore­ground speech stim­uli in­cluded 40 TED Talks cov­er­ing a range of top­ics, with 20 fe­male and 20 male pre­sen­ters, each speak­ing in a va­ri­ety of English ac­cents. All speech streams were root-mean-squared (RMS) nor­mal­ized to re­duce dif­fer­ences be­tween male and fe­male voices. Each of the 4-talker back­ground bab­ble sig­nals was ob­tained by sum­ming the au­dio sig­nals of four sep­a­rate TED talks. The long-term av­er­age spec­trum of the bab­ble noise was then ad­justed to align with the over­all spec­trum of both male and fe­male fore­ground speak­ers, to pre­vent in­con­sis­ten­cies in mask­ing.

EEG data pre­pro­cess­ing

Neural data were an­a­lyzed with cus­tom scripts in MATLAB soft­ware (MathWorks), based on pub­licly avail­able scripts and re­sources shared as part of the CNSP ini­tia­tive (Cognition and Natural Sensory Processing; https://​cn­sp­work­shop.net). Neural sig­nals were first band-pass fil­tered be­tween 0.5 Hz and 8 Hz, us­ing a zero-phase shift Butterworth fil­ters of or­der 4, and then down­sam­pled from 512 to 64 Hz. Spherical spline in­ter­po­la­tion was ap­plied to re­place chan­nels that were three stan­dard de­vi­a­tions away from the mean. EEG was then re-ref­er­enced to the av­er­age of the two mas­toid chan­nels.

Speech fea­tures

The cur­rent study aimed to char­ac­ter­ize the neural track­ing of a dy­namic multi-talker sce­nario by mea­sur­ing the re­la­tion­ship be­tween EEG data and var­i­ous fea­tures of the fore­ground speech stim­uli, re­lated to their acoustic and lex­i­cal prop­er­ties. To model the speech acoustics, the au­dios’ broad­band am­pli­tude en­velopes were ex­tracted by tak­ing the ab­solute of the Hilbert trans­form. In or­der to model the lex­i­cal prop­er­ties of speech, the tran­scribed stim­uli and their au­dios were first au­to­mat­i­cally aligned us­ing the WebMAUS Basic aligner [78 – 80], which iden­ti­fied time­stamps cor­re­spond­ing to the start and end of each word. The re­sult­ing au­to­matic align­ment was saved in the TextGrid for­mat and ad­justed man­u­ally, when nec­es­sary, us­ing the Praat soft­ware [81]. The time stamps were then used to build bi­nary word on­set vec­tors in MATLAB. While bi­nary word on­set vec­tors rep­re­sent in­for­ma­tion re­lated to word seg­men­ta­tion, they can also be mod­u­lated ac­cord­ing to each word’s sur­prisal or en­tropy value to rep­re­sent higher-or­der se­man­tic in­for­ma­tion. Word sur­prisal is a mea­sure of how un­ex­pected a word is given its pre­ced­ing lin­guis­tic con­text, and it can be com­puted us­ing Large Language Models (LLMs), as the neg­a­tive log­a­rithm of the prob­a­bil­ity of that word given the pre­vi­ous con­text. Word en­tropy, on the other hand, mea­sures how un­cer­tain or un­pre­dictable the next word is. Here, we used a pre­trained open-source LLM, Mistral-7B-v0.1 [82] to ex­tract word prob­a­bil­i­ties, and then com­puted lex­i­cal sur­prisal and lex­i­cal en­tropy val­ues for each word.

When con­sid­er­ing the dy­namic na­ture of the at­ten­tion switch­ing task, the de­f­i­n­i­tion of what con­text to in­clude for the cur­rent word pre­dic­tion be­comes a non-triv­ial prob­lem. From the ma­chine per­spec­tive, given a word w be­long­ing to, e.g., the left stream, the LLM would pre­dict it more eas­ily when pro­vided with all the avail­able lin­guis­tic con­text from the left stream. However, from the neural/​be­hav­ioral per­spec­tive, since par­tic­i­pants were flex­i­bly re-ori­ent­ing their at­ten­tion be­tween left and right streams, the op­ti­mal con­text would be im­pacted by the at­ten­tion switch and, po­ten­tially, store in­for­ma­tion of pre­vi­ously at­tended blocks. To com­pare these con­text-ac­cu­mu­la­tion al­ter­na­tives, we rep­re­sented con­text ac­cord­ing to four al­ter­na­tive rep­re­sen­ta­tions. A ma­chine-ideal model, Oracle, con­sid­ers as con­text all words pre­ced­ing the cur­rent word in one par­tic­u­lar stream, whether they were at­tended or un­at­tended. As such, this model is un­aware of the switch of at­ten­tion. Among more neu­rally-plau­si­ble and switch-aware con­text ac­cu­mu­la­tion mod­els, we con­structed the Attention model, which in­cor­po­rates as con­text any pre­vi­ously at­tended block from both left and right speech streams, and the Speaker Specific model, which dis­plays a speech stream bias, whereby only pre­vi­ously at­tended blocks of the same stream are in­cluded as con­text for lex­i­cal pre­dic­tion of the cur­rent block. Similarly to Attention and Speaker Specific, the fourth model, Reset, is switch-aware, but not con­text-aware. In fact, it does not keep track of any pre­vi­ous speech block, nei­ther at­tended nor un­at­tended, and in­stead re­sets the con­text fol­low­ing each at­ten­tion switch cue.

Temporal Response Function (TRF) and analy­sis pro­ce­dure

Quad-rotor drone shots taken low to the ground are dif­fi­cult: GPS al­ti­tude is fairly rough on ac­cu­racy, and ob­sta­cle avoid­ance can get sig­nif­i­cantly more dif­fi­cult ver­sus just fly­ing over the every­thing. Cinema rover drones are less com­mon but do get around a num­ber of these prob­lems, es­pe­cially if the sub­jects are not high off the ground. As a fan of kart­ing, light­weight con­trap­tions and pho­tog­ra­phy this seems like a pretty good pro­ject: Tackle a me­chan­i­cally sta­bi­lized video plat­form, pi­lot it re­motely and cap­ture some out­door ac­tion.

Taking a look at the Chassis

I picked up this from BMI sur­plus [link], dur­ing an ad­ven­ture with Jake Hecla [link] and the mys­te­ri­ous Arsenio [link]. For some quick back-story, BMI sur­plus is this in­cred­i­bly in­ter­est­ing sur­plus em­po­rium in Massachusetts, while they do not have tours, you can pick up items that you pur­chase ahead of time. We were for­tu­nate to get some time to browse in­side the in­cred­i­bly dense ar­rays of ma­chines, gad­gets and giz­mos. A lot of this stuff ap­pears to come from Lincoln Laboratory, the friendly neigh­bor­hood spooks. I was able to hag­gle a bit and pur­chased the mys­tery RC chas­sis for ~50 USD + tax, along with some other items.

There was no ac­tual in­for­ma­tion about this thing, what it was used for, why it had a bizarre Z-Axis lin­ear ac­tu­a­tor bolted to the top. I did some dig­ging but was un­able to find any write-ups, tech­ni­cal pa­pers or any­thing about this ‘thing’. Maybe they were sur­vey­ing, maybe it was a mo­bile de­vice to take im­ages at the door height of a car? No idea

Fortunately the ac­tu­a­tor was a sim­ple DC brushed mo­tor lin­ear ac­tu­a­tor, which was easy to con­trol. Behold a scaled dolly shot of this con­trap­tion in-ac­tion.

There is an an­i­mated video here tag.

While this thing was in­ter­est­ing, there was no way that I was go­ing to even try putting a full sized cam­era gim­bal on this thing, so af­ter a few screws the Z-Axis was re­moved, un­der­neath we see a wa­ter-jet plate on, the most com­i­cally du­bi­ous 4 – 40 stand­offs I have ever seen. 4 inch long 4 – 40 stand-offs is wild.

Hardware Mock Up

For the pur­poses of de­ter­min­ing what this could look like, and if it was go­ing to be too un­wieldy, I opted to tem­porar­ily use the ex­ist­ing mount­ing plate, make a 3D printed adapter and get a hint as to what I’d be work­ing with.

First up was to re­move the han­dle as­sem­bly that’s na­tive to the Movi M10, which in­her­ently flips this whole gim­bal up­side-down. I was some­what con­cerned with how much load­ing would now be on these four M3 screws, but that’s some­thing that fu­ture-Dane needs to re­solve. Shown be­low is the gim­bal, with the mount point shown. Likely, go­ing for­ward I would prob­a­bly end up pick­ing up the cen­tral 4 screws and try and me­chan­i­cally pro­vide some ad­di­tional sup­port.

I used some quick ther­mal in­serts to pick up on the long flat por­tions of the ex­ist­ing chas­sis. By us­ing flat-head screws, I can take up a bit of tol­er­ance mis­match, and when loose it can slide back and for­ward down the top plat­form. This print was a place­holder, but it did let me get a good vi­sual of how tall this whole stack-up would be.

Finally, the first test fit to see what the gim­bal looked like on the chas­sis. While this is a quick mock up it did pro­vide some in­sight as to how quickly the height of the whole as­sem­bly could eas­ily creep up if it was­n’t ac­tively be­ing con­strained. The lower the cam­era and as­sem­bly height, the lower the ef­fec­tive ve­hi­cle cen­ter of grav­ity re­mains, in­creas­ing its sta­bil­ity while help­ing re­duce rollover forces. Making a mock-up may take time but it re­ally lets you skip an it­er­a­tion step as in­stead of an ill-de­fined cad model you now have some­thing to in­ter­act with on the lab bench.

There is an an­i­mated video here tag.

It is huge. I took the op­por­tu­nity to also do some place­ment tests for where the bat­tery mounts could end up. As it was fairly ap­par­ent, us­ing bat­ter­ies as bumpers is not a great plan. The two packs would likely need to lay as low as pos­si­ble but still re­main ac­ces­si­ble for hot-swap­ping two packs at a time. The only rea­son­able spot would be along the sides, while not in­ter­fer­ing with the gim­bal or the ground clear­ance.

New Mechanicals

A more struc­tural gim­bal mount

The floppy alu­minum mounted on some adorable 4 – 40 stand­offs was not go­ing to do it, I needed some­thing much more struc­tural. While the gim­bal it­self is not me­chan­i­cally heavy, its mount point does need to be as low as pos­si­ble to help mit­i­gate flip­ping. Ideally the Gimbal-camera as­sem­bly is re­move-able from the frame, such that i can test and tweak the mo­tor con­trol tun­ing, fend­ers and the like with­out putting every­thing else through hell. Time to fire up the CNC, turn some proper stand­offs and rigidly con­nect to the frame.

The ba­sic plot is to pro­vide a solid foun­da­tion as close as me­chan­i­cally pos­si­ble to the frame for the gim­bal and shock mount. Fortunately the sub-frame base plate is made of alu­minum. I’m go­ing to again opt for stand-of­f’s to el­e­vate the plat­form above the drive mo­tor, but just use some large round-stock to pro­vide a se­cure mount.

There is an an­i­mated video here tag.

After some quick ma­chin­ing, drilling and tap­ping we have our new el­e­vated gim­bal plat­form, with coun­ter­sunk M8 flat head screws. This in­ten­tion­ally barely clears the drive mo­tor, and leaves the cen­tral round part of the spring damper mount re­cessed to keep the Z-offset height as low as pos­si­ble.

The spac­ing of the stand­offs is nom­i­nally tied to the clos­est po­si­tions that I could pick up that were co-pla­nar with­out bump­ing into the mo­tor mount / servo mounts. While this is slightly aft of cen­ter, it does per­mit space for the some­what heavy bat­ter­ies to live, ide­ally re­sult­ing in the to­tal mass bal­ance to­wards the cen­ter.

After ver­i­fy­ing lo­ca­tion a few times, I punched some holes through the chas­sis and used some M8 screws with flanges to firmly at­tach the new plat­form to the chas­sis. These were ini­tially just tight­ened to a few new­ton me­ters, but on the fi­nal as­sem­bly did re­ceive some mild loc­tite to help with vi­bra­tion in­duced loss of ten­sion.

Fenders for Ice Racing

One of the big is­sues with rac­ing on a slushy sur­face is the slush get­ting every­where . I do not have a tra­di­tional chas­sis for this ve­hi­cle, so it’s up to me to fig­ure out how to con­tain the slush, while not be­ing too in­flex­i­ble. This is a great op­tion for 3D printed parts, as it’s a lot of odd shapes and con­tours, how­ever, this is also a bat­tle bots crossover episodes , and oddly, flex­i­ble things should out-per­form sta­tic things. Initially, for it­er­at­ing and quick pro­to­typ­ing, the fend­ers are nor­mal PLA.

Comically, up to this point, I have never ac­tu­ally printed with com­mer­cial TPU, which is the go-to plas­tic flex­i­ble elas­tomer. I cant think of a bet­ter use case than fend­ers on a RC car frame. Just like a com­muter-bike, the more of the wheel path that is cov­ered, the less can get sprayed onto the cam­era and gim­bal. The plot is to pickup a hard mount, which in this case is some alu­minum an­gle stock, use a stan­dard PLA part to pro­vide a mount­ing point close to the wheel, and then tran­si­tion into a TPU part around the wheel.

I had ini­tially tried 85A TPU and it was just a bit too fid­dly to re­li­ably print, with­out re-do­ing my fil­a­ment spool hold­ers to be lower re­sis­tance. Any re­sis­tance was caus­ing un­der-ex­tru­sion and it was dif­fi­cult to man­age. I opted for switch­ing to 95A fil­a­ment, which is stiffer, and did a sub­se­quent slow week­end print to make flex­i­ble tire fend­ers for the rear. The rear mounts are mir­rored but both mate with three M6 threaded flange screws. After fid­dling with set­tings to get a re­li­able print on the fend­ers, its a good idea to start plan­ning a TPU front bumper to help keep this thing from get­ting smashed too eas­ily.

The print time for a sin­gle wheel fender was ap­proach­ing a day and a half on a Prusa MK4. For this spe­cific print, I opted for high wall count 25% in­fill, this should re­sult in a fairly stiff part that’s able to ab­sorb im­pacts.

While it would have been ideal if all four fend­ers were the same part, the steer­ing up front re­quires a lot more clear­ance, re­sult­ing in a larger ra­dius. The printed part does pick up the same hard mount point

There is an an­i­mated video here tag.

I did­n’t men­tion how awk­ward re­mov­ing the sup­port ma­te­r­ial is for large TPU prints. Its time con­sum­ing just due to how im­pact and force ab­sorbent it is. The layer-layer ad­he­sion is amaz­ing.

There is an an­i­mated video here tag.

Finally a spool up of the chas­sis on stand-off blocks, fu­eled by the two se­ries DeWalt bat­ter­ies. As I learned from a friend, full RPM in this sit­u­a­tion is out­side of the mo­tor and drive-train specs, given that i was now run­ning 10S / ~40v. The short blip of full speed was plenty to see how fright­en­ing this mon­ster would be­come.

There is an an­i­mated video here tag.

To mount the side plates, i used a long tip marker to in­di­cate where the print would align with the chas­sis and used a punch to trans­fer those lo­ca­tions so they could be sub­se­quently tapped.

There is an an­i­mated video here tag.

With the holes in the gim­bal riser tapped and the chas­sis clear­ance holes drilled, it’s time to put every­thing to­gether. Due to the right side of the ve­hi­cle’s sim­plic­ity, I’m opt­ing to in­stall its cover plate first, there’s only two wires to worry about. Time to put the elec­tric screw­driver to work. Six M3 screws grab the alu­minum top-plate and six sub­se­quent M3 screws at­tach the bot­tom of the print to the frame.

This video has au­dio: Click to un-mute

Battery Mounts

Both sides of the gim­bal mount­ing plate fit in sep­a­rate, blue, printed plates that hold the main power switch, pre-charge and bat­tery mount points. The prints pick up tapped M3 holes above and con­tain M3 ther­mal in­serts be­low to pickup screws from the chas­sis.

The printed mounts then mate to an off the shelf, in­jec­tion molded DeWalt Battery ter­mi­nal, con­nect­ing with four M4 screws. With the bat­tery latched in, it is sur­pris­ingly abuse tol­er­ant. Shown be­low is a stan­dard DeWalt 6AH 20V bat­tery mod­ule. The or­ange cover cap­tures any ex­posed wiring pre­sent from the bat­tery adapter. Ideally the gap-space gets cov­ered up to help mit­i­gate ice slush ingress, but that’s a prob­lem for fu­ture Dane.

There is an an­i­mated video here tag.

For the left side of the chas­sis cover we have a lot more go­ing on. The main power switch, pre-charge and pack volt­age in­di­ca­tion is pre­sent, with as­so­ci­ated wiring on the back­side. Behind this are the dc/​dc con­vert­ers that pro­vide steer­ing power and in­di­ca­tion light power, along with our re­mote con­trolled re­lay for head­lights / tail lights.

“Headlights and Tail lights”

Having some vi­sual in­di­ca­tion on this con­trap­tion is help­ful, es­pe­cially with how quickly dusk ap­pears. A small head­light and rear fac­ing red lights should be a quick ad­di­tion:

For a head­light i opted for this wa­ter­proof small mod­ule, in­tended as a 3rd party au­to­mo­tive light. It for­tu­nately takes a wide range of op­er­at­ing volt­ages so i can put to use a 15W dc/​dc mod­ule that’s been col­lect­ing dust. For the tail lights I’m also opt­ing to re-use some 12V in­di­ca­tors pur­chased for a pro­ject from ages ago. While they are not in­cred­i­bly bright they are vis­i­ble from a rea­son­able dis­tance.

We do have a num­ber of chan­nels avail­able on this ra­dio, in­clud­ing switches. Having the abil­ity to dis­able light­ing, if it were in­ter­fer­ing with the cam­era or caus­ing re­flec­tions, would be use­ful, es­pe­cially re­motely. To im­ple­ment this I’m opt­ing to go sim­ple, a very ba­sic RC con­trolled re­lay.

With the sim­ple 3d printed brack­ets in­stalled, I was pretty happy with how things turned out.

There is an an­i­mated video here tag.

Sorting out HD FPV

There’s re­ally four op­tions for long range video links at the mo­ment, low res­o­lu­tion low la­tency ana­log, high de­f­i­n­i­tion high price DJI hard­ware, pre­vi­ous gen­er­a­tion niche cin­ema hard­ware and Open Source Build the whole thing so­lu­tions.

As of writ­ing this, the FCC has banned most of DJI’s hard­ware of­fer­ings [Link]. Given that pre­sent-gen­er­a­tion DJI hard­ware is not banned, but fu­ture gen­er­a­tions are, the prices have be­come quite high. For ref­er­ence, the trans­mit­ter alone is 1100 USD. We want some­thing equiv­a­lent that hits all three ideals: low la­tency, high res­o­lu­tion and

Let’s look at the niche cin­ema hard­ware of yes­ter­year and see if any gad­gets are avail­able for low ru­ble.

Enter the CONNEX by Amimon

This gad­get was re­leased in 2015, roughly a decade ago, but the specs are re­ally re­mark­able, es­pe­cially for the time. 1KM of range? -10C op­er­at­ing rat­ing? Not me­chan­i­cally enor­mous? sounds great!

Here’s a quick overview of the fea­tures di­rect from the man­ual. 1KM / 0.6 mile range is fan­tas­tic for that res­o­lu­tion & la­tency, and 5.8ghz an­tenna hard­ware is now fairly easy to come by. Given that we don’t re­ally know the ori­en­ta­tion of the rover in re­la­tion to the pi­lot, we’re stuck with omni an­ten­nas on the rover. The rover, un­like a quadro­tor drone, is also phys­i­cally on the ground, with the an­ten­nas barely 40 cm from the sur­face. Real world tests will likely net a shorter range but this is al­ready an ex­cel­lent start.

This is pretty ex­cel­lent, and they do ap­pear for ~100 – 200$ used on eBay, but Wait, why have I never heard of these things? Amimon was pur­chased by Teradek, who makes a very sim­i­lar item just for 5X the price. Awesome.

I pur­chased a set of trans­mit­ter and ground sta­tion ra­dios from eBay and got to work sort­ing out how I would in­te­grate them to this ve­hi­cle. One of the dan­gers of work­ing with “older” hard­ware is not the hard­ware it’s the sup­port soft­ware.

Narrators voice “there were soft­ware is­sues”

The specs are quite im­pres­sive on pa­per, specif­i­cally the <1ms la­tency . I was also im­pressed by the -10 Celsius rat­ing. On the rover side of the fence we need to pipe Mini HDMI from the cam­era into the “Air Unit” along with ~14V from the Gimbal bat­tery.

A copy of the man­ual for the Connex Amimon is avail­able here [link], with a lo­cal copy here [link]

Lets build a dis­play and FPV re­ceiver mount

Monitors for out­door use can be a bit tricky, you are in­her­ently try­ing to beat the sun. I have been a fairly big fan of Liliput and opted for their 7″ 1800 nit mon­i­tor, it sup­ports 1080P and is cov­ered in 1/4 – 20 mount points. They just work, have a wide in­put volt­age range, and are so much more rugged than mys­tery 4-character ama­zon brands.

The Connex re­ceiver is some­what large and the an­ten­nas do need to be fac­ing up­wards, so let’s stick the whole thing on the back of the Liliput mon­i­tor. There are four M2 threaded holes on the back­side of the Connex re­ceiver, so our part will pick up the sides of the Liliput and pro­vide a place for four screws to mate to the re­ceiver, hug­ging the back of the mon­i­tor. Fortunately, all the in­put/​out­puts of the re­ceiver are on the sides, so as long as we prop­erly me­chan­i­cally con­strain the ca­bling we should have a pretty ex­cel­lent lit­tle setup.

After some it­er­a­tion and fit-test­ing, I came up with a slightly more me­chan­i­cally ro­bust part, us­ing some epox­ied in M3 screws to act as me­chan­i­cal stiff­en­ers on the parts that con­nect to the side of the mon­i­tor. M3 threaded in­serts pro­vide spots to help con­strain the HDMI and power ca­bles, while keep­ing the path to the switch and link con­nec­tors ac­ces­si­ble.

Now that we have a mon­i­tor and re­ceiver for the hand­held con­trols por­tion of this pro­ject. For the ra­dio, I’m opt­ing to use a Taranis X7, mostly be­cause I picked one up at Guardian from the left­over cruft pile. I’m a big fan of the X7, I used it for SnowBot [link]. My only qualm is that there are no places to mount ex­ter­nal gad­gets or giz­mos, if this had some M6 or 1/4 – 20 threaded mount points it would be ex­cel­lent.

Long ago, Guardian Agriculture was Kiwi Agriculture

I needed a ra­dio for re­mote con­trol and planned to just have a portable dis­play that would fol­low along with that re­mote. Yes, as men­tioned by FRED, I could set up a ground sta­tion and a tri­pod but the prob­a­bil­ity of me knock­ing over a tri­pod out in the cold is very high. So let’s start out with the only ac­tual mount point on the X7, the neck-strap mount.

We also have one more ‘hard point’ that we can pick up, the an­tenna pro­tru­sion that’s in­jec­tion molded into the case. If we can pick up a hard point mount there and the neck-strap, and con­tour closely to the case we should be set for at least a first pass at a mon­i­tor mount.

After a num­ber of it­er­a­tions, I ended up with an M6 long ther­mal in­sert to pickup the neck­lace mount and a heav­ily walled hole to pickup the an­tenna mount. This breaks out into one M4 ther­mal in­sert for the mon­i­tor and two aux­il­iary M3 ther­mal in­serts for “whatever sub­se­quent spac­ing mod­i­fi­ca­tions i need”. Note that I opted for high in­fill and high wall-count for this part as the lever arm from the mon­i­tor is quite high.

It did end up work­ing out fairly well, es­pe­cially for a first pass, the mon­i­tor is quite heavy and the stock bendy mount was quite lim­ited, so some more it­er­at­ing to do.

Configuring the Connex Amimon

The nom­i­nal pair­ing process for these two ra­dios is fairly straight­for­ward, and does not re­quire any com­pan­ion soft­ware. To pair the pro­ce­dure is fairly straight­for­ward:

Apply power to air unit

press and hold link for 5 sec­onds, or un­til it starts fast-blink­ing

Apply power to ground sta­tion unit (preferably with a mon­i­tor con­nected)

press and hold link for 5 sec­onds, or un­til it starts fast-blink­ing

Follow the on screen di­rec­tions on the air unit and it will show a progress bar for pair­ing

I fol­lowed these di­rec­tions and alas, no dice. The ground unit sat in pair­ing mode for 5+ min­utes and then timed out.

I did some dig­ging and found the con­fig­u­ra­tion tool. It did “just work” right out of the box, which is great for ~10 yr old soft­ware, but I came upon my first dilemma. The ground unit and the air unit had wildly dif­fer­ent firmware ver­sions . I did at­tempt dif­fer­ent vari­a­tions of the pair­ing pro­ce­dure, but alas each go they see each other but refuse to pair. The soft­ware tool did have an up­date fea­ture, but it was too smart, it na­tively pings a server to check for new, com­pat­i­ble, firmware ver­sions. Those servers are un­for­tu­nately gone. Shown be­low is the “No Server Connection!” mes­sage, also show­ing the mis­match be­tween the Air Unit and the Ground Unit.

Time to find some help. Between 2016 and now, the ini­tial cre­ator of this hard­ware / soft­ware was ab­sorbed into Teledek, which gen­er­ally re­sults in pre­vi­ous hard­ware get­ting shelved. I was able to get in touch with a sup­port en­gi­neer and was given the best sup­port email ever: there’s an of­fline up­date mode

Enabling Offline Update Mode for the Connex Amimon

To en­able of­fline up­date mode here is the pro­ce­dure:

Download the Connex Management Tool

The tool is avail­able from the ven­dor here [Link] and a lo­cal copy is avail­able here [Link]

Extract the man­age­ment tool and in­stall

For the pur­poses of this write up, lets as­sume you are run­ning win­dows

Place a blank file in the pro­gram di­rec­tory

Create a file “local.txt” with no con­tents in the pro­gram folder, C:\Program Files (x86)\Amimon\Connex

Grab the lat­est firmware files

The lat­est firmware files are avail­able here [link] and a backup copy is avail­able here [Link]. Download and save lo­cally

Unzip the firmware, you should end up with all of the firmware op­tions for US, EU and over­seas.

Launch the Connex Software

Click up­date and browse to the cor­rect firmware

With the se­cret of­fline up­date mode, we now have the abil­ity to push these units to the same firmware ver­sion. I started with the air unit and then com­pleted with the ground unit. Shown be­low is the process, un­for­tu­nately OBS screen grab missed the ‘open a win­dow to browse to the ac­tual file’, but for ref­er­ence I used “PR_ID_UAV10100US000_PR_NAME_ConnexUS_VER_4_5_61.amn” as the fi­nal tar­get firmware for both units.

There is an an­i­mated video here tag.

It works!

Propulsion Electronics

This did come with a cas­tle cre­ations mo­tor con­troller, how­ever, I did want some­thing that I was able to ad­just set points for, and have some flex­i­bil­ity go­ing for­ward re­gard­ing au­ton­omy. I opted for a VESC 75V100, nom­i­nally as I had one avail­able and had used one on a pre­vi­ous pro­ject. The 75V100 is very bud­get friendly, and just re­quires some sil­i­cone glue to make it ro­bust enough to sur­vive shock and vibe. Internally there is a large elec­trolytic ca­pac­i­tor that has no me­chan­i­cal con­straints.

For the ini­tial spool-ups, I used a bench sup­ply at 30V, which hon­estly is in­ad­vis­able. Bench sup­plies are not four quad­rant de­vices and re-gen cur­rents from the mo­tor spool­ing down can over-volt the sup­ply and cause is­sues with the con­troller. Nominally I was mostly in­ter­ested in ver­i­fy­ing that the VESC could run the mo­tor sen­sor-less

Fortunately, even with a ba­sic tune we got what we were look­ing for, mo­tor char­ac­ter­is­tics: 5 mOhm phase re­sis­tance and low phase in­duc­tance. With this ba­sic in­for­ma­tion we can do a quick spool up test, mak­ing sure to not spool down quickly.