hckrnws

The article fails to distinguish between hard real-time and soft real-time.

If you're controlling a bandsaw, you've got a hard real-time application. You can't miss your window for the next instruction.

Most user interfaces are soft real-time. Occasionally missing your window is fine. And so it is OK to do things like hash lookups whose average performance is O(1), and whose worst case performance is O(n). Ditto for dynamically resizing arrays. As long as you're back in time, most of the time, it can be OK.

The problem isn't that we code UIs as soft real-time. It is that the soft bit gets squishier with each layer of abstraction. And there is a feedback loop where people get more and more used to slow applications, so nobody is concerned if their application is unnecessarily slow as well.

Compounding this is the fact that we often measure performance in terms of throughput, instead of latency. Therefore, every single device and peripheral is willing to lose a bit of latency. It happens at all levels. The classic I like to quote is http://www.stuartcheshire.org/rants/latency.html.

And the result is that modern applications on modern hardware are less responsive than older applications on older hardware. Fixing it is a question of fixing a lot of little problems. And, as we used to joke about Microsoft, what's the point of developing fault-tolerant software when we've already developed fault-tolerant users?

Agreed here. While I wish more frameworks did what Android did, and allow you to literally block file and network access on the main thread (opt-in strict mode that I wish was default enabled)... worrying about paged memory in an active GUI is so, so much further down the hard-real-time branch of things that the effort/reward ratio is literally insane in almost all cases. It may be a fun challenge for some, but it'll absolutely never be a major use case, nor should it be.

Even if you rebuilt the the entire OS and libraries on a separation of sync vs async code, you still come to the inevitable problem of propagating delays to the user in a predictable manner.

So I've pressed a button to load a file that's usually really fast and this time nothing happens because the async call is taking it's time. Is that being represented to the user in some meaningful way? Is the button disabled until the operation completes? Is it stuck in the down position to show I can't press it again, or does it spring back up and show a loading dialog? Is that any more clear in a real time system then it would be in a blocking situation?

The problem is not blocking vs realtime, it's about programmers not understanding all the state their code can run in, and it's not clear that a realtime system would save the user when they fall into that unknown state.

UI was pretty much solved in the 90's including the problem of instant feedback. Then abandoned for shiny stuff.

Personally I set up i3 to open most windows asynchronously, so my flow isn't interrupted. It's great, but takes a bit getting used to windows not randomly stealing focus. It's not for everyone though.

Can you be more specific about what was “solved” in the 90s? The platforms upon which apps are run now and the technologies as well as the expected capabilities of those apps have drastically changed. Not all UI development has changed just for being shiny.

I see little reason why building a UI today cannot be a superset of what was solved in the 90s so I’m curious to know what that solved subset looks like to you

Just start with the fact that all desktop programs had a menu bar where you could find every feature - and at least on Windows - also a shortcut for that feature.

This was broken with Ribbon and the hamburger menus that every application seems to have switched to for no other reason it seems than to copy Chrome.

To be fair Ribbon is somewhat useable again, but the in the first version I have no idea how people were supposed to find the open and save functions :-)

Other problems:

Tooltips are gone. Yes, I can see they are hard to get right on mobile, but why remove them on desktop while at the same time when the help files and the menus were removed?

The result is even power users like me have to hunt through internet forums to figure out how to use simple features.

Back in the nineties I could also insert a hyphen-if-needed (I have no idea what it is called but the idea is that in languages like Norwegian and German were we create new words by smashing other words together it makes sense to put in invisible hyphens that activates whenever the word processor needs to break the word and disappears when the word is on the start or in the middle of a line and doesn't have to be split.)

Back in the nineties I was a kid on a farm. Today I am a 40+ year-old consultant who knows all these things used to be possible but the old shortcuts are gone and I cannot even figure out if it these features exist anymore as documentation is gone, tooltips are gone and what documentation exist is autotranslated to something so ridiculously bad that I can hardly belive it. (In one recent example I found Microsoft had consistently translated the word for "sharing" (sharing a link) with the word for "stock" (the ones you trade).

IMO ribbon menus are when implemented correctly actually better than a menu bar. It gives icons and the benefit of a GUI to old text-menus.

Hamburger menus I disagree with but sort of understand the logic of - they're basically like making 'fullscreen mode' the default mode, and then the hamburger menu button just sort of temporarily toggles that off. It makes perfect sense on mobile (I don't think that's what you're talking about though), and on the desktop it can make sense in a web browser when you have, essentially, 3 sets of chrome - you have the desktop window, the browser's chrome, and then the website's chrome all before you get to the website's content.

A related detail, even things like icon design have gone in a strange direction. In the interest of simplicity they've gone from recognizable to rather amorphous blobs. A button for print has gone from a clearly recognizable image of a printer, enough you could probably even guess the model number, to the icon being a rounded square with another rounded square sticking out the middle top. Many of these newer icons are just too abstract and similar to one another to be recognizable, IMO, and I think the user experience suffers.

And since saving to a floppy is not a thing anymore...

What’s a printer? Does it make cheap iPads?

You list a bunch of unrelated things that has absolutely nothing to do with the topic: UIs back in the day had less latency, by not caring about accessibility, internationalization, etc - but I’m quite sure they were way worse off in terms of properly handling blocking operations: you surely know the Solitaire effect

At the very least, you knew that when the UI locks up, it's actually doing something. These days, previously-blocking operations may be run async, but the result is that, every now and then, the UI will miss the signal that the async job completed (or failed). An UI that regularly desyncs from reality is a relatively new problem.

> At the very least

I don’t care if the food is bad because it is too salty, or because it is overcooked, I still won’t eat it.

You'll reconsider when your only alternative is food that's cooked perfectly, but also bacteriologically contaminated.

There were all kinds of standards, guidelines that made people recognize what the program is doing and how to operate it. Nowadays, UIs are mostly defective, trying its best to hinder effective usage and hide/strip functionality. There are of course progress too, but something was lost that make applications today much less intuitive and hard/tough to use.

I don't understand what specifically is of concern here. Do you have exact examples?

Some platforms publish a list of recommended guidelines which are effectively a standard. For example here's one from Apple about when and how to use charts in an application: https://developer.apple.com/design/human-interface-guideline...

Some of it may be found in older HIG sources sure: https://en.m.wikipedia.org/wiki/Human_interface_guidelines

Also they were called GUI standards or UI at the time.

The modern equivalents called UX isn't reflecting the same conglomeration of standards and conventions though. So not talking about the newer stuff.

I'm no expert on it, and it required specialized expertise. It's been abandoned for mobile interfaces and the modern UX stuff, which often optimizes for design over functionality.

If you've never used old software, it's hard to explain. But old Apple or Microsoft GUI standards would cover the basics, but you'd also need to study the applications and how they presented their GUI.

While I broadly agree with the UX/HIG/design guideline issues that are common in modern software... literally none of them have anything to do with the technicals of how quickly the UI reacts to actions and renders the next state. You can have responsive UI in all of them. And all the modern ones also say "jank is bad, don't do that".

Back in the days™ there were widget libraries as part of the OS, which followed the guidelines by the OS vendor. This gave a foundation for somewhat similar behavior and standardisation of behavior. This gives usability.

Nowadays man's applications are web apps, build without such frameworks, with less UI research and even where frameworks are used they are often built with a somewhat mobile first approach.

One made up example that attempts to embody the complaint:

If I visited a site dedicated to hamburgers today, I would not be surprised if the "Log Out" button was presented as an image of a hot dog. It would be a mystery to me what that hot dog did until I clicked on it.

Compare this to 90's UI, where it would pretty much be unheard of to do something like that. It would have been a joke, or a novelty. These days that sort of ambiguous interface isn't presented as a joke or a novelty - it's the real product.

For example, Apple's human interface guidelines mandated that you have to give the user instant feedback (I think they even talked about how many milliseconds of delay are tolerable). A correctly programmed application in OS 9 would give instant feedback on each menu item selected, button pressed, and so on.

They later gave this up and almost everything else in their very reasonable guidelines based on actual research when they switched to OS X in a hurry and multithreaded everything. The early Mail application was a disaster, for example. Generally, people started complaining a lot about the spinning beach ball of death.

In contrast, modern UX guidelines are mostly about web design, how to make web pages look fancy. They also recommend instant feedback, but many libraries and application designs don't really support it.

You are replying out of context: if your file is on a network drive it doesn't matter if you have shiny UI, or text based terminal, you gonna wait for the round trip with your UI being unresponsive

That doesn't follow. Your UI can be responsive while the round trip happens. The application won't literally stop working 100%.

Depending on the application. Take single-window editor (think notepad) for example. If the "load" command is blocking, what can you do?

You cannot allow editing - the existing buffer will be replaced once load completes. You can show the menu, but most options should be disabled. And it will be pretty confusing for user to see existing file remain in read-only mode after "open" command.

The most common solution if you expect loads to be slow is a modal status box which blocks entire UI, but maybe shows progress + cancel button. This definitely helps, but also a lot of extra code, which may not be warranted if usual loads are very fast.

Well, notepad is a bad example because it has tabs now. If the load command is blocking, show an indicator of that inside the tab. The user can switch to the other tab and have a full editing experience - this buffer obviously will not be replaced.

If you pick another single-window app, my response is: that is a decision they chose to make. They can also choose to go multi-window or tabbed just like notepad did.

At the very least: allow the user to close the window or cancel the in-progress load.

You still don’t block the render thread.

Yep. Non-blocking threads was a thing in the 90's also. BeOS probably the poster-child of that.

Only if the programmer wasn't on fast local storage when they tested their code.

> solved in the 90s

Ya, you know, unless you didn't speak English, needed accessibility, had a nonstandard screen size, had a touch screen, etc

My biggest issue with i3 (which I generally love and have used for 5+ years) is that if I switch to a workspace, launch an application with dmenu, and switch to a different workspace before the application loads, the application will load on the current workspace instead of the workspace I originally launched the application from. Anyone have a solution?

If you use the same workspace for the same apps, I believe people configure i3 to force apps on the same workspace.

Not my cup of tea, but I got the same problem.

Going to try this: https://faq.i3wm.org/question/2828/open-application-and-fix-...

Same with Sway.

Anything stealing focus should be a crime. It's the reason I use i3 as well. When I use Mac OS X sometimes, things just popping to the front to offer updates or whatever makes me want to just drop the laptop in the thrashcan. Who comes up with crap like that?

Sounds great, how did you do that? What's the config option?

I believe it may depend on i3 version, but you can make it work with the no_focus command, and ie. any window title, all, etc.: https://askubuntu.com/questions/1379653/is-it-possible-to-st...

An asynchronous action should always show immediate feedback (button disabled + a loading spinner or similar). You should not be assuming that the potentially slow operation will be fast.

Exactly this, doing it right is possible but vastly more complicated and requires tons more UI work.

It's not impossible - tape drives jukeboxes with 5 minute delays aren't a new invention.

I miss running Photon on QNX, which, being a hard real-time OS, did have upper bounds on timing for many operations. Photon was a GUI for near real time interfaces for hard real time programs. It was far smoother than what we see today.

* No paging. Entire program in memory at all times. Although it was possible to put a paging library inside an application and let it manage its own paging, which was done for gcc.

* A real time CPU dispatcher. Real time priorities were strictly preemptive. Unblock a higher priority task, it starts now, not when the dispatcher gets around to it.

* The usual test for a hard real time OS is that you have an interrupt routine that senses an external pin. It unblock a user level task when the pin goes high. The user level task turns on an output pin. You hook up a scope and a square wave generator, and watch the latency. If there are outlier values on the scope, something is broken. One implication is that stuff below the OS, such as anything running in system management mode after boot, has to be eliminated. It cannot be allowed to steal cycles.

* While real time work is going on, you can still run compiles and web browsers. They get preempted. I was impressed that that worked. Our real time application had a hardware stall timer, and if the commands coming out were late, a relay tripped and everything shut down.

So, it's absolutely possible to do this. Downsides:

* You're way out of the mainstream.

* You're going through the general case every time. Fast paths are unwanted.

* Everything is always on. Power consumption is constant. No CPU slowdowns, no sleep modes, no battery saving. This is just fine when you're running a rolling mill, and 99+% of the power is driving the heavy equipment. Not so good when you're running a credit card terminal. Battery powered devices cannot run well in this mode.

> Everything is always on. Power consumption is constant. No CPU slowdowns, no sleep modes, no battery saving. This is just fine when you're running a rolling mill, and 99+% of the power is driving the heavy equipment. Not so good when you're running a credit card terminal. Battery powered devices cannot run well in this mode.

Is there no possibility for compromise here?

A credit card terminal sits idle 99%+ of the time, does little actual work, and whatever work it actually does, it's in a reaction to an input event (such as user pressing a button, or a command being sent from cash register). CPU power states are, to my understanding, something you can switch between many times in a fraction of a second. Outside of talking to the network, duration of which is by nature unpredictable, everything else falls into two modes - "slow mode" for listening to events (and perhaps doing whatever is needed to keep the network connection alive), and "fast mode" for processing those events. Sounds to me you could put a real-time guarantee on everything except the networking parts, and keep the CPU slow/sleeping for most of the time.

> Everything is always on. Power consumption is constant.

That was the case in ye olden days of single-core CPUs, but is it still true today?

I can easily imagine a computer that has one always-on core for kernel chores and hardware management, and a dozen dormant cores that wake up when userland tasks need those cycles, all running in hard real time. Or even better, a mix of real-time for UI & audio on some cores + non-time-bound jobs on others.

There's the problem of what does the power management, shut down, and start up of dormant parts of the CPU. If that steals cycles from the OS, it can interfere with real time processing.

There's no reason that CPUs couldn't be designed to avoid such interference, but don't expect to find that on x86. Not sure about ARM.

I like this perspective.

In the demoscene days, we used to obsess over 60 Hz guaranteed framerates. Code was constantly being profiled using raster bars. Our effects might have been crappy, but all was buttery smooth.

Then someone decided it would be a good idea to create a common user interface that would work on many different framerates and resolutions, and all was lost. Or people would try for complex 3D graphics and sacrifice smoothness.

Some people tried to convince us that humans have a 150ms built-in visual processing delay, and all would be fine. This is not the case, and we are still stuck with mediocre animations. The content has improved a lot though :)

> Code was constantly being profiled using raster bars.

For those wondering: we'd change the background color from, say, black to blue for the "physics" part, to green for the "drawing" part, to purple for the "audio rendering" part... All in the same frame, while the frame was being drawn.

So you'd see on the border of your screen (usually outside of the drawing area you'd have access to) approximately which percentage of each frame was eaten by "physics", graphics, audio etc. and how close you were to missing a frame.

And it was quite "hectic" typically these color "raster" bars would jump around quite some from one frame to another.

But yeah there was indeed something very special about having a pixel-perfect scrolling for a 2D game running precisely at the refresh rate (50 Hz or 60 Hz back then). And it wasn't just scrolling: games running on fixed hardware often had characters movement set specifically in "pixels per frame". It felt so smooth.

Something has definitely been lost when the shift to 3D games happened: even playing, say, Counter-Strike at 99 fps (was it even doing that? I don't remember) wasn't the same.

People don't understand that their 144 Hz monitor running a 3D game still doesn't convey that smoothness that my vintage arcade cab does when running an old 2D game with pixel-perfect scrolling while never skipping a frame.

A lot of that smoothness was the lack of sample-and-hold blur on CRTs. Gaming LCDs now have modes that strobe the backlight to simulate this effect. Additionally, the importance of sample-and-hold blur declines as the frame rate increases. A 360Hz LCD displaying 360fps motion can look reasonably sharp even without strobing.

Blur Busters has detailed explanations, e.g.:

https://blurbusters.com/blur-busters-law-amazing-journey-to-...

> People don't understand that their 144 Hz monitor running a 3D game still doesn't convey that smoothness that my vintage arcade cab does when running an old 2D game with pixel-perfect scrolling while never skipping a frame.

The actual motion-to-photon delay (measurable with a high-speed camera) in a well optimized competitive FPS is somewhere around 20ish ms or less these days, so it's on par with it in regards to the input lag, and overall it's far smoother because of the higher frame rate, better displays, and way more thought put into tiny nuances competitive players complain about. A proper setup feels extremely smooth, responsive, and immediate.

> The actual motion-to-photon delay (measurable with a high-speed camera) in a well optimized competitive FPS is somewhere around 20ish ms or less these days, so it's on par with it in regards to the input lag

I.e. more than two frames between input and output. That is quite a lot, and AFAIK a large regression.

It’s the exact same amount of time - why would the frame count matter?

1 - 2 frames is a lot of time and latency this big is perceptible in many tasks, fast-paced videogames being just one of them.

It was in reference to high refresh rate screens - 1-2 frames is not much at 144+ Hz.

There’s a big difference in latency on a 10k$ machine and my “non-budget but on a budget” 3k$ machine. You can’t realistically expect people to pay pro-gaming prices.

> People don't understand that their 144 Hz monitor running a 3D game still doesn't convey that smoothness that my vintage arcade cab does when running an old 2D game with pixel-perfect scrolling while never skipping a frame.

There are great examples even outside of the gaming world. I wish there were videos around on how fluid were Scala Multimedia presentations on a bare Amiga 500 in the 90s. An 8MHz clocked machine (7.16 MHz here in the EU for PAL synchronization) that in that field would outperform gear costing at least an order of magnitude more. It was hugely successful back in the day in many local TV stations.

Not just that. You hit a key and the game could react in literally the next frame. There was an immediacy you don't get now.

Ever since the xbox360 era of games the game world has been updating in parallel with the rendering of the prior frame at the very least, that pipeline is quite commonly 3 frames long now. That is a lot of latency to get some macro concurrency.

Metal Slug on the PS4 was jarringly unresponsive compared to my expectations, though I'm not sure I've played it on an arcade machine in 20 years so maybe it's my memory or physiology at least partly to blame.

If I recall correctly, the PS4 controller has a disgusting amount of input delay. I noticed this most strongly on the FFX Remaster where you need to hit timer events before the arrow enters the shaded box.

It doubly screws you in the chocobo racing section because the controls assume no delay, meaning you can’t respond to hazards in time and constantly overcorrect movement.

Some multiplayer network games memoise the game state. When an input event arrives from a remote player it is applied to the game state that existed at the time of the event. Then the state is fast-forwarded to the present time and the game continues. Ideally you shouldn't notice this happening.

The obvious fix for laggy input is to apply the same processing to local input, but I've not heard of anyone doing this.

(I think this technique is known as netcode.)

Early versions of Android had this feature (Android 3-ish, 4-ish?)

Touch events were run through a filter to predict where the touch position would be some number of frames in the future -- presumably to compensate for page-flipping delays. Not sure it made much difference, mostly because smooth animations on old versions of Android were extremely difficult because of inadequate CPUs/GPUs.

The classes used to do predictive tracking don't seem to be used anymore in current best practice/current API sets.

I think native PS4 games did something like this to work around it. Final Fantasy X was a PS2 game ported to PS4, so I doubt they re-wrote their controller code to that extent, especially given how noticeable it was.

Strangely enough, I had the same experience with diablo 4 until I turned down some of the graphics settings. I'm not sure if it was some DLSS / frame generation thing or what, but the game felt sloppy to me. Especially in comparison to something like diablo 2 which feels razor sharp.

I wish modern game developers & graphics pipelines would spend more effort optimizing input latency. I'll stop noticing the graphics 2 minutes into playing the game. But if the input is laggy, I'll feel that the entire time I'm playing. And the feeling of everything being a bit laggy will linger long after I stop playing.

I’m totally with you. It has a generic art style, and with optimized settings, looks like goop. There’s never going to be a game that feels like how it feels teleporting through a whole act in D2. Everything loads instantly. And it keeps doing that even at high cast speeds.

> But if the input is laggy, I'll feel that the entire time I'm playing.

You've already bought the game at this point so most devs don't care.

They care a lot if they get bad reviews online. That translates directly into sales.

An insane amount of money and effort that goes into most AAA games. It seems like a stretch to accuse the industry of not caring about the quality of the games they make.

Well, yes, I explained myself horribly.

What I meant is that most people are way more sensitive to graphics than anything else. A lot of people will never play a game that is ugly but don't seem to mind laggy inputs.

There are a lot of games that, even without "real" input lag, have tons of animation dampening/inertia to the point that it makes them IMHO unresponsive, just to improve the "cinematic feel" and avoid characters instantly changing direction.

If you prefer responsiveness to having the player character spin be perfectly smooth, well, sucks to be you, because prettier games sell better.

Obviously this doesn't apply to fighting games or competitive shooters or anything like that where responsiveness is the point of the game.

> What I meant is that most people are way more sensitive to graphics than anything else. A lot of people will never play a game that is ugly but don't seem to mind laggy inputs.

Maybe. I wouldn’t be surprised most people are much more sensitive to input latency than we think. But when there’s input latency, they have no idea that that’s the problem so instead they attribute the slushy feel of the game on something else. Or just don’t enjoy the game as much without knowing why. Game developers talk a lot about “game feel” and this is the kind of thing they’re talking about.

Yeah, there's a lot of games where I don't care about lag. Donut County is a little drifty? Great, that fits the aesthetic. But some games really need to be tight.

The Metal Slug games are already at a disadvantage because they run at 30fps, unusually low for arcade games. All else being equal, lower frame rate gives higher input latency.

FWIW I took this impetus to check my TV settings, and poking around improved things. It's still not the arcade but it's better.

I think the difference is that 3D is inherently continuous (because it is simulating a 3d space) and requires another layer of abstraction on top of the controllers to work intuitively. 2D on the other hand is pixel based which is usually discrete (the world units are uniform in pixels), and also the character movement is 1:1 mapping on the controller. These two factors make 2D feel more responsive, and I'm guessing that once VR gets better it will close the abstraction gap and make the controller mapping (body/hand movement) 1:1.

Using pixel coordinates for 2D space is like sizing web elements in pixels - it's just plain wrong and will look vanishingly small on future higher-resolution monitors.

It's just plain wrong from today's perspective. Back when pixel sizes were predictable, one could exploit this at all levels.

Pixel artists would (and still continue to) consider every individual pixel, performing anti-aliasing by hand. Animations would be timed in relative to the (fixed) framerate, leading to optimal smoothness.

Reaching this level of fluidity is only recently possible again, at the cost of vastly higher hardware requirements. Most people haven't even experienced this, because of the suboptimal period we have been in for some time now.

You win some, you lose some.

I was speaking a little more abstractly. For 2D, even if you use an intermediate coordinate system, the transformation between the two can be mapped 1:1 with the controller (in theory, not that you need to), because controllers are 2D in design (specifically the directional arrows and joysticks, which have 2D coordinates). I think allows for more natural responsiveness. If controllers had 3D joysticks, or eventually when VR gets better at hand motion, then 3D responsiveness will improve.

Wasn't it common practice on older systems like, say, the SNES Sega Genesis? A design responsive to multiple resolutions wasn't really a thing back then as far as I know.

Imagine the richness of retro gaming today if they had had the foresight to design for resolution independence. Now only a vanishingly tiny subset of the best of the best games have the financial incentive or dedicated modding community to release in high definition.

At 14 I discovered that tiny response times are essential by observing terminal echos from a TCP client and server connection. I could clearly see a 50ms difference, down to about 15ms if I recall correctly. It wasn't until years later that I discvered high response+refresh monitors and from then on anything under 120FPS is just laggy if it's a game.

> One of the fundamental problems is that many UI applications on Windows, Linux, and macOS call functions that are not specified to run in a bounded amount of time. Here’s a basic example: many applications don’t think twice about doing file IO in a UI event handler. That results in a tolerable amount of latency most of the time on standard disk drives but what if the file is stored on a network drive? It could take much longer than a second to service the file request. This will result in a temporarily hung application with the user not knowing what is happening. The network drive is operating correctly, the UI application isn’t.

Say what you will about JavaScript, but a great thing it's done for modern software is making async programming default and ergonomic. With most APIs, you couldn't block on IO if you tried. Which means web UIs never do

Huh? It used to be very common to find that a web page was stuck processing some script. Still isn't that uncommon, all told.

You are mostly right that you couldn't block on IO, necessarily, but you could still hose up the event thread quite heavily.

If you're actually doing busy logic then it'll block the UI thread, but you have to work pretty hard to do that, and IO won't block the UI thread

All you need to is `await` an IO operation on the UI thread.

My favorite is when someone builds a ton of filtering capability locally that loads all of the data and then filters it down. Was a common mistake many editors (such as emacs) used to make, so not at all blaming incompetence. But is absolutely a trap.

Not true. The await statement will kick the operation over to the event loop to wait for a response, and the thread itself will keep humming along (pulling other events off the loop, handling animations and scrolling, etc)

Not on await, no, unless the caller/awaiter is async too?

The function containing the `await` waits, but the thread does not; it switches to doing other things:

- Page scrolling/rendering

- If the user triggers an event, like clicking on a button, that JS event handler is independent from the waiting function and can run while it's waiting

- Same thing if some other `await`ed IO resolves; that function can resume while the other one is still waiting

This is why async/await exists. It allows our code to tell the runtime "let me know when this is ready, otherwise do whatever you want in the meantime". Many languages have a similar feature now, but in JavaScript it's especially fundamental to the language and ecosystem, which means code gets written this way by default, which means JS programs don't have the problem mentioned in the OP by default.

I'm curious how this works with event handlers. Suppose you have one on a button that will fetch a number from a server and show it. For fun, assume the server is basically fizzbuzz, but with a 2 second delay for every multiple of 3/5 and a 1 second delay otherwise. If I click the button 4 times quickly (less than a second for all clicks), what happens?

I'm assuming it is on the programmer to do some smart debouncing in the event handler?

Every click will immediately fire a request, and they will come back in whatever order they come back in based on the server's timing (and network latency, and whatever else)

Done the straightforward way, you will end up displaying whichever one took the longest to return. I've run into this class of bug before, and since then I make sure to design my primitives to guard against it (always assign the latest-requested instead of latest-received result into state)

Awesome, thanks for confirming my thoughts there. Was hoping to prototype a test soon.

Okay, I think I am confused about how this works in JS. I've written my fair share of event-driven code, but managed to avoid working with actual async/await. My understanding was that calling an async function does what you describe, but awaiting it literally blocks the caller thread - because what else could it block? Your description sounds like something I'd expect of a yield instruction.

It blocks the logic flow of the current function, but in a way that can be resumed later

If it helps you think about it, any await statement can be converted straightforwardly to an old-fashioned promise resolution with a callback:

  async function foo1() {
    doStuff()

    const res = await doIO()

    return 'Result: ' + res
  }

  function foo2() {
    doStuff()

    return doIO().then(res => {
      return 'Result: ' + res
    })
  }

That's what I thought. But then, unless foo2() is an async function, the entire thread that's running foo2() will be blocked, will it not?

Nope. It returns a promise, which other logic can then hang a callback off of. But the promise is returned immediately (if just resolves and calls the callback later, like any event listener would)

The reality is that optimizing for the common scenario just gives such significant benefits that the worst-case scenario becomes practically infeasible to even consider.

And it's not just OSes that don't particularly care about real-time - modern processors don't either. e.g., in the common scenarios it's possible to load maybe eight 8-byte values per nanosecond (maybe eight 32-byte values if you're doing SIMD), but if they're out of cache, it could take hundreds of nanoseconds per byte. Many branches will be predicted and thus not cost affect latency or throughput at all, but some will be mispredicted & delay execution by a dozen or so nanoseconds. On older processors, a float add could take magnitudes more time if it hit subnormals.

If you managed to figure out the worst-case timings of everything on modern processors, you'd end up at pre-2000s speeds.

And virtual memory isn't even the worst OS-side thing - disk speed is, like, somewhat bounded. But the number of processes running in parallel to your app is not, so if there are a thousand processes that want 100% of all CPU cores, your UI app will necessarily be able to utilize only 0.1% of the CPU. [edit note: the original article has been edited to have a "Real-time Scheduling" section, but didn't originally]

So, to get real-time UIs you'd need to: 1. revert CPUs to pre-2000s speeds; 2. write the apps to target such; 3. disable ability to run multiple applications at the same time. Noone's gonna use that.

I feel like this comment is from an alternate reality where the Windows lock screen doesn't just drop all your inputs for a random period of at least a few seconds when you try to enter your password.

It is very much true that there's a very large amount of things that could be improved on the status quo with rather minimal amounts of effort, that could resolve a vast majority of the brokenness of responsiveness of UIs. But decreasing dropped frames/inputs by 10x or 100x is still just simply not gonna make it "real-time".

I remember the windows lock screen being particularly annoying (an animation had to finish..?) but on Linux Mint I've always been able to start typing away immediately, so much so that I start blindly typing in the password before my monitor has finished turning on. Properly queueing events (keyboard ones at least) should be pretty simple to do properly, but of course many things still get it wrong.

Ooh I know how to fix this one! gpedit.msc > Computer Configuration > Administrative Templates > Control Panel > Personalization > Do not display the lock screen > Enabled

Yes, the screen still locks, it just doesn't eat your keystrokes anymore. If you have Windows Home edition without gpedit.msc there's a registry key you can set instead, you'll have to google it.

I’m a Windows fan, but this bugs me so much, prior to Windows 8 I could go Ctrl Alt Del and instantly start typing my password, now I have to wait a couple of seconds before entering being able to enter my PIN.

To be fair, SDDM has the same issue, at least on multi monitor setups.

I agree with the main gist of the article and a lot of the points, but disagree on a few specifics.

1) mlock isn’t meant to be called on all your memory, just something that needs it to operate correctly. The situation the author described where the system comes to a halt as memory contents are paged in and out of memory/disk is (subjectively) worse when the only option is for the OOM killer to begin reaping processes everywhere (which would happen if all apps took this advice).

2) the following excerpt is not how any sane modern OS scheduler works:

> Imagine you have multiple background process running at 100% CPU, then a UI event comes in to the active UI application. The operating system may block for 100ms * N before allowing the UI application to process the event, where N is the number of competing background processes, potentially causing a delayed response to the user that violates the real-time constraint

Modern schedulers calculate priority based off whether the process/thread yielded its remaining execution time in the previous round or was forcibly evicted. Background processes are additionally run at a penalty. The foreground window (on OSes with internal knowledge of such a thing) or terminal owner in the current login session group gets a priority boost. Threads blocked waiting for input events get a massive priority boost.

(But the point stands and it might be a whole lot longer than n * 100ms if drivers or kernel modules are doing stuff.)

> The situation the author described where the system comes to a halt as memory contents are paged in and out of memory/disk is (subjectively) worse when the only option is for the OOM killer to begin reaping processes everywhere.

This one's interesting since your outcome often depends on what hardware you have. On systems with slow IO, i.e. a slow HDD, it's possible for swapping to make a system entirely unusable for minutes, whereas if swap is disabled the OOM killer is able to kick in and solve the issue in less than a minute. That's the difference between being able to keep most of your work open and none of your work open (because the alternative is being forced to reboot).

But in the era when spinning rust startup disks were in use everywhere, no app would autosave. I can’t imagine the carnage if MS Word or Excel were just violently killed at the first sign of memory pressure back in the day.

The “funny” thing is that I can still regularly hit the “sweet” spot where Linux completely freezes with 8GB of rams (with zram) on a fast SSD.. I think the paging logic still has some assumptions that are only true for hard drives, and that put the system into a frenzy - only REISUB works at that point.

It is well known that general purpose operating systems do not allow real time applications. The reason is people prefer an OS that can do a lot of things well enough, vs a system that can do one thing in real time. Real time OS's exist, and you would never want to build a desktop environment on them.

Also, the OS will never become the bottleneck for any general purpose software because you'll never have a team of programmers good enough to make it so. All the performance issues be due to mistakes in the application itself.

You do not want real time systems design for a GUI, its a lot more work and it wastes the vast majority of processing power to guarantee performance. You just need to be a careful about the amount of work you do in event handlers and split off heavy or IO work and split big updates. The problem is all concurrency is a lot harder than serial and so it tends to be something that is delayed until its really necessary so a lot of "not really good enough" handlers end up causing minor annoying delays.

You don't want to do the design work, but you do want the results.

Maybe some sort of magical framework will figure out how to get the latter without the work in the future.

As a human, I would like to be able to treat user interfaces as predictable, physical objects.

Unfortunately programs are built out of math and so I can’t usually do this.

Apple iOS actually does do a pretty good job of it, and that is a huge differentiator that I think is a large part of why iPhones feel great to use.

Thank the dispatch queue and Xcode will yell at you if you do anything other than UI and very trivial computes on the main UI thread

Some of Apple's biggest perf wins in the last several years involved removing dispatch asynchrony from code paths that really don't need to be asynchronous.

Obviously if you're writing an event-loop web server, then your file IO needs to be non-blocking or you're hosed. On the other hand, if you're reading a small configuration file from disk after your app launches, the most responsive option may well be to just read the bytes on the current thread and continue.

https://web.archive.org/web/20190606075031/https://twitter.c...

https://web.archive.org/web/20211005132519/https://twitter.c...

> when I realized that most mainstream desktop UI applications were fundamentally broken.

Oh boy

> File system IO functions belong to a class of functions called blocking functions.

Hasn’t non-blocking IO been a major feature for about a decade now??

Someone has not told Microsoft | It is not as simple as that.

On some Windows machines with network-mounted drives, the File-Print-to-pdf dialog takes *minutes* to become responsive, even when all currently open files are on a local drive.

This is the kind of thing the author is talking about. The programmers of that dialog box probably just called a generic "open file dialog" library function, without researching its worst-case performance.

In turn the library writers probably blithely coded something like "check if all mounted drives are accessible", without stopping to consider the worst-case performance.

The file picker on Windows should be thought of as more like a process than a function call. It has to 1) read the metadata of the current directory’s files 2) look up which icon to use for each 3) load the icon 3) call apps that may modify the file icon overlay (like “synced” or “not synced” cloud icons for Cloud file systems like Google Drive) 4) read the registry for various settings 5) load which pinned files a user set up - etc etc. All this involves dozens of disk reads, calling into third party code, etc. A lot of this may be cached but who knows.

The alternative to this is to roll a barebones file picker - there might even be one available in the Windows API.

You're right, Sysinternals Process Monitor will show lots of stuff going on for your example.

> Someone has not told Microsoft

More like the part of Microsoft that implemented non-blocking I/O for Windows some time ago never bothered to tell the part of Microsoft that writes the generic Windows UI code for things like the open file dialog. Or for Microsoft Office applications, for that matter; I still see Word and Excel block the UI thread when opening a file from a network drive, even though Windows has perfectly good asynchronous file I/O API calls.

> never bothered to tell the part of Microsoft that writes the generic Windows UI code

no, they just don't care.

More like three decades. But you'll need to send the memo to small time players like Microsoft, who continue doing all kinds of stupid blocking IO on every possible user interaction because who even fucking knows.

> Hasn’t non-blocking IO been a major feature for about a decade now??

This doesn't make people use it, and even the ones that try to use it might erroneously expect that open(2) will return in less than a second.

Im happy even with JetBrains products. They display "working" popup on blocking operations, and do a lot of stuff in the background, which is not blocking my work.

I've finally learned to love VS Code, but the new UI of Clion is pretty damn decent

I believe it's pretty easy to resolve -- declare one thread as "UI Thread", and forbid doing any IO on it.

Android (and I believe iOS does too) enforce that.

It's up to the developer to show a meaningful message/animation if an IO operation takes noticeable time.

> Android (and I believe iOS does too) enforce that.

This is absolutely not true. Android simply detects that there has been no progress on the UI thread "for a few seconds" before force-closing the app [1]. By this time, the interaction has been janky/frozen for WAY too long. If you have seen bad iOS scrolling and lock-ups, you know this as well.

I have worked on mobile software for these apps that have billions of users. When I pointed out how much stuff ran on the UI thread, there was a collective "this is just the way it is" response and life went on.

It's super-depressing.

-----

[1] "Performing long operations in the UI thread, such as network access or database queries, blocks the whole UI. When the thread is blocked, no events can be dispatched, including drawing events.

From the user's perspective, the application appears to hang. Even worse, if the UI thread is blocked for more than a few seconds, the user is presented with the "application not responding" (ANR) dialog."

https://developer.android.com/guide/components/processes-and...

Attempting a network operation on the Android UI thread will throw the android.os.NetworkOnMainThreadException exception, and has done since API 11 (Honeycomb) in 2011.

https://developer.android.com/reference/android/os/NetworkOn...

Interesting, probably my IDE (Android Studio) only warned me about that, but I took the warning for an error and never even tried to do the wrong way.

Correct. Looking at the Windows side of things, most of the UWP APIs are designed to be async, meaning they will never block the UI thread.

The author is giving the OS a unfair rep here. All three major OS has solutions to this problem. It's more up to application makers to use the tools correctly.

Visual Studio Code has a huge realtime problem. I find myself typing ahead by ten or 15 characters most of the time before changes show up in the UI. At which point either my brain-typeahead buffer overflows, or keystrokes start getting dropped (I'm not sure which).

Not sure if this is a problem specifically on Linux or not. I like to think that on Windows, UI-thread/process priority boosts take care of this problem, but I suspect not entirely.

I've been meaning to turn off Intellisense to see if my productivity will increase. What I know for sure: my productivity in VSCode/C++ is hugely tanked compared to Android Studio/java.

That's extremely slow it's in millisecs for me, how can you even work with that latency? I think it could be a graphics driver issue, I notice VSCode lagging sometimes when I have some WebGL app running next to it.

Or switch to something not built on top of Electron, like Sublime.

We have UI models that allow us to through long-running operations onto a background thread so that we can render perfectly animated hypno-toad cursors Or, more currently, perfectly animated cute-wait animations to keep people entertained while long-running operations complete. (Only a matter of time before wait-animations incorporate mini-games to play while while you're waiting, I think). But...

I'm not sure we have UI models for dealing with more complicated asynchronous operations on a document model though.

Old versions of Microsoft word used to run pagination and line-breaking on a background thread. The UI would keep a couple of lines around the edit cursor up-to-date, with precisely page layout for anything more than a couple of lines after the edit cursor running asynchronously. (It probably still does, but we probably notice it less these days).

And various elaborate schemes for keeping local and cloud object models synchronized asynchronously.

Oh. And I guess we have compilers and editors that collaborate to take snapshots of code when a compile starts, while allowing editing to continue, and to run Intellisense analysis in the background in the face of continually updating edit buffers. Incredibly complicated stuff, done mostly by brute force of intellect, I think.

Beyond that, I don't think we have a general theory of what to do in user interfaces when there isn't enough CPU to go around.

Windows Phone 7 fixed this in a way - it had a UI thread that could not be used for anything but UI updates. Your event handlers would be executed on a different thread. As long as you showed the loader, the rest of the app would work perfectly smooth.

It's been few years since then so I am light on details.

---

Also, as much as devs hate javascript, it mostly solves this by having 1 thread and offloading all system calls to another via callbacks? So the solution to your issue is: Use electron. (Just kidding, but there is some truth in it)

Web browsers should add a DOM method called `addEventStream()` to supplement `addEventListener()`. It would still accept an event type—e.g. `button.addEventStream('click')`—but would not accept a handler function. It would just return an async iterator of event objects. Backpressure on the iterator would place the element into a `:waiting` state similar to `:disabled` in which the element would become non-interactive. All UI event handling becomes a data processing pipeline.

> This is a transparent process that is not under control of the application. Thus, if any given memory access can block on IO from a disk drive, that means the system is fundamentally not real-time, therefore UI applications on such a system are fundamentally broken.

Not really. It means they chose to degrade performance when too much memory is used, rather than crash.

All options available when "out of memory" are bad. They thought that was the least bad option.

> So correct UI applications cannot call any blocking function from their main threads.

Author is redefining "correct" to mean "has the property I care about, which in this case is performance".

There are many desirable distinct properties of a computer system: correctness, performance, security, etc.

"Correct" usually means something like "gives the right answer". It has nothing to do with performance.

For a game like Street Fighter, you can present to someone the time at which the first frame begins and all of the input events within the game and they can calculate the unique game outcome that results. It's a bit embarrassing that we cannot have the same property in applications that we interact with using our keyboards all day, and we must constantly look at them to see if they are behaving as if we have sent the inputs we have sent.

In my experience, "Correct" means "satisfies a specification". The specification is often about the final result, but it's not limited to it.

As an example, I would call a function called "mergeSort" that takes O(n^2) time incorrect.

You can have it fast, or correct, or both, or neither.

Here's an extremely fast sorting algorithm that's incorrect:

  function cobbalSort(arr) { return arr; }

I believe I agree.

The iPhone UI must have either a realtime UI (or something pretty snappy). It responds to finger inputs in a bounded way.

Dispatch queue. Android has the equivalent thing.

More than that even. Since iOS 1, the majority of compute-intense work after a touch event has happened in an entirely different process, which has the highest OS-scheduler priority. This is what CoreAnimation is based on. Unlike a game engine or other "realtime" app, the OS itself is doing everything possible including preempting your main-thread to ensure that anything animating takes priority.

> Dispatch queue. Android has the equivalent thing.

Must be disabled on Samsung flagships, as these are the ones I have experience with, and I wouldn't say touch response is bounded on them, not for any useful definition of "response" and "bounded".

The last mobile device I used that had sane input response was a Sony Ericsson K800i feature phone. Input processing on it was fast, and most importantly, consistent. That means predictable - I could do most stuff on that phone while keeping it in my pocket, because I muscle-memorized the input sequences and how long every action took. Such a thing is impossible on smartphones.

> This doesn’t seem like a common problem but whole system “out of memory” conditions are not that uncommon. When the system is in this state, it starts rapidly paging memory onto the hard disk. UI applications will be affected and this will cause your system to hang without warning and with no way to intervene since keypresses cannot be processed. From a user standpoint, this is worse than a kernel panic. This type of failure has happened to me multiple times on Linux so I know it’s a problem there. Perhaps Windows and macOS engineers have already considered this issue but I doubt it.

In my experience this is only a problem on Linux. Windows and mac will simply inform you that the system is running out of memory and then you can kill things to set it right. No hung system.

If this article is true, I wonder if newer frameworks like Android's Compose are fundamentally flawed, at least when used in languages with garbage collection.

Essentially, composable can recompose with every frame, like for an animation. But, in certain circumstances, this will cause allocations for every frame.

For example, a modifier that is scoped to something like BoxScope. You can't hoist it out of the composable, it has to be in BoxScope. If that scope is animated on every frame, that modifier get re-allocated, every frame. That could be a lot of pressure on the GC.

Edit: Then again, its hard doing anything realtime in GC languages like Java / Kotlin, maybe its possible if doing 0 allocations per event.

Yes, these frameworks are fundamentally broken. Even a framework with extensive usage like React doesn’t work for real-time applications. Or at least your only option is to manipulate the DOM directly for the performant real time parts.

So is the moral of the story is to build an entire OS like erlang's actor model? Doesn't that work by using lightweight threads and messaging passing between them. The supervisor gives each thread a certain amount of time to run and then puts the current thread on the back burner while the next thread runs for it's allotted time. I remember hearing it in an erlang talk by Joe Armstrong on Youtube. I can't remember which one though.

Sounds like you'd end up with a microservices architecture. Not a fan.

Yes.

UI should be run in a separate processor in real time.

I am tired of clicking to see the screen change after I click and the click registers on the new screen, not what I clicked on.

Total overkill. If the screen loading is slower than a frame because of...

- I/O like file access or networks, you can punt to an async framework like Rust's Tokio

- CPU number crunching like cracking a password, you can punt to a worker thread (like Tokio's `spawn_blocking`)

The kernel scheduler is smart enough to give time slices to the UI thread even if the password-cracking thread is trying to eat as much CPU as possible, or, God forbid, the network thread is _sleeping_ until a packet arrives. (Networking is a waste of a thread. Most requests and responses are small enough that the CPU could process them on the UI thread, it's just that sleeping the whole thread until the message shows up is goofy)

It's not a lack of processors, it's the fact that good multi-threading and UI totally changes the architecture of a program, and most of us aren't trained for it or incentivized to implement it.

I disagree that a dedicated I/O processor is overkill. We spend a lot of transistors and code to solve problems that occur very rarely. Obscure race conditions come to mind. Putting the UI in real time would save me hassle every day, so it's worth the effort.

Adding USB ports to a video card would achieve 99% of the hardware requirements. The other 1% is made up of things I have overlooked.

The software would be more complicated, but it would not require rewriting applications. It would mostly be recoding OSes.

Also, the security benefits of linking output the screen to input from the keyboard and mouse in real time would be substantial.

Yet another reason to ditch GC based languages by default.

I felt like by reading the headline alone, I know exactly what this is talking about

Edit: I was completely wrong, but the article is very interesting.

Is this article another way of saying that the user doesn't like to wait more than 100ms when they perform an action?