I am always glad to see people pursuing creating their own programming language. More people should give it a try, IMHO.
A tricky thing that comes up with Rust comparisons is that often, Rust has a feature that's weird or hard to use, but it's because that's the only solution that makes sense within the constraints Rust has placed upon itself. Clicking some links gets me to https://git.yzena.com/Yzena/Yc/src/branch/master/docs/yao/de...
> Yao's biggest goals are: Correctness, Convenience, Performance, in that order.
Having clear goals is a great thing when making a language, because they give you guidance on what is appropriate to include and what is not.
Rust has certainly demonstrated having similar goals, but in a slightly different order: Correctness, Performance, and then Convenience. So it wouldn't shock me if Yao could do some things better than Rust, in accordance with its goals. But that also means that sometimes, Rust would be useful where Yao cannot be. Everything is tradeoffs.
Incidentally, I actually think that figuring out what your values are, and what your needs are, is a great way to pick a programming language. Decide what matters to you as an engineer, and then find a language that shares similar values. I gave a conference talk a few years back on this idea, and how I viewed Rust's values at the time https://www.infoq.com/presentations/rust-tradeoffs/
> I am always glad to see people pursuing creating their own programming language. More people should give it a try, IMHO.
I must agree that it's certainly lots of fun and a learning experience, but I must also always think of this amusing quote I once read somewhere: "What is a gentleman? A computer scientist who doesn't invent his own programming language."
This is what I want more people participating in "language wars" to understand. There is not much sense to compare projects that pursue completely different goals.
And having a background in programming language design surely makes understanding of solutions and compromises a lot easier.
I have said this recently a few times: I have found that most of the time, various people involved in working on languages themselves have far kinder attitudes towards other languages than their respective communities tend to. I've always chalked it up to some sort of collegial professionalism, but maybe understanding the tradeoffs more deeply is an aspect of it too.
> A tricky thing that comes up with Rust comparisons is that often, Rust has a feature that's weird or hard to use, but it's because that's the only solution that makes sense within the constraints Rust has placed upon itself.
Hah I find the same thing about Haskell. People are a bit less charitable about Haskell's quirks though I find..
The place I work definitely has a culture of whining :P
- Is there really a major use case for function traits? Is more elaboration of the type system actually useful?
- The original "Rewriting Rust" article discussed replacing Rust's macro system. That's not mentioned here. If you had to write your macros in Rust, to be run in compile time, would that be a win? I doubt it. Each macro would start with a call to a parser, and then you generate code as strings. This would pay off only for complex cases such as a regular expression compiler. Rust already has compile-time build programs, where you can run Rust during the build process. (I use that to take in an API definition from another source and generate Rust.)
- "Runtime capabilities" seems to be an attempt to build a container system into the language. It's not clear that helps much. That's really the OS's job. Protecting programs against themselves is nice, but not usually the problem. I'd like to see something in Android that gives an app file access with less access than "all files". Like "Can read what was installed for this app, can write working subdirectory for this app, and nothing else."
Fair criticisms, John. I just want to address macros.
Yao has something like Rust's macros. They are called "keywords," because even the built-in keywords are implemented that way. (That is why Yao can just not have a `while` keyword.)
My Yao-based build system, Rig, uses keywords to implement a build DSL. [1]
> Is there really a major use case for function traits?
Absolutely. They can be used for function overloading, and for manually creating closures from structs, which can be more ergonomic sometimes. Whether you want those features is another matter. The rust compiler supports them but doesn't expose them to stable rust.
> 'd like to see something in Android that gives an app file access with less access than "all files".
SELinux has been able to solve that problem in Android for a long, long time now. Which is more appropriate, as you say it should be the job of the OS.
The problem is the age old one where users want convenience at the cost of security.
Interesting response. I'm as curious as anyone else as to how Yao will improve upon Rust, but here is what I've observed from following several (relatively) recently created languages (Julia, Nim, Mojo, JAX):
* Language creators start with high hopes and big promises. "We've learned from the past! This will solve all the problems of previous X,Y,Z languages! It will be so much easier to use!"
* The language begins to be designed, implemented, and developed. Problems arise. Tradeoffs must be made. Invariably, the new language develops its own set of limitations, shortcomings, and weaknesses. These are inevitable with any new programming language.
* Ultimately the language either finds a userbase who are willing to overlook or work around the limitations and pitfalls, or it fades away.
* If it remains, eventually another new language arises saying "We've learned from the past! This will solve all the problems of previous X,Y,Z languages! It will be so much easier to use!" And the cycle repeats.
Now, is this a bad thing? I would argue no, not necessarily. Every time a new language is created, our vision of what is possible is expanded. We learn new lessons about what works and what doesn't work in programming languages.
So in short I'm looking forward to see how the state of the art is advanced by Yao. If it gets to a decent working state, I might even try it out if I have time. However, in light of the above statements forgive me for being skeptical of the following claim from Yao's website [1]:
> As powerful as C, As flexible as Lisp, As easy as Python, As provable as Ada/SPARK, More reliable than Rust, With the time semantics of HAL/S.
Fwiw, Zig is one of the few languages on my radar that's on the uptick. At least in my circle, I know people from firmware to VM development that moved from Rust over to Zig for its flexibility. There' also Bun that managed to nearly be a drop-in replacement for Node in a very short development cycle, which they claim is due to Zig. (I am not associated with Zig nor have a lot of experience with it yet)
I've heard great things about Zig as well. The one difference between it and Rust is the amount of industry and financial investment to date. Rust has had literally millions of dollars of buy-in from some of the biggest tech corporations on Earth, which is why I'm more confident that it will likely stick around.
This is one area where, for example, Julia has struggled. In many ways it's a fantastic language for scientific research, but it simply has never received nearly as much funding or investment as, e.g., Python.
I also appreciate the way that Zig focuses on tooling, particularly around cross-compilation. That's something that is usually a "nice to have" rather than "must have" particularly in young projects, and for users that need it, it's a huge win.
On the gripping hand, Rust is the first language to popularize (not invent, hold your tomatoes) many terrific programming ideas under one wrapper. It has demonstrated how much the compiler can do to prove code is correct. Yet…it has some warts. I am quite interested in what lessons can be learned to smooth out some of the Rust edges for a language which does not have “replace C++” as the guiding principle.
> The Rust async book mentions three more methods: […]
Hah, I wrote this section and was deeply aware of these problems at the time. Let’s just say it didn’t fit in the introduction. In fact, a lot of the understanding of this issue has been expressed later, blog post by blog post, largely without coordination. It’s almost eerie to see again and again people encountering the same issue, and spending huge effort to address it (this one takes the prize though - making your own language).
Let me try a to narrativize it: we have a fundamental problem with unrestricted concurrent control flow (very similar to how we used to have unrestricted sequential control flow when we had goto). This problem exists in most programming languages, but we mostly pretend it’s not there (and live with the subtle bugs instead). However, with Rust’s unique ownership- and safety model, the lack of structured concurrency was an unavoidable problem, at collision course with Rust itself. In the Rust we have today (without structured concurrency), a bunch of expected and reasonable behaviors cannot be achieved (such as borrowing from a parent task).
Do you know of any longer form content that walks through this area, or maybe just a listing of blogs. I have a feeling the resources are relatively scattered over various sites and across time, so any tips on search terms to use if a deep dive into this problem (both in Rust and generically in programming language semantics) intrigues me?
Sibling already gave great pointers to structured concurrency in general.
For rust specifically the links in the post were good, some I hadn’t seen that came up in the last 2 years.
Perhaps someone from deep PL background has more rigorous papers and such to point to? I highly doubt this is being discovered now for the first time – it wouldn’t surprise me if there’s a whole academic sub-branch about similar things. Given how important concurrency is, and how difficult it is to get right, there’s a lot of opportunity for academia to tackle the problem space. We sure as hell need it.
Reference counting is ultimately a GC, although not a tracing GC which is the most common kind. I also don't really see the appeal of not having a GC in a language like that. If it doesn't compete with C/C++ for performance and low level support then not having a GC is no longer an advantage.
That said Koka still remains very cool for the effect system though, and I would really like to see it in a mainstream language!
> This is actually the root of the function color problem: async is exactly backwards! Instead of marking asynchronous functions, languages should have had us mark synchronous functions, and then async should have been the default!
What happens when you need to add a new trait that all existing functions must be changed to have? That's why async is generally an attribute.
The answer is that we'd have to say that all functions have some default trait set that can be extended automatically as the language evolves, except those anti-traits they have. Then we could say that a function is async meaning "not sync", or even we could write !sync instead of async.
I rather like the idea that you have to write `!sync` instead of `async`.
Every unmarked function can do anything. If you add a new thing that functions can do, you assume unmarked functions can do it (even if they don't), and you add a trait for the negative.
Yeah, you shouldn't believe it yet, but here is the status of each:
* "As powerful as C": unproven. I might need something like Rust's `unsafe`, and even though I have something in mind for that, it requires that any program be expressible in Dynamic Restricted Structured Concurrency, an open problem. I am working on a proof, though.
* "As flexible as Lisp": I think this is proven. Yao has equivalents for both macros (keywords) and reader macros (lexing modes). My build system uses its own keywords to implement a build DSL [1]. The shell sublanguage uses a lexing mode, and another makes it possible to embed JSON.
* "As easy as Python": in retrospect, this isn't quite possible, but I hope to get 90% of the way.
* "As provable as Ada/SPARK": I'll let you read the design in [2] and decide for yourself. But Yao will also have contracts.
* "More reliable than Rust": unproven, but I think the lack of async will go a long way.
* "With the time semantics of HAL/S": unproven. HAL/S was what the Shuttle software was written in [3]. It was hard real-time. Because Yao will be distributed in IR form [4], the back end could leverage knowledge of worst-case instruction latencies to calculate worst-case response times. This same thing will also allow using only constant-time instructions for cryptography, using a `constant_time` function trait.
Worst case execution times are generally unknowable, even at codegen. It depends on things like the specific microarchitectural details, what other instructions are currently executing, the memory hierarchy, what instructions have been executed before, how the power supply is responding, what the thermal environment is like, etc.
HAL/S dealt with these problems by a combination of programmer/validation discipline, custom hardware designed for hard real-time usecases, and generous support from the underlying RTOS (FCOS). Cryptography code doesn't deal with it, it just avoids data dependent nondeterminism, which is good enough for those use cases.
My idea is to cause a compile error if the backend can't use all instructions for which it knows the worst case latency or if it can't use purely constant-time instructions.
That's essentially every instruction though. You can't write a program that doesn't access memory for example, and every memory access is nondeterministic in wall clock time.
In the crypto example, the backend would have to be sophisticated enough to know when the same addresses would be accessed every time, like bitslicing in AES. [1]
The devil absolutely is in the details. This is why I say Yao's capabilities on this front are unproven.
> * "As provable as Ada/SPARK": I'll let you read the design in [2] and decide for yourself. But Yao will also have contracts.
Without being too self-indulgent, I'm not sure there is that big of a gap between the two in provability, there are now a huge array of verifiers for Rust code which are being used to verify real code: SAT/SMT solvers, kernels, memory allocators etc...
Hats off, you really did in fact a deep analysis of Rust conceptual errors by searching alternative ways to fix Rust async. Also by understanding and describing the decisions you made and what are their architectural implications (I liked the good, the bad and the ugly part).
While I can't appreciate the entirety of this language because I can't grasp all the technical subtilities, I really like the core ideas at the basis of Yao and the spirit you put into it. I hope I can play with this language some day. And wish you good luck with this nice project.
> "As flexible as Lisp": I think this is proven. Yao has equivalents for both macros (keywords) and reader macros (lexing modes).
First-class symbols (incl. uninterned symbols)? Classes and functions can be redefined at runtime? User-available parsing (read) and runtime compilation (compile)? Restarts? CLOS and MOP? Communication with the compiler (declarations and ClTl2 *-information)?
I don't mean to sound too abrasive, but reducing Lisp flexibility to macros is a bit much. In any case, good luck with your project!
> Classes and functions can be redefined at runtime?
No.
> User-available parsing (read) and runtime compilation (compile)?
Yes, though runtime compilation is not implemented yet.
> Restarts?
Yes, though they are only implemented in the runtime; the keywords don't exist yet.
> CLOS and MOP?
No, the object system will be more recognizable to non-Lispers.
> Communication with the compiler (declarations and ClTl2 *-information)?
I wasn't able to find info on these. I will keep looking, and I would appreciate pointers, but the answer is probably no.
Edit: after skimming [1], the answer on declarations is probably yes, for the most part. Still searching for *-information stuff.
Edit 2: After skimming [2], Yao originally didn't have any concept of stuff like the *-information functions, but those are good ideas, and I will add them.
Huh, thanks for replying seriously to my snark, that's more than what I expected!
About CLOS, I suggest at least making method combination more flexible than C++'s override. A way to run both the parent and child methods is often needed (maybe in a specific order, maybe with a way to collect all the return values).
> Huh, thanks for replying seriously to my snark, that's more than what I expected!
Quite frankly, the snark was warranted. I'm not a Lisper, but I do know more, and I should have been more precise.
> About CLOS, I suggest at least making method combination more flexible than C++'s override. At least a way to run both the parent and child methods is often needed (maybe in a specific order, maybe with a way to collect all the return values).
I agree with this. However, Yao will not have inheritance; instead, it will be more like Go: composition over inheritance. And interfaces. So I could be wrong, but without inheritance, I'm not sure the entire complexity of the CLOS is needed. Do correct me, though.
> Declarations are basically C/C++'s pragmas and __attribute__, but standardized and using a seamless syntax.
That's such a nicer description than in CLtL2. And yes, Yao will have them.
If those are just for giving info to macros, Yao already has that; the lexer will tell keywords if the given token is a package, a keyword, an operator, a function, a type, etc., along with other info.
> However, Yao will not have inheritance; instead, it will be more like Go: composition over inheritance. And interfaces. So I could be wrong, but without inheritance, I'm not sure the entire complexity of the CLOS is needed. Do correct me, though.
I'm sadly not very well versed with this alternative to OOP. But something tells me that a more "basic" and less dynamic scheme like that is probably best for performance reasons, since you'd either need a Julia-like JIT or a granular "dynamism knob" like https://github.com/marcoheisig/fast-generic-functions to truly amortize the cost of CLOS-like dispatch and polymorphism; though magic can be done, as the "Fast generic dispatch for Common Lisp" paper shows.
> if those are just for giving info to macros, Yao already has that
Huh! If this includes types deduced by compiler inference and not just user declarations (since Yao seems gradually typed, to my eyes), that'd be pretty impressive.
Hope you'll be able to keep your spirits high, this is a lifetime's work you seem to be looking forward to ;)
> I'm sadly not very well versed with this alternative to OOP. But something tells me that a more "basic" and less dynamic scheme like that is probably best for performance reasons, since you'd either need a Julia-like JIT or a granular "dynamism knob" like https://github.com/marcoheisig/fast-generic-functions to truly amortize the cost of CLOS-like dispatch and polymorphism; though magic can be done, as the "Fast generic dispatch for Common Lisp" paper shows.
You know, for someone who isn't "well versed" with it, you certainly nailed exactly why I chose it. Also, simplicity. The "as easy as Python" is only possible if I remove complecting [1] things like inheritance.
> Huh! If this includes types deduced by compiler inference and not just user declarations (since Yao seems gradually typed, to my eyes), that'd be pretty impressive.
It does, actually. But Yao is not gradually typed, unfortunately. Too complecting.
Given the development model of Lisp is modifying a live image of a running program, I have to assume you limited this statement to the availability of macros.
Serious mode: it's good that people are exploring the programming language space, both tiny and big changes can mean a world of difference in reducing bugs and making programs nicer both for the user to run and the developer to create and maintain.
Serious mode off. When will we get Yao in the Linux kernel?
Odd post, and it landed me on the Yzena website which is very surreal. It seems to be intentionally hard from the website to figure out what Yzena is or why anyone would want to buy a license for it (or the things it makes?). There was a link to a repo full of perl scripts, though.
According to the website I'm not allowed to ask about this though ;)
Is there something like this but without concurrency? I don't need concurrency or multithreading, and most of my pain with Rust has come from the fact that it prevents me from doing things that I know are correct because it thinks my code will run in a multithreaded environment (it will not).
It seems like Yao's whole project is solving the concurrency problem.
I'm like, why not drop it entirely (if performance isn't one of the language's top priorities anyway)? What does keeping it gain that's so important? (Context: I don't understand concurrency and multithreading.)
Yao's concurrency will be explicit. You will not get concurrency unless you open a threadset and spawn threads. Yao scripts run in a single thread by default.
I focused on solving the concurrency problem because if it is solved, single thread code is solved as well.
Famous last words. Ever needed a timeout? Cancelation? Respond to Ctrl+C (SIGINT)? Your UI has an event handler? Client server architectures? All of those things need concurrency, somewhere.
That said, you could argue that I’m being nitpicky. A combination of the OS and language runtime (eg JavaScript) can keep you mostly isolated from the complexity explosion of parallelism and concurrency. But it’s still there.
> it prevents me from doing things that I know are correct because it thinks my code will run in a multithreaded environment
Could you elaborate on that? In your case you don't need to worry about Send/Sync, you can use Cell/RefCell, etc. The borrow checker isn't about threading, it's useful in a purely single-threaded context too.
> But there is an even harder problem: the C10M Problem. Is it possible to use RSC to handle 10 million connections, of which 1 million need some processing every second?
(RSC being restricted structured concurrency.)
IMO the C10M problem requires: a) user-land network stacks or unikernel servers, b) NPROC (or fewer) threads or processes, c) evented, continuation-passing style code in each of those threads or processes.
I've written one such program (though without (a)), a HTTP read-only file server that does (b) and (c) and which supports "tailing" files by doing `GET`s with `Range: bytes=0-` (or some other starting offset, but no ending offset), and it scales to tens of thousands of concurrent clients trivially. To get to millions of clients I would need to use H/3 and QUIC, and also do (a). (That server also supports ETags and conditional requests. Tailing `GET`s end when the file is unlinked or renamed away. Essentially it is a poor person's Kafka.) This program stores state in a per-client data structure and in the form of a pointer to a continuation closure that is `{data structure, handler closure}` that is registered with the event system, and each handler does only non-blocking operations (if you can think of filesystem operations as non-blocking anyways, though one could also use aio).
I bet this works well with RSC if all the state is kept in statically allocated objects. In my program all state is kept in dynamically allocated heap objects, but I could have statically allocated them instead, perhaps at program startup, or else at compile time.
The point here is to absolutely minimize the representation of state. In a thread-per-client implementation the state is smeared on each thread's stack and also any necessary heap objects. In a CPS implementation the state is fully explicit, with none of it in any stack frames -- the programmer is in charge of producing the most compact state representation, and it's easy because the state has to be fully explicit.
I view evented CPS and thread-per-client as the ends of a concurrency spectrum where green threads and co-routines are somewhere in the middle. If you want C10M you will probably end up wanting evented CPS. Green threads / fibers can get close to CPS if you work to minimize state anyways.
>Instead, Yao will be compiled to an LLVM-like IR (which already exists), and that’s how it will be distributed. Users will have to do the final compile step on their local machines.
Is this referring to the compiler itself, or programs written in Yao?
If the goal is portability, Google pursued the idea of distributing cross-platform LLVM IR via PNaCl and ultimately abandoned it, because LLVM IR isn't really designed for portability.
Without sounding too much like a Cathy Woods, I'm wondering if AI can be leveraged to help with all of the non-core aspects of creating new languages? For example, really why does every language really need a json parser, or http requests library. If one were to examine the machine code instructions involved in each, they would likely be rather similar after all. Writing all of that stuff is an enormous lift for the community. Tooling is similarly "undifferentiated heavy lifting" in a way. Also, moving existing code bases to new languages is still hard.
If there were better ways to achieve these things, there would be much less inertia in the language space.
> Instead of marking asynchronous functions, languages should have had us mark synchronous functions […] Although that wouldn’t remove the possibility of bugs when calling a blocking function in async code,
So mark "synchronous" (i.e., non-yielding) and "asynchronous" (i.e., non-blocking). Then we can have the synchronous colour, the asynchronous colour, functions that are both (e.g. pure CPU-bound tasks, like arithmetic), and functions that are neither (e.g. buggy code).
Recent and related:
Rewriting Rust - https://news.ycombinator.com/item?id=41654871 - Sept 2024 (385 comments)
I am always glad to see people pursuing creating their own programming language. More people should give it a try, IMHO.
A tricky thing that comes up with Rust comparisons is that often, Rust has a feature that's weird or hard to use, but it's because that's the only solution that makes sense within the constraints Rust has placed upon itself. Clicking some links gets me to https://git.yzena.com/Yzena/Yc/src/branch/master/docs/yao/de...
> Yao's biggest goals are: Correctness, Convenience, Performance, in that order.
Having clear goals is a great thing when making a language, because they give you guidance on what is appropriate to include and what is not.
Rust has certainly demonstrated having similar goals, but in a slightly different order: Correctness, Performance, and then Convenience. So it wouldn't shock me if Yao could do some things better than Rust, in accordance with its goals. But that also means that sometimes, Rust would be useful where Yao cannot be. Everything is tradeoffs.
Incidentally, I actually think that figuring out what your values are, and what your needs are, is a great way to pick a programming language. Decide what matters to you as an engineer, and then find a language that shares similar values. I gave a conference talk a few years back on this idea, and how I viewed Rust's values at the time https://www.infoq.com/presentations/rust-tradeoffs/
This was based off of bcantrill's Platform as a Reflection of Values, which was very influential on me. https://www.youtube.com/watch?v=Xhx970_JKX4
If you've ever heard about Oxide's focus on values, this is some of the older background on that.
I wish I could upvote you twice.
I decided on my values because of bcantrill's talk. Everyone should watch it.
A more complete list of my values is at https://git.yzena.com/Yzena/Yc/src/branch/master/docs/adl/00... .
Could you elaborate on value #0 being users (and what it means to prioritize users over programmers)?
Good question. I will add that.
It means that users' convenience comes before my convenience. (I am the programmer in question.)
This means that if there is something I could do to make things easier on users, I do it, even if it is a lot more work for me.
An example is https://git.yzena.com/Yzena/Yc/commit/c9086af225d42b2f125959... . Comments in Yao do not use the `#` character, but I added the shebang to the parser specially so that scripts could be run just like any others scripts.
This is nice to see. I often came across opposite sentiment here. "Dev time is precious, user is OK with un-performant/etc. software."
Thank you for your kind words!
> I am always glad to see people pursuing creating their own programming language. More people should give it a try, IMHO.
I must agree that it's certainly lots of fun and a learning experience, but I must also always think of this amusing quote I once read somewhere: "What is a gentleman? A computer scientist who doesn't invent his own programming language."
I've never heard that, and that's very funny.
This is what I want more people participating in "language wars" to understand. There is not much sense to compare projects that pursue completely different goals.
And having a background in programming language design surely makes understanding of solutions and compromises a lot easier.
I have said this recently a few times: I have found that most of the time, various people involved in working on languages themselves have far kinder attitudes towards other languages than their respective communities tend to. I've always chalked it up to some sort of collegial professionalism, but maybe understanding the tradeoffs more deeply is an aspect of it too.
> A tricky thing that comes up with Rust comparisons is that often, Rust has a feature that's weird or hard to use, but it's because that's the only solution that makes sense within the constraints Rust has placed upon itself.
Hah I find the same thing about Haskell. People are a bit less charitable about Haskell's quirks though I find..
The place I work definitely has a culture of whining :P
Mostly an ad for Yao.
- Is there really a major use case for function traits? Is more elaboration of the type system actually useful?
- The original "Rewriting Rust" article discussed replacing Rust's macro system. That's not mentioned here. If you had to write your macros in Rust, to be run in compile time, would that be a win? I doubt it. Each macro would start with a call to a parser, and then you generate code as strings. This would pay off only for complex cases such as a regular expression compiler. Rust already has compile-time build programs, where you can run Rust during the build process. (I use that to take in an API definition from another source and generate Rust.)
- "Runtime capabilities" seems to be an attempt to build a container system into the language. It's not clear that helps much. That's really the OS's job. Protecting programs against themselves is nice, but not usually the problem. I'd like to see something in Android that gives an app file access with less access than "all files". Like "Can read what was installed for this app, can write working subdirectory for this app, and nothing else."
Fair criticisms, John. I just want to address macros.
Yao has something like Rust's macros. They are called "keywords," because even the built-in keywords are implemented that way. (That is why Yao can just not have a `while` keyword.)
My Yao-based build system, Rig, uses keywords to implement a build DSL. [1]
And keywords are fast to compile.
[1]: https://rigbuild.dev/build.rig5/#keywords
> Is there really a major use case for function traits?
Absolutely. They can be used for function overloading, and for manually creating closures from structs, which can be more ergonomic sometimes. Whether you want those features is another matter. The rust compiler supports them but doesn't expose them to stable rust.
> 'd like to see something in Android that gives an app file access with less access than "all files".
SELinux has been able to solve that problem in Android for a long, long time now. Which is more appropriate, as you say it should be the job of the OS.
The problem is the age old one where users want convenience at the cost of security.
Interesting response. I'm as curious as anyone else as to how Yao will improve upon Rust, but here is what I've observed from following several (relatively) recently created languages (Julia, Nim, Mojo, JAX):
* Language creators start with high hopes and big promises. "We've learned from the past! This will solve all the problems of previous X,Y,Z languages! It will be so much easier to use!"
* The language begins to be designed, implemented, and developed. Problems arise. Tradeoffs must be made. Invariably, the new language develops its own set of limitations, shortcomings, and weaknesses. These are inevitable with any new programming language.
* Ultimately the language either finds a userbase who are willing to overlook or work around the limitations and pitfalls, or it fades away.
* If it remains, eventually another new language arises saying "We've learned from the past! This will solve all the problems of previous X,Y,Z languages! It will be so much easier to use!" And the cycle repeats.
Now, is this a bad thing? I would argue no, not necessarily. Every time a new language is created, our vision of what is possible is expanded. We learn new lessons about what works and what doesn't work in programming languages.
So in short I'm looking forward to see how the state of the art is advanced by Yao. If it gets to a decent working state, I might even try it out if I have time. However, in light of the above statements forgive me for being skeptical of the following claim from Yao's website [1]:
> As powerful as C, As flexible as Lisp, As easy as Python, As provable as Ada/SPARK, More reliable than Rust, With the time semantics of HAL/S.
[1]: https://git.yzena.com/Yzena/Yc/src/branch/master/docs/yao/ma...
Fwiw, Zig is one of the few languages on my radar that's on the uptick. At least in my circle, I know people from firmware to VM development that moved from Rust over to Zig for its flexibility. There' also Bun that managed to nearly be a drop-in replacement for Node in a very short development cycle, which they claim is due to Zig. (I am not associated with Zig nor have a lot of experience with it yet)
I've heard great things about Zig as well. The one difference between it and Rust is the amount of industry and financial investment to date. Rust has had literally millions of dollars of buy-in from some of the biggest tech corporations on Earth, which is why I'm more confident that it will likely stick around.
This is one area where, for example, Julia has struggled. In many ways it's a fantastic language for scientific research, but it simply has never received nearly as much funding or investment as, e.g., Python.
At 8 years (1999) python did not have any big investment or even big tech backers. IIRC the first real python success story was dropbox
At 8 years there's a major JavaScript bundler and Uber using zig.
As someone who starting using Rust from before it reached 1.0, it's insanely funny to me to see comments like this.
People said the exact same kind of thing about Rust at the time!
I also appreciate the way that Zig focuses on tooling, particularly around cross-compilation. That's something that is usually a "nice to have" rather than "must have" particularly in young projects, and for users that need it, it's a huge win.
Also Zig wants to have better memory safety before v1 - there's an issue tracking that in the zig org but I couldn't find it
On the gripping hand, Rust is the first language to popularize (not invent, hold your tomatoes) many terrific programming ideas under one wrapper. It has demonstrated how much the compiler can do to prove code is correct. Yet…it has some warts. I am quite interested in what lessons can be learned to smooth out some of the Rust edges for a language which does not have “replace C++” as the guiding principle.
Author here.
You are absolutely right to be skeptical. I wrote that a long time ago.
I did detail the current status of those claims in a comment: https://news.ycombinator.com/item?id=41672316 .
> The Rust async book mentions three more methods: […]
Hah, I wrote this section and was deeply aware of these problems at the time. Let’s just say it didn’t fit in the introduction. In fact, a lot of the understanding of this issue has been expressed later, blog post by blog post, largely without coordination. It’s almost eerie to see again and again people encountering the same issue, and spending huge effort to address it (this one takes the prize though - making your own language).
Let me try a to narrativize it: we have a fundamental problem with unrestricted concurrent control flow (very similar to how we used to have unrestricted sequential control flow when we had goto). This problem exists in most programming languages, but we mostly pretend it’s not there (and live with the subtle bugs instead). However, with Rust’s unique ownership- and safety model, the lack of structured concurrency was an unavoidable problem, at collision course with Rust itself. In the Rust we have today (without structured concurrency), a bunch of expected and reasonable behaviors cannot be achieved (such as borrowing from a parent task).
Do you know of any longer form content that walks through this area, or maybe just a listing of blogs. I have a feeling the resources are relatively scattered over various sites and across time, so any tips on search terms to use if a deep dive into this problem (both in Rust and generically in programming language semantics) intrigues me?
https://vorpus.org/blog/notes-on-structured-concurrency-or-g...
https://dubroy.com/blog/handling-user-input-with-structured-...
The first one is particularly good. Both of those have links to more topics.
Sibling already gave great pointers to structured concurrency in general.
For rust specifically the links in the post were good, some I hadn’t seen that came up in the last 2 years.
Perhaps someone from deep PL background has more rigorous papers and such to point to? I highly doubt this is being discovered now for the first time – it wouldn’t surprise me if there’s a whole academic sub-branch about similar things. Given how important concurrency is, and how difficult it is to get right, there’s a lot of opportunity for academia to tackle the problem space. We sure as hell need it.
This discussion is seriously lacking in references to Koka language[0].
Koka is memory safe without using traditional GC, has effects, and is pretty cool over all.
[0] https://koka-lang.github.io/koka/doc/index.html
Reference counting is ultimately a GC, although not a tracing GC which is the most common kind. I also don't really see the appeal of not having a GC in a language like that. If it doesn't compete with C/C++ for performance and low level support then not having a GC is no longer an advantage.
That said Koka still remains very cool for the effect system though, and I would really like to see it in a mainstream language!
> This is actually the root of the function color problem: async is exactly backwards! Instead of marking asynchronous functions, languages should have had us mark synchronous functions, and then async should have been the default!
What happens when you need to add a new trait that all existing functions must be changed to have? That's why async is generally an attribute.
The answer is that we'd have to say that all functions have some default trait set that can be extended automatically as the language evolves, except those anti-traits they have. Then we could say that a function is async meaning "not sync", or even we could write !sync instead of async.
I rather like the idea that you have to write `!sync` instead of `async`.
The post answers that.
Every unmarked function can do anything. If you add a new thing that functions can do, you assume unmarked functions can do it (even if they don't), and you add a trait for the negative.
Thanks, I got interrupted in my reading.
Wow, about Yao:
"In fact, it might be better said that Yao is meant to be:
As powerful as C, As flexible as Lisp, As easy as Python, As provable as Ada/SPARK, More reliable than Rust, With the time semantics of HAL/S."
Even if I don't really believe in this incredible statement, I'm really curious to see what can be the result.
Author here.
Yeah, you shouldn't believe it yet, but here is the status of each:
* "As powerful as C": unproven. I might need something like Rust's `unsafe`, and even though I have something in mind for that, it requires that any program be expressible in Dynamic Restricted Structured Concurrency, an open problem. I am working on a proof, though.
* "As flexible as Lisp": I think this is proven. Yao has equivalents for both macros (keywords) and reader macros (lexing modes). My build system uses its own keywords to implement a build DSL [1]. The shell sublanguage uses a lexing mode, and another makes it possible to embed JSON.
* "As easy as Python": in retrospect, this isn't quite possible, but I hope to get 90% of the way.
* "As provable as Ada/SPARK": I'll let you read the design in [2] and decide for yourself. But Yao will also have contracts.
* "More reliable than Rust": unproven, but I think the lack of async will go a long way.
* "With the time semantics of HAL/S": unproven. HAL/S was what the Shuttle software was written in [3]. It was hard real-time. Because Yao will be distributed in IR form [4], the back end could leverage knowledge of worst-case instruction latencies to calculate worst-case response times. This same thing will also allow using only constant-time instructions for cryptography, using a `constant_time` function trait.
[1]: https://rigbuild.dev/build.rig5/#keywords
[2]: https://gavinhoward.com/2024/05/what-rust-got-wrong-on-forma...
[3]: https://www.fastcompany.com/28121/they-write-right-stuff
[4]: https://gavinhoward.com/2024/09/rewriting-rust-a-response/#d...
Worst case execution times are generally unknowable, even at codegen. It depends on things like the specific microarchitectural details, what other instructions are currently executing, the memory hierarchy, what instructions have been executed before, how the power supply is responding, what the thermal environment is like, etc.
HAL/S dealt with these problems by a combination of programmer/validation discipline, custom hardware designed for hard real-time usecases, and generous support from the underlying RTOS (FCOS). Cryptography code doesn't deal with it, it just avoids data dependent nondeterminism, which is good enough for those use cases.
Yes, you are correct.
My idea is to cause a compile error if the backend can't use all instructions for which it knows the worst case latency or if it can't use purely constant-time instructions.
That's essentially every instruction though. You can't write a program that doesn't access memory for example, and every memory access is nondeterministic in wall clock time.
Yes.
In the crypto example, the backend would have to be sophisticated enough to know when the same addresses would be accessed every time, like bitslicing in AES. [1]
The devil absolutely is in the details. This is why I say Yao's capabilities on this front are unproven.
[1]: https://www.bearssl.org/constanttime.html#constant-time-in-b...
> * "As provable as Ada/SPARK": I'll let you read the design in [2] and decide for yourself. But Yao will also have contracts.
Without being too self-indulgent, I'm not sure there is that big of a gap between the two in provability, there are now a huge array of verifiers for Rust code which are being used to verify real code: SAT/SMT solvers, kernels, memory allocators etc...
I agree. I think getting provability right would bring reliability along with it.
Thank you very much for your detailed answer. Will look into your links right now.
Hats off, you really did in fact a deep analysis of Rust conceptual errors by searching alternative ways to fix Rust async. Also by understanding and describing the decisions you made and what are their architectural implications (I liked the good, the bad and the ugly part).
While I can't appreciate the entirety of this language because I can't grasp all the technical subtilities, I really like the core ideas at the basis of Yao and the spirit you put into it. I hope I can play with this language some day. And wish you good luck with this nice project.
This is one of the nicest, yet honest, compliments I have ever received. Thank you!
> "As flexible as Lisp": I think this is proven. Yao has equivalents for both macros (keywords) and reader macros (lexing modes).
First-class symbols (incl. uninterned symbols)? Classes and functions can be redefined at runtime? User-available parsing (read) and runtime compilation (compile)? Restarts? CLOS and MOP? Communication with the compiler (declarations and ClTl2 *-information)?
I don't mean to sound too abrasive, but reducing Lisp flexibility to macros is a bit much. In any case, good luck with your project!
> First-class symbols (incl. uninterned symbols)?
Yes.
> Classes and functions can be redefined at runtime?
No.
> User-available parsing (read) and runtime compilation (compile)?
Yes, though runtime compilation is not implemented yet.
> Restarts?
Yes, though they are only implemented in the runtime; the keywords don't exist yet.
> CLOS and MOP?
No, the object system will be more recognizable to non-Lispers.
> Communication with the compiler (declarations and ClTl2 *-information)?
I wasn't able to find info on these. I will keep looking, and I would appreciate pointers, but the answer is probably no.
Edit: after skimming [1], the answer on declarations is probably yes, for the most part. Still searching for *-information stuff.
Edit 2: After skimming [2], Yao originally didn't have any concept of stuff like the *-information functions, but those are good ideas, and I will add them.
[1]: https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node105.html
[2]: https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html
Huh, thanks for replying seriously to my snark, that's more than what I expected!
About CLOS, I suggest at least making method combination more flexible than C++'s override. A way to run both the parent and child methods is often needed (maybe in a specific order, maybe with a way to collect all the return values).
Declarations are basically C/C++'s pragmas and __attribute__, but standardized and using a seamless syntax. The ClTl2 additions I'm talking about are ways for the compiler to give lexical information to macros: see https://github.com/Zulu-Inuoe/trivial-cltl2 and https://github.com/alex-gutev/cl-environments
> Huh, thanks for replying seriously to my snark, that's more than what I expected!
Quite frankly, the snark was warranted. I'm not a Lisper, but I do know more, and I should have been more precise.
> About CLOS, I suggest at least making method combination more flexible than C++'s override. At least a way to run both the parent and child methods is often needed (maybe in a specific order, maybe with a way to collect all the return values).
I agree with this. However, Yao will not have inheritance; instead, it will be more like Go: composition over inheritance. And interfaces. So I could be wrong, but without inheritance, I'm not sure the entire complexity of the CLOS is needed. Do correct me, though.
> Declarations are basically C/C++'s pragmas and __attribute__, but standardized and using a seamless syntax.
That's such a nicer description than in CLtL2. And yes, Yao will have them.
> The ClTl2 additions I'm talking about are ways for the compiler to give lexical information to macros: see https://github.com/Zulu-Inuoe/trivial-cltl2 and https://github.com/alex-gutev/cl-environments
If those are just for giving info to macros, Yao already has that; the lexer will tell keywords if the given token is a package, a keyword, an operator, a function, a type, etc., along with other info.
> However, Yao will not have inheritance; instead, it will be more like Go: composition over inheritance. And interfaces. So I could be wrong, but without inheritance, I'm not sure the entire complexity of the CLOS is needed. Do correct me, though.
I'm sadly not very well versed with this alternative to OOP. But something tells me that a more "basic" and less dynamic scheme like that is probably best for performance reasons, since you'd either need a Julia-like JIT or a granular "dynamism knob" like https://github.com/marcoheisig/fast-generic-functions to truly amortize the cost of CLOS-like dispatch and polymorphism; though magic can be done, as the "Fast generic dispatch for Common Lisp" paper shows.
> if those are just for giving info to macros, Yao already has that
Huh! If this includes types deduced by compiler inference and not just user declarations (since Yao seems gradually typed, to my eyes), that'd be pretty impressive.
Hope you'll be able to keep your spirits high, this is a lifetime's work you seem to be looking forward to ;)
> I'm sadly not very well versed with this alternative to OOP. But something tells me that a more "basic" and less dynamic scheme like that is probably best for performance reasons, since you'd either need a Julia-like JIT or a granular "dynamism knob" like https://github.com/marcoheisig/fast-generic-functions to truly amortize the cost of CLOS-like dispatch and polymorphism; though magic can be done, as the "Fast generic dispatch for Common Lisp" paper shows.
You know, for someone who isn't "well versed" with it, you certainly nailed exactly why I chose it. Also, simplicity. The "as easy as Python" is only possible if I remove complecting [1] things like inheritance.
> Huh! If this includes types deduced by compiler inference and not just user declarations (since Yao seems gradually typed, to my eyes), that'd be pretty impressive.
It does, actually. But Yao is not gradually typed, unfortunately. Too complecting.
[1]: https://www.youtube.com/watch?v=LKtk3HCgTa8
> As flexible as Lisp": I think this is proven.
Given the development model of Lisp is modifying a live image of a running program, I have to assume you limited this statement to the availability of macros.
> Given the development model of Lisp is modifying a live image of a running program
I would say it is modifying the running program itself. An image is the saved version of the heap. Saving/loading images is optional.
But with Lisp one can also compile source files to compiled files. In rare occasions it is also possible to compile whole programs.
Yes, sorry. I should clarify that.
People interested in this should also make sure to checkout Hylo, https://www.hylo-lang.org
There are several presentations on the design and mechanisms. Eg
- https://www.youtube.com/watch?v=ws-Z8xKbP4w (Val: A Safe Language to Interoperate with C++ - Dimitri Racordon - CppCon 2022)
- https://www.youtube.com/watch?v=oFupPFniD9s ([IWACO23] Borrow checking Hylo)
Serious mode: it's good that people are exploring the programming language space, both tiny and big changes can mean a world of difference in reducing bugs and making programs nicer both for the user to run and the developer to create and maintain.
Serious mode off. When will we get Yao in the Linux kernel?
> Serious mode off. When will we get Yao in the Linux kernel?
As soon as someone pays me long enough to get it there. :)
I only have 3 hours max a day (usually only 2) to work on Yao. And I have other projects I want to do as well.
But if someone paid, I sure could find the desire to work on it.
>Serious mode off. When will we get Yao in the Linux kernel?
Yaol can get involved and help with that, yaol know?
Serious_mode_still_off? true
Odd post, and it landed me on the Yzena website which is very surreal. It seems to be intentionally hard from the website to figure out what Yzena is or why anyone would want to buy a license for it (or the things it makes?). There was a link to a repo full of perl scripts, though.
According to the website I'm not allowed to ask about this though ;)
I only have one Perl script in that repo...
Anyway, Yzena is my business. I am not a very good marketer, and the website is bad.
I also don't think I will get any business, but eh.
> that language is coming into existence right now! It is called Yao.
Wasn't it called Mojo? It's been a while since I saw traction of that new language over here.
Anyhow, it is interesting how many new languages are appearing every day. The bar is indeed constantly moving upwards.
Mojo is completely unrelated, created by Chris Lattner and his team at Modular AI.
Is there something like this but without concurrency? I don't need concurrency or multithreading, and most of my pain with Rust has come from the fact that it prevents me from doing things that I know are correct because it thinks my code will run in a multithreaded environment (it will not).
It seems like Yao's whole project is solving the concurrency problem.
I'm like, why not drop it entirely (if performance isn't one of the language's top priorities anyway)? What does keeping it gain that's so important? (Context: I don't understand concurrency and multithreading.)
Good question.
Yao's concurrency will be explicit. You will not get concurrency unless you open a threadset and spawn threads. Yao scripts run in a single thread by default.
I focused on solving the concurrency problem because if it is solved, single thread code is solved as well.
> I don't need concurrency or multithreading
Famous last words. Ever needed a timeout? Cancelation? Respond to Ctrl+C (SIGINT)? Your UI has an event handler? Client server architectures? All of those things need concurrency, somewhere.
That said, you could argue that I’m being nitpicky. A combination of the OS and language runtime (eg JavaScript) can keep you mostly isolated from the complexity explosion of parallelism and concurrency. But it’s still there.
> it prevents me from doing things that I know are correct because it thinks my code will run in a multithreaded environment
Could you elaborate on that? In your case you don't need to worry about Send/Sync, you can use Cell/RefCell, etc. The borrow checker isn't about threading, it's useful in a purely single-threaded context too.
Not OP but don’t a lot of async libraries require things to be Send because they assume they’ll run on a multithreaded executor?
> I don't need concurrency or multithreading, > ... > Context: I don't understand concurrency and multithreading.
I get not needing it, but if you don't understand concurrency or multithreading how are you so certain you don't need concurrency or multithreading?
Slightly off-topic, but TFA links to https://gavinhoward.com/2024/05/what-rust-got-wrong-on-forma... which says:
> But there is an even harder problem: the C10M Problem. Is it possible to use RSC to handle 10 million connections, of which 1 million need some processing every second?
(RSC being restricted structured concurrency.)
IMO the C10M problem requires: a) user-land network stacks or unikernel servers, b) NPROC (or fewer) threads or processes, c) evented, continuation-passing style code in each of those threads or processes.
I've written one such program (though without (a)), a HTTP read-only file server that does (b) and (c) and which supports "tailing" files by doing `GET`s with `Range: bytes=0-` (or some other starting offset, but no ending offset), and it scales to tens of thousands of concurrent clients trivially. To get to millions of clients I would need to use H/3 and QUIC, and also do (a). (That server also supports ETags and conditional requests. Tailing `GET`s end when the file is unlinked or renamed away. Essentially it is a poor person's Kafka.) This program stores state in a per-client data structure and in the form of a pointer to a continuation closure that is `{data structure, handler closure}` that is registered with the event system, and each handler does only non-blocking operations (if you can think of filesystem operations as non-blocking anyways, though one could also use aio).
I bet this works well with RSC if all the state is kept in statically allocated objects. In my program all state is kept in dynamically allocated heap objects, but I could have statically allocated them instead, perhaps at program startup, or else at compile time.
The point here is to absolutely minimize the representation of state. In a thread-per-client implementation the state is smeared on each thread's stack and also any necessary heap objects. In a CPS implementation the state is fully explicit, with none of it in any stack frames -- the programmer is in charge of producing the most compact state representation, and it's easy because the state has to be fully explicit.
I view evented CPS and thread-per-client as the ends of a concurrency spectrum where green threads and co-routines are somewhere in the middle. If you want C10M you will probably end up wanting evented CPS. Green threads / fibers can get close to CPS if you work to minimize state anyways.
>Yao can’t be compiled in the traditional way.
>Instead, Yao will be compiled to an LLVM-like IR (which already exists), and that’s how it will be distributed. Users will have to do the final compile step on their local machines.
Is this referring to the compiler itself, or programs written in Yao?
Programs written in Yao.
If the goal is portability, Google pursued the idea of distributing cross-platform LLVM IR via PNaCl and ultimately abandoned it, because LLVM IR isn't really designed for portability.
I designed my own, uncreatively called Yvm. Yao already compiles to it.
Without sounding too much like a Cathy Woods, I'm wondering if AI can be leveraged to help with all of the non-core aspects of creating new languages? For example, really why does every language really need a json parser, or http requests library. If one were to examine the machine code instructions involved in each, they would likely be rather similar after all. Writing all of that stuff is an enormous lift for the community. Tooling is similarly "undifferentiated heavy lifting" in a way. Also, moving existing code bases to new languages is still hard.
If there were better ways to achieve these things, there would be much less inertia in the language space.
> Instead of marking asynchronous functions, languages should have had us mark synchronous functions […] Although that wouldn’t remove the possibility of bugs when calling a blocking function in async code,
So mark "synchronous" (i.e., non-yielding) and "asynchronous" (i.e., non-blocking). Then we can have the synchronous colour, the asynchronous colour, functions that are both (e.g. pure CPU-bound tasks, like arithmetic), and functions that are neither (e.g. buggy code).
[dead]
Glad I waited before starting with Rust. Now I can just jump straight to Yao. Rust is so early 2020s.
Please don't jump to Yao! The only thing it's good for right now is to replace shell scripts.
You're still using Yao? Pfff.
> Pfff
Now I want a language named that.
> You're still using Yao? Pfff.
One-up flex master right here. Unbeatable.
Hip crowd's already moved on to Tcl, I know (with the advancements in v9.0 and all).
Tcl is yesterday's news. Hip crowd moved to artisanal ASM.
Excuse me, but real programmers use butterflies
https://xkcd.com/378/