#dt - duck tape for your unix pipes

1 messages · Page 1 of 1 (latest)

acoustic urchin
#

Some questions from @sweet sonnet from #pet-languages

im a huge fan of concatenative langs and have written & rewritten all too many forths, and im really curious if you had any firsthand experience with them before dt?

Yes I'm definitely a fan. I actually built a whole integration for Factor into Amazon's internal build system to help me learn Amazon's thing. (Kinda Bazel/Buck/Blaze-ish, but much older) No one is using it of course, which was part of the reason I chose Factor, there's such a tiny chance that anyone would take my proof-of-concept and build something in production with it

dt is actually a rewrite and fork of a different concatenative language (For now called "Rail") that I don't quite know exactly what I'll do with it, but I have some ideas on how something concatenative could be useful as a compiler backend, and have been reading up on work by Anton Ertl to understand stack-based compiler optimizations... Probably have a few years to really understand and digest this stuff tho

I've also done lots of experimenting with Forth, although nothing for actual work (yet!) but I believe hacking and hobby work can be just as "real." I got a physical copy of "Thinking Forth" right on my desk, and have been following @warm grove and his Assembly Nights projects for a while. Maybe someday I'll do something like a JonesForth for

or was the idea more based on intuitions from day-to-day shell usage ("hey, this is really concatenative-y...") and then working backwards from there?

It's also this. I've been working deep in the bootstrapping layer of Amazon's code base for a few years, and it's a lot of shell to get things bootstrapped. Also I spent a lot of time automating my own work so I can get to much higher levels of productivity. That, and I've contributed to Linux distros and such, and also I'm a Linux geek, so the shell is my life.

#

I doubt I've found the "final form" of dt, but I think I've got a reasonable core that I can call 1.0 and I can build on for a while until I understand how other humans might use it too

#

dt - duck take for your unix pipes

#

PS I like keeping the name "duck tape" vs "duct tape" ambiguous; that is like half the discussion on the largest HN thread lol

acoustic urchin
#

dt - duck tape for your unix pipes

untold tree
acoustic urchin
#

Thanks! I'll fix that, I've been kinda rushing on the docs. Let me know if anyone finds any others

#

I guess if something is not in the table of contents, mdbook will not render it

sweet sonnet
# acoustic urchin Some questions from <@142835397301501953> from <#863860986745192478> > im a hu...

Yes I'm definitely a fan. I actually built a whole integration for Factor into Amazon's internal build system to help me learn Amazon's thing. (Kinda Bazel/Buck/Blaze-ish, but much older) No one is using it of course, which was part of the reason I chose Factor, there's such a tiny chance that anyone would take my proof-of-concept and build something in production with it
ooh sounds interesting, ill have to estalk investigate when ive got the bandwidth im curious, are custom build systems a Builder Tools rite of passage or just an extremely beneficial extracurricular? id been longing to join BT for ages but the powers that be dont seem to be very generous when it comes to headcount, especially since layoffs and austerity mode

#

dt is actually a rewrite and fork of a different concatenative language (For now called "Rail") that I don't quite know exactly what I'll do with it,
too relatable. dt actually being useful is something that really drew my attention. concat langs are just kinda... pointless in the real world, but fish and nushell have proven that sh compatibility isnt necessary to be useful. do you actually daily drive dt yet?

#

Anton Ertl
just skimmed some of his papers. some seem kinda interesting. i found https://sci-hub.ru/https://doi.org/10.1002/cpe.1016 pretty neat. its basically "what if we did old school forth AOT compilation as a JIT?". not the best compiler by any means, but its a really simple approach to making a portable-ish JIT. every time i see stuff like this, it gets me brain going on evil hacky ways to optimize forth. i always keep ending up with "where can i sneakily un-forth this forth code?", but unless you heavily constrain the backend's stack-oriented intermediate representation (like you have: no variadics, etc), then theres just way too much observable behavior that optimizing well sounds hard/impossible. however.. if you do have those contraints along with quotation-based control flow, then i suppose that a stack representation is probably as expressive as single static assignment with phi nodes (what llvm uses).

#

Maybe someday I'll do something like a JonesForth for
i would highly recommend hacking up a basic forth in assembly if youve got a spare weekend. i had learned mips assembly in college, but forthing was how i taught myself x86. so much fun! brad rogriguez has a super informative series of article called Moving Forth that goes over the "magic" that other resources do a poor job of explaining (especially DOES>. holy cow that word takes a few attempts to really grok). as a nice side benefit, he uses older assembly languages different enough that its not just copy pasting like jonesforth is :) https://www.bradrodriguez.com/papers/index.html

#

It's also this. I've been working deep in the bootstrapping layer of Amazon's code base for a few years, and it's a lot of shell to get things bootstrapped. Also I spent a lot of time automating my own work so I can get to much higher levels of productivity. That, and I've contributed to Linux distros and such, and also I'm a Linux geek, so the shell is my life.
and that doesnt make you despise shell with a burning passion? youre a stronger man than i
speaking of bootstrapping, my new team has finally solved the problem: every package is a NoOpBuild and you just refer to the scripts/ directory and an incorrectly documented list of system dependencies 🙃. oh, and dont forget the git submodules that point to codecommit repos, so you better have the git remote helper installed too 🙃. once i better understand our mono-ish repo build tooling, im on the first flight to Peru.

acoustic urchin
#

Are you an Amazonian too?

sweet sonnet
#

sengir@ :^)

acoustic urchin
#

I'm former-Amazon as of last Friday, was hiljusti@

sweet sonnet
#

i was the one asking about cflags during the brain dump

acoustic urchin
#

Oh right during the Zig meeting -- last week was a whirlwind of many things

sweet sonnet
#

same. last week was also my last week at my old role

acoustic urchin
#

id been longing to join BT for ages

I recommend it if only to grow skills. Holy moley does it push you to grow your skills. I've had multiple CRs in Amazon where more than 5 PEs were directly invested and commenting and etc. Especially these teams: SCCAM, Language Experience, Build Execution, with SCCAM being most challenging

#

talk to Robert Knight if you need a foot in the door for conversations

#

do you actually daily drive dt yet?

Not as a shell, but yes I've used it for non-trivial shell scripting kinda things. Last day of Amazon I used it to automate out patches to some Java 3p imports that had unused dependency resolutions

#

every time i see stuff like this, it gets me brain going on evil hacky ways to optimize forth.

I think the most promising idea on optimized Forth that I've seen so far is meow5. Basically: always aggressively inline compilation

sweet sonnet
acoustic urchin
#

He's a real good manager. Next person I could refer to is Jesus Ruiz-Scougall -- he's the only director I know with a github account lol

sweet sonnet
#

I recommend it if only to grow skills. Holy moley does it push you to grow your skills. I've had multiple CRs in Amazon where more than 5 PEs were directly invested and commenting and etc. Especially these teams: SCCAM, Language Experience, Build Execution, with SCCAM being most challenging
i love estalking buildertools CRs and its great seeing all the high viz stuff. but i especially loved seeing PEs fighting the good fight for Peru by dirtying their hands with updating, fixing, migrating ancient packages that everyone was too afraid to touch for whatever reason

acoustic urchin
#

oh yeah

#

a lot of that still needs to happen

#

but hands-on PEs is the name of the game in BT

#

in a not-stepping-on-toes kind of way

#

I wish, though, that I had a chance to work on Kuiper too. Especially if I could have done some embedded stuff THAT WOULD GO TO SPACE lol

#

I live near leblstev@ and we get the occasion to talk every once in a while

sweet sonnet
# acoustic urchin > every time i see stuff like this, it gets me brain going on evil hacky ways to...

thats basically what OG single pass forth compilers do, but the problem is that you really need additional passes to benefit from inlining. with modern branch prediction, branches can be basically free if you arent doing anything bone=headed, like jumping to a register offset. a lot of older forth implementation tutorials will misguide you to absolutely torturing your branch predictor with indirect threaded code ITC. thats another reason why i like brad's articles :)

sweet sonnet
acoustic urchin
#

something like that haha

sweet sonnet
# acoustic urchin in a not-stepping-on-toes kind of way

i love it! elsewhere i usually either saw two kinds of PEs:

  1. the man with the plan--they sketch out a prototype implementation of a thoughtful solution to a really complicated problem, then that serves as a spec of sorts for others to actually implement it
  2. too busy barraged by middle management meetings, holding office hours, and forwarding emails to really do anything on their own
sweet sonnet
acoustic urchin
#

Maybe some day haha

sweet sonnet
acoustic urchin
#

Very general stuff like the effects of cosmic radiation, or the state of proprietary software used to simulate satellites

sweet sonnet
# acoustic urchin Maybe some day haha

i will admit, i think the org really needs some Builder Tools peeps who really know their shit and how to best amazon-ify this whole mess of codebase. ive heard that others teams have brazil deps, others have peru deps, and like i said we use git submodules. i have no clue if anyone actually uses build fleet. 😉
i was referred to someone working on fixing this mess, but theyre ooo for another week or two. really excited to see what the longterm plan is

sweet sonnet
acoustic urchin
#

Almost all organizations at Amazon eventually end up with some local BT equivalent. Some "Platform Reliability" or "Infrastructure" or "Release Engineering" team -- especially subsidiaries. They'll work with BT on their needs, and pick up the work that's too specialized to whatever they do

sweet sonnet
#

or maybe that team already exists, idk. this is my second day lmao

#

also, completely unrelated anecdote that i find friggin hilarious (read: terrifying): following the "how to build" guide for our codebase and just doing the default "brain=off" instructions for each step will have you wgeting a random executable from github into your dev-dsk's /usr/bin 🙃

#

uh, i mean i love concatenative languages. dt is really cool

acoustic urchin
#

yay concatenative languages!

acoustic urchin
#

BTW I would recommend casual users to wait until

  1. Zig releases at 0.11.0
  2. Your package manager of choice supports Zig 0.11.0
  3. dt is packaged for your package manager. (I will probably create packages at least for Nix, AUR, Void, FreeBSD, and Homebrew)
#

If anyone is using it seriously before that point, let me know what you're doing so I don't break you inadvertently

#

Before then I'm considering some features "stable" only after I've documented the intended behavior (and make sure the real behavior matches the intent!)

acoustic urchin
acoustic urchin
acoustic urchin
#

Phew that was fun! But I really needed to rehearse a demo more if I do this again

analog flax
acoustic urchin
#

Definitely a huge inspiration

#

I feel like Forth is still ahead of its time

viral trench
#

I was trying dt out (nice project btw!), and i noticed that the gcd rosetta code example doesnt seem to work properly:

dt 1.3.1
Learn from my mistakes - someone should.
» [\x: [\y: x x y % gcd] x do?] \gcd def
» 15 24 gcd pl
9
»

acoustic urchin
#

Oh shoot

#

What did I do?

acoustic urchin
#

not tested extensively but I think this works. I think the previous one only worked by accident on some cases

viral trench
#

Nice, this seems to work properly! 👌

acoustic urchin
#

😮‍💨 phew

acoustic urchin
#

dogfooding project of the night:

acoustic urchin