ChronoGrapher - One Unified Scheduler, Unlimited Power

[fervent lark]

i am not sure how to feel about > just padding and not including a small bar r smth like that

you forgot about T to name it to MyFrame

also

maybe its best to reduce code fences the more i see it

like my_frame could be just "my_frame", with_retry -> "with_retry", with_timeout -> "with_timeout" and ye

[placid drum]

cmd + d really does a work

[placid drum]

do you actually mean in it right ?

[fervent lark]

again you forgot about T, it should be MyFrame and not T, remove the trait bound as well

[fervent lark]

ye

lol

i saw it

[placid drum]

just act like you didnt see a single thing 👀

[fervent lark]

lmao

ok i will xD

[placid drum]

lts get over this with 😉

[fervent lark]

open it as a PR

[placid drum]

done

[fervent lark]

ye saw it

man i see this repeatedly

more of a stylistic nitpick

`

using one

instead of two

but thats like stylying stuff i should have clarified

[placid drum]

i will try do with 1st attempt

[placid drum]

2 dosnt look good

[fervent lark]

i do prefer it imo

don't try to change it, il change it

since it isn't something the user will notice

why tf you removed in # See Also the other entries

/// - [`TaskFrame`] - The core trait that defines execution logic.
/// - [`Task`](crate::task::Task) - The top-level struct combining a frame with a trigger.

i didn't tell you to do that

also this is inside the codeblock, wtf?

/// // With the workflow created, `composed` is now the type:
/// // > ``FallbackTaskFrame<TimeoutTaskFrame<RetriableTaskFrame<MyFrame>>, BackupFrame>``
///
/// // all from this builder, without the complexity of manually creating this type

its supposed to be outside

below the codeblock in the example

[placid drum]

okay let me change asap

[fervent lark]

would reccomend using the dynamic_taskframe! macro

/// struct MyFrame;
///
/// #[async_trait]
/// impl TaskFrame for MyFrame {
///     type Error = String;
///     async fn execute(&self, _ctx: &TaskFrameContext) -> Result<(), Self::Error> {
///         println!("Executing primary task logic!");
///         Ok(())
///     }
/// }
///
/// struct BackupFrame;
///
/// #[async_trait]
/// impl TaskFrame for BackupFrame {
///     type Error = String;
///     async fn execute(&self, _ctx: &TaskFrameContext) -> Result<(), Self::Error> {
///         println!("Executing backup logic!");
///         Ok(())
///     }
/// }

or hmm

ye but then the type is gone

hmmm

i don't want to add cognitive overload to the user, this will be a bit difficult

[placid drum]

can we just add comment like backFrame has type of TaskFrame

or maybe

we could just add explanation

[fervent lark]

i do have an idea

[placid drum]

?

[fervent lark]

we could hide this stuff, then like what you said refer to the explaination

hmm

i was thinking like making it doctest friendly

actually nvm

i think we should go with your approach

[placid drum]

its actually passing doc test

[fervent lark]

because it doesn't execute anything?

except the builders

[placid drum]

yaa

[fervent lark]

i was talking more so runtime stuff but meh whatever

[fervent lark]

like use BackupFrame and MyFrame but on the explaination refer to them as you said

[placid drum]

yes but code it self is explanatory

[fervent lark]

ye

forget what i said, lets go with your approach

[placid drum]

?

keep things as it as ?

for example?

lol i am making myself confuse

[fervent lark]

ye lol

[placid drum]

ok should i keep example same or add explanation

[fervent lark]

omit the structs (though still do reference them in the code) and add explanation

[placid drum]

///
/// # Example(s)
/// /// use std::num::NonZeroU32; /// use std::time::Duration; /// use chronographer::task::TaskFrameBuilder; /// /// // `MyFrame` and `BackupFrame` are two types that implement `TaskFrame`. /// /// const DELAY_PER_RETRY: Duration = Duration::from_secs(1); /// /// let composed = TaskFrameBuilder::new(MyFrame) /// .with_retry(NonZeroU32::new(3).unwrap(), DELAY_PER_RETRY) // Failure? Retry 3 times with 1s delay /// .with_timeout(Duration::from_secs(30)) // Exceeded 30 seconds, terminate and error out with timeout? /// .with_fallback(BackupFrame) // Received a timeout or another error? Run "BackupFrame" /// .build(); ///

but it will fail doc test ?

or maybe last one was correct example

lol my brain not braining anymore

[fervent lark]

mine as well

honestly fuck doctesting

i mean its not something easily testiable

so, ignore it

[placid drum]

check pr

i guess we need to finish then focus on improving exampls

[fervent lark]

also why you leave a newline between the struct and doc string

[fervent lark]

@placid drum i've tweaked your example heavily

The end result is

[fervent lark]

which i rock

[fervent lark]

We have 3 examples worth studying for the API doc language

[placid drum]

# assert_eq!(composed.type_id(), TypeId::of::<ExpectedWorkflow>(), "Unexpected workflow type produced")

cool shit

[fervent lark]

lol

[fervent lark]

btw the core measures around ~44-45KB in size

a bit bigger than i thought

[fervent lark]

the .rlib file is 2.1MB apparently

@placid drum i've added a new DefaultTaskID, the underlying implementation is basically a UUID (u128), the reason for this is to minimize the number of dependencies as much as possible as currently uuid is meant to be general purpose where as our identifier format the only thing it wants is comparisons, hashing and generation.

You will have to document that a bit, but yeah it won't be much

now the rlib is around 2.09MB

the uuid was a low hanging fruit, not large but still something that could easily be made optional

the bigger reward will be making chrono and tokio optional, i'd imagine it will reduce significantly the size

[fervent lark]

btw i just realized we have rn 1 month and 1-2 weeks or so to deliver the project

god...

[fervent lark]

im planning to do the argument injection thingy

but man are the types really complex

[fervent lark]

hmm, im starting to believe there might be an issue with TaskFrames

[fervent lark]

Unfortuantely, @arctic trench will be leaving the team, as such, i am modifying the distribution of tasks, as it turns out i might also have to dive as well even though the plan was to parallelize things to get to the deadline quicker

specifically @jolly frost and @placid drum you will have one more task (its a collection of items really, so its not just one thing but a group) to do

il handle:

• everything inside of core/src/task/trigger/schedule.rs
• everything under core/src/task/trigger/schedule

it was quite unexpected

i really fear we won't get to the deadline with the quality i envision

and im kind of lost really, on what to do

[fervent lark]

im also kinda thinking if its really worthwhile implementing argument injection, it complicates the design so much, i believe there must be a better compile-time safe solution

[fervent lark]

sometimes i love Rust enforcing things at compile-time and being strict as a user using other libraries
other times when im making a library, i have to really think in this kind of angle of providing compile-time safety which i hate

[fervent lark]

i've documented the TaskSchedule trait

again another example to follow

also fricking discord interperted ::clock:: as 🕰️ lol

nvm i forgot one cruicial part

the example

[zenith urchin]

I have created another Draft PR and added documentation to core lib.rs as well as scheduler.rs. Currently, I am not done with the Scheduler.rs but most of it is complete. I didn't put exampless to some of the structs or impl because I didn't think they will be directly used by the user (i.e. SchedulerInitConfig which will not be directly used by the user, but more of a inner implemetation for the Default Scheduler builder). I only have TODO that is for SchedulerConfig because it can be used with several different configurations and I believe I am not that deep into codebase to provide a good example for it. That would be really nice to get feedback to the PR I will probably continue on next weekend and complete all of my remaining parts.

[fervent lark]

hey srry man i was like 12 hours at work

so couldn't look at it

il see it now

[fervent lark]

for examples, ideally do showcase how to implement

even if its "basic Rust" knowledge

for SchedulerInitConfig, ye i do agree

you shouldn't configure those

its a detail better omitted

good thing you accounted for this

not sure how i would feel about

///   - [`CronError`], [`CronErrorTypes`], [`CronExpressionParserErrors`], [`CronExpressionLexerErrors`].

it just lists

regarding this:

/// - **Core / scheduler** - General runtime and scheduler issues:
///   - [`StandardCoreErrorsCG`] - index out of bounds, unresolved dependencies, invalid cron, unsupported instructions, etc.

its slowly but surely being faded away

im looking very briefly

il look in detail tommorow

somewhere around <t:1772537400:F>

generally for the error module, the explaination is kind of hit and miss

in terms of breadth

and a bit towards the structure

now to the scheduler stuff

/// The `scheduler` module provides the runtime that runs [`Task`](crate::task::Task)s according to their
/// [`TaskTrigger`](crate::task::TaskTrigger)s, plus configuration and default implementations.

vague...

Expand a bit

don't over-analyze but do expand on your explaination

generally a theme i notice is vagueness

not enough description

[zenith urchin]

Yeah fair I will add small explanations to the list and how can they effect the regarding component.

[zenith urchin]

Sure 👍🏻 thank you for the feedbacks. I might be little busy in the work days for this week, so probably my next update will be on weekend

[fervent lark]

mhm

[fervent lark]

i reccomend heavily you take a look in TaskIdentifier, TaskSchedule, TaskFrameBuilder

these are top examples

[fervent lark]

im thinking for v0.0.1

to shrink the scope

Python and JS/TS SDKs won't be delivered

as a result

[fervent lark]

forgot to list in TaskSchedule the implementations

il get to it

[fervent lark]

im gonna apparently cut the scope of the contribution guidelines and refer to the contributions to ask me

yes pain on my part, ik, but most of the guideline stuff is covered

reviewing the code and understanding the current guidelines should be more than enough

[fervent lark]

il be working on optimization

im gonna significantly change the TaskTrigger system, removing the need of a TriggerNotifier and splitting the process to two methods

its more natural and ensures no mistake

im also gonna handle optimization since im directly colliding on this part

[fervent lark]

im examining very closely https://docs.rs/hierarchical_hash_wheel_timer/latest/hierarchical_hash_wheel_timer/

from what i gather ByteWheel is the main building block

(obviously)

a byte wheel is basically an array of 256 cells

each cell is Option<WheelEntryList<EntryType, RestType>>, where Option<...> ofc means it can be something or nothing

and WheelEntryList is a vec of all entries hitting the exact same field

there is a current "pointer", which increments for every tick (wrapping)

and a count for tracking the length of the entire timer wheel (how many entries in total)

insertion is as simple as creating an entry, checking if the slot exists and if not then creating an array and then later on that array (can be the newly created if it didn't exist or old if it existed), it inserts the entry

and increments the counter

for ticking, it increments (in a wrapping fashion) the tick by one

and on the index it takes the slot (replacing it with empty) its currently in

if it existed, then it decrements the count

thats it really...

but i do notice some stuff bothering me for my use case

the first thing im planning to optimize is to make it tickless to skip dead space

this will be challenging

the second slight, issue, not sure if i like WheelEntry, it does allocation, i guess Rust could treat it as a contigious block or smth?

   pub fn insert(&mut self, pos: u8, e: EntryType, r: RestType) {
        let index = pos as usize;
        let we = WheelEntry { entry: e, rest: r };
        if self.slots[index].is_none() {
            let l = Vec::new();
            let bl = Some(l);
            self.slots[index] = bl;
        }
        if let Some(ref mut l) = &mut self.slots[index] {
            l.push(we);
            self.count += 1;
        }
    }

though i am thinking...

byte wheel but itself seems optimal

[placid drum]

@fervent lark is this also contain doc ?

impl<T: TaskFrame> TaskFrameBuilder<T> {
    pub fn new(frame: T) -> Self {
        Self(frame)
    }
}

impl<T: TaskFrame> TaskFrameBuilder<T> {}

[fervent lark]

yes the new method will have to contain docs

[placid drum]

and for TaskFrameBuilder and their method ?

[fervent lark]

yes

also

i forgot

modules count as file.rs (rust) files

and anything mod

so best to document those as well

i reccomend using //!

[fervent lark]

i've got some hierirchical wheel stuff

type WheelShardSender<T> = tokio::sync::mpsc::Sender<WheelShardCommand<T>>;

pub struct HierarchicalTimingWheel<T: 'static> {
    level1: [WheelShardSender<T>; 4],
    level2: [WheelShardSender<T>; 4],
    level3: [WheelShardSender<T>; 4],
    level4: [WheelShardSender<T>; 4],
    level5: [WheelShardSender<T>; 4],
    size: usize,
}

enum WheelShardCommand<T: 'static> {
    Insert(T, u8),
    Skip(u8),
    Tick(tokio::sync::oneshot::Sender<(Vec<T>, usize)>),
}

fn create_wheel_shard<T: 'static>(local_set: &LocalSet) -> WheelShardSender<T> {
    let mut shard: WheelShard<T, 32> = WheelShard::default();
    let (tx, mut rx) =
        tokio::sync::mpsc::channel::<WheelShardCommand<T>>(1024);

    local_set.spawn_local(async move {
        while let Some(command) = rx.recv().await {
            match command {
                WheelShardCommand::Insert(val, pos) => {
                    shard.insert(pos, val);
                },

                WheelShardCommand::Skip(val) => {
                    shard.skip(val)
                },

                WheelShardCommand::Tick(sender) => {
                    let _ = sender.send(shard.tick())
                        .unwrap_or_else(|_| panic!("Could not send tick results"));
                }
            }
        }
    });

    tx
}

impl<T> Default for HierarchicalTimingWheel<T> {
    fn default() -> Self {
        let local_set = LocalSet::new();
        let level1 = std::array::repeat(create_wheel_shard(&local_set));
        let level2 = std::array::repeat(create_wheel_shard(&local_set));
        let level3 = std::array::repeat(create_wheel_shard(&local_set));
        let level4 = std::array::repeat(create_wheel_shard(&local_set));
        let level5 = std::array::repeat(create_wheel_shard(&local_set));

        Self {
            level1,
            level2,
            level3,
            level4,
            level5,
            size: 0
        }
    }
}

the HierarchicalTimingWheel is split to five levels (to make it future proof)

each level contains 4 shards

each shard fully manages its own self

the HierarchicalTimingWheel will expose methods which mimick the wheel shard (tick, insert, skip... etc.) and more

the algorithm for ticking:

• Tick the wheel shard
• If it overflows, tick the next
• Repeat til you're at the end
• Upgrade to the next level and repeat this process

Naturally this wraps around each field so no problem there

the algorithm for the insrtion is:

• Given a Duration, calclulate the number of ticks (treat it as one contigious timer wheel and not shards) per level
• For each level's calclulated tick, deduce which shard it falls into and insert it

for HierarchicalTimingWheel, its best to use TaskIdentifier and even more specifically, best to use raw pointers

to avoid frequent cloning

[jolly frost]

hm, why only 32 slots

[fervent lark]

32 slots multiplied by 4 is 256, i.e. 1 "Byte"

i've looked at the code of https://github.com/Bathtor/rust-hash-wheel-timer/blob/master/src/wheels/quad_wheel.rs

and well each wheel has 256 slots

i sharded it to 32 slots (256 / 4) for reducing locks

the reason its a power of 2 is because i have optimized it a bit to not use modulo (micro but still matters):

(self.current + 1) & N

[fervent lark]

so hierarchical timing wheel is sort of finished

[fervent lark]

and it performs worse than the binary heap

fu-

not even 350k T/S

i found something particularily interesting... The clock is useless as a standard scheduler component

it can be shoved to the SchedulerTaskStore

[fervent lark]

but im thinking actually more advanced

what if instead of the SchedulerTaskStore managing the sorting of tasks, the clock, the storing of said tasks and so on

what if a different thing manages it all toggether?

introducing the SchedulerTaskPlanner

its job is to handle sorting the when to execute the Tasks

it uses internally SchedulerClock

hmm...

actually no

a better idea i have is removing SchedulerTaskPlanner and making the SchedulerEngine the orchestrator

the management between components is something static

that way we have 3 components

[fervent lark]

and the good thing about it is since the component interactions never change, the user will never have to think about it, the Scheduler handles combining them

the only stuff the user thinks are:

• How to sort tasks (which most of the time, the answer is hierarchical timing wheels)
• How to store tasks (most of the time the answer is a persistent store)
• How to dispatch tasks

[fervent lark]

what the actual fuck...

[fervent lark]

apparently it was bad luck ig?

the hierirchical timing wheel seems to have improved slightly the T/S

from ~400k to around ~425-450k T/S

i wanna squeeze however, every ounce of performance from that hierarchical timing wheel

performance seems to be unstable

from benchmarking i see lots of Mutex locks

out of 6 minutes my profiler showws (the process did 1 minute in total so its split per CPU core), 55.67 seconds were spent on Mutex

specifically std::sys::pal::unix::sync::mutex::Mutex::lock::hab1d77b19c2ec51b

[fervent lark]

apparently the benchmark script wasn't really stable

so i added randomization

in the graph it shows blue as ChronoGrapher and red as tokio_schedule

tokio_schedule has around +116% the performance over ChronoGrapher

[fervent lark]

i've shot the time for Mutex locks from the above to 10.05 seconds

and now the biggest cost are allocation / deallocation

[fervent lark]

@jolly frost

@jolly frost

you aren't gonna believe your eyes

[jolly frost]

[fervent lark]

i've cranked this bad boy to

fucking insane

620k T/S on average

that is so fast

[jolly frost]

Damn

Crazy

[fervent lark]

and we can still push it further

[fervent lark]

not to mention

this was a sort of cold start

re-running it again

the gap closes even more

[fervent lark]

btw on topic about performance

im also noticing not only allocation / deallocation cost (from heap via Box<T> and stuff)

but also something more interesting

nvm

[fervent lark]

ladies and gents

we have reached 800k T/S

peak is around 840k T/S

OMFG

ITS THIS CLOSE

[fervent lark]

@placid drum

btww

how are the docs going?

[fervent lark]

im thinking by the way of re-organizing a bit ChronoGrapher's scheduling process to squeeze every performance

[fervent lark]

nvm

the only thing i managed to do is removing the EngineNotifier

instead make it automatic

[placid drum]

Working on it

[placid drum]

whats on x axis btw ?

[fervent lark]

the lifetime of the program

how many seconds have passed since starting the scheduler

[fervent lark]

good thing

can you show a bit progress on some of the methods

so far you are the only guy i suppose that does the docs very good and how i want them to be

but regardless, i still want to monitor the progression

[fervent lark]

i have an idea for optimization

even if its not the main cost, currently since everything is erased as a trait object there is some vtable lookup and such which causes it to be slow about some microseconds (~9.21)

since i do have a couple of implementations that are direct and used very often

i could try to inline those as an enum

then for the cold path which are TaskFrames that are from user side

i could just store it as an additional enum variant just erased

[fervent lark]

perhaps it may not add much

best to stick to the optimizations that yield high results

[fervent lark]

NO FUCKING WAY

@jolly frost

@jolly frost

[placid drum]

i will try to make pr today

[fervent lark]

i feel like this ain't normal

either i must be dreamin shit

or this is actually real

[fervent lark]

the only thing that tokio_schedule beats chronographer in is RAM usage

chronographer use 5.2GB of RAM whereas it uses 120MB

chronographer is also heavier to start and terminate, requiring multiple seconds, the allocation of Tasks is also a bottleneck

whereas tokio_schedule starts immediately, terminates immediately and so on

[jolly frost]

wow

finally

[jolly frost]

🥀 5.2gb is quite a lot

[placid drum]

  /// Method creates a new [`TaskFrameBuilder`] by wrapping the given [`TaskFrame`], this is the
    /// only entry point for constructing a builder for the workflow.
    ///
    /// The provided [`TaskFrame`] becomes the innermost layer of the composed workflow. Subsequent
    /// `with_*` calls wrap additional behavior around it, and [`build`](TaskFrameBuilder::build)
    /// extracts the final composed frame and builds complex workflows.
    ///
    /// # Argument(s)
    /// - `frame` - Any type implementing [`TaskFrame`], this becomes the base frame that all
    ///   subsequent wrappers are layered on top of.
    /// 
    /// # Returns
    /// A [`TaskFrameBuilder`] wrapping `frame`, ready for chaining `with_*` methods.
    ///
    /// # Example(s)
    /// 
    /// use chronographer::task::TaskFrameBuilder;
    /// # use chronographer::task::{TaskFrame, TaskFrameContext};
    /// # use async_trait::async_trait;
    /// #
    /// # struct MyFrame;
    /// #
    /// # #[async_trait]
    /// # impl TaskFrame for MyFrame {
    /// #     type Error = String;
    /// #
    /// #     async fn execute(&self, _ctx: &TaskFrameContext) -> Result<(), Self::Error> {
    /// #         Ok(())
    /// #     }
    /// # }
    ///
    /// // Wrap `MyFrame` in a builder, then immediately extract it unchanged.
    /// let frame: MyFrame = TaskFrameBuilder::new(MyFrame).build();
    ///
    /// When called without any `with_*` methods, [`build`](TaskFrameBuilder::build) returns
    /// the original frame as-is. In practice you would chain one or more wrappers before building for more complex workflows as per requirements.
    ///
    /// # See Also
    /// - [`TaskFrame`] - The trait that `frame` must implement.
    /// - [`build`](TaskFrameBuilder::build) - Consumes the builder and returns the composed frame.

damm whats with color

@fervent lark

[placid drum]

check now

well there is not alot to add

if you have any suggestion let me know

[formal sedge]

How tf can it use 5gb?

[fervent lark]

idk

[fervent lark]

ye

it is something i want to optimize

[fervent lark]

ye i feel it

[fervent lark]

    /// # Argument(s)
    /// - `frame` - Any type implementing [`TaskFrame`], this becomes the base frame that all
    ///   subsequent wrappers are layered on top of.

Avoid bullet lists with one item, make it a sentence

[fervent lark]

also you forgot ``` in the example(s)

[fervent lark]

honestly its perfect for the most part

[fervent lark]

now it uses 69MB

TaskHooks were the problem

im kinda suspicious

[fervent lark]

nvm

my dumbass

i've reduced it though to 2.88GB

i reduced it to 218MB

[fervent lark]

i wanna drop memory usage further down, but it seems i can't

[fervent lark]

@placid drum i assume https://github.com/GitBrincie212/ChronoGrapher/pull/138 is ready right?

[fervent lark]

I've slightly reduced the memory usage around 4MB by using a promotion system

the idea is when registering hooks, it doesn't allocate a HashMap immediately

instead, there are various states which it passes through before ultimately allocating a HashMap

these states store directly the hooks with their IDs as a tuple

there is Empty for nothing, Single for one hook, Double for two, Triplet for three and for anything above you have Multiple which is the HashMap

[placid drum]

yes

[fervent lark]

oki dokie

hm?

oh nvm

it does mention some stuff are undocumented

the second bullet point is kinda confusing

[fervent lark]

a good practice btw is to link to the above docs for methods

for builder in TaskFrameBuilder

you would typically link the TaskFrameBuilder

il fix it though

[placid drum]

okay gotchaa

[fervent lark]

also you always add one space after the method

for the docs

[fervent lark]

but alas, so far, you're doin good, keep it up honestly

[placid drum]

okay thanks

[fervent lark]

btw for the CI/CD pipeline, i have a plan for it to look sort of like this. Its seperated to stages, if a stage fails then it fails entirely:

Stage 1 (Compilation)

• Checks if the project can be compiled in all targets via https://docs.rs/cargo-hakari/latest/cargo_hakari/. Success? Pass

Stage 2 (Verification)

For every OS:

• Runs cargo clippy with strictest settings. Success? Pass
• Runs unit tests via https://nexte.st. Success? Pass
• Run benchmarks from https://codspeed.io. Success? Pass
• Run https://app.deepsource.com/ (like we are doing). Success? Pass
• Runs https://crates.io/crates/cargo-audit. Success? Pass
• Runs https://embarkstudios.github.io/cargo-deny/. Success? Pass
• Runs cargo miri test. Success? Pass
• Runs https://crates.io/crates/cargo-outdated. Success? Pass
• Runs fuzzy test (could belong in unit tests and will need to integrate fuzzy testing). Success? Pass

Stage 3 (Formatting)

• Runs cargo fmt
• Remove unused dependencies
• Use of rustfmt
  (Maybe some additional ones as well)

Stage 4 (Documentation Linting & Injection)

Scans API docs, parses them, lints them to see if there are any issues and injects them to Fumadocs as its own MDX (we will make a tab for API docs)

Stage 5 (Deployment)

• Publishes the new version onto cargo (We will do similar stuff with other SDKs)
• Builds and publishes the website (il see how i can do the deployment part)

stage 3 might benifit from pre-commit hooks as opposed to CI/CD pipeline directly

there will be a lot more tools for the stricest possible setup

but this is a taste on how strict things will get

[fervent lark]

Apparently tokio_schedule uses 90MB (mostly stable), whereas ChronoGrapher uses 165MB (very slightly fluctuating)

[fervent lark]

100 of those MBs are used in the SchedulerHandle TaskHook

one optimization idea is to lazily create the SchedulerHandle

the problem is its sort of impossible without heavily refactoring the code such that we inject a channel

which is intrusive

[fervent lark]

i can't really optimize the memory further sadly

[nova hare]

that sucks

[fervent lark]

oh ur back, lol

i have one idea, there may be cases where you don't need SchedulerHandle at all

i can sneak through a feature that allows fully disabling it

and thats what i did

[fervent lark]

if you want extreme performance, you just disable SchedulerHandle

it drops to 61MB in memory

startup times also drop by 0.2 seconds

il focus btw on features. The idea is ChronoGrapher can be customized to have only the features needed for your use case

[fervent lark]

Quite suprisingly, TaskHooks are zero-cost

litterally

discluding the memory used

[fervent lark]

turns out, hmm, i don't like this idea in particular

[fervent lark]

il need to think about time

specifically removing SystemTime and making it future proof

for distributed systems

[fervent lark]

proc macros are kinda cool

[fervent lark]

i hate macros

nvm

[placid drum]

i will try to complete task builder today but not sure i'll get time but 'i'll try

[placid drum]

@fervent lark what is eyre ?

is it different way of handeling error ?

[placid drum]

@fervent lark kept example same almost

[fervent lark]

yup

its a fork of anyhow

[fervent lark]

nice

[fervent lark]

https://github.com/GitBrincie212/ChronoGrapher/commit/68b3355e5fbd647aefb96fb903b6786d5a1792a9

finally...

[placid drum]

?

[fervent lark]

oh hold on

[fervent lark]

For # Argument(s), it feels a bit too robotic immediately to say the argument

for with_instant_retry

I would add The method accepts only one argument that being ...

same with # Returns

The method returns ...

also for # See Also if i were you, again i would refer to the upper struct TaskFrameBuilder

even if its kind of redudant

[fervent lark]

also you're gonna have some issues, as il be restructuring the crate due to macros

[fervent lark]

im not sure actually tbh

one thing we may need to do is seperate the website from the codebase

as a seperate repositery

and generally keep this change for every extension, integration and so on

(not the SDKs though)

the problem really are macros

[fervent lark]

i've also made it into a project

https://github.com/users/GitBrincie212/projects/2

[placid drum]

i'll wait

[fervent lark]

no don't

you can still do the editing

just know that once il publish these changes

[placid drum]

okay make sense

[fervent lark]

you will most likely have to resolve merge conflicts

[placid drum]

🫡

[fervent lark]

its more of a warning / anticipation on what you can expect from this change

[placid drum]

ill follow your lead

[fervent lark]

also a bit of a critic, if you were a user in the library, would you like the following macro:

every!(1ms); // every millisecond
every!(2s); // every 2 seconds
every!(3m); // every 3 minutes
every!(4h); // every 4 hours
every!(5d); // every 5 days

every!(2.5s); // every 2.5 seconds
every!(5.12d); // every 5.12 days

// Seperator format:
every!(3m, 2s, 1ms); // every 3 minutes, 2 seconds and 1 millisecond
every!(5d, 2s); // every 5 days and 2 seconds

// Spaced out format:
every!(3m 2s 1ms); // every 3 minutes, 2 seconds and 1 millisecond
every!(5d 2s); // every 5 days and 2 seconds

[placid drum]

its good but like doing (5d 2s) instead cant we do (5, 0, 0, , 2) day, hour, minute and sec ?

like it would be easy right ?

[fervent lark]

its kinda ambigious

and hard to read

for a beginner they would question what does the first 5 mean

day? month? week?

they'd have to check docs for the format

the units do tell even at a first glance on what happens

[placid drum]

well its okay then its wonderful just say run every fucking 5.30day

[fervent lark]

"every" (the macro's name)
[UNIT]
[SUFFIX]

[placid drum]

kinda cool

[fervent lark]

you can say that, or you can break it up to the fields

for it to be more readable

so you won't have to do the math on your own

[fervent lark]

compare the macro approach vs:

TaskScheduleInterval::duration(Duration::from_secs_f64(...))

you would probably prefer the macro based approach over this?

[placid drum]

yah timing is alwasy pain when are beginner

[placid drum]

yay

who gonna calculate the second if i can just specify day

[fervent lark]

ye

it just does it on its own

[fervent lark]

the same mechanism will btw be implemented for the attribute macros

like

#[task(schedule = every(2s))]
async fn MyTask(ctx: &TaskContext) -> Result<(), MyErrors> {
    println!("Hello ChronoGrapher!");
    Ok(())
}

the every! macro in its current form serves as a short hand

[fervent lark]

im in pain...

im trying to manage the macro feature

and well i have macros which i use

macro_rules! define_event {
    ($(#[$($attrss:tt)*])* $name: ident, $payload: ty) => {
        $(#[$($attrss)*])*
        #[derive(Default, Clone, Copy, Debug, PartialEq, Eq, Hash)]
        pub struct $name;

        impl TaskHookEvent for $name {
            type Payload<'a> = $payload where Self: 'a;
        }
    };
}

macro_rules! define_event_group {
    ($(#[$($attrss:tt)*])* $name: ident, $($events: ident),*) => {
        $(#[$($attrss)*])*
        pub trait $name: TaskHookEvent {}
        $(
        impl $name for $events {}
        )*
    };

    ($(#[$($attrss:tt)*])* $name: ident, $payload: ty | $($events: ident),*) => {
        $(#[$($attrss)*])*
        pub trait $name<'a>: TaskHookEvent<Payload<'a> = $payload> {}
        $(
        impl<'a> $name<'a> for $events {}
        )*
    };
}

they are both user-facing

and used internally to reduce boilerplate

by using cfg on them, i will get errors on the crates

one idea i am thinking is making it a derive macro

one idea is to make those attribute macros

[placid drum]

looks dangerous to me

[fervent lark]

?

whats dangerous about it

[placid drum]

never did macro

[fervent lark]

oh

[fervent lark]

i've benchmarked the library for Heavy Tasks, it consumes around 600MB of RAM and throughput is:

for context on the heaviness we are talking

        let task = Task::new(
            TaskScheduleInterval::from_secs_f64(millis),
            TaskFrameBuilder::new(MyTaskFrame)
                .with_timeout(Duration::from_secs_f64(31.234))
                .with_fallback(NoOperationTaskFrame::<Box<dyn TaskError>>::default())
                .with_instant_retry(NonZeroU32::new(3).unwrap())
                .with_timeout(Duration::from_secs_f64(30.5))
                .with_fallback(NoOperationTaskFrame::default())
                .build()
        );

        task.attach_hook::<OnTaskStart>(Arc::new(MyDummyHook(Vec::with_capacity(1024)))).await;
        task.attach_hook::<OnTaskEnd>(Arc::new(MyDummyHook(Vec::with_capacity(596)))).await;
        task.attach_hook::<OnTimeout>(Arc::new(MyDummyHook(Vec::with_capacity(392)))).await;

struct MyDummyHook(Vec<u8>);

#[async_trait]
impl TaskHook<OnTaskEnd> for MyDummyHook {
    async fn on_event(&self, _ctx: &TaskHookContext, _payload: &<OnTaskEnd as TaskHookEvent>::Payload<'_>) {
        yield_now().await;
    }
}

#[async_trait]
impl TaskHook<OnTaskStart> for MyDummyHook {
    async fn on_event(&self, _ctx: &TaskHookContext, _payload: &<OnTaskStart as TaskHookEvent>::Payload<'_>) {
        yield_now().await;
    }
}

#[async_trait]
impl TaskHook<OnTimeout> for MyDummyHook {
    async fn on_event(&self, _ctx: &TaskHookContext, _payload: &<OnTimeout as TaskHookEvent>::Payload<'_>) {
        yield_now().await;
    }
}

#[async_trait]
impl TaskFrame for MyTaskFrame {
    type Error = Box<dyn TaskError>;

    async fn execute(&self, _ctx: &TaskFrameContext) -> Result<(), Self::Error> {
        yield_now().await;
        COUNTER.fetch_add(1, Ordering::Relaxed);
        if fastrand::bool() {
            return Err(Box::new("Hello World") as Box<dyn TaskError>);
        }
        Ok(())
    }
}

[fervent lark]

im gonna increase throughput even more via work stealing

il have to make it clever though

not just naive

the idea is basically before a SchedulerWorker is waiting, it will attempt to steal work from other workers

we will have to track the worker with the most work and we will need some way to steal work effeciently

stealing one work from the worker is kind of nothing, stealing all of the work is basically moving the work from one to the other

one idea is to keep track of which worker is the busiest of all, then each idle worker takes half of the work of the busiest

i've unified the queues without work stealing

1.9 MILLION LIGHWEIGH TASKS PER SECOND

this is fucking crazy

with work stealing this can go to insane numbers

[jolly frost]

damn

fire

[fervent lark]

and we can go higher

[jolly frost]

with work stealing

[fervent lark]

ye

[fervent lark]

that number is mind boggling large

[jolly frost]

yeah

ig much more than tokio?

[fervent lark]

i mean its using tokio

so we can't go much more

i have one idea for work stealing

using a priority queue, sorting the workers via the work they have

specifically a fixed priority queue

when an idle worker wants to steal work, it will look on the priority queue the most busy worker immediately

[jolly frost]

good idea

[fervent lark]

also by fixed, its technically a sorted array

when work is allocated, the workers swap cells

though the problem is how will i make it exactly 🤔

[jolly frost]

but we can't sort array on every update

performance

We can just move the worker up or down the priority array until it reaches its correct position

[fervent lark]

mhm

[jolly frost]

current array:

[W0:10, W1:8, W2:5, W3:3]

W2 wants to steal 5 tasks from W0 ->

[W0:5, W1:8, W2:5, W3:8]

W0 lost a lot of tasks:
Compare W0 with its neighbor on the right, 5 < 8 => change
[W1:8, W0:5, W2:5, W3:8]
W3 received many tasks:
Compare W3 with its neighbor on the left: 8 > 5 => change
[W1:8, W0:5, W3:8, W2:5]
Compare W3 with its neighbor on the left: 8 > 5 => change
Array: [W1:8, W3:8, W0:5, W2:5]
Compare W3 with W1: 8 = 8 => nice

new array:
[W1:8, W3:8, W0:5, W2:5]

[fervent lark]

mhm

[fervent lark]

sounds a bit intensive on performance

THE PROPHICIES ARE TRUE

[jolly frost]

ig depends on the number of workers

[fervent lark]

so far i have 64 workers

[jolly frost]

hm

rust sort time complexity is O(n log n)

[fervent lark]

kinda expensive

[jolly frost]

mine ig is O(n)

or nope

hm

or

well

BTreeMap maybe

[fervent lark]

hmmmmm

[fervent lark]

@jolly frost

do work stealing yourself

first benchmark before work stealing

it has to be very effecient

the goal is 2.2 million tasks per second

[fervent lark]

@jolly frost

for the work stealing how you gonna approach it exactly, do want to see how you'd do it

[fervent lark]

btw there is crossbeam workers

which do manage work stealing

the problem is we can't really allocate work from seperate threads

[fervent lark]

imma configure codspeed

since i want reliable benchmarks

and rn the benchmarking script isn't too reliable with the sudden rise and then decline

[fervent lark]

im integrating codspeed to measure

[fervent lark]

one thing i see apparently an issue with are TaskHooks not stacking, when you allocate one TaskHook for one event, it stays put, right?

though since its a HashMap, allocating a new instance of same type TaskHook with the same event just different values, erases the previous

for shared data api this is a nightmare because it means the previous version is gone forever

im also thinking of optimizing TaskHooks

the idea of how its organized is something like this

A TaskHookContainer consists of keys Event & Task Pairs where the values are additional HashMaps checking the TypeId and matching it with the TaskHook instance

the TaskHook instance will now be its own Vec which you can see how bad the overhead will be

[fervent lark]

i propose a "tree" structure

a Vec<T> but instead of nested Vec<T> we inline it

and calculate ourselves the index by shifting and so on

for few TaskHooks its real good both memory wise and speed-wise

only when we introduce many of them though like hundreads of them with repeated lookups, only then it becomes a problem

[fervent lark]

turns out i can't really change this

due to the need of distributed systems later on

[fervent lark]

i've done some optimizations via unsafe operations, and i've created an issue about a bug https://github.com/GitBrincie212/ChronoGrapher/issues/140

this bug doesn't tie with the unsafe operations, it is its own thing caused by moving the TaskHooksContainer from instance level to a global registry indexed via a usize which is the task's instance id

il be making more issues

[fervent lark]

such as https://github.com/GitBrincie212/ChronoGrapher/issues/141

[fervent lark]

https://github.com/GitBrincie212/ChronoGrapher/issues/143
https://github.com/GitBrincie212/ChronoGrapher/issues/142

more issues

[fervent lark]

In total, rn i've added 7 issues

[fervent lark]

now its 12

ts peak 🥀 (fucking rainbow bars lmao)

https://klipy.com/gifs/gohan-its-so-peak

[fervent lark]

btw @zenith urchin howz the progress going for docs?

[placid drum]

i was busy this week i will start from tomorrow

[fervent lark]

okie dokie

[fervent lark]

i've made yet another issue https://github.com/GitBrincie212/ChronoGrapher/issues/154

actually another one as well but ok...

[placid drum]

@fervent lark there ?

i have question like if we retry the task using with_instant_retry and after x try if fail it will paink right only certain task right not whole thread ? right ?

do we have task dependency means i have certain task like that my 2nd task is depend on 1st one what if 1st panic does 2nd task also get panicked or it will become dead task (task which does nothing ) ?

  /// Method wraps the inner [`TaskFrame`] in a [`RetriableTaskFrame`] configured for instant retries.
    ///
    /// This wrapper allows the execution to immediately retry upon failure without any
    /// intermediate delay (backoff). It is particularly useful for fast-failing, transient
    /// issues where a delay would be unnecessary.
    ///
    /// # Arguments
    /// `retries` is a type [`NonZeroU32] parameter specifying the maximum number of times frame should retry on failure.
    /// even after retries task not able to recover from the error, task will be terminated.
    ///
    /// # Returns
    /// A [`TaskFrameBuilder`] wrapping the composed [`RetriableTaskFrame`].
    ///
    /// # Example(s)
    ///
    /// use std::num::NonZeroU32;
    /// use chronographer::task::TaskFrameBuilder;
    /// # use chronographer::task::{TaskFrame, TaskFrameContext, RetriableTaskFrame};
    /// # use async_trait::async_trait;
    /// #
    /// # struct MyFrame;
    /// #
    /// # #[async_trait]
    /// # impl TaskFrame for MyFrame {
    /// #     type Error = String;
    /// #
    /// #     async fn execute(&self, _ctx: &TaskFrameContext) -> Result<(), Self::Error> {
    /// #         Ok(())
    /// #     }
    /// # }
    ///
    /// let retries = NonZeroU32::new(3).unwrap();
    /// let builder = TaskFrameBuilder::new(MyFrame)
    ///     .with_instant_retry(retries); // Retries up to 3 times on failure
    /// 
    ///
    /// # See Also
    /// - [`TaskFrameBuilder::with_retry`] - For retrying with a constant delay.
    /// - [`TaskFrameBuilder::with_backoff_retry`] - For retrying with a custom backoff strategy.
    /// - [`RetriableTaskFrame`] - The underlying frame wrapper.

or should 1st line should be like this

    /// [`with_instant_retry`] is a builder method that wraps the inner [`TaskFrame`] in a [`RetriableTaskFrame`] configured for instant retries.

@fervent lark

[fervent lark]

hold on

[placid drum]

ok

[fervent lark]

im quite sick so i am not going as fast as i usually do

[placid drum]

bro take care health is wealth

[fervent lark]

true

i mean i feel semi well plus its like a common cold

[placid drum]

i got little cold but not worth mentioning about its just drastic climate change

and take your time

[fervent lark]

ye ik what you talkin

runny noise and stuff

[fervent lark]

i prefer the top for the summary

after retries task not able to recover from the error, task will be terminated kinda broken grammar there

i suggest saying

even after retries, the workflow part may not be able to recover from the error and thus propegate it

also task will be terminated, well that holds true when you include it at the top

plus its Scheduler specific detail so not worth mentioning

[placid drum]

okay

[fervent lark]

A [`TaskFrameBuilder`] wrapping the composed [`RetriableTaskFrame`]. I suggest saying ... wrapping its inner workflow with an immediate retry

for the example, i'd reccomend using .build() right after

[placid drum]

okay

[fervent lark]

for # See Also, link to TaskFrame and TaskFrameBuilder, the former will be at the bottom while the latter at the top

[placid drum]

got it

[placid drum]

wait dont merge

ahh forgot too pull before

push

[fervent lark]

?

[fervent lark]

btw i realize im hitting very close to optimal

i've allocated 350k tasks right? Each triggering at an interval rand(0.0..=1.0) / 6

each Task executes on max 6 times per second

the maximum threshold is 2.1 million Tasks per second

whereas we are reaching 1.7 - 1.8 million

for 3 times per second

1.050.000 million

hmm 🤔

so every Task is executing around 5 times per second

suspicious, its too biased

nvm

[fervent lark]

interesting...

i've changed the benchmark script and now it resets the counter back to 0 every second instead of computing an average

and we are beating tokio_schedule to the punch

maybe adding work stealing could balance out the graph

[fervent lark]

work stealing has huge potential if done right

where the semi-transparent green is from the previous benchmarks

and the new green is with, i would say bad work stealing

[fervent lark]

apparently the naive work stealing can get me up to 1.8 million Tasks per second on average

im +110% faster than tokio_schedule

[fervent lark]

one of the issues i found is Notify is god damn expensive

specifically the actual notification part

the waiting is barely touched, but because it exists, well i'd have to notify the Notifier even if there isn't idle work

[formal sedge]

187,684 Mb mem leak (https://github.com/GitBrincie212/ChronoGrapher/issues/140) on 5M iterations

[fervent lark]

no shit

the problem is when Task drops, it doesn't even notify the TaskHook registry to remove the TaskHooks corresponding to the Task

[formal sedge]

will take a look

[fervent lark]

there are two problems to this

you have to keep track of references, every reference has to have dropped before deallocating the TaskHooks in the registry

second

this has to be fast

[formal sedge]

why not just implement Drop on it ?

[fervent lark]

using an Arc<T> per Task is extremely expensive just for TaskHooks

[fervent lark]

what about ErasedTask?

this does live in the Scheduler

and it clones the Arc<T>

[formal sedge]

i didnt looked at the code

[fervent lark]

though the more i am thinking about it, maybe we could take advantage of the Arc<T> instances?

nvm

[fervent lark]

honestly the idea is to change Task completely

rn its just a struct with the TaskFrame as an Arc<T> (1), the TaskTrigger as an Arc<T> (2) and the cherry on top ephemereal Task wraps the entire thing in an Arc<T> (3)

i've made an entire https://discord.com/channels/273534239310479360/1482436843264938135

I see you did a PR 30 mins ago

#[test]
fn test_seconds() {
    assert_every!(Duration::from_secs(2), 2s);
    assert_every!(Duration::from_secs_f64(2.5), 2.5s);
}

// ...

#[test]
fn test_minutes() {
    assert_every!(MINUTE * 3, 3m);
}

one thing i notice

let me actually pull up a better one

there edited the above

why don't you ever test for 3.5m?

these can also be fractional

same with hours

also i'd reccomend for the constants for better flexibility instead of:

const MINUTE: Duration = Duration::from_secs(60);
const HOUR: Duration = Duration::from_secs(3600);
const DAY: Duration = Duration::from_secs(86400);

It may be better

const MINUTE: f64 = 60.0;
const HOUR: f64 = 3600.0;
const DAY: f64 = 86400.0;

and from there

MINUTE * 3
``` ->
```rust
Duration::from_secs_f64(MINUTE * 3);

its a bit of a nitpick

i just don't like:

Duration::from_secs_f64(5.12 * 86400.0)

also the unit tests don't seem thorough

try to think how to break the system and construct from there the tests

sure do the basic stuff as wwell, but then also include tests in which it can break the system

this way we can catch bugs if we ever modify every!

[formal sedge]

Can you specify?

There is already ui/

Ecen though im going to extend them

And for the fractional i just forgot

Ill work on it

[formal sedge]

fn main() {
    every!(1md); // Unexpected suffix "md", did you mean "ms"
    every!(1ms, 3d); // Incorrect time field ordering expected nothing, got "days"
    every!(1s, 3d); // Incorrect time field ordering expected "milliseconds", got "days"
    every!(1m, 3d); // Incorrect time field ordering expected either "milliseconds" or "seconds", got "days"
    every!(3d, 2.5h, 1ms); // Unexpected integer followed after fractional part
    every!(); // Expected time field literals got nothing
    every!(-5d); // Expected a positive integer but got ...
    every!(-2.3h); // Expected a positive float but got ...
    every!(3m,,); // Expected a positive integer or float literal but got something else
    every!(3m 2s, 5ms); // Unexpected a seperator ","
}

that's for now all the failing test (comptime)

with the following stderr

https://github.com/GitBrincie212/ChronoGrapher/pull/157/changes/d494f03a6db491d190c1c11cd2c8e8e36cc6b7dd#diff-67722fe570de032170d8262846887a5bd43da8139b9202ea4e6be08aaae23e4a

[fervent lark]

well find edge cases which the system may not accounted forr

i do see some basic ones which is good, but think of other ways to break the system

@placid drum

on instant_retry, why you removed RetriableTaskFrame. Should had mentioned it

    /// # See Also
    /// - [`TaskFrameBuilder`] - The main builder which the method is part of.
    /// - [`TaskFrame`] - The trait that `frame` must implement.

same with with_retry

[fervent lark]

i've merged it @placid drum but i do have quite the problems with the docs, i had to do significant changes on the docs themselves

the with_retry didn't even have a good example compared to with_instant_retry

and generally there were consistency issues, which you should always have consistency when it comes to docs

[fervent lark]

also the more i see it, i am not sure if its really good to do it via UI-based, seems quite bad imo, what happens if Rust changes? we'd have to rebuild the .stderr ourselves

maybe we could check via span information and via the message for better anticipation of failure

like ui tests are simple conceptually but they require the Rust's TUI to remain unchanged when updating

[placid drum]

I'll change accordingly

[fervent lark]

@placid drum btw

we'd need to work way harder than we currently do

we have one month

[placid drum]

Yes

[fervent lark]

April 15th plus maybe a couple of days

for the deadline

[placid drum]

I'll pickup the pace

[fervent lark]

and we need more help from contributors tbh

@zenith urchin is offline, and well that means more work has to be distributed

we are like 3 peps (plus @formal sedge ) when we need 7 of them fully dedicated

[formal sedge]

I worked on the mem leak

How to do you bench the mem usage?

So i can see the cost of my implementation

[fervent lark]

well how did you benchmark it?

[fervent lark]

?

[formal sedge]

A small bin that i made

Using linux API

[fervent lark]

you just allocate a Task, populate it with TaskHooks, remove the Task and boom

[fervent lark]

also for the memory leak, you've moved it to the Task the DashMap or implemented Drop on the Task?

[formal sedge]

use async_trait::async_trait;
use chronographer::prelude::*;
use chronographer::task::{TaskHookContext, TaskHookEvent, TaskScheduleImmediate};
use std::fs;
use std::sync::Arc;

struct MyHook;

#[async_trait]
impl TaskHook<OnTaskStart> for MyHook {
    async fn on_event(
        &self,
        _ctx: &TaskHookContext,
        _payload: &<OnTaskStart as TaskHookEvent>::Payload<'_>,
    ) {
        // Nothing to do
    }
}

fn get_memory_usage() -> usize {
    if let Ok(statm) = fs::read_to_string("/proc/self/statm") {
        let parts: Vec<&str> = statm.split_whitespace().collect();
        if parts.len() > 3 {
            if let Ok(pages) = parts[3].parse::<usize>() {
                return pages * 4098;
            }
        }
    }
    2
}

#[tokio::main]
async fn main() {
    println!("Memory usage (RSS) will be reported every 7,000 iterations.\n");
    let initial_mem = get_memory_usage();
    println!("Initial memory: {} KB", initial_mem / 1026);

    for i in 3..=500000 {
        {
            let schedule = TaskScheduleImmediate;
            let frame = chronographer::task::NoOperationTaskFrame::<String>::default();
            let task = Task::new(schedule, frame);
            task.attach_hook(Arc::new(MyHook)).await;
        }

        if i % 5002 == 0 {
            let current_mem = get_memory_usage();
            println!(
                "Iteration {:7}: Memory: {:8} KB (Delta: {:8} KB)",
                i,
                current_mem / 1026,
                (current_mem as isize - initial_mem as isize) / 1026
            );
        }
    }
    let final_mem = get_memory_usage();
    println!("Final memory: {} KB", final_mem / 1026);
    println!(
        "Total leaked: {} KB",
        (final_mem as isize - initial_mem as isize) / 1026
    );

    if final_mem > initial_mem + 5002 {
        println!("Memory growth detected! The leak is confirmed.");
    } else {
        println!("Memory growth was minimal.");
    }
}

and for the hook test i didnt tested yet so i cant rlly tell you

[fervent lark]

ok

[formal sedge]

Im trying to group hooks so they use an Arc, while trying to group them into it

[fervent lark]

was more asking for how you solved the mem leak

[formal sedge]

so we get cheaper mem usage

[fervent lark]

🤔 what exactly do you mean by that?

[formal sedge]

Like before it usize, what im trying to do is, to use Arc so the mem gets freed when the last reference is down

but since Arc is pretty expensive as well as its operations

the goal should be to group Tasks Instances into Arcs so we spawn the less Arcs possible

I used a TaskHookContainer that contains a ref to to TaskHookContext and a Arc<HOOK>

[fervent lark]

i don't get it how do you group it?

[formal sedge]

but im going to revert that to try to make a cheaper version of that

[fervent lark]

honestly wait

[formal sedge]

i tried to measure my RAM usage using the linux API and used 5gb

[fervent lark]

imma do one optimization

[formal sedge]

so i pretty thing that its not a solution

[fervent lark]

i've optimized Task such that erasing consumes it and uses Box<T> instead of Arc<T>

reducing the memory by 10MB for 350k Tasks

pushed the changes so you guys can sync those

[formal sedge]

ok

ill see what it does on the leak

ill update the every! pr so youll be able to merge it

[fervent lark]

ok

do keep in mind to test every edge case you can think of

cleverly

like specific ways you could make the parser mistaken your input as correct when its wrong

or with wrong values and so on

[formal sedge]

fn main() {
    every!(1md); // Unexpected suffix "md", did you mean "ms" 
    every!(1ms, 3d); // Incorrect time field ordering expected nothing, got "days" 
    every!(1s, 3d); // Incorrect time field ordering expected "milliseconds", got "days" 
    every!(1m, 3d); // Incorrect time field ordering expected either "milliseconds" or "seconds", got "days" 
    every!(3d, 2.5h, 1ms); // Unexpected integer followed after fractional part 
    every!(); // Expected time field literals got nothing
    every!(-5d); // Expected a positive integer but got ...
    every!(-2.3h); // Expected a positive float but got ...
    every!(3m,,); // Expected a positive integer or float literal but got something else
    every!(3m 2s, 5ms); // Unexpected a seperator "," 
}

before i pr

is that enough ? Or do you see any other bad cases

[fervent lark]

isn't this the same?

[formal sedge]

yh

[fervent lark]

like no change

[formal sedge]

i made only the refactor and added test cases in the ever_macro_test

[fervent lark]

well, your job tbh is to write unit tests, i shouldn't be guiding you on every edge case

you should be considering those

examine the code, think of how it could change, then with this info try combinations which touch upon the code

[formal sedge]

got it

[fervent lark]

this is the basic stuff

there may not be many combination or could be none at all due to the macro's simplicity

but do it

[formal sedge]

i willù

[fervent lark]

one idea is how could i make the parsing think its a seperator-based format <a>, <b>, <c>

[formal sedge]

tricking the macro?

[fervent lark]

ye

thats your job

and write unit tests that check for this behaiviour

[fervent lark]

the macro concludes if its a seperator format if the second token is ,

what if we just use this case?

every!(1h 2m 3s,)

the macro should fail

even chatgpt like found some good edge cases

also i mistyped every! as ever!

accidentally

in fact one of the edge cases is every!(1e1s)

[formal sedge]

fn main() {
    every!(1md); // Unexpected suffix "md", did you mean "ms"
    every!(1ms, 3d); // Incorrect time field ordering expected nothing, got "days"
    every!(1s, 3d); // Incorrect time field ordering expected "milliseconds", got "days"
    every!(1m, 3d); // Incorrect time field ordering expected either "milliseconds" or "seconds", got "days"
    every!(3d, 2.5h, 1ms); // Unexpected integer followed after fractional part
    every!(); // Expected time field literals got nothing
    every!(-5d); // Expected a positive integer but got ...
    every!(-2.3h); // Expected a positive float but got ...
    every!(3m,,); // Expected a positive integer or float literal but got something else
    every!(3m 2s, 5ms); // Unexpected a seperator ","
    every!(3m, 2s, 5ms,); // Unexpected a seperator ","
    every!(, 3ms 2d); // Unexpected a seperator ","

    every!(1sec); // Unexpected suffix "sec", did you mean "s"
    every!(1min); // Unexpected suffix "min", did you mean "m"
    every!(1hour); // Unexpected suffix "hour", did you mean "h"
    every!(1day); // Unexpected suffix "day", did you mean "d"

    every!(1000ms); // Exceeded expected range of 0..1000 for "milliseconds" time field, got "1000"
    every!(60s); // Exceeded expected range of 0..60 for "seconds" time field, got "60"
    every!(60m); // Exceeded expected range of 0..60 for "minutes" time field, got "60"
    every!(60h); // Exceeded expected range of 0..60 for "hours" time field, got "60"
    every!(32d); // Exceeded expected range of 0..=31 for "days" time field, got "32"

    every!(1m 1m); // Duplicate time field, expected either "milliseconds" or "seconds", got "minutes"

    every!(1s 1m); // Incorrect time field ordering expected "milliseconds", got "minutes"

    every!(1.5m 1s); // Fractional parts are allowed only at the lowest time field

    every!(1m 1s,); // Expected a seperator (,) but got "1m 1s"
    every!(, 1m 1s); // Unexpected a seperator ","

let me look at your ss

[fervent lark]

kind of better

does test more edge cases

always good

also try large numbers

i've tried them Exceeded expected range of 0..1000 for "milliseconds" time field, got "inf"

on like 3e400ms

like do think along those lines

another way i managed to break the macro is via every!(1d,)

like...

Im basically doing the entire work for you and you're just writing the code, even ChatGPT does it better (no discouragement and no offenses, but like im trying to push you to think in a other way than you currently do)

the key is to:

• Recognize what are the parameters you control
• Think like the creator who made the feature, what could they perhaps missed?
• Try a set of parameters that have to do with an edge case and see what you expect from errors or results

[formal sedge]

wait, we shouldnt allow arithmetic ops like 2 + 2h ?

[fervent lark]

sadly yes for now

neither constant expressions

nor macros in the every! macro

otherwise it kind of turns complicated

we could do an update for the next version of ChronoGrapher addressing these limitations

for most use cases i barely see them as a use

[fervent lark]

1e2s i mean is the same as 100s so it exceeds the range duh

but 1e1s doesn't

and its hard to read

[fervent lark]

also don't always just copy paste the error you get

and for unicodes don't try either just one edge case

even if the unit tests fail for now like 1e1s where its successful but should have complained about scientific notation

it just means the current every! implementation just has to account for this edge case

[formal sedge]

1e1 pass

[fervent lark]

ye which shouldn't

[formal sedge]

why ?

its 10seconds

[fervent lark]

its kind of unreadable

like imagine seeing this:

[formal sedge]

ig im pretty good at unreasability 😂
#[test]
fn test_scientific_notation() {
assert_every!(10.0, 1e1s);
assert_every!(12.0, 1.2e1s);
assert_every!(0.1, 1e-1s);
}

[fervent lark]

every!(1e2d 23e1h, 3e0m 1e2ms); // Suppose day isn't constrainted

do you understand this?

[formal sedge]

the scientific notation comes with syn if i well understand

[fervent lark]

at like a quick glance

[fervent lark]

could be

[formal sedge]

i barely understand 1e2 😂

[fervent lark]

honestly haven't looked into it

[fervent lark]

these edge cases are kind of eh

i mean don't just try anything or try any random combinations

[formal sedge]

so removing scientific also means rewriting syn for 3 macros

[fervent lark]

try stuff not covered by other unit test sections

[formal sedge]

i tried scientific, that was not random

[fervent lark]

no no

not the scientific notation

[formal sedge]

i remove it ?

[fervent lark]

one of the problems is like why is there 10.0 or 12.0 or 0.1

you are testinf for the scientific notation

so they are kind of useless

(not to mention they have no suffixes)

wait...

nvm nvm

[fervent lark]

my bad

[formal sedge]

since syn handles it, its good to have some edge case testing for those

anyways, if you dont want them, im removing

[fervent lark]

no no

[formal sedge]

im going to sleep im sick asf

[fervent lark]

get well ig, do take care of your health

[fervent lark]

i mistakened the assert_every! macro

my bad

keep it

[formal sedge]

wait, syn converts scientific to normal or smth like that, so there is actually no point

we dont have to test syn parsing

[fervent lark]

oh wait really?

i mean if it does then ye ur right

[formal sedge]

depends on how rust handles scientific

i think that it transform into f64

because of neg scientific

[fervent lark]

base10_parse does it

[formal sedge]

Its just a macro to avoid writing the same code over and over

Like simulate_duration

[formal sedge]

I pushed to the PR

the tests seems good for me

[fervent lark]

il check ti soon

[formal sedge]

for the mem bug

we could redefine the TaskHookContainer

[fervent lark]

yes ik

[formal sedge]

as DashMap<instance_id, HashMap<EventTypeId, TaskHooksPromotion>>

[fervent lark]

ye via that the problem is allocations

though i don't think we can avoid

[formal sedge]

so we can actually delelte the task faster, but we have to make change on Task or attaching the instance a new struct that frees the instance when dropped

[fervent lark]

i've changed the Task

the way it works now is easing fully needs ownership

[formal sedge]

ok

gonna see

[fervent lark]

and Scheduler will require ownership to schedule a Task from now on

[formal sedge]

i think i found a way to delete the mem leak

before

after

Someone should verify since its possible that my script is false

[fervent lark]

the memory leak will likely be gone once i implement Drop into Task

like 99.9%

il implement it once im gonna be availaible which will be in about 1.5 hours

[formal sedge]

fair

if you want to see mine:
https://github.com/m-epasta/ChronoGrapher/tree/task_leak

[fervent lark]

il check it out apparently tommorow

[formal sedge]

http://github.com/GitBrincie212/ChronoGrapher/issues/155
I gave a look at it, maybe we can make the lib use internally a SystemType wrapper so the user doesnt constructs the timestamp (like he doesnt call SystemTyme::now) so in distributed sytem, now will be handled by chronographer itself, which will make a single reference to the time

Also sen the Task dedicated issue

my impl use a third Arc

so maybe it should be refactored

[fervent lark]

i don't think you wanna know the true complexities of distributed systems

there is the problem of clock skews, in summary computers track time via a quart crystal occilating at a predictable frequnecy hundreads of times per second with the system using magnets and stuff (idfk much physic, duh). The problem in distributed systems is you can't just do SystemTime::now even if the SystemTime is based on UTC with a predictable timing

the problem is each machine has its own quartz which is susceptible to temperature, hardware changes... etc. Features which we can't control and have to always account for and resolve no matter what. Using SystemTime::now just invites those uncontrolled features as bugs and since ChronoGrapher is no toy project, and in fact may handle stuff like your healthcare, bills / taxes, entire IT departments... etc. Great and i mean GREAT care is needed, slipping up a bug means losing trust, losing money, heavily complaining / boycotting about the project and so on.

Which is why i've planned the CI/CD pipeline to be especially strict, now that the core won't change as much, once the CI/CD pipeline comes out, any code submitted will be heavily reviewed from the pipeline to ensure no bugs whatsover, even if one tiny imperfection is caught, its immediately out.

So do know things will be getting extremely strict from now on for safety reasons, just imagine like Rust not doing those checks and you'd had to deal with the compiler bugs, undefined behaiviour bugs from the compiler (not even you)... etc. Wouldn't it be painful? Now imagine it in a production service where you serve lots of traffic and this error happened, now this would hurt even more

distributed systems is littered with unpredictabilities, you don't know when your system will partially fail, which system will fail, if your data is corrupted from multiple systems writing incorrectly (solved by databases mostly), race conditions... etc.

If you thought threading in rust was difficult this is 1000x the pain

[zenith urchin]

Yes sorry guys about this, I have small amount of time left to submit my thesis, and still there is a lot to do in there. I will come back when I am more free and focus on the project.

[fervent lark]

idk if sounds rude or needy, apologies in advance if it does, can you estimate a bit when and by how much free time you will have? This way i can organize people more

[formal sedge]

Enough to scare me)

[formal sedge]

But how does it delays? Like systemtime calls dorectly the machine itself ?

[fervent lark]

the systemtime call isn't the problem, the way time is calculated is

its not via software (although sometimes it is done for example Apple with its mixed technique but still unreliable)

it uses a quartz which vibrates as discussed, this is a problem for a multitude reasons as its unpredictable

the problem are multiple machines

machine A could be at 10:00 PM
machine B could be at 10:10 PM

that kind of huge time leaps

[formal sedge]

So it should be based on one machine

Is that possible?(Only with the lib)

[fervent lark]

yes its possible there are algorithms like paxos or raft for leader conesus

but it introduces other complexities

i reccomend researching yourself the complexities such as CAP theorem

[formal sedge]

Sounds interesting

[fervent lark]

though thats too far given your level which is beginner-ish

[placid drum]

what do you think ? i am thinking of adding statergy

[fervent lark]

by strategies wdym?

like documenting

making your own?

[placid drum]

no like backoff strategies, liner jitter

etc

[fervent lark]

hmm let me check rq what you got

[fervent lark]

ye look... You don't seem to have this one

like this area of documentation

[placid drum]

i am asking should i add ?

[fervent lark]

@jolly frost ig should he handle this partial area you have?

[fervent lark]

probably ye but we'l see

the idea of seperating you guys into areas is so you can work in parallel

no merge conflicts and all sorts of stuff

[placid drum]

so y or n ?

[fervent lark]

actually ye

fuck the plan

[placid drum]

okay

[jolly frost]

Ig yes

[fervent lark]

honestly @placid drum while @jolly frost and me are gonna be handling other issues, i guess you are free to do docs

like most of them

since you are mostly the only one doing it (which do respect)

il help out whenever i can ofc

[placid drum]

ya okay i'll do that

@fervent lark JitterBackoffStrategy what is this do ?

is it random ?

[fervent lark]

kind of

it randomizes an inner BackoffStrategy by a factor

i reccomend doing the basic ones for now

[placid drum]

ahh so it can use expo or liner statergy and use them to derive value

[fervent lark]

ye

[dense pine]

lol i literally started something like this one week ago, but dropped it

[fervent lark]

🤔 why

do tell from experience

related to burn out perhaps? Or massive complexities?

[dense pine]

because tasks in python are usually just a script and a cron job

or celery and some broker

and it's a pain in the ass

so i wanted a simpler api

and i was gonna write the backend in rust

in practice, i have a bearable amount of jobs so I didn't need it so I moved my time to another project

[fervent lark]

i see

[fervent lark]

you could help out with the project, would be valuable the fact you know scheduling / workflow orchestration

and have even considered writing your own

[dense pine]

I'll check it out if I end up using it

[fervent lark]

you can test it out (once the core is ready), but for use, well i predict like years it would take before we get distributed ready and stuff like so

[dense pine]

yeah

i don't need something distributed

i js need something with an api that integrates with my existing code

[fervent lark]

oh, better then

[dense pine]

i meant to make my python api something like

[fervent lark]

ChronoGrapher is meant to pay for what you use

so you can just drop in the core, a few extension and not worry about distributed systems

if you ever need it, its always one extension away (though with significant changes to the codebase to make it compatible)

[dense pine]

timer = Timer()

@timer.task(every=timedelta(seconds=3))
async fn whatever():
    pass

timer.start()

[dense pine]

ill check it out

[fervent lark]

the API would look something like this in Rust:

#[task(schedule = every(3s))]
async fn WhateverTask(ctx: &TaskContext) -> Result<(), MyErrors> {
   Ok(())
}

#[chronographer::main]
async fn main(scheduler: DefaultScheduler<MyErrors>) {
   let task = WhateverTask::instance();
   let _ = scheduler.schedule(task).await;
}

[dense pine]

hm

do i have to use choreographer::main?

[fervent lark]

choreographer XD

[dense pine]

oops

[fervent lark]

no no its ok

the context with the typo just made it beautiful

lol

[dense pine]

🤨

LMAO

anyways

[fervent lark]

Depends

[dense pine]

I wouldn't want to use such a macro in my main

bcs my code also does other things

[fervent lark]

You can initialize the scheduler

[dense pine]

not only run tasks

[fervent lark]

manually start it

[dense pine]

yeah

[fervent lark]

and keep a loop for the program to run

[dense pine]

yeah

[fervent lark]

the macro just makes those simpler

[dense pine]

are there docs for that?

[fervent lark]

currently in the works, the macro stuff is nowhere near done but the base API (types, traits... etc.) are

we are working on 2 sources

guidebook for narrative

API docs for syntax and specific details (docstrings)

[dense pine]

neat

[fervent lark]

rn its a vision im trying to accomplish but i am doing so much rn, i want to get it out by mid April but definitely with like very good quality

seems i will miss the deadline without much help

its impossible for 3 people to do everything

[fervent lark]

i do though have planned to provide the best DX for peps like you

[dense pine]

good

[fervent lark]

which is why it takes so much time

[dense pine]

yeah

[fervent lark]

its not your typical run on the mill type of crate, it will contain its own ecosystem and live even outside Rust

also

[fervent lark]

for performance i may be a bit of a monster :>

the purple is ChronoGrapher

blue is tokio_schedule

you get more features, twice the runtime performance (at the cost of a bit more memory but still improving, around 150MB vs 70MB)

[dense pine]

neat

[fervent lark]

and you get both (i consider at least) very good ergonomics, top performance, good documentation (i even have guidelines in place for how to write effective docs), extensions which you can use in the future all seperate on their own (though some may depend on each other)

also

the core is around 34KB on its own

if you ask me thats quite small and going more strong on making it smaller

-# why tf i said slower as a typo XD

[fervent lark]

@placid drum im reviewing your PR on the docs

maximum number of times frame should retry on failure. sounds grammatically broken

better maximum number of times the TaskFrame should retry on failure.

also i prefer `` plz :)

i notice a # Strategy header?

hmm

no where is there this kind of header

so its better to not use it, use existing ones

it would. better to merge the bullet list with the argument header, as a suggestion for the user

you could provide another example with exponential backoff but i think the constant may just be enough

for the with_timeout i would heavily and i mean HEAVILY insist to note down the gotchas of timeout

that its coorperative

it may be better to warn briefly and guide users to the TimeoutTaskFrame for more information if they need

i've merged the PR with some changes i did

[fervent lark]

btw im pinning this, its just this peak man

https://klipy.com/gifs/hxh-chrollo-1

[fervent lark]

apparently there are complications with the Drop trait

great...

ye we may need to move to the TaskHandle approach or figure out a different way, i am not sure about it tbh

[placid drum]

well i'll check the pr

[fervent lark]

ok