Introduction

I apologize for the under-construction vibe that this site gives off. I am building it incrementally, and even I don't know how it will end up looking.

Recently, while writing code for this site, I decided to switch from writing unit tests after writing implementation to using Test Driven Development. So now I have unit tests that were written afterwards and unit tests that were written before implementation. And the tests look very different.

First differences

Here are the tests for a module I wrote earlier without TDD.

    #[tokio::test]
    async fn test() {
        drop(env_logger::try_init());

        log::info!("Creating test database");
        let pool = SERVER.connect().await;
        let mut mgr = mgt::Manager::new(); 
        mgr.add(mod_info());
        mgr.add(users::mod_info());
        mgr.migrate(&pool).await
            .expect("Migration failed");

        log::info!("Creating user user1");
        let user1 = users::create(&pool, "user1").await
            .expect("Unable to create user");

        log::info!("Creating new session");
        let session = create(&pool, &user1.uid, 3).await
            .expect("Unable to create new session");
        assert_eq!(session.uid, user1.uid);

        log::info!("Checking for active session");
        let chk_value = get(&pool, &session.sessionid).await
            .expect("Unable to get session via sessionid");
        assert_eq!(chk_value, session);

        log::info!("Testing session deletion");
        delete(&pool, &session.sessionid).await
            .expect("Unable to delete session");
        let chk_value = get(&pool, &session.sessionid).await;
        assert!(matches!(chk_value, Err(Error::RowNotFound)));

        log::info!("Testing foreign key");
        users::delete(&pool, &user1.uid).await
            .expect("Unable to delete user");
        let chk_value = create(&pool, &user1.uid, 3).await;
        assert!(matches!(chk_value, Err(Error::Database(_))));

        log::info!("Testing session pruning");
        let user1 = users::create(&pool, "user1").await
            .expect("Unable to create user");
        for _ in 0..5 {
            create(&pool, &user1.uid, 3).await
                .expect("Unable to create new session");
        }
        log::info!("Assertions for session pruning");
        let chk_value = list(&pool).await.unwrap();
        assert_eq!(chk_value.len(), 4);
    }

It is hard to figure out in a glance what the test is checking for. It is also not obvious what the module does from just the tests. The code is written in Rust, and if you did not know Rust, the logging statements might be the only thing that tells you what the test is doing.

Here are the tests for the first module I wrote with TDD.

    #[tokio::test]
    async fn test_get() {
        let (pool, user1, _) = init().await;
        let form = test_blog();

        let id = create(&pool, &user1.uid, &form).await
            .expect("create");
        let blog = get(&pool, id, Access::All).await
            .expect("get");
        assert_eq!(&blog.title, &form.title);
        assert_eq!(&blog.body, &form.body);
        assert_eq!(&blog.author_name, &user1.username);
    }

    #[tokio::test]
    async fn test_get_others() {
        let (pool, user1, user2) = init().await;
        let form = test_blog();

        let id = create(&pool, &user1.uid, &form).await
            .expect("create");
        let result = get(&pool, id, Access::Mine(user2.uid.clone())).await;
        assert!(matches!(result, Err(sqlx::Error::RowNotFound)));
    }

    #[tokio::test]
    async fn test_modify() {
        let (pool, user1, _) = init().await;
        let mut form = test_blog();

        let id = create(&pool, &user1.uid, &form).await
            .expect("create");

        form.body = "Modified body".into();
        modify(&pool, id, &form, Access::All).await
            .expect("modify");
        let blog = get(&pool, 1, Access::All).await
            .expect("get");
        assert_eq!(&blog.body, &form.body);
    }

    #[tokio::test]
    async fn test_modify_others() {
        let (pool, user1, user2) = init().await;
        let mut form = test_blog();

        let id = create(&pool, &user1.uid, &form).await
            .expect("create");

        form.body = "Modified body".into();
        let access = Access::Mine(user2.uid.clone());
        let result = modify(&pool, id, &form, access).await;
        assert!(matches!(result, Err(sqlx::Error::RowNotFound)));
    }

    #[tokio::test]
    async fn test_index() {
        let (pool, user1, _) = init().await;
        let form = test_blog();

        create(&pool, &user1.uid, &form).await
            .expect("create");

        let idx = index(&pool, Access::All).await
            .expect("index");
        assert_eq!(idx.len(), 1);
        let idx = index(&pool, Access::Mine(user1.uid.clone())).await
            .expect("index");
        assert_eq!(idx.len(), 1);
    }

    #[tokio::test]
    async fn test_index_others() {
        let (pool, user1, user2) = init().await;
        let form = test_blog();

        create(&pool, &user1.uid, &form).await
            .expect("create");

        let idx = index(&pool, Access::Mine(user2.uid.clone())).await
            .expect("index");
        assert_eq!(idx.len(), 0);
    }

It is much more obvious that the module handles displaying and editing blog posts on this site. It is also obvious that it cannot delete blog posts yet.

However, if I look at the names of the tests, I see a cartesian product.

I do not like it. To me it looks like there is redundancy involved, and a change in the implementation might require multiple tests to be updated.

It is possible that this is actually good code and I haven't realized the merits yet. It is also possible that this is actually bad code and I did TDD wrong.

More complex modules

So far I do not have experience in using TDD for more complex code. In the past, I have written tests that involve loops or tree-generation algorithms to check each possible case. I wonder how such tests will work with TDD involved. But then I also lack any experience in how to grade tests written with TDD. I guess only time and experience will tell.

What is this "life" thing are you talking about?

Are you one of those "I like to travel" people?

"Singing"?

"Dancing"?

Popular media?

No, you explored nothing. You just copied everyone else.

Now this is not an inherently bad choice to make, but please stop telling others to "get a life" or "touch grass". They are having fun. Or until you told them what they found fun is not actually fun.

I don't know how many tourists are 'tourists', who got gaslit into tourism, leaving their true interests undiscovered.

It is good to have life outside work, but are you actually having life outside work?

The probability of "I like to travel", and gaslighting

Out of all people,

• How many people truely enjoy tourism?
• How many people only 'like' tourism?
• How many people don't like tourism but still travel just to get along?
• How many just don't?

It seems like basically everyone likes tourism.

Statistically speaking, how is this possible? There are so many niches to dive into. How is it always tourism? How is it always singing and dancing?

I once had a group of men hold me down, insisting that they wouldn't let me go until I sing my 'favorite song'. They genuinely had no ill will, they just couldn't accept the fact that songs aren't my thing.

More than 10 people; cannot accept that someone may not like songs.

This probably amounts to 93.9 percent of the entire population. How can this be possible?

Tribal mechanisms

Turns out, the cake is real. Life, on the other hand, is not.

How do most people live?

• Wake up
• The usual morning routine. By usual I mean the sedentary one.
• Go to work.
• Return from work.
• Watch TV, shorts, movies, serials, stuff
• Sleep
• Repeat

Is this a problem? No. But if you act like you are inherently superior than anyone who wants to be different, then we have a problem.

There are so many niches to explore, that evaluating all of them is basically impossible in a human timescale. Convergence on the 'one true thing' is impossible. I think, if people truely chose what they want to do, everyone would choose something different.

So why popular media is ... popular? Why most people don't workout? How does everyone like singing and dancing?

We are social creatures. We want to fit in our tribe. This means, enjoying the same media as those around us ... hence popular media, visiting the same places and taking pictures 'gotta catch them all' style except for your own breath ... hence tourism, partying hard... hence fast food and booze, and mediocre singing and dancing by those who 'love' it ... just like everyone else.

If you truely love it, why aren't you good at it?

We don't need to be tribal.

We can all be different.

And we can all embrace each other for being different.

And we can stop bullying others for being different.

And we can all trust each other despite being different.

If you are different, to me it means that you are curious. You explored around what can be done, instead of simply copying others. You have questioned what is considered as normal, regardless of consequences.

And I am interested. I want to listen to you. Please tell me what makes you different.

Appendix 1

Let:

x: the fraction of people that accept that someone may not like songs.
E: accepts that someone may not like songs
10 random samples taken, 10 times E is false.
To find: x

Let f(x) : probability distribution of x Lets assume perfect uncertainity, so f(x) is 1.

Let f_1(x) be probability density of x when sample 1 is false

\begin{align*}
f_1(x) &= \frac{f(x)  (1 - x)}
         {\int\limits_{0}^{1} f(y) (1 - y) dy}
\\
       &=   \frac{1-x} 
           {y - \frac{y^2}{2}}
         \bigg|_0^1
\\
       &= \frac{1-x}{0.5}
\\
       &= 2(1-x)
\end{align*}

so when first sample is false, x = 0 is more likely than x = 0.5 and x = 1 cannot happen.

\begin{align*}

f_{10}(x) &= \frac{(1-x)^{10}}
               {\int\limits_{0}^{1} (1-y)^{10}dy} \\
\text{Substitute:} \\
           z             &= 1-y   \\
           \frac{dz}{dy} &= -1    \\
             dy          &= -dz   \\
\text{After substitution:} \\
f_{10}(x) &= \frac{(1-x)^{10}}
                  {\int\limits_{1}^{0} z^{10} (-dz)} \\
          &= \frac{(1-x)^{10}}
                  {-\frac{z^{11}}{11}\bigg|_{1}^{0}} \\
          &= \frac{(1-x)^{10}}
                  {\frac{1}{11}} \\
          &= 11 (1-x)^{10}

\end{align*}

Let w be the median of the probability distribution.

\begin{aligned}
     \int\limits_{0}^{w} 11(1-x)^{10} dx 
       &= \int\limits_{w}^{1} 11(1-x)^{10} dx \\
=>   \int\limits_{0}^{w} (1-x)^{10} dx 
       &= \int\limits_{w}^{1} (1-x)^{10} dx \\
\\
\text{Substitute:} \\
                 y &= 1-x \\
             dy/dx &= -1 => dx = -dy \\
       At x = 0: y &= 1 \\
       At x = w: y &= 1-w \\
       At x = 1: y &= 0  \\
\text{After substitution:} \\
\int\limits_{1}^{1-w} y^{10} dy &= \int\limits_{1-w}^{1} y^{10} dy \\
=> y^{11} \bigg|_{1}^{1-w} &= y^{11} \bigg|_{1-w}^{0} \\
=> (1-w)^{11} - 1 &= -(1-w)^{11} \\
=> 2(1-w)^{11} &= 1 \\
=> 1-w &= (\frac{1}{2})^{\frac{1}{11}} \\
=> w &= 1 - [ (\frac{1}{2})^{\frac{1}{11}} ] \\
=> w &~=  0.061
\end{aligned}

6.1% of population accepts that someone may not like songs. Or, 93.9% of the population cannot accept that someone may not like songs.

Accidentally testing a private API

In my last article, I talked about using Test Driven Development while writing code for this site.

So I wanted to measure how well this blog is performing (speed and efficiency stuff, not popularity). The idea is simple,

1. The server maintains atomic integers representing performance counters.
2. I insert code at measurement points to increment the performance counters.
3. Every 5 minutes, the server takes a snapshot of the performance counters, and reset the counters to 0.
4. The server stores the snapshot into the database

Measurement code (runs every request)
   |
   |  Atomic increment  
   V
Performance counters
   | 
   |  Atomic replace with 0
   V
Sampler (runs every 5 minutes in an hour on wall clock, beginning at 00)
   |
   |  INSERT INTO pm_counters VALUES (...);
   V
Database

One of the stated benefits of TDD is that it helps you improve the design of your code. I did not talk about code, because the problem was trivial. I already had design in mind, so I would write the same code regardless of the method used.

This time, it is different. I was going to write sampler code, and I had no design in mind. The code needs to trigger at multiples of 5 minutes on a wall clock, so a simple 5 minute sleep loop isn't gonna cut it.

Ah yes, a clock. Abstracting one would be interesting (read: difficult). Oh I know, why don't I just tell the implementation what the time is, and the implementation should return the time when to trigger it again.

This is the first attempt at the tests:

#[cfg(test)]
mod test {
...

    #[test]
    fn next_trigger_from_arbitrary() {
        let mut sampler = init(rfc3339utc("2000-01-01T11:59:17")); //< We tell it the current time
        sampler.trigger();
        assert_eq!(sampler.trig_time, rfc3339utc("2000-01-01T12:00:00")); //< We check the next trigger time
        assert_eq!(sampler.storage.len(), 0);
    }

    #[test]
    fn next_trigger_from_trigger() {
        let mut sampler = init(rfc3339utc("2000-01-01T12:00:00"));
        sampler.trigger();
        assert_eq!(sampler.trig_time, rfc3339utc("2000-01-01T12:05:00"));
        assert_eq!(sampler.storage.len(), 1);
    }
}

This is the code that passes the tests:

pub struct Sampler {
    trig_time: DateTime<Utc>, //< Next wakeup time
    duration_s: u32, //< Duration between triggering, in seconds (upto 3600)
    counters: Arc<Counters>,
    storage: Vec<Row>, 
}

impl Sampler {
    pub fn trigger(&mut self) {
        let time = self.trig_time.time();
        let hour = time.hour();
        let minute = time.minute();
        let second = time.second();

        //Only retain accuracy upto seconds. 
        let round_time = self.trig_time.date_naive()
            .and_hms_opt(hour, minute, second)
            .expect("Unable to round off sub-second part");
        self.trig_time = DateTime::from_utc(round_time, Utc);

        let seconds_in_hour = second + (minute * 60);
        let extra_seconds = seconds_in_hour % self.duration_s; //5min

        if extra_seconds == 0 {
            //Sample statistics
            let stats = self.counters.take();
            self.storage.push(Row {
                end_time: self.trig_time, 
                counters: stats,
            });
        }

        //Save the next trigger time. 
        self.trig_time = self.trig_time
            .add(Duration::seconds((self.duration_s - extra_seconds) as i64));
    }
}

And this is the untested function I ended up writing to drive it.

pub async fn sampler_task(
    counters: Arc<Counters>, 
    duration_s: u32,
    retain_days: u32,
    pool: PgPool
) {
    let mut sampler = Sampler {
        trig_time: Utc::now(),
        counters,
        duration_s,
        storage: Vec::new(),
    };

    sampler.trigger();
    loop {
        //Cleanup
        let retain_time = Utc::now() - Duration::days(retain_days as i64);
        if let Err(e) = model::cleanup(&pool, retain_time).await {
            log::warn!("Unable to perform cleanup: {}", e);
        }

        //sleep till the trigger time
        let duration = match (sampler.trig_time - Utc::now()).to_std() {
            Ok(value) => value,
            Err(_) => {
                log::warn!("Lag? trigger time is {}", sampler.trig_time);
                sampler.trig_time = Utc::now();
                sampler.trigger();
                (sampler.trig_time - Utc::now()).to_std()
                    .expect("Unable to figure out sleep time")
            }
        };

        tokio::time::sleep(duration).await;
        sampler.trigger();

        //Insert into database
        for row in sampler.storage.drain(0..) {
            if let Err(e) = model::insert(&pool, &row).await {
                log::warn!("Unable to insert into database: {}", e);
            }
        }
    }
}

What? Code under test is smaller than the untested driver code. I ended up only calling the untested function from outside the module; thus all the code I wrote tests against is effectively private API.

When I initially read about TDD, I thought TDD makes it hard to build bad designs. This is just a quick and easy easy screw-up.

The large, untested function bothers me additionally. Ideally, we'd have integration tests to guard code like this, but let's be real... we deal with legacy code where majority of code has dodgy unit tests, if they exist at all. We'd be lucky to have a working CI setup that can run integration tests. It would be nicer if our unit tests could test more, even if ideally it shouldn't.

I made a note to myself, sometime later I'll try again from scratch.

Further progress

Since past few months, I have been writing as much code as I can using TDD.

At my day job, I was urgently asked to write a small service in Go. I did not know Go at that time. Multiple interfaces I had to use were not determined at that time, and solution to the problem it needed to solve was also not known. I cannot go into the details, but imagine a farmer handed you a buffalo one morning and asked you to take it to a faraway place. As soon as the farmer left, the buffalo starts acting erratically. You try out everything you know; you ask your friends, but the buffalo just wouldn't do what you want it to do. Eventually the farmer himself arrived, but he too cannot control the buffalo. It is an absolute chaos.

Over one month, the service code changed so much, there was almost nothing left in common compared to my initial implementation done in the first week. But throughout the time, I was able to drastically change behavior in less than a day... which I had to do at-least a dozen times.

I have made a few notes,

• TDD makes it significantly easier to learn new programming languages. It helped me quite a bit while learning Go and enabled me to start writing production code under a short notice. I think continuing to practice TDD is worth it for this benefit alone.
• Hardest part of TDD is starting to write a new code while also not having any reference for how its interface should look like. Conversely, modifying an existing module with existing tests is the easiest.
• When I write a new module, initially its quality is the poorest. The more I modify the module, the easier it becomes to change. Without TDD, I have to rely on experience to ensure that code remains easy to change throughout all modifications.
• The signs that TDD sends you about your design is not always in the form of tests being hard to write.
• Tests written after writing code have 70-80% coverage. With TDD, code has almost 100% coverage, all without trying to chase coverage.
• Code written with TDD is initially more complex than needed. The additional complexity is only justified when adding more functionality and then TDD takes the lead. TDD does not make sense for temporary/disposable code.
• We do not manually call the functions that represent the test cases, so these function names are as good as comments. We can go nuts on naming the tests, and we probably should. It is possible to speed up troubleshooting by naming the tests properly, though I have yet to learn how to properly take advantage of this. This leads neatly to the next note.
• A significant reduction in printf-debugging. The better the failing test point towards where I messed up, the less likely I am to add log statements, use a debugger, etc. Before using TDD, my tests routinely had logging statements, now they don't.
• Stress reduction. Those moments where you put all your brains into figuring out how to write this code are all gone, now I develop each module incrementally, adding one requirement fragment at a time.

Attempt 2 at performance monitoring

With some experience gained, I set out to write the performance monitoring code again.

pub trait Clock {
    fn now(&self) -> DateTime<Utc>;
    fn sleep(&mut self, duration: Duration) -> BoxFuture<'_, ()>;
}

pub trait Sampler {
    fn reset(&mut self) -> BoxFuture<'_, ()>;
    fn sample(&mut self, end_time: DateTime<Utc>) -> BoxFuture<'_, ()>;
}

pub async fn sampling_task(
    settings: Settings,
    mut clock: impl Clock,  //< Sleep and getting current time is handled via a trait.
    mut sampler: impl Sampler, //< It turns out that the code relevant for
                               //< timing does not actually need to handle
                               //< the counters directly.
) {
    let interval = Duration::seconds(settings.interval_s as i64);
    let tolerance = Duration::seconds(settings.tolerance_s as i64);

    let mut target = Utc.timestamp_opt(0, 0).unwrap();
    let mut store_row = false;
    loop {
        let now = clock.now();
        let diff = target - now;
        if diff < -tolerance || diff > (interval + tolerance) {
            store_row = false;
            target = next_target(now, settings.interval_s);
        } else if diff > tolerance {
            clock.sleep(diff).await;
        } else {
            if store_row {
                sampler.sample(target).await;
            } else {
                sampler.reset().await;
            }
            store_row = true;

            target += interval;
        }
    }
}

fn next_target(time: DateTime<Utc>, interval_s: u32) -> DateTime<Utc> {
    let hour = time.hour();
    let minute = time.minute();
    let second = time.second();

    let seconds_in_hour = second + (minute * 60);
    let extra_seconds = seconds_in_hour % interval_s; //5min
    let sleep_time = interval_s - extra_seconds;

    DateTime::from_utc(
        time.date_naive()
            .and_hms_opt(hour, minute, second)
            .expect("Unable to round off sub-second part"),
        Utc
    ) + Duration::seconds(sleep_time as i64)
}

Turns out that the sampler does not need to handle the atomic integers directly either.

pub trait Delegate: 'static + Send {
    fn reset(&mut self);
    fn sample(&mut self, stats: &mut Statistics);
}

...
impl<ErrorHandler> Sampler for SamplerImpl<ErrorHandler>
where ErrorHandler: Fn(sqlx::Error) + Send {
    fn reset(&mut self) -> BoxFuture<'_, ()> {
        Box::pin(async move {
            for x in &mut self.delegates {
                x.reset();
            }
        })
    }

    fn sample(&mut self, end_time: DateTime<Utc>) -> BoxFuture<'_, ()> {
        Box::pin(async move {
            let mut stats = Statistics::default();
            for x in &mut self.delegates {
                x.sample(&mut stats);
            }
            let row = Row {
                end_time,
                counters: stats,
            };
            if let Err(e) = model::insert(&self.pool, &row).await {
                (self.error_handler)(e);
            }
        })
    }
}

Such a design allowed for more sources of performance data. I could use anything that fit the Delegate trait. e.g I added some statistics from /proc

impl Delegate for Procfs {
    fn sample(&mut self, stats: &mut Statistics) {
        let res : Result<(), String> = (|| {
            let proc_self_stat = procfs::process::Process::myself()
                .map_err(|e| format!("Cannot read /proc/self: {}", e))?
                .stat()
                .map_err(|e| format!("Cannot read /proc/self/stat: {}", e))?;

            let load_average = procfs::LoadAverage::new()
                .map_err(|e| format!("Cannot read /proc/loadavg: {}", e))?;

            stats.rss = Some(((proc_self_stat.rss * procfs::page_size())
                              /1024) as i64);
            stats.load_avg = Some((load_average.five * 1000.0) as i64);
            stats.utime = Some((((proc_self_stat.utime - self.cur_utime) * 1000)
                               / procfs::ticks_per_second()) as i64);
            stats.stime = Some((((proc_self_stat.stime - self.cur_stime) * 1000)
                               / procfs::ticks_per_second()) as i64);

            self.cur_utime = proc_self_stat.utime;
            self.cur_stime = proc_self_stat.stime;
            Ok(())
        })();
        if let Err(msg) = res {
            log::warn!("{}", msg);
        }
    }
    fn reset(&mut self) {
        self.sample(&mut Statistics::default());
    }
}

But how further can we push that method?

Design

The goal, in case it wasn't clear yet, is to write a Testcontainers-like library that works well with Podman. Testcontainers is a library that starts your test dependencies in a container and stop them after you are done using them.

Starting a container is easy, stopping them is the hard part.

Our new library will replace the entrypoint of the container with our own, and use that entrypoint to terminate the container from within. I am calling the new library Disposables.

Given that we have our own entrypoint inside the container, there is a lot more we can do than just terminate the container after use.

The entrypoint can do more

Testcontainers library is written in many different programming languages. All features of Testcontainers need to be re-implemented in each programming language. We however can take advantage of running our own entrypoint inside the container, and add feature implementation in the entrypoint. This way a lot of implementation only needs to be done once, instead of re-implementing them in each programming language.

For example, waiting for containers to become ready is entirely done within the entrypoint, and only readiness notification is sent back to the test suite.

I have some future plans to implement some Testcontainers features, like Testcontainers modules. I think I only have to write the implementation once and only write the API for each programming language, overall I predict 'modules' will be significantly less work to support.

Conclusion

You can try out the library now at https://github.com/akashrawal/disposables. Right now only Java and Rust languages are available, as I am basically only developing based on my use cases.

I have tested Disposables with Docker daemon configured with user namespaces and with SELinux. Where Testcontainers has issues, Disposables just works, no configuration needed.

user@localhost:~/selinux-test$ export DISPOSABLES_ENGINE=docker
user@localhost:~/selinux-test$ cargo test                                                             
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.02s
     Running unittests src/main.rs (target/debug/deps/selinux_test-a03052e3d408ceef)

running 1 test
test test::nginx ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.86s

user@localhost:~/selinux-test$ export DISPOSABLES_ENGINE=podman
user@localhost:~/selinux-test$ cargo test
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.02s
     Running unittests src/main.rs (target/debug/deps/selinux_test-a03052e3d408ceef)

running 1 test
test test::nginx ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.73s

user@localhost:~/selinux-test$ sestatus 
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33

Changes are inevitable.

Your software works well today. Tomorrow your customer might ask for a new feature. Security vulnerabilities may be discovered in your software or libraries and frameworks your software uses. Your software may need to adapt to newer standards or interoperate with other software built recently. Or your management is asking you to increase test coverage.

Either way, I have seen simple changes being delayed by days or weeks, just because the engineers don't have courage to change the code. It is fruitless to be afraid however.

There will be issues.

Legacy code often does not have tests to provide a safety net. The code can fail in ways you might not foresee. You'll uncover bugs that have laid dormant for years. When change is necessary, be prepared for these outcomes.

Recognize the patterns.

Have respect for those who wrote the code. They were limited by what they knew at that time and did their best. So there will be patterns. However bad they are today, those patterns were the 'good practices' they knew at that time. Learn to recognize them.

Setup a fast build system.

This will help you change the code easier by reducing your code/test cycle time. If you learnt to recognize the patterns, this might be easier than you think.

Prioritize

Always improve the code that you changed in some way. Focus more on improving code that is changed more frequently. Refactoring the code that has no business reason to change is often fruitless, so channel your energy elsewhere.

Testing

Use automated testing to reduce the likelihood of adding defects. Add approval tests to avoid changing behavior of existing code. You might still break things in brand-new ways, so be prepared.

There will be payoffs.

You'll be the one who can fix complex issues in the codebase. You'll lay the groundwork for setting up automated testing that catches real bugs. You'll enable the rest of the team to work more quickly and deliver sooner.

However since 2022, Deno is trying to imitate Node more and more, and this is destroying Deno's ecosystem and undermining its own goal.

Users' Perspective

When Deno was announced in 2020, Deno was its own thing. Deno bet hard on ESM, re-used web APIs and metas wherever possible, pushed for URL imports instead of node_modules, supported executing typescript files without tsx or tsconfig.json and so on.

"If Deno implemented Node APIs and tried to imitate Node and NPM ways of doing things, existing libraries and frameworks written using Node will automatically work in Deno and thus adopting Deno will be easier." I don't know who said this, but someone must have said this.

What has happened instead, is that Deno trying to imitate Node has disincentivized formation of any practical ecosystem for Deno, while the existing libraries and frameworks are unreliable when used with Deno.

I tried using Next.js via Deno some time back, and Next.js dev server crashed when Turbopack is enabled. There is a workaround, so for the time being that issue is solved. But today there is another issue, type checking (and LSP) for JSX is broken.

This is my experience with using Node libraries with Deno. Every hour of work is accompanied with another hour (sometimes more) of troubleshooting the libraries themselves.

I think this is the consequence of trying to imitate something you are not. Deno is trying to be compatible with Node. but there are gaps in the said compatibility. I think achieving compatibility with Node is hard, and the gaps in compatibility will stay for a long time.

For example, at the time of writing, FileHandle.readLines is not implemented in Deno.

import fs from 'node:fs/promises';

const hd = await fs.open(Deno.args[0]);
for await (const line of hd.readLines()) {
	console.log("Line: ", line);
}

The above script crashes despite having no issues with Typescript.

$ deno check test.ts
Check file://path/to/test.ts
$ deno run -R test.ts input.txt
error: Uncaught (in promise) TypeError: hd.readLines(...) is not a function or its return value is not async iterable
for await (const line of hd.readLines()) {
                            ^
    at file://path/to/test.ts:4:29
$

Using NPM libraries is also typically accompanied with a complete disregard for Deno's security features. You just end up running deno with -A all the time.

Library devs' Perspective

Deno 1.0 is released, and library devs are excited to join the ecosystem. Projects like drollup, denodb, drizzle-deno are started,

But then Deno announces Node and NPM compatibility and all that momentum is gone.

Now, it seems like Deno's practical ecosystem is limited to first party libraries like @std and Fresh, libraries on JSR, and a small subset of libaries on NPM that works on Deno.

If you look at the situation from library or framework dev's perspective, it all seems reasonable. Most of them are not new to Javascript; they are much more familiar with Node than with Deno.

When Deno is announced, some of them might want to contribute to Deno's ecosystem. But then Deno announces Node and NPM compatibility, and now there is not enough incentive to develop software for Deno. It doesn't matter that Node compatibility is spotty, because they'd rather just go back to using Node like they're used to. Supporting multiple runtimes is painful. If you want to understand the pain, ask anyone who tried to ship any cross platform application written in C or C++.

Deno should have promoted its own API

If the competition is trying to be more like Node, Node is the winner.

There is a lesson to be learned here. If you are trying to replace a legacy system, don't re-implement the same legacy system. Instead, put the burden of backwards-compatibility on the legacy system.

Deno aimed to uncomplicate Javascript. (Deno's homepage literally says that.) By trying to mimic Node, Deno has unintentionally put Node's complexity problem at the center of the stage. And now, it cannot be removed. Instead of being a brand new thing, Deno ended up being a less reliable variant of Node.

Deno should have supported its own API on top of Node instead. Since Deno controls its API, supporting its own API on Node would be simpler than supporting Node APIs. For library and framework developers, libraries made for Deno would work on Node and there would be no need to support multiple runtimes.

This would have resulted in a much larger ecosystem of software made for Deno which is more reliable and free of Node's legacy.

Background

For a homelab, all I have is an Intel NUC acting as a singular server. It is working as great as it could, but the single server setup leaves more to be desired. Every time I take it down for maintenance, for the time the server is down, I lose access to the services running on the server, and the internet. I wish to upgrade to a cluster of servers so that I could have additional servers to pick up the slack automatically. I could set up a Pacemaker cluster to ensure that only one active server will handle my default route and all my services. But then I would be paying for N servers and getting the benefit of one. If I have N servers, I want the capacity of N servers, or at-least a decent fraction of N. After some searching I figured out that VyOS and other router software can achieve this, but I don't want the complexity of managing N different routers. I want 1 router. One, distributed router.

Baby steps first

The goal

First, let's figure out a way to reliably discover each node's immediate neighbors, and via how many ways we can connect to them.

Each node sends its information, like its name, public keys, etc via IPv6 link local multicast packets. I'll call these packets advertisements. I choose IPv6 link local because no manual IP address configuration is required. The kernel automatically assigns IPv6 link local addresses for each network interface.

When another node (B) receives the multicast packets, it knows that it can receive packets from the former node (A), but what about the other way round? For that, node B then establishes a TCP connection to node A. If node B can connect to node A, we can conclude that a bidirectional communication is possible between node A and node B.

We also need to deal with a potential race condition. What if both node A and node B connect to each other simultaneously? We need a tie breaker to decide which connection to keep. All we need to do is to assign a random number called rank to each connection, and simply keep the connection with highest rank.

But what if the two nodes are connected by more than one links? What if there is a switch in between? The number of TCP connections will increase quadratically, and before long we'd be eating file descriptors for breakfast. So, instead of establishing a new TCP connection for each unique advertisement, the two nodes will only maintain a single TCP connection. The two nodes share advertisements received from each other. Thus for each pair of nodes A and B node A has two sets of advertisements:

• Advertisements received from node B over multicast UDP
• Node A's advertisements reported as received by node B over TCP connection

The intersection of these two sets will provide all working bidirectional communication paths between node A and node B (without routing, that is.)

An example

Consider the following scenario:

 -----------------                                            -----------------
 |               |  eth0(fe80::a0)           eth0(fe80::b0)   |               |
 |      Node     |--------------------------------------------|      Node     |
 |               |                                            |               |
 |       A       |  eth1(fe80::a1)           eth1(fe80::b1)   |       B       |
 |               |--------------------------------------------|               |
 |               |                                            |               |
 -----------------                                            -----------------

There are two network interfaces attached to each node, namely eth0 and eth1. The network interfaces are shown above with their respective link local addresses.

What happens if Node A sends an advertisement via eth0?

  ----------                                   ----------
  | Node A |                                   | Node B |
  ----------                                   ----------
      |                                            |
      | Advertisement(name: A)                     |
      |------------------------------------------->|
      | eth0                                  eth0 |
      |                                            |
      |                                TCP connect |
      |<-------------------------------------------|
      | eth0                                  eth0 |
      |                                            |

Once node B receives the advertisement, it establishes a TCP connection back to Node A. Now that TCP connection is established, both nodes A and B are aware that bidirectional communication is possible between the two nodes via eth0 network interface.

Discovery of the second link (eth1 -- eth1) happens without creating another TCP connection.

  ----------                                                    ----------
  | Node A |                                                    | Node B |
  ----------                                                    ----------
      |                                                             |
      | Advertisement(name: A)                                      |
      |------------------------------------------------------------>|
      | eth0                                                   eth0 |
      |                                                             |
      |                                                 TCP connect |
      |<------------------------------------------------------------|
      | eth0                                                   eth0 |
      |                                                             |
      | Advertisement(name: A)                                      |
      |------------------------------------------------------------>|
      | eth1                                                   eth1 |
      |                                                             |
      |         Heartbeat(received_adverts: [fe80::a1 -> fe80::b1]) |
      |<------------------------------------------------------------|
      | eth0                                                   eth0 |
      |                                                             |
      |                                      Advertisement(name: B) |
      |<------------------------------------------------------------|
      | eth1                                                   eth1 |
      |                                                             |
      | Heartbeat(received_adverts: [fe80::b1 -> fe80::a1])         |
      |------------------------------------------------------------>|
      | eth0                                                   eth0 |
      |                                                             |

The nodes share the received advertisements over the already established TCP connection. They aren't sent on demand, but rather as part of periodic heartbeat messages sent over the TCP connection.

Testing time

Network namespaces, without root

Linux has network namespaces. Basically you divide the Linux network stack into partitions; each partition behaves as a separate computer from network point of view. You can easily do this using unshare --net command and you'll have a shell running in a new, isolated network namespace.

# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth0@if2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether 2a:66:b1:f2:e8:80 brd ff:ff:ff:ff:ff:ff link-netnsid 0
# unshare --net
# ip link show
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
# 

You can make these isolated namespaces reachable by either moving real network interfaces into them, or by creating veth pairs and moving one of them into the network namespace.

One small problem, you need to be root to unshare a network namespace, and you don't want to run your test suite as root.

$ unshare --net
unshare: unshare failed: Operation not permitted
$

Fortunately, we have user namespaces. For this article all you need to understand is that by unshareing a user namespace, you can create a sandbox where your process has root access within the box, but not outside it.

$ unshare --user --map-root-user
# ls -l .bashrc /usr/bin/bash
-rw-r--r-- 1 root   root       933 Feb  6  2023 .bashrc
-rwxr-xr-x 1 nobody nobody 1112880 Jan 16 16:18 /usr/bin/bash
# 

In a nutshell, my UID has been mapped to root and all other UIDs have been made inaccessible. This does not give me any special privileges.

# : > /usr/bin/bash
-bash: /usr/bin/bash: Permission denied

But the important bit, can I unshare a network namespace now?

$ unshare --user --map-root-user
# ls -l .bashrc /usr/bin/bash
-rw-r--r-- 1 root   root       933 Feb  6  2023 .bashrc
-rwxr-xr-x 1 nobody nobody 1112880 Jan 16 16:18 /usr/bin/bash
# unshare --net
# ip link show
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
# 

Yes!

Remote controlled worker process

User namespaces solves the problem of emulating a network situation without any virtualization or any container runtime (like the one shown in the first figure), but how would you set them up? There is nsenter but it needs actual root user to work.

$ unshare --user --map-root-user --net sleep infinity &
[1] 208920
$ nsenter --net=/proc/208920/ns/net -- ip link show
nsenter: reassociate to namespace 'ns/net' failed: Operation not permitted
$ sudo nsenter --net=/proc/208920/ns/net -- ip link show
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
$ 

And by the way /proc/<pid>/ns/net file represents the network namespace of a given process.

So I made a remote controlled worker process to execute commands from within a network namespace, which connects to the test suite listening at a unix domain socket.

The first worker process is run in a user namespace via unshare; I call it the hub process. Once the hub process connects to the test suite, it can be made to spawn additional worker processes, each running under separate network namespaces via unshare. All the worker processes connect to the same unix domain socket and can be controlled individually and in the right order determined by the test suite.

Why do I need a hub process? That is because each network namespace needs to be under a common user namespace so that network interfaces can be moved between them.

$ unshare --user --map-root-user --net sleep infinity &
[1] 211416
$ unshare --user --map-root-user --net
# ip link add eth0 type veth peer name temp
# ip link set temp netns 211416 name eth0
RTNETLINK answers: Operation not permitted
# 

Having a hub process makes it easy to have a common user namespace.

$ unshare --user --map-root-user
# unshare --net sleep infinity &
[1] 211458
# ip link add eth0 type veth peer name temp
# ip link set temp netns 211458 name eth0
# ip link show
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth0@if2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 66:e8:e1:8a:99:c0 brd ff:ff:ff:ff:ff:ff link-netnsid 0

Which means, finally I have a decent way to set up a test scenario, while keeping up the readability of test cases.

#[tokio::test]
async fn ping() {
    let mut hub = Hub::new().await.unwrap();

    let mut a = hub.namespace().await.unwrap();
    let mut b = hub.namespace().await.unwrap();

    //Connect two namespaces
    a.connect("eth0", "eth0", &mut b).await.unwrap();

    //Assign IP addresses
    a.sh("ip addr add 192.168.0.1/24 dev eth0").await.unwrap();
    a.sh("ip link set eth0 up").await.unwrap();
    b.sh("ip addr add 192.168.0.2/24 dev eth0").await.unwrap();
    b.sh("ip link set eth0 up").await.unwrap();

    //Ping test
    a.sh("ping -c 1 192.168.0.2").await.unwrap();
}

You can check out the code behind it at https://gitlab.com/akash_rawal/nwlab2/-/tree/master/src/worker?ref_type=heads.

If you browse the rest of the repository, you can also find tests and implementation of the basic neighbor discovery algorithm that I was talking about.

Conclusion

My first attempt to write tests comprised of using separate code blocks to initialize each network namespace, and then make them callable from main. That grew ugly very quickly. I mean just look at it.

pub fn routine() -> Routine {
    Routine::from(|| async move {
        cmd::run(&mut cmd::new_ns(".a")).await.unwrap();
    }).push(".a", Routine::from(|| async move {
        cmd::init_netns().await.unwrap();

        //Spawn a child in new network namespace and send it the other end
        let mut child = cmd::fork_ns(".c").spawn().unwrap();

        //Connect to it
        let pid = child.id().unwrap();
        cmd::new_veth("eth0", "eth0", pid).await.unwrap();

        //Assign IP address
        cmd::bash("ip addr add 172.16.0.1/16 dev eth0").await.unwrap();
        cmd::bash("ip link set eth0 up").await.unwrap();

        //Run echo client
        crate::svc::echo_client("172.16.0.2:7").await.unwrap();

        //Wait for child
        child.wait().await.unwrap();

        log::info!("Client ended");
    }).push(".c", Routine::from(|| async {
        cmd::init_netns().await.unwrap();
        
        //Wait for the network link
        cmd::wait_for_link("eth0").await.unwrap();
        log::info!("From child:");
        cmd::bash("ip link show").await.unwrap();

        //Assign IP address
        cmd::bash("ip addr add 172.16.0.2/16 dev eth0").await.unwrap();
        cmd::bash("ip link set eth0 up").await.unwrap();

        //Run echo server
        crate::svc::echo_server_1("172.16.0.2:7").await;

        log::info!("Server ended");
    })))
}

It is a very similar test case, but you see that cmd::wait_for_link("eth0") in there? That is a busy wait that repeatedly checks for whether the given network interface exists. I am quite happy with how easily I can set up test scenarios. The neighbor discovery itself is not the star of this show; it takes ~5 seconds to converge and needs a lot more work.

How it all started

I was once building a service that checks a MariaDB cluster for common problems and do some basic repair work automatically. So I needed a MariaDB instance to test my project. For that, I used Testcontainers. Testcontainers is a library to run external dependencies in ephemeral containers, so no need to setup environment variables or managing test servers. Go check out their homepage.

I set up a quick test to check out how it works... and we have a problem.

2024/08/18 21:26:46 🐳 Creating container for image testcontainers/ryuk:0.7.0
2024/08/18 21:26:46 ✅ Container created: 912f450a9bc3
2024/08/18 21:26:46 🐳 Starting container: 912f450a9bc3
2024/08/18 21:26:46 ✅ Container started: 912f450a9bc3
2024/08/18 21:26:46 ⏳ Waiting for container id 912f450a9bc3 image: testcontainers/ryuk:0
.7.0. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms}
2024/08/18 21:26:46 failed accessing container logs: Error response from daemon: can not 
get logs from container which is dead or marked for removal
--- FAIL: TestMain (0.41s)

Ended up wasting the rest of the day troubleshooting the issue, recognizing that the issue is with testcontainers/ryuk container not being able to do its work, trying some workaround, and then grudgingly disabling the user namespace feature that I had enabled in Docker daemon.

Testcontainers' implementation detail: its reaper

When you start containers within your tests, it is desirable to reliably terminate them.

Many programming languages offer deterministic destructors. (Or similar features like Go's defer statement, or Rust's Drop trait) Some test runners offer functionality to run user-defined cleanup function at the end of each test, or at the end of all tests.

None of these are reliable. If the test crashes or terminates abnormally, these containers will not be cleaned up.

The solution made by Testcontainers is docker.io/testcontainers/ryuk. From the dockerhub description, using it involves only 6 simple steps.

1. Start it:

 $ docker run -v /var/run/docker.sock:/var/run/docker.sock -e RYUK_PORT=8080 -p 8080:8080 docker.io/testcontainers/ryuk

2. Connect via TCP:

 $ nc localhost 8080

3. Send some filters:

 label=testing=true&health=unhealthy
 ACK
 label=something
 ACK

4. Close the connection

5. Send more filters with "one-off" style:

 printf "label=something_else" | nc localhost 8080

6. See containers/networks/volumes deleted after 10s:

Its weakness

It requires access to the docker socket.

Access to the docker socket usually implies root access on the system the docker daemon runs on.

This also means that ryuk cannot function if it is running in a user namespace or a similarly unprivileged situation.

I am not a fan of tests requiring root access. If anything, I want software development work to be as contained as I can make it. If a test goes haywire, I don't want to waste my next few hours of my life restoring my PC from backup. It is not even a ridiculous concern, I once encountered a buggy test case that performed rm -rf $HOME because of hardcoded Windows paths which didn't work on linux.

Podman is a very attractive option to docker. It requires no root access, needs no daemons, and only setup it requires is setting up /etc/subuid and /etc/subgid, which likely your operating system already does.

But testcontainers requires a docker socket, and at the time of writing, its podman support is experimental.

How do we reliably cleanup dependencies running in containers reliably?

Just replace the entrypoint

For each dependency container, we can replace its entrypoint to include a cleanup functionality.

This is what it needs to do:

1. Fork off the prior entrypoint.
2. Open a TCP socket and wait for a connection
3. Wait for the connection to close.
4. When the connection closes, simply quit.
5. As the new entrypoint that quit is the PID 1 of the container, the container terminates.

The following was the first iteration of such an entrypoint.

#!/bin/sh 

#Run the base container's entrypoint 
"$@" & 

#Exit after the keep-alive connection is closed 
exec socat TCP-LISTEN:4,accept-timeout=15 EXEC:cat

This is what the test fixture does:

1. Start the container with replaced entrypoint.
2. Connect to TCP port 4 and keep the connection open.
3. At the end of test suite, close the TCP connection. Or the connection is closed by the operating system if the test crashes.

I have a more robust version of the entrypoint at https://gitlab.com/akash_rawal/selfterm/-/tree/master/test_entrypoint. In the gitlab project you can also find prebuilt images for MariaDB and Postgres container images.

Conclusion

Testcontainers is an awesome project and I like the idea, but I think the implementation for cleaning up containers is too complicated and a bit flawed. Its reliance on the docker daemon is an ongoing problem. But with a dash of trickery this can be turned into a false statement. I hope we get more tooling for testing which is compatible with user namespaces or Podman, or similarly require less privileges.

What else, test with safety, folks!