• pixxelkick@lemmy.world
    link
    fedilink
    arrow-up
    4
    ·
    1 day ago

    Reproducing the bug with an automated test is harder, its code you can run that tests your other code.

    But allows you to just 1 click run it and get a yes/no “is this still broken” output without having to manually reproduce it by hand each time.

    Whats important is this is in the domain of what LLMs can actually work with, the output of the test is something they can parse and iterate on until it works.

    They execute the command to run the test, check the output, and keep working til the test passes.

    They can add additional tests to help isolate the problem, or strip down the existing test until its doing the absolute bare min steps to reproduce, in order to narrow the scope of whats causing it.

    But when your test involves stuff running in the kernel of an OS, your automated tests meed to effectively be code you write that bootstraps a virtual machine up and manipulates and observes that second machines kernel…

    You can do it, but its one of the most complicated forms of automated tests to design and run!