Programming

16975 readers

1288 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Follow the programming.dev instance rules
Keep content related to programming in some way
If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities [email protected]

founded 1 year ago

MODERATORS

[email protected]

-2

Open-Source, Language-Agnostic Mutation Testing Tool Using LLM Agents (github.com)

submitted 2 months ago by [email protected] to c/[email protected]

3 comments fedilink hide all child comments

Check out our open-source, language-agnostic mutation testing tool using LLM agents here: https://github.com/codeintegrity-ai/mutahunter

Mutation testing is a way to verify the effectiveness of your test cases. It involves creating small changes, or “mutants,” in the code and checking if the test cases can catch these changes. Unlike line coverage, which only tells you how much of the code has been executed, mutation testing tells you how well it’s been tested. We all know line coverage is BS.

That’s where Mutahunter comes in. We leverage LLM models to inject context-aware faults into your codebase. As the first AI-based mutation testing tool, Mutahunter surpasses traditional “dumb” AST-based methods. Our AI-driven approach provides a full contextual understanding of the entire codebase, enabling it to identify and inject mutations that closely resemble real vulnerabilities. This ensures comprehensive and effective testing, significantly enhancing software security and quality.

We’ve added examples for JavaScript, Python, and Go (see /examples). It can theoretically work with any programming language that provides a coverage report in Cobertura XML format (more supported soon) and has a language grammar available in TreeSitter.

Check it out and let us know what you think! We’re excited to get feedback from the community and help developers everywhere improve their code quality.

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 2 points 2 months ago* (last edited 2 months ago) (2 children)

As the first AI-based mutation testing tool, Mutahunter surpasses traditional “dumb” AST-based methods. Our AI-driven approach provides a full contextual understanding of the entire codebase,

This is where I call bullshit. The AST is a precise representation of the code which should be easier to analyse and modify instead of the raw text. If you only rely on processing the text I have a hard time believing your AI has a deep understanding of the contextual interdependencies of the different parts of the code.

Edit: it kinda does use the AST if one of the preconditions is a treesitter grammar? Maybe the marketing wording is just very unfortunate?

[–] [email protected] 2 points 2 months ago (1 children)

Hey our bad, the wording is not clear. We do make use of the AST to give the AI more context of the codebase.

[–] [email protected] 1 points 2 months ago

I guess it's also my bad. After reading the text with a more neutral mindset it's not that clear cut as I initially read it. My bs detector tends to get a bit sensitive when AI comes up and I interpreted the wording in the worst way possible.