Summary

Our test shows that AI-generated MATLAB code improves significantly when the model is given MATLAB-specific skills, but static analysis remains necessary for catching and fixing residual coding-standard violations. The combination of the MATLAB Agentic Toolkit and CC4M reduced detected guideline violations from 37 to just 2.

The workflow with AI skills for code generation and static analysis for assurance, reduces developer intervention and improves compliance before code reaches the CI/CD pipeline.

Introduction

We tested the MATLAB Agentic Toolkit and our Code Checker for MATLAB (CC4M) in a MATLAB code generation workflow, to see how much they improve the generated code.

For the trial we are using the unsloth/Qwen3.6-27B-MTP-GGUF:Q6_K model running on-premise on a llama.cpp server, being accessed from OpenCode Desktop, with MATLAB R2024a available via the Model Context Protocol (MCP).

We tested the following resources in the code generation workflow:

• MATLAB Agentic Toolkit: defines skills that the model can use to write better code.
• CC4M: made available as an MCP tool to improve the generated code via a static check (using a development version).

To compare the effects of these resources, we are generating code in three phases:

1. Set a benchmark by letting the model create a PLAN.md from a prompt and generate the code without any of the extra resources.
2. Use the MATLAB Agentic Toolkit and see how this improves the plan and code.
3. Use CC4M as an MCP tool to detect and fix coding guidelines violations.

The prompt that was used as a starting point for the model:

Write a PLAN.md for writing a MATLAB function that simulates the throw of a ball.

The ball throw should be simulated with a ODE and include the effect of drag and horizontal wind

The function must have the following inputs: throw speed, throw angle, and wind speed.
Add unit tests to validate the function.

When the generated code complies with the requirements and passes all tests, use the run_cc4m tool to check that the code complies with the standards.

Phase 1 - Benchmark

The model made the following plan based on the prompt:

• PLAN.md

and generated the following code and tests:

• simulate_ball_throw.m
• test_simulate_ball_throw.m

Evaluation

• On first glance, the plan and function appear to be correctly implemented.

• Throwing the ball horizontally or downward was not identified by the model as a special case. This caused an MCP timeout error when the tests were run, as the ode45 method ran indefinitely in the testHorizontalThrow test. This needed to be repaired manually, as the timeout error did not provide the model with information of the cause of the timeout.

• The tests contain hallucinations on how to assert the test conditions and thus all throw errors.

  assert(testCase, abs(results.range) < 1, 'FailureMessage', 'Error message');

  The model was not able to recover, as it did not get the feedback needed to do so. It needed an intervention by providing the help of testCase.verifyTrue to fix the tests.

• With the fix, the tests reach good code coverage, although they could be extended with some extra special cases, like throwing down or back.

  One of the failing tests was 'fixed' by first reducing the throwing speed and then relaxing the condition in the test to ensure it passes. [...]

  

Phase 2 - Adding MATLAB Agentic Toolkit skills

For the second phase, we made the MATLAB Agentic Toolkit SKILL.md files available via the OpenCode configuration and requested the model to create a new plan with these skills:

I have provided extra skills that should help you write better MATLAB code. Recreate the plan from the previous instructions using these new skills

The model responded with

...
Now I have a thorough understanding of the skills and the workspace. Let me rewrite the plan.
...

and wrote a new plan with the following differences, as (accurately) described by the model:

• PLAN.md

The code and tests that were generated from the updated plan are available via the following links:

• simulateBallThrow.m
• tsimulate_ball_throw.m

Evaluation

• On first glance, the plan, function and tests appear to be correctly implemented.
• The updated code makes good use of the arguments block, although the plan made up the validator function: mustBeScalar.
• The tests make better use of the available TestCase class verify functions, but the model made up the verifyFields and verifyContains methods.
• The function filename was originally in snake_case, while the function name was in lowerCamelCase, which triggers a MATLAB CodeAnalyzer warning.

These issues were detected when the tests were run on the MCP. The model kept fixing the errors and rerunning the tests until all tests passed.

• Note: The horizontal or downward ball throw problem with the ODE also needed to be fixed manually in this case.

Phase 3 - Improve code with CC4M

The coding standards defined in the Toolkit have been largely applied in the generated code. However, some guidelines appear to have not made it to the model's context and thus have not been applied in the code. An extra check and fix cycle can make the code comply with the guidelines.

So, in this phase we ran CC4M with the MATLAB Coding Guidelines implemented as the predefined MathWorksGuidelines_v1 configuration with 62 checks that is shipped with CC4M. With the guideline violations detected by CC4M, the model was able to resolve most* violations and improve the code:

• simulateBallThrow.m

* The nested functions were not converted to separate functions by the model, as the Local functions guideline is only a recommendation and would require a restructuring of the code. The Local functions guideline of the MathWorks also allows for using local functions, although having extra tests for the ballOde function would be a good addition to the test suite.

Evaluation

Checking and improving the code with CC4M was quick. Especially when compared with writing and fixing the code and tests based on the original plan.md file. The changes made to the code make sense and improve compliance, but do not change the code much.

For the ball throw simulation functions that were generated in the three phases, CC4M analysis found the following number of violations. These numbers show that giving the model MATLAB skills overall improves the code quality, but that a static code check step further reduces guideline violations.

Rule ID	Phase 1	Phase 2	Phase 3
Argument validation	1	-	-
Avoid shadowing	-	1	-
Blank lines around local functions	4	-	-
Function name casing (main/local/nested)	1	-	-
Local functions	2	-	2
Nested functions	-	2	-
Number of function inputs	1	1	-
Parentheses in mathematical expressions	1	2	-
Spaces after commas, & semicolons	1	3	-
Spaces around multiply, divide, & exponent operators	16	12	-
Use of break, continue, & return	1	1	-
Use of literal values	5	4	-
Variable name casing	4	-	-
Total violations	37	26	2
Number of rules	11	8	1
Number of lines	100	124	128
Executable lines	60	68	73

Our verdict

MATLAB code generated without any additional resources is likely to be okay for simple use cases. It needs extra guidance on how to do the assertions in tests, though. It may also require extra guidance when the model is making mistakes, so it is best to give it smaller tasks and keep it under supervision.

Using the MATLAB Agentic Toolkit greatly improves the quality of the code that is written and will likely remove most needs for user interventions. We have not tested code generation using any of the more specialized MATLAB Toolboxes, but the Toolkit provides skills for those. Given how much it improved the generated tests, it is likely to also greatly improve the code quality for those use cases.

The tests that are generated with the Toolkit give good coverage and are a decent starting point. They should be expanded upon to ensure the generated functions work for the full operational scope.

Having the Toolkit's coding guidelines in the model context improves adherence in the code, but a static code check with CC4M in a post-process step still detects and fixes many violations. Going forward, we will use the MATLAB Agentic Toolkit to write the code and tests, and a CC4M step to improve compliance with our own guidelines. This reduces the number of violations found in our CC4M CI/CD pipeline, which the developer would otherwise need to fix in the merge request.