Setup practices detection with the AI Agent
Understand how your custom practices can be detected in your IDE, your CI/CD pipelines and code reviews, using the Packmind AI Agent.
How do practice detection work?
The Packmind AI Agent relies on AI to generate a deterministic program to detect violations of a coding practice. (Think of it as a custom Eslint rule for instance, generated by AI).
To process a practice, the AI Agent will need at least:
A description for the practice (NB: that will be in most cases generated by Packmind AI)
A negative example.
As soon as a practice is validated, the AI Agent will assess the practice. Once assessed the agent starts the workflow.
The practice might be labeled as Undetectable if it is overly generic, or if detecting it reliably requires making too many assumptions.
Upon updating the practice you will be able to reassess it from the Detection popover
The AI Agent defines a two-step workflow. You can configure and monitor it in the Detection page of a practice.
You can open the Detection page by clicking the Settings button in the Detection popover.
0. Practice Assessment
The initial step in assessing whether a practice can be detected in code is to evaluate if the practice description is thorough and supported by sufficient examples. This ensures clarity in understanding the intent and applicability of the practice, facilitating an effective generation of detection guidelines.
1. Detection Guidelines and unit tests
Detection guidelines, generated first by the AI Agent, indicate how violations of the practice can be detected in a source code using the code syntax. While the description of the practice focuses on explaining its intent, the guidelines only target the identification of its violation.
When guidelines are generated the first time, few unit tests are also generated. These are useful to help the AI Agent to understand your practice (read more on unit tests below).
Keep in mind that you can edit both guidelines and unit tests at any time if you consider some adjustments are needed🤞.
2. Program generation
Once guidelines are generated, and every time they are updated, the AI Agent runs a new program generation.
The program must fulfill the following requirements to be successful:
All negative examples (added manually, unit tests, or generated by Packmind AI) must be detected by the program.
All positive examples (added manually, unit tests, or generated by Packmind AI) must NOT be detected by the program.
Define unit tests for the detection process
Unit tests are code snippets that either contain or do not contain a violation of a coding practice. This helps to document common cases where violations should be detected or not. It's helpful, for instance, when you've identified a false positive and want to document why this case should not be detected as a violation.
When you open the Detection page of a practice, you'll find a section called Unit Tests
In the Unit Tests section you can:
Add, edit, or remove unit tests.
Run the test suite to ensure that the program still behaves as expected.
Outdated Detection Status and False Positives
When a new code example is added to an existing practice, Packmind’s analysis program may or may not detect it as a violation. If the program fails to detect the example, the associated practice will be marked as Outdated in the Packmind interface and a corresponding warning will be shown.
In this state:
the practice will no longer be detected in IDEs or via the CLI
developers won’t receive feedback based on it
To resolve this:
a Packmind user must regenerate the program for the affected practice
they must ensure that the new example is correctly identified as a violation after regeneration
Once the regeneration is successful, the practice will be detected again in the IDE and CLI tools as expected.
Can I refine the scope of exclusions and inclusions for a practice?
When you open the Detection page of a practice, you'll find a section called Detection Filters.
You can add one or several patterns both for inclusions and exclusions.
Packmind supports glob patterns, here are some examples:
**/*.spec.ts
Match files ending with .spec.ts
**/*Controller*
Match files which name containsController
**/infra/**/*.ts
Match.tsfiles under a directory named infra,directly or not (can be a sub-sub directory for instance)
**/infra/*.ts
Match .tsfiles directly under a directory named infra
apps/**/*
Match all files inside the root directory apps
What if I don't use AI?
If you cannot use AI in your context for compliance reasons, you can still use regular expressions or Semgrep patterns. Go to the Detection popover of a coding practice to open the configuration menu.
Detection Statuses
In the Practices section, a coding practice in Packmind has one of the several detection statuses:
To configure: The practice has no detection configuration yet.
Configured: The practice can be detected for violation ✅
Disabled: The practice has been manually disabled for detection
Outdated: The current program does not cover all unit tests and examples. This can happen when users update practice examples or unit tests. In this case, you can regenerate a new program.
In progress: The AI Agent is currently generating a program to detect violations of the practice.
Failed: The AI Agent could not succeed in generating a program. You'll have to ensure your practice examples are fine and refine the detection guidelines if needed.
Where are violations detected?
Violations of practices are pushed to developers in 3 different channels:
In IDEs, thanks to our IDE extensions;
During code reviews, thanks to our web browsers extensions;
When using the CLI, locally or in the CI/CD process.
Can I integrate reports with other tools?
Currently, the CLI supports two formatters for the output file:
The SonarQube generic format
The SARIF format (Static Analysis Results Interchange Format), which you can integrate with tools that support it
Please reach us to suggest new formats.
Does Packmind store source code on server side?
The answer is simple: no. The source code is sent through our plugins or the CLI through a secure SSL connection, but once our engine has analyzed the code, it's just not stored at all in our database. The database only stores source code related to a best practice description.
Last updated
Was this helpful?