VerifyML
See the introduction, the maker also suggests to use an front-end tool to collaboratively change the model card. Model Card Editor this is not open-source and also the developer suggests in this issue to not use this tool but to use tools like AIVerify. This checklist only looks at the verifyML python toolkit and not the web interface.
Functionality
| Requirement | Priority | Fulfilled | Comments |
|---|---|---|---|
| The tool allows users to conduct technical tests on algorithms or models, including assessments of performance, bias, and fairness. To facilitate these tests, users can input relevant datasets, | M | 1 | The tool does allow a few standardized tests, specified here |
| The tool allows users to choose which tests to perform. | M | 1 | In code the user is free to choose any test |
| The tool allows users to fill out questionnaires to conduct impact assessments for AI. For example IAMA or ALTAI. | M | 0 | |
| The tool can generate a human readable report. | M | 1 | The tool can visualize model cards that are generated by it |
| The tools works with a standardized report format, that it can read, write, and update. | M | 1 | It generates html which can be imported by a machine |
| The tool supports plugin functionality so additional tests can be added easily. | S | 1 | Any test can be ran by the user itself and the output imported in the model card generated by the tool |
| The tool allows to create custom reports based on components. | S | 0 | It doesn't offer any standardization in what to put in the report |
| It is possible to add custom components for reports. | S | 1 | Anything can be put in the model card, which makes it very flexible |
| The tool provides detailed logging, including tracking of different model versions, changes in impact assessments, and technical test results for individual runs. | S | 0 | Not ouf of the box, but this could be written in code by the owner of the algorithm |
| The tool supports saving progress. | S | 1 | Once the modelcard is generated it could be loaded in again and be changed |
| The tool can be used on an isolated system without an internet connection. | S | 1 | Once the tool is imported in python it can be used without an internet connection |
| The tool offers options to discuss and document conversations. For example, to converse about technical tests or to collaborate on impact assessments. | C | 0 | Assessments are not supported |
| The tool operates with complete data privacy; it does not share any data or logging information. | C | 1 | It does not do this |
| The tool allows extension of report formats functionality. | C | 1 | As it exports html, it can also be transferred to json or markdown |
| The tool can be integrated in a CI/CD flow. | C | 1 | The automated tests could be ran in the CI/CD tool to generated a model card |
| The tool can be offered as a (cloud) service where no local installation is required. | C | 0 | The python tool itself not, but a frontend which needs to be developed yes |
| It is possible to define and automate workflows for repetitive tasks. | C | 1 | As it is written in python this can be automated easily |
| The tool offers pre-built connectors or low-code/no-code integration options to simplify the integration process. | C | 0 | The tool does this not |
total_score = 42
Reliability
| Requirement | Priority | Fulfilled | Comments |
|---|---|---|---|
| The tool operates consistently and reliably, meaning it delivers the same expected results every time you use it. | M | 1 | Once you have located the right (older) libraries it runs pretty smoothly and reliably |
| The tool recovers automatically from common failures. | S | 0 | Library dependencies needs to be solved by yourself as this is not handled by the tool (especially graphs) |
| The tool recovers from failures quickly, minimizing data loss, for example by automatically saving intermediate test progress results. | S | 0 | It does not store any intermediary results |
| The tool handles errors gracefully and informs users of any issues. | S | 0 | It just breaks, you need to explicitly export the model card for it to saved |
| The tool provides clear error messages and instructions for troubleshooting. | S | 0 | The error messages are python error messages unrelated to the tool |
total_score = 4
Usability
| Requirement | Priority | Fulfilled | Comments |
|---|---|---|---|
| The tool possess a clean, intuitive, and visually appealing UI that follows industry standards. | S | 0 | There is no user interface |
| The tool provides clear and consistent navigation, making it easy for users to find what they need. | S | 0 | There is no user interface |
| The tool is responsive and provides instant feedback. | S | 0 | There is no user interface |
| The user interface is multilingual and supports at least English. | S | 0 | There is no user interface |
| The tool offers keyboard shortcuts for efficient interaction. | C | 0 | There is no user interface |
| The user interface can easily be translated into other languages. | C | 0 | There is no user interface |
total_score = 0
Help & Documentation
| Requirement | Priority | Fulfilled | Comments |
|---|---|---|---|
| The tool provides comprehensive online help documentation with searchable functionalities. | S | 0.5 | The documentation is quite concise and helpful, but it is outdated |
| The tool offers context-sensitive help within the application. | C | 0 | No context info whatsoever |
| The online documentation includes video tutorials and training materials for ease of learning. | C | 0 | Just documentation |
| The project provides readily available customer support through various channels (e.g., email, phone, online chat) to address user inquiries and troubleshoot issues. | C | 0 | The people who worked on the tool are quick to respond to issues, but they don't support the tool anymore |
total_score = 1.5
Performance Efficiency
| Requirement | Priority | Fulfilled | Comments |
|---|---|---|---|
| The tool operates efficiently and minimize resource utilization. | M | 1 | Very lightweight tool, as it is a python package |
| The tool responds to user actions instantly. | M | 1 | When run, it returns instantly |
| The tool is scalable to accommodate increased user base and data volume. | S | 1 | This would be installed distributed and therefore would be scalable, with large datasets it is still very quick |
total_score = 11
Maintainability
| Requirement | Priority | Fulfilled | Comments |
|---|---|---|---|
| The tool is easy to modify and maintain. | M | 1 | The tool itself it not so large and written with tools we are all quite aware of |
| The tool adheres to industry coding standards and best practices to ensure code quality and maintainability. | M | 1 | The repository has poetry, pre-commit hooks, has a CI, and looks well structured |
| The code is written in a common, widely adopted and supported and actively used and maintained programming language. | M | 1 | in Python and jupyter notebooks |
| The project provides version control for code changes and rollback capabilities. | M | 1 | It is hosted on Github |
| The project is open source. | M | 1 | Apache 2.0 license |
| It is possible to contribute to the source. | S | 0 | The project is not active supported anymore, so we would need to make a fork and make that the main source |
| The system is modular, allowing for easy modification of individual components. | S | 0.5 | The idea of a model card is pretty modular, and can be changed any way we like. Adding assessments in the tool would be quite the effort |
| Diagnostic tools are available to identify and troubleshoot issues. | S | 1 | Just standard python troubleshooting tools |
total_score = 24.5
Security
| Requirement | Priority | Fulfilled | Comments |
|---|---|---|---|
| The tool must protect data and system from unauthorized access, use, disclosure, disruption, modification, or destruction. | M | 0 | not applicable |
| Regular security audits and penetration testing are conducted. | S | 0 | As the tool is not actively maintained anymore |
| The tool enforce authorization controls based on user roles and permissions, restricting access to sensitive data and functionalities. | C | 0 | As this is a local import only, this is managed by the developer |
| Data encryption is used for sensitive information at rest and in transit. | C | 0 | Intermediary data is not stored, and the end result is put in html with no encryption |
| The project allows for regular security audits and penetration testing to identify vulnerabilities and ensure system integrity. | C | 1 | It does not block this for users to do this |
| The tool implements backup functionality to ensure data availability in case of incidents. | C | 0 | Not supported |
total_score = 2
Compatibility
| Requirement | Priority | Fulfilled | Comments |
|---|---|---|---|
| The tool is compatible with existing systems and infrastructure. | M | 1 | It can be easily imported and installed in python |
| The tool supports industry-standard data formats and protocols. | M | 1 | Standardized tests are used and the output format is html |
| The tool operates seamlessly on supported operating systems and hardware platforms. | S | 1 | As it is a python tool, anywhere where python can run this can also be run |
| The tool supports commonly used data formats (e.g., CSV, Excel, JSON) for easy data exchange with other systems and tools. | S | 1 | This can be imported |
| The tool integrates with existing security solutions. | C | 0 | It does not do such a thing |
total_score = 14
Accessibility
| Requirement | Priority | Fulfilled | Comments |
|---|---|---|---|
| The tool is accessible to users with disabilities, following relevant accessibility standards (e.g., WCAG). | S | 0 | You need to be a programmer to use it, and that is not your typical user with disabilities |
total_score = 0
Portability
| Requirement | Priority | Fulfilled | Comments |
|---|---|---|---|
| The tool support a range of operating systems (e.g., Windows, macOS, Linux) commonly used within an organization. | S | 0.5 | If you can run python, which is not always possible within the government for example |
| The tool minimizes dependencies on specific hardware or software configurations, promoting flexibility. | S | 1 | As it is a python tool |
| The tool offers a cloud-based deployment option or be compatible with cloud environments for scalability and accessibility. | S | 0 | It is not offered as a cloud-based option |
| The tool adheres to relevant cloud security standards and best practices. | S | 0 | On the github nothing is mentioned about security and for the cloud version it is not applicable |
total_score = 4.5
Deployment
| Requirement | Priority | Fulfilled | Comments |
|---|---|---|---|
| The tool has an easy and user-friendly installation and configuration process. | S | 0.2 | You need to have some developer knowledge and also knowledge about the technical tests to use |
| The tool has on-premise or cloud-based deployment options to cater to different organizational needs and infrastructure. | S | 0 | Not applicable |
total_score = 0.6
Legal & Compliance
| Requirement | Priority | Fulfilled | Comments |
|---|---|---|---|
| It is clear how the tool is funded to avoid improper influence due to conflicts of interest | M | 1 | It was developed during a competition and it does not receive funding anymore |
| The tool is compliant with relevant legal and regulatory requirements. | S | 1 | Under the apache 2.0 license |
| The tool adheres to (local) data privacy regulations like GDPR, ensuring the protection of user data. | S | 1 | Data will stay local |
| The tool implements appropriate security measures to comply with industry regulations and standards. | S | 0 | The repo does not speak about security at all |
| The tool is licensed for use within the organization according to the terms and conditions of the license agreement. | S | 1 | Under the apache 2.0 license |
| The tool respects intellectual property rights and avoid copyright infringement issues. | S | 1 |
total_score = 16