AI Model Validation and Edge Computing

July 26, 2022 1 Comment

By Stephen LaBella, Director of Government Sales

Artificial intelligence and machine learning in edge computing require testing data sets to provide accuracy and efficiency. How can this be done? Processing data sets and information through a model to make sure it achieves its intended purpose is done through a process known as model validation.

Model validation is the process that is carried out after model training, where the trained model is evaluated with a testing data set. The test data may or may not include the same data set from which the training set is procured. There are two types of Model validation techniques to analyze information from data sets for models. They are in-sample validation and out-of-sample validation.

In-sample validation is testing with the same dataset that is used to build the model. Out-of-sample validation is testing the data from a new dataset that is not used to build the model.

We can analyze how effective our model is and if it achieves its intended purpose by model validation. The main goal of a machine learning model is to learn from the training examples so that the model can generalize the learning and/or new information to new instances that it has not yet seen. It is important to follow several steps to have a successful model validation. These include the following:

1. Choosing a machine learning algorithm
2. Choosing hyperparameters for the model
3. Making sure to “fit” the model for training data
4. Use the model to predict labels for new data

In machine learning (ML), the term parameters refer to something that can be learned by the algorithm during training, and hyperparameters refers to something that is passed to the algorithm. Once the accuracy score for the model is tested, if the score is low, we can change the value of the hyperparameters used and retest until we get a decent accuracy rating.

There are various forms of model validation techniques. These include:

• Train/test split
• K-fold cross-validation
• Leave-one-out cross validation
• Leave-one-group-out cross validation
• Nested cross validation
• Time-series cross validation
• Wilcoxon signed-rank test
• McNemar’s test
• 5X2CV paired t-test
• 5X2CV combined f-test

There are various ways of validating a model and no single method works in all scenarios. It is imperative, however, that the US Department of Defense (DOD) prioritize safe, secure, trusted, and responsible AI development and deployment for model validation.  Developing artificial intelligence for national security and defense requires state-of-the-art solutions with the most cutting-edge innovations. One Stop Systems is addressing the priorities for AI development with solutions that strengthen the nexus between the US DOD and industry partners to achieve the optimum level of AI/ML capabilities.

AI Applications

In many artificial intelligence firms, there are challenges to demonstrating the success of model performance in support of intended operational objectives. Many firms recognize these challenges and focus on solutions for model validation and testing at the edge.

CalypsoAI has built VESPR Validate to provide AI developers and adopters with testing to ensure models are validated and deployment ready. VESPR Validate tests under several different categories including data corruptions, weather simulation, and adversarial threats. Cybersecurity for machine learning algorithms is different from traditional cyber domains. Forms of testing to support this are data privacy protection and model evasion attack algorithms.

By testing model validation at the edge, firms can compare a sample of their own data and watch how algorithms perform against another set of tests. For example, an organization may build an object recognition algorithm to identify ships, but simultaneously wants to know if the algorithm will successfully identify a ship underwater. Edge computing allows for independent AI model verification and validation. The benefits include demonstrating model performance thresholds relative to operational conditions and models are not easily degraded by adversarial attacks.

Calypso's VESPR Validate provides AI developers and adopters automation tools to test and validate AI/ML models. CalypsoAI addresses one of the biggest issues facing AI --operationalizing AI models. Through VESPR Validate, decision-makers gain an understanding of how a model will perform under operational conditions. Validate runs perturbations against training data to measure how a model will perform under degraded conditions. The result is an explainable outcome that is understood by all stakeholders and provides a set of standards providing an acceptable range by which a model may be trusted to be deployed.

VESPR Validate also tests against adversarial attacks on data and models. Running security tests is often omitted in the testing process and is foundational to trusting artificial intelligence results.

CalypsoAI’s and One Stop Systems mission is to build trust in AI through independent testing and validation. We solve one of the biggest issues facing AI: machine learning models not getting deployed into production. Through CalypsoAI’s automated testing and validation solution, decision-makers gain the performance visibility needed to confidently deploy their models into production. This ensures the success of the AI strategy while drastically reducing the amount of risk, time, and money spent manually testing and validating models.

AI and ML algorithms can analyze data much faster and more thoroughly than human operators, enabling faster and more accurate decision-making. There are many challenges to operationalizing AI/ML. Nevertheless, to succeed at implementing this technology, results must make AI trustworthy. Trust in AI is built upon two pillars, robustness and explainability: 

1. There must be the certainty that AI can’t be deceived by adversaries. There is a high chance that adversaries are already manipulating scenes and pixels to delude AI, both to avoid being recognized/identified (e.g., missile launching systems), as well as to create fake images (e.g., create a fake bridge to disrupt a mission).
2. Decision-makers need to understand how the AI thinks and how it reaches the desired outcome, otherwise, they will not accept it. Without the buy-in of the decision-makers, AI may never reach its full potential.

One Stop Systems Solutions

Rigel Edge Supercomputer is One Stop Systems’ HPC solution, ideal for performing model validation tests for independent AI model validation and verification. Rigel’s optimized PCIe architecture enables maximum data throughput, avoiding bandwidth bottlenecks. Rigel’s lightweight, rugged, thermally optimized, and compact design brings the power of the data center to the rugged edge, whether airborne, marine, or terrestrial.   

GAS-Rugged - HPC Solution features the most advanced GPU, allowing customers to consolidate training, inference, and analytics into a unified deployable AI solution at the edge.  With available rack and flange mounting options, adaptable power subsystem and unheard-of performance in a modest 23” depth aluminum enclosure, the GAS-R excels in demanding autonomous driving, vehicle, and airborne applications with true “Datacenter in the Sky” capability.

As demand continues to grow for high-performance computing solutions at the edge with GPU technology, the United States Department of Defense continues to find trusted solutions for defense needs based on their model validation, verification, and accreditation standards.

The US DOD is preparing new rules and regulations as model validation and verification will plays a larger role in autonomous systems and military solutions for defense. Updated autonomous weapons rules coming for the Pentagon: Exclusive details - Breaking Defense.

Artificial intelligence/machine learning and data science models can generate the correct output if the exact question is asked to produce this output. However, there is a concern about the lack of trust in these models. This is where model validation and verification come into play. It provides trusted results generated by these models by both mathematical and logical comparison with the actual output. Model validation testing at the edge provides for different tests and data, by checking the validity of the model in a looping manner. The process of validity checking automated at the edge is a big advantage to confirm the trust in the AI/ML model.

Click the buttons below to share this blog post!

Return to the main Blog page

1 Response

Leave a comment

Comments will be approved before showing up.

Name *

Email *

Comment *

---

Also in One Stop Systems Blog