Quantization of Onnx model

I am trying to quantize an ONNX model using the onnxruntime quantization tool. My code is below for quantization: import onnx from quantize import quantize, QuantizationMode # Load the onnx model...

ONNX Runtime C# does not remember the state of LSTM networks

I exported a trained LSTM neural network from this example from Matlab to ONNX. Then I try to run this network with ONNX Runtime C#. However, it looks like I am doing something wrong and the...

NN model format conversion tflite -> onnx

I'd like to convert the hosted models TensorFlow-Lite hosted models mainly the mobilenets into ONNX format. So I'd like to try the quantized version of those hosted models and run them with...

Inference of onnx model (opset11) in Windows 10 c++?

In order to use my custom TF model through WinML, I converted it to onnx using the tf2onnx converter. The conversion finally worked using opset 11. Unfortunately I cannot load the model in the...

Why cant I use ONNX Runtime training with pytorch?

When I run from onnxruntime.capi.ort_trainer import ORTTrainer as stated at https://github.com/microsoft/onnxruntime/#training-start, I get this error: ModuleNotFoundError: No module named...

How to merge two ONNX deep learning models

I have two models that are in ONNX format. Both models are similar (both are pre-trained deep learning models, ex. ResNet50 models). The only difference between them is that the last layers are...

Setting up ONNX Runtime on Ubuntu 20.04 (C++ API)

I am currently in the midst of trying to get my image processing programs to work on Ubuntu (coming from windows). I have successfully built and linked the OpenCV and Boost libraries to work with...

Trouble building onnxruntime from source - FileNotFoundError

I am trying to build onnxruntime from source, but I run into the following problem. After executing git clone https://github.com/microsoft/onnxruntime.git cd onnxruntime ./build.bat I run into...

What is tensorflow concrete function outputs correspond to structured_outputs?

I trained my customized ssd_mobilenet_v2 using TensorFlow2 Object Detection API. After training completed, I used exporter_main_v2.py to export a saved_model of my customized model. If I load...

How do you run a ONNX model on a GPU?

I'm trying to run an ONNX model import onnxruntime as ort import onnxruntime.backend model_path = "model.onnx" #https://microsoft.github.io/onnxruntime/ ort_sess =...

How to know input/output layer names and sizes for Pytorch model?

I have Pytorch model.pth using Detectron2's COCO Object Detection Baselines pretrained model R50-FPN. I am trying to convert the .pth model to onnx. My code is as follows. import io import numpy...

Unsupported ONNX opset version: 11

I'm following this guide to convert darknet to onnx. However, I'm facing the following error: "C:\Users\Scott\Anaconda3\envs\pytorch_yolov4\lib\site-packages\torch\onnx\symbolic_helper.py", line...

Difference between WinML and OnnxRuntime for WPF in C#

To package trained Onnx models with a WPF .Net Core 3.1 app, I'm wondering if there are any difference to these two methods: Microsoft.ML.OnnxRuntime and Microsoft.AI.MachineLearning (WinML)?...

Memory corruption when using OnnxRuntime with OpenVINO on the Intel MyriadX and Raspberry Pi 4B

I'm trying to run Inference on the Intel Compute Stick 2 (MyriadX chip) connected to a Raspberry Pi 4B using OnnxRuntime and OpenVINO. I have everything set up, the openvino provider gets...

Unable to load DLL 'onnxruntime' or one of its dependencies ML.NET

I get this onnx from Python but I received error in context.Transforms.ApplyOnnxModel() - "Unable to load DLL 'onnxruntime' or one of its dependencies: The specified module could not be found....

How to do multiple inferencing on onnx(onnxruntime) similar to sklearn

I want to infer outputs against many inputs from an onnx model using onnxruntime in python. One way is to use the for loop but it seems a very trivial and a slow method. Is there a way to do the...

How to use ONNX model in C++ code on Linux?

I train some Unet-based model in Pytorch. It take an image as an input, and return a mask. After training i save it to ONNX format, run it with onnxruntime python module and it worked like a charm...

Yolov4 onnxruntime C++

I need to deploy a yolov4 inference model and I want to use onnxruntime with tensorRT backend. I don't know how to post process yolov4 detection result in C++. I have a sample written in python...

How do I convert to ONNX a Spark ML model with multiple input columns and use it for scoring dynamic batch size?

I converted a logistic regression model with dynamic batch size from Spark ML to ONNX using this: initial_types = [('Features', FloatTensorType([None, 5]))] onnx_model = convert_sparkml(s_clf,...

Can I combine two ONNX graphs together, passing the output from one as input to another?

I have a model, exported from pytorch, I'll call main_model.onnx. It has an input node I'll call main_input that expects a list of integers. I can load this in onnxruntime and send a list of ints...

Running threads in parallel takes more time then sequential execution in python

I have two ONNX deep learned models. I want to run both the models parallelly. I am using threads from python. But surprisingly it is taking more time then running both the models...

Running Multiple ONNX Model for Inferencing in Parallel in Python

Is there a way to run multiple ONNX models in parallel and use multiple cores available? Currently, I have trained two ONNX models and want to infer using them. I have used threading from Python...

ONNX Quantized Model Type Error: Type 'tensor(float16)'

I converted onnx model from float32 to float16 by using this script. from onnxruntime_tools import optimizer optimized_model = optimizer.optimize_model("model_fixed.onnx", model_type='bert_tf',...

How to use onnxruntime with .ort model in Android Studio

I'm trying to create an Android App that incorporates a Machine Learning Model. I had an onnx model, along with a Python script file, two json files with the label names, and some numpy data for...

Parse an ONNX model using C++. Extract layers, input and output shape from an onnx model using c++

I'm trying to extract data like input layers, output layers and their shapes from an onnx model. I know there is python interface to do this. I want to do something similar to this code but in...

Onnxruntime vs PyTorch

I have trained YOLO-v3 tiny on my custom dataset using PyTorch. For comparing the inferencing time, I tried onnxruntime on CPU along with PyTorch GPU and PyTorch CPU. The average running times...

How to get onnx format from pretrained GPT2 models?

I'm trying to transform KoGPT2 model, which is pretrained by GPT2, to onnx format in order to change the model to tensorflow format. I used convert_graph_to_onnx in transformers but it didn't work...

Unexpected input data type. Actual: (tensor(double)) , expected: (tensor(float))

I am learning this new ONNX framework that allows us to deploy the deep learning (and others) model into production. However, there is one thing I am missing. I thought that the main reason for...

Inferencing tensorflow-trained model using ONNX in C++?

I am trying to recreate the work done in this video, CppDay20Interoperable AI: ONNX & ONNXRuntime in C++ (M. Arena, M.Verasani).The github repository for the demo code is here .So far I have...

Problems installing python packages on Mac M1

I want to install python packages listed in the requirements file of my github repo. However, I have problems installing those python packages into my conda environment. First of all, I installed...