How can I check if a record exists when passing a dataframe to SQL in pandas?

Background I'm building an application that passes data from a CSV to a MS SQL database. This database is being used as a repository for all my enterprise's records of this type (phone calls)....

basic pyodbc bulk insert

In a python script, I need to run a query on one datasource and insert each row from that query into a table on a different datasource. I'd normally do this with a single insert/select statement...

Flask-SQLAlchemy - how do sessions work with multiple databases?

I'm working on a Flask project and I am using Flask-SQLAlchemy. I need to work with multiple already existing databases. I created the "app" object and the SQLAlchemy one: from flask import...

one hot encoding for frequent values only

I am looking to do one hot encoding to a column, but only for those that are very frequent. All that are below a threshold T will be put in their own category. My strategy was to create a...

pyodbc.connect() works, but not sqlalchemy.create_engine().connect()

I am attempting to write a Python script that can take Excel sheets and import them into my SQL Server Express (with Windows Authentication) database as tables. To do this, I am using pandas to...

How to install pymssql on MacOS Sierra

Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-J1I0ox/pymssql/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n',...

AttributeError: module 'odbc' has no attribute 'connect' - python with pydev

I am very new to python and I just can't seem to find an answer to this error. When I run the code below I get the error AttributeError: module 'odbc' has no attribute 'connect' However, the...

Speeding up pandas.DataFrame.to_sql with fast_executemany of pyODBC

I would like to send a large pandas.DataFrame to a remote server running MS SQL. The way I do it now is by converting a data_frame object to a list of tuples and then send it away with pyODBC's...

Creating a python spark dataframe from pyodbc rows

I'm trying to do something simple here. Get rows from pyodbc and use this as an input for creating a dataframe import pyodbc import sys import csv connection = pyodbc.connect("DSN=MySQL") cursor...

Pandas to_sql changing datatype in database table

Has anyone experienced this before? I have a table with "int" and "varchar" columns - a report schedule table. I am trying to import an excel file with ".xls" extension to this table using a...

"Insert Into" statement causing errors due to "Parameter 7 (""): The supplied value is not a valid instance of data type float."

I'm loading a batch of CSV files into a SQL Server table using Python one row at a time. The files each contain a number of free text fields and erroneous data which I trim and rename before...

'pyodbc.Cursor' object has no attribute 'fast_executemany'

I am having a weird problem, I have a web app that is using fast_executemany in order to insert into the database , when running it on the local host it works with no problem but when deploying it...

Connection problem Linux Apache+PHP to MS SQL Server using WindowsAuth / Kerberos ("No Kerberos credentials available")

Following setup: I have a Linux (Debian) machine as a webserver (apache) and a MS SQL Server on Windows. Furthermore I am not able to get a SQL user for this, only authentication through...

Python Flask as Windows Service

I am trying to get a Flask app to run as a Service in Windows. I have already tried to implement a solution as suggested here and here without success. I have a simple folder with just two...

How do I access my Azure SQL database (not in a container) from within a docker container

I have a SQL Server running on docker. I'm trying to connect to it from a web app written in python3 with pymssql. It's working until I run the same app inside a docker container. I get this...

pyodbc module not found

I am facing a situation where modules like pyodbc which used to work earlier (3 weeks back) are not working anymore in my laptop. I am repeatedly getting Module not found error. To resolve it by...

Delete rows with pyodbc freezes and doesn't delete rows

I am writing a script to remove rows from a SQL Azure database. When I execute this script in Jupyter notebook it runs forever and never deletes the rows. Any ideas why this script is not...

Can't open lib 'ODBC Driver 17 for SQL Server' : file not found

I am fairly new to Python and Azure web apps. Any help is appreciated. My setup: Program: Visual Studio code Language: Python-Flask Cloud provider: Microsoft Azure Database: Azure SQL...

Change where pyodbc expects libodbc.2.dylib to live (changing default odbc file locations)

When importing pyodbc ❯ python >>> import pyodbc Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError:...

Python - connect to hive using Pyodbc

I am trying to connect to hive server using PyODBC however I am getting error. Not sure how to resolve it. HiveServer is running on EMR. Code import pyodbc cnxnstr =...

Django and Azure SQL key error 'deferrable' when start migrate command

I try to connect Django to Azure SQL and have the error KeyError: deferrable when I start to migrate command. I can't find a resolution for this issue. I use this...

Connecting to SQL Server database from Windows on ARM64

I'm using a Surface Pro X (2020), and attempting to connect to a remote SQL Server database. I have installed the ODBC drivers (which are x86 at this point, but installed fine under emulation)....

How to write pandas dataframe to HDFS cluster using pyodbc?

I am using DSN to connect my local python to an HDFS cluster: import pyodbc with pyodbc.connect("DSN=CDH_HIVE_PROD", autocommit=True) as conn: df = pd.read_sql("""Select * from...

python packages are not installed throughCI/CD pipelines,import error , package not found

Introduction and Explaination i am trying deploy my code to azure function app (linux based , python code ) through CI.CD pipelines. at first it worked fine but after few deployments, i am import...

Getting Error "[01000] [unixODBC][Driver Manager]Can't open lib 'SQL Server Native Client 11.0' : file not found (0) (SQLDriverConnect)

I am trying to connect to SQL Server Database. This is my script: import sys import pandas as pd import pysftp import pyodbc sq_conn = pyodbc.connect("Driver={SQL Server Native Client 11.0};" ...

Python connect to ODBC 2.x database

I'm trying to connect to an old database that has its own equally old ODBC driver (which is no longer maintained) # The driver is 32 bit, so I'm using Python 3.8.7 (32bit) con =...

Pyodbc on M1 Macs

I am trying to connect to a Microsoft sql server database using pyodbc. I keep getting the error Error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server'...

Python ODBC (pyodbc) connection to MSSQL server fails with error "Can't open lib ... file not found"- but driver file is there

I am trying to get pyodbc to connect to SQL Server with this test script: import pyodbc server = 'ms.sql.server' database = 'database' username = 'username' password =...

Client unable to establish connection, macOS, django, odbc 17, pyodbc

Everything was working until I restarted my laptop. I have this setup: python 3.8.9 django 3.1.1 pyodbc 4.0.30 pyodbc.drivers() shows this: ['ODBC Driver 17 for SQL Server'] SQLServer openssl...

spyder's kernel dies when I try to make a plot

I just updated spyder to 5.1.5 and all it takes is these 4 lines of code to break the kernel: import numpy as np import matplotlib.pyplot as plt arr = np.arange(0,100) plt.plot(arr) Other plots...