I wanted to share some of the results with you. To determine Which Python library gives us faster speed, we did a bulk insert into Aurora PostgreSQL using various library and result are mentioned in video

Article
[ Ссылка ]

GitHub Code:

[ Ссылка ]


Conclusion
SqlAlchemy should always be the primary option when working with and entering bulk items into AWS Aurora because it is obvious from testing that it is a faster approach to insert data into Aurora PostgreSQL. When compared to psycopg2(executemany), VS SqlAlchemy is almost 60 to 70% faster. Comparing batch Size 30,000 Bulk Insert using psycopg2(executemany) we found it took around 1248 seconds vs when using psycopg2(execute_batch_method ) took 19.4 seconds VS SQLAlchemy took only 1.5 seconds.



References

Aurora PostgreSQL Insert Many Performances Test Using Various Python Library. Accessed 27 Oct. 2022.

“SQLAlchemy.” Pypi, pypi.org/project/SQLAlchemy. Accessed 27 Oct. 2022.

“Psycopg2.” Psycopg2, pypi.org/project/psycopg2. Accessed 27 Oct. 2022.

“Pandas to PostgreSQL Using Psycopg2: Bulk Insert Performance Benchmark.” Naysan, naysan.ca/2020/05/09/pandas-to-postgresql-using-psycopg2-bulk-insert-performance-benchmark. Accessed 27 Oct. 2022.

“Improve Your Psycopg2 Executions for PostgreSQL in Python.” Datacareer, www.datacareer.de/blog/improve-your-psycopg2-executions-for-postgresql-in-python. Accessed 27 Oct. 2022.


#aws #cloud #cloudcomputing #azure #devops #technology #python #amazonwebservices #linux #amazon #programming #awscloud #cybersecurity #coding #googlecloud #developer #kubernetes #bigdata #datascience #microsoft #machinelearning #software #java #tech #it #gcp #awstraining #javascript #security #docker

свернуть