Looking for advice. I have a CSV file that's 1.9gb (2.6m rows) and attempted to upload it to MySQL but the server keeps timing out. I've installed Python, ran a script to split the CSV and now have 58 files ready to upload to the database. Any idea how to upload this efficiently as opposed to 1 file at a time?

串文

2024-12-04 09:31

讚

回覆

轉發

作者

粉絲

串文

54+

讚

回覆

轉發

24小時粉絲增長

無資料

互動率

(讚 + 回覆 + 轉發) / 粉絲數

114.29%

回覆 (BETA)

最先回覆的內容
發文後	用戶	內容
2 小時內	_nre_s	if order is not important, just multithread and upload in parallel. Also keep the pointer of last write entry and retry from that point on failure/timeout. 1.9gb/2.6m row is fairly small number to upload.
4 小時內	Aaron Jay aaronmacbaron	Write another python script to automate the process and build a thread pool so that whenever a thread is free you can queue up another script. I’ve done this for a migration from on-prem to AWS from a postgresql 8 to 15.6.
4 小時內	Derek derek_yjk	Use BigQuery
10 小時內	David Vanni dax85x	With phpmyadmin on PHP server i just raised the timeout and file size in php.ini and myadmin info then all went easy, got like 3gb db in one shot
15 小時內	Ahmad Khalil asuramaru2142	Deploy nocodb offline, connect to the remote db, upload csv and let nocodb handle uploading part. Eaaaassssyyyyy
19 小時內	mai el.p.iam	Wondering if it's not simplest to just use python to connect to the server then do a loop to upload. It's not the most elegant but gets the job done in probably the same amount of time trying to set up something more complicated
一天內	Anak Tech Melek Bisnis 💻💡💰 anaktechmelekbisnis	Ask claude 😁