|
This falls under the 'checking to make sure I'm not leaving my drink on the roof of my car before driving off' category of suggestions.
You may want to make sure the database connection is not closing and reconnecting between each record. This is something I've run into back in my old Access days. 
|
|
|
|
|
I had a similar issue where I needed to import 10's of thousands of records every day, most of them duplicates. We only inserted if the record didn't already exist. It was a serious performance bottleneck.
I generated hashes for all of the existing rows and added an index for the new hash column. Now I calculate the hash of the values in each new row to be inserted and compare the hash to existing rows. I skip the insert if an existing identical row already exists. Using the index solved the insert performance problem for me.
I don't recommend using MD5 or similar simple hashes because it's too easy to create duplicate hashes.
|
|
|
|
|
- Figure out the bottle neck.
Since it does not happen in Dev, but does in production, I suspect it is the duplicate checking. If you size dev the same as production will it reproduce in dev?
Also, do you have more fault tolerance/active nodes in production? If you are doing a few thousand small transactions in a high availability setup, it will take longer.
Do you perform all of your duplicate checks in a single query? Or one import row at a time?
If you setup good, unique indexes, then you can skip the duplicate checking and let the DB do it for you. This would point you toward a commit per import row.
We had an import feature that ended up with an unnecessary exponential complexity. It was fine during testing but started getting really slow at only 150 rows.
|
|
|
|
|
#Worldle #267 1/6 (100%)
🟩🟩🟩🟩🟩🎉
https://worldle.teuteuf.fr
easy one
"A little time, a little trouble, your better day"
Badfinger
|
|
|
|
|
Wordle 484 6/6
🟨⬛⬛🟨🟩
⬛🟨🟨⬛🟩
🟩⬛🟩⬛🟩
🟩⬛🟩⬛🟩
🟩⬛🟩🟨🟩
🟩🟩🟩🟩🟩
|
|
|
|
|
Wordle 484 2/6
🟨🟨⬜🟩🟩
🟩🟩🟩🟩🟩
My first 2!
Software rusts. Simon Stephenson, ca 1994. So does this signature. me, 2012
|
|
|
|
|
Wordle 484 3/6
⬜🟨⬜⬜⬜
⬜🟨🟩⬜🟩
🟩🟩🟩🟩🟩
Not my first 3 but still proud
"A little time, a little trouble, your better day"
Badfinger
|
|
|
|
|
Wordle 484 4/6
🟩⬛🟩⬛🟩
🟩⬛🟩⬛🟩
🟩🟩🟩⬛🟩
🟩🟩🟩🟩🟩
No comment! 
|
|
|
|
|
Wordle 484 5/6
🟨🟨⬜🟨⬜
🟨🟨🟩⬜⬜
⬜⬜🟩🟩🟩
⬜⬜🟩🟩🟩
🟩🟩🟩🟩🟩
|
|
|
|
|
Wordle 484 5/6
⬜🟨⬜⬜⬜
⬜⬜⬜🟨🟩
🟩⬜🟩⬜🟩
🟩🟩🟩⬜🟩
🟩🟩🟩🟩🟩
The order of my last two guesses was a coinflip
"If we don't change direction, we'll end up where we're going"
|
|
|
|
|
Wordle 484 4/6
🟨⬜⬜🟨🟩
🟩🟩🟩⬜🟩
🟩🟩🟩⬜🟩
🟩🟩🟩🟩🟩
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
Wordle 484 4/6
⬛⬛🟩⬛🟩
🟩🟩🟩⬛🟩
🟩🟩🟩⬛🟩
🟩🟩🟩🟩🟩
|
|
|
|
|
Wordle 484 5/6
⬜⬜⬜⬜🟨
🟨⬜⬜⬜🟩
🟩🟨⬜⬜🟩
🟩⬜🟩🟨🟩
🟩🟩🟩🟩🟩
|
|
|
|
|
Wordle 484 6/6
⬜⬜🟩⬜🟩
🟩⬜🟩⬜🟩
🟩⬜🟩⬜🟩
🟩⬜🟩⬜🟩
🟩⬜🟩⬜🟩
🟩🟩🟩🟩🟩
Phew
Life should not be a journey to the grave with the intention of arriving safely in a pretty and well-preserved body, but rather to skid in broadside in a cloud of smoke, thoroughly used up, totally worn out, and loudly proclaiming “Wow! What a Ride!" - Hunter S Thompson - RIP
|
|
|
|
|
|
But not so for iron, which is ironic.
|
|
|
|
|
|
I thought the adjective for iron was "feral".
|
|
|
|
|
So should the adjective for copper be capric?
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows.
-- 6079 Smith W.
|
|
|
|
|
I think this thread is very, very thin.
"A little time, a little trouble, your better day"
Badfinger
|
|
|
|
|
Ok! Just very thin.
"A little time, a little trouble, your better day"
Badfinger
|
|
|
|
|
|
Now do AI code generation. 
To err is human. Fortune favors the monsters.
|
|
|
|
|
Use every open-access Github repository as training? 
|
|
|
|
|
Please post it in Free Tools Discussion Boards[^] as well, here will get in the second page really, really fast.
M.D.V. 
If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about?
Help me to understand what I'm saying, and I'll explain it better to you
Rating helpful answers is nice, but saying thanks can be even nicer.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
