Page 1 of 1

Slow Import - Bad SQL or What???

PostPosted: Mon Aug 03, 2015 6:33 pm
by Osensnolf
Importing data into my database is very slow. I have several shared servers with old Interspire Email Marketer on it and they are faster than my primary Interspire Email Marketer which has 32GB RAM.

I am afraid that this is going to cause more problems later.

It imports about 10 records per second.

My other Interspire Email Marketer's do 80-110 records per second using the same test data. All servers have the same Interspire Email Marketer suppression list and about the same amount of contacts.

Is something wrong or does something need to be optimized??

Anyone have any suggestions?

(Feeling like Maborak has left Interspire Email Marketer addons.)

Re: Slow Import - Bad SQL or What???

PostPosted: Thu Aug 06, 2015 10:29 am
by mmm
What is your DBF setup?

Re: Slow Import - Bad SQL or What???

PostPosted: Mon Aug 31, 2015 3:30 pm
by MaBoRaK
Osensnolf wrote:Importing data into my database is very slow. I have several shared servers with old Interspire Email Marketer on it and they are faster than my primary Interspire Email Marketer which has 32GB RAM.

I am afraid that this is going to cause more problems later.

It imports about 10 records per second.

My other Interspire Email Marketer's do 80-110 records per second using the same test data. All servers have the same Interspire Email Marketer suppression list and about the same amount of contacts.

Is something wrong or does something need to be optimized??

Anyone have any suggestions?

(Feeling like Maborak has left Interspire Email Marketer addons.)


Everything depends of your Subscribers data. The process to import is:

- Check if the email is valid ( is only php code )
- Check if the email is a bounce ( mysql process #1 )
- Check if the email has been suppressed ( mysql process #2 )
- Another validation with the database ( mysql process #3 )
- Once all validations passed, the email is configured with the default custom fields ( mysql process #4 )

Mysql process #1 and #2 are the common BOTTLENECK in the import process, because the email needs to be compared with a large bounce table or/and large suppression table.

Re: Slow Import - Bad SQL or What???

PostPosted: Mon Aug 31, 2015 5:09 pm
by Osensnolf
After not receiving a reply, I decided to install a new version of Interspire Email Marketer. I then imported almost all of the data from the tables in the old (slow) Interspire Email Marketer SQL into the new SQL and the problem was solved.

So far the speed is as fast as it should be. Not sure what was causing it to be slow.

Good to know in case anyone else ever has this issue.

Re: Slow Import - Bad SQL or What???

PostPosted: Tue Sep 01, 2015 7:11 pm
by mmm
If you have a backup of that slow Interspire Email Marketer, for testing purposes, it would be great to learn if these would help:
a) cpanel dbf fix
b) phpMyAdmin dbf fix
c) Interspire Email Marketer dbf fix

Re: Slow Import - Bad SQL or What???

PostPosted: Wed Sep 02, 2015 2:54 am
by MaBoRaK
loading......

I have a beta import working with CLI in Installer 4.5 >

Try with:

Code: Select all
installer/api/cli.php --import --file=THEVFILE.CSV --listid=THELISTID

For now, the import works only with single emails and not with multi-column CSV files.

connection closed.

Re: Slow Import - Bad SQL or What???

PostPosted: Wed Sep 02, 2015 3:41 pm
by mmm
[INFO][error]: Cannot open the file

Perms are 644.

What is the required format of the file,
as having only emails in there did not work,
as having "Email Address" in the header and all emails in quotation marks did not work either (as in the exported files),
as having "Email Address" in the header and all emails in quotation marks and each line end with a comma did not work either (as in the exported files).

Ideally, if you could, please always provide examples, that would be much appreciated.
:)