Slow Import - Bad SQL or What???

Forum to discuss Installation and Configuration Issues

Slow Import - Bad SQL or What???

Postby Osensnolf » Mon Aug 03, 2015 6:33 pm

Importing data into my database is very slow. I have several shared servers with old Interspire Email Marketer on it and they are faster than my primary Interspire Email Marketer which has 32GB RAM.

I am afraid that this is going to cause more problems later.

It imports about 10 records per second.

My other Interspire Email Marketer's do 80-110 records per second using the same test data. All servers have the same Interspire Email Marketer suppression list and about the same amount of contacts.

Is something wrong or does something need to be optimized??

Anyone have any suggestions?

(Feeling like Maborak has left Interspire Email Marketer addons.)
Osensnolf
 
Posts: 82
Joined: Tue Dec 24, 2013 3:28 pm

Re: Slow Import - Bad SQL or What???

Postby mmm » Thu Aug 06, 2015 10:29 am

What is your DBF setup?
mmm
 
Posts: 106
Joined: Sat Apr 11, 2015 4:36 pm

Re: Slow Import - Bad SQL or What???

Postby MaBoRaK » Mon Aug 31, 2015 3:30 pm

Osensnolf wrote:Importing data into my database is very slow. I have several shared servers with old Interspire Email Marketer on it and they are faster than my primary Interspire Email Marketer which has 32GB RAM.

I am afraid that this is going to cause more problems later.

It imports about 10 records per second.

My other Interspire Email Marketer's do 80-110 records per second using the same test data. All servers have the same Interspire Email Marketer suppression list and about the same amount of contacts.

Is something wrong or does something need to be optimized??

Anyone have any suggestions?

(Feeling like Maborak has left Interspire Email Marketer addons.)


Everything depends of your Subscribers data. The process to import is:

- Check if the email is valid ( is only php code )
- Check if the email is a bounce ( mysql process #1 )
- Check if the email has been suppressed ( mysql process #2 )
- Another validation with the database ( mysql process #3 )
- Once all validations passed, the email is configured with the default custom fields ( mysql process #4 )

Mysql process #1 and #2 are the common BOTTLENECK in the import process, because the email needs to be compared with a large bounce table or/and large suppression table.
CEO in Maborak Technologies and Interspire Addons Developer
MaBoRaK
Site Admin
 
Posts: 120
Joined: Thu Jul 18, 2013 10:41 am

Re: Slow Import - Bad SQL or What???

Postby Osensnolf » Mon Aug 31, 2015 5:09 pm

After not receiving a reply, I decided to install a new version of Interspire Email Marketer. I then imported almost all of the data from the tables in the old (slow) Interspire Email Marketer SQL into the new SQL and the problem was solved.

So far the speed is as fast as it should be. Not sure what was causing it to be slow.

Good to know in case anyone else ever has this issue.
Osensnolf
 
Posts: 82
Joined: Tue Dec 24, 2013 3:28 pm

Re: Slow Import - Bad SQL or What???

Postby mmm » Tue Sep 01, 2015 7:11 pm

If you have a backup of that slow Interspire Email Marketer, for testing purposes, it would be great to learn if these would help:
a) cpanel dbf fix
b) phpMyAdmin dbf fix
c) Interspire Email Marketer dbf fix
mmm
 
Posts: 106
Joined: Sat Apr 11, 2015 4:36 pm

Re: Slow Import - Bad SQL or What???

Postby MaBoRaK » Wed Sep 02, 2015 2:54 am

loading......

I have a beta import working with CLI in Installer 4.5 >

Try with:

Code: Select all
installer/api/cli.php --import --file=THEVFILE.CSV --listid=THELISTID

For now, the import works only with single emails and not with multi-column CSV files.

connection closed.
CEO in Maborak Technologies and Interspire Addons Developer
MaBoRaK
Site Admin
 
Posts: 120
Joined: Thu Jul 18, 2013 10:41 am

Re: Slow Import - Bad SQL or What???

Postby mmm » Wed Sep 02, 2015 3:41 pm

[INFO][error]: Cannot open the file

Perms are 644.

What is the required format of the file,
as having only emails in there did not work,
as having "Email Address" in the header and all emails in quotation marks did not work either (as in the exported files),
as having "Email Address" in the header and all emails in quotation marks and each line end with a comma did not work either (as in the exported files).

Ideally, if you could, please always provide examples, that would be much appreciated.
:)
mmm
 
Posts: 106
Joined: Sat Apr 11, 2015 4:36 pm


Return to Installation and Configuration

 


  • Related topics
    Replies
    Views
    Last post

Who is online

Users browsing this forum: No registered users and 1 guest