Support archive | go to Low

exp_low_reorder_orders randomly erases

Bransin Anderson 2 Jan 2014 23:37 problem, pending

This has been a recurring issue that seems to happen at random. It's happened about 3 times in the last year. No entries return using the exp:low_reorder:entries tag.

The exp_low_reorder_orders table seems to erase itself for some strange reason. A quick fix is to resave the set and the exp_low_reorder_orders table repopulates again. However, this is not a permanent fix and we need to get to the bottom of why this might be happening.

Any ideas?

EE 2.6.1 20130506
Low Reorder 2.2.1

Set Label: Entries: Products, Product Order
Set name: set_1
Set ID: 1
New entries: Prepend
Check "Clear cache" checkbox?: yes
Show expired entries: yes
Show future entries: yes
Show sticky entries: no
Channels: Products
Statuses: Open
Categories: Show entries per single category. Configurator, Products

Replies

Low 3 Jan 2014 09:47

So there's no way to trigger this behaviour? Does it happen after saving an entry? What extensions do you have installed that make use of the entry_submission_end hook?

Also, can you try and update to LR 2.2.2 just to be sure?
Bransin Anderson 7 Jan 2014 22:24

We'll go ahead and update LR to 2.2.2 to be sure. It's very difficult to find when it happens randomly like this. The client doesn't even know when exactly this happens.

The following extensions use the entry_submission_end hook.

Super_search_ext, priority 10, version 2.1.3
Low_reorder_ext, priority 5, version 2.2.1
Structure_ext, priority 10, version 3.3.12
Ce_cache_ext, priority 9, version 1.9

What I have noticed is that it fails on the row where the cat_id is zero. Maybe that's just coincidental.
Low 8 Jan 2014 12:49

Hmm, the add-on should remove the category 0 row for a set that has "Show entries per single category". I'll need to take a look myself. Please send SuperAdmin login credentials to hi at gotolow dot com if you can.
RealJustinLong 8 Jan 2014 13:32

Hi Low,

I have just sent Super Admin details over to you.

Cheers,
Justin
Bransin Anderson 19 Feb 2014 00:41

This has occurred again. This time without the cat_id 0 being stored.

Extensions that use the entry_submission_end hook with priority. Low reorder executes first. Perhaps a later priority will have an impact?

Low_reorder_ext: 5
CE_cache_ext: 9
Super_search_ext: 10
Structure_ext: 10

I've scoured the error logs and haven't found anything yet. Also found that it stores about 9-10 records in the exp_low_reorder_orders and gives up. Trying to determine if I can narrow this to a specific category or entry causing this.

There are some category names that have the pipe character. In what order are the cat_id's stored in the exp_low_reorder? I'm trying to figure out if it's a particular cat_id or entry that is causing this.

We run DataGrab every hour to update existing entries. The option, "Update existing entries is set" when a duplicate is found. There is no entry_submission_end hook with this module however.

Any other insights would be helpful.
Low 19 Feb 2014 09:49

Hmm. The category names or order in which they are saved shouldn't matter at all. I'm dealing with cat IDs I'm getting from EE's API or from the POST itself.

What is your auto_assign_cat_parents setting?

The strange thing is, the only place where sets are completely removed like that, is when there are no entries at all in it, or when you explicitly remove a set in the CP. You can try and remove line #228 in ext.low_reorder.php, which reads
```
ee()->low_reorder_order_model->remove_rogues();
```
...but should only remove rows that have a single pipe in the sort_order field.

Also, the order of the extension priority shouldn't matter. In fact, I reckon it's even better if LR is the first one in the list.

It almost looks like something else is removing the rows...
Bransin Anderson 6 Mar 2014 18:11

Hi Low.

Was able to narrow it down to the Channel Entries API. I'm able to replicate this locally when running a DataGrab import manually.

DataGrab uses the Channel Entries API update_entry method for each entry in a function around line 501 datagrab_model.php.

Api_channel_entries.php
$this->api_channel_entries->update_entry

Note: update_entry is now deprecated, and it simply calls the method 'save_entry'. Save_entry uses the entry_submission_end.

The save_entry method calls the function _update_entry which deletes the category_posts around line 1730.
and then runs the function _sync_related which saves the categories for that entry.

entry_submission_end is the last thing to run for each entry which triggers the low reorder extension to update the data in exp_low_reorder_orders.

The DataGrab import works correctly, and the entries have their category assignments. I'm still trying to figure out where the choke is happening. Not sure yet if this is an issue with DataGrab, an EE API issue, or Low Reorder. Maybe a combination of all 3 :P

Bransin Anderson 6 Mar 2014 19:48

print_r($new_orders) in ext.low_reorder.php when running DataGrab.
This is a long list, but here is an excerpt where we can see the categories lose their entries on each pass. Categories 110 and 291 for example. However, once finishing the DataGrab import, all entries have their categories.

I've also disabled all extensions that use the entry_submission_end except Low_reorder_ext. The entry_submission_end is within the channel entries api function update_entry which is used by DataGrab on each entry.

print_r($new_orders) in Low Reorder when running DataGrab import. The entries wither away from the categories.

 
 Array 
 ( 
     [1-299] => | 
 ) 
 Array 
 ( 
     [1-90] => |259|258| 
     [1-89] => |259|258| 
     [1-38] => |259|258| 
 ) 
 Array 
 ( 
     [1-90] => |259| 
     [1-89] => |259| 
     [1-38] => |259| 
 ) 
 Array 
 ( 
     [1-90] => | 
     [1-89] => | 
     [1-38] => | 
 ) 
 Array 
 ( 
     [1-39] => |267|266|265|264|261| 
 ) 
 Array 
 ( 
     [1-39] => |267|266|265|264| 
 ) 
 Array 
 ( 
     [1-39] => |267|266|265| 
 ) 
 Array 
 ( 
     [1-39] => |267|266| 
 ) 
 Array 
 ( 
     [1-39] => |267| 
 ) 
 Array 
 ( 
     [1-39] => | 
 ) 
 Array 
 ( 
     [1-92] => |274|273|272| 
     [1-68] => |274|273|272| 
 ) 
 Array 
 ( 
     [1-92] => |274|273| 
     [1-68] => |274|273| 
 ) 
 Array 
 ( 
     [1-92] => |274| 
     [1-68] => |274| 
 ) 
 Array 
 ( 
     [1-92] => | 
     [1-68] => | 
 ) 
 Array 
 ( 
     [1-297] => |1500| 
 ) 
 Array 
 ( 
     [1-296] => |404| 
 ) 
 Array 
 ( 
     [1-296] => | 
 ) 
 Array 
 ( 
     [1-93] => | 
     [1-292] => | 
 ) 
 Array 
 ( 
     [1-94] => | 
     [1-33] => | 
 ) 
 Array 
 ( 
     [1-96] => |466|288|284|286|285|283|282|281|279|278| 
     [1-41] => |1600|466|1068|1067|1567|288|285|284|286|533|279|283|282|281|280|278| 
 ) 
 Array 
 ( 
     [1-96] => |466|288|284|286|285|283|282|281|279| 
     [1-41] => |1600|466|1068|1067|1567|288|285|284|286|533|279|283|282|281|280| 
 ) 
 Array 
 ( 
     [1-96] => |466|288|284|286|285|283|282|281| 
     [1-41] => |1600|466|1068|1067|1567|288|285|284|286|533|283|282|281|280| 
 ) 
 Array 
 ( 
     [1-41] => |1600|466|1068|1067|1567|288|285|284|286|533|283|282|281| 
 ) 
 Array 
 ( 
     [1-96] => |466|288|284|286|285|283|282| 
     [1-41] => |1600|466|1068|1067|1567|288|285|284|286|533|283|282| 
 ) 
 Array 
 ( 
     [1-96] => |466|288|284|286|285|283| 
     [1-41] => |1600|466|1068|1067|1567|288|285|284|286|533|283| 
 ) 
 Array 
 ( 
     [1-96] => |466|288|284|286|285| 
     [1-41] => |1600|466|1068|1067|1567|288|285|284|286|533| 
 ) 
 Array 
 ( 
     [1-96] => |466|288|286|285| 
     [1-41] => |1600|466|1068|1067|1567|288|285|286|533| 
 ) 
 Array 
 ( 
     [1-96] => |466|288|286| 
     [1-41] => |1600|466|1068|1067|1567|288|286|533| 
 ) 
 Array 
 ( 
     [1-96] => |466|288| 
     [1-41] => |1600|466|1068|1067|1567|288|533| 
 ) 
 Array 
 ( 
     [1-41] => |1600|466|1068|1067|1567|288| 
 ) 
 Array 
 ( 
     [1-96] => |466| 
     [1-41] => |1600|466|1068|1067|1567| 
 ) 
 Array 
 ( 
     [1-43] => |1601|1600|290| 
 ) 
 Array 
 ( 
     [1-43] => |1601|1600| 
 ) 
 Array 
 ( 
     [1-44] => |292| 
 ) 
 Array 
 ( 
     [1-44] => | 
 ) 
 Array 
 ( 
     [1-99] => |297|296|295|294| 
     [1-45] => |297|296|295|294| 
 ) 
 Array 
 ( 
     [1-99] => |297|296|295| 
     [1-45] => |297|296|295| 
 ) 
 Array 
 ( 
     [1-99] => |297|296| 
     [1-45] => |297|296| 
 ) 
 Array 
 ( 
     [1-99] => |297| 
     [1-45] => |297| 
 ) 
 Array 
 ( 
     [1-99] => | 
     [1-45] => | 
 ) 
 Array 
 ( 
     [1-100] => |299| 
     [1-42] => |1567|299| 
 ) 
 Array 
 ( 
     [1-100] => | 
     [1-42] => |1567| 
 ) 
 Array 
 ( 
     [1-101] => | 
     [1-87] => | 
 ) 
 Array 
 ( 
     [1-102] => | 
     [1-293] => | 
 ) 
 Array 
 ( 
     [1-104] => |308|309|303|307|306|305|304| 
     [1-71] => |309|308|303|307|306|305|304| 
 ) 
 Array 
 ( 
     [1-104] => |308|309|307|306|305|304| 
     [1-71] => |309|308|307|306|305|304| 
 ) 
 Array 
 ( 
     [1-104] => |308|309|307|306|305| 
     [1-71] => |309|308|307|306|305| 
 ) 
 Array 
 ( 
     [1-104] => |308|309|307|306| 
     [1-71] => |309|308|307|306| 
 ) 
 Array 
 ( 
     [1-104] => |308|309|307| 
     [1-71] => |309|308|307| 
 ) 
 Array 
 ( 
     [1-104] => |308|309| 
     [1-71] => |309|308| 
 ) 
 Array 
 ( 
     [1-104] => |309| 
     [1-71] => |309| 
 ) 
 Array 
 ( 
     [1-104] => | 
     [1-71] => | 
 ) 
 Array 
 ( 
     [1-105] => |312|311| 
     [1-72] => |312|311| 
 ) 
 Array 
 ( 
     [1-105] => |312| 
     [1-72] => |312| 
 ) 
 Array 
 ( 
     [1-105] => | 
     [1-72] => | 
 ) 
 Array 
 ( 
     [1-73] => |314| 
 ) 
 Array 
 ( 
     [1-261] => | 
     [1-73] => | 
 ) 
 Array 
 ( 
     [1-107] => | 
     [1-74] => |1105|1104|1103|316| 
 ) 
 Array 
 ( 
     [1-74] => |1105|1104|1103| 
 ) 
 Array 
 ( 
     [1-75] => |497|496|494|318| 
 ) 
 Array 
 ( 
     [1-75] => |497|496|494| 
 ) 
 Array 
 ( 
     [1-109] => |324|323|322|320|321| 
     [1-76] => |324|323|322|321|320| 
 ) 
 Array 
 ( 
     [1-109] => |324|323|322|321| 
     [1-76] => |324|323|322|321| 
 ) 
 Array 
 ( 
     [1-109] => |324|323|322| 
     [1-76] => |324|323|322| 
 ) 
 Array 
 ( 
     [1-109] => |324|323| 
     [1-76] => |324|323| 
 ) 
 Array 
 ( 
     [1-109] => |324| 
     [1-76] => |324| 
 ) 
 Array 
 ( 
     [1-109] => | 
     [1-76] => | 
 ) 
 Array 
 ( 
     [1-113] => |330|329|328|327|326| 
     [1-77] => |330|329|328|327|326| 
 ) 
 Array 
 ( 
     [1-113] => |330|329|328|327| 
     [1-77] => |330|329|328|327| 
 ) 
 Array 
 ( 
     [1-113] => |330|329|328| 
     [1-77] => |330|329|328| 
 ) 
 Array 
 ( 
     [1-113] => |330|329| 
     [1-77] => |330|329| 
 ) 
 Array 
 ( 
     [1-113] => |330| 
     [1-77] => |330| 
 ) 
 Array 
 ( 
     [1-113] => | 
     [1-77] => | 
 ) 
 Array 
 ( 
     [1-110] => |437|333|332| 
     [1-291] => |437|333|332| 
 ) 
 Array 
 ( 
     [1-110] => |437|333| 
     [1-291] => |437|333| 
 ) 
 Array 
 ( 
     [1-110] => |437| 
     [1-291] => |437| 
 ) 
 Array 
 ( 
     [1-110] => | 
     [1-291] => | 
 ) 
 Array 
 ( 
     [1-114] => |440|444|443|441|439| 
 ) 
 Array 
 ( 
     [1-111] => |444|441| 
     [1-114] => |440|444|443|441| 
 ) 
 Array 
 ( 
     [1-114] => |444|443|441| 
 ) 
 Array 
 ( 
     [1-111] => |444| 
     [1-114] => |444|443| 
 ) 
 Array 
 ( 
     [1-114] => |444| 
 ) 
 Array 
 ( 
     [1-111] => | 
     [1-114] => | 
 ) 
 Array 
 ( 
     [1-37] => |337|336| 
 ) 
 Array 
 ( 
     [1-37] => |337| 
 ) 
 Array 
 ( 
     [1-37] => | 
 ) 
 Array 
 ( 
     [1-36] => | 
 ) 
 Array 
 ( 
     [1-79] => |355|354|353|352|351|350|349|348|347|346|342|341|340| 
 ) 
 Array 
 ( 
     [1-79] => |355|354|353|352|351|350|349|348|347|346|342|341| 
 ) 
 Array 
 ( 
     [1-79] => |355|354|353|352|351|350|349|348|347|346|342| 
 ) 
 Array 
 ( 
     [1-79] => |355|354|353|352|351|350|349|348|347|346| 
 ) 
 Array 
 ( 
     [1-79] => |355|354|353|352|351|350|349|348|347| 
 ) 
 Array 
 ( 
     [1-79] => |355|354|353|352|351|350|349|348| 
 ) 
 Array 
 ( 
     [1-79] => |355|354|353|352|351|350|349| 
 ) 
 Array 
 ( 
     [1-79] => |355|354|353|352|351|350| 
 ) 
 Array 
 ( 
     [1-79] => |355|354|353|352|351| 
 ) 
 Array 
 ( 
     [1-79] => |355|354|353|352| 
 ) 
 Array 
 ( 
     [1-79] => |355|354|353| 
 ) 
 Array 
 ( 
     [1-79] => |355|354| 
 ) 
 Array 
 ( 
     [1-79] => |355| 
 ) 
 Array 
 ( 
     [1-79] => | 
 ) 
 Array 
 ( 
     [1-80] => |1498|1496|357|1492| 
 ) 
 Array 
 ( 
     [1-80] => |1498|1496|1492| 
 ) 
 Array 
 ( 
     [1-81] => | 
 ) 
 Array 
 ( 
     [1-271] => |361|360|362| 
 ) 
 Array 
 ( 
     [1-271] => |361|362| 
 ) 
 Array 
 ( 
     [1-271] => |362| 
 ) 
 Array 
 ( 
     [1-271] => | 
 ) 
 Array 
 ( 
     [1-83] => | 
 ) 
 Array 
 ( 
     [1-85] => | 
 ) 
 Array 
 ( 
     [1-41] => |1600|466|1068|1567| 
 ) 
 Array 
 ( 
     [1-41] => |1600|466|1567| 
 ) 
 Array 
 ( 
     [1-41] => |1600|1567| 
     [1-96] => | 
 ) 
 Array 
 ( 
     [1-75] => |497|496| 
 ) 
 Array 
 ( 
     [1-75] => |496| 
 ) 
 Array 
 ( 
     [1-75] => | 
 ) 
 Array 
 ( 
     [1-74] => |1105|1104| 
 ) 
 Array 
 ( 
     [1-74] => |1105| 
 ) 
 Array 
 ( 
     [1-74] => | 
 ) 
 Array 
 ( 
     [1-298] => | 
 ) 
 Array 
 ( 
     [1-307] => | 
 ) 
 Array 
 ( 
     [1-80] => |1498|1496| 
 ) 
 Array 
 ( 
     [1-80] => |1498| 
 ) 
 Array 
 ( 
     [1-80] => | 
 ) 
 Array 
 ( 
     [1-297] => | 
 ) 
 Array 
 ( 
     [1-41] => |1600| 
     [1-42] => | 
 ) 
 Array 
 ( 
     [1-29] => | 
 ) 
 Array 
 ( 
     [1-41] => | 
     [1-43] => |1601| 
 ) 
 Array 
 ( 
     [1-43] => | 
 )

Attached screenshot shows the result after running DataGrab and commenting out the line:
ee()->low_reorder_order_model->remove_rogues()

Bransin Anderson 6 Mar 2014 22:32

Around line 253 in ext.low_reorder.php I have a question with the code below involving the str_replace. If I remove str_replace and assign $new_orders to the value it works.

I'm trying to determine if I would affect anything by changing this?. I updated an existing entry and did not find any duplicates in that sort_order row and ran the DataGrab import again without a hitch. Hrmmmm..
```
 
 if (strpos($val, $needle) !== FALSE) 
 { 
  $new_orders[$key] = str_replace($needle, '|', $val); 
 } 
 
```
to
```
 
 if (strpos($val, $needle) !== FALSE) 
 { 
  $new_orders[$key] = $val;  
 } 
 
```
Low 7 Mar 2014 07:44

Interesting. I'll see if I can get a copy of DataGrab and I'll try to reproduce.

The str_replace is there to remove the entry ID for entries that are no longer assigned to that particular category. For example, entry A has category X and Y, so its ID appears in both X and Y. If you change the categories to X and Z, I'll need to remove the ID from Y. That's what the str_replace does.
Low 7 Mar 2014 11:32

Can you send me an example import file to hi at gotolow dot com? So far my tests have been inconclusive.
Low 7 Mar 2014 17:10

Hmm, running your import and outputting $new_orders just gives me a bunch of empty arrays. Is the output you pasted here the same for each import? What are your DataGrab settings for that import?
Bransin Anderson 7 Mar 2014 17:34

Configure Import in DataGrab

Default Fields
==============
Title: name

Custom Fields
==============
Product Title: name
Mage SKU: sku
Mage Price: base_price
Mage ID: id

Categories
==========
Add categories to the category group: Products (label)

Default category value: null
Category group: Products: null
Category delimiter: ,

Add categories to the category group: Configurator (label)

Default category value: null
Category group: Configurator: null
Category delimiter: ,

Check for duplicate entries
===========================
Use this field to check for duplicates: Mage ID
Update existing entries: Checked
Add a timestamp to this field: Last Import
Delete entries with old timestamp: 86400

Other Settings
==============
Author Field Value: Screen Name
Offset(in seconds): 2
Import in batches: 0

Ran the import again with the print_r($new_orders) around line 267. Array keys are in a different order but the values are the same.
Low 7 Mar 2014 17:47

Pfff, still no dice. What's your DataGrab version? And do you have any add-ons that extend it?
Bransin Anderson 7 Mar 2014 18:03

DataGrab 1.8.0
EE v2.6.1 - Build Date: 20130506
Low Reorder 2.2.2

I've looked and I don't see anything that could extend DataGrab. I did an entire project search to see if these hooks were being used anywhere.

ajw_datagrab_pre_import
ajw_datagrab_modify_data
ajw_datagrab_post_import

The tests I've been doing, I've disabled all extensions that use the entry_submission_end except low reorder. This is so bizarre! Are you able to successfully import the entries?

The import doesn't assign any categories. It updates the entry data that is already assigned to a category and leaves the categories alone. Could that have anything to do with it? More hrmmmmm.
Low 7 Mar 2014 18:08

Very strange. I have been testing with EE 2.7.3, but the rest is the same. Getting expected results there, no errors, no strange receding entry IDs...

I'll try and recreate it in my 2.6 branch and report back later.
Bransin Anderson 7 Mar 2014 18:12

If it helps, I can have this setup in a dev environment where things can be torn apart.
Low 7 Mar 2014 18:17

Sure, send over the login creds for that. Will take a look over the weekend.