Landsat-8 scene_list repeated lines

[maabdelghaffar]

Description:
After executing Landsat8SearchOperator for many times using the new search criteria (startdate-enddate, etc...), repeated records appeared in the result-set as follows:

[.....,
('LC08_L1TP_150037_20170723_20170729_01_T1', 'LC81500372017204LGN00', 'https://s3-us-west-2.amazonaws.com/landsat-pds/c1/L8/150/037/LC08_L1TP_150037_20170723_20170729_01_T1/index.html'), ('LC08_L1TP_150037_20170723_20170729_01_T1', 'LC81500372017204LGN00', 'https://s3-us-west-2.amazonaws.com/landsat-pds/c1/L8/150/037/LC08_L1TP_150037_20170723_20170729_01_T1/index.html'),
.......]

Investigation:
After exploring the issue, CSV file scene_list which is retrieved from AWS got some repeated lines.

Proposed solution:
Update the UpdateSceneList plugin to make sure there is no repeated line in the CSV before importing it to the DB