Check which timestamps are correct!

[153957]

When switching the way the timestamps are created in the csv files, from calendar.timegm(date.utctimetuple()) to {{ date|date:'U' }} the timestamps changed!
It just so happens to have changed the timestamps by either 3600 or 7200 seconds (UTC +1 and +2, i.e. the timezone and DST in the Netherlands). I will check which one was the correct one..

P.S. I hate timezones and DST.

[davidfokkema]

Me too. Pain in the *ss.

[153957]

Probably fixed with 2fd6331, but may need to reprocess all configs.
But we will need to reprocess all events anyway; use actual trigger settings and increased coincidence window. Perhaps also ensure traces are filtered before determining arrival times.

[153957]

Currently the time shown in http://data.hisparc.nl/api/station/510/config/ is wrong.
It is "2015-11-27T16:34:42", but I started DAQ mode at 15:34 UTC.
Perhaps the field shows date using the server local timezone?
Downloading the Source tsv it currently gives the timestamp 1448638482.
This is correct, when I use sapphire.transformation.clock.gps_to_datetime it gives the expected time of 15:34.

[153957]

I checked one of the returned timestamps (with the current method of |date:"U") and compared this to the int timestamp in the raw data, they matched. So that should be the correct timestamp, even though the conversion of the timestamp into the database created a local time datetime object.

[153957]

Not entirely fixed yet.

[153957]

Can Perhaps be fixed by adding:

from django.utils import timezone
[...].replace(tzinfo=timezone.utc)

where [...] is the datetime object from a configuration.timestamp..
This specifically states that the object is in UTC, otherwise during template rendering the datetime object will be considered local time (I think)..

[153957]

I checked if the timestamps in the GPS Source TSV from station 501 are equal to those in the raw datastore. Most match, several can not be found, and some recent configs do not match. The following script was used (can be used for other stations):

from sapphire.transformations.clock import gps_to_datetime
from sapphire import Station
import tables

station_number = 501
station = Station(station_number)
cluster = station.cluster().lower()
timestamps = station.gps_locations['timestamp']
node_path = '/hisparc/cluster_%s/station_%d' % (cluster, station_number)

for ts in timestamps:
    path = gps_to_datetime(ts).strftime('%Y/%-m/%Y_%-m_%-d.h5')
    with tables.open_file(path, 'r') as data:
        try:
            cts = data.get_node(node_path, 'config').col('timestamp')
        except:
            print ts, 'No source found'
            continue
        if not ts in cts:
            print ts, cts

[153957]

I've recently seen issues with incorrect timestamps in TSV sources, possibly because the server is now running in Mountain Time in Utah, the offsets are about 7-8 hours offset.