Skip to content

Commit

Permalink
test
Browse files Browse the repository at this point in the history
  • Loading branch information
Simran Mattu committed Sep 17, 2024
1 parent a3bb5c9 commit 42f655d
Show file tree
Hide file tree
Showing 3 changed files with 126 additions and 122 deletions.
246 changes: 125 additions & 121 deletions data/migrate/errors-backfilling.csv
Original file line number Diff line number Diff line change
@@ -1,122 +1,126 @@
Error Code,Error Type,Message Template,Notes
1,Warning,Not a text file
2,Error,No core metadata tables found. Not an Extended CSV file
3,Error,Missing required table #{table}
4,Error,Excess table #{table} does not belong in {dataset} file
5,Error,Missing required field #{table}.{field}
6,Warning,Excess field {field} does not belong in table {table}
7,Error,Required field #{table}.{field} is null or empty
8,Warning,Optional field #{table}.{field} is null or empty
9,Error,Table #{table} has no fields
10,Error,Table #{table} has no fields,Placeholder for optional-table-specific error
11,Error,Required table #{table} contains no data
12,Warning,Optional table #{table} contains no data
13,Error,No non-core data tables found
14,Warning,Unexpected empty line between table header and fields
15,Error,Unrecognized data {row}
16,Warning,Improper delimiter used '{separator}' corrected to '\,' (comma)
20,Warning,#{table} field {oldfield} capitalization should be {newfield}
21,Warning,#{table} corrected to {newtable} using aliases
22,Warning,#{table} field {oldfield} corrected to {newfield} using aliases
23,Warning,#{table}.{field} value corrected to {newvalue} using aliases
25,Warning,#{table} row has more values than #{table} has columns
26,Error,Fewer than minimum {bound} occurrences of table #{table} found
27,Error,More than maximum {bound} occurrences of table #{table} found
28,Error,Fewer than minimum {bound} number of rows in table #{table}
29,Error,More than maximum {bound} number of rows in table #{table}
30,Warning,#{table}.Time separator '{separator}' corrected to ':' (colon)
31,Error,Failed to parse #{table}.Time {component}: contains invalid characters
32,Warning,#{table}.Time corrected from 12-hour clock to 24-hour YYYY-MM-DD format
33,Warning,#{table}.Time {component} is not within allowable range [{lower}]-[{upper}]
34,Warning,#{table}.Date separator '{separator}' corrected to '-' (hyphen)
35,Error,#{table}.Date not in YYYY-MM-DD format: missing separators
36,Error,#{table}.Date is incomplete
37,Error,#{table}.Date not in YYYY-MM-DD format: too many separators
38,Error,Failed to parse #{table}.Date {component}: contains invalid characters
39,Error,#{table}.Date {component} is not within allowable range [{lower}]-[{upper}],Reserved for year and month components
40,Error,#{table}.Date day is not within allowable range [{lower}]-[{upper}]
41,Warning,#{table}.UTCOffset separator '{separator}' corrected to ':' (colon)
42,Warning,#{table}.UTCOffset {component} is less than 2 digits long
43,Warning,#{table}.UTCOffset {component} is missing\, default value is '00' (zero)
44,Warning,Missing sign in #{table}.UTCOffset\, default '+' (plus)
45,Warning,Invalid sign in #{table}.UTCOffset\, replacing with '{sign}'
46,Warning,#{table}.UTCOffset is a series of zeroes\, should be '+00:00:00'
47,Error,Failed to parse #{table}.UTCOffset: contains invalid characters
50,Warning,Missing #CONTENT.Class\, default is 'WOUDC'
51,Error,#CONTENT.Class {value} failed to validate against registry
52,Error,#CONTENT.Category {value} failed to validate against registry
53,Warning,#CONTENT.Level should be {value} according to present tables
54,Warning,#CONTENT.Level {oldvalue} should be a decimal number ({newvalue})
55,Error,Failed to parse #CONTENT.Level: contains invalid characters
56,Error,Unknown #CONTENT.Level for dataset {dataset}
57,Warning,#CONTENT.Form {oldvalue} should be integral ({newvalue})
58,Error,Failed to parse #CONTENT.Form: contains invalid characters
59,Error,Cannot resolve missing or invalid #CONTENT.Form
60,Error,Unknown #CONTENT.Form for dataset {dataset} and level {level}
61,Error,Cannot assess expected table set: #CONTENT.{field} unknown
62,Warning,Missing #DATA_GENERATION.Date\, defaults to processing date
63,Warning,Missing #DATA_GENERATION.Version\, defaults to {default}
64,Warning,#DATA_GENERATION.Version is not within allowable range [{lower}]-[{upper}]
65,Warning,#DATA_GENERATION.Version does not have exactly decimal place
66,Error,Failed to parse #DATA_GENERATION.Version: contains invalid characters
67,Error,#DATA_GENERATION.Agency failed to validate against registry
70,Warning,#PLATFORM.ID is not 3 digits long
71,Error,#PLATFORM.ID not found in registry
72,Error,#PLATFORM.Type in file does not match registry
73,Error,#PLATFORM.Name in file does not match registry
74,Error,#PLATFORM.Country in file does not match registry
75,Warning,Ship #PLATFORM.Country should be 'XY' to meet ISO-3166 standards
76,Error,Failed to parse #LOCATION.{field}: contains invalid characters,Reserved for fields Latitude and Longitude
77,Warning,Failed to parse #LOCATION.Height: contains invalid characters
78,Error,#LOCATION.{field} is not within allowable range [{lower}]-[{upper}],Reserved for fields Latitude and Longitude
79,Warning,#LOCATION.Height is not within allowable range [{lower}]-[{upper}]
80,Warning,#LOCATION.{field} in file does not match registry,Reserved for fields Latitude and Longitude
81,Warning,#LOCATION.Height in file does not match registry
82,Error,Null value found for #INSTRUMENT.Name
83,Warning,Null value found for #INSTRUMENT.Model
84,Warning,Null value found for #INSTRUMENT.Number
85,Error,#INSTRUMENT.Name not found in registry
86,Error,#INSTRUMENT.Model not found in registry
87,Error,Instrument failed to validate against registry
88,Error,Deployment {ident} not found in registry
89,Error,Failed to parse #{table}.{field} due to errors: {reason}
90,Warning,Inconsistent Time values between #TIMESTAMP tables
91,Warning,#{table}.Date cannot be more recent than #DATA_GENERATION.Date,Reserved for TIMESTAMP tables
92,Warning,#{table}.Date cannot be more recent than #DATA_GENERATION.Date,Reserved for non-TIMESTAMP tables
93,Warning,First #TIMESTAMP.Time cannot be more recent than other time(s)
95,Error,Submitted file #DATA_GENERATION.Date is earlier than previously submitted version
96,Error,Submitted file version and #DATA_GENERAION.Date identical to previously submitted file
97,Error,Submitted #DATA_GENERATION.Date is identical to previously submitted file
98,Error,Submitted version number is identical to previously submitted file
99,Warning,ECC instrument serial number different from previous submission
101,Error,No ozone data in #DAILY table
102,Warning,#DAILY.Date found in non-chronological order
103,Warning,#DAILY.Date has different year than #TIMESTAMP.Date
104,Warning,Duplicate observations found in #DAILY table for Date=(date)
105,Warning,Multiple observations found with #DAILY.Date {date}
106,Warning,#TIMESTAMP.Date before #DAILY does not equal first date of #DAILY
107,Warning,#TIMESTAMP.Date after #DAILY does not equal last date of #DAILY
108,Warning,More than two #TIMESTAMP tables found in file
109,Warning,#TIMESTAMP table after #DAILY is missing\, deriving based on requirements
110,Warning,Missing #MONTHLY table\, deriving based on requirements
111,Warning,Missing value for #MONTHLY.{field}\, deriving based on requirements
112,Warning,#MONTHLY.{field} differs from derived value
113,Error,Cannot derive #MONTHLY table: missing #DAILY.ColumnO3
114,Warning,#OBSERVATIONS.Time found in non-chronological order
115,Warning,Duplicate observations found in #OBSERVATIONS table for Time={time}
116,Warning,Multiple observations found with #OBSERVATIONS.Time {time}
118,Warning,#{table}.Date found in non-chronological order,Reserved for Umkehr data tables
119,Warning,Duplicate observations found in #{table} for Date={date},Reserved for Umkehr data tables
120,Warning,Multiple observations found with #{table}.Date {date},Reserved for Umkehr data tables
121,Warning,#TIMESTAMP.Date before #{table} does not equal first date of #{table},Reserved for Umkehr data tables
122,Warning,#TIMESTAMP.Date after #{table} does not equal last date of #{table},Reserved for Umkehr data tables
123,Warning,#TIMESTAMP table after #{table} is missing\, deriving based on requirements,Reserved for Umkehr data tables
125,Error,Lidar table counts are uneven between #OZONE_PROFILE and #OZONE_SUMMARY
126,Error,Spectral table counts are uneven between #TIMESTAMP\, #GLOBAL\, and #{summary_table}
200,Warning,Data file successfully persisted
201,Warning,New instrument added
202,Warning,New deployment added
203,Warning,New station name added
204,Warning,New contribution added
209,Error,Data file failed to validate
1000,Error,Unassigned error message
101,Warning,Not a text file
102,Error,No core metadata tables found. Not an Extended CSV file
103,Warning,Unexpected empty line between table header and fields
104,Warning,Improper delimiter used '{separator}' corrected to '\,' (comma)
105,Warning,#{table} field {oldfield} capitalization should be {newfield}
106,Warning,#{table} corrected to {newtable} using aliases
107,Warning,#{table} field {oldfield} corrected to {newfield} using aliases
108,Warning,#{table}.{field} value corrected to {newvalue} using aliases
109,Warning,#{table}.Time separator '{separator}' corrected to ':' (colon)
110,Warning,#{table}.Time corrected from 12-hour clock to 24-hour YYYY-MM-DD format
111,Warning,#{table}.Date separator '{separator}' corrected to '-' (hyphen)
112,Error,#{table}.Date not in YYYY-MM-DD format: missing separators
113,Error,#{table}.Date is incomplete
114,Error,#{table}.Date not in YYYY-MM-DD format: too many separators
115,Warning,#{table}.UTCOffset separator '{separator}' corrected to ':' (colon)
116,Warning,#{table}.UTCOffset {component} is less than 2 digits long
117,Warning,#{table}.UTCOffset {component} is missing\, default value is '00' (zero)
118,Warning,Missing sign in #{table}.UTCOffset\, default '+' (plus)
119,Warning,Invalid sign in #{table}.UTCOffset\, replacing with '{sign}'
120,Warning,#{table}.UTCOffset is a series of zeroes\, should be '+00:00:00'
121,Error,Cannot derive #MONTHLY table: missing #DAILY.ColumnO3
122,Error,Lidar table counts are uneven between #OZONE_PROFILE and #OZONE_SUMMARY
123,Error,Spectral table counts are uneven between #TIMESTAMP\, #GLOBAL\, and #{summary_table}
201,Error,Missing required table #{table}
202,Error,Excess table #{table} does not belong in {dataset} file
203,Error,Missing required field #{table}.{field}
204,Error,Required field #{table}.{field} is null or empty
205,Warning,Optional field #{table}.{field} is null or empty
206,Error,Table #{table} has no fields
207,Error,Table #{table} has no fields,Placeholder for optional-table-specific error
208,Error,Required table #{table} contains no data
209,Warning,Optional table #{table} contains no data
210,Error,No non-core data tables found
211,Error,Unrecognized data {row}
212,Warning,#{table} row has more values than #{table} has columns
213,Error,Fewer than minimum {bound} occurrences of table #{table} found
214,Error,More than maximum {bound} occurrences of table #{table} found
215,Error,Fewer than minimum {bound} number of rows in table #{table}
216,Error,More than maximum {bound} number of rows in table #{table}
217,Warning,#CONTENT.Level should be {value} according to present tables
218,Warning,#CONTENT.Level {oldvalue} should be a decimal number ({newvalue})
219,Warning,#CONTENT.Form {oldvalue} should be integral ({newvalue})
220,Error,Cannot assess expected table set: #CONTENT.{field} unknown
221,Warning,Missing #DATA_GENERATION.Date\, defaults to processing date
222,Warning,#DATA_GENERATION.Version does not have decimal place
223,Error,Null value found for #INSTRUMENT.Name
224,Warning,Null value found for #INSTRUMENT.Model
225,Warning,Null value found for #INSTRUMENT.Number
226,Warning,Inconsistent Time values between #TIMESTAMP tables
227,Warning,#{table}.Date cannot be more recent than #DATA_GENERATION.Date,Reserved for non-TIMESTAMP tables
228,Warning,First #TIMESTAMP.Time cannot be more recent than other time(s)
229,Error,Submitted file #DATA_GENERATION.Date is earlier than previously submitted version
230,Error,No ozone data in #DAILY table
231,Warning,#DAILY.Date found in non-chronological order
232,Warning,#DAILY.Date has different year than #TIMESTAMP.Date
233,Warning,Duplicate observations found in #DAILY table for Date=(date)
234,Warning,Multiple observations found with #DAILY.Date {date}
235,Warning,#TIMESTAMP.Date before #DAILY does not equal first date of #DAILY
236,Warning,#TIMESTAMP.Date after #DAILY does not equal last date of #DAILY
237,Warning,More than two #TIMESTAMP tables found in file
238,Warning,#TIMESTAMP table after #DAILY is missing\, deriving based on requirements
239,Warning,Missing #MONTHLY table\, deriving based on requirements
240,Warning,Missing value for #MONTHLY.{field}\, deriving based on requirements
241,Warning,#MONTHLY.{field} differs from derived value
242,Warning,#OBSERVATIONS.Time found in non-chronological order
243,Warning,Duplicate observations found in #OBSERVATIONS table for Time={time}
244,Warning,Multiple observations found with #OBSERVATIONS.Time {time}
245,Warning,#{table}.Date found in non-chronological order,Reserved for Umkehr data tables
246,Warning,Multiple observations found with #{table}.Date {date},Reserved for Umkehr data tables
247,Warning,#TIMESTAMP.Date before #{table} does not equal first date of #{table},Reserved for Umkehr data tables
248,Warning,#TIMESTAMP.Date after #{table} does not equal last date of #{table},Reserved for Umkehr data tables
249,Warning,#TIMESTAMP table after #{table} is missing\, deriving based on requirements,Reserved for Umkehr data tables
250,Warning,Excess field {field} does not belong in table {table}
251,Warning,Duplicate observations found in #{table} for Date={date},Reserved for Umkehr data tables
301,Error,Failed to parse #{table}.Time {component}: contains invalid characters
302,Error,Failed to parse #{table}.Date {component}: contains invalid characters
303,Error,#{table}.Date {component} is not within allowable range [{lower}]-[{upper}],Reserved for year and month components
304,Error,#{table}.Date day is not within allowable range [{lower}]-[{upper}]
305,Error,Failed to parse #{table}.UTCOffset: contains invalid characters
306,Warning,Missing #CONTENT.Class\, default is 'WOUDC'
307,Error,#CONTENT.Class {value} failed to validate against registry
308,Error,#CONTENT.Category {value} failed to validate against registry
309,Error,Unknown #CONTENT.Level for dataset {dataset}
310,Error,Failed to parse #CONTENT.Level: contains invalid characters
311,Error,Failed to parse #CONTENT.Form: contains invalid characters
312,Error,Cannot resolve missing or invalid #CONTENT.Form
313,Error,Unknown #CONTENT.Form for dataset {dataset} and level {level}
314,Warning,Missing #DATA_GENERATION.Version\, defaults to {default}
315,Warning,#DATA_GENERATION.Version is not within allowable range [{lower}]-[{upper}]
316,Error,Failed to parse #DATA_GENERATION.Version: contains invalid characters
317,Error,#DATA_GENERATION.Agency failed to validate against registry
318,Warning,#PLATFORM.ID is not 3 digits long
319,Error,#PLATFORM.ID not found in registry
320,Error,#PLATFORM.Type in file does not match registry
321,Error,#PLATFORM.Name in file does not match registry
322,Error,#PLATFORM.Country in file does not match registry
323,Warning,Ship #PLATFORM.Country should be 'XY' to meet ISO-3166 standards
324,Warning,Failed to parse #LOCATION.Height: contains invalid characters
325,Error,#LOCATION.{field} is not within allowable range [{lower}]-[{upper}],Reserved for fields Latitude and Longitude
326,Warning,#LOCATION.Height is not within allowable range [{lower}]-[{upper}]
327,Warning,#LOCATION.{field} in file does not match registry,Reserved for fields Latitude and Longitude
328,Warning,#LOCATION.Height in file does not match registry
329,Warning,Null value found for #INSTRUMENT.Model
330,Warning,Null value found for #INSTRUMENT.Number
331,Error,#INSTRUMENT.Name not found in registry
332,Error,#INSTRUMENT.Model not found in registry
333,Error,Instrument failed to validate against registry
334,Error,Deployment {ident} not found in registry
335,Error,Failed to parse #{table}.{field} due to errors: {reason}
336,Warning,#{table}.Date cannot be more recent than #DATA_GENERATION.Date,Reserved for TIMESTAMP tables
337,Warning,#{table}.Date cannot be more recent than #DATA_GENERATION.Date,Reserved for non-TIMESTAMP tables
338,Warning,ECC instrument serial number different from previous submission
339,Error,Failed to parse #LOCATION.{field}: contains invalid characters,Reserved for fields Latitude and Longitude
340,Warning,#{table}.Time {component} is not within allowable range [{lower}]-[{upper}]
401,Error,Submitted file version and #DATA_GENERAION.Date identical to previously submitted file
402,Error,Submitted #DATA_GENERATION.Date is identical to previously submitted file
403,Error,Submitted version number is identical to previously submitted file
404,Warning,ECC instrument serial number different from previous submission
405,Warning,Data file successfully persisted
406,Warning,New instrument added
407,Warning,New deployment added
408,Warning,New station name added
409,Warning,New contribution added
410,Error,Data file failed to validate
1000,Error,Unassigned error message
File renamed without changes.
2 changes: 1 addition & 1 deletion woudc_data_registry/tests/test_report_generation.py
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ def test_uses_error_definition(self):
"""Test that error/warning feedback responds to input files"""

# The two error files below have different error types for error 1.
all_errors = resolve_test_data_path('config/Updated_Errors.csv')
all_errors = resolve_test_data_path('config/errors.csv')

with report.OperatorReport(SANDBOX_DIR) as op_report:
op_report.read_error_definitions(all_errors)
Expand Down

0 comments on commit 42f655d

Please sign in to comment.