Reliability of date metadata in riksdagen-corpus

[Lauler]

Hi!

I've made a first pass on matching text from speeches in riksdagen-corpus with audio files from debates in the Riksdag from 1966-2002. I am at a point where I am sanity checking the results, and would appreciate input on the reliability of the metadata.

The premise is as follows:

• I have audio files whose only metadata is the dates the audio files were recorded (sometime multiple possible dates for a file).
• The only way I can enumerate possible combinations of text from speeches which may be present in audio files is by joining the datasets by date ranges. This creates a one-to-many, and sometimes many-to-many relationship. However, it does reduce the search space substantially.

I made a table below with some statistics of the total number of speeches, and the number of matched speeches grouped by year. I have a couple of questions regarding this which I hope you could provide some input on:

1. What is your sense or best estimate of the reliability of the "dates" listed in riksdagen-corpus protocols? Are they generally reliably extracted?
2. Do you have any intuition regarding whether the variation in total number of speeches and total number of words is natural? For example the total number of speeches/words drops by quite a significant margin in the mid 70s.

year	hours	nr_speeches_total	nr_matched_speeches	matched_speeches_frac	nr_words_total	nr_matched_words	matched_words_frac
1966	134.91	7557	1765	0.23	3617584	875874	0.24
1967	521.44	9009	6938	0.77	4402289	3392561	0.77
1968	443.56	8391	6320	0.75	3938987	2898893	0.74
1969	447.10	8762	6486	0.74	4071915	2933517	0.72
1970	353.04	8660	5074	0.59	4228332	2372048	0.56
1971	254.71	7519	4112	0.55	3769730	1845929	0.49
1972	246.06	8036	3958	0.49	4020891	1779679	0.44
1973	212.48	7635	3658	0.48	3893746	1534418	0.39
1974	184.25	6888	3307	0.48	3299849	1334757	0.40
1975	179.59	6579	3065	0.47	3050821	1267293	0.42
1976	165.83	5459	2973	0.54	2479230	1158626	0.47
1977	150.72	5779	2814	0.49	2511684	1067905	0.43
1978	199.95	7293	3721	0.51	3156465	1426693	0.45
1979	249.91	9470	4906	0.52	3985952	1833886	0.46
1980	259.49	9867	5230	0.53	4179315	1913346	0.46
1981	258.94	8659	4994	0.58	3605324	1883822	0.52
1982	237.56	8873	4751	0.54	3772018	1709017	0.45
1983	304.81	10451	6075	0.58	4069562	2174155	0.53
1984	343.20	10175	6876	0.68	3950887	2439652	0.62
1985	287.91	9902	5988	0.60	3809341	2040371	0.54
1986	304.86	9838	6028	0.61	3794095	2150256	0.57
1987	283.64	10347	5610	0.54	3959095	1991215	0.50
1988	314.38	10248	6440	0.63	4038696	2226916	0.55
1989	345.88	11680	7223	0.62	4313674	2495575	0.58
1990	312.95	11758	6837	0.58	4343658	2293327	0.53
1991	309.69	11321	7405	0.65	3978245	2301311	0.58
1992	324.64	14066	9036	0.64	4151762	2484895	0.60
1993	161.67	14049	4840	0.34	4331595	1272118	0.29
1994	209.23	11538	5205	0.45	3712036	1389569	0.37
1995	191.94	11619	5332	0.46	3697275	1452850	0.39
1996	166.38	11757	4686	0.40	3648873	1297080	0.36
1997	165.93	12273	4498	0.37	4037814	1335680	0.33
1998	166.60	11677	4309	0.37	3771918	1320000	0.35
1999	171.91	13269	4789	0.36	4111736	1383887	0.34
2000	160.53	13636	4240	0.31	4397766	1290869	0.29
2001	206.11	13579	5303	0.39	4416784	1647964	0.37
2002	41.07	13333	1047	0.08	4307296	329538	0.08

[fredrik1984]

Great work!

I am not really sure about the dates. Some records (protokoll) could also cover a couple of days. @ninpnin do you know about the quality of our coverage of the dates?

Regarding the second question, I am not sure I understand what you mean. But one important thing is that there is a difference between the oral speeches and the printed speeches in the records. This is because the parliamentary stenographers who turn verbatim speech to edited (readable text). Sometimes the differences is not big but sometimes it is a big difference.

[Lauler]

The second question was referring to the fact that the total number of speeches in the dataset drops to 5459 in 1976. This seems anomalous as there are a higher number of speeches in the preceding and following years.

@BobBorges informed me in an e-mail conversation that v.1.0.0 of The Swedish Parliament Corpus is out, and that you've switched repositories to https://github.com/swerik-project/the-swedish-parliament-corpus .

For reference, in v.1.0.0 there are a lot more segmented speeches in the mid 70s:

https://github.com/swerik-project/the-swedish-parliament-corpus/blob/main/stats/prot-speeches/prot-speeches.csv

I think I will try running this all again with the new version of the parliament corpus, and additionally also relaxing the date ranges for the speeches I am not able to match in the first pass of the data.

[Lauler]

By the way, I think you should advertise the fact that v.1.0.0 is out and is hosted in a different repository in the README of this repo.

[fredrik1984]

Ah, got it. We have seen some variation in the speech data but over so many years that your data indicate. There are some strange thing going on around 1976/1977 when the parliament switch from full year to autumn-to-summer year.

I can also say that we are working on improving the segmentation of speeches right now. Hence, in the next releases, there might be even better mapping between speech recordings and printed speech records.

And yes, it is on my to-do to do so!

[Lauler]

I can also say that we are working on improving the segmentation of speeches right now. Hence, in the next releases, there might be even better mapping between speech recordings and printed speech records.

Sounds great!

[fredrik1984]

Excellent!

[MansMeg]

@BobBorges and @ninpnin , do you have a take on question 1. Is it worthwhile to file this as an issue?

[BobBorges]

I think the dates are pretty reliable. One challenge, though, is that some records contain multiple dates, but it sounds like @Lauler already has that covered to some extent.

[MansMeg]

@Lauler if you find any incorrect dates it would be great for us to know. Ideally you could fix them and do a PR.

[Lauler]

I asked the same set of questions both here and to Riksdagen, as the root cause of non-matches can potentially arise from multiple sources:

1. Dates may be wrong in the filenames of the media files (filenames seem to have been manually created). Some files cover implausibly wide date ranges, and in most of these cases it appears like someone erroneously input the wrong decade/year/month for one of the dates present in the filename.
2. Dates may be wrong in riksdagen-corpus.
3. Speeches maybe be segmented incorrectly.

As for fixing the dates: I don't know which source I can trust as the truth for the date even when I get matches (aside from consulting the original source documents which are parsed by riksdagen-corpus). I think riksdagen-corpus probably has the more reliable dates. If there's a match for the audio & text of a speech, it's just a matter of choosing the most reliable source. And that can be updated at a later stage as well.

This discussion thread (and similar questions to Riksdagen regarding media files) is mostly about getting some info that allows me to determine a suitable search space for performing the matches. If I can trust the dates, I only need to make 2 million comparisons between text+audio files, If I run every possible combination naïvely it's over 2 billion comparisons.

[ninpnin]

The newer (1971-> maybe?) docDates come directly from riksdagen's metadata, while for the data before that they are scraped from margin notes. If there are OCR errors in the dates, some dates may be incorrect, but I think that is rare. We are not aware of any issues with this as of now.

If this is an important concern, we can perform statistical quality control. It would probably even be pretty fast, sampling 1 protocol per year/chamber, and checking the date manually.

[ninpnin]

I went through one protocol per decade:

• There are were a few OCR errors, that added extra days within the same month (18th instead of 13th, for example)
• Riksdagen dates start from the 1990s

[ninpnin]

Based on this, I think we want a gold standard for the dates: randomly sample one protocol per year and chamber, and annotate its true dates.