[WIP] feeds merge workflow

apps/merge-media-sources/Dockerfile

@@ -0,0 +1,21 @@
+#
+# Identify duplicate feeds or media sources and merge them into one entry, updating
+# the feed and media source references to point to the new entry.
+#
+
+FROM gcr.io/mcback/common:latest
+
+# Copy sources
+COPY src/ /opt/mediacloud/src/merge-media-sources/
+ENV PERL5LIB="/opt/mediacloud/src/merge-media-sources/perl:${PERL5LIB}" \
+    PYTHONPATH="/opt/mediacloud/src/merge-media-sources/python:${PYTHONPATH}"
+
+# Copy worker scripts
+COPY bin /opt/mediacloud/bin
+
+USER mediacloud
+
+# Set a failing CMD because we'll be using the same image to run feeds merge + media merge,
+# so the user is expected to set "command" in docker-compose.yml to run a specific worker.
+
+CMD ["SET_CONTAINER_COMMAND_TO_ONE_OF_THE_WORKERS"]

apps/merge-media-sources/README.md

@@ -0,0 +1,11 @@
+# Merging media sources
+
+## TODO
+
+* Create sample database with fake data
+* Test running the same activity multiple times
+* If an activity throws an exception, its message should get printed out to the console as well (in addition to
+  Temporal's log)
+* Track failed workflows / activities in Munin
+* Instead (in addition to) of setting `workflow_run_timeout` in `test_workflow.py`, limit retries of the individual
+  activities too so that when they fail, we'd get a nice error message printed to the test log