GIT authentication is not working during cluster setup

[jcunhasilva]

I created the following cluster using the airflow operator deployed directly from Google Cloud Marketplace:

apiVersion: airflow.k8s.io/v1alpha1
kind: AirflowCluster
metadata:
  name: my-airflow-cluster
spec:
  executor: Kubernetes
  ui:
    replicas: 1
    version: "1.10.1"
  scheduler:
    version: "1.10.1"
  worker:
    version: "1.10.1"
  config:
    airflow:
      AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 100
  dags:
    subdir: "pipelines/"
    git:
      repo: "https://github.com/MyOrg/my-repo.git"
      once: false
      branch: master
      user: MY_USER
      cred:
        name: MY_GIT_SECRET
  airflowbase:
    name: my-airflow-base

Once the cluster is up and running, I get the following error in the git-sync side container:
Cloning into '/git'...\nfatal: could not read Username for 'https://github.com': No such device or address\n"

So it seems the container is struggling with GIT authentication. When inspecting the container's environment variables I can see that these variables were created:

GIT_PASSWORD	
defined as key password of MY_GIT_SECRET
GIT_USER	
my_user

I stumbled upon this issue in the git-sync project where other people had issues setting up authentication:
kubernetes/git-sync#126

In this issue we have the following configuration

...
    - name: GIT_SYNC_USERNAME
      valueFrom:
        secretKeyRef:
          name: git-creds
          key: username
    - name: GIT_SYNC_PASSWORD
      valueFrom:
        secretKeyRef:
          name: git-creds
          key: password
...

so it seems the airflow-operator is using the wrong prefix for the authentication ("GIT_" instead of "GIT_SYNC_")

[jcunhasilva]

It seems that this problem is already fixed in the master branch:
891e94f

Is it possible to update the Google Cloud version with this fix?
I deployed directly from here:
https://console.cloud.google.com/kubernetes/application

@barney-s the last marketplace build is from July 2018:
https://console.cloud.google.com/gcr/images/cloud-marketplace/GLOBAL/google/airflow-operator?gcrImageListsize=30

Can we have an update to this image?

[barney-s]

Will work on updating the cloud marketplace.
Are you able to work with the latest master for now ?

[jcunhasilva]

Hi @barney-s
Yes I was able to deploy it manually using the master branch.

However I was forced to use the mysql spec configurations because both the Postgres and CloudSQL specs are not working.

If I use the CloudSQL spec I get this error when the database connection is starting:
raise Exception("Global variable explicit_defaults_for_timestamp needs to be on (1) for mysql"
We don't have any way of setting this flag in the CloudSQL UI.

If I use the Postgres spec I get a similar error:
FATAL: unrecognized configuration parameter "explicit_defaults_for_timestamp

Should I create a new issue for this?
Thank you.

[barney-s]

Thanks @jcunhasilva
The postgres issue is fixed (#30)
CloudSQL issue is tricky. I have opened a bug to track it: #48

[barney-s]

Feel free to reopen if you for tracking cloud marketplace availability.