Skip to content

Commit

Permalink
Release - 0.3.0 (#45)
Browse files Browse the repository at this point in the history
* Update README.md

* Updated gettext errors

* Removed unused variables

* Fixed linting/formatting issues

* Fixed linting/formatting issues

* Update deploy_heroku.yml

* Rename deploy_heroku.yml to deploy_heroku_staging.yml

* Create deploy_heroku_prod.yaml

* Update deploy_heroku_staging.yml

* As a user I can sign up and sign in with a valid e-mail and password  (#28)

* [#6 #16] As a user I can sign in with a valid e-mail and password

* Remove unused files

* Remove unsused update function for User

* Add session for user after log in

* Add unique email constraint on Users table

* Add user sign in

* Add current sign in status for user

* Add user sign out functionality (not in backlog)

* Remove / refactored code

* Remove coverage check for currently unused plug

* Add controller tests

* Prepare ExMachina for testing

* User sign out now displays a message

* Change session deletion method to ensure persistence of message to user upon sign out

* Refactored password hashing function so it can be used in future tests

* Refactor fixture to use ExMachina and Faker for data generation

* Modify ExUnit tests to conform to standards

* Tidy template pages with correct formatting

* Remove comments and cleaned up code

* Merge migrations into single file for User schema

* Remove comments and cleaned up code

* Correct English used in ExUnit test case

* Move secret_key-base file to environment variable for production

* Change multiple alias identifiers from one line to multiple to satisy codebase

* Make blank line seperation more consistent in the tests

* Add feature test case for User log in

* [#6 #7 #16 #22] Add account tests for invalid cases and fixed formatting

* [#6 #7 #16 #22] Add account tests for invalid cases and fixed formatting

* [#6 #7 #16 #22] Refactor code according to feedback

* [#6 #7 #16 #22] Refactor code according to feedback

* [#6 #7 #16 #22] Remove code form AuthController to AuthHelper to reflect the functionality

* [#6 #7 #16 #22] Remove auto-generated function spec

* [#6 #7 #16 #22] Moved Account context into accounts folder and account schema into its own folder to improve structure

* [#6 #7 #16 #22] Add tests for user changesets for negative paths

* [#6 #7 #16 #22] Add tests for user changesets for negative paths

* [#6 #7 #16 #22] Change refute to assert false for testing outcomese

* [#6 #7 #16 #22] Changed from pattern matching to double equals to match exact output when required. Move value being tested against to the right side

* [#3] [UI] As a user, I can upload a CSV file containing keywords which will then be used to search on Google (#31)

* [#6 #16] As a user I can sign in with a valid e-mail and password

* Remove unused files

* Remove unsused update function for User

* Add session for user after log in

* Add unique email constraint on Users table

* Add user sign in

* Add current sign in status for user

* Add user sign out functionality (not in backlog)

* Remove / refactored code

* Remove coverage check for currently unused plug

* Add controller tests

* Prepare ExMachina for testing

* User sign out now displays a message

* Change session deletion method to ensure persistence of message to user upon sign out

* Refactored password hashing function so it can be used in future tests

* Refactor fixture to use ExMachina and Faker for data generation

* Modify ExUnit tests to conform to standards

* Tidy template pages with correct formatting

* Remove comments and cleaned up code

* Merge migrations into single file for User schema

* Remove comments and cleaned up code

* Correct English used in ExUnit test case

* Move secret_key-base file to environment variable for production

* Change multiple alias identifiers from one line to multiple to satisy codebase

* Make blank line seperation more consistent in the tests

* Add feature test case for User log in

* [#6 #7 #16 #22] Add account tests for invalid cases and fixed formatting

* [#6 #7 #16 #22] Add account tests for invalid cases and fixed formatting

* [#6 #7 #16 #22] Refactor code according to feedback

* [#6 #7 #16 #22] Refactor code according to feedback

* [#6 #7 #16 #22] Remove code form AuthController to AuthHelper to reflect the functionality

* [#6 #7 #16 #22] Remove auto-generated function spec

* [#6 #7 #16 #22] Moved Account context into accounts folder and account schema into its own folder to improve structure

* [#6 #7 #16 #22] Add tests for user changesets for negative paths

* [#6 #7 #16 #22] Add tests for user changesets for negative paths

* [#3] Add endpoint and controller for handling keyword upload

* [#3] Add template files for uploading files, including upload form

* [#3] Add link to keywords page in navigation

* [#3] Remove coverall and comments from authenticated plug to prepare for tests

* [#3] Add tests for keyword controller and test csv file

* [#3] Add tests for ensure_authenticated plug

* Resolved merge conflict

* [#3] Format code

* [#3] Change keywords fixture file name and changed template to show 1000 keywords limit

* [#3] Remove blank line and re-order assert tests for ensure_authenticated plug tests

* [#3] Add an additional test to ensure unauthenticated users are unable to upload a keywords file

* [#3] Format test

* [#18] [Backend] As a user, I can upload a CSV file containing keywords which will be stored (#33)

* [#6 #16] As a user I can sign in with a valid e-mail and password

* Remove unused files

* Remove unsused update function for User

* Add session for user after log in

* Add unique email constraint on Users table

* Add user sign in

* Add current sign in status for user

* Add user sign out functionality (not in backlog)

* Remove / refactored code

* Remove coverage check for currently unused plug

* Add controller tests

* Prepare ExMachina for testing

* User sign out now displays a message

* Change session deletion method to ensure persistence of message to user upon sign out

* Refactored password hashing function so it can be used in future tests

* Refactor fixture to use ExMachina and Faker for data generation

* Modify ExUnit tests to conform to standards

* Tidy template pages with correct formatting

* Remove comments and cleaned up code

* Merge migrations into single file for User schema

* Remove comments and cleaned up code

* Correct English used in ExUnit test case

* Move secret_key-base file to environment variable for production

* Change multiple alias identifiers from one line to multiple to satisy codebase

* Make blank line seperation more consistent in the tests

* Add feature test case for User log in

* [#6 #7 #16 #22] Add account tests for invalid cases and fixed formatting

* [#6 #7 #16 #22] Add account tests for invalid cases and fixed formatting

* [#6 #7 #16 #22] Refactor code according to feedback

* [#6 #7 #16 #22] Refactor code according to feedback

* [#6 #7 #16 #22] Remove code form AuthController to AuthHelper to reflect the functionality

* [#6 #7 #16 #22] Remove auto-generated function spec

* [#6 #7 #16 #22] Moved Account context into accounts folder and account schema into its own folder to improve structure

* [#6 #7 #16 #22] Add tests for user changesets for negative paths

* [#6 #7 #16 #22] Add tests for user changesets for negative paths

* [#3] Add endpoint and controller for handling keyword upload

* [#3] Add template files for uploading files, including upload form

* [#3] Add link to keywords page in navigation

* [#3] Remove coverall and comments from authenticated plug to prepare for tests

* [#3] Add tests for keyword controller and test csv file

* [#3] Add tests for ensure_authenticated plug

* Resolved merge conflict

* [#3] Format code

* [#3] Change keywords fixture file name and changed template to show 1000 keywords limit

* [#3] Remove blank line and re-order assert tests for ensure_authenticated plug tests

* [#3] Add an additional test to ensure unauthenticated users are unable to upload a keywords file

* [#3] Format test

* [#18] Add NimbleCSV

* [#18] Add Keyword Controller and helper function for validate/parse csv

* [#18] Add Keyword Controller tests and additional test files for invalid cases

* [#18] Change invalid file format to invalid file extension

* [#18] Initial KeywordUpload Schema, associations and tests

* [#18] Initial code to carry out the mass insertions of keywords into the table for the User. Needs refactor wip

* [#18] Refactor KeywordUpload changeset to Use __MODULE__ as default argument

* #[18] Slight refactor of keyword saving for user wip

* Refactor Keyword context name and add one test

* [#18] Change alias to fix formatting errors

* [#18] Add test to Keyword Controller to verify an uplaod of two keywords returns the correct count to the user

* [#18] Add additional empty line for csv files

* [#18] Change from using length to Enum.count() for counting list size

* [#18] Change from using string field to text for keyword html storage to remove character limit

* [#18] Add positive test result for KeywordUpload changeset

* [#18] Remove unnecessary conn.halts from keyword controller

* [#18] Remove comments and changed grammar in test cases for Keywords

* [#18] Change name and status fields of KeywordUpload to be to not accept null

* [#18] Refactor parsing of keywords into correct structure for bulk inserts wip

* [#18] Add two further KeywordUpload changeset tests to ensure a KeywordUpload has to have an existing user

* [#1] [#20] As a user, I can view a list of my previously uploaded keywords (#35)

* #[1] Add Context function to retreive list of uploaded keywords for a particular user

* #[1] Add Controller and template to show the list of uploaded keywords for the user

* #[1] Change github action trigger from Pull Request to Push to allow staging and prod

* #[1] Change github action trigger from Pull Request to Push to allow staging and prod

* [#1] Add Uploaded field to display for each KeywordUpload and format using Calendar module

* #[1] Add test for KeywordView for formatting timestamp

* #[1] Change Repo.list_all to return the inserted Keywords

* #[1] Write tests for fetching KeywordUploads for a particular user

* #[1] Add KeywordUpload Factory to tests for listing KeywordUploads for a User

* [#1] Refactor keyword test using pipe operator to make it cleaner

* [#1] Fix formatting on keywords index template file

* [#1] Clean up keyword template file

* [#1] Remove external Calendar library dependencies due to built-in functionality in Elxiir

* [#1] Made keyword test title more explicit

* Remove prod.secret.exe config import to allow deployment (#36)

* [#24] [Backend] Retrieve and store HTML response using background job on Google Search with uploaded keywords (#39)

* Release - 0.2.0 (#37)

* Update README.md

* Updated gettext errors

* Removed unused variables

* Fixed linting/formatting issues

* Fixed linting/formatting issues

* Update deploy_heroku.yml

* Rename deploy_heroku.yml to deploy_heroku_staging.yml

* Create deploy_heroku_prod.yaml

* Update deploy_heroku_staging.yml

* As a user I can sign up and sign in with a valid e-mail and password  (#28)

* [#6 #16] As a user I can sign in with a valid e-mail and password

* Remove unused files

* Remove unsused update function for User

* Add session for user after log in

* Add unique email constraint on Users table

* Add user sign in

* Add current sign in status for user

* Add user sign out functionality (not in backlog)

* Remove / refactored code

* Remove coverage check for currently unused plug

* Add controller tests

* Prepare ExMachina for testing

* User sign out now displays a message

* Change session deletion method to ensure persistence of message to user upon sign out

* Refactored password hashing function so it can be used in future tests

* Refactor fixture to use ExMachina and Faker for data generation

* Modify ExUnit tests to conform to standards

* Tidy template pages with correct formatting

* Remove comments and cleaned up code

* Merge migrations into single file for User schema

* Remove comments and cleaned up code

* Correct English used in ExUnit test case

* Move secret_key-base file to environment variable for production

* Change multiple alias identifiers from one line to multiple to satisy codebase

* Make blank line seperation more consistent in the tests

* Add feature test case for User log in

* [#6 #7 #16 #22] Add account tests for invalid cases and fixed formatting

* [#6 #7 #16 #22] Add account tests for invalid cases and fixed formatting

* [#6 #7 #16 #22] Refactor code according to feedback

* [#6 #7 #16 #22] Refactor code according to feedback

* [#6 #7 #16 #22] Remove code form AuthController to AuthHelper to reflect the functionality

* [#6 #7 #16 #22] Remove auto-generated function spec

* [#6 #7 #16 #22] Moved Account context into accounts folder and account schema into its own folder to improve structure

* [#6 #7 #16 #22] Add tests for user changesets for negative paths

* [#6 #7 #16 #22] Add tests for user changesets for negative paths

* [#6 #7 #16 #22] Change refute to assert false for testing outcomese

* [#6 #7 #16 #22] Changed from pattern matching to double equals to match exact output when required. Move value being tested against to the right side

* [#3] [UI] As a user, I can upload a CSV file containing keywords which will then be used to search on Google (#31)

* [#6 #16] As a user I can sign in with a valid e-mail and password

* Remove unused files

* Remove unsused update function for User

* Add session for user after log in

* Add unique email constraint on Users table

* Add user sign in

* Add current sign in status for user

* Add user sign out functionality (not in backlog)

* Remove / refactored code

* Remove coverage check for currently unused plug

* Add controller tests

* Prepare ExMachina for testing

* User sign out now displays a message

* Change session deletion method to ensure persistence of message to user upon sign out

* Refactored password hashing function so it can be used in future tests

* Refactor fixture to use ExMachina and Faker for data generation

* Modify ExUnit tests to conform to standards

* Tidy template pages with correct formatting

* Remove comments and cleaned up code

* Merge migrations into single file for User schema

* Remove comments and cleaned up code

* Correct English used in ExUnit test case

* Move secret_key-base file to environment variable for production

* Change multiple alias identifiers from one line to multiple to satisy codebase

* Make blank line seperation more consistent in the tests

* Add feature test case for User log in

* [#6 #7 #16 #22] Add account tests for invalid cases and fixed formatting

* [#6 #7 #16 #22] Add account tests for invalid cases and fixed formatting

* [#6 #7 #16 #22] Refactor code according to feedback

* [#6 #7 #16 #22] Refactor code according to feedback

* [#6 #7 #16 #22] Remove code form AuthController to AuthHelper to reflect the functionality

* [#6 #7 #16 #22] Remove auto-generated function spec

* [#6 #7 #16 #22] Moved Account context into accounts folder and account schema into its own folder to improve structure

* [#6 #7 #16 #22] Add tests for user changesets for negative paths

* [#6 #7 #16 #22] Add tests for user changesets for negative paths

* [#3] Add endpoint and controller for handling keyword upload

* [#3] Add template files for uploading files, including upload form

* [#3] Add link to keywords page in navigation

* [#3] Remove coverall and comments from authenticated plug to prepare for tests

* [#3] Add tests for keyword controller and test csv file

* [#3] Add tests for ensure_authenticated plug

* Resolved merge conflict

* [#3] Format code

* [#3] Change keywords fixture file name and changed template to show 1000 keywords limit

* [#3] Remove blank line and re-order assert tests for ensure_authenticated plug tests

* [#3] Add an additional test to ensure unauthenticated users are unable to upload a keywords file

* [#3] Format test

* [#18] [Backend] As a user, I can upload a CSV file containing keywords which will be stored (#33)

* [#6 #16] As a user I can sign in with a valid e-mail and password

* Remove unused files

* Remove unsused update function for User

* Add session for user after log in

* Add unique email constraint on Users table

* Add user sign in

* Add current sign in status for user

* Add user sign out functionality (not in backlog)

* Remove / refactored code

* Remove coverage check for currently unused plug

* Add controller tests

* Prepare ExMachina for testing

* User sign out now displays a message

* Change session deletion method to ensure persistence of message to user upon sign out

* Refactored password hashing function so it can be used in future tests

* Refactor fixture to use ExMachina and Faker for data generation

* Modify ExUnit tests to conform to standards

* Tidy template pages with correct formatting

* Remove comments and cleaned up code

* Merge migrations into single file for User schema

* Remove comments and cleaned up code

* Correct English used in ExUnit test case

* Move secret_key-base file to environment variable for production

* Change multiple alias identifiers from one line to multiple to satisy codebase

* Make blank line seperation more consistent in the tests

* Add feature test case for User log in

* [#6 #7 #16 #22] Add account tests for invalid cases and fixed formatting

* [#6 #7 #16 #22] Add account tests for invalid cases and fixed formatting

* [#6 #7 #16 #22] Refactor code according to feedback

* [#6 #7 #16 #22] Refactor code according to feedback

* [#6 #7 #16 #22] Remove code form AuthController to AuthHelper to reflect the functionality

* [#6 #7 #16 #22] Remove auto-generated function spec

* [#6 #7 #16 #22] Moved Account context into accounts folder and account schema into its own folder to improve structure

* [#6 #7 #16 #22] Add tests for user changesets for negative paths

* [#6 #7 #16 #22] Add tests for user changesets for negative paths

* [#3] Add endpoint and controller for handling keyword upload

* [#3] Add template files for uploading files, including upload form

* [#3] Add link to keywords page in navigation

* [#3] Remove coverall and comments from authenticated plug to prepare for tests

* [#3] Add tests for keyword controller and test csv file

* [#3] Add tests for ensure_authenticated plug

* Resolved merge conflict

* [#3] Format code

* [#3] Change keywords fixture file name and changed template to show 1000 keywords limit

* [#3] Remove blank line and re-order assert tests for ensure_authenticated plug tests

* [#3] Add an additional test to ensure unauthenticated users are unable to upload a keywords file

* [#3] Format test

* [#18] Add NimbleCSV

* [#18] Add Keyword Controller and helper function for validate/parse csv

* [#18] Add Keyword Controller tests and additional test files for invalid cases

* [#18] Change invalid file format to invalid file extension

* [#18] Initial KeywordUpload Schema, associations and tests

* [#18] Initial code to carry out the mass insertions of keywords into the table for the User. Needs refactor wip

* [#18] Refactor KeywordUpload changeset to Use __MODULE__ as default argument

* #[18] Slight refactor of keyword saving for user wip

* Refactor Keyword context name and add one test

* [#18] Change alias to fix formatting errors

* [#18] Add test to Keyword Controller to verify an uplaod of two keywords returns the correct count to the user

* [#18] Add additional empty line for csv files

* [#18] Change from using length to Enum.count() for counting list size

* [#18] Change from using string field to text for keyword html storage to remove character limit

* [#18] Add positive test result for KeywordUpload changeset

* [#18] Remove unnecessary conn.halts from keyword controller

* [#18] Remove comments and changed grammar in test cases for Keywords

* [#18] Change name and status fields of KeywordUpload to be to not accept null

* [#18] Refactor parsing of keywords into correct structure for bulk inserts wip

* [#18] Add two further KeywordUpload changeset tests to ensure a KeywordUpload has to have an existing user

* [#1] [#20] As a user, I can view a list of my previously uploaded keywords (#35)

* #[1] Add Context function to retreive list of uploaded keywords for a particular user

* #[1] Add Controller and template to show the list of uploaded keywords for the user

* #[1] Change github action trigger from Pull Request to Push to allow staging and prod

* #[1] Change github action trigger from Pull Request to Push to allow staging and prod

* [#1] Add Uploaded field to display for each KeywordUpload and format using Calendar module

* #[1] Add test for KeywordView for formatting timestamp

* #[1] Change Repo.list_all to return the inserted Keywords

* #[1] Write tests for fetching KeywordUploads for a particular user

* #[1] Add KeywordUpload Factory to tests for listing KeywordUploads for a User

* [#1] Refactor keyword test using pipe operator to make it cleaner

* [#1] Fix formatting on keywords index template file

* [#1] Clean up keyword template file

* [#1] Remove external Calendar library dependencies due to built-in functionality in Elxiir

* [#1] Made keyword test title more explicit

* Remove prod.secret.exe config import to allow deployment (#36)

* [#24] Add client to interface with Google Search to carry out queries and receive HTML respose

* [#24] Add HTTPoison dependency for Google Search Client

* [#24] Add KeywordUpload functions and changesets for updating the status and HTML

* [#24] Add Oban job and worker for fetching HTML for KeywordUpload and updating its status

* [#24] Modify KeywordController to handle creation of KeywordUpload jobs via helper function

* [#24] Refactor GoogleSearchClient function name

* [#24] Fix oban job not scheduling with delay

* [#24] Fix oban job not scheduling with delay

* [#24] Refactor delay time to function argument

* [#24] Add test using ExVCR to ensure GoogleSearchClient returns valid response

* [#24] Add test using ExVCR to ensure GoogleSearchClient returns valid response

* [#24] Add tests for the worker that performs the retrieving of the HTML and updating the status

* [#24] Remove pattern matching on HTTPoison error result due to current testability issues

* [#24] Add additional tests for KeywordUpload changesets for updating the status and html

* [#24] Add additional tests for KeywordUpload context functions for updating the status and html

* [#24] Change from Enum.zip_with function to Enum.with_index to generate delays for each job to improve readability

* [#24] Add additional assertion for the status prior to change to make it more explicit that the status gets changed successfully

* [#24] Make test title for the valid html changeset result more explicit

* [#24] Change the KeywordUpload status to failed when max attempts have been reached

* [#24] Create test for the job creation helper function to ensure jobs are inserted with delay

* [#24] Reverted one line multi alias to conform to formatting warning

* [#24] Change function guard to simpler pattern match for keyword upload worker attempts

* [#24] Clean up search worker using pipes

* [#24] Update job creation helper file name to include the suffix of helper

* [#24] Update job creation helper file name to include the suffix of helper

* [#24] Rename module SearchWorker to KeywordSearchWorker to reflect file name

* [#24] Explicitly set uploaded keyword status during status update tests

* [#24] Change function name insert_keyword_upload_html to update_keyword_upload_html

* [#24] Clean up formatting of job_creation_helper_test

* [#24] Clea up pattern matching to make code cleaner for checking keyword uplaod status

* [#24] Add error response to GoogleSearchClient for 500 server errors with associated stub cassette test

* [#24] Fix coding style on keyword search worker

* [#24] Add additional error response and test for GoogleSearchClient to handle unhandled responses

* [#24] Add additional error response and test for GoogleSearchClient to handle unhandled responses

* [Chore] [#40] Update project structure and naming according to Nimble standards (#41)

* Release - 0.2.0 (#37)

* Update README.md

* Updated gettext errors

* Removed unused variables

* Fixed linting/formatting issues

* Fixed linting/formatting issues

* Update deploy_heroku.yml

* Rename deploy_heroku.yml to deploy_heroku_staging.yml

* Create deploy_heroku_prod.yaml

* Update deploy_heroku_staging.yml

* As a user I can sign up and sign in with a valid e-mail and password  (#28)

* [#6 #16] As a user I can sign in with a valid e-mail and password

* Remove unused files

* Remove unsused update function for User

* Add session for user after log in

* Add unique email constraint on Users table

* Add user sign in

* Add current sign in status for user

* Add user sign out functionality (not in backlog)

* Remove / refactored code

* Remove coverage check for currently unused plug

* Add controller tests

* Prepare ExMachina for testing

* User sign out now displays a message

* Change session deletion method to ensure persistence of message to user upon sign out

* Refactored password hashing function so it can be used in future tests

* Refactor fixture to use ExMachina and Faker for data generation

* Modify ExUnit tests to conform to standards

* Tidy template pages with correct formatting

* Remove comments and cleaned up code

* Merge migrations into single file for User schema

* Remove comments and cleaned up code

* Correct English used in ExUnit test case

* Move secret_key-base file to environment variable for production

* Change multiple alias identifiers from one line to multiple to satisy codebase

* Make blank line seperation more consistent in the tests

* Add feature test case for User log in

* [#6 #7 #16 #22] Add account tests for invalid cases and fixed formatting

* [#6 #7 #16 #22] Add account tests for invalid cases and fixed formatting

* [#6 #7 #16 #22] Refactor code according to feedback

* [#6 #7 #16 #22] Refactor code according to feedback

* [#6 #7 #16 #22] Remove code form AuthController to AuthHelper to reflect the functionality

* [#6 #7 #16 #22] Remove auto-generated function spec

* [#6 #7 #16 #22] Moved Account context into accounts folder and account schema into its own folder to improve structure

* [#6 #7 #16 #22] Add tests for user changesets for negative paths

* [#6 #7 #16 #22] Add tests for user changesets for negative paths

* [#6 #7 #16 #22] Change refute to assert false for testing outcomese

* [#6 #7 #16 #22] Changed from pattern matching to double equals to match exact output when required. Move value being tested against to the right side

* [#3] [UI] As a user, I can upload a CSV file containing keywords which will then be used to search on Google (#31)

* [#6 #16] As a user I can sign in with a valid e-mail and password

* Remove unused files

* Remove unsused update function for User

* Add session for user after log in

* Add unique email constraint on Users table

* Add user sign in

* Add current sign in status for user

* Add user sign out functionality (not in backlog)

* Remove / refactored code

* Remove coverage check for currently unused plug

* Add controller tests

* Prepare ExMachina for testing

* User sign out now displays a message

* Change session deletion method to ensure persistence of message to user upon sign out

* Refactored password hashing function so it can be used in future tests

* Refactor fixture to use ExMachina and Faker for data generation

* Modify ExUnit tests to conform to standards

* Tidy template pages with correct formatting

* Remove comments and cleaned up code

* Merge migrations into single file for User schema

* Remove comments and cleaned up code

* Correct English used in ExUnit test case

* Move secret_key-base file to environment variable for production

* Change multiple alias identifiers from one line to multiple to satisy codebase

* Make blank line seperation more consistent in the tests

* Add feature test case for User log in

* [#6 #7 #16 #22] Add account tests for invalid cases and fixed formatting

* [#6 #7 #16 #22] Add account tests for invalid cases and fixed formatting

* [#6 #7 #16 #22] Refactor code according to feedback

* [#6 #7 #16 #22] Refactor code according to feedback

* [#6 #7 #16 #22] Remove code form AuthController to AuthHelper to reflect the functionality

* [#6 #7 #16 #22] Remove auto-generated function spec

* [#6 #7 #16 #22] Moved Account context into accounts folder and account schema into its own folder to improve structure

* [#6 #7 #16 #22] Add tests for user changesets for negative paths

* [#6 #7 #16 #22] Add tests for user changesets for negative paths

* [#3] Add endpoint and controller for handling keyword upload

* [#3] Add template files for uploading files, including upload form

* [#3] Add link to keywords page in navigation

* [#3] Remove coverall and comments from authenticated plug to prepare for tests

* [#3] Add tests for keyword controller and test csv file

* [#3] Add tests for ensure_authenticated plug

* Resolved merge conflict

* [#3] Format code

* [#3] Change keywords fixture file name and changed template to show 1000 keywords limit

* [#3] Remove blank line and re-order assert tests for ensure_authenticated plug tests

* [#3] Add an additional test to ensure unauthenticated users are unable to upload a keywords file

* [#3] Format test

* [#18] [Backend] As a user, I can upload a CSV file containing keywords which will be stored (#33)

* [#6 #16] As a user I can sign in with a valid e-mail and password

* Remove unused files

* Remove unsused update function for User

* Add session for user after log in

* Add unique email constraint on Users table

* Add user sign in

* Add current sign in status for user

* Add user sign out functionality (not in backlog)

* Remove / refactored code

* Remove coverage check for currently unused plug

* Add controller tests

* Prepare ExMachina for testing

* User sign out now displays a message

* Change session deletion method to ensure persistence of message to user upon sign out

* Refactored password hashing function so it can be used in future tests

* Refactor fixture to use ExMachina and Faker for data generation

* Modify ExUnit tests to conform to standards

* Tidy template pages with correct formatting

* Remove comments and cleaned up code

* Merge migrations into single file for User schema

* Remove comments and cleaned up code

* Correct English used in ExUnit test case

* Move secret_key-base file to environment variable for production

* Change multiple alias identifiers from one line to multiple to satisy codebase

* Make blank line seperation more consistent in the tests

* Add feature test case for User log in

* [#6 #7 #16 #22] Add account tests for invalid cases and fixed formatting

* [#6 #7 #16 #22] Add account tests for invalid cases and fixed formatting

* [#6 #7 #16 #22] Refactor code according to feedback

* [#6 #7 #16 #22] Refactor code according to feedback

* [#6 #7 #16 #22] Remove code form AuthController to AuthHelper to reflect the functionality

* [#6 #7 #16 #22] Remove auto-generated function spec

* [#6 #7 #16 #22] Moved Account context into accounts folder and account schema into its own folder to improve structure

* [#6 #7 #16 #22] Add tests for user changesets for negative paths

* [#6 #7 #16 #22] Add tests for user changesets for negative paths

* [#3] Add endpoint and controller for handling keyword upload

* [#3] Add template files for uploading files, including upload form

* [#3] Add link to keywords page in navigation

* [#3] Remove coverall and comments from authenticated plug to prepare for tests

* [#3] Add tests for keyword controller and test csv file

* [#3] Add tests for ensure_authenticated plug

* Resolved merge conflict

* [#3] Format code

* [#3] Change keywords fixture file name and changed template to show 1000 keywords limit

* [#3] Remove blank line and re-order assert tests for ensure_authenticated plug tests

* [#3] Add an additional test to ensure unauthenticated users are unable to upload a keywords file

* [#3] Format test

* [#18] Add NimbleCSV

* [#18] Add Keyword Controller and helper function for validate/parse csv

* [#18] Add Keyword Controller tests and additional test files for invalid cases

* [#18] Change invalid file format to invalid file extension

* [#18] Initial KeywordUpload Schema, associations and tests

* [#18] Initial code to carry out the mass insertions of keywords into the table for the User. Needs refactor wip

* [#18] Refactor KeywordUpload changeset to Use __MODULE__ as default argument

* #[18] Slight refactor of keyword saving for user wip

* Refactor Keyword context name and add one test

* [#18] Change alias to fix formatting errors

* [#18] Add test to Keyword Controller to verify an uplaod of two keywords returns the correct count to the user

* [#18] Add additional empty line for csv files

* [#18] Change from using length to Enum.count() for counting list size

* [#18] Change from using string field to text for keyword html storage to remove character limit

* [#18] Add positive test result for KeywordUpload changeset

* [#18] Remove unnecessary conn.halts from keyword controller

* [#18] Remove comments and changed grammar in test cases for Keywords

* [#18] Change name and status fields of KeywordUpload to be to not accept null

* [#18] Refactor parsing of keywords into correct structure for bulk inserts wip

* [#18] Add two further KeywordUpload changeset tests to ensure a KeywordUpload has to have an existing user

* [#1] [#20] As a user, I can view a list of my previously uploaded keywords (#35)

* #[1] Add Context function to retreive list of uploaded keywords for a particular user

* #[1] Add Controller and template to show the list of uploaded keywords for the user

* #[1] Change github action trigger from Pull Request to Push to allow staging and prod

* #[1] Change github action trigger from Pull Request to Push to allow staging and prod

* [#1] Add Uploaded field to display for each KeywordUpload and format using Calendar module

* #[1] Add test for KeywordView for formatting timestamp

* #[1] Change Repo.list_all to return the inserted Keywords

* #[1] Write tests for fetching KeywordUploads for a particular user

* #[1] Add KeywordUpload Factory to tests for listing KeywordUploads for a User

* [#1] Refactor keyword test using pipe operator to make it cleaner

* [#1] Fix formatting on keywords index template file

* [#1] Clean up keyword template file

* [#1] Remove external Calendar library dependencies due to built-in functionality in Elxiir

* [#1] Made keyword test title more explicit

* Remove prod.secret.exe config import to allow deployment (#36)

* [#40] Update .gitignore

* [#40] Changed context naming to plural form, and updated folder and module naming consistency with tests

* [#40] Add newline for end of .gitignore file

* [#40] Refactored tests and Exmachina to build the User through the Keyword factory insteado f separately in the tests

* [#24] [Backend] Parse the HTML and store URL data for the Keyword Upload search for the User (#43)

* [#24] Initial schema and migrations for storing URL data for Keyword Uploads

* [#24] Add initial tests for validating url data into changeset

* [#24] Clean up Keyword Upload factory

* [#24] Change to realistic uploaded search data to ensure adwords appear on page

* [#24] Remove doc comment

* [#24] Initial adword parsing wip

* [#24] Initial insertion of url data for keyword upload

* [#24] Refactor keyword parsing

* [#24] Refactor keyword parsing

* [#24] Modified test to check for completed status

* [#24] Add custom cassettes to test parser

* [#24] Add factory for creating url data for search results

* [#24] Add tests for parsing html links

* [#24] Add test for creating search results from url data

* [#24] Add tests for parsing html links

* [#24] Add additional test data to ensure bottom ads are displayed for cassettes

* [#24] Add additional test for bottom adwords parsing

* [#24] Add additional tests to ensure errors are created for invalid search url data

* [#24] Refactored search_result_url_date to search_result_url

* [#24] Refactor search_result_url_data to search_result_url

* [#24] Refactor naming of variables and functions related to url data

* [#24] Add blank line to avoid warnings for csv

* [#24] Move HTTPoison and cassette config out of test files to DataCse

* [#24] Move cassete files to correct location

* [#24] Refactor keyword parser for readability

* [#24] Clean up keyword URL parsing further

* [#24] Remove custom cassettes as not required

* [#24] Refactor google search result parser

* [24] Change name of google url parsing function

* [#24] Change vcr cassette names to clarify its usage

* [#24] Change name of functions used to parse individual urls from adwords

* Remove files that I accidently accepted back in merge conflict?
  • Loading branch information
liamstevens111 authored Jun 2, 2022
1 parent ba2617e commit 4bed138
Show file tree
Hide file tree
Showing 41 changed files with 792 additions and 101 deletions.
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,7 @@ google_search_data_viewer-*.tar
npm-debug.log
/assets/node_modules/

.DS_Store

# Ignore .iex.exs files in case you like to edit your preload commands locally.
.iex.exs
2 changes: 1 addition & 1 deletion config/config.exs
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ config :phoenix, :json_library, Jason
config :google_search_data_viewer, Oban,
repo: GoogleSearchDataViewer.Repo,
plugins: [Oban.Plugins.Pruner],
queues: [default: 10]
queues: [default: 10, keyword_search: 10]

# Import environment specific config. This must remain at the bottom
# of this file so it overrides the configuration defined above.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
defmodule GoogleSearchDataViewer.Accounts.Account do
defmodule GoogleSearchDataViewer.Account.Accounts do
import Ecto.Query, warn: false

alias GoogleSearchDataViewer.Accounts.Passwords
alias GoogleSearchDataViewer.Accounts.Schemas.User
alias GoogleSearchDataViewer.Account.Passwords
alias GoogleSearchDataViewer.Account.Schemas.User
alias GoogleSearchDataViewer.Repo

def list_users, do: Repo.all(User)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
defmodule GoogleSearchDataViewer.Accounts.Passwords do
defmodule GoogleSearchDataViewer.Account.Passwords do
def hash_password(password), do: Bcrypt.hash_pwd_salt(password)

def verify_password(password, hashed_password), do: Bcrypt.verify_pass(password, hashed_password)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
defmodule GoogleSearchDataViewer.Accounts.Schemas.User do
defmodule GoogleSearchDataViewer.Account.Schemas.User do
use Ecto.Schema

import Ecto.Changeset

alias GoogleSearchDataViewer.Accounts.Passwords
alias GoogleSearchDataViewer.Keywords.Schemas.KeywordUpload
alias GoogleSearchDataViewer.Account.Passwords
alias GoogleSearchDataViewer.Keyword.Schemas.KeywordUpload

schema "users" do
field :email, :string
Expand Down
22 changes: 22 additions & 0 deletions lib/google_search_data_viewer/keyword/google_search_client.ex
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
defmodule GoogleSearchDataViewer.Keyword.GoogleSearchClient do
@base_url "https://www.google.com/search?q="
@headers [
{"User-Agent",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36"}
]

def get_html(keyword) do
search_url = @base_url <> URI.encode(keyword)

case HTTPoison.get(search_url, @headers) do
{:ok, %HTTPoison.Response{status_code: 200, body: body}} ->
{:ok, body}

{:ok, %HTTPoison.Response{status_code: 500}} ->
{:error, "Internal server error"}

{:ok, response = %HTTPoison.Response{status_code: _}} ->
{:error, response}
end
end
end
50 changes: 50 additions & 0 deletions lib/google_search_data_viewer/keyword/google_search_parser.ex
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
defmodule GoogleSearchDataViewer.Keyword.GoogleSearchParser do
@css_search_selectors %{
top_adwords: "#tads > .uEierd a.sVXRqc",
top_non_adwords: ".MhgNwc a",
non_adwords: ".yuRUbf a",
bottom_adwords: "#bottomads .uEierd a.sVXRqc"
}

def parse_html_urls(html) do
{_, parsed_html} = Floki.parse_document(html)

[]
|> parse_top_adwords(parsed_html)
|> parse_top_non_adwords(parsed_html)
|> parse_non_adwords(parsed_html)
|> parse_bottom_adwords(parsed_html)
end

defp parse_top_adwords(url_stats, parsed_html) do
parsed_html
|> Floki.find(@css_search_selectors.top_adwords)
|> Floki.attribute("href")
|> Enum.map(fn url -> %{url: url, is_adword: true, is_top_adword: true} end)
|> Enum.concat(url_stats)
end

defp parse_top_non_adwords(url_stats, parsed_html) do
parsed_html
|> Floki.find(@css_search_selectors.top_non_adwords)
|> Floki.attribute("href")
|> Enum.map(fn url -> %{url: url, is_adword: false, is_top_adword: false} end)
|> Enum.concat(url_stats)
end

defp parse_non_adwords(url_stats, parsed_html) do
parsed_html
|> Floki.find(@css_search_selectors.non_adwords)
|> Floki.attribute("href")
|> Enum.map(fn url -> %{url: url, is_adword: false, is_top_adword: false} end)
|> Enum.concat(url_stats)
end

defp parse_bottom_adwords(url_stats, parsed_html) do
parsed_html
|> Floki.find(@css_search_selectors.bottom_adwords)
|> Floki.attribute("href")
|> Enum.map(fn url -> %{url: url, is_adword: true, is_top_adword: false} end)
|> Enum.concat(url_stats)
end
end
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
defmodule GoogleSearchDataViewer.Keywords.Keyword do
defmodule GoogleSearchDataViewer.Keyword.Keywords do
import Ecto.Query, warn: false

alias GoogleSearchDataViewer.Keywords.Schemas.KeywordUpload
alias GoogleSearchDataViewer.Keyword.Schemas.KeywordUpload
alias GoogleSearchDataViewer.Repo

def get_keyword_upload(id), do: Repo.get(KeywordUpload, id)

def get_keyword_uploads_for_user(user) do
KeywordUpload
|> where(user_id: ^user.id)
Expand All @@ -12,6 +14,18 @@ defmodule GoogleSearchDataViewer.Keywords.Keyword do
|> Repo.all()
end

def update_keyword_upload_status(keyword_upload, status) do
keyword_upload
|> KeywordUpload.status_changeset(status)
|> Repo.update()
end

def update_keyword_upload_html(keyword_upload, html) do
keyword_upload
|> KeywordUpload.html_changeset(%{html: html})
|> Repo.update()
end

def insert_keyword_uploads(attrs) do
Repo.insert_all(KeywordUpload, attrs, returning: true)
end
Expand All @@ -29,7 +43,7 @@ defmodule GoogleSearchDataViewer.Keywords.Keyword do
end)
|> Enum.map(fn params -> create_changeset_and_parse(params) end)
|> Enum.map(&Map.from_struct/1)
|> Enum.map(fn params -> Map.drop(params, [:__meta__, :user, :id]) end)
|> Enum.map(fn params -> Map.drop(params, [:__meta__, :user, :id, :search_result_urls]) end)
|> Enum.map(fn params -> insert_timestamps(params) end)
end

Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
defmodule GoogleSearchDataViewer.Keywords.Schemas.KeywordUpload do
defmodule GoogleSearchDataViewer.Keyword.Schemas.KeywordUpload do
use Ecto.Schema

import Ecto.Changeset

alias GoogleSearchDataViewer.Accounts.Schemas.User
alias GoogleSearchDataViewer.Account.Schemas.User
alias GoogleSearchDataViewer.Keyword.Schemas.SearchResultUrl

schema "keyword_uploads" do
field :name, :string
Expand All @@ -13,6 +14,7 @@ defmodule GoogleSearchDataViewer.Keywords.Schemas.KeywordUpload do
values: [:pending, :inprogress, :completed, :failed],
default: :pending

has_many :search_result_urls, SearchResultUrl
belongs_to :user, User

timestamps()
Expand All @@ -24,4 +26,14 @@ defmodule GoogleSearchDataViewer.Keywords.Schemas.KeywordUpload do
|> validate_required([:name, :user_id])
|> assoc_constraint(:user)
end

def html_changeset(keyword_upload \\ %__MODULE__{}, html) do
keyword_upload
|> cast(html, [:html])
|> validate_required([:html])
end

def status_changeset(keyword_upload \\ %__MODULE__{}, status) do
change(keyword_upload, status: status)
end
end
24 changes: 24 additions & 0 deletions lib/google_search_data_viewer/keyword/schemas/search_result_url.ex
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
defmodule GoogleSearchDataViewer.Keyword.Schemas.SearchResultUrl do
use Ecto.Schema

import Ecto.Changeset

alias GoogleSearchDataViewer.Keyword.Schemas.KeywordUpload

schema "search_result_urls" do
field :url, :string
field :is_adword, :boolean, default: false
field :is_top_adword, :boolean, default: false

belongs_to :keyword_upload, KeywordUpload

timestamps()
end

def changeset(search_result_url \\ %__MODULE__{}, attrs) do
search_result_url
|> cast(attrs, [:url, :is_adword, :is_top_adword, :keyword_upload_id])
|> validate_required([:url, :keyword_upload_id])
|> assoc_constraint(:keyword_upload)
end
end
10 changes: 10 additions & 0 deletions lib/google_search_data_viewer/keyword/search_results.ex
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
defmodule GoogleSearchDataViewer.Keyword.SearchResults do
alias GoogleSearchDataViewer.Keyword.Schemas.SearchResultUrl
alias GoogleSearchDataViewer.Repo

def create_search_results(url_stats) do
url_stats
|> SearchResultUrl.changeset()
|> Repo.insert()
end
end
Original file line number Diff line number Diff line change
@@ -1,19 +1,22 @@
defmodule GoogleSearchDataViewerWeb.KeywordController do
use GoogleSearchDataViewerWeb, :controller

alias GoogleSearchDataViewer.Keywords.Keyword
alias GoogleSearchDataViewer.Keyword.Keywords
alias GoogleSearchDataViewerWeb.KeywordHelper
alias GoogleSearchDataViewerWorker.Keyword.Keywords, as: WorkerKeywords

def index(conn, _params) do
keywords = Keyword.get_keyword_uploads_for_user(conn.assigns.current_user)
keywords = Keywords.get_keyword_uploads_for_user(conn.assigns.current_user)
render(conn, "index.html", keywords: keywords)
end

def upload(conn, %{"file" => file}) do
case KeywordHelper.validate_and_parse_keyword_file(file) do
{:ok, keywords} ->
{keyword_count, _keywords} =
Keyword.create_keyword_uploads(keywords, conn.assigns.current_user)
{:ok, parsed_keywords} ->
{keyword_count, keyword_uploads} =
Keywords.create_keyword_uploads(parsed_keywords, conn.assigns.current_user)

WorkerKeywords.create_keyword_upload_jobs_with_delay(keyword_uploads)

conn
|> put_flash(:info, "File successfully uploaded. #{keyword_count} keywords uploaded.")
Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
defmodule GoogleSearchDataViewerWeb.SessionController do
use GoogleSearchDataViewerWeb, :controller

alias GoogleSearchDataViewer.Accounts.Account
alias GoogleSearchDataViewer.Account.Accounts

def new(conn, _params), do: render(conn, "new.html")

def create(conn, %{"email" => email, "password" => password}) do
case Account.validate_email_and_password(email, password) do
case Accounts.validate_email_and_password(email, password) do
{:ok, user} ->
conn
|> GoogleSearchDataViewerWeb.AuthHelper.sign_in(user)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
defmodule GoogleSearchDataViewerWeb.UserController do
use GoogleSearchDataViewerWeb, :controller

alias GoogleSearchDataViewer.Accounts.Account
alias GoogleSearchDataViewer.Accounts.Schemas.User
alias GoogleSearchDataViewer.Account.Accounts
alias GoogleSearchDataViewer.Account.Schemas.User

def new(conn, _params) do
changeset = User.changeset(%User{}, %{})
Expand All @@ -11,7 +11,7 @@ defmodule GoogleSearchDataViewerWeb.UserController do
end

def create(conn, %{"user" => user_params}) do
case Account.create_user(user_params) do
case Accounts.create_user(user_params) do
{:ok, user} ->
conn
|> GoogleSearchDataViewerWeb.AuthHelper.sign_in(user)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ defmodule GoogleSearchDataViewerWeb.EnsureAuthenticatedPlug do
import Plug.Conn
import Phoenix.Controller

alias GoogleSearchDataViewer.Accounts.Account
alias GoogleSearchDataViewer.Account.Accounts
alias GoogleSearchDataViewerWeb.Router.Helpers, as: Routes

def init(opts), do: opts
Expand All @@ -26,5 +26,5 @@ defmodule GoogleSearchDataViewerWeb.EnsureAuthenticatedPlug do

defp get_user(nil), do: nil

defp get_user(user_id), do: Account.get_user(user_id)
defp get_user(user_id), do: Accounts.get_user(user_id)
end
4 changes: 2 additions & 2 deletions lib/google_search_data_viewer_web/plugs/put_current_user.ex
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
defmodule GoogleSearchDataViewerWeb.PutCurrentUserPlug do
import Plug.Conn

alias GoogleSearchDataViewer.Accounts.Account
alias GoogleSearchDataViewer.Account.Accounts

def init(opts), do: opts

def call(conn, _opts) do
user_id = get_session(conn, :user_id)

user = user_id && Account.get_user(user_id)
user = user_id && Accounts.get_user(user_id)

assign(conn, :current_user, user)
end
Expand Down
Empty file.
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
defmodule GoogleSearchDataViewerWorker.Keyword.KeywordSearchWorker do
use Oban.Worker,
queue: :keyword_search,
max_attempts: 3,
unique: [period: 30]

alias GoogleSearchDataViewer.Keyword.GoogleSearchClient
alias GoogleSearchDataViewer.Keyword.GoogleSearchParser
alias GoogleSearchDataViewer.Keyword.Keywords
alias GoogleSearchDataViewer.Keyword.SearchResults

@max_attempts 3

@impl Oban.Worker
def perform(%Oban.Job{args: %{"keyword_id" => keyword_id}, attempt: @max_attempts}) do
{:ok, _} =
keyword_id
|> Keywords.get_keyword_upload()
|> Keywords.update_keyword_upload_status(:failed)

{:error, "max attempts reached, attempt: #{@max_attempts}"}
end

@impl Oban.Worker
def perform(%Oban.Job{args: %{"keyword_id" => keyword_id}}) do
{_, keyword_upload} =
keyword_id
|> Keywords.get_keyword_upload()
|> Keywords.update_keyword_upload_status(:inprogress)

{:ok, html_response} = GoogleSearchClient.get_html(keyword_upload.name)

{:ok, _} = Keywords.update_keyword_upload_html(keyword_upload, html_response)

html_response
|> GoogleSearchParser.parse_html_urls()
|> Enum.map(fn url_stats -> Map.put(url_stats, :keyword_upload_id, keyword_upload.id) end)
|> Enum.each(fn search_result_url ->
SearchResults.create_search_results(search_result_url)
end)

Keywords.update_keyword_upload_status(keyword_upload, :completed)

:ok
end
end
12 changes: 12 additions & 0 deletions lib/google_search_data_viewer_worker/keyword/keywords.ex
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
defmodule GoogleSearchDataViewerWorker.Keyword.Keywords do
alias GoogleSearchDataViewerWorker.Keyword.KeywordSearchWorker

def create_keyword_upload_jobs_with_delay(keyword_uploads, delay \\ 3) do
keyword_uploads
|> Enum.with_index()
|> Enum.map(fn {keyword_upload, index} ->
KeywordSearchWorker.new(%{keyword_id: keyword_upload.id}, schedule_in: index * delay)
end)
|> Oban.insert_all()
end
end
Loading

0 comments on commit 4bed138

Please sign in to comment.