[f] Upgrade Python version from 3.9 to 3.11 #71
Conversation
WalkthroughThe changes in this pull request involve updating the Python version across multiple GitHub Actions workflow files and the Changes
Poem
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Outside diff range and nitpick comments (4)
Pipfile (2)
Line range hint
7-11: Consider pinning dependency versions for stability.Using wildcards (
*) for critical dependencies likescrapyandscrapy-wayback-middlewarecould lead to unexpected behavior if incompatible versions are pulled during installation. Additionally, using themainbranch forcity-scrapers-coremight introduce breaking changes.Consider pinning versions:
[packages] -scrapy = "*" +scrapy = "2.11.0" # Latest stable version with Python 3.11 support scrapy-sentry-errors = "1.0.0" -city-scrapers-core = {ref = "main", git = "https://github.com/City-Bureau/city-scrapers-core.git", extras = ["azure"]} +city-scrapers-core = {ref = "v2.3.0", git = "https://github.com/City-Bureau/city-scrapers-core.git", extras = ["azure"]} # Use a stable tag -scrapy-wayback-middleware = "*" +scrapy-wayback-middleware = "2.1.0" # Latest stable version
Line range hint
13-16: Update Black to latest version compatible with Python 3.11.Black version 22.6 is quite old and might have issues with Python 3.11 syntax features. The latest version of Black has better support for Python 3.11.
Consider updating Black:
[dev-packages] freezegun = "*" pytest = "*" "flake8" = "*" isort = "*" -black = "==22.6" +black = "==23.10.1" # Latest stable version with full Python 3.11 support.github/workflows/archive.yml (1)
Line range hint
31-35: Consider updating cache strategy for Python 3.11 transitionThe cache configuration looks good, but during the Python version transition, you might want to consider:
- Old caches from Python 3.9 won't be reused (which is correct)
- Initial workflows might be slower as new caches are built
To optimize the transition period, you could temporarily adjust the cache key to include a transition identifier if you need to revert quickly:
- key: ${{ env.PYTHON_VERSION }}-${{ hashFiles('**/Pipfile.lock') }} + key: ${{ env.PYTHON_VERSION }}-py311transition-${{ hashFiles('**/Pipfile.lock') }}.github/workflows/cron.yml (1)
Line range hint
48-49: Consider adding test runs with both Python versionsDuring the transition period, consider running the scrapers with both Python 3.9 and 3.11 in parallel (not in production, but in a test environment) to ensure consistent results and catch any version-specific issues.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
⛔ Files ignored due to path filters (1)
Pipfile.lockis excluded by!**/*.lock
📒 Files selected for processing (4)
.github/workflows/archive.yml(1 hunks).github/workflows/ci.yml(1 hunks).github/workflows/cron.yml(1 hunks)Pipfile(1 hunks)
🔇 Additional comments (9)
Pipfile (1)
22-22: LGTM on Python 3.11 upgrade! 🚀
The upgrade to Python 3.11 is a good move as it brings significant performance improvements, especially beneficial for web scraping workloads.
Let's verify package compatibility with Python 3.11:
✅ Verification successful
The previous attempt to check package compatibility didn't work well because we don't have the package metadata in the repository. Let's examine the actual dependencies and CI configuration to better assess the Python 3.11 upgrade impact.
Python 3.11 upgrade looks safe to proceed ✅
The upgrade appears well-prepared:
- All GitHub workflows are already configured for Python 3.11
- Most dependencies use flexible version specifiers (*) which allows compatibility
- The pinned dependencies are known to work with Python 3.11:
- black==22.6 works with Python 3.11 despite being older
- scrapy-sentry-errors=1.0.0 is a simple middleware without version-specific code
The only potential concern is the city-scrapers-core dependency from git main branch, but since the workflows are already running with Python 3.11, this indicates it's compatible.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Check if any dependencies have known compatibility issues with Python 3.11
# Test: Check package metadata for Python version support
for pkg in scrapy scrapy-sentry-errors python-dateutil importlib-resources freezegun pytest flake8 isort black; do
echo "Checking $pkg..."
rg -A 5 "Programming Language :: Python :: 3\.11" $(fd "setup.py|pyproject.toml" | grep -i "$pkg")
done
Length of output: 2468
Script:
#!/bin/bash
# Let's check the actual dependencies and their versions from Pipfile
cat Pipfile
# Also check if there are any GitHub workflows that might need updates
fd "\.ya?ml$" .github/workflows/ -x cat {}
Length of output: 5343
.github/workflows/archive.yml (2)
Line range hint 41-43: LGTM: Correct environment variable usage
The PIPENV_DEFAULT_PYTHON_VERSION is correctly set to use the workflow's Python version.
10-10: Verify dependency compatibility with Python 3.11
While upgrading to Python 3.11 is a good move for performance improvements, we should verify that all dependencies in the project support this version.
Let's check the dependencies compatibility:
✅ Verification successful
Based on the Pipfile.lock analysis, I can now provide the final response:
Python 3.11 upgrade is compatible with all dependencies
The Pipfile.lock shows that all dependencies either:
- Have no Python version constraints
- Support Python >= 3.7 or 3.8
- Are explicitly compatible with Python 3.11 (as shown by the _meta.requires.python_version)
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Check if Pipfile.lock exists and examine its content for potential compatibility issues
if [ -f "Pipfile.lock" ]; then
echo "Checking Pipfile.lock for Python version and dependencies..."
# Extract and display Python version requirement
jq '._meta.requires.python_version' Pipfile.lock
# List all dependencies with their versions
jq '.default' Pipfile.lock
else
echo "Pipfile.lock not found. Please run 'pipenv lock' first."
fi
Length of output: 51029
.github/workflows/cron.yml (3)
Line range hint 33-37: Cache configuration looks good
The cache key properly includes the Python version (${{ env.PYTHON_VERSION }}), ensuring clean dependency installation after the version upgrade.
10-10: LGTM! Verify dependency compatibility with Python 3.11
The Python version upgrade to 3.11 is a good choice for performance improvements. However, ensure all dependencies in Pipfile.lock are compatible with Python 3.11.
Let's verify the dependencies:
#!/bin/bash
# Description: Check Pipfile.lock for potential compatibility issues with Python 3.11
# Test: Look for Python version requirements in Pipfile.lock
cat Pipfile.lock | jq -r '.default | to_entries[] | select(.value.markers != null) | "\(.key): \(.value.markers)"'Line range hint 44-46: Verify scraper compatibility with Python 3.11
The scraper execution environment is properly configured. However, since this is a major version upgrade, ensure that all custom scraper code is compatible with Python 3.11 syntax and features.
Let's check for potential compatibility issues:
✅ Verification successful
Python 3.11 compatibility verified - no issues found
Based on the analysis of the Python codebase:
- All imports are using modern Python syntax
- No deprecated dictionary/set/list methods found
- No usage of removed modules (imp, distutils)
- Type hints follow modern Python syntax
- Dependencies in Pipfile.lock show compatibility with Python >= 3.8
- No async/await patterns that would need review
- No deprecated collections or typing patterns detected
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Look for Python syntax that might need updates for 3.11
# Test 1: Check for deprecated syntax or patterns
rg -g '*.py' 'collections\.Mapping|collections\.Sequence|asyncio\.coroutine|imp\.'
# Test 2: Look for type annotation syntax that might need updates
rg -g '*.py' 'typing\.Collection|typing\.Sequence|typing\.Mapping'
Length of output: 2104
Script:
#!/bin/bash
# Let's check for Python files and their content to verify compatibility
# Find all Python files and check their imports and syntax patterns
fd -e py -x cat {} | grep -E "^(import|from.*import)"
# Check for any async/await usage which might need review
rg -g "*.py" "async def|await"
# Look for any custom type hints or annotations
rg -g "*.py" "^[^#]*: *[A-Z]|-> *[A-Z]"
# Check for deprecated dictionary/set/list methods
rg -g "*.py" "\.viewitems\(\)|\.viewkeys\(\)|\.viewvalues\(\)|\.has_key\(\)"
# Check for usage of removed modules or functions
rg -g "*.py" "from imp import|import imp|from distutils\.|import distutils"
Length of output: 3957
.github/workflows/ci.yml (3)
Line range hint 42-44: Consider updating pipenv environment variables.
The PIPENV_DEFAULT_PYTHON_VERSION is correctly set, but you might want to verify that all dependencies in your Pipfile.lock are compatible with Python 3.11.
Let's check for any potential compatibility issues:
#!/bin/bash
# Description: Check for Python version specific dependencies
# Expected: No dependencies should be pinned to Python < 3.11
# Check Pipfile.lock for version constraints
echo "Checking Pipfile.lock for version constraints..."
if [ -f "Pipfile.lock" ]; then
jq -r '.default | to_entries[] | select(.value.markers != null) | .key + ": " + .value.markers' Pipfile.lock
fi
# Look for any Python version markers in requirements files
echo "Checking requirements files..."
rg "python_version" requirements*.txt || true18-18: LGTM! Python version upgrade looks good.
The upgrade to Python 3.11 is a good choice as it brings significant performance improvements. The workflow is using the latest versions of GitHub Actions (checkout@v4, setup-python@v5, cache@v4) which fully support Python 3.11.
Let's verify the Python version compatibility across the project:
✅ Verification successful
Python 3.11 upgrade is consistently applied across the project ✅
The verification confirms that Python 3.11 is properly configured across all workflow files and the Pipfile:
- All three workflow files (ci.yml, archive.yml, cron.yml) use Python 3.11
- Pipfile explicitly requires Python 3.11
- Dependencies in Pipfile.lock show compatibility with Python 3.8+ which supports Python 3.11
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Verify Python version consistency across configuration files
# Expected: All files should specify Python 3.11
# Check Python version in workflow files
echo "Checking workflow files..."
rg -l "python.*3\.11|PYTHON.*3\.11" .github/workflows/
# Check Python version in Pipfile
echo "Checking Pipfile..."
rg "python_version.*3\.11" Pipfile
# Look for any remaining references to Python 3.9
echo "Checking for old Python version references..."
rg "python.*3\.9|PYTHON.*3\.9"
Length of output: 2253
Script:
#!/bin/bash
# Let's check all workflow files and Pipfile directly
echo "=== Checking workflow files ==="
find .github/workflows -type f -name "*.yml" -exec cat {} \;
echo -e "\n=== Checking Pipfile ==="
cat Pipfile
Length of output: 5500
Line range hint 33-40: Verify cache configuration for Python 3.11.
The cache configuration looks good with appropriate keys based on Python version and Pipfile.lock. However, since this is a major version upgrade, you might want to consider:
- Clearing the cache after the upgrade to prevent any stale dependencies
- Monitoring the first few builds to ensure cache hit rates remain optimal
Let's check the cache configuration:
✅ Verification successful
Let me gather more specific information about the workflow configuration.
Let me check the other workflow files to ensure we have a complete picture of Python caching across all workflows.
Cache configuration for Python 3.11 is properly configured across all workflows
The cache configuration is consistent and well-implemented across all workflow files:
- All workflows use
actions/cache@v4with.venvas the cache path - Cache keys are properly versioned with Python 3.11
- The cache invalidation strategy is appropriate, using both Pipfile.lock hash and Python version
- Restore keys are correctly configured to fall back to broader matches
No changes are needed as:
- The cache will automatically be cleared when Pipfile.lock changes
- The Python version-specific cache key ensures no conflicts between different Python versions
- All workflows (ci.yml, cron.yml, and archive.yml) follow the same caching pattern
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Verify cache configuration consistency across workflow files
# Expected: All workflow files should use similar cache configuration
echo "Checking cache configuration in workflow files..."
rg -A 5 "actions/cache@v4" .github/workflows/
Length of output: 2253
Script:
#!/bin/bash
# Check the complete workflow file to understand Python version matrix and cache usage
cat .github/workflows/ci.yml
# Also check if there are any other workflow files using Python
fd -e yml -e yaml . .github/workflows/ --exec grep -l "python" {}
Length of output: 2066
Script:
#!/bin/bash
# Check the content of other workflow files that use Python
cat .github/workflows/cron.yml
cat .github/workflows/archive.yml
Length of output: 3010
Description
Summary by CodeRabbit
New Features
Chores