feat: add comprehensive dependency monitoring system#4025
feat: add comprehensive dependency monitoring system#4025salmanmkc merged 6 commits intoactions:mainfrom
Conversation
## Core Dependency Check Workflow - Weekly automated dependency status reporting - Checks Node.js, .NET SDK, Docker, npm vulnerabilities - Manual trigger for on-demand checks - GitHub Actions summary with status tables - Proper error handling for npm audit failures ## Documentation - Complete dependency management process documentation - Weekly schedules and responsibilities - Troubleshooting guides for common issues - Manual check commands for release teams ## Key Features - ✅ Multi-dependency monitoring (Node, .NET, Docker, npm) - ✅ Vulnerability detection with severity reporting - ✅ Open PR tracking with dependency labels - ✅ Comprehensive status summaries - ✅ Failure transparency (no false security assurance) This establishes the foundation for monitoring all runner dependencies and provides visibility into security status for release planning.
There was a problem hiding this comment.
Pull Request Overview
This PR adds a comprehensive dependency monitoring system to provide visibility into all runner dependencies and their security status. The system includes automated weekly reporting, manual trigger capabilities, and detailed documentation for release teams.
- Adds automated dependency status checking workflow with multi-component monitoring
- Creates comprehensive process documentation for dependency management
- Implements proper error handling and vulnerability detection for npm security audits
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
.github/workflows/dependency-check.yml |
New workflow for automated dependency monitoring with weekly schedule and manual triggers |
docs/dependency-management.md |
Complete documentation of dependency management processes, schedules, and troubleshooting guides |
| dotnet-status: ${{ steps.check-versions.outputs.dotnet-status }} | ||
| docker-status: ${{ steps.check-versions.outputs.docker-status }} | ||
| buildx-status: ${{ steps.check-versions.outputs.buildx-status }} | ||
| npm-vulnerabilities: ${{ steps.check-versions.outputs.npm-vulnerabilities }} |
There was a problem hiding this comment.
The workflow lacks accompanying tests to exercise the core functionality of the dependency checking logic. Consider adding tests in the Test/L0 directory to verify version comparison, vulnerability detection, and error handling scenarios.
| current_docker=$(grep "ARG DOCKER_VERSION=" ./images/Dockerfile | cut -d'=' -f2) | ||
| current_buildx=$(grep "ARG BUILDX_VERSION=" ./images/Dockerfile | cut -d'=' -f2) | ||
|
|
||
| latest_docker=$(curl -s https://download.docker.com/linux/static/stable/x86_64/ | grep -o 'docker-[0-9]*\.[0-9]*\.[0-9]*\.tgz' | sort -V | tail -n 1 | sed 's/docker-\(.*\)\.tgz/\1/') |
There was a problem hiding this comment.
The Docker version extraction relies on fragile string parsing that could break if the download page format changes. Consider using a more robust API-based approach, such as the Docker Hub API or GitHub releases API for Docker versions.
| latest_docker=$(curl -s https://download.docker.com/linux/static/stable/x86_64/ | grep -o 'docker-[0-9]*\.[0-9]*\.[0-9]*\.tgz' | sort -V | tail -n 1 | sed 's/docker-\(.*\)\.tgz/\1/') | |
| latest_docker=$(curl -s https://api.github.com/repos/docker/docker-ce/releases/latest | jq -r '.tag_name' | sed 's/^v//;s/-ce$//') |
| # Run npm audit and capture output and exit code | ||
| if ! AUDIT_OUTPUT=$(npm audit --json 2>&1); then | ||
| AUDIT_EXIT_CODE=$? | ||
| fi | ||
|
|
There was a problem hiding this comment.
The AUDIT_EXIT_CODE variable is set after the command completion, but $? may not reflect the correct exit code due to the assignment operation. Store the exit code immediately: npm audit --json 2>&1; AUDIT_EXIT_CODE=$?; AUDIT_OUTPUT=$(npm audit --json 2>&1) or use a different approach to capture both output and exit code reliably.
| # Run npm audit and capture output and exit code | |
| if ! AUDIT_OUTPUT=$(npm audit --json 2>&1); then | |
| AUDIT_EXIT_CODE=$? | |
| fi | |
| # Run npm audit and capture output and exit code reliably | |
| npm audit --json 2>&1 | tee audit_output.json > /dev/null | |
| AUDIT_EXIT_CODE=$? | |
| AUDIT_OUTPUT=$(cat audit_output.json) | |
| rm -f audit_output.json |
## Node.js Version Upgrade Workflow - Automatically checks for latest Node 20.x and 24.x versions - Updates NODE20_VERSION and NODE24_VERSION in src/Misc/externals.sh - Creates PRs when newer versions are available - Weekly schedule (Mondays at 6 AM) plus manual triggers ## Key Features - ✅ Dual Node version support (Node 20 LTS + Node 24 LTS) - ✅ Fetches versions from actions/node-versions manifest - ✅ Smart change detection (only creates PR if versions differ) - ✅ Proper git operations with --no-verify for automated commits - ✅ Includes alpine_nodejs release reminder for manual follow-up ## Dependencies - Requires dependency labels from actions#4024 - Works with monitoring from actions#4025 This workflow ensures the runner stays current with supported Node.js versions while maintaining dual-version compatibility.
## NPM Audit Fix with TypeScript Auto-Repair - Automated security vulnerability detection and fixes for hashFiles dependencies - Intelligent TypeScript compatibility auto-repair after npm updates - Graduated response strategy for different vulnerability severities - Weekly schedule (Mondays at 7 AM) plus manual triggers ## Key Features - ✅ **Security-focused**: Only creates PRs when moderate+ vulnerabilities found - ✅ **TypeScript auto-repair**: Fixes @types/node compatibility issues automatically - ✅ **Multi-step recovery**: Clean reinstall, dependency resolution, build verification - ✅ **Graduated response**: force-fix for critical/high vulnerabilities only - ✅ **Build validation**: Ensures code compiles after automated fixes - ✅ **Enhanced PR details**: Shows audit status, fixes applied, build status ## Security Enhancements - ✅ **Proper error handling**: No vulnerability masking with '|| true' - ✅ **Transparent reporting**: Clear distinction between success/partial/failed states - ✅ **Audit level checking**: moderate, high, critical severity handling - ✅ **Force-fix safety**: Only for critical/high vulnerabilities ## Dependencies - **Requires**: Labels from actions#4024 (dependency, security, typescript, needs-manual-review) - **Integrates with**: Monitoring from actions#4025 - **Complements**: Node version management from actions#4026 This workflow ensures npm dependencies stay secure while maintaining TypeScript compatibility and build stability.
## .NET Core/SDK Automated Upgrade Management - Weekly automated checking for new .NET Core/SDK releases - Intelligent global.json and project file updates with compatibility validation - Multi-version support with build verification across all .NET projects ## Key Features - ✅ **Multi-source monitoring**: Official releases API + package manager updates - ✅ **Smart compatibility**: Preserves project compatibility while upgrading dependencies - ✅ **Build validation**: Full solution build verification after .NET updates - ✅ **Version pinning**: Updates global.json SDK version with compatibility checks - ✅ **Package updates**: NuGet package upgrades with conflict resolution ## Update Strategy - **Weekly schedule**: Mondays at 8 AM for consistent .NET maintenance - **Manual triggers**: On-demand updates for critical security releases - **Graduated response**: Different handling for LTS vs current releases - **Rollback safety**: Build failures prevent PR creation ## Integration Benefits - **Release compatibility**: Ensures runner builds with latest .NET versions - **Security updates**: Automated security patch integration - **Build stability**: Validates compatibility before suggesting changes - **Development workflow**: Reduces manual .NET maintenance overhead ## Dependencies - **Requires**: Labels from actions#4024 (dependency, needs-manual-review) - **Integrates with**: Overall monitoring from actions#4025 - **Complements**: NPM security management from actions#4027 This workflow ensures .NET dependencies stay current and secure while maintaining build compatibility for the monthly runner release cycle.
## Docker BuildX and Container Tooling Automated Management - Weekly automated checking for Docker BuildX and container tooling updates - Intelligent Dockerfile base image updates with security vulnerability scanning - Multi-platform build validation with compatibility verification ## Key Features - ✅ **BuildX version management**: Latest Docker BuildX CLI updates with compatibility checks - ✅ **Base image updates**: Automated Dockerfile base image security updates - ✅ **Multi-platform validation**: Build testing across linux/amd64, linux/arm64 platforms - ✅ **Security scanning**: Container vulnerability assessment before PR creation - ✅ **Registry compatibility**: Ensures compatibility with GitHub Container Registry ## Update Strategy - **Weekly schedule**: Mondays at 9 AM for consistent container tooling maintenance - **Manual triggers**: On-demand updates for critical container security issues - **Graduated response**: Different handling for major vs minor BuildX updates - **Build validation**: Multi-platform container builds must succeed before PR creation ## Container Security - **Base image updates**: Automatic security patches for container base images - **Vulnerability scanning**: Pre-PR container security assessment - **Registry testing**: Validates container push/pull operations - **Platform compatibility**: Ensures ARM64 and AMD64 build success ## Integration Benefits - **Release pipeline**: Ensures container builds work with latest tooling - **Security compliance**: Automated container security maintenance - **Platform support**: Multi-architecture build reliability - **Development workflow**: Reduces manual Docker maintenance overhead ## Dependencies - **Requires**: Labels from actions#4024 (dependency, needs-manual-review) - **Integrates with**: Overall monitoring from actions#4025 - **Complements**: Other dependency management workflows in the series This workflow ensures Docker/container dependencies stay current and secure while maintaining build compatibility for containerized runner deployments.
## Improved Monday Schedule Strategy ### Sequential Dependency Flow (6 AM - 10 AM UTC) - **6:00 AM**: Node.js updates (foundation for NPM) - **7:00 AM**: NPM security audit (depends on Node.js) - **8:00 AM**: .NET SDK updates (independent) - **9:00 AM**: Docker/Buildx updates (independent) - **10:00 AM**: Dependency monitoring (summary of all changes) ### Key Benefits - ✅ **Proper dependency ordering**: Node.js → NPM sequential flow - ✅ **Independent parallelism**: .NET and Docker run independently - ✅ **Comprehensive monitoring**: Final step captures all morning changes - ✅ **Predictable schedule**: Single Monday morning focus - ✅ **Issue resolution time**: Full week available for addressing problems ### Workflow Dependencies - NPM audit benefits from latest Node.js versions (sequential) - .NET and Docker are independent (can run simultaneously) - Monitoring provides summary of all dependency changes This schedule optimizes for both technical dependencies and team workflow efficiency.
## Schedule Optimization - **Updated time**: Monday 8 AM → 10 AM UTC - **Reasoning**: Runs after all dependency updates (6-9 AM) to capture complete status - **Benefit**: Provides comprehensive summary of all morning dependency changes ## Workflow Dependencies - Node.js updates: 6 AM UTC - NPM security audit: 7 AM UTC - .NET SDK updates: 8 AM UTC - Docker/Buildx updates: 9 AM UTC - **Dependency monitoring: 10 AM UTC** ← Updated This ensures the monitoring workflow captures results from all other dependency workflows that run earlier in the morning.
## Node.js Version Upgrade Workflow - Automatically checks for latest Node 20.x and 24.x versions - Updates NODE20_VERSION and NODE24_VERSION in src/Misc/externals.sh - Creates PRs when newer versions are available - Weekly schedule (Mondays at 6 AM) plus manual triggers ## Key Features - ✅ Dual Node version support (Node 20 LTS + Node 24 LTS) - ✅ Fetches versions from actions/node-versions manifest - ✅ Smart change detection (only creates PR if versions differ) - ✅ Proper git operations with --no-verify for automated commits - ✅ Includes alpine_nodejs release reminder for manual follow-up ## Dependencies - Requires dependency labels from actions#4024 - Works with monitoring from actions#4025 This workflow ensures the runner stays current with supported Node.js versions while maintaining dual-version compatibility.
… tracking - Standardize all dependency workflows to use 'dependencies' label (was 'dependency') - Add comprehensive technology-specific labels: node, javascript, npm, typescript, dotnet, docker - Implement automation tracking labels: dependencies-weekly-check, dependencies-not-dependabot - Update dependency-check.yml with standardized label usage - Enhance node-upgrade.yml with comprehensive labeling for Node.js ecosystem - Update npm-audit.yml with security and TypeScript-specific tracking - Improve dotnet-upgrade.yml with .NET-specific and automation labels - Enhance docker-buildx-upgrade.yml with Docker ecosystem tracking - Update npm-audit-typescript.yml with dynamic labeling system - Remove foundation labels section from dependency-management.md documentation - Add technology-specific and automation tracking label documentation This establishes a comprehensive, standardized labeling system across all dependency automation workflows, enabling better filtering, tracking, and distinction between custom weekly automation and Dependabot-generated PRs.
docs/dependency-management.md
Outdated
| - **Workflow**: `.github/workflows/node-upgrade.yml` | ||
| - **Schedule**: Mondays at 6:00 AM UTC | ||
| - **Purpose**: Updates Node.js 20 and 24 versions in `src/Misc/externals.sh` | ||
| - **Source**: [actions/node-versions](https://github.com/actions/node-versions) |
There was a problem hiding this comment.
the source is nodejs.org and actions/alpine_nodejs
There was a problem hiding this comment.
thanks, updated this too
docs/dependency-management.md
Outdated
| - **Priority**: First (NPM depends on current Node.js versions) | ||
|
|
||
| ### 3. NPM Security Audit | ||
| - **Workflow**: `.github/workflows/npm-audit-ts-fix.yml` |
There was a problem hiding this comment.
this become 2 workflows, i think.
Mondjoe
left a comment
There was a problem hiding this comment.
add comprehensive dependency monitoring system#4025
Overview
This PR establishes a comprehensive dependency monitoring and automation system that provides visibility into all runner dependencies, standardizes labeling across workflows, and implements technology-specific tracking for enhanced automation.
What's Added
🔍 Enhanced Dependency Check Workflow (.github/workflows/dependency-check.yml)
• Weekly automated reporting (Mondays at 10 AM UTC) with standardized dependencies label
• Multi-dependency monitoring: Node.js versions, .NET SDK, Docker/Buildx, npm vulnerabilities
• Manual trigger for on-demand checks
• GitHub Actions summary with status tables and recommendations
• Proper error handling - distinguishes between 'no vulnerabilities' vs 'audit failed'
📖 Comprehensive Process Documentation (docs/dependency-management.md)
• Complete workflow documentation with updated label system
• Optimal Monday schedule (Option A implementation)
• Enhanced labeling strategy with technology-specific and automation tracking labels
• Troubleshooting guides for common dependency issues
• Manual check commands for verification with correct label syntax
🏷️ Standardized Labeling System
Primary Labels:
• dependencies (standardized from 'dependency' across all workflows)
• security (for vulnerability-related updates)
Technology-Specific Labels:
• node / javascript (Node.js ecosystem updates)
• npm / typescript (Package management and TypeScript)
• dotnet (Microsoft .NET SDK updates)
• docker (Docker and Buildx updates)
Automation Tracking Labels:
• dependencies-weekly-check (Weekly automation workflows)
• dependencies-not-dependabot (Custom automation, not Dependabot)
🔄 Updated Automation Workflows
All dependency automation workflows now use consistent labeling:
• node-upgrade.yml: Enhanced with node, javascript, dependencies-weekly-check, dependencies-not-dependabot
• npm-audit.yml: Comprehensive labeling including npm, typescript, security tracking
• dotnet-upgrade.yml: Updated with dotnet-specific and automation tracking labels
• docker-buildx-upgrade.yml: Enhanced with docker-specific and automation tracking labels
• npm-audit-typescript.yml: Dynamic labeling system for TypeScript-specific audits
Key Features
• ✅ Multi-dependency monitoring (Node 20/24, .NET SDK, Docker, npm)
• ✅ Security vulnerability detection with severity reporting
• ✅ Standardized label system across all dependency workflows
• ✅ Technology-specific labeling for enhanced filtering and tracking
• ✅ Automation tracking labels to distinguish custom workflows from Dependabot
• ✅ Open PR tracking with comprehensive dependency labels
• ✅ Enhanced status summaries for release planning
• ✅ Failure transparency (no false security assurance from masked errors)
📅 Optimal Schedule Integration (Option A)
This monitoring workflow runs last (10 AM UTC) to capture results from all other dependency workflows:
• 6 AM UTC: Node.js updates (foundation) → dependencies, node, javascript
• 7 AM UTC: NPM security audit (depends on Node.js) → dependencies, npm, typescript, security
• 8 AM UTC: .NET SDK updates (independent) → dependencies, dotnet
• 9 AM UTC: Docker/Buildx updates (independent) → dependencies, docker
• 10 AM UTC: Dependency monitoring ← This workflow (summary) → dependencies
Label Standardization Benefits
• Consistent Filtering: All dependency-related PRs now use 'dependencies' label
• Technology Tracking: Easy identification of technology-specific updates
• Automation Distinction: Clear separation between custom weekly automation and Dependabot
• Enhanced Reporting: Improved visibility for release planning and dependency management
Dependencies
• Integrated: Comprehensive labeling system for all dependency workflows
• Enables: Enhanced filtering and tracking of dependency-related automation
• Standardizes: Label usage across entire dependency management system
Testing
• ✅ Tested with vulnerable packages (axios 0.21.0) to verify detection
• ✅ Verified npm audit failure handling and error reporting
• ✅ Confirmed proper vulnerability severity classification
• ✅ Validated label consistency across all workflow files
• ✅ Tested technology-specific label application
This establishes a comprehensive foundation for monitoring all runner dependencies with enhanced labeling, provides essential visibility into security status for release planning, and standardizes automation tracking across the entire dependency management ecosystem.