Add a skill for LLMs on how to get the azp logs for analysis (#86765)

2026-06-19 07:35:52 +00:00 · 2026-04-09 11:45:43 -05:00
parent 54cdaedfbc
commit fb8c4d3177
4 changed files with 158 additions and 24 deletions
@@ -0,0 +1,99 @@
+---
+description: Download Azure Pipelines CI logs for analysis
+argument-hint: <pr_number|build_id|build_url>
+allowed-tools: [Bash(gh pr view:*), Bash(gh pr checks:*), Bash(ls:*), Read, Grep]
+---
+
+Azure Pipelines Logs Downloader
+================================
+
+Download Azure Pipelines CI logs for analyzing test failures and CI issues.
+
+**IMPORTANT**: Always ask the user before downloading logs. The download may take 5-10 minutes (or longer for large CI runs)
+depending on the number of jobs and log size.
+
+Usage
+-----
+
+```bash
+/azp-logs <pr_number|build_id|build_url>
+```
+
+Arguments
+---------
+
+- `pr_number`: GitHub PR number (will extract build ID from latest CI run)
+- `build_id`: Azure Pipelines build ID (numeric)
+- `build_url`: Full Azure Pipelines URL (e.g., <https://dev.azure.com/ansible/ansible/_build/results?buildId=12345>)
+
+Implementation
+--------------
+
+This command uses the existing `hacking/azp/download.py` script to download CI logs.
+
+**Before running**: Always confirm with the user before downloading logs. Inform them that:
+- The download may take 5-10 minutes for a full CI run (potentially longer for very large runs)
+- Logs will be saved to a directory named after the build ID
+- The download size can be 10-50MB depending on the number of jobs
+
+Process Steps
+-------------
+
+1. **Ask user for confirmation**: Explain what will be downloaded and estimated time
+
+2. **Determine build ID**:
+   - If given a PR number: Use `gh pr checks <number>` to get the Azure Pipelines URL
+   - If given a URL: Pass it directly to download.py (it extracts buildId automatically)
+   - If given a build ID: Use directly
+
+3. **Download logs**: Run `hacking/azp/download.py` with appropriate flags:
+
+   ```bash
+   ./hacking/azp/download.py <build_id_or_url> --console-logs -v
+   ```
+
+4. **Analyze logs**: After download completes, examine logs in `<build_id>/` directory:
+   - Grep for common failure patterns: `FAILED`, `ERROR`, `Traceback`
+   - Focus on logs from failed jobs (check job names)
+   - Compare with ansibot comments for context
+
+Download Script Options
+-----------------------
+
+The `hacking/azp/download.py` script supports:
+- `--console-logs`: Download console logs (recommended for CI failure analysis)
+- `--artifacts`: Download test artifacts
+- `--run-metadata`: Download run metadata JSON
+- `--all`: Download everything
+- `--match-job-name <regex>`: Filter to specific jobs
+- `--match-artifact-name <regex>`: Filter to specific artifacts
+- `-v, --verbose`: Show what is being downloaded
+- `-t, --test`: Dry run (show what would be downloaded)
+
+For most CI failure analysis, use `--console-logs` to get the log files.
+
+Common Analysis Patterns
+------------------------
+
+After downloading logs to `<build_id>/` directory:
+
+```bash
+# Find all errors and failures
+grep -r "FAILED\|ERROR\|Traceback" <build_id>/
+
+# Find specific test failures
+grep -r "FAILED test" <build_id>/
+
+# Find sanity test failures
+grep -r "The test" <build_id>/ | grep -i "failed"
+
+# List all downloaded log files
+ls -lh <build_id>/
+```
+
+Notes
+-----
+
+- Logs are downloaded to a directory named after the build ID
+- Console logs are named after the job hierarchy (e.g., "Job Name Stage Name.log")
+- The Ansible project is public, so no authentication is required
@@ -42,6 +42,7 @@ gh pr view <number> --comments                            # Check for ansibot CI
 gh pr checks <number>                                     # Get Azure Pipelines URLs
 gh pr checkout <number>                                   # Switch to PR branch
 gh pr diff <number>                                       # See all changes
+/azp-logs <number>                                        # Download CI logs for PR
 ```

 **Container Selection:**
@@ -151,6 +152,43 @@ This shows:
 4. For sanity test failures, the error messages usually indicate exactly what needs to be fixed
 5. For test failures, run the same tests locally using `ansible-test` to reproduce and debug

+**5. Downloading Azure Pipelines logs for analysis:**
+
+When CI failures need deeper investigation beyond what's visible in ansibot comments or the web UI, use the `/azp-logs` skill:
+
+```bash
+# Download logs using PR number (automatically finds latest build)
+/azp-logs <pr_number>
+
+# Or use build ID directly from gh pr checks output
+/azp-logs <build_id>
+
+# Or use the full Azure Pipelines URL
+/azp-logs https://dev.azure.com/ansible/ansible/_build/results?buildId=12345
+```
+
+The skill uses `hacking/azp/download.py` to download console logs into a directory named after the build ID.
+
+**After downloading, analyze the logs:**
+- Grep for common failure patterns: `grep -r "FAILED\|ERROR\|Traceback" <build_id>/`
+- Focus on logs from failed jobs identified in `gh pr checks` output
+- Compare error messages with ansibot comments to get full context
+- Sanity test failures usually have clear error messages with file:line references
+- Integration/unit test failures may require examining full test output and tracebacks
+
+**Advanced usage:**
+The download script supports filtering and customization:
+
+```bash
+# Download only logs matching specific job names
+./hacking/azp/download.py <build_id> --console-logs --match-job-name "Sanity.*"
+
+# Download artifacts and metadata too
+./hacking/azp/download.py <build_id> --all
+```
+
+See `.claude/skills/azp-logs/SKILL.md` for complete documentation.
+
 ## PR Review Guidelines

 ### PR Review Checklist
@@ -27,21 +27,16 @@ import json
 import os
 import re
 import io
+import shutil
 import zipfile
-
-import requests
+import urllib.request
+import urllib.error

 try:
    import argcomplete
 except ImportError:
    argcomplete = None

-# Following changes should be made to improve the overall style:
-# TODO use new style formatting method.
-# TODO use requests session.
-# TODO type hints.
-# TODO pathlib.
-

 def main():
    """Main program body."""
@@ -134,9 +129,8 @@ def download_run(args):

    if args.run_metadata:
        run_url = 'https://dev.azure.com/ansible/ansible/_apis/pipelines/%s/runs/%s?api-version=6.0-preview.1' % (args.pipeline_id, args.run)
-        run_info_response = requests.get(run_url)
-        run_info_response.raise_for_status()
-        run = run_info_response.json()
+        with urllib.request.urlopen(run_url) as run_info_response:
+            run = json.load(run_info_response)

        path = os.path.join(output_dir, 'run.json')
        contents = json.dumps(run, sort_keys=True, indent=4)
@@ -148,9 +142,8 @@ def download_run(args):
            with open(path, 'w') as metadata_fd:
                metadata_fd.write(contents)

-    timeline_response = requests.get('https://dev.azure.com/ansible/ansible/_apis/build/builds/%s/timeline?api-version=6.0' % args.run)
-    timeline_response.raise_for_status()
-    timeline = timeline_response.json()
+    with urllib.request.urlopen('https://dev.azure.com/ansible/ansible/_apis/build/builds/%s/timeline?api-version=6.0' % args.run) as timeline_response:
+        timeline = json.load(timeline_response)
    roots = set()
    by_id = {}
    children_of = {}
@@ -185,17 +178,20 @@ def download_run(args):

    if args.artifacts:
        artifact_list_url = 'https://dev.azure.com/ansible/ansible/_apis/build/builds/%s/artifacts?api-version=6.0' % args.run
-        artifact_list_response = requests.get(artifact_list_url)
-        artifact_list_response.raise_for_status()
-        for artifact in artifact_list_response.json()['value']:
+        with urllib.request.urlopen(artifact_list_url) as artifact_list_response:
+            artifact_list = json.load(artifact_list_response)
+
+        for artifact in artifact_list['value']:
            if artifact['source'] not in allowed or not args.match_artifact_name.match(artifact['name']):
                continue
            if args.verbose:
                print('%s/%s' % (output_dir, artifact['name']))
            if not args.test:
-                response = requests.get(artifact['resource']['downloadUrl'])
-                response.raise_for_status()
-                archive = zipfile.ZipFile(io.BytesIO(response.content))
+                with urllib.request.urlopen(artifact['resource']['downloadUrl']) as response:
+                    with io.BytesIO() as buffer:
+                        shutil.copyfileobj(response, buffer)
+                        buffer.seek(0)
+                        with zipfile.ZipFile(buffer) as archive:
                            archive.extractall(path=output_dir)

    if args.console_logs:
@@ -220,9 +216,9 @@ def download_run(args):
            if args.verbose:
                print(log_path)
            if not args.test:
-                log = requests.get(r['log']['url'])
-                log.raise_for_status()
-                open(log_path, 'wb').write(log.content)
+                with urllib.request.urlopen(r['log']['url']) as log:
+                    with open(log_path, 'wb') as log_file:
+                        shutil.copyfileobj(log, log_file)


 if __name__ == '__main__':
@@ -1,4 +1,5 @@
 .claude/commands/review.md pymarkdown!skip  # Claude Code command with YAML frontmatter
+.claude/skills/azp-logs/SKILL.md pymarkdown!skip  # Claude Code skill with YAML frontmatter
 .github/ISSUE_TEMPLATE/internal_issue.md pymarkdown!skip
 lib/ansible/_internal/_wrapt.py black!skip  # vendored code
 lib/ansible/config/base.yml no-unwanted-files