aboutsummaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)Author
2018-07-09update.py: check whether branch existed when nocheckoutrbt/li_recsRobert Yang
Fixed: Assume there is no master branch in hello layer: $ update.py -l hello -b master INFO: Skipping update of layer hello - branch master doesn't exist This is correct since hello layer doesn't have master branch, but when --nocheckout: $ update.py -l hello -b master --nocheckout [snip] INFO: Sorting layers for branch mater: WARNING: Cannot find required collections on branch master: WARNING: hello: LAYERDEPENDS: <snip> This is incorrect, this patch fixed the problem, now it skips it since the branch doesn't exists when --nocheckout. Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
2018-07-09update.py: add layers when RECOMMENDS isn't satisfiedRobert Yang
When layer_a RECOMMENDS layer_b, try to add layer_b before layer_a, but if layer_b is not found, still add layer_a. And print summary error mesage: $ update.py -b master ERROR: Issues found on branch master: openembedded-core: Added without LAYERRECOMMENDS meta-secure-env: Failed to add since LAYERDEPENDS is not satisfied Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
2018-07-09utils.py: fix checkout_repo when no HEADRobert Yang
Fixed: $ git clone <url> warning: remote HEAD refers to nonexistent ref, unable to checkout. $ git rev-parse HEAD HEAD fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree. Use '--' to separate paths from revisions, like this: 'git <command> [<revision>...] -- [<file>...]' Catch the error and avoid that. And use "git reset --hard" to replace of "git reset --hard HEAD", HEAD is default for git reset, so they are the same, but the later one reports error when remote HEAD doesn't exist: $ git reset --hard HEAD fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree. [snip] $ git reset --hard No errors. Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
2018-07-09update_layer.py: avoid calling setup_core_layer_sys_path() when --initialRobert Yang
Fixed: $ update.py -b <new_branch> [snip] NOTE: Starting bitbake server... Traceback (most recent call last): File "update_layer.py", line 471, in main utils.setup_core_layer_sys_path(settings, branch.name) File "/buildarea1/lyang1/layerindex-web/layerindex/utils.py", line 376, in setup_core_layer_sys_path core_layerdir = os.path.join(core_repodir, core_layerbranch.vcs_subdir) AttributeError: 'NoneType' object has no attribute 'vcs_subdir' [snip] This is because core_layerbranch is not in database yet for completely new branch, so it is None and we will get the error. Avoid calling setup_core_layer_sys_path() when "update_layer.py --initial" will fix the problem. And also only add core layer's sys path when it is present, since core layer may not be added yet for completely new branch. Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
2018-06-06rrs_upstream_history.py: fix set_regexes functionYi Zhao
REGEX, REGEX_URI and GITTAGREGEX are replaced by UPSTREAM_CHECK_REGEX, UPSTREAM_CHECK_URI and UPSTREAM_CHECK_GITTAGREGEX Signed-off-by: Yi Zhao <yi.zhao@windriver.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-06-06rrs_upstream_email.py: enable set log levelYi Zhao
Add logger.setLevel to enable set log level. Signed-off-by: Yi Zhao <yi.zhao@windriver.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-07Add CSV export for layer recipesPaul Eggleton
Add the ability to export the recipe listing for a layer to a CSV file for importing into external tools. At the moment we include name, version and license, but there is a parameter that lets you specify the fields to include in the URL if desired. Implements [YOCTO #12722]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-07Drop old raw recipe export codePaul Eggleton
We're about to replace this with a proper CSV export function, so we don't need this dead code hanging around anymore. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04TODO: add RRS itemsPaul Eggleton
Add some stuff still left to be done after the merging. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs: fix unique constraint on RecipeMaintainerHistory sha1 fieldPaul Eggleton
Although it's unlikely to be an issue, technically we shouldn't be insisting the sha1 field be unique globally, just within each layerbranch, so adjust the constraints. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs: admin: validate that email address fields are setPaul Eggleton
If automated emails are enabled, we need to ensure that the other email fields are populated, so validate that. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs: add flag to MaintenancePlan to specify layer-wide maintainersPaul Eggleton
Most layers do not track maintenance on a per-recipe basis, and for those layers we will hide some of the per-recipe maintainer features and on the recipe detail show the layer maintainer(s) as the maintainer(s) of the recipe. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs_upstream_history: make more robustPaul Eggleton
Avoid exceptions / blank versions during recipe upstream crashing the entire script. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs_upgrade_history: skip commits that don't touch the layerPaul Eggleton
If we're in a repository containing multiple layers, we don't care about commits that don't affect the layer we are processing, so skip those commits rather than passing them to upgrade_history_internal.py which will ignore them (which is significantly slower). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs: validate that a layerbranch is only part of one planPaul Eggleton
The processing code can't currently handle if a layerbranch is part of more than one plan, so disallow that. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04templates/rrs: replace use of = with ==Paul Eggleton
I can't quite tell which Django version broke this, but in any case based on what's in pagination.html it seems we ought to have been using == already. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs/urls: Use new urlpatterns list syntaxPaul Eggleton
The patterns() function is deprecated in Django 1.8 and gone in 1.10, so we should switch over to the new list format. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs: link maintenance/upstream history to layerbranchPaul Eggleton
RecipeUpstreamHistory was not linked to the layer it was produced from, which meant that it wasn't easy to query for a different maintenance plan (i.e. a different layer) and thus the maintenance plan selection on the recipe list didn't really work. Add a link field, populate it in a migration and then make it required. We had added a link earlier from RecipeMaintainerHistory to LayerBranch but it was optional; for the same reasons we now populate it and make it required. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs: drop a couple of unused functions from Raw classPaul Eggleton
These two functions aren't being used anywhere. In the interest of having as little code directly reading the database using SQL as possible, drop them. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs/tools: add -p/--plan optionPaul Eggleton
Add an option to specify which maintenance plan to operate on (largely for debugging purposes). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs: improve admin for Release/Milestone objectsPaul Eggleton
* Ensure the Release and Milestone names are separated by a space when listing Milesones * Include the maintenance plan name in the name shown for each Release/Milestone * Allow filtering Releases/Milestones by maintenance plan Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs: duplicate releases from first plan when adding a new planPaul Eggleton
It's a pain to have to add all the releases when adding a new maintenance plan. Since these are likely to be the same (or similar) for every plan, then duplicate them across from the first plan when you save a new one. Also add "default" milestones on the assumption that other layers probably won't want to use the 4-milestone split per release, but there do have to be some milestone records, so just create one milestone for each release. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs: default python2/3 environments for new maintenance plan layer branchesPaul Eggleton
It's a bit of a pain to have to set the two python environment fields on every record in order to have things set correctly, and it can easily get forgotten, so try to set them automatically by default (assuming reasonable naming). Note that this does introduce an annoying behaviour whereby if you click "Add another Maintenance plan layer branch" and then decide you don't want it, the admin form will insist you fill in the fields unless you clear out the python2/3 environment fields. I'm not sure how to fix that, so I'm leaving it as-is for now. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs: releases should be unique by plan and name, not just namePaul Eggleton
I missed changing this constraint when adding the plan field. We want to be able to have the same named release on another plan. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs: add charts pagePaul Eggleton
Add some basic charts to show recipe upstream / patch status. Implements [YOCTO #7909]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs: add patch listing to recipe list and detailPaul Eggleton
Expose the newly added patch information in the RRS: * Add a table to the recipe detail listing the patches for the recipe * Add pending / total counts to the recipe list page Implements [YOCTO #7909]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04Implement patch trackingPaul Eggleton
Collect information about patches applied by a recipe, and record each patch along with the upstream status, presenting them in the recipe detail. Implements [YOCTO #7909]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs: restore full-width pagesPaul Eggleton
Upon consideration, for the width of the information we want to present we do actually want full-width pages for the RRS. When this was changed earlier in the rrs branch it was changed in the base template, but we want to keep the same style elsewhere, so put a block in that will let use the "container-fluid" style (full width) in the RRS pages and "container" by default everywhere else. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04Turn URLs into links in layer notesPaul Eggleton
It's often useful to add a link to another page for more information when adding a layer note, so turn any included URLs into actual links. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs_upgrade_history: improve checkout logicPaul Eggleton
* Consolidate the code for checking out a repository, using the newly added utils.checkout_repo() function * Check out a layer's dependencies, not just the layer itself * Only check out if the desired revision isn't already checked out (mostly useful for bitbake which we would otherwise be checking out much more frequently than necessary since it may not have changed even if we've moved to a new commit in the layer). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04utils: add common function to check out a specific git revisionPaul Eggleton
Checking out a revision in the bitbake/layer repositories is something we are doing in a few places, so add a checkout_repo() function that does this, ensuring that we don't get tripped up by any junk files, and avoids checking out if the repository is already at the desired revision (thus avoiding the clean operation and e.g. preserving any .pyc files that aren't stale and would speed things up a little). Note that we do the clean before checking out in case there are untracked files that are tracked in the commit we are checking out. In addition to adding this function, change the existing code that we use in the update script to check out a layer use the new function. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04models: implement layerindex method to get recursive dependenciesPaul Eggleton
It would be useful in some scenarios to get the complete list of recursive dependencies for a layer, so add a function to do that. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04Implement layer web repo commit URLPaul Eggleton
The Recipe Reporting System needs to be able to provide links to commits in the web interface for the repository, but we can only do this if we have a custom template URL just like we do for file/tree links, since it's different for different git web interfaces. Add support in all the various places for such a URL and make use of it in the RRS. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs: support new source structure in recipe detailPaul Eggleton
In the rrs branch we used to just store SRC_URI in a field, however we now have a proper model to store sources, so use that in the RRS recipe detail page. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04Save recipe source URLsPaul Eggleton
Save each remote SRC_URI so we can use these for the recipe reporting system. This replaces an earlier implementation in the rrs branch where we simply stored SRC_URI verbatim. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04admin: use more dynamic method of setting recipe read-only fieldsPaul Eggleton
Every time we add something that links to Recipe we had to add it to the exclusions list in the readonly_fields line for RecipeAdmin (and ClassicRecipeAdmin), which is tedious and easily forgotten. We can avoid this by looking at each field and excluding it by its attributes rather than having a hardcoded list of names. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04Add a link from the Tools drop-down to the RRS if enabledPaul Eggleton
If the RRS is enabled, then add a link to it in the tools menu. I don't expect this to be used a lot, but otherwise the only way you'd get there would be either externally or via the link from the layer detail. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs: Add link to layer detail to breadcrumb on recipe detailPaul Eggleton
Add a convenience link to the layer detail to the breadcrumb (also as an indicator, since it's possible to have more than one layer in the maintenance plan). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs: handle linking maintainershipPaul Eggleton
Provide a mechanism set the maintainer for things like gcc-cross-<arch> to the same as gcc. (We do have entries in the .inc file for these, however they aren't useful as they don't match the recipe name when we parse it, and due to the fact that RecipeMaintainer objects link directly to Recipe objects, we can't handle entries that don't map to a real recipe). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs/views: replace mimetype with content_typePaul Eggleton
Adapt to this change that happened sometime in Django 1.7. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs_maintainer_history: avoid unnecessary checkoutsPaul Eggleton
We don't actually need to check out the repository until we actually analyse a commit, so avoid doing so. Additionally, there's not much point in checking out master at the end, let the next script invocation do that if needed (if it needs to, it should since otherwise there's no guarantee what state the repository is in). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs_maintainer_history: add --fullreload optionPaul Eggleton
Add an option that deletes all maintainer history records for the current layer branch so that they can then be reloaded from scratch. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs_upgrade_history: drop final cleanPaul Eggleton
We don't really need to clean up the repository, we'll take care of it on subsequent runs. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs_upgrade_history: handle only .inc file changingPaul Eggleton
If a recipe upgrade only consists of a .inc file changing, we were not picking it up, since we were only looking specifically for recipes (.bb). If a .inc file changes, assume all .bb files in the same directory (if any) should be parsed to look for an upgrade. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs/tools: ensure recipe parsing code deletes temporary dirPaul Eggleton
load_recipes() was leaving files around in /tmp; on my Fedora system this eventually resulted in /tmp running out of space which we do not want. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04README.rrs: updatePaul Eggleton
Drop outdated information, fix typos/spelling/grammar, improve formatting and add some new steps. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs_upgrade_history: Add debug option to process specific commitPaul Eggleton
Sometimes it's useful to be able to re-try processing a particular commit (generally in conjunction with --dry-run), so add an option to enable that. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs_upgrade_history: ignore files outside of the layerPaul Eggleton
We were parsing recipes that were in the repository but not inside the actual layer we're dealing with (e.g. we have meta-selftest within the OE-Core repository, containing a number of recipes that are only intended for testing purposes and should not be looked at by this script). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs_upgrade_history: show text of any exception that blocks parsingPaul Eggleton
If an exception occurs during parsing, let's actually see what kind of exception it was in the output. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04rrs_upgrade_history: handle broken version numbersPaul Eggleton
In OE-Core revision e0531174119bff21e9014b95ed1bbd0e1c01af26 we accidentally committed a new e2fsprogs recipe with ..bb at the end of its name instead of .bb. This was fixed immediately afterwards, but when the RRS hits this commit, it doesn't fail immediately, but the bogus version "1.43." gets into the database and all subsequent commits touching the e2fsprogs recipe cause bb.utils.vercmp_part() to blow up because one of the version parts in the "previous" version in the database is apparently empty. To work around this and any similar issues, just reject any change that results in such a broken version string (on the assumption that it'll be corrected in a subsequent commit and thus we will get to re-parse the recipe then and therefore not miss the upgrade.) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>