10000 Create a notion of "Projects" · Issue #1738 · fossology/fossology · GitHub
[go: up one dir, main page]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a notion of "Projects" #1738

Open
NicolasToussaint opened this issue Jun 9, 2020 · 7 comments
Open

Create a notion of "Projects" #1738

NicolasToussaint opened this issue Jun 9, 2020 · 7 comments
Labels
enhancement
8000

Comments

@NicolasToussaint
Copy link
Member

Description

For multiple reasons, I would like to be able to identify uploads to projects.

A project would be defined by:

  • a project ID (as primary key)
  • a project name
  • a root folder in Fossology
  • one or more Group IDs and associated access.
  • a URL / ID / Free field to link the project to an external tool

As a suggestion, a new table entry could be created containing all info mentioned above.

This could be used, for example:

  • to gather multiple uploads under the same umbrella, and make bulk changes changes to all uploads / sub-directories that belong to a given project
  • to associate a link to a ticketing system or project management solution
  • to produce metrics with Work-In-Progress dashboard

In terms of UI integration:

  • Add an optional selection of project for new uploads
  • Add pages to list, edit, delete projects (in Organize or Admin menu)
  • Display the project name in the "Folder Navigation" window, and in the yellow band

Anyone has an opinion on such a feature ?
@deveaud-m
Copy link
Contributor

@NicolasToussaint I like the idea! In my team we use the concept of a "clearing request", this is simply a collection of uploads that belong to a specific product/project and needs to be scanned together, very similar to the concept you describe here.

We tried using the tag feature of Fossology to label uploads belonging together:

  1. set the tag using the file info tab
  2. use the search UI to collect all uploads using a specific tag

While both steps work well, we are missing following features:

  • the ability to filter uploads after a given tag on the main browser
  • the ability to manage tags via REST API (create/delete)

Extending the tag feature could be another option to implement "projects".

@jeremiah
Copy link

I wonder if there is a need to consider how the various licenses work together in a project? While I think a lot will depend on how the project is built with regard to dependencies nonetheless the "project" grouping might be able to provide an overall scenario for how a given project with multiple licenses is licensed overall and which license applies to which software component.

@swinslow
Copy link
Contributor

Hi all, I have a wrapper framework around existing Fossology (3.6.0) that I use for project scanning, to try to accomplish something along these lines. The code is at https://github.com/swinslow/scaffold -- it is unfortunately lacking in documentation at the moment, mostly because I've just been using it myself and haven't written it up. This is probably overkill for many folks' use cases, but sharing here in case it's at all helpful.

To use it, you set up a config.json file to define project configuration:

  • where the code is (currently can pull from GitHub or from Gerrit repos)
  • which repos to include or exclude
  • project license policies -- categories of licenses, different aliases for the licenses (based on how Fossology encodes them in SPDX export), and which ones to flag for review

You also set up a corresponding file listing out different texts which corresponds to particular licenses (this is for use in automated monkbulk scanning runs).

Then, the framework automatically runs through the following steps:

  • check a listing of the repos currently in the GitHub org / Gerrit set of repos (and pause / warn if there are new ones not currently tracked)
  • pull the code from the repo(s), clear out .git/ directories and optionally others, and zip it together so that there is one zip file per project
    • this way, when I'm doing the clearing in Fossology, I can do it with several repos bundled together, rather than separately for each repo
  • upload the code to a Fossology server and run a bunch of agents, including the configured monkbulk matches (all using fossdriver
    • this includes automatically running the reuser agent on the results, based on the previously cleared scan for this project
  • I then go and manually clear the scan results directly in Fossology, and then mark it as "cleared"
  • the framework then exports the results as an SPDX tag-value document (also using fossdriver) and then parses it, applying the categorized licenses
  • there is also an ongoing file where I have writeups of what action items the developers should take to address particular findings; this file is applied to the categorized license findings
  • when everything is done, it creates and uploads:
    • HTML reports of the findings and overall license summary
    • XLSX detailed report of categorized licenses for each file
    • SPDX file
  • optionally, it can also file and update Jira tickets if the project is using Jira to track issues like this

I have no idea if any of this is useful or relevant to what you're looking at :) I'm sure several parts are specific to the particular workflow I've got. But I'm happy to share more details and/or actually write up some documentation on it, if that would be helpful.

@NicolasToussaint
Copy link
Member Author

Many thanks for your detailed thoughts on this !

@deveaud-m Tags look good, I had never used them before, but it looks interesting to build on top of that.
I will need to investigate, as it requires a few steps to actually be able to use them... more info here for the curious.

@jeremiah In my case, a project will often be composed of various components that with various "use cases" (front-end, back-end, mobile, embedded, standalone software). So we will still want to analyse the findings separately.
Which does not mean that the possibility to aggregate the findings does not make sense :)

@swinslow Many thanks for sharing and describing your work!
I think that injecting this notion of project inside Fossology may in fact simplify your framework.
We do intent to glue services together too (like ticketing, list the actions to be communicated to the projects, ...), your framework will certainly be an inspiration.

@swinslow
Copy link
Contributor

@NicolasToussaint agreed -- if Fossology had a concept of "projects", I expect it would be very helpful.

I've found it particularly useful to be able to combine pulls from multiple repos into a single Fossology upload, so that I can do things like run bulk text matches, reuse prior results, export combined SPDX files, etc., at the "project" level. That has made the process faster for me when I'm dealing with a larger number of repos, rather than clearing them one-by-one.

So I'd suggest that if there's a way to run agents / actions across multiple uploads in a Fossology "project," that might be useful for at least some use cases.

@jeremiah
Copy link

So I'd suggest that if there's a way to run agents / actions across multiple uploads in a Fossology "project," that might be useful for at least some use cases.

+1

I like the idea of a sort of "policy engine" across a diverse source code that may or may not become a binary but rather has another, maybe higher level, cohesion.

@anantnipunge
Copy link

@NicolasToussaint
+1
@jeremiah
I've found it particularly useful to be able to combine pulls from multiple repos into a single Fossology upload, so that I can do things like run bulk text matches, reuse prior results, export combined SPDX files, etc., at the "project" level. That has made the process faster for me when I'm dealing with a larger number of repos, rather than clearing them one-by-one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jeremiah @swinslow @deveaud-m @NicolasToussaint @anantnipunge
0