Support for bundle inline recipe configuration #206
Comments
|
I think this is a must have. A simple solution would be to add a It's more convenient for a developer to manage the recipes from his repository. And it will allow private recipes easily. I'll have a look at the source code this week to see if it's something that could be implemented easily or not. |
|
To me, inline recipes are like DI by annotations: something we don't want to support because it's putting the responsibility in the wrong hands. |
I disagree on the DI through annotations part, it feels more like configuring routes to me, it provides the link between the bundle (a route) and the framework (router). The responsibility is not in the wrong hands, it's still the author's responsibility to make it work, except that inlining it, will make the author's life a lot easier.
The consumer won't even notice this unless the consumer opens the directory containing this config.
Having them in different git repositories is quite cumbersome, a lot of extra work and a lot of problems keeping things in sync and test them properly. With a dedicated repository, there's need for a staging environment to test if your recipe works. With inline repositories, you merely have to install the fork or branch.
Yes, this is indeed a benefit of the collective repository. Long story short: I don't want the repository to be removed, I think it should still be leading. However, Symfony is recommending us to use flex, supply all kind of fancy tools, and we still have to do it the old fashioned way, because we can't open-source the majority of our bundles. We can't use a private github repository either, because then we still have to place our code external, rather than in-house. On top of that, maintaining an in-house repository, still has all the overhead I've mentioned, which we are not looking for. When we have version 10.0.0 of a bundle, we have no need to keep the config of 3.0.0. Yes, we can delete those directories, but if we have 2 majors running side-by-side (for a few days max), there's so much extra work and overhead, that maintaining it, is not worth it. There's just too much that can (and thus will) go wrong when you keep it in an external repository. |
|
"We can't use a private github repository either, because then we still have to place our code external": only the recipes repository must be on Github. |
|
@fabpot You're right, but it would still "leak" outside, whi 8000 ch is the problem |
|
If you think it's a security problem to let developer manage their configuration (without being validated) we could make this option optin or ask the end user if he really wants to use the recipe (like you did for contrib https://github.com/symfony/flex/blob/master/src/Flex.php#L158) |
|
There are 3 types of packages:
I think, that
But, I do not like any: SE contains a lot of extra files, and plugin is a crutch =( |
|
@Pierstoval Anyway, i've created a composer command, that allows to apply recipe manually right from the package. That command does not use |
|
@covex-nn There is now a solution for private repositories, but read my entire comment to know the limitations. @iltar, @Grafikart I had already explained the problem of private repositories and presented solutions, but the Symfony team explained very well their motivations and their reasons for refusing to use any other means than the official and contrib recipes on Symfony.sh. Of course, like you, I proposed the addition of the manifest in the component, what you call an 'inline configuration @fabpot I understand that the Symfony team wants to talk with the developers of third-party components to make the Symfony ecosystem better, and therefore to lock it as it wishes, it is legitimate, but debatable. Given that the Symfony team does not want to take responsibility for a likely bad quality on the configuration of the third-party component (because not verified by them or the community), I tried to express my vision of what Symfony Flex can represent for the third-party developers. I invite you to read my very (too?) long comment) in the issue #70, as well as the following different answers. I sincerely believe that Symfony Flex should move in this direction, despite the reluctance of the Symfony team. @nicolas-grekas On this subject, and with all due respect, say that @nicolas-grekas Of course, Symfony Flex should not offer everything and anything, but I think it should purpose a common base, and that's why I see much more Symfony Flex as a great automation tool to standardize the project configuration system, than a simple tool dedicated to the simplification of the Symfony's project configurations, despite what the team say, and even if you have added directly into the plugin a special configurator for the Symfony Bundles ( From this discussion, I therefore proposed in the issue #173 to add the ability to extend Symfony Flex, just as we can extend Composer, and I have propose in the PR #174, a simpler implementation than that presented in this issue. Several advantages to this solution:
Of course, the Symfony team does not agree to add a feature that for them, will not be used by the community. This results in an unnecessary maintenance cost for a feature that is not really useful for the majority of developers, and I can understand if they are right. In ths case, it's up to us developers of the Symfony community, to show that this feature is claimed, if you find it interesting. And yes, I come back to the charge on this subject, but it's for a good cause ;-). I sincerely believe that this feature will be widely used for many projects, especially large open source or private modular projects. Symfony Flex could become an excellent tool for the configuration of projects and allow the community to design new Flex-based tools to do things that the Symfony's team did not initially envision or want to support, such as a plugin that supports recipes installed in the component. It is obvious that I will still be satisfied if you only accept the inline configuration, although I think that giving the opportunity to extend Flex would be better. Thank you for your great works, and have a good end-of-year parties :-) |
|
Hi, |
|
Flex recipes are validated and that's 100% part of the Flex experience. Inline recipes bypass any such validation. |
|
@nicolas-grekas : True, only recipes are placed on github but what is the sense in putting a recipe on a github repository when the only projects that would be using it are in a private self hosted gitlab intance for example... |
|
P.S. : Then again I might be wrong but I don't think I would be the only one thinking that way. Only time will tell. But I strongly believe that not giving an alternative for private packages (and by that I mean not just a non free one) will be somewhat a blocker for the maintainers of privately owned packages. Note : the issue could also be approached the other way around : the symfony.sh process could be adapted to either be able to run on a remote git based repository or to be able to run against recipes from a remote git based repository (publicly available by connecting to it using credentials, which could be those of a github account)... I do not have any insights on how the process in question works so I can only make suggestions, which could be possible or not. |
|
I strongly agree with @rrajkomar - not being able to use our private packages in the build process is a strong blocker. We have a lot of internal packages that we cannot (not permitted by the customers) host anywhere else but that need to be integrated in Symfony. Think of a monitoring package, some connection to internal systems and so on. If we can't load them via Flex, this takes the fun out of it and will possibly break Flex, because we still need to register stuff manually (which is by far more prone to errors than having our own flex recipes and packages which we take our own responsibility for. As an alternative to this, we could of course maintain our own standard edition of Symfony, but that's kind of the opposite of what Symfony 4 should become... For us, the ability to load internal packages that we maintain on our local gitlab server is a deal breaker for Flex usage. |
|
I think inline recipes would be great and when detected it could ask the user if it wants to use them. I'm too very interested in creating recipes for some of our internal bundles. The alternative solution which I'm testing with now is to run a custom version of flex server https://github.com/aurimasniekis/flex-server and a private (non-github) repository where I store the recipes. This seems to work very good but I run into issues with recipes which would require sessions and security changes which is currently unsupported. |
|
The current private recipes beta does mention a future price of 249€/month. If the unconfirmed intention of having private recipes as a paid feature, It would make sense they are opposing any inline recipe configuration. It would be sad, as I was hoping to use this feature for our private repositories. |
|
@tseho Thank you for your link, I thought the integration would be free as nothing was mentioned in this way in the links provided by @fabpot in my issue #71, and that your page is not referenced on the Symfony.sh website. So now I understand better their confirmed intentions, and this justifies much more the unwillingness of the Symfony team to add other solution allowing to use recipes for private packages. This remains a choice, and I do not despair to see the Symfony team reconsider their choice, allowing the community to add functionality to Flex with plugins (#174), if they do not want to support them natively (this issue). However, other service with the same principle, does not force their users to subscribe to a paid offer to use an open source tool with private repositories. Take for example Composer and Packagist, they propose the service Private Packagist to manage private packages, but it is still possible to use VCS Repositories, private servers Satis or Packagist, "inlines" packages, Artifacts, Path directories, etc. ... to manage private dependencies. Same for NPM with NPM Registry, they offer to manage private packages with a paid subscription, but you can still use Git/Github/etc links in the dependencies. In both cases, we do not have to use their servers if we wish. Consequently, we subscribe to their services by choice (simplicity, speed, security, serenity, etc ...) and not because we are constrained technically. I think that Symfony Flex should offer the same principle, and provide a solution that does not impose their proprietary Symfony.sh service. Of course, I understand and I encourage the promotion of this service, because it is important for SensioLabs SAS to diversify its income, allowing in return, to provide open source and proprietary quality tools (and everything that goes around). For example, SensioLabs' Blackfire is a great proprietary SaaS tool that is priced right. But concerning Flex and Symfony.sh, impose the service and propose it at a price of 249 EUR / mounth (297 USD), it's simply incomprehensible, especially since the services of the same type are rather between 7 and 16 USD per month. @nicolas-grekas To answer your sentence
For a service at 297 USD / month (in case it would be the final price, and not counting the 25 USD / month for Github if I use another service), I quickly justify the development of a plugin to work with private repositories. And I did not wait to realize it: Less than 4 hours on my free time to make a Composer plugin to use inline recipes, to define a package in So I do with, because in the end, my profitability is much higher, but I will prefer to see this capability natively integrated in Flex. It goes without saying that I am always open to making changes to my PR if you want to offer the capability to extend Flex for other plugins. Of course, I know that the Flex's extensibility is another issue, but it is closely related to this one. @fabpot, @nicolas-grekas, the Symfony Core' team, please, reconsider the orientation of Symfony Flex. Thank you |
|
@francoispluchino did you consider the flex server I mentioned before? I think it does exactly what you want I guess. |
|
I will just mention one thing: there is no relationships between this issue and the paid offering. I've given the reasons why I'm opposed to this proposal. I still stand by them. I don't see why I would need to change the orientation of Flex. I have good reasons to think that the direction you want to take is not the right one. Now, discussions are good. Discussions are open. I'm demonstrated more than once that people can convince me to change my initial plans. But quite frankly, I don't even understand why one would think that I would consciously limit a tool just to be able to propose a paid offering. 14 years of free work for the community. Think about this. Anyway, have a good day. I'm done for now. I will go out, take a break. |
|
@fabpot Sorry I joined the conversation with an unrelated pricing. I was surprised by the amount since pricing of SensioLabs tools, like Blackfire, have always been fair. I have been following the progress of Flex since you announced it and I have great hopes for it. Flex will still be a welcome tool for open-source projects and curated recipes. |
|
@fabpot We misunderstood each other, so I will try to add a clearer explanation to my comments. When I talk about orientation, I'm talking about features, I'm talking about giving the Symfony community the opportunity to do more, by bringing their little stone to this edifice that you've managed to wear and do grow over the years. I appreciate and respect your work a lot, so please do not make me say what I did not say :-). So when I talk about orientation for Symfony Flex, I'm not talking about forcing people to use a paid service, but the orientation to only use recipes from Symfony.sh, and so, consequently, to limit the recipes for the open source packages or being forced to use Symfony.sh blocked on Github for private recipes (which obliges all those who do not use Github for private repositories to subscribe in addition for this service). When I say confirmed intention, however, I'm talking about the intention to do pay for the Flex Server service after the Beta-test phase. Confirmed because it is an information on a official page of this service. And as I said, it's a perfectly normal logic that does not bother me (we can always discuss the price of the service, but that's another subject). I indicate that I understand better the reluctance of the team because precisely, this point is not negligible and for a company, it is justifiable to find an economic model or new services especially when there is like you, a consequent work for an open source project requiring more and more resources. I also do not think it's wise to focus solely on the community (but yes, we must not forget him). So this is not a criticism as you think, on the contrary, but a statement of choice should not be neglected in the development of a tool, whether open source or proprietary. On the other hand, what bothers me is to leave only the ability to set the recipes on Symfony.sh, which is a paradox for a tool to make Symfony Framework more flexible. Of course, I include in my comment the capabilities to extend Flex, because if it were possible, the question would not arise, and those wishing to use this method or another, could do it. It would not be harmful for Symfony Flex if some used this method, because it would concern them, and only them, but not other users of Symfony Flex. To return to inline recipes, your 3 arguments are interesting but questionable, and as you wish, I will take them resume:
This is not an argument, but as I already said, I excited to discover your new ideas!
I gladly accept your argument, even if I do not consider it imperative to have an updated recipe to be a necessity, because the recipe has to work with the package version. If it is not the case, it is a bug and the recipe must be fixed. Besides, a recipe can copy files from the installed package. And of course, the file is not overwritten when updating a new version (normal, it's for the config). So if this file is buggy, in the end, it is the recipe that is buggy. It can be updated in the package or on a server. I'm sorry, but it's the same problem, the end user must do the correction by hand. And for those who install the package with an updated recipe, the file is still in the package, if it is not as updated, the problem will still be present. So a new version of the package must be published. So in this logic, a configuration should not move for a given version, but indeed, the configuration could move in the case where the configuration of the project must evolve (like Symfony 2.x to 3.0, and 3.x to 4.0). Moreover, on this subject, In this case, how will we do with the recipes ? You create a new remote repository, and you update Flex to be compatible by automatically detecting the version of the Symfony project to automatically select the good repositories? Otherwise, a simple variable in the composer.json file may be much simpler, or something else, because I did not look at it in more detail, but of course, we are not there.
As I've already explained several times, I understand this argument, and you've added precisions for the recipes-contrib, but for inline recipes for private packages, it's useless. I specify it, because we never know, I never said that the server mode was useless and it was necessary to use only inline recipes (see below).
I always agree with you for open-source packages, and a server is therefore mandatory and relevant to achieve this functionality for all existing packages, and should not be directly associated with tools such as Flex. However, I do not agree with you about private packages for one or more private projects, because if we create independent private packages, it is for reuse on the same type of project 99% of the time, because otherwise the code is placed in the project's I do not think it is relevant to be limited to one distribution channel for recipes. For me, Symfony.sh must be the official remote repository of recipes, and it would be good to propose several types of repositories for recipes, just a 67E6 s we can use different types of repositories for Composer, knowing that the official Packagist.org repository is enabled by default. As I said in this comment, I think that it would be very useful to propose a flexible approach on repositories, by defining Symfony.sh as official remote repository, and to allow to add packages like recipe repositories, but also of inline recipes directly in the package of the library, and of course allow to define a custom private server. You do not agree with me on this point, and I can understand it and in any case accept it, but in this case, can you please, reconsider the possibility to the third party plugins, to extend the functionality of Flex, as we can extend the functionalities of Composer? Outside the argument of @nicolas-grekas regarding the cost of maintenance to keep a stable public API, I do not see why the concept of this idea is rejected. This concerns issue #173, but it flows from the issue of the private repositories and this issue, even if it goes much further in the value proposition. On this subject, I invite you to re-read the issue #70 and mainly my proposals in this comment, especially this one, and this last. Thank you again, and I hope I have convinced you that my intentions were good. |
|
I just want to also add my two cents: we are an agency and therefore are quite often bootstrapping new projects.
So in general using Sf4 without Flex is a clear DX downgrade in comparison to using Sf3 with the standard edition. So you can say "well, just don't use flex", but that also implies "please use our more complicated product" which goes against the main idea behind Flex, as I recall. I have the feeling that while providing a more opinionated framework we also have lost the possibility to configure some things or opt out from them. This also includes things like being forced to use So while in general I love the way Sf(4) is going, I just have the feeling that we are currently stomping on the valid use cases of several users (myself included).
I am 10000% sure, that here are at least 10+ people willing to implement this feature as soon it is green lighted here. |
|
@apfelbox maybe you missed
That's only 20% of the sentence: you missed the "and maintain" part. How many of them would be willing to make their own Flex-like alternative? That's basically the question. |
You are right, that is indeed still possible. I was rather referring to the "best of both worlds": having a minimal project and the comfort that is provided by Flex as well as not being "locked in" to certain restrictions that are currently still in place by Flex. (which are mainly by design, though) The step away from a huge config file where 90% of the lines are commented out was a good one. But the only way (?) to have this managable is by using Flex – which imposes its own restrictions.
That is indeed the question. But that is a discussion without actual result: some say "there is nobody", some say "yeah, there is/will be somebody" – so not very goal oriented. Especially as it seems that there might even be an alternative to this whole topic by just allowing Flex itself to be extended in certain key places, as @francoispluchino proposed. So that the support for in-package manifests can be added via a plugin. |
|
@apfelbox As @nicolas-grekas says, there is a skeleton to get something similar to Symfony Standard Edition with However you have a very cool novelty allowing unpack a pack, and therefore simply remove parts that you do not want to use. @fabpot explained it very well in this post. With this vision, we can easily replace SSE with Flex. So I would not say it's DX downgrade, it's even the opposite, but it's frustrating in its limitations. To explain myself, I will take your example with With SSE, you clone and modify SSE to create your own project template to quickly bootstrap a new project. You have lots of code, config files, etc ... For your example, you modify the composer.json file, and the So now with Flex, the skeleton is only a Flex will copy the recipe for What happens during the installation process with your new template skeleton, and more precisely when running the So to fix it, you have to modify the 2 files, which is not problematic for the installation of a new project with the official template, but totally ineffective for a system allowing to design project templates using Flex flexibility. Because currently, to solve the problem, you end up redoing a pseudo SSE based on Flex including all modified files ... And if you have several different templates, well, you multiply the number of modifications by the number of project templates, like SSE. In this case, we can say that it is a DX downgrade against Flex, since we lose all its power and its updated recipes. So sorry, I'm going to come back to my work and my vision again, because I recently migrated my tool to merge it with Flex because there is a very strong similarity in the methodology as mentioned above:
So using Flex, I deleted some configurators, I have a lot less code to maintain, I can use the power of Flex, while adding my features (custom configurators, inline recipes and package repositories for recipes, etc). You understand that I hacked Flex because it is impossible to extend it for the moment. It works fine until Flex changes without warning its internal APIs, which is normal since there is no system to extend it. However, I consider that the game is worth it because I hate to use tools that do 90% of the same work with a similar approach, and that I deeply believe that Flex can reach a lot more people than if I propose my own plugin to replace Flex. Maybe someday you'll understand why it's so important for me to extend Flex. So, regarding the problem mentioned above in my message: adding the ability to extend Flex, we could create plugins for Flex to:
I do not think that these features should be natively integrated with Flex, but I think that these features can be useful for many people, but not all. Therefore it would be preferable to outsource them with plugins. To achieve all that I have indicated, the PR #174 needs to be updated. @fabpot, @nicolas-grekas My additional examples are still not enough for you to accept my value proposition? Given that the discussion is rather about the Flex extensibility and not just about the inline recipes, I suggest continuing on the issue #173 and closing this issue. What do you think? PS. I know, my comments are really too long right now :-) |
|
The discussion is big, I have a simple question: we have our company private bundles git repositories on our private servers. We want to integrate that bundles in our Symfony private application on Flex way. Is there any free way todo that?! |
|
@zorang We are currently implementing our own solution for this. It is self hosted and allows private recipes from a private repo behind a firewall. For the private bundles themselves, you should use satis, toran or private packagist. See here: https://github.com/moay/symfony-flex-server I am still working on this, give a week and it should be ready to be used. I'll release a beta. Currently, the flex functionality works, but it still needs some tweaking, some tests and a proper documentation. EDIT: This answers @zorang's question, but I think, it does not actually touch this issue. |
|
Looks good, we are using https://github.com/aurimasniekis/flex-server and also works very well :) |
|
@slootjes Yes, we do too for now, but the project neither well coded, nor supported nor compliant to the current version of flex. F.i. it does not support |
|
Good to know, I've starred the repo 👍 |
|
I think it's time Symfony core developers listen to us. I still don't understand why the need of centralise this tool instead of giving freedom to the developers to decide. I agree that for official packages it makes sense, but not for other packages. It makes no sense for me to prevent developers to use the flex tool for their own private packages, and it doesn't make sense having to maintain the recipes in a separated repository. Unnecessary burocracy. It's the responsibility of the end user/developer to decide what to include in his/her project and to verify that it works and it is secure. For private packages seems that the only solution is to create our own |
|
There is definitely merit in having central repositories of recipes is to ensure quality (reliability, validity, and "officiality" if that's even a word) at least to some degree. However I agree that "third-party" (not listed in a core or contrib repositories) recipes need to be implemented, because currently recipes cannot be implemented for closed-source/private packages. It would greatly improve the experience for digital agencies working with Symfony on multiple projects. If "third-party" recipes are implemented somehow, it should be to allow flexibility in the cases where it's absolutely needed without sacrificing the benefits of centralized repositories. ProposalI'm not a part of the Flex project, so I don't know off the top of my head how complex something like the following would be - I offer it purely as an idea, not hey-please-implement-this-okay-thanks. Currently, the
{
"extra": {
"symfony": {
"allow-contrib": true,
"allow-untrusted": [
"vendor/package-name",
"vendor/package-name2"
]
}
}
}Obviously this comes with its own down-sides, just as having to host your own private recipes repository has its downsides. I'm not the one to make this kind of decision, but considering how often this issue keeps cropping up it would be nice to see a solution to this problem (whatever that may be) in the future. |
|
I wish I could have such Composer plugin that would read a recipe from inside the package and then play it. |
|
Just out of curiosity I did a PR where I added support for inline recipe, only ~20 lines of code is required for this to work. How it works ? easy. This proof of concept is still a work in progress but adding support for local recipes should be definitely quite easy to do and I might dedicate more time if I see some interest. |
|
This is not something that is going to happen. Call it opinionated - it is. That aligns to what recipes are from our pov: opinionated ways of using some libraries. i.e., not something that belongs to the same domain as the code - but that belongs to packaging. See also #572 (comment) Thank you for submitting the discussion anyway. |
Now that Flex has matured a bit, I would like to bring this point up again. Currently, having dozens of internal bundles (and several open-source), having dozens of applications using those bundles, we prefer to follow the recommendations and standards as much as possible. This means that we will start using Flex from 4.0 on.
The selling point of flex, is the ability to provide an "installation" into the application by using manifests. However, this being in an external repository, poses several issues:
To me, these are a few blockers that prevent Flex from being useful for me. I have dozens of bundles that have a mandatory configuration, but I won't be able to utilize flex to increase the Developer experience. It will actually make behavior different between the recipe'd packages and internal packages when it comes to installing, making it annoying to find out whether or not you still have to add some config file for example.
I'm not sure if this is a technical limitation, but I would love to be able to do the exact opposite, inline the configuration in my package. This gives me several benefits:
The directory structure of an inlined flex config could be as simple as adding a
.flexdirectory or something similar.The text was updated successfully, but these errors were encountered: