Move ports to subdirectory #2401
Comments
|
The name hardware_ports isn't really correct though, even a bit confusing, since minimal/unix/windows are not exactly hardware-bound |
|
@tannewt: No revolutions please. If you would like to contribute to MicroPython, but never did before (as https://github.com/micropython/micropython/graphs/contributors shows), we have a list of useful things to do just for such folks: #1331 . I'd especially like to draw attention to the need to restore test coverage to be back to 90%+, that's where more hands would really help. |
|
Simply "ports" would be a better name IMHO, and I agree it would be nice to neaten up the directory structure. |
|
@pfalcon I'd be happy to help with the testing coverage over the long term. I'd love your opinion on the ports subdirectory though because I'm adding samd21 support to micropython. My current code adds it as another top level but it'd be cleaner IMO if they were all in a subdirectory. Code is here: https://github.com/adafruit/micropython |
|
@tannewt : You can read MicroPython's author opinion on that here: #1907 . I personally think that it's far from the top-priority issue in uPy development, and an obvious, existing way "just works". That way being: fork repo, maintain your port, work together with uPy community on resolving general uPy issues, when it's clear to everyone that yourself and your port are there to stay (i.e. being maintained daily, you'll be around to answer users' questions), it can be merged to mainline. It seems that you're on that way already, and that's great. Problem of "too many ports" can be resolved when there will be indeed too many well maintained ports, and there're a lot of work for everyone to do to get there yet. In the meantime, I'm strongly -N or renaming something, because it complicates usage of "git log" - a tool, which should be daily friend of everyone who contributes to uPy (or any other project for that matter). |
|
@pfalcon thanks for the link. That background is really useful. I came into this assuming centralizing is the right way to do it. Having copies in different repos risks making sharing of work (code and documentation) harder. The points about noisy git log and repo size make total sense though. Has anyone already made a micropython port which separates micropython well that I can use as an example? I was looking at mynewt and they do something similar. Their SAMD21 support is separated into this repo for example: https://github.com/runtimeinc/mynewt_arduino_zero |
|
@tannewt it would be nice to eventually clean up the top-level directory structure. But we can already address your (valid) concerns re sharing work without doing a restructure: a lot of the shared code is slowly being moved to the lib/ directory (see lib/utils/pyexec for example), and the common modules are in extmod (eg machine stuff, fatfs stuff). |
|
@dpgeorge is there a reason not to do the restructure I suggest? I'm willing to do the work if we can decide how to restructure things. (Splitting into separate repos was another suggestion I saw.) I didn't mean to get into the resharing discussion here though I'd love to have it elsewhere. |
Main 2 reasons not to do it are:
Of course, the sooner we do such a restructure the better because points 1&2 above (and everything else) just get more entrenched over time. @tannewt what are your main reasons for wanting to restructure? The work to do it is insignificant compared with the "fallout". |
|
Perhaps a good compromise would be to use the new structure for the new ports (with clear instructions on how to do it), and once that has proved working well, maybe gradually start to move the other ports? |
|
I'm not sure why 1 is an issue. Doesn't 2 is tricky. One option is to submit changes to other forks as well or simply explaining what the change was and how to do it yourself. I come from a monolithic codebase setting where you just warn everyone and make sure big changes have landed before the switch. The reason I thought of doing it is simply to separate core micropython from port specific micropython. Doing that makes it clearer which directories are for ports and which are micropython. Once thats clearer you can start imposing stronger assumptions of the port directory layout, implementation APIs and file naming. If we could go further and structure it in a way where core micropython is in a separate repo from port specific stuff that could be even better IMO. How do you decide what parts of micropython get pulled into the microbit version? @deshipu I'm not sure that provides the clarity benefit though. |
microbit is a special case because of requirements from the BBC and the yotta build system, and I was just copying needed files into the microbit repo. But BBC requirements and yotta are now gone (effectively) and it would be nice to bring the microbit port (actually a nRF51 port, with microbit being a board within that port) into the main repo. That's a very low priority thing though. |
|
That would be awesome to reintegrate it. Someone just pushed nrf52 support
|
I think so too - restructuring may cause some "fallout" but the sooner the better IMO. The number of ports (and users wanting to port..) seem to be consistently increasing, so thats great! If the structure of the repository is hindering devs to do the ports that would be a pitty. |
|
@tannewt what I proposed at least prevents this entrenching and buys us some time In addition, the external ports can move to the new layout gradually, when they have time, and not be forced by sudden changes in the core. |
|
@deshipu Thats true! If we introduce it now, then new ports at least can go there. What external ports are you referring to? What is your preference for the new layout? Seemed like a few people earlier preferred the directory name |
|
I think my favorite is the idea of keeping each port in a separate repository, and having the micropython core as a subrepository there. It also lets the authors review and merge their pull request without having commit rights in the main repository, which could greatly spread the reviewing work. |
|
I went ahead and split off the esp8266 port just to see what it would take: https://github.com/deshipu/micropython-port-esp8266 (it would probably still have its own readme and docs and maybe even tests) |
|
It could also have the history of all the deleted files pruned, then it would be really fast to check out. |
|
I like both idea's, restructure and separate repo's. Better now than later when it is still complicated. and all boards in folder "arch" as separate repo. |
|
I have to admit I didn't think about doing it that way -- putting the ports as submodules of the main repository. My example puts the main repository as a submodule of each port, and after some consideration, I think this is actually better, because:
|
|
@toolmacher That is an interesting idea. However, I agree with @deshipu that it would be too difficult to keep the port in sync with the apis of the core. In fact, I think that moving ports into the main repo is the way to go. Why? Having all ports in one place will make the existence of each port very well known and make it easier to unify shared APIs. Imagine a change that standardizes the machine.I2C api between all of the ports by sharing a lower level C API:
So, I prefer 3 because its the easiest way to unify the machine APIs. |
I see no real problem with that. There will always be ports of lower priority, or even completely abandoned by their authors. I see no reason why they should block development of the core.
I think you are optimizing for a very rare case here. In fact, so rare, that it didn't even happen so far in the history of this repo. The idea of an atomic commit that updates in one go the API, all the places where it is used, all the documentation that refers to it and all the tests is an alien idea for some of the developers here. In fact even at this moment there are some broken drivers and documentation that weren't updated after an API change. I don't think it makes sense to optimize for something that never happens. Even if a proper review process for everyone was introduced and this was enforced, this kind of change should be really rare in a normal project. On the other hand, with ports each in their own repository, you can have different sets of people with merge rights, faster code reviews and thus potentially quicker development. A smaller team of people can also easier agree on what to do, and what the standards for the particular port should be. It's also easier to test ideas in one port, and then implement them everywhere if they prove to work well. You also don't need to check out a lot of unneeded code to work on a particular port -- you only download what you actually need, that is, the port's code and the core MicroPython. The whole move only makes sense if it will allow the MicroPython project to scale better. Putting everything in one place doesn't work very well for this. |
I don't think its as rare as you think. Several times when working on teensy or stmhal, I've made changes which broke the other and the only reason I've noticed is because we build them both with travis. @pfalcon has also made changes which broke stuff other than what he was working on (I'm not picking on Paul - anybody that works on larger projects does this from time to time), and the reason I noticed was because I have my own travis builds which build a bunch of additional targets that the regular travis build doesn't. |
|
I like the idea of having everything in one repo so that it's easy to change things and keep (relative) consistency. I like the idea of separate repos so the core is lighter, it can be quicker to develop and allows fine grained access control. Right now I'm thinking that the project is still too immature to split it up. We need to stick together (as a team and as a code base) to hammer out deep issues that have long-term consequences like the machine API and consistent/portable drivers. IMO those are the 2 main things that we should work on in the coming months to solidify uPy's foundations (the core is already very stable, will always be defined by upstream CPython, and will slowly evolve). And to work out the details of these APIs we need a single repo. Then, once the foundations are set down and the docs are written, the ports will be built around them and things will be pretty consistent. So, although it would be nice to restructure the top-level dir (and git blame still works on moved files, still has past history, but not git log) I don't think it adds any real benefit right now. Yes it's a bit confusing to a new comer, but once you get used to it it's really fine. Once we have a more mature project we can reconsider what to do (either move or split to other repos). @tannewt what concerns do you have leaving the structure as-is? Regarding the samd21 port, one thing that would definitely be moved would be the CMSIS library, it would go from stmhal/cmsis to lib/cmsis so it could be reused by other ports. Actually, in the process of moving we'd upgrade it to the latest version. |
|
@dpgeorge If everyone wants to leave it as is thats fine with me. I think its more important to have one repo. My concern is just that as ports get added the purpose of each top level folder gets less and less clear. Easing the ramp up time to newcomers is a good thing because the more hands working on micropython the faster it can get better. Having a subdirectory for ports also lessens the overhead of additional ports because it doesn't clutter the top level. That makes it easier to centralize the ports. |
|
@tannewt I agree, and I'd like to clean up the top-level dir eventually. Right now though we are not adding any new ports, and soon if we add samd21 that'll only be 1 extra. Beyond that I'd say we should revisit this issue. |
|
@dpgeorge if we want to do it eventually why not now? |
|
Because eventually we might want to do it a different way (eg separate repos). Unless we can come to an agreement now on the structure that is relatively future proof. |
|
it's hard to figure out where to find PyBoard "port". And yes... it's better to make a proper project structure now rather than after there will be even more ports/forks/etc. Technical debt is growing every day |
|
All ports are now moved to the ports/ directory. |
What do you all think about moving ports to a separate subdirectory? I'll do the work if we all agree it'd be worthwhile.
It'd take the top level from
To:
Hardware ports would be:
The text was updated successfully, but these errors were encountered: