E528 fix(statistics): allow at_port to be specified manually by coroa · Pull Request #1386 · PyPSA/PyPSA · GitHub
[go: up one dir, main page]

Skip to content

fix(statistics): allow at_port to be specified manually#1386

Merged
FabianHofmann merged 22 commits intoPyPSA:masterfrom
open-energy-transition:feat/statistics-flexible-at-port
Feb 23, 2026
Merged

fix(statistics): allow at_port to be specified manually#1386
FabianHofmann merged 22 commits intoPyPSA:masterfrom
open-energy-transition:feat/statistics-flexible-at-port

Conversation

@coroa
Copy link
Member
@coroa coroa commented Oct 9, 2025

Changes proposed in this Pull Request

For retrieving statistics in respect to a specific port, we'd want the
at_port statistics argument be able to accept a single port or a list of
ports (as the type arguments already said they would).

This means that for links with an efficiency of 0.9, they could be accounted for at the output capacity, for instance:

In [29]: n.optimize.expressions.capacity("Link", at_port="1")
Out[29]:
LinearExpression [carrier: 1]:
------------------------------
[DC]: +0.9 Link-p_nom[Norwich Converter] + 0.9 Link-p_nom[Norway Converter] + 0.9 Link-p_nom[Bremen Converter] + 0.9 Link-p_nom[DC link]

@FabianHofmann Does that make sense?

It's still marked as a draft, as we'll have to add documentation first.

Checklist

  • Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
  • Unit tests for new features were added (if applicable).
  • A note for the release notes doc/release_notes.rst of the upcoming release is included.
  • I consent to the release of this PR's code under the MIT license.

@coroa
Copy link
Member Author
coroa commented Oct 9, 2025

The match statement is consistent with the minimal python version 3.10. https://peps.python.org/pep-0636/

10BC0

@FabianHofmann
Copy link
Contributor

yes, I like that feature very much!

@gianvicolux
Copy link
Contributor

I added a raise ValueError when at_port is defined by means of the prefix "bus". I also updated the docstrings in pypsa/statistics/expressions.py.

Now, the complete documentation must be updated, too.

@coroa @FabianHofmann

@coroa
Copy link
Member Author
coroa commented Oct 15, 2025

@gianvicolux I think if you add a small release note this is ready for review.

Copy link
Member Author
@coroa coroa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, with this it's good from my side.

@coroa coroa requested review from FabianHofmann and lkstrp October 15, 2025 10:07
@coroa coroa marked this pull request as ready for review October 15, 2025 10:08
Copy link
Member
@lkstrp lkstrp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't notices that at_port is just this:

if not at_port:
    ports = [ports[0]]

which makes this PR a bug fix and not a feature.

Since we are touching it already, I'd make more changes and turn it into a bug fix and a new feature:

Which ports to consider:
- True: All ports of components
- False: Exclude first port ("bus"/"bus0")
- False: Consider only first port ("bus"/"bus0")
- str or list of str: Specific ports to include
    Note: use port number only, without 'bus' prefix (e.g., '1' instead of 'bus1')

is mixing types and is not intuitive.
What about removing the boolean and just allowing a list of ports? The default could be set to None, which would map to c.ports. 'bus' and 'bus0' can be treated equally.

We are also now in 1.0 territory, which means:

  • that we want to still allow True and False and map them to all ports or just the first one, raising a deprecation warning.
  • don't wanna just raise a ValueError for at_port="bus0" which very buggy worked before, but would fail now.

Also why kicking the 'bus' from 'bus1', but only allowing '1' but not 1 as int? Because that's what is returned from c.ports?

Features
--------

* New feature for retrieving statistics in respect to a specific port: ``at_port`` statistics argument is able to accept a single port or a list of ports (as the type arguments already said they would). Note: use port number only, without 'bus' prefix (e.g., '1' instead of 'bus1') (https://github.com/PyPSA/PyPSA/pull/1386).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this file. This needs to go in docs/release-notes.md now and with markdown formatting so (- for bullet points, (<!-- md:pr 1386 -->) to link the pr and single '`' for code formatting.

@coroa
Copy link
Member Author
coroa commented Oct 15, 2025

I didn't notices that at_port is just this:

if not at_port:
    ports = [ports[0]]

which makes this PR a bug fix and not a feature.

when i touched it first, i considered it as a feature, i only noticed the docstrings late in the process. It's true that the docstring is not implemented.

Since we are touching it already, I'd make more changes and turn it into a bug fix and a new feature:

Which ports to consider:
- True: All ports of components
- False: Exclude first port ("bus"/"bus0")
- False: Consider only first port ("bus"/"bus0")
- str or list of str: Specific ports to include
    Note: use port number only, without 'bus' prefix (e.g., '1' instead of 'bus1')

is mixing types and is not intuitive. What about removing the boolean and just allowing a list of ports? The default could be set to None, which would map to c.ports. 'bus' and 'bus0' can be treated equally.

I think, you need an option to specify all ports (across multiple component types) and this should not be the default, because it easily leads to double-counting and it was not the default up to now.

What about

at_port: Literal["all"] | int | Sequence[int] = [0]

as preferred type and thus deviating from c.ports (which i don't like).

We are also now in 1.0 territory, which means:

  • that we want to still allow True and False and map them to all ports or just the first one, raising a deprecation warning.

Sure.

  • don't wanna just raise a ValueError for at_port="bus0" which very buggy worked before, but would fail now.

This did not work before, it returned effectively the result of at_port=c.ports including double-counting potential and all. I think people should know they made a major mistake.

Also why kicking the 'bus' from 'bus1', but only allowing '1' but not 1 as int? Because that's what is returned from c.ports?

Indeed, I like consistent naming and c.ports is the closest official definition i know. It's not a definition i like, but ambiguity is worse imho.

@lkstrp
Copy link
Member
lkstrp commented Oct 16, 2025

I am happy with:

at_port: Literal["all"] | int | Sequence[int] = [0]

And I agree, c.ports is not great. It returns "" for single ports right now. Which is good for primitive column name building, but not as a argument set. What would you change there? Just switching it to return 0, 1, 2 .. ?

@coroa
Copy link
< 8000 /span>
Member Author
coroa commented Oct 16, 2025

I am happy with:

at_port: Literal["all"] | int | Sequence[int] = [0]

And I agree, c.ports is not great. It returns "" for single ports right now. Which is good for primitive column name building, but not as a argument set. What would you change there? Just switching it to return 0, 1, 2 .. ?

I think either numbers 0, 1, 2 would be good or actual attribute names: ["bus"] or ["bus0", "bus1"]. What about having a second c.port_attrs or c.port_cols which returns the latter in additon?

@lkstrp
Copy link
Member
lkstrp commented Oct 16, 2025

c.port_cols would clutter the API a bit, but since we won't align "bus0" with "bus" any time soon, I am sold

@coroa
Copy link
Member Author
coroa commented Oct 16, 2025

Hmm .. i thought some more. There will be recurringly the need to accept both types of inputs, port_cols and port_nums. What about a new dataclass:

type PortsLike = Literal["all"] | int | Sequence[int] | str | Sequence[str]

@dataclass
class Ports:
    nums: list[int]
    c: Component

    @classmethod
    def from_like(cls, ports: PortsLike, c: Component):
         match ports:
              ...
  
          return cls(nums, ct)

    def __iter__(self):
        return iter(self.nums)

    def cols(self):
        ...

and c.ports returns that instead. Not sure about transition path. What do you think?

@lkstrp
Copy link
Member
lkstrp commented Oct 16, 2025

I think we should keep it simple. Returning a data class could cause confusion for a use case as simple as this. It also adds overhead. Accepting both types can be done everywhere with a simple ports = _as_ports(ports) or an internal data class if you think that's better. There are not even that many use cases for it right now, just for statistics I think? Do you have any other potential stuff in mind where we need ports?

@coroa
Copy link
Member Author
coroa commented Oct 17, 2025

Not sure what's the difference between internal, external. here. I'd expect the overhead to be negligible.

You'll need at least two methods _as_ports(:PortsLike) -> Sequence[int] and _as_port_cols(:PortsLike, :Component) -> Sequence[str].

My thinking was that the dataclass would make that more discoverable, but the use is mainly internal anyway.

The problem with ports also comes up frequently when formulating custom constraints, so making that easier would be a good thing.

@lkstrp lkstrp marked this pull request as draft November 17, 2025 08:17
@lkstrp lkstrp closed this Feb 5, 2026
@coroa
Copy link
Member Author
coroa commented Feb 5, 2026

I still need something like this, is there a particular reason that you did close the PR?

@lkstrp lkstrp reopened this Feb 5, 2026
@coroa
Copy link
Member Author
coroa commented Feb 16, 2026

@lkstrp Happy if you can have a look. The diff was just against the wrong branch.

@FabianHofmann
Copy link
Contributor
FabianHofmann commented Feb 16, 2026

Nice refactoring, there is one thing we have to be aware of though.

If I see correctly, there is now an inconsistency how bus_carrier and at_port interact. When bus_carrier is specified the expression should loop over all ports and filter out the ones matching the bus carrier. For this case, at_port should automatically resolve to "all".. Now, most functions have at_port "hardcoded" to "bus0", meaning bus_carrier filtering silently misses connections on other ports.

We could instead keep None as the at_port default and centralize the implicit default in _aggregate_components.

Copy link
Member
@lkstrp lkstrp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The API get's also much better, thank you!

We could instead keep None as the at_port default and centralize the implicit default in _aggregate_components.

I think this would be the cleanest

ports = [ports[0]]
if at_port is False or at_port is None:
warnings.warn(
f"Passing `at_port={at_port}` is deprecated. Use `at_port=0` instead.",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add version 1.2 and removed in 2.0, just check format of other deprecation warnings

@coroa
Copy link
Member Author
coroa commented Feb 17, 2026

Nice refactoring, there is one thing we have to be aware of though.

If I see correctly, there is now an inconsistency how bus_carrier and at_port interact. When bus_carrier is specified the expression should loop over all ports and filter out the ones matching the bus carrier. For this case, at_port should automatically resolve to "all".. Now, most functions have at_port "hardcoded" to "bus0", meaning bus_carrier filtering silently misses connections on other ports.

We could instead keep None as the at_port default and centralize the implicit default in _aggregate_components.

I think, i only chose at_port="bus0" where there was at_port=False before. An example for the previous at_port=None is optimal_capacity:

There i have at_port=None as default,

https://github.com/PyPSA/PyPSA/pull/1386/changes#diff-ce7e31f541948b44c5a77e2e6d35c8ba65265254f2d1e97174cd234b2a548cdaR1112

and then a:

if at_port is None:
    at_port = "all" if bus_carrier else "bus0"

https://github.com/PyPSA/PyPSA/pull/1386/changes#diff-ce7e31f541948b44c5a77e2e6d35c8ba65265254f2d1e97174cd234b2a548cdaR1191-R1192

Where do you think there is an inconsistency?

@lkstrp
Copy link
Member
lkstrp commented Feb 17, 2026

Now I am confused. This PR is also on an old master where docstrings did not match the signature.

You indeed don't introduce any inconsistencies, they have been in there before. In my understanding we should just default to None in any expression/ statistic which also takes bus_carrier. This gets then always correctly resolved via

if at_port is None:
    at_port = "all" if bus_carrier else "bus0"

capex for example was defaulting to False, you c CE38 hanged it now to "bus0" which is fine. But it should have been None in the first place. So this is an additional bug fix.

@coroa
Copy link
Member Author
coroa commented Feb 17, 2026

Now I am confused. This PR is also on an old master where docstrings did not match the signature.

Is it? capex on the latest master is:

def capex( # noqa: D417
self,
components: str | Sequence[str] | None = None,
groupby_method: Callable | str = "sum",
aggregate_across_components: bool = False,
groupby: str | Sequence[str] | Callable = "carrier",
at_port: bool | str | Sequence[str] = False,
carrier: str | Sequence[str] | None = None,
bus_carrier: str | Sequence[str] | None = None,
nice_names: bool | None = None,
drop_zero: bool | None = None,
round: int | None = None,
cost_attribute: str = "capital_cost",
) -> pd.DataFrame:
"""Calculate the **capital expenditure**.
Includes newly installed and existing assets, measured in the specified
currency.
Parameters
----------
components : str | Sequence[str] | None, default=None
Components to include in the calculation. If None, includes all one-port
and branch components. Available components are 'Generator', 'StorageUnit',
'Store', 'Load', 'Line', 'Transformer' and'Link'.
groupby_method : Callable | str, default="sum"
Function to aggregate groups when using the groupby parameter.
Any pandas aggregation function can be used.
aggregate_across_components : bool, default=False
Whether to aggregate across components. If there are different components
which would be grouped together due to the same index, this is avoided.
groupby : str | Sequence[str] | Callable, default="carrier"
How to group components:
- `False`: No grouping, return all components individually
- string or list of strings: Group by column names from [c.static][pypsa.Components]
- callable: Function that takes network and component name as arguments
at_port : bool | str | Sequence[str], default=False
Which ports to consider:
- True: All ports of components
- False: Exclude first port ("bus"/"bus0")
- str or list of str: Specific ports to include

and on this PR:

https://github.com/open-energy-transition/PyPSA/blob/1cb1500a6a9cd3dc18fd1f430b440622202d8520/pypsa/statistics/expressions.py#L601-L641

Where do you think is the inconsistency?

If you mean the difference in treatment between capex, optimal_capacity and energy_balance in regards to at_port and bus_carrier is convoluted, i wholeheartedly agree. I did not understand why one would want to choose the defaults as they are, but i assumed there would be a good reason.

Sounds like there might not be one. Maybe @FabianHofmann, you can clarify if there was intent. Or if you think your rule above:

We could instead keep None as the at_port default and centralize the implicit default in _aggregate_components

should actually be general.

@FabianHofmann
Copy link
Contributor
FabianHofmann commented Feb 18, 2026

@coroa sorry for being late here. the problem is that when specifying at_port=bus0 and not to False as before the loop over the ports will not go beyond bus0. that is when the user wants to extract the information for a bus_carrier wo explicitly passing at_port=False or None, the code will only check if bus0 has that bus carrier. does that make sense?

EDIT: Just wrote a test that checks the behavior, it confirms that this would be a breaking change. Fine if I push it?

@FabianHofmann FabianHofmann moved this from Todo to In Progress in PyPSA - Roadmap Feb 18, 2026
@coroa
Copy lin 3D88 k
Member Author
coroa commented Feb 19, 2026

@FabianHofmann Feel free to push the test.

Copy link
Contributor
@FabianHofmann FabianHofmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coroa this is very nice now. long overdue to tackle the inconsistencies. I adjusted everything I wanted, happy to pull this in. As a follow-up we should adjust the docstrings in optimization/expressions.py.

@lkstrp you want to have another look?

Which ports to consider:
- None: Automatically set to "all" if bus_carrier is specified, otherwise "bus0"
- "all": All ports of components
- "bus0": Consider only first port
Copy link
Contributor

Choose a reason for hiding this comment

3D88 The reason will be displayed to describe this comment to others. Learn more.

this line (coming from me) does not add a lot. happy to remove it again

@FabianHofmann FabianHofmann enabled auto-merge (squash) February 23, 2026 09:10
@FabianHofmann FabianHofmann merged commit a837d9b into PyPSA:master Feb 23, 2026
24 of 26 checks passed
@github-project-automation github-project-automation bot moved this from In Progress to Done in PyPSA - Roadmap Feb 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants

0