8000 Inconsistencies in handling of block quotes and pre-formatted code blocks · Issue #515 · tdlib/telegram-bot-api · GitHub
[go: up one dir, main page]
Skip to content

Inconsistencies in handling of block quotes and pre-formatted code blocks #515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Bibo-Joshi opened this issue Jan 7, 2024 · 6 comments
Closed

Inconsistencies in handling of block quotes and pre-formatted code blocks #515

Bibo-Joshi opened this issue Jan 7, 2024 · 6 comments

Comments

@Bibo-Joshi
Copy link

Hi there. First of all, let me say that I'm exited for the API 7.0 release! It brings functionality that's been eagerly awaited from the community
The team of python-telegram-bot.org is currently working on integrating these changes into the python library. During this process, I noticed that the new block quote formatting options shows several inconsistencies in the handling of line breaks. The key points are

  • even though the resulting message may look the same in the messanger, the reported entities and the reported message text may differ depending on the input for the Bot API
  • cosecutive block quotes without separating blank line can not be achieved with MD V2.

The second point is unfortunate, but I can understand that HTML is in general just more flexible than MD. The first point may not seem to bad on first glance, because a user can still generate the rendered result they like. However, the inconsistencies make it rather difficult to work with updating messages, parsing & combining content of several messages and similar use cases. As notable example, python-telegram-bot provides utility functionality that tages a Message object and computes text including formatting markers in the desired markup language, with the idea being that

await bot.send_message(chat_id=chat_id, text=message.text_html/markdown_v2, parse_mode=ParseMode.HTM/MARKDOWN_V2)

produces the same rendered result as visible for message itself. This output can then be used to process the message further within Telegram or e.g. render the message in external programs. The observed inconsistencies make the implementation hard for us (see also discussion in python-telegram-bot/python-telegram-bot#4038)

Let me describe the observed inconsistencies in code. The reference implementation is written for python-telegram-bot version 20.7, but ofc the same results can be achieved with plain HTTP requests.

import asyncio

from telegram import Bot, Message
from telegram.constants import ParseMode


async def main():
    data: list[list[tuple[str, ParseMode]]] = [
        # These produce the same rendered result but different entities
        [
            (">A\nB", ParseMode.MARKDOWN_V2),
            ("<blockquote>A</blockquote>\nB", ParseMode.HTML),
        ],
        # These produce the same rendered result but different entities
        # The first HTML version also produces a different reported message text
        [
            ("ABC\n>DEF\nGHI", ParseMode.MARKDOWN_V2),
            ("ABC<blockquote>DEF</blockquote>GHI", ParseMode.HTML),
            ("ABC\n<blockquote>DEF</blockquote>\nGHI", ParseMode.HTML),
        ],
        # The Markdown V2 results are different from each other and from the HTML result
        # In particular, the HTML result can not be achieved with Markdown V2
        # The HTML versions produce the same rendered result but different entities
        # The reported message text coincides only for the first and third case
        [
            (">ABC\n>DEF\n>GHI", ParseMode.MARKDOWN_V2),
            (">ABC\n\n>DEF\n\n>GHI", ParseMode.MARKDOWN_V2),
            (
                "<blockquote>ABC</blockquote>\n<blockquote>DEF</blockquote>\n<blockquote>GHI"
                "</blockquote>",
                ParseMode.HTML,
            ),
            (
                "<blockquote>ABC</blockquote><blockquote>DEF</blockquote><blockquote>GHI"
                "</blockquote>",
                ParseMode.HTML,
            ),
        ],
    ]

    async with Bot("TOKEN") as bot:
        for case in data:
            messages: list[Message] = []
            for text, pars_mode in case:
                messages.append(
                    await bot.send_message(
                        chat_id=chat_id,
                        text=text,
                        parse_mode=pars_mode,
                    )
                )

            for message in messages:
                print(repr(message.text))
            for message in messages:
                print(message.entities)

            await bot.send_message(
                chat_id=chat_id,
                text=20 * "-",
            )
            print(20 * "-")


if __name__ == "__main__":
    asyncio.run(main())

Output:

'A\nB'
'A\nB'
(MessageEntity(length=2, offset=0, type=<MessageEntityType.BLOCKQUOTE>),)
(MessageEntity(length=1, offset=0, type=<MessageEntityType.BLOCKQUOTE>),)
--------------------
'ABC\nDEF\nGHI'
'ABCDEFGHI'
'ABC\nDEF\nGHI'
(MessageEntity(length=4, offset=4, type=<MessageEntityType.BLOCKQUOTE>),)
(MessageEntity(length=3, offset=3, type=<MessageEntityType.BLOCKQUOTE>),)
(MessageEntity(length=3, offset=4, type=<MessageEntityType.BLOCKQUOTE>),)
--------------------
'ABC\nDEF\nGHI'
'ABC\n\nDEF\n\nGHI'
'ABC\nDEF\nGHI'
'ABCDEFGHI'
(MessageEntity(length=11, offset=0, type=<MessageEntityType.BLOCKQUOTE>),)
(MessageEntity(length=4, offset=0, type=<MessageEntityType.BLOCKQUOTE>), MessageEntity(length=4, offset=5, type=<MessageEntityType.BLOCKQUOTE>), MessageEntity(length=3, offset=10, type=<MessageEntityType.BLOCKQUOTE>))
(MessageEntity(length=3, offset=0, type=<MessageEntityType.BLOCKQUOTE>), MessageEntity(length=3, offset=4, type=<MessageEntityType.BLOCKQUOTE>), MessageEntity(length=3, offset=8, type=<MessageEntityType.BLOCKQUOTE>))
(MessageEntity(length=3, offset=0, type=<MessageEntityType.BLOCKQUOTE>), MessageEntity(length=3, offset=3, type=<MessageEntityType.BLOCKQUOTE>), MessageEntity(length=3, offset=6, type=<MessageEntityType.BLOCKQUOTE>))
--------------------

Screenshot from Telegram Desktop Windows (Version 4.14.3 x64)

image

While investigating these inconsistencies, I became aware that some of them in fact already apply for the pre-formatted code blocks, which so far we just hadn't noticed. Let me demonstrate also this.

In the above reference implementation, replace the data with

data: list[list[tuple[str, ParseMode]]] = [
        # These produce the same rendered result and the same entities
        [
            ("```A```\nB", ParseMode.MARKDOWN_V2),
            ("```\nA```\nB", ParseMode.MARKDOWN_V2),
            ("<pre>A</pre>\nB", ParseMode.HTML),
        ],
        # These produce the same rendered result but the second version produces different entities
        # and different reported message text
        [
            ("ABC\n```DEF```\nGHI", ParseMode.MARKDOWN_V2),
            ("ABC<pre>DEF</pre>GHI", ParseMode.HTML),
            ("ABC\n<pre>DEF</pre>\nGHI", ParseMode.HTML),
        ],
        # These produce the same rendered result, but the reported entities and message text depend
        # on whether the newlines characters are present
        [
            ("```ABC```\n```DEF```\n```GHI```", ParseMode.MARKDOWN_V2),
            ("```ABC``````DEF``````GHI```", ParseMode.MARKDOWN_V2),
            (
                "<pre>ABC</pre>\n<pre>DEF</pre>\n<pre>GHI</pre>",
                ParseMode.HTML,
            ),
            (
                "<pre>ABC</pre><pre>DEF</pre><pre>GHI</pre>",
                ParseMode.HTML,
            ),
        ],
    ]

Output:

'A\nB'
'A\nB'
'A\nB'
(MessageEntity(length=1, offset=0, type=<MessageEntityType.PRE>),)
(MessageEntity(length=1, offset=0, type=<MessageEntityType.PRE>),)
(MessageEntity(length=1, offset=0, type=<MessageEntityType.PRE>),)
--------------------
'ABC\nDEF\nGHI'
'ABCDEFGHI'
'ABC\nDEF\nGHI'
(MessageEntity(length=3, offset=4, type=<MessageEntityType.PRE>),)
(MessageEntity(length=3, offset=3, type=<MessageEntityType.PRE>),)
(MessageEntity(length=3, offset=4, type=<MessageEntityType.PRE>),)
--------------------
'ABC\nDEF\nGHI'
'ABCDEFGHI'
'ABC\nDEF\nGHI'
'ABCDEFGHI'
(MessageEntity(length=3, offset=0, type=<MessageEntityType.PRE>), MessageEntity(length=3, offset=4, type=<MessageEntityType.PRE>), MessageEntity(length=3, offset=8, type=<MessageEntityType.PRE>))
(MessageEntity(length=3, offset=0, type=<MessageEntityType.PRE>), MessageEntity(length=3, offset=3, type=<MessageEntityType.PRE>), MessageEntity(length=3, offset=6, type=<MessageEntityType.PRE>))
(MessageEntity(length=3, offset=0, type=<MessageEntityType.PRE>), MessageEntity(length=3, offset=4, type=<MessageEntityType.PRE>), MessageEntity(length=3, offset=8, type=<MessageEntityType.PRE>))
(MessageEntity(length=3, offset=0, type=<MessageEntityType.PRE>), MessageEntity(length=3, offset=3, type=<MessageEntityType.PRE>), MessageEntity(length=3, offset=6, type=<MessageEntityType.PRE>))
--------------------

Screenshot from Telegram Desktop Windows (Version 4.14.3 x64)

image

I'm aware that the formatting functionality can be viewed as "working as expected" and one can argue that parsing problems of the entities should be handled by any Bot API wrapper itself. Still, I want to point out these discrepancies to you and emphasize that IMO it consistent parsing (same rendered results have same entities & text) would improve the usability of the Bot API .
@Bibo-Joshi Bibo-Joshi mentioned this issue Jan 7, 2024
Merged
10 tasks
@Bibo-Joshi Bibo-Joshi changed the title Inconsistencies in handling of block quotes and c Inconsistencies in handling of block quotes and pre-formatted code blocks Jan 7, 2024
@levlam
Copy link
Contributor
levlam commented Jan 8, 2024 •
edited
Loading

This output can then be used to process the message further within Telegram or e.g. render the message in external programs.

It is very unlikely that you can use Markdown output in external programs. For internal usages in most cases it is much better to manually specify text entities instead of trying to construct corresponding Markdown/HTML markup. This is supported for more than 3 years now and should be used instead of text_markdown/text_html.

# These produce the same rendered result but different entities

Blockquotes and pre-formatted blocks should start on a new line and end before a new line. If they aren't then apps will still show them as if they are, but it is up to the app, how this is achieved. You may see a different number of empty lines in different places in different apps in the latter case.

@Bibo-Joshi
Copy link
Author
Bibo-Joshi commented Jan 8, 2024 •
edited
Loading

Thanks for the swift reply.

It is very unlikely that you can use Markdown output in external programs.

Copy-pasting the code snippets from https://core.telegram.org/bots/api#formatting-options to both GitHub and StackOverflow, I can see that two widely-used interpreters can display most parts of TGs formatting options correctly without a need for changes.
In fact, the need to adapt/leave out some of TGs formatting options for processing in external programs actually highlights the use cases of methods like text_md/html. If you want to display TG messages in reStructuredText, Ascii-doc, or other formats, you'll have to translate the entities into markup symols or at least in other entities-like datastructures that the external program can reliably understand.

For internal usages in most cases it is much better to manually specify text entities instead of trying to construct corresponding Markdown/HTML markup. This is supported for more than 3 years now and should be used instead of text_markdown/text_html.

I see a "in most cases" there 😉 A use case that I have in mind is adding a prefix-text to an existing message, where both the prefix-text and the message contain formatting entities. Having to shift the entities in the messages by a computed offset and calculating offset+length for the prefix-text, all in utf-16, is far more implementation effort than simply writing html/md_formatted_prefix + message.text_markdown/html.
Nevertheless, if this is TGs official standpoint, we'll have to evaluate how much maintenance overhead text_html/markdown is worth for us.

Markdown

bold *text
italic *text
underline
strikethrough
||spoiler||
bold italic bold italic bold strikethrough ||italic bold strikethrough spoiler|| underline italic bold bold
inline URL
inline mention of a user
👍
inline fixed-width code

pre-formatted fixed-width code block

pre-formatted fixed-width code block written in the Python programming language

Block quotation started
Block quotation continued
The last line of the block quotation

HTML

bold, bold
italic, italic
underline, underline
strikethrough, strikethrough, strikethrough
spoiler, spoiler
bold italic bold italic bold strikethrough italic bold strikethrough spoiler underline italic bold bold
inline URL
inline mention of a user
👍
inline fixed-width code

pre-formatted fixed-width code block
pre-formatted fixed-width code block written in the Python programming language
Block quotation started\nBlock quotation continued\nThe last line of the block quotation

Same on SO

image

Blockquotes and pre-formatted blocks should start on a new line and end before a new line. If they aren't then apps will still show them as if they are, but it is up to the app, how this is achieved.

I would be very glad if this was documented at https://core.telegram.org/bots/api#formatting-options. This would alleviate at least some of the inconsistencies. Frankly, questions on the Bot API arising from shifting responsibility to the client without having that documented in the API is establishing itself as a pattern (See also #429 and #428) :/

@levlam
Copy link
Contributor
levlam commented Jan 9, 2024

can display most parts of TGs formatting options correctly without a need for changes.

But definitely not all of them. There will be multiple issues, for example, with bold/italic, blockquotes, or character escaping. Not even mentioning that classic Markdown and HTML are space-agnostic and Bot API isn't.

A use case that I have in mind is adding a prefix-text to an existing message

This can be easily done by shifting entities and library can provide a helper for that, which receives two text with entities and returns their concatenation. Implementation of such function is much simpler than trying to revert Markdown formatting.

I would be very glad if this was documented at https://core.telegram.org/bots/api#formatting-options.

This isn't a strong requirement, otherwise, it would be checked server-side. I also can't guarantee that non-official apps will correctly move the blocks to a new paragraph as intended.

@Bibo-Joshi
Copy link
Author

But definitely not all of them. There will be multiple issues

I agree that displaying tg messages in external programs can not reliably work without the need for some adaption. But as explained above, this makes text_md/html more valuable, not less valuable.

library can provide a helper for that, which receives two text with entities and returns their concatenation.

This still leaves the user with having to construct entity objects for the prefix and calling a method with 4 arguments, while string concatenation is a more simple operation.

This isn't a strong requirement, otherwise, it would be checked server-side. I also can't guarantee that non-official apps will correctly move the blocks to a new paragraph as intended.

If behavior of non-official clients is expected to have problems with block entities that do not start/end on a new line, wouldn't this be even more reason to document this, at least as a guideline?

In any case, it become clear that you don't see a need to improve the inconsistencies and we'll have to live with that.

@levlam
Copy link
Contributor
levlam commented Jan 10, 2024

In any case, it become clear that you don't see a need to improve the inconsistencies and we'll have to live with that.

There is no way to improve the way the resulting message may look, because this depend on the user's app. But consecutive block quotes without separating blank line could be supported in Bot API.

@levlam
Copy link
Contributor
levlam commented Feb 26, 2024

The ability to create consequent quotes using MarkdownV2 was added in Bot API 7.1 using a zero-length entity or separators between the entities. The corresponding examples were added to the documentation.

@levlam levlam closed this as completed Feb 26, 2024
@Bibo-Joshi Bibo-Joshi mentioned this issue Jul 12, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Bibo-Joshi @levlam
0