moment/src/backend/utils.py

173 lines
4.8 KiB
Python
Raw Normal View History

2019-12-19 22:46:16 +11:00
# SPDX-License-Identifier: LGPL-3.0-or-later
2019-12-19 00:42:56 +11:00
"""Various utilities that are used throughout the package."""
2019-11-12 00:26:35 +11:00
import collections
import html
import inspect
import io
import json
Big performance refactoring & various improvements Instead of passing all sorts of events for the JS to handle and manually add to different data models, we now handle everything we can in Python. For any change, the python models send a sync event with their contents (no more than 4 times per second) to JS, and the QSyncable library's JsonListModel takes care of converting it to a QML ListModel and sending the appropriate signals. The SortFilterProxyModel library is not used anymore, the only case where we need to filter/sort something now is when the user interacts with the "Filter rooms" or "Filter members" fields. These cases are handled by a simple JS function. We now keep separated room and timeline models for different accounts, the previous approach of sharing all the data we could between accounts created a lot of complications (local echoes, decrypted messages replacing others, etc). The users's own account profile changes are now hidden in the timeline. On startup, if all events for a room were only own profile changes, more events will be loaded. Any kind of image format supported by Qt is now handled by the pyotherside image provider, instead of just PNG/JPG. SVGs which previously caused errors are supported as well. The typing members bar paddings/margins are fixed. The behavior of the avatar/"upload a profile picture" overlay is fixed. Config files read from disk are now cached (TODO: make them reloadable again). Pylint is not used anymore because of all its annoying false warnings and lack of understanding for dataclasses, it is replaced by flake8 with a custom config and various plugins. Debug mode is now considered on if the program was compiled with the right option, instead of taking an argument from CLI. When on, C++ will set a flag in the Window QML component. The loading screen is now unloaded after the UI is ready, where previously it just stayed in the background invisible and wasted CPU. The overall refactoring and improvements make us now able to handle rooms with thousand of members and no lazy-loading, where previously everything would freeze and simply scrolling up to load past events in any room would block the UI for a few seconds.
2019-08-11 22:01:22 +10:00
import xml.etree.cElementTree as xml_etree # FIXME: bandit warning
2019-12-06 01:00:23 +11:00
from datetime import timedelta
from enum import Enum
Big performance refactoring & various improvements Instead of passing all sorts of events for the JS to handle and manually add to different data models, we now handle everything we can in Python. For any change, the python models send a sync event with their contents (no more than 4 times per second) to JS, and the QSyncable library's JsonListModel takes care of converting it to a QML ListModel and sending the appropriate signals. The SortFilterProxyModel library is not used anymore, the only case where we need to filter/sort something now is when the user interacts with the "Filter rooms" or "Filter members" fields. These cases are handled by a simple JS function. We now keep separated room and timeline models for different accounts, the previous approach of sharing all the data we could between accounts created a lot of complications (local echoes, decrypted messages replacing others, etc). The users's own account profile changes are now hidden in the timeline. On startup, if all events for a room were only own profile changes, more events will be loaded. Any kind of image format supported by Qt is now handled by the pyotherside image provider, instead of just PNG/JPG. SVGs which previously caused errors are supported as well. The typing members bar paddings/margins are fixed. The behavior of the avatar/"upload a profile picture" overlay is fixed. Config files read from disk are now cached (TODO: make them reloadable again). Pylint is not used anymore because of all its annoying false warnings and lack of understanding for dataclasses, it is replaced by flake8 with a custom config and various plugins. Debug mode is now considered on if the program was compiled with the right option, instead of taking an argument from CLI. When on, C++ will set a flag in the Window QML component. The loading screen is now unloaded after the UI is ready, where previously it just stayed in the background invisible and wasted CPU. The overall refactoring and improvements make us now able to handle rooms with thousand of members and no lazy-loading, where previously everything would freeze and simply scrolling up to load past events in any room would block the UI for a few seconds.
2019-08-11 22:01:22 +10:00
from enum import auto as autostr
from pathlib import Path
from types import ModuleType
from typing import Any, Dict, Tuple, Type
from uuid import UUID
Big performance refactoring & various improvements Instead of passing all sorts of events for the JS to handle and manually add to different data models, we now handle everything we can in Python. For any change, the python models send a sync event with their contents (no more than 4 times per second) to JS, and the QSyncable library's JsonListModel takes care of converting it to a QML ListModel and sending the appropriate signals. The SortFilterProxyModel library is not used anymore, the only case where we need to filter/sort something now is when the user interacts with the "Filter rooms" or "Filter members" fields. These cases are handled by a simple JS function. We now keep separated room and timeline models for different accounts, the previous approach of sharing all the data we could between accounts created a lot of complications (local echoes, decrypted messages replacing others, etc). The users's own account profile changes are now hidden in the timeline. On startup, if all events for a room were only own profile changes, more events will be loaded. Any kind of image format supported by Qt is now handled by the pyotherside image provider, instead of just PNG/JPG. SVGs which previously caused errors are supported as well. The typing members bar paddings/margins are fixed. The behavior of the avatar/"upload a profile picture" overlay is fixed. Config files read from disk are now cached (TODO: make them reloadable again). Pylint is not used anymore because of all its annoying false warnings and lack of understanding for dataclasses, it is replaced by flake8 with a custom config and various plugins. Debug mode is now considered on if the program was compiled with the right option, instead of taking an argument from CLI. When on, C++ will set a flag in the Window QML component. The loading screen is now unloaded after the UI is ready, where previously it just stayed in the background invisible and wasted CPU. The overall refactoring and improvements make us now able to handle rooms with thousand of members and no lazy-loading, where previously everything would freeze and simply scrolling up to load past events in any room would block the UI for a few seconds.
2019-08-11 22:01:22 +10:00
import filetype
from aiofiles.threadpool.binary import AsyncBufferedReader
from nio.crypto import AsyncDataT as File
from nio.crypto import async_generator_from_data
Big performance refactoring & various improvements Instead of passing all sorts of events for the JS to handle and manually add to different data models, we now handle everything we can in Python. For any change, the python models send a sync event with their contents (no more than 4 times per second) to JS, and the QSyncable library's JsonListModel takes care of converting it to a QML ListModel and sending the appropriate signals. The SortFilterProxyModel library is not used anymore, the only case where we need to filter/sort something now is when the user interacts with the "Filter rooms" or "Filter members" fields. These cases are handled by a simple JS function. We now keep separated room and timeline models for different accounts, the previous approach of sharing all the data we could between accounts created a lot of complications (local echoes, decrypted messages replacing others, etc). The users's own account profile changes are now hidden in the timeline. On startup, if all events for a room were only own profile changes, more events will be loaded. Any kind of image format supported by Qt is now handled by the pyotherside image provider, instead of just PNG/JPG. SVGs which previously caused errors are supported as well. The typing members bar paddings/margins are fixed. The behavior of the avatar/"upload a profile picture" overlay is fixed. Config files read from disk are now cached (TODO: make them reloadable again). Pylint is not used anymore because of all its annoying false warnings and lack of understanding for dataclasses, it is replaced by flake8 with a custom config and various plugins. Debug mode is now considered on if the program was compiled with the right option, instead of taking an argument from CLI. When on, C++ will set a flag in the Window QML component. The loading screen is now unloaded after the UI is ready, where previously it just stayed in the background invisible and wasted CPU. The overall refactoring and improvements make us now able to handle rooms with thousand of members and no lazy-loading, where previously everything would freeze and simply scrolling up to load past events in any room would block the UI for a few seconds.
2019-08-11 22:01:22 +10:00
2019-11-13 00:26:43 +11:00
Size = Tuple[int, int]
Big performance refactoring & various improvements Instead of passing all sorts of events for the JS to handle and manually add to different data models, we now handle everything we can in Python. For any change, the python models send a sync event with their contents (no more than 4 times per second) to JS, and the QSyncable library's JsonListModel takes care of converting it to a QML ListModel and sending the appropriate signals. The SortFilterProxyModel library is not used anymore, the only case where we need to filter/sort something now is when the user interacts with the "Filter rooms" or "Filter members" fields. These cases are handled by a simple JS function. We now keep separated room and timeline models for different accounts, the previous approach of sharing all the data we could between accounts created a lot of complications (local echoes, decrypted messages replacing others, etc). The users's own account profile changes are now hidden in the timeline. On startup, if all events for a room were only own profile changes, more events will be loaded. Any kind of image format supported by Qt is now handled by the pyotherside image provider, instead of just PNG/JPG. SVGs which previously caused errors are supported as well. The typing members bar paddings/margins are fixed. The behavior of the avatar/"upload a profile picture" overlay is fixed. Config files read from disk are now cached (TODO: make them reloadable again). Pylint is not used anymore because of all its annoying false warnings and lack of understanding for dataclasses, it is replaced by flake8 with a custom config and various plugins. Debug mode is now considered on if the program was compiled with the right option, instead of taking an argument from CLI. When on, C++ will set a flag in the Window QML component. The loading screen is now unloaded after the UI is ready, where previously it just stayed in the background invisible and wasted CPU. The overall refactoring and improvements make us now able to handle rooms with thousand of members and no lazy-loading, where previously everything would freeze and simply scrolling up to load past events in any room would block the UI for a few seconds.
2019-08-11 22:01:22 +10:00
auto = autostr
class AutoStrEnum(Enum):
2019-11-12 00:26:35 +11:00
"""An Enum where auto() assigns the member's name instead of an int.
Example:
>>> class Fruits(AutoStrEnum): apple = auto()
>>> Fruits.apple.value
"apple"
"""
2019-12-19 00:42:56 +11:00
@staticmethod
def _generate_next_value_(name, *_):
return name
2019-11-12 00:26:35 +11:00
def dict_update_recursive(dict1: dict, dict2: dict) -> None:
2019-12-19 00:42:56 +11:00
"""Deep-merge `dict1` and `dict2`, recursive version of `dict.update()`."""
# https://gist.github.com/angstwad/bf22d1822c38a92ec0a9
2019-11-12 00:26:35 +11:00
for k in dict2:
if (k in dict1 and isinstance(dict1[k], dict) and
isinstance(dict2[k], collections.Mapping)):
dict_update_recursive(dict1[k], dict2[k])
else:
dict1[k] = dict2[k]
Big performance refactoring & various improvements Instead of passing all sorts of events for the JS to handle and manually add to different data models, we now handle everything we can in Python. For any change, the python models send a sync event with their contents (no more than 4 times per second) to JS, and the QSyncable library's JsonListModel takes care of converting it to a QML ListModel and sending the appropriate signals. The SortFilterProxyModel library is not used anymore, the only case where we need to filter/sort something now is when the user interacts with the "Filter rooms" or "Filter members" fields. These cases are handled by a simple JS function. We now keep separated room and timeline models for different accounts, the previous approach of sharing all the data we could between accounts created a lot of complications (local echoes, decrypted messages replacing others, etc). The users's own account profile changes are now hidden in the timeline. On startup, if all events for a room were only own profile changes, more events will be loaded. Any kind of image format supported by Qt is now handled by the pyotherside image provider, instead of just PNG/JPG. SVGs which previously caused errors are supported as well. The typing members bar paddings/margins are fixed. The behavior of the avatar/"upload a profile picture" overlay is fixed. Config files read from disk are now cached (TODO: make them reloadable again). Pylint is not used anymore because of all its annoying false warnings and lack of understanding for dataclasses, it is replaced by flake8 with a custom config and various plugins. Debug mode is now considered on if the program was compiled with the right option, instead of taking an argument from CLI. When on, C++ will set a flag in the Window QML component. The loading screen is now unloaded after the UI is ready, where previously it just stayed in the background invisible and wasted CPU. The overall refactoring and improvements make us now able to handle rooms with thousand of members and no lazy-loading, where previously everything would freeze and simply scrolling up to load past events in any room would block the UI for a few seconds.
2019-08-11 22:01:22 +10:00
async def is_svg(file: File) -> bool:
2019-12-19 00:42:56 +11:00
"""Return whether the file is a SVG (`lxml` is used for detection)."""
2019-11-12 00:26:35 +11:00
chunks = [c async for c in async_generator_from_data(file)]
with io.BytesIO(b"".join(chunks)) as file:
try:
_, element = next(xml_etree.iterparse(file, ("start",)))
return element.tag == "{http://www.w3.org/2000/svg}svg"
except (StopIteration, xml_etree.ParseError):
return False
Big performance refactoring & various improvements Instead of passing all sorts of events for the JS to handle and manually add to different data models, we now handle everything we can in Python. For any change, the python models send a sync event with their contents (no more than 4 times per second) to JS, and the QSyncable library's JsonListModel takes care of converting it to a QML ListModel and sending the appropriate signals. The SortFilterProxyModel library is not used anymore, the only case where we need to filter/sort something now is when the user interacts with the "Filter rooms" or "Filter members" fields. These cases are handled by a simple JS function. We now keep separated room and timeline models for different accounts, the previous approach of sharing all the data we could between accounts created a lot of complications (local echoes, decrypted messages replacing others, etc). The users's own account profile changes are now hidden in the timeline. On startup, if all events for a room were only own profile changes, more events will be loaded. Any kind of image format supported by Qt is now handled by the pyotherside image provider, instead of just PNG/JPG. SVGs which previously caused errors are supported as well. The typing members bar paddings/margins are fixed. The behavior of the avatar/"upload a profile picture" overlay is fixed. Config files read from disk are now cached (TODO: make them reloadable again). Pylint is not used anymore because of all its annoying false warnings and lack of understanding for dataclasses, it is replaced by flake8 with a custom config and various plugins. Debug mode is now considered on if the program was compiled with the right option, instead of taking an argument from CLI. When on, C++ will set a flag in the Window QML component. The loading screen is now unloaded after the UI is ready, where previously it just stayed in the background invisible and wasted CPU. The overall refactoring and improvements make us now able to handle rooms with thousand of members and no lazy-loading, where previously everything would freeze and simply scrolling up to load past events in any room would block the UI for a few seconds.
2019-08-11 22:01:22 +10:00
async def svg_dimensions(file: File) -> Size:
2019-12-19 00:42:56 +11:00
"""Return the width and height, or viewBox width and height for a SVG.
2019-11-12 00:26:35 +11:00
If these properties are missing (broken file), ``(256, 256)`` is returned.
"""
chunks = [c async for c in async_generator_from_data(file)]
with io.BytesIO(b"".join(chunks)) as file:
attrs = xml_etree.parse(file).getroot().attrib
try:
width = round(float(attrs.get("width", attrs["viewBox"].split()[3])))
except (KeyError, IndexError, ValueError, TypeError):
width = 256
try:
height = round(float(attrs.get("height", attrs["viewBox"].split()[4])))
except (KeyError, IndexError, ValueError, TypeError):
height = 256
return (width, height)
async def guess_mime(file: File) -> str:
2019-12-19 00:42:56 +11:00
"""Return the file's mimetype, or `application/octet-stream` if unknown."""
2019-11-12 00:26:35 +11:00
if isinstance(file, io.IOBase):
file.seek(0, 0)
elif isinstance(file, AsyncBufferedReader):
await file.seek(0, 0)
try:
first_chunk: bytes
async for first_chunk in async_generator_from_data(file):
break
else:
return "inode/x-empty" # empty file
# TODO: plaintext
mime = filetype.guess_mime(first_chunk)
return mime or (
"image/svg+xml" if await is_svg(file) else
"application/octet-stream"
)
finally:
if isinstance(file, io.IOBase):
file.seek(0, 0)
elif isinstance(file, AsyncBufferedReader):
await file.seek(0, 0)
def plain2html(text: str) -> str:
2019-12-19 00:42:56 +11:00
"""Convert `\\n` into `<br>` tags and `\\t` into four spaces."""
2019-11-12 00:26:35 +11:00
return html.escape(text)\
.replace("\n", "<br>")\
.replace("\t", "&nbsp;" * 4)
def serialize_value_for_qml(value: Any, json_lists: bool = False) -> Any:
2019-12-19 00:42:56 +11:00
"""Convert a value to make it easier to use from QML.
Returns:
2019-11-12 00:26:35 +11:00
2019-12-19 00:42:56 +11:00
- Return the member's actual value for `Enum` members
- A `file://...` string for `Path` objects
- Strings for `UUID` objects
- A number of milliseconds for `datetime.timedelta` objects
- The class `__name__` for class types.
- `ModelItem.serialized` for `ModelItem`s
2019-11-12 00:26:35 +11:00
"""
if json_lists and isinstance(value, list):
return json.dumps(value)
if hasattr(value, "serialized"):
return value.serialized
if hasattr(value, "__class__") and issubclass(value.__class__, Enum):
return value.value
if isinstance(value, Path):
return f"file://{value!s}"
if isinstance(value, UUID):
return str(value)
2019-12-06 01:00:23 +11:00
if isinstance(value, timedelta):
return value.total_seconds() * 1000
if inspect.isclass(value):
return value.__name__
return value
def classes_defined_in(module: ModuleType) -> Dict[str, Type]:
2019-12-19 00:42:56 +11:00
"""Return a `{name: class}` dict of all the classes a module defines."""
2019-11-12 00:26:35 +11:00
return {
m[0]: m[1] for m in inspect.getmembers(module, inspect.isclass)
if not m[0].startswith("_") and
m[1].__module__.startswith(module.__name__)
}