Big performance refactoring & various improvements

Instead of passing all sorts of events for the JS to handle and manually
add to different data models, we now handle everything we can in Python.

For any change, the python models send a sync event with their
contents (no more than 4 times per second) to JS, and the QSyncable
library's JsonListModel takes care of converting it to a QML ListModel
and sending the appropriate signals.

The SortFilterProxyModel library is not used anymore, the only case
where we need to filter/sort something now is when the user interacts
with the "Filter rooms" or "Filter members" fields. These cases are
handled by a simple JS function.

We now keep separated room and timeline models for different accounts,
the previous approach of sharing all the data we could between accounts
created a lot of complications (local echoes, decrypted messages
replacing others, etc).

The users's own account profile changes are now hidden in the timeline.
On startup, if all events for a room were only own profile changes, more
events will be loaded.

Any kind of image format supported by Qt is now handled by the
pyotherside image provider, instead of just PNG/JPG.
SVGs which previously caused errors are supported as well.

The typing members bar paddings/margins are fixed.

The behavior of the avatar/"upload a profile picture" overlay is fixed.

Config files read from disk are now cached (TODO: make them reloadable
again).

Pylint is not used anymore because of all its annoying false warnings
and lack of understanding for dataclasses, it is replaced by flake8 with
a custom config and various plugins.

Debug mode is now considered on if the program was compiled with
the right option, instead of taking an argument from CLI.
When on, C++ will set a flag in the Window QML component.

The loading screen is now unloaded after the UI is ready, where
previously it just stayed in the background invisible and wasted CPU.

The overall refactoring and improvements make us now able to handle
rooms with thousand of members and no lazy-loading, where previously
everything would freeze and simply scrolling up to load past events
in any room would block the UI for a few seconds.

.gitignore

@@ -13,3 +13,4 @@ dist
 .qmake.stash
 Makefile
 harmonyqml
+harmonyqml.pro.user

.gitmodules

@@ -1,3 +1,3 @@
-[submodule "submodules/SortFilterProxyModel"]
+[submodule "submodules/qsyncable"]
-	path = submodules/SortFilterProxyModel
+	path = submodules/qsyncable
-	url = https://github.com/oKcerG/SortFilterProxyModel
+	url = https://github.com/benlau/qsyncable

TODO.md

@@ -2,20 +2,26 @@
   - `QCoreApplication::setAttribute(Qt::AA_EnableHighDpiScaling)`
 
 - Refactoring
+  - Remove copyrights
+  - Remove clip props when possible
+  - `property list<thing>`
+  - See Loader.enabled and async
   - Use [Animators](https://doc.qt.io/qt-5/qml-qtquick-animator.html)
   - Sendbox
-  - Use .mjs modules
   - SignIn/RememberAccount screens
     - SignIn must be in a flickable
-  - Don't bake in size properties for components
   - Unfinished work in button-refactor branch
     - Button can get "hoverEnabled: false" to let HoverHandlers work
   - Room Sidepane
     - Hide when window too small
     - Also save/load its size
-  - When qml syntax highlighting supports string interpolation, use them
+  - When qml syntax highlighting supports ES6 string interpolation, use them
 
 - Fixes
+  - Terrible performance using `QT_QPA_PLATFORM=wayland-egl`, must use `xcb`
+  - Reloading config files (cache)
+  - Ignore @ when filtering members
+  - Tiny invisible scrollbar
   - Update state.json page when accepting an invite
   - Run import in thread and AsyncClient.olm functions, they block async loop
   - Handle import keys errors
@@ -28,10 +34,6 @@
   - Message position after daybreak delegate
   - Keyboard flicking against top/bottom edge
   - Don't strip user spacing in html
-  - Past events loading (limit 100) freezes the GUI - need to move upsert func
-    to a WorkerScript
-  - `MessageDelegate.qml:63: TypeError: 'reloadPreviousItem' not a function`
-  - Horrible performance for big rooms
   - [hr not working](https://bugreports.qt.io/browse/QTBUG-74342)
 
 - UI
@@ -40,6 +42,9 @@
       - Accept/cancel buttons
       - Transitions
 
+  - Combine events so they take less space
+    - After combining is implemented, no need to hide our own profile changes.
+  - Room last activity time in RoomDelegate
   - When starting a long task, e.g. importing keys, quitting the page,
     and coming back, show the buttons as still loading until operation is done
   - Make invite/left banners look better in column mode
@@ -59,12 +64,12 @@
       - Support \ escaping
   - Improve avatar tooltips position, add stuff to room tooltips (last msg?)
   - Accept drag and dropping a picture in account settings to set avatar
-  - When all the events loaded on beginning in a room are name/avatar changes,
-    no last event room text is displayed (use sync filter?)
 
   - Show something when connection is lost or 429s happen
   - "Rejoin" LeftBanner button if room is public
   - Daybreak color
+  - Conversation breaks: show time of first new msg after break instead of big
+    blank space
   - Replies
   - `pyotherside.atexit()`
   - Sidepane
@@ -82,7 +87,9 @@
   - Leave room
   - Forget room warning popup
   - Prevent using the SendBox if no permission (power levels)
-  - Spinner when loading past room events, images or clicking buttons
+    - Prevent using an alias if that user is not in the room or no permission
+  - Spinner when loading account, past room events, images or clicking buttons
+    - Show account page as loading until profile initially retrieved
   - Theming
     - Don't create additional lines in theme conversion (braces)
     - Recursively merge default and user theme
@@ -114,20 +121,17 @@
   - Links preview
 
 - Client improvements
+  - Image provider: on failed conversion, way to show a "broken image" thumb?
   - Config file format
-  - Set Qt.application.* stuff from C++
-  - [debug mode](https://docs.python.org/3/library/asyncio-dev.html)
   - Initial sync filter and lazy load, see weechat-matrix `_handle_login()`
     - See also `handle_response()`'s `keys_query` request
   - Direct chats category
-  - On sync, check messages API, if a limited sync timeline was received
   - Markdown: don't turn #things (no space) and `thing\n---` into title,
     disable `__` syntax for bold/italic
   - Push instead of replacing in stack view (remove getMemberFilter when done)
   - `<pre>` scrollbar on overflow
   - When inviting someone to direct chat, room is "Empty room" until accepted,
     it should be the peer's display name instead.
-  - See `Qt.callLater()` potential usages
   - Animate RoomEventDelegate DayBreak apparition
   - Room subtitle: show things like "*Image*" instead of blank, etc
 

harmonyqml.pro

@@ -5,7 +5,7 @@ DEFINES += QT_DEPRECATED_WARNINGS
 CONFIG += warn_off c++11 release
 dev {
     CONFIG -= warn_off release
-    CONFIG += debug
+    CONFIG += debug qml_debug declarative_debug
 }
 
 BUILD_DIR = build
@@ -24,7 +24,7 @@ TARGET = harmonyqml
 
 # Libraries includes
 
-include(submodules/SortFilterProxyModel/SortFilterProxyModel.pri)
+include(submodules/qsyncable/qsyncable.pri)
 
 
 # Custom functions

live-reload.sh

@@ -5,9 +5,14 @@
 
 # no_embedded (resources) is used to speed up the compilation
 
+export DISPLAY=${1:-:0}
+export QT_QPA_PLATFORM=xcb
+
+CFG='dev no_embedded'
+
 while true; do
     find src harmonyqml.pro -type f |
     entr -cdnr sh -c \
-        'qmake CONFIG+="dev no_embedded" && make && ./harmonyqml --debug'
+        "qmake harmonyqml.pro CONFIG+='$CFG' && make && ./harmonyqml"
     sleep 0.2
 done

src/main.cpp

@@ -12,9 +12,20 @@
 int main(int argc, char *argv[]) {
     QApplication app(argc, argv);
 
+    QApplication::setOrganizationName("harmonyqml");
+    QApplication::setApplicationName("harmonyqml");
+    QApplication::setApplicationDisplayName("HarmonyQML");
+    QApplication::setApplicationVersion("0.1.0");
+
     QQmlEngine engine;
     QQmlContext *objectContext = new QQmlContext(engine.rootContext());
 
+#ifdef QT_DEBUG
+    objectContext->setContextProperty("debugMode", true);
+#else
+    objectContext->setContextProperty("debugMode", false);
+#endif
+
     QQmlComponent component(
         &engine,
         QFileInfo::exists("qrc:/qml/Window.qml") ?

src/python/__init__.py

@@ -1,4 +1,4 @@
 # Copyright 2019 miruka
 # This file is part of harmonyqml, licensed under LGPLv3.
 
-from .app import APP
+from .app import APP  # noqa

src/python/__main__.py

@@ -1,2 +1,6 @@
 # Copyright 2019 miruka
 # This file is part of harmonyqml, licensed under LGPLv3.
+
+from .app import APP
+
+APP.test_run()

src/python/app.py

@@ -2,20 +2,20 @@
 # This file is part of harmonyqml, licensed under LGPLv3.
 
 import asyncio
+import logging as log
 import signal
 from concurrent.futures import Future
 from operator import attrgetter
-from pathlib import Path
 from threading import Thread
-from typing import Any, Coroutine, Dict, List, Optional, Sequence
+from typing import Coroutine, Sequence
 
 import uvloop
 from appdirs import AppDirs
 
-import pyotherside
+from . import __about__, pyotherside
+from .pyotherside_events import CoroutineDone
 
-from . import __about__
+log.getLogger().setLevel(log.INFO)
-from .events.app import CoroutineDone, ExitRequested
 
 
 class App:
@@ -30,15 +30,26 @@ class App:
         self.image_provider = ImageProvider(self)
         pyotherside.set_image_provider(self.image_provider.get)
 
-        self.loop        = asyncio.get_event_loop()
+        self.loop = asyncio.get_event_loop()
+
+        if not pyotherside.AVAILABLE:
+            self.set_debug(True, verbose=True)
+
         self.loop_thread = Thread(target=self._loop_starter)
         self.loop_thread.start()
 
 
-    def is_debug_on(self, cli_flags: Sequence[str] = ()) -> bool:
+    def set_debug(self, enable: bool, verbose: bool = False) -> None:
-        debug      = "-d" in cli_flags or "--debug" in cli_flags
+        if verbose:
-        self.debug = debug
+            log.getLogger().setLevel(log.DEBUG)
-        return debug
+
+        if enable:
+            log.info("Debug mode enabled.")
+            self.loop.set_debug(True)
+            self.debug = True
+        else:
+            self.loop.set_debug(False)
+            self.debug = False
 
 
     def _loop_starter(self) -> None:
@@ -53,11 +64,11 @@ class App:
 
     def _call_coro(self, coro: Coroutine, uuid: str) -> None:
         self.run_in_loop(coro).add_done_callback(
-            lambda future: CoroutineDone(uuid=uuid, result=future.result())
+            lambda future: CoroutineDone(uuid=uuid, result=future.result()),
         )
 
 
-    def call_backend_coro(self, name: str, uuid: str, args: Sequence[str] = ()
+    def call_backend_coro(self, name: str, uuid: str, args: Sequence[str] = (),
                          ) -> None:
         self._call_coro(attrgetter(name)(self.backend)(*args), uuid)
 
@@ -72,22 +83,31 @@ class App:
 
 
     def pdb(self, additional_data: Sequence = ()) -> None:
-        # pylint: disable=all
+        ad = additional_data  # noqa
-        ad = additional_data
+        rl = self.run_in_loop  # noqa
-        rl = self.run_in_loop
+        ba = self.backend  # noqa
-        ba = self.backend
+        mo = self.backend.models # noqa
         cl = self.backend.clients
-        tcl = lambda user: cl[f"@test_{user}:matrix.org"]
+        tcl = lambda user: cl[f"@test_{user}:matrix.org"]  # noqa
+
+        from .models.items import Account, Room, Member, Event, Device  # noqa
 
         import json
-        jd = lambda obj: print(json.dumps(obj, indent=4, ensure_ascii=False))
+        jd = lambda obj: print(  # noqa
+            json.dumps(obj, indent=4, ensure_ascii=False),
+        )
 
-        print("\n=> Run `socat readline tcp:127.0.0.1:4444` in a terminal "
+        log.info("\n=> Run `socat readline tcp:127.0.0.1:4444` in a terminal "
-              "to connect to pdb.")
+                 "to connect to pdb.")
         import remote_pdb
         remote_pdb.RemotePdb("127.0.0.1", 4444).set_trace()
 
 
+    def test_run(self) -> None:
+        self.call_backend_coro("load_settings", "")
+        self.call_backend_coro("load_saved_accounts", "")
+
+
 # Make CTRL-C work again
 signal.signal(signal.SIGINT, signal.SIG_DFL)
 

src/python/backend.py

@@ -2,15 +2,20 @@
 # This file is part of harmonyqml, licensed under LGPLv3.
 
 import asyncio
+import logging as log
 import random
-from typing import Dict, List, Optional, Set, Tuple
+from typing import DefaultDict, Dict, List, Optional, Tuple, Union
 
 import hsluv
 
+import nio
+
 from .app import App
-from .events import users
-from .html_filter import HTML_FILTER
 from .matrix_client import MatrixClient
+from .models.items import Account, Device, Event, Member, Room
+from .models.model_store import ModelStore
+
+ProfileResponse = Union[nio.ProfileGetResponse, nio.ProfileGetError]
 
 
 class Backend:
@@ -22,12 +27,19 @@ class Backend:
         self.ui_settings    = config_files.UISettings(self)
         self.ui_state       = config_files.UIState(self)
 
+        self.models = ModelStore(allowed_key_types={
+            Account,             # Logged-in accounts
+            (Device, str),       # Devices of user_id
+            (Room,   str),       # Rooms for user_id
+            (Member, str),       # Members in room_id
+            (Event,  str, str),  # Events for account user_id for room_id
+        })
+
         self.clients: Dict[str, MatrixClient] = {}
 
-        self.past_tokens:        Dict[str, str] =  {}    # {room_id: token}
+        self.profile_cache: Dict[str, nio.ProfileGetResponse] = {}
-        self.fully_loaded_rooms: Set[str]       = set()  # {room_id}
+        self.get_profile_locks: DefaultDict[str, asyncio.Lock] = \
-
+                DefaultDict(asyncio.Lock)  # {user_id: lock}
-        self.pending_profile_requests: Set[str] = set()
 
 
     def __repr__(self) -> str:
@@ -42,11 +54,11 @@ class Backend:
                            device_id:  Optional[str] = None,
                            homeserver: str = "https://matrix.org") -> str:
         client = MatrixClient(
-            backend=self, user=user, homeserver=homeserver, device_id=device_id
+            self, user=user, homeserver=homeserver, device_id=device_id,
         )
         await client.login(password)
-        self.clients[client.user_id] = client
+        self.clients[client.user_id]         = client
-        users.AccountUpdated(client.user_id)
+        self.models[Account][client.user_id] = Account(client.user_id)
         return client.user_id
 
 
@@ -55,13 +67,15 @@ class Backend:
                             token:      str,
                             device_id:  str,
                             homeserver: str = "https://matrix.org") -> None:
+
         client = MatrixClient(
             backend=self,
-            user=user_id, homeserver=homeserver, device_id=device_id
+            user=user_id, homeserver=homeserver, device_id=device_id,
         )
         await client.resume(user_id=user_id, token=token, device_id=device_id)
-        self.clients[client.user_id] = client
+
-        users.AccountUpdated(client.user_id)
+        self.clients[client.user_id]         = client
+        self.models[Account][client.user_id] = Account(client.user_id)
 
 
     async def load_saved_accounts(self) -> Tuple[str, ...]:
@@ -83,8 +97,8 @@ class Backend:
     async def logout_client(self, user_id: str) -> None:
         client = self.clients.pop(user_id, None)
         if client:
+            self.models[Account].pop(client.user_id, None)
             await client.logout()
-            users.AccountDeleted(user_id)
 
 
     async def logout_all_clients(self) -> None:
@@ -115,20 +129,26 @@ class Backend:
         return (settings, ui_state, theme)
 
 
-    async def request_user_update_event(self, user_id: str) -> None:
+    async def get_profile(self, user_id: str) -> ProfileResponse:
-        if not self.clients:
+        if user_id in self.profile_cache:
-            await self.wait_until_client_exists()
+            return self.profile_cache[user_id]
 
-        client = self.clients.get(
+        async with self.get_profile_locks[user_id]:
-            user_id,
+            if not self.clients:
-            random.choice(tuple(self.clients.values()))
+                await self.wait_until_client_exists()
-        )
-        await client.request_user_update_event(user_id)
 
+            client = self.clients.get(
+                user_id,
+                random.choice(tuple(self.clients.values())),
+            )
 
-    @staticmethod
+            response = await client.get_profile(user_id)
-    def inlinify(html: str) -> str:
+
-        return HTML_FILTER.filter_inline(html)
+            if isinstance(response, nio.ProfileGetError):
+                log.warning("%s: %s", user_id, response)
+
+            self.profile_cache[user_id] = response
+            return response
 
 
     @staticmethod

src/python/config_files.py

@@ -3,12 +3,14 @@
 
 import asyncio
 import json
+import logging as log
 from pathlib import Path
 from typing import Any, Dict
 
 import aiofiles
 from dataclasses import dataclass, field
 
+from . import pyotherside
 from .backend import Backend
 from .theme_parser import convert_to_qml
 from .utils import dict_update_recursive
@@ -23,9 +25,10 @@ class ConfigFile:
     backend:  Backend = field(repr=False)
     filename: str     = field()
 
+    _cached_read: str = field(default="", init=False, compare=False)
+
     @property
     def path(self) -> Path:
-        # pylint: disable=no-member
         return Path(self.backend.app.appdirs.user_config_dir) / self.filename
 
 
@@ -34,7 +37,13 @@ class ConfigFile:
 
 
     async def read(self):
-        return self.path.read_text()
+        if self._cached_read:
+            log.debug("Returning cached config %s", type(self).__name__)
+            return self._cached_read
+
+        log.debug("Reading config %s at %s", type(self).__name__, self.path)
+        self._cached_read = self.path.read_text()
+        return self._cached_read
 
 
     async def write(self, data) -> None:
@@ -76,7 +85,6 @@ class Accounts(JSONConfigFile):
 
 
     async def add(self, user_id: str) -> None:
-        # pylint: disable=no-member
         client = self.backend.clients[user_id]
 
         await self.write({
@@ -85,7 +93,7 @@ class Accounts(JSONConfigFile):
                 "homeserver": client.homeserver,
                 "token":      client.access_token,
                 "device_id":  client.device_id,
-            }
+            },
         })
 
 
@@ -119,14 +127,12 @@ class UIState(JSONConfigFile):
 
     @property
     def path(self) -> Path:
-        # pylint: disable=no-member
         return Path(self.backend.app.appdirs.user_data_dir) / self.filename
 
 
     async def default_data(self) -> JsonData:
         return {
             "collapseAccounts":    {},
-            "collapseCategories":  {},
             "page":                "Pages/Default.qml",
             "pageProperties":      {},
             "sidePaneManualWidth": None,
@@ -137,7 +143,6 @@ class UIState(JSONConfigFile):
 class Theme(ConfigFile):
     @property
     def path(self) -> Path:
-        # pylint: disable=no-member
         data_dir = Path(self.backend.app.appdirs.user_data_dir)
         return data_dir / "themes" / self.filename
 
@@ -148,7 +153,9 @@ class Theme(ConfigFile):
 
 
     async def read(self) -> str:
-        # pylint: disable=no-member
+        if not pyotherside.AVAILABLE:
+            return ""
+
         if self.backend.app.debug:
             return convert_to_qml(await self.default_data())
 

src/python/events/app.py

@@ -1,19 +0,0 @@
-# Copyright 2019 miruka
-# This file is part of harmonyqml, licensed under LGPLv3.
-
-from typing import Any
-
-from dataclasses import dataclass, field
-
-from .event import Event
-
-
-@dataclass
-class ExitRequested(Event):
-    exit_code: int = 0
-
-
-@dataclass
-class CoroutineDone(Event):
-    uuid:   str = field()
-    result: Any = None

src/python/events/event.py

@@ -1,29 +0,0 @@
-# Copyright 2019 miruka
-# This file is part of harmonyqml, licensed under LGPLv3.
-
-from enum import Enum
-from typing import Any
-
-from dataclasses import dataclass
-
-import pyotherside
-
-
-@dataclass
-class Event:
-    def __post_init__(self) -> None:
-        # CPython >= 3.6 or any Python >= 3.7 needed for correct dict order
-        args = [
-            # pylint: disable=no-member
-            self._process_field(getattr(self, field))
-            for field in self.__dataclass_fields__  # type: ignore
-        ]
-        pyotherside.send(type(self).__name__, *args)
-
-
-    @staticmethod
-    def _process_field(value: Any) -> Any:
-        if hasattr(value, "__class__") and issubclass(value.__class__, Enum):
-            return value.value
-
-        return value

src/python/events/rooms.py

@@ -1,108 +0,0 @@
-# Copyright 2019 miruka
-# This file is part of harmonyqml, licensed under LGPLv3.
-
-from datetime import datetime
-from typing import Any, Dict, List, Sequence, Type
-
-from dataclasses import dataclass, field
-
-import nio
-from nio.rooms import MatrixRoom
-
-from .event import Event
-
-
-@dataclass
-class RoomUpdated(Event):
-    user_id:        str                      = field()
-    category:       str                      = field()
-    room_id:        str                      = field()
-    display_name:   str                      = ""
-    avatar_url:     str                      = ""
-    topic:          str                      = ""
-    members:        Sequence[Dict[str, Any]] = ()
-    typing_members: Sequence[str]            = ()
-    inviter_id:     str                      = ""
-
-
-    @classmethod
-    def from_nio(cls,
-                 user_id:  str,
-                 category: str,
-                 room:     MatrixRoom,
-                 info:     nio.RoomInfo) -> "RoomUpdated":
-
-        typing: List[str] = []
-
-        if hasattr(info, "ephemeral"):
-            for ev in info.ephemeral:
-                if isinstance(ev, nio.TypingNoticeEvent):
-                    typing = ev.users
-
-        name = room.name or room.canonical_alias
-
-        if not name:
-            name = room.group_name()
-            name = "" if name == "Empty room?" else name
-
-        members = [{"userId": m.user_id, "powerLevel": m.power_level}
-                   for m in room.users.values()]
-
-        return cls(
-            user_id        = user_id,
-            category       = category,
-            room_id        = room.room_id,
-            display_name   = name,
-            avatar_url     = room.gen_avatar_url or "",
-            topic          = room.topic or "",
-            inviter_id     = getattr(room, "inviter", "") or "",
-            members        = members,
-            typing_members = typing,
-        )
-
-
-@dataclass
-class RoomForgotten(Event):
-    user_id:  str = field()
-    room_id:  str = field()
-
-
-@dataclass
-class RoomMemberUpdated(Event):
-    room_id: str  = field()
-    user_id: str  = field()
-    typing:  bool = field()
-
-
-@dataclass
-class RoomMemberDeleted(Event):
-    room_id: str  = field()
-    user_id: str  = field()
-
-
-# Timeline
-
-@dataclass
-class TimelineEventReceived(Event):
-    event_type:    Type[nio.Event] = field()
-    room_id:       str             = field()
-    event_id:      str             = field()
-    sender_id:     str             = field()
-    date:          datetime        = field()
-    content:       str             = field()
-    is_local_echo: bool            = False
-
-    target_user_id: str = ""
-
-    @classmethod
-    def from_nio(cls, room: MatrixRoom, ev: nio.Event, **fields
-                ) -> "TimelineEventReceived":
-        return cls(
-            event_type = type(ev),
-            room_id    = room.room_id,
-            event_id   = ev.event_id,
-            sender_id  = ev.sender,
-            date       = datetime.fromtimestamp(ev.server_timestamp / 1000),
-            target_user_id = getattr(ev, "state_key", "") or "",
-            **fields
-        )

src/python/events/users.py

@@ -1,64 +0,0 @@
-# Copyright 2019 miruka
-# This file is part of harmonyqml, licensed under LGPLv3.
-
-from datetime import datetime
-from enum import Enum
-
-from dataclasses import dataclass, field
-
-from nio.rooms import MatrixUser
-
-from .event import Event
-
-# Logged-in accounts
-
-@dataclass
-class AccountUpdated(Event):
-    user_id: str = field()
-
-
-@dataclass
-class AccountDeleted(Event):
-    user_id: str = field()
-
-
-# Accounts and room members details
-
-@dataclass
-class UserUpdated(Event):
-    user_id:      str  = field()
-    display_name: str  = ""
-    avatar_url:   str  = ""
-
-    @classmethod
-    def from_nio(cls, user: MatrixUser) -> "UserUpdated":
-        return cls(
-            user_id      = user.user_id,
-            display_name = user.display_name or "",
-            avatar_url   = user.avatar_url or "",
-        )
-
-
-# Devices
-
-class Trust(Enum):
-    blacklisted = -1
-    undecided   = 0
-    trusted     = 1
-
-
-@dataclass
-class DeviceUpdated(Event):
-    user_id:        str      = field()
-    device_id:      str      = field()
-    ed25519_key:    str      = field()
-    trust:          Trust    = Trust.undecided
-    display_name:   str      = ""
-    last_seen_ip:   str      = ""
-    last_seen_date: datetime = field(default_factory=lambda: datetime(1, 1, 1))
-
-
-@dataclass
-class DeviceDeleted(Event):
-    user_id:   str = field()
-    device_id: str = field()

src/python/html_filter.py

@@ -34,7 +34,7 @@ class HtmlFilter:
 
         # hard_wrap: convert all \n to <br> without required two spaces
         self._markdown_to_html = mistune.Markdown(
-            hard_wrap=True, renderer=MarkdownRenderer()
+            hard_wrap=True, renderer=MarkdownRenderer(),
         )
 
         self._markdown_to_html.block.default_rules = [
@@ -56,7 +56,7 @@ class HtmlFilter:
 
         if not outgoing:
             text = re.sub(
-                r"(^\s*&gt;.*)", r'<span class="greentext">\1</span>', text
+                r"(^\s*&gt;.*)", r'<span class="greentext">\1</span>', text,
             )
 
         return text
@@ -84,15 +84,15 @@ class HtmlFilter:
             text = re.sub(
                 r"<(p|br/?)>(\s*&gt;.*)(!?</?(?:br|p)/?>)",
                 r'<\1><span class="greentext">\2</span>\3',
-                text
+                text,
             )
 
         return text
 
 
     def sanitize_settings(self, inline: bool = False) -> dict:
-        # https://matrix.org/docs/spec/client_server/latest.html#m-room-message-msgtypes
+        # https://matrix.org/docs/spec/client_server/latest#m-room-message-msgtypes
-        # TODO: mx-reply, audio, video
+        # TODO: mx-reply, audio, video, the new hidden thing
 
         inline_tags = {"font", "a", "sup", "sub", "b", "i", "s", "u", "code"}
         tags        = inline_tags | {
@@ -103,8 +103,7 @@ class HtmlFilter:
         }
 
         inlines_attributes = {
-            # TODO: translate font attrs to qt html subset
+            "font": {"color"},
-            "font": {"data-mx-bg-color", "data-mx-color"},
             "a":    {"href"},
             "code": {"class"},
         }
@@ -119,9 +118,10 @@ class HtmlFilter:
             "attributes": inlines_attributes if inline else attributes,
             "empty": {} if inline else {"hr", "br", "img"},
             "separate": {"a"} if inline else {
-                "a", "p", "li", "table", "tr", "th", "td", "br", "hr"
+                "a", "p", "li", "table", "tr", "th", "td", "br", "hr",
             },
             "whitespace": {},
+            "keep_typographic_whitespace": True,
             "add_nofollow": False,
             "autolink": {
                 "link_regexes": self.link_regexes,
@@ -135,25 +135,26 @@ class HtmlFilter:
                 sanitizer.tag_replacer("em", "i"),
                 sanitizer.tag_replacer("strike", "s"),
                 sanitizer.tag_replacer("del", "s"),
-                sanitizer.tag_replacer("span", "font"),
-                self._remove_empty_font,
                 sanitizer.tag_replacer("form", "p"),
                 sanitizer.tag_replacer("div", "p"),
                 sanitizer.tag_replacer("caption", "p"),
                 sanitizer.target_blank_noopener,
+                self._process_span_font,
             ],
             "element_postprocessors": [],
             "is_mergeable": lambda e1, e2: e1.attrib == e2.attrib,
         }
 
 
-    def _remove_empty_font(self, el: HtmlElement) -> HtmlElement:
+    @staticmethod
-        if el.tag != "font":
+    def _process_span_font(el: HtmlElement) -> HtmlElement:
+        if el.tag not in ("span", "font"):
             return el
 
-        settings = self.sanitize_settings()
+        color = el.attrib.pop("data-mx-color", None)
-        if not settings["attributes"]["font"] & set(el.keys()):
+        if color:
-            el.clear()
+            el.tag = "font"
+            el.attrib["color"] = color
 
         return el
 
@@ -191,7 +192,7 @@ class HtmlFilter:
     @staticmethod
     def _is_image_path(link: str) -> bool:
         return bool(re.match(
-            r".+\.(jpg|jpeg|png|gif|bmp|webp|tiff|svg)$", link, re.IGNORECASE
+            r".+\.(jpg|jpeg|png|gif|bmp|webp|tiff|svg)$", link, re.IGNORECASE,
         ))
 
 

src/python/image_provider.py

@@ -2,30 +2,35 @@
 # This file is part of harmonyqml, licensed under LGPLv3.
 
 import asyncio
+import logging as log
 import random
 import re
+from dataclasses import dataclass, field
 from io import BytesIO
 from pathlib import Path
-from typing import Tuple
+from typing import Optional, Tuple
 from urllib.parse import urlparse
 
 import aiofiles
-from dataclasses import dataclass, field
 from PIL import Image as PILImage
 
 import nio
-import pyotherside
 from nio.api import ResizingMethod
 
-Size      = Tuple[int, int]
+from . import pyotherside, utils
-ImageData = Tuple[bytearray, Size, int]  # last int: pyotherside format enum
+from .pyotherside import ImageData, Size
+
+POSFormat = int
 
 CONCURRENT_DOWNLOADS_LIMIT = asyncio.BoundedSemaphore(8)
 
+with BytesIO() as img_out:
+    PILImage.new("RGBA", (1, 1), (0, 0, 0, 0)).save(img_out, "PNG")
+    TRANSPARENT_1X1_PNG = (img_out.getvalue(), pyotherside.format_data)
+
 
 @dataclass
 class Thumbnail:
-    # pylint: disable=no-member
     provider: "ImageProvider" = field()
     mxc:      str             = field()
     width:    int             = field()
@@ -70,7 +75,6 @@ class Thumbnail:
 
     @property
     def local_path(self) -> Path:
-        # pylint: disable=bad-string-format-type
         parsed = urlparse(self.mxc)
         name   = "%s.%03d.%03d.%s" % (
             parsed.path.lstrip("/"),
@@ -81,14 +85,41 @@ class Thumbnail:
         return self.provider.cache / parsed.netloc / name
 
 
-    async def download(self) -> bytes:
+    async def read_data(self, data: bytes, mime: Optional[str],
+                       ) -> Tuple[bytes, POSFormat]:
+        if mime == "image/svg+xml":
+            return (data, pyotherside.format_svg_data)
+
+        if mime in ("image/jpeg", "image/png"):
+            return (data, pyotherside.format_data)
+
+        try:
+            with BytesIO(data) as img_in:
+                image = PILImage.open(img_in)
+
+                if image.mode == "RGB":
+                    return (data, pyotherside.format_rgb888)
+
+                if image.mode == "RGBA":
+                    return (data, pyotherside.format_argb32)
+
+                with BytesIO() as img_out:
+                    image.save(img_out, "PNG")
+                    return (img_out.getvalue(), pyotherside.format_data)
+
+        except OSError as err:
+            log.warning("Unable to process image: %s - %r", self.http, err)
+            return TRANSPARENT_1X1_PNG
+
+
+    async def download(self) -> Tuple[bytes, POSFormat]:
         client = random.choice(
-            tuple(self.provider.app.backend.clients.values())
+            tuple(self.provider.app.backend.clients.values()),
         )
         parsed = urlparse(self.mxc)
 
         async with CONCURRENT_DOWNLOADS_LIMIT:
-            response = await client.thumbnail(
+            resp = await client.thumbnail(
                 server_name = parsed.netloc,
                 media_id    = parsed.path.lstrip("/"),
                 width       = self.server_size[0],
@@ -96,37 +127,37 @@ class Thumbnail:
                 method      = self.resize_method,
             )
 
-        if isinstance(response, nio.ThumbnailError):
+        if isinstance(resp, nio.ThumbnailError):
-            # Return a transparent 1x1 PNG
+            log.warning("Downloading thumbnail failed - %s", resp)
-            with BytesIO() as img_out:
+            return TRANSPARENT_1X1_PNG
-                PILImage.new("RGBA", (1, 1), (0, 0, 0, 0)).save(img_out, "PNG")
-                return img_out.getvalue()
 
-        body = response.body
+        body, pos_format = await self.read_data(resp.body, resp.content_type)
-
-        if response.content_type not in ("image/jpeg", "image/png"):
-            with BytesIO(body) as img_in, BytesIO() as img_out:
-                PILImage.open(img_in).save(img_out, "PNG")
-                body = img_out.getvalue()
 
         self.local_path.parent.mkdir(parents=True, exist_ok=True)
 
         async with aiofiles.open(self.local_path, "wb") as file:
-            await file.write(body)
+            # body might have been converted, always save the original image.
+            await file.write(resp.body)
 
-        return body
+        return (body, pos_format)
+
+
+    async def local_read(self) -> Tuple[bytes, POSFormat]:
+        data = self.local_path.read_bytes()
+        with BytesIO(data) as data_io:
+            return await self.read_data(data, utils.guess_mime(data_io))
 
 
     async def get_data(self) -> ImageData:
         try:
-            body = self.local_path.read_bytes()
+            data, pos_format = await self.local_read()
-        except FileNotFoundError:
+        except (OSError, IOError, FileNotFoundError):
-            body = await self.download()
+            data, pos_format = await self.download()
 
-        with BytesIO(body) as img_in:
+        with BytesIO(data) as img_in:
             real_size = PILImage.open(img_in).size
 
-        return (bytearray(body), real_size, pyotherside.format_data)
+        return (bytearray(data), real_size, pos_format)
 
 
 class ImageProvider:
@@ -141,7 +172,13 @@ class ImageProvider:
         if requested_size[0] < 1 or requested_size[1] < 1:
             raise ValueError(f"width or height < 1: {requested_size!r}")
 
+        try:
+            thumb = Thumbnail(self, image_id, *requested_size)
+        except ValueError as err:
+            log.warning(err)
+            data, pos_format = TRANSPARENT_1X1_PNG
+            return (bytearray(data), (1, 1), pos_format)
+
         return asyncio.run_coroutine_threadsafe(
-            Thumbnail(self, image_id, *requested_size).get_data(),
+            thumb.get_data(), self.app.loop,
-            self.app.loop
         ).result()

src/python/matrix_client.py

@@ -12,18 +12,15 @@ from datetime import datetime
 from enum import Enum
 from pathlib import Path
 from types import ModuleType
-from typing import DefaultDict, Dict, Optional, Type, Union
+from typing import DefaultDict, Dict, Optional, Set, Tuple, Type, Union
 from uuid import uuid4
 
-import filetype
-
 import nio
-from nio.rooms import MatrixRoom
 
-from . import __about__
+from . import __about__, utils
-from .events import rooms, users
-from .events.rooms import TimelineEventReceived
 from .html_filter import HTML_FILTER
+from .models.items import Account, Event, Member, Room
+from .models.model_store import ModelStore
 
 
 class UploadError(Enum):
@@ -38,19 +35,11 @@ class MatrixClient(nio.AsyncClient):
                  user:       str,
                  homeserver: str           = "https://matrix.org",
                  device_id:  Optional[str] = None) -> None:
-        # TODO: ensure homeserver starts with a scheme://
 
-        from .backend import Backend
+        store = Path(backend.app.appdirs.user_data_dir) / "encryption"
-        self.backend: Backend = backend
-
-        self.sync_task: Optional[asyncio.Future] = None
-
-        self.send_locks: DefaultDict[str, asyncio.Lock] = \
-                DefaultDict(asyncio.Lock)  # {room_id: lock}
-
-        store = Path(self.backend.app.appdirs.user_data_dir) / "encryption"
         store.mkdir(parents=True, exist_ok=True)
 
+        # TODO: ensure homeserver starts by a scheme://
         # TODO: pass a ClientConfig with a pickle key
         super().__init__(
             homeserver = homeserver,
@@ -59,12 +48,31 @@ class MatrixClient(nio.AsyncClient):
             store_path = store,
         )
 
+        from .backend import Backend
+        self.backend: Backend    = backend
+        self.models:  ModelStore = self.backend.models
+
+        self.sync_task:           Optional[asyncio.Future] = None
+        self.first_sync_happened: asyncio.Event            = asyncio.Event()
+
+        self.send_locks: DefaultDict[str, asyncio.Lock] = \
+                DefaultDict(asyncio.Lock)  # {room_id: lock}
+
+        self.past_tokens:        Dict[str, str] = {}     # {room_id: token}
+        self.fully_loaded_rooms: Set[str]       = set()  # {room_id}
+        self.loaded_once_rooms:  Set[str]       = set()  # {room_id}
+
+        self.local_echoes_uuid: Set[str]       = set()
+        self.resolved_echoes:   Dict[str, str] = {}  # {event_id: echo_uuid}
+
+        self.skipped_events: DefaultDict[str, int] = DefaultDict(lambda: 0)
+
         self.connect_callbacks()
 
 
     def __repr__(self) -> str:
         return "%s(user_id=%r, homeserver=%r, device_id=%r)" % (
-            type(self).__name__, self.user_id, self.homeserver, self.device_id
+            type(self).__name__, self.user_id, self.homeserver, self.device_id,
         )
 
 
@@ -86,17 +94,10 @@ class MatrixClient(nio.AsyncClient):
             with suppress(AttributeError):
                 self.add_event_callback(getattr(self, f"on{name}"), class_)
 
-
+        self.add_ephemeral_callback(
-    async def start_syncing(self) -> None:
+            self.onTypingNoticeEvent, nio.events.TypingNoticeEvent,
-        self.sync_task = asyncio.ensure_future(
-            self.sync_forever(timeout=10_000)
         )
 
-        def callback(task):
-            raise task.exception()
-
-        self.sync_task.add_done_callback(callback)
-
 
     @property
     def default_device_name(self) -> str:
@@ -107,19 +108,19 @@ class MatrixClient(nio.AsyncClient):
 
     async def login(self, password: str, device_name: str = "") -> None:
         response = await super().login(
-            password, device_name or self.default_device_name
+            password, device_name or self.default_device_name,
         )
 
         if isinstance(response, nio.LoginError):
-            print(response)
+            log.error(response)
         else:
-            await self.start_syncing()
+            await self.start()
 
 
     async def resume(self, user_id: str, token: str, device_id: str) -> None:
         response = nio.LoginResponse(user_id, device_id, token)
         await self.receive_response(response)
-        await self.start_syncing()
+        await self.start()
 
 
     async def logout(self) -> None:
@@ -131,27 +132,29 @@ class MatrixClient(nio.AsyncClient):
         await self.close()
 
 
-    async def request_user_update_event(self, user_id: str) -> None:
+    async def start(self) -> None:
-        if user_id in self.backend.pending_profile_requests:
+        def on_profile_response(future) -> None:
-            return
+            resp = future.result()
-        self.backend.pending_profile_requests.add(user_id)
+            if isinstance(resp, nio.ProfileGetResponse):
+                account                 = self.models[Account][self.user_id]
+                account.profile_updated = datetime.now()
+                account.display_name    = resp.displayname or ""
+                account.avatar_url      = resp.avatar_url or ""
 
-        response = await self.get_profile(user_id)
+        ft = asyncio.ensure_future(self.backend.get_profile(self.user_id))
+        ft.add_done_callback(on_profile_response)
 
-        if isinstance(response, nio.ProfileGetError):
+        def on_unexpected_sync_stop(future) -> None:
-            log.warning("%s: %s", user_id, response)
+            raise future.exception()
 
-        users.UserUpdated(
+        self.sync_task = asyncio.ensure_future(
-            user_id        = user_id,
+            self.sync_forever(timeout=10_000),
-            display_name   = getattr(response, "displayname", "") or "",
-            avatar_url     = getattr(response, "avatar_url", "") or "",
         )
-
+        self.sync_task.add_done_callback(on_unexpected_sync_stop)
-        self.backend.pending_profile_requests.discard(user_id)
 
 
     @property
-    def all_rooms(self) -> Dict[str, MatrixRoom]:
+    def all_rooms(self) -> Dict[str, nio.MatrixRoom]:
         return {**self.invited_rooms, **self.rooms}
 
 
@@ -162,36 +165,48 @@ class MatrixClient(nio.AsyncClient):
             text   = text[1:]
 
         if text.startswith("/me ") and not escape:
-            event_type = nio.RoomMessageEmote
+            event_type = nio.RoomMessageEmote.__name__
             text       = text[len("/me "): ]
             content    = {"body": text, "msgtype": "m.emote"}
             to_html    = HTML_FILTER.from_markdown_inline(text, outgoing=True)
-            echo_html  = HTML_FILTER.from_markdown_inline(text)
         else:
-            event_type = nio.RoomMessageText
+            event_type = nio.RoomMessageText.__name__
             content    = {"body": text, "msgtype": "m.text"}
             to_html    = HTML_FILTER.from_markdown(text, outgoing=True)
-            echo_html  = HTML_FILTER.from_markdown(text)
 
         if to_html not in (html.escape(text), f"<p>{html.escape(text)}</p>"):
             content["format"]         = "org.matrix.custom.html"
             content["formatted_body"] = to_html
 
-        TimelineEventReceived(
+        uuid = str(uuid4())
-            event_type    = event_type,
+        self.local_echoes_uuid.add(uuid)
-            room_id       = room_id,
+
-            event_id      = f"local_echo.{uuid4()}",
+        our_info        = self.models[Member, room_id][self.user_id]
+        display_content = content.get("formatted_body") or content["body"]
+
+        local = Event(
+            client_id      = f"echo-{uuid}",
+            event_id       = "",
+            event_type     = event_type,
+            date           = datetime.now(),
+            content        = display_content,
+            inline_content = HTML_FILTER.filter_inline(display_content),
+            is_local_echo  = True,
+
             sender_id     = self.user_id,
-            date          = datetime.now(),
+            sender_name   = our_info.display_name,
-            content       = echo_html,
+            sender_avatar = our_info.avatar_url,
-            is_local_echo = True,
         )
+        for user_id in self.models[Account]:
+            if user_id in self.models[Member, room_id]:
+                self.models[Event, user_id, room_id][f"echo-{uuid}"] = local
 
         async with self.send_locks[room_id]:
             response = await self.room_send(
                 room_id                   = room_id,
                 message_type              = "m.room.message",
                 content                   = content,
+                tx_id                     = uuid,
                 ignore_unverified_devices = True,
             )
 
@@ -199,44 +214,62 @@ class MatrixClient(nio.AsyncClient):
                 log.error("Failed to send message: %s", response)
 
 
-    async def load_past_events(self, room_id: str, limit: int = 25) -> bool:
+    async def load_past_events(self, room_id: str) -> bool:
-        if room_id in self.backend.fully_loaded_rooms:
+        if room_id in self.fully_loaded_rooms:
             return False
 
+        await self.first_sync_happened.wait()
+
         response = await self.room_messages(
             room_id = room_id,
-            start   = self.backend.past_tokens[room_id],
+            start   = self.past_tokens[room_id],
-            limit   = limit,
+            limit   = 100 if room_id in self.loaded_once_rooms else 25,
         )
 
+        self.loaded_once_rooms.add(room_id)
         more_to_load = True
 
-        if self.backend.past_tokens[room_id] == response.end:
+        if self.past_tokens[room_id] == response.end:
-            self.backend.fully_loaded_rooms.add(room_id)
+            self.fully_loaded_rooms.add(room_id)
             more_to_load = False
 
-        self.backend.past_tokens[room_id] = response.end
+        self.past_tokens[room_id] = response.end
 
         for event in response.chunk:
             for cb in self.event_callbacks:
                 if (cb.filter is None or isinstance(event, cb.filter)):
-                    await cb.func(
+                    await cb.func(self.all_rooms[room_id], event)
-                        self.all_rooms[room_id], event, from_past=True
-                    )
 
         return more_to_load
 
 
+    async def load_rooms_without_visible_events(self) -> None:
+        for room_id in self.models[Room, self.user_id]:
+            asyncio.ensure_future(
+                self._load_room_without_visible_events(room_id),
+            )
+
+
+    async def _load_room_without_visible_events(self, room_id: str) -> None:
+        events = self.models[Event, self.user_id, room_id]
+        more   = True
+
+        while self.skipped_events[room_id] and not events and more:
+            more = await self.load_past_events(room_id)
+
+
     async def room_forget(self, room_id: str) -> None:
         await super().room_forget(room_id)
-        rooms.RoomForgotten(user_id=self.user_id, room_id=room_id)
+        self.models[Room, self.user_id].pop(room_id, None)
+        self.models.pop([Event, self.user_id, room_id], None)
+        self.models.pop([Member, room_id], None)
 
 
     async def upload_file(self, path: Union[Path, str]) -> str:
         path = Path(path)
 
         with open(path, "rb") as file:
-            mime = filetype.guess_mime(file)
+            mime = utils.guess_mime(file)
             file.seek(0, 0)
 
             resp = await self.upload(file, mime, path.name)
@@ -253,7 +286,7 @@ class MatrixClient(nio.AsyncClient):
         return UploadError.unknown.value
 
 
-    async def set_avatar_from_file(self, path: Union[Path, str]
+    async def set_avatar_from_file(self, path: Union[Path, str],
                                   ) -> Union[bool, str]:
         resp = await self.upload_file(path)
 
@@ -264,30 +297,153 @@ class MatrixClient(nio.AsyncClient):
         return True
 
 
+    # Functions to register data into models
+
+    async def set_room_last_event(self, room_id: str, item: Event) -> None:
+        room = self.models[Room, self.user_id][room_id]
+
+        if room.last_event is None:
+            room.last_event = item.__dict__
+            return
+
+        for_us       = item.target_id in self.backend.clients
+        is_member_ev = item.event_type == nio.RoomMemberEvent.__name__
+
+        # If there were no better events available to show previously
+        prev_is_member_ev = \
+            room.last_event["event_type"] == nio.RoomMemberEvent.__name__
+
+        if is_member_ev and for_us and not prev_is_member_ev:
+            return
+
+        if item.date < room.last_event["date"]:  # If this is a past event
+            return
+
+        room.last_event = item.__dict__
+
+
+    async def register_nio_room(self, room: nio.MatrixRoom, left: bool = False,
+                               ) -> None:
+        # Generate the room name
+        name = room.name or room.canonical_alias
+        if not name:
+            name = room.group_name()
+            name = "" if name == "Empty room?" else name
+
+        # Add room
+        try:
+            last_ev = self.models[Room, self.user_id][room.room_id].last_event
+        except KeyError:
+            last_ev = None
+
+        self.models[Room, self.user_id][room.room_id] = Room(
+            room_id        = room.room_id,
+            display_name   = name,
+            avatar_url     = room.gen_avatar_url or "",
+            topic          = room.topic or "",
+            inviter_id     = getattr(room, "inviter", "") or "",
+            left           = left,
+            filter_string  = " ".join({name, room.topic or ""}).strip(),
+            last_event     = last_ev,
+        )
+
+        # Add the room members to the added room
+        new_dict = {
+            user_id: Member(
+                user_id       = user_id,
+                display_name  = room.user_name(user_id)  # disambiguated
+                                if member.display_name else "",
+                avatar_url    = member.avatar_url or "",
+                typing        = user_id in room.typing_users,
+                power_level   = member.power_level,
+                filter_string = " ".join({
+                    member.name, room.user_name(user_id),
+                }).strip(),
+            ) for user_id, member in room.users.items()
+        }
+        self.models[Member, room.room_id].update(new_dict)
+
+
+    async def get_member_name_avatar(self, room_id: str, user_id: str,
+                                    ) -> Tuple[str, str]:
+        try:
+            item = self.models[Member, room_id][user_id]
+        except KeyError:  # e.g. user is not anymore in the room
+            info = await self.backend.get_profile(user_id)
+
+            return (info.displayname or "", info.avatar_url or "") \
+                   if isinstance(info, nio.ProfileGetResponse) else \
+                   ("", "")
+        else:
+            return (item.display_name, item.avatar_url)
+
+
+    async def register_nio_event(self,
+                                 room:    nio.MatrixRoom,
+                                 ev:      nio.Event,
+                                 content: str) -> None:
+
+        await self.register_nio_room(room)
+
+        sender_name, sender_avatar = \
+            await self.get_member_name_avatar(room.room_id, ev.sender)
+
+        target_id = getattr(ev, "state_key", "") or ""
+
+        target_name, target_avatar = \
+            await self.get_member_name_avatar(room.room_id, target_id) \
+            if target_id else ("", "")
+
+        # Create Event ModelItem
+        item = Event(
+            client_id      = ev.event_id,
+            event_id       = ev.event_id,
+            event_type     = type(ev).__name__,
+            content        = content,
+            inline_content = HTML_FILTER.filter_inline(content),
+            date          = datetime.fromtimestamp(ev.server_timestamp / 1000),
+
+            sender_id     = ev.sender,
+            sender_name   = sender_name,
+            sender_avatar = sender_avatar,
+
+            target_id     = target_id,
+            target_name   = target_name,
+            target_avatar = target_avatar,
+        )
+
+        # Add the Event to model
+        if ev.transaction_id in self.local_echoes_uuid:
+            self.resolved_echoes[ev.event_id] = ev.transaction_id
+            self.local_echoes_uuid.discard(ev.transaction_id)
+            item.client_id = f"echo-{ev.transaction_id}"
+
+        elif ev.sender in self.backend.clients:
+            client = self.backend.clients[ev.sender]
+
+            # Wait until our other account has no more pending local echoes,
+            # so that we can know if this event should replace an echo
+            # from that client by finding its ID in the resolved_echoes dict.
+            # Server only gives back the transaction ID to the original sender.
+            while client.local_echoes_uuid:  # while there are pending echoes
+                await asyncio.sleep(0.1)
+
+            with suppress(KeyError):
+                item.client_id = f"echo-{client.resolved_echoes[ev.event_id]}"
+
+        self.models[Event, self.user_id, room.room_id][item.client_id] = item
+
+        await self.set_room_last_event(room.room_id, item)
+
+
     # Callbacks for nio responses
 
     async def onSyncResponse(self, resp: nio.SyncResponse) -> None:
-        up = rooms.RoomUpdated.from_nio
-
-        for room_id, info in resp.rooms.invite.items():
-            room = self.invited_rooms[room_id]
-
-            for member in room.users.values():
-                users.UserUpdated.from_nio(member)
-
-            up(self.user_id, "Invites", room, info)
-
         for room_id, info in resp.rooms.join.items():
-            room = self.rooms[room_id]
+            if room_id not in self.past_tokens:
-
+                self.past_tokens[room_id] = info.timeline.prev_batch
-            for member in room.users.values():
-                users.UserUpdated.from_nio(member)
-
-            if room_id not in self.backend.past_tokens:
-                self.backend.past_tokens[room_id] = info.timeline.prev_batch
-
-            up(self.user_id, "Rooms", room, info)
 
+        # TODO: way of knowing if a nio.MatrixRoom is left
         for room_id, info in resp.rooms.leave.items():
             # TODO: handle in nio, these are rooms that were left before
             # starting the client.
@@ -299,7 +455,12 @@ class MatrixClient(nio.AsyncClient):
                 if isinstance(ev, nio.RoomMemberEvent):
                     await self.onRoomMemberEvent(self.rooms[room_id], ev)
 
-            up(self.user_id, "Left", self.rooms[room_id], info)
+            await self.register_nio_room(self.rooms[room_id], left=True)
+
+        if not self.first_sync_happened.is_set():
+            asyncio.ensure_future(self.load_rooms_without_visible_events())
+
+        self.first_sync_happened.set()
 
 
     async def onErrorResponse(self, resp: nio.ErrorResponse) -> None:
@@ -310,55 +471,45 @@ class MatrixClient(nio.AsyncClient):
             log.warning(repr(resp))
 
 
-    # Callbacks for nio events
+    # Callbacks for nio room events
-
     # Content: %1 is the sender, %2 the target (ev.state_key).
-    # pylint: disable=unused-argument
 
-    async def onRoomMessageText(self, room, ev, from_past=False) -> None:
+    async def onRoomMessageText(self, room, ev) -> None:
         co = HTML_FILTER.filter(
             ev.formatted_body
-            if ev.format == "org.matrix.custom.html" else html.escape(ev.body)
+            if ev.format == "org.matrix.custom.html" else html.escape(ev.body),
         )
-        TimelineEventReceived.from_nio(room, ev, content=co)
+        await self.register_nio_event(room, ev, content=co)
 
 
-    async def onRoomMessageEmote(self, room, ev, from_past=False) -> None:
+    async def onRoomMessageEmote(self, room, ev) -> None:
         co = HTML_FILTER.filter_inline(
             ev.formatted_body
-            if ev.format == "org.matrix.custom.html" else html.escape(ev.body)
+            if ev.format == "org.matrix.custom.html" else html.escape(ev.body),
         )
-        TimelineEventReceived.from_nio(room, ev, content=co)
+        await self.register_nio_event(room, ev, content=co)
 
 
-    # async def onRoomMessageImage(self, room, ev, from_past=False) -> None:
+    async def onRoomCreateEvent(self, room, ev) -> None:
-        # import json; print("RMI", json.dumps( ev.__dict__ , indent=4))
-
-    # async def onRoomEncryptedImage(self, room, ev, from_past=False) -> None:
-        # import json; print("REI", json.dumps( ev.__dict__ , indent=4))
-
-
-    async def onRoomCreateEvent(self, room, ev, from_past=False) -> None:
         co = "%1 allowed users on other matrix servers to join this room." \
              if ev.federate else \
              "%1 blocked users on other matrix servers from joining this room."
-        TimelineEventReceived.from_nio(room, ev, content=co)
+        await self.register_nio_event(room, ev, content=co)
 
 
-    async def onRoomGuestAccessEvent(self, room, ev, from_past=False) -> None:
+    async def onRoomGuestAccessEvent(self, room, ev) -> None:
         allowed = "allowed" if ev.guest_access else "forbad"
         co      = f"%1 {allowed} guests to join the room."
-        TimelineEventReceived.from_nio(room, ev, content=co)
+        await self.register_nio_event(room, ev, content=co)
 
 
-    async def onRoomJoinRulesEvent(self, room, ev, from_past=False) -> None:
+    async def onRoomJoinRulesEvent(self, room, ev) -> None:
         access = "public" if ev.join_rule == "public" else "invite-only"
         co     = f"%1 made the room {access}."
-        TimelineEventReceived.from_nio(room, ev, content=co)
+        await self.register_nio_event(room, ev, content=co)
 
 
-    async def onRoomHistoryVisibilityEvent(self, room, ev, from_past=False
+    async def onRoomHistoryVisibilityEvent(self, room, ev) -> None:
-                                          ) -> None:
         if ev.history_visibility == "shared":
             to = "all room members"
         elif ev.history_visibility == "world_readable":
@@ -373,20 +524,22 @@ class MatrixClient(nio.AsyncClient):
                         json.dumps(ev.__dict__, indent=4))
 
         co = f"%1 made future room history visible to {to}."
-        TimelineEventReceived.from_nio(room, ev, content=co)
+        await self.register_nio_event(room, ev, content=co)
 
 
-    async def onPowerLevelsEvent(self, room, ev, from_past=False) -> None:
+    async def onPowerLevelsEvent(self, room, ev) -> None:
         co = "%1 changed the room's permissions."  # TODO: improve
-        TimelineEventReceived.from_nio(room, ev, content=co)
+        await self.register_nio_event(room, ev, content=co)
 
 
-    async def get_room_member_event_content(self, ev) -> Optional[str]:
+    async def process_room_member_event(self, room, ev) -> Optional[str]:
         prev            = ev.prev_content
         now             = ev.content
         membership      = ev.membership
         prev_membership = ev.prev_membership
+        ev_date         = datetime.fromtimestamp(ev.server_timestamp / 1000)
 
+        # Membership changes
         if not prev or membership != prev_membership:
             reason = f" Reason: {now['reason']}" if now.get("reason") else ""
 
@@ -421,17 +574,12 @@ class MatrixClient(nio.AsyncClient):
             if membership == "ban":
                 return f"%1 banned %2 from the room.{reason}"
 
-
+        # Profile changes
-        if ev.sender in self.backend.clients:
-            # Don't put our own name/avatar changes in the timeline
-            return None
-
         changed = []
 
         if prev and now["avatar_url"] != prev["avatar_url"]:
             changed.append("profile picture")  # TODO: <img>s
 
-
         if prev and now["displayname"] != prev["displayname"]:
             changed.append('display name from "{}" to "{}"'.format(
                 prev["displayname"] or ev.state_key,
@@ -439,6 +587,19 @@ class MatrixClient(nio.AsyncClient):
             ))
 
         if changed:
+            # Update our account profile if the event is newer than last update
+            if ev.state_key == self.user_id:
+                account = self.models[Account][self.user_id]
+
+                if account.profile_updated < ev_date:
+                    account.profile_updated = ev_date
+                    account.display_name    = now["displayname"] or ""
+                    account.avatar_url      = now["avatar_url"] or ""
+
+            if ev.state_key in self.backend.clients or len(room.users) > 50:
+                self.skipped_events[room.room_id] += 1
+                return None
+
             return "%1 changed their {}.".format(" and ".join(changed))
 
         log.warning("Invalid member event - %s",
@@ -446,48 +607,65 @@ class MatrixClient(nio.AsyncClient):
         return None
 
 
-    async def onRoomMemberEvent(self, room, ev, from_past=False) -> None:
+    async def onRoomMemberEvent(self, room, ev) -> None:
-        co = await self.get_room_member_event_content(ev)
+        co = await self.process_room_member_event(room, ev)
 
-        if co is not None:
+        if co is None:
-            TimelineEventReceived.from_nio(room, ev, content=co)
+            # This is run from register_nio_event otherwise
+            await self.register_nio_room(room)
+        else:
+            await self.register_nio_event(room, ev, content=co)
 
 
-    async def onRoomAliasEvent(self, room, ev, from_past=False) -> None:
+    async def onRoomAliasEvent(self, room, ev) -> None:
         co = f"%1 set the room's main address to {ev.canonical_alias}."
-        TimelineEventReceived.from_nio(room, ev, content=co)
+        await self.register_nio_event(room, ev, content=co)
 
 
-    async def onRoomNameEvent(self, room, ev, from_past=False) -> None:
+    async def onRoomNameEvent(self, room, ev) -> None:
         co = f"%1 changed the room's name to \"{ev.name}\"."
-        TimelineEventReceived.from_nio(room, ev, content=co)
+        await self.register_nio_event(room, ev, content=co)
 
 
-    async def onRoomTopicEvent(self, room, ev, from_past=False) -> None:
+    async def onRoomTopicEvent(self, room, ev) -> None:
         co = f"%1 changed the room's topic to \"{ev.topic}\"."
-        TimelineEventReceived.from_nio(room, ev, content=co)
+        await self.register_nio_event(room, ev, content=co)
 
 
-    async def onRoomEncryptionEvent(self, room, ev, from_past=False) -> None:
+    async def onRoomEncryptionEvent(self, room, ev) -> None:
-        co = f"%1 turned on encryption for this room."
+        co = "%1 turned on encryption for this room."
-        TimelineEventReceived.from_nio(room, ev, content=co)
+        await self.register_nio_event(room, ev, content=co)
 
 
-    async def onOlmEvent(self, room, ev, from_past=False) -> None:
+    async def onOlmEvent(self, room, ev) -> None:
-        co = f"%1 sent an undecryptable olm message."
+        co = "%1 sent an undecryptable olm message."
-        TimelineEventReceived.from_nio(room, ev, content=co)
+        await self.register_nio_event(room, ev, content=co)
 
 
-    async def onMegolmEvent(self, room, ev, from_past=False) -> None:
+    async def onMegolmEvent(self, room, ev) -> None:
-        co = f"%1 sent an undecryptable message."
+        co = "%1 sent an undecryptable message."
-        TimelineEventReceived.from_nio(room, ev, content=co)
+        await self.register_nio_event(room, ev, content=co)
 
 
-    async def onBadEvent(self, room, ev, from_past=False) -> None:
+    async def onBadEvent(self, room, ev) -> None:
-        co = f"%1 sent a malformed event."
+        co = "%1 sent a malformed event."
-        TimelineEventReceived.from_nio(room, ev, content=co)
+        await self.register_nio_event(room, ev, content=co)
 
 
-    async def onUnknownBadEvent(self, room, ev, from_past=False) -> None:
+    async def onUnknownBadEvent(self, room, ev) -> None:
-        co = f"%1 sent an event this client doesn't understand."
+        co = "%1 sent an event this client doesn't understand."
-        TimelineEventReceived.from_nio(room, ev, content=co)
+        await self.register_nio_event(room, ev, content=co)
+
+
+    # Callbacks for nio ephemeral events
+
+    async def onTypingNoticeEvent(self, room, ev) -> None:
+        # Prevent recent past typing notices from being shown for a split
+        # second on client startup:
+        if not self.first_sync_happened.is_set():
+            return
+
+        self.models[Room, self.user_id][room.room_id].typing_members = sorted(
+            room.user_name(user_id) for user_id in ev.users
+            if user_id not in self.backend.clients
+        )

src/python/models/__init__.py

@@ -0,0 +1,9 @@
+# Copyright 2019 miruka
+# This file is part of harmonyqml, licensed under LGPLv3.
+
+from typing import Tuple, Type, Union
+
+from .model_item import ModelItem
+
+# Type[ModelItem] or Tuple[Type[ModelItem], str...]
+SyncId = Union[Type[ModelItem], Tuple[Union[Type[ModelItem], str], ...]]

src/python/models/items.py

@@ -0,0 +1,119 @@
+# Copyright 2019 miruka
+# This file is part of harmonyqml, licensed under LGPLv3.
+
+from dataclasses import dataclass, field
+from datetime import datetime
+from typing import Any, Dict, List, Optional
+
+from ..html_filter import HTML_FILTER
+from .model_item import ModelItem
+
+
+@dataclass
+class Account(ModelItem):
+    main_key = "user_id"
+
+    user_id:         str      = field()
+    display_name:    str      = ""
+    avatar_url:      str      = ""
+    profile_updated: datetime = field(default_factory=datetime.now)
+
+    def __lt__(self, other: "Account") -> bool:
+        name       = self.display_name or self.user_id[1:]
+        other_name = other.display_name or other.user_id[1:]
+        return name < other_name
+
+
+@dataclass
+class Room(ModelItem):
+    main_key = "room_id"
+
+    room_id:        str       = field()
+    display_name:   str       = ""
+    avatar_url:     str       = ""
+    topic:          str       = ""
+    inviter_id:     str       = ""
+    left:           bool      = False
+    filter_string:  str       = ""
+    typing_members: List[str] = field(default_factory=list)
+
+    last_event: Optional[Dict[str, Any]] = None  # Event __dict__
+
+    def __lt__(self, other: "Room") -> bool:
+        # Left rooms may still have an inviter_id, check left first.
+        if self.left and not other.left:
+            return True
+
+        if other.inviter_id and not self.inviter_id:
+            return True
+
+        name       = self.display_name or self.room_id
+        other_name = other.display_name or other.room_id
+        return name < other_name
+
+
+@dataclass
+class Member(ModelItem):
+    main_key = "user_id"
+
+    user_id:       str  = field()
+    display_name:  str  = ""
+    avatar_url:    str  = ""
+    typing:        bool = False
+    power_level:   int  = 0
+    filter_string: str  = ""
+
+    def __lt__(self, other: "Member") -> bool:
+        name       = self.display_name or self.user_id[1:]
+        other_name = other.display_name or other.user_id[1:]
+        return name < other_name
+
+
+@dataclass
+class Event(ModelItem):
+    main_key = "event_id"
+
+    client_id:      str      = field()
+    event_id:       str      = field()
+    event_type:     str      = field()  # Type[nio.Event]
+    content:        str      = field()
+    inline_content: str      = field()
+    date:           datetime = field()
+
+    sender_id:     str = field()
+    sender_name:   str = field()
+    sender_avatar: str = field()
+
+    target_id:     str = ""
+    target_name:   str = ""
+    target_avatar: str = ""
+
+    is_local_echo: bool = False
+
+    def __post_init__(self) -> None:
+        self.inline_content = HTML_FILTER.filter_inline(self.content)
+
+
+    def __lt__(self, other: "Event") -> bool:
+        # Sort events from newest to oldest. return True means return False.
+        # Local echoes always stay first.
+        if self.is_local_echo and not other.is_local_echo:
+            return True
+
+        if other.is_local_echo and not self.is_local_echo:
+            return False
+
+        return self.date > other.date
+
+
+@dataclass
+class Device(ModelItem):
+    main_key = "device_id"
+
+    device_id:      str  = field()
+    ed25519_key:    str  = field()
+    trusted:        bool = False
+    blacklisted:    bool = False
+    display_name:   str  = ""
+    last_seen_ip:   str  = ""
+    last_seen_date: str  = ""

src/python/models/model.py

@@ -0,0 +1,116 @@
+# Copyright 2019 miruka
+# This file is part of harmonyqml, licensed under LGPLv3.
+
+import logging as log
+import time
+from threading import Lock, Thread
+from typing import Any, Dict, Iterator, List, MutableMapping, Optional
+
+from . import SyncId
+from ..pyotherside_events import ModelUpdated
+from .model_item import ModelItem
+
+
+class Model(MutableMapping):
+    def __init__(self, sync_id: SyncId) -> None:
+        self.sync_id:  SyncId               = sync_id
+        self.sortable: Optional[bool]       = None
+        self._data:    Dict[Any, ModelItem] = {}
+
+        self._changed:     bool   = False
+        self._sync_lock:   Lock   = Lock()
+        self._sync_thread: Thread = Thread(target=self._sync_loop, daemon=True)
+        self._sync_thread.start()
+
+
+    def __repr__(self) -> str:
+        try:
+            from pprintpp import pformat
+        except ImportError:
+            from pprint import pformat  # type: ignore
+
+        if isinstance(self.sync_id, tuple):
+            sid = (self.sync_id[0].__name__, *self.sync_id[1:])  # type: ignore
+        else:
+            sid = self.sync_id.__name__  # type: ignore
+
+        return "%s(sync_id=%s, sortable=%r, %s)" % (
+            type(self).__name__, sid, self.sortable, pformat(self._data),
+        )
+
+
+    def __str__(self) -> str:
+        if isinstance(self.sync_id, tuple):
+            sid = (self.sync_id[0].__name__, *self.sync_id[1:])  # type: ignore
+        else:
+            sid = self.sync_id.__name__  # type: ignore
+
+        return f"{sid!s}: {len(self)} items"
+
+
+    def __getitem__(self, key):
+        return self._data[key]
+
+
+    def __setitem__(self, key, value: ModelItem) -> None:
+        new = value
+
+        if key in self:
+            existing = dict(self[key].__dict__)  # copy to not alter with pop
+            merged   = {**existing, **value.__dict__}
+
+            existing.pop("parent_model", None)
+            merged.pop("parent_model", None)
+
+            if merged == existing:
+                return
+
+            new = type(value)(**merged)  # type: ignore
+
+        new.parent_model = self
+
+        with self._sync_lock:
+            self._data[key] = new
+            self._changed   = True
+
+
+    def __delitem__(self, key) -> None:
+        with self._sync_lock:
+            del self._data[key]
+            self._changed = True
+
+
+    def __iter__(self) -> Iterator:
+        return iter(self._data)
+
+
+    def __len__(self) -> int:
+        return len(self._data)
+
+
+    def _sync_loop(self) -> None:
+        while True:
+            time.sleep(0.25)
+
+            if self._changed:
+                with self._sync_lock:
+                    log.debug("Syncing %s", self)
+                    ModelUpdated(self.sync_id, self.serialized())
+                    self._changed = False
+
+
+    def serialized(self) -> List[Dict[str, Any]]:
+        if self.sortable is True:
+            return [item.__dict__ for item in sorted(self._data.values())]
+
+        if self.sortable is False or len(self) < 2:
+            return [item.__dict__ for item in self._data.values()]
+
+        try:
+            return [item.__dict__ for item in sorted(self._data.values())]
+        except TypeError:
+            self.sortable = False
+        else:
+            self.sortable = True
+
+        return self.serialized()

src/python/models/model_item.py

@@ -0,0 +1,31 @@
+# Copyright 2019 miruka
+# This file is part of harmonyqml, licensed under LGPLv3.
+
+from typing import ClassVar, Optional
+
+
+class ModelItem:
+    main_key: ClassVar[str] = ""
+
+    def __init_subclass__(cls) -> None:
+        if not cls.main_key:
+            raise ValueError("Must specify main_key str class attribute.")
+
+        super().__init_subclass__()
+
+
+    def __new__(cls, *_args, **_kwargs) -> "ModelItem":
+        from .model import Model
+        cls.parent_model: Optional[Model] = None
+        return super().__new__(cls)
+
+
+    def __setattr__(self, name: str, value) -> None:
+        super().__setattr__(name, value)
+
+        if name != "parent_model" and self.parent_model is not None:
+            with self.parent_model._sync_lock:
+                self.parent_model._changed = True
+
+    def __delattr__(self, name: str) -> None:
+        raise NotImplementedError()

src/python/models/model_store.py

@@ -0,0 +1,57 @@
+# Copyright 2019 miruka
+# This file is part of harmonyqml, licensed under LGPLv3.
+
+from typing import Dict, Iterator, MutableMapping, Set, Tuple, Type, Union
+
+from dataclasses import dataclass, field
+
+from . import SyncId
+from .model_item import ModelItem
+from .model import Model
+
+KeyType = Union[Type[ModelItem], Tuple[Type, ...]]
+
+@dataclass(frozen=True)
+class ModelStore(MutableMapping):
+    allowed_key_types: Set[KeyType] = field()
+
+    data: Dict[SyncId, Model] = field(init=False, default_factory=dict)
+
+
+    def __getitem__(self, key: SyncId) -> Model:
+        try:
+            return self.data[key]
+        except KeyError:
+            if isinstance(key, tuple):
+                for i in key:
+                    if not i:
+                        raise ValueError(f"Empty string in key: {key!r}")
+
+                key_type  = (key[0],) + \
+                            tuple(type(el) for el in key[1:])  # type: ignore
+            else:
+                key_type = key  # type: ignore
+
+            if key_type not in self.allowed_key_types:
+                raise TypeError(f"{key_type!r} not in allowed key types: "
+                                f"{self.allowed_key_types!r}")
+
+            model          = Model(key)
+            self.data[key] = model
+            return model
+
+
+    def __setitem__(self, key, item) -> None:
+        raise NotImplementedError()
+
+
+    def __delitem__(self, key: SyncId) -> None:
+        del self.data[key]
+
+
+    def __iter__(self) -> Iterator[SyncId]:
+        return iter(self.data)
+
+
+    def __len__(self) -> int:
+        return len(self.data)

src/python/pyotherside.py

@@ -0,0 +1,30 @@
+# Copyright 2019 miruka
+# This file is part of harmonyqml, licensed under LGPLv3.
+
+import logging as log
+from typing import Tuple, Callable
+
+AVAILABLE = True
+try:
+    import pyotherside
+except ModuleNotFoundError:
+    log.warning("pyotherside module is unavailable.")
+    AVAILABLE = False
+
+Size      = Tuple[int, int]
+ImageData = Tuple[bytearray, Size, int]  # last int: pyotherside format enum
+
+format_data:     int = pyotherside.format_data if AVAILABLE else 0
+format_svg_data: int = pyotherside.format_svg_data if AVAILABLE else 1
+format_rgb888:   int = pyotherside.format_rgb888 if AVAILABLE else 2
+format_argb32:   int = pyotherside.format_argb32 if AVAILABLE else 3
+
+
+def send(event: str, *args) -> None:
+    if AVAILABLE:
+        pyotherside.send(event, *args)
+
+
+def set_image_provider(provider: Callable[[str, Size], ImageData]) -> None:
+    if AVAILABLE:
+        pyotherside.set_image_provider(provider)

src/python/pyotherside_events.py

@@ -0,0 +1,57 @@
+# Copyright 2019 miruka
+# This file is part of harmonyqml, licensed under LGPLv3.
+
+from dataclasses import dataclass, field
+from enum import Enum
+from typing import Any, Dict, List, Union
+
+from . import pyotherside
+from .models import SyncId
+
+
+@dataclass
+class PyOtherSideEvent:
+    def __post_init__(self) -> None:
+        # CPython >= 3.6 or any Python >= 3.7 needed for correct dict order
+        args = [
+            self._process_field(getattr(self, field))
+            for field in self.__dataclass_fields__  # type: ignore
+        ]
+        pyotherside.send(type(self).__name__, *args)
+
+
+    @staticmethod
+    def _process_field(value: Any) -> Any:
+        if hasattr(value, "__class__") and issubclass(value.__class__, Enum):
+            return value.value
+
+        return value
+
+
+@dataclass
+class ExitRequested(PyOtherSideEvent):
+    exit_code: int = 0
+
+
+@dataclass
+class CoroutineDone(PyOtherSideEvent):
+    uuid:   str = field()
+    result: Any = None
+
+
+@dataclass
+class ModelUpdated(PyOtherSideEvent):
+    sync_id: SyncId               = field()
+    data:    List[Dict[str, Any]] = field()
+
+    serialized_sync_id: Union[str, List[str]] = field(init=False)
+
+    def __post_init__(self) -> None:
+        if isinstance(self.sync_id, tuple):
+            self.serialized_sync_id = [
+                e.__name__ if isinstance(e, type) else e for e in self.sync_id
+            ]
+        else:
+           self.serialized_sync_id = self.sync_id.__name__
+
+        super().__post_init__()

src/python/utils.py

@@ -2,10 +2,14 @@
 # This file is part of harmonyqml, licensed under LGPLv3.
 
 import collections
-from enum import auto as autostr
+import xml.etree.cElementTree as xml_etree  # FIXME: bandit warning
 from enum import Enum
+from enum import auto as autostr
+from typing import IO, Optional
 
-auto = autostr  # pylint: disable=invalid-name
+import filetype
+
+auto = autostr
 
 
 class AutoStrEnum(Enum):
@@ -22,3 +26,19 @@ def dict_update_recursive(dict1, dict2):
             dict_update_recursive(dict1[k], dict2[k])
         else:
             dict1[k] = dict2[k]
+
+
+def is_svg(file: IO) -> bool:
+    try:
+        _, element = next(xml_etree.iterparse(file, ("start",)))
+        return element.tag == "{http://www.w3.org/2000/svg}svg"
+    except (StopIteration, xml_etree.ParseError):
+        return False
+
+
+def guess_mime(file: IO) -> Optional[str]:
+    if is_svg(file):
+        return "image/svg+xml"
+
+    file.seek(0, 0)
+    return filetype.guess_mime(file)

src/qml/Base/HAvatar.qml

@@ -35,7 +35,7 @@ HRectangle {
         text: name ? name.charAt(0) : "?"
         font.pixelSize: parent.height / 1.4
 
-        color: Utils.hsla(
+        color: Utils.hsluv(
            name ? Utils.hueFrom(name) : 0,
            name ? theme.controls.avatar.letter.saturation : 0,
            theme.controls.avatar.letter.lightness,

src/qml/Base/HListModel.qml

@@ -2,124 +2,14 @@
 // This file is part of harmonyqml, licensed under LGPLv3.
 
 import QtQuick 2.12
-import SortFilterProxyModel 0.2
+import QSyncable 1.0
 
-SortFilterProxyModel {
+JsonListModel {
-    // To initialize a HListModel with items,
+    id: model
-    // use `Component.onCompleted: extend([{"foo": 1, "bar": 2}, ...])`
+    source: []
+    Component.onCompleted: if (! keyField) { throw "keyField not set" }
 
-    id: sortFilteredModel
+    function toObject(itemList=listModel) {
-
-    property ListModel model: ListModel {}
-    sourceModel: model  // Can't assign a "ListModel {}" directly here
-
-    function append(dict)           { return model.append(dict) }
-    function clear()                { return model.clear() }
-    function insert(index, dict)    { return model.inset(index, dict) }
-    function move(from, to, n=1)    { return model.move(from, to, n) }
-    function remove(index, count=1) { return model.remove(index, count) }
-    function set(index, dict)       { return model.set(index, dict) }
-    function sync()                 { return model.sync() }
-    function setProperty(index, prop, value) {
-        return model.setProperty(index, prop, value)
-    }
-
-    function extend(newItems) {
-        for (let item of newItems) { model.append(item) }
-    }
-
-    function getIndices(whereRolesAre, maxResults=null, maxTries=null) {
-        // maxResults, maxTries: null or int
-        let results = []
-
-        for (let i = 0; i < model.count; i++) {
-            let item    = model.get(i)
-            let include = true
-
-            for (let role in whereRolesAre) {
-                if (item[role] != whereRolesAre[role]) {
-                    include = false
-                    break
-                }
-            }
-
-            if (include) {
-                results.push(i)
-                if (maxResults && results.length >= maxResults) {
-                    break
-                }
-            }
-
-            if (maxTries && i >= maxTries) {
-                break
-            }
-        }
-        return results
-    }
-
-    function getWhere(rolesAre, maxResults=null, maxTries=null) {
-        let items = []
-
-        for (let indice of getIndices(rolesAre, maxResults, maxTries)) {
-            items.push(model.get(indice))
-        }
-        return items
-    }
-
-    function forEachWhere(rolesAre, func, maxResults=null, maxTries=null) {
-        for (let item of getWhere(rolesAre, maxResults, maxTries)) {
-            func(item)
-        }
-    }
-
-    function upsert(
-        whereRolesAre, newItem, updateIfExist=true, maxTries=null
-    ) {
-        let indices = getIndices(whereRolesAre, 1, maxTries)
-
-        if (indices.length == 0) {
-            model.append(newItem)
-            return model.get(model.count)
-        }
-
-		let existing = model.get(indices[0])
-        if (! updateIfExist) { return existing }
-
-		// Really update only if existing and new item have a difference
-		for (var role in existing) {
-			if (Boolean(existing[role].getTime)) {
-				if (existing[role].getTime() != newItem[role].getTime()) {
-					model.set(indices[0], newItem)
-					return existing
-				}
-			} else {
-				if (existing[role] != newItem[role]) {
-					model.set(indices[0], newItem)
-					return existing
-				}
-			}
-		}
-        return existing
-    }
-
-    function pop(index) {
-        let item = model.get(index)
-        model.remove(index)
-        return item
-    }
-
-    function popWhere(rolesAre, maxResults=null, maxTries=null) {
-        let items = []
-
-        for (let indice of getIndices(rolesAre, maxResults, maxTries)) {
-            items.push(model.get(indice))
-            model.remove(indice)
-        }
-        return items
-    }
-
-
-    function toObject(itemList=sortFilteredModel) {
         let objList = []
 
         for (let item of itemList) {

src/qml/Base/HPage.qml

@@ -8,6 +8,7 @@ import "../SidePane"
 
 SwipeView {
     default property alias columnChildren: contentColumn.children
+
     property alias page: innerPage
     property alias header: innerPage.header
     property alias footer: innerPage.header
@@ -81,7 +82,6 @@ SwipeView {
 
             HColumnLayout {
                 id: contentColumn
-                spacing: theme.spacing
                 width: innerFlickable.width
                 height: innerFlickable.height
             }

src/qml/Base/HRoomAvatar.qml

@@ -4,16 +4,13 @@
 import QtQuick 2.12
 
 HAvatar {
-    property string userId: ""
+    property string displayName: ""
-    property string roomId: ""
+    property string avatarUrl: ""
 
-    readonly property var roomInfo: rooms.getWhere({userId, roomId}, 1)[0]
+    name: displayName[0] == "#" && displayName.length > 1 ?
+          displayName.substring(1) :
+          displayName
 
-    // Avoid error messages when a room is forgotten
+    imageUrl: avatarUrl ? ("image://python/" + avatarUrl) : null
-    readonly property var dname: roomInfo ? roomInfo.displayName : ""
+    toolTipImageUrl: avatarUrl ? ("image://python/" + avatarUrl) : null
-    readonly property var avUrl: roomInfo ? roomInfo.avatarUrl : ""
-
-    name: dname[0] == "#" && dname.length > 1 ? dname.substring(1) : dname
-    imageUrl: avUrl ? ("image://python/" + avUrl) : null
-    toolTipImageUrl: avUrl ? ("image://python/" + avUrl) : null
 }

src/qml/Base/HTextField.qml

@@ -26,9 +26,8 @@ TextField {
         border.color: field.activeFocus ? focusedBorderColor : borderColor
         border.width: bordered ? theme.controls.textField.borderWidth : 0
 
-        Behavior on color { HColorAnimation { factor: 0.5 } }
+        Behavior on color { HColorAnimation { factor: 0.25 } }
-        Behavior on border.color { HColorAnimation { factor: 0.5 } }
+        Behavior on border.color { HColorAnimation { factor: 0.25 } }
-        Behavior on border.width { HNumberAnimation { factor: 0.5 } }
     }
 
     selectByMouse: true

src/qml/Base/HUserAvatar.qml

@@ -5,23 +5,16 @@ import QtQuick 2.12
 
 HAvatar {
     property string userId: ""
-    readonly property var userInfo: userId ? users.find(userId) : ({})
+    property string displayName: ""
+    property string avatarUrl: ""
 
     readonly property var defaultImageUrl:
-        userInfo.avatarUrl ? ("image://python/" + userInfo.avatarUrl) : null
+        avatarUrl ? ("image://python/" + avatarUrl) : null
 
     readonly property var defaultToolTipImageUrl:
-        userInfo.avatarUrl ? ("image://python/" + userInfo.avatarUrl) : null
+        avatarUrl ? ("image://python/" + avatarUrl) : null
 
-    name: userInfo.displayName || userId.substring(1)  // no leading @
+    name: displayName || userId.substring(1)  // no leading @
     imageUrl: defaultImageUrl
     toolTipImageUrl:defaultToolTipImageUrl
-
-    //HImage {
-        //id: status
-        //anchors.right: parent.right
-        //anchors.bottom: parent.bottom
-        //source: "../../icons/status.svg"
-        //sourceSize.width: 12
-    //}
 }

src/qml/Chat/Banners/LeftBanner.qml

@@ -6,7 +6,6 @@ import "../../Base"
 
 Banner {
     property string userId: ""
-    readonly property var userInfo: users.find(userId)
 
     color: theme.chat.leftBanner.background
 

src/qml/Chat/Chat.qml

@@ -4,29 +4,26 @@
 import QtQuick 2.12
 import QtQuick.Layouts 1.12
 import "../Base"
+import "../utils.js" as Utils
 
 HPage {
     id: chatPage
 
-    property bool ready: roomInfo && ! roomInfo.loading
+    property string userId: ""
+    property string roomId: ""
 
-    property var roomInfo: null
+    property bool ready: roomInfo !== "waiting"
+
+    readonly property var roomInfo: Utils.getItem(
+        modelSources[["Room", userId]] || [], "room_id", roomId
+    ) || "waiting"
     onRoomInfoChanged: if (! roomInfo) { pageStack.showPage("Default") }
 
-    readonly property string userId: roomInfo.userId
-    readonly property string category: roomInfo.category
-    readonly property string roomId: roomInfo.roomId
-
-    readonly property var senderInfo: users.find(userId)
-
     readonly property bool hasUnknownDevices: false
-         //category == "Rooms" ?
-         //Backend.clients.get(userId).roomHasUnknownDevices(roomId) : false
 
-    header: RoomHeader {
+    header: Loader {
         id: roomHeader
-        displayName: roomInfo.displayName
+        source: ready ? "RoomHeader.qml" : ""
-        topic: roomInfo.topic
 
         clip: height < implicitHeight
         width: parent.width
@@ -37,18 +34,7 @@ HPage {
     page.leftPadding: 0
     page.rightPadding: 0
 
-
     Loader {
-        Timer {
-            interval: 200
-            repeat: true
-            running: ! ready
-            onTriggered: {
-                let info = rooms.find(userId, category, roomId)
-                if (! info.loading) { roomInfo = Qt.binding(() => info) }
-            }
-        }
-
         source: ready ? "ChatSplitView.qml" : "../Base/HBusyIndicator.qml"
 
         Layout.fillWidth: ready

src/qml/Chat/ChatSplitView.qml

@@ -28,25 +28,21 @@ HSplitView {
         }
 
         InviteBanner {
-            visible: category == "Invites"
+            id: inviteBanner
-            inviterId: roomInfo.inviterId
+            visible: Boolean(inviterId)
+            inviterId: chatPage.roomInfo.inviter_id
 
             Layout.fillWidth: true
         }
 
-        //UnknownDevicesBanner {
-            //visible: category == "Rooms" && hasUnknownDevices
-            //
-            //Layout.fillWidth: true
-        //}
-
         SendBox {
             id: sendBox
-            visible: category == "Rooms" && ! hasUnknownDevices
+            visible: ! inviteBanner.visible && ! leftBanner.visible
         }
 
         LeftBanner {
-            visible: category == "Left"
+            id: leftBanner
+            visible: chatPage.roomInfo.left
             userId: chatPage.userId
 
             Layout.fillWidth: true
@@ -56,7 +52,8 @@ HSplitView {
     RoomSidePane {
         id: roomSidePane
 
-        activeView: roomHeader.activeButton
+        activeView: roomHeader.item ? roomHeader.item.activeButton : null
+
         property int oldWidth: width
         onActiveViewChanged:
             activeView ? restoreAnimation.start() : hideAnimation.start()
@@ -89,7 +86,9 @@ HSplitView {
         collapsed: width < theme.controls.avatar.size + theme.spacing
 
         property bool wasSnapped: false
-        property int referenceWidth: roomHeader.buttonsWidth
+        property int referenceWidth:
+            roomHeader.item ? roomHeader.item.buttonsWidth : 0
+
         onReferenceWidthChanged: {
             if (! chatSplitView.manuallyResized || wasSnapped) {
                 if (wasSnapped) { chatSplitView.manuallyResized = false }

src/qml/Chat/RoomHeader.qml

@@ -6,9 +6,6 @@ import QtQuick.Layouts 1.12
 import "../Base"
 
 HRectangle {
-    property string displayName: ""
-    property string topic: ""
-
     property alias buttonsImplicitWidth: viewButtons.implicitWidth
     property int buttonsWidth: viewButtons.Layout.preferredWidth
     property var activeButton: "members"
@@ -29,14 +26,14 @@ HRectangle {
 
         HRoomAvatar {
             id: avatar
-            userId: chatPage.userId
+            displayName: chatPage.roomInfo.display_name
-            roomId: chatPage.roomId
+            avatarUrl: chatPage.roomInfo.avatar_url
             Layout.alignment: Qt.AlignTop
         }
 
         HLabel {
             id: roomName
-            text: displayName
+            text: chatPage.roomInfo.display_name
             font.pixelSize: theme.fontSize.big
             color: theme.chat.roomHeader.name
             elide: Text.ElideRight
@@ -53,7 +50,7 @@ HRectangle {
 
         HLabel {
             id: roomTopic
-            text: topic
+            text: chatPage.roomInfo.topic
             font.pixelSize: theme.fontSize.small
             color: theme.chat.roomHeader.topic
             elide: Text.ElideRight

src/qml/Chat/RoomSidePane/MemberDelegate.qml

@@ -4,14 +4,13 @@
 import QtQuick 2.12
 import QtQuick.Layouts 1.12
 import "../../Base"
+import "../../utils.js" as Utils
 
 HInteractiveRectangle {
     id: memberDelegate
     width: memberList.width
     height: childrenRect.height
 
-    property var memberInfo: users.find(model.userId)
-
     Row {
         width: parent.width - leftPadding * 2
         padding: roomSidePane.currentSpacing / 2
@@ -24,7 +23,9 @@ HInteractiveRectangle {
 
             HUserAvatar {
                 id: avatar
-                userId: model.userId
+                userId: model.user_id
+                displayName: model.display_name
+                avatarUrl: model.avatar_url
             }
 
             HColumnLayout {
@@ -32,7 +33,7 @@ HInteractiveRectangle {
 
                 HLabel {
                     id: memberName
-                    text: memberInfo.displayName || model.userId
+                    text: model.display_name || model.user_id
                     elide: Text.ElideRight
                     verticalAlignment: Qt.AlignVCenter
 

src/qml/Chat/RoomSidePane/MembersView.qml

@@ -3,7 +3,6 @@
 
 import QtQuick 2.12
 import QtQuick.Layouts 1.12
-import SortFilterProxyModel 0.2
 import "../../Base"
 import "../../utils.js" as Utils
 
@@ -13,23 +12,11 @@ HColumnLayout {
         bottomMargin: currentSpacing
 
         model: HListModel {
-            sourceModel: chatPage.roomInfo.members
+            keyField: "user_id"
-
+            source: Utils.filterModelSource(
-            proxyRoles: ExpressionRole {
+                modelSources[["Member", chatPage.roomId]] || [],
-                name: "displayName"
+                filterField.text
-                expression: users.find(userId).displayName || userId
+            )
-            }
-
-            sorters: StringSorter {
-                roleName: "displayName"
-            }
-
-            filters: ExpressionFilter {
-                function filterIt(filter, text) {
-                    return Utils.filterMatches(filter, text)
-                }
-                expression: filterIt(filterField.text, displayName)
-            }
         }
 
         delegate: MemberDelegate {}

src/qml/Chat/SendBox.qml

@@ -4,6 +4,7 @@
 import QtQuick 2.12
 import QtQuick.Layouts 1.12
 import "../Base"
+import "../utils.js" as Utils
 
 HRectangle {
     function setFocus() { areaScrollView.forceActiveFocus() }
@@ -11,9 +12,12 @@ HRectangle {
     property string indent: "    "
 
     property var aliases: window.settings.writeAliases
-    property string writingUserId: chatPage.userId
     property string toSend: ""
 
+    property string writingUserId: chatPage.userId
+    readonly property var writingUserInfo:
+        Utils.getItem(modelSources["Account"] || [], "user_id", writingUserId)
+
     property bool textChangedSinceLostFocus: false
 
     property alias textArea: areaScrollView.area
@@ -57,6 +61,8 @@ HRectangle {
         HUserAvatar {
             id: avatar
             userId: writingUserId
+            displayName: writingUserInfo.display_name
+            avatarUrl: writingUserInfo.avatar_url
         }
 
         HScrollableTextArea {
@@ -166,6 +172,13 @@ HRectangle {
                 })
 
                 area.Keys.onPressed.connect(event => {
+                    if (event.modifiers == Qt.MetaModifier) {
+                        // Prevent super+key from sending the key as text
+                        // on xwayland
+                        event.accepted = true
+                        return
+                    }
+
                     if (event.modifiers == Qt.NoModifier &&
                         event.key == Qt.Key_Backspace &&
                         ! textArea.selectedText)

src/qml/Chat/Timeline/EventContent.qml

@@ -19,7 +19,9 @@ Row {
 
         HUserAvatar {
             id: avatar
-            userId: model.senderId
+            userId: model.sender_id
+            displayName: model.sender_name
+            avatarUrl: model.sender_avatar
             width: hideAvatar ? 0 : 48
             height: hideAvatar ? 0 : collapseAvatar ? 1 : 48
         }
@@ -52,8 +54,8 @@ Row {
                 width: parent.width
                 visible: ! hideNameLine
 
-                text: senderInfo.displayName || model.senderId
+                text: Utils.coloredNameHtml(model.sender_name, model.sender_id)
-                color: Utils.nameColor(avatar.name)
+                textFormat: Text.StyledText
                 elide: Text.ElideRight
                 horizontalAlignment: onRight ? Text.AlignRight : Text.AlignLeft
 
@@ -74,7 +76,7 @@ Row {
                       Qt.formatDateTime(model.date, "hh:mm:ss") +
                       "</font>" +
                       // local echo icon
-                      (model.isLocalEcho ?
+                      (model.is_local_echo ?
                        "&nbsp;<font size=" + theme.fontSize.small +
                        "px>⏳</font>" : "")
 

src/qml/Chat/Timeline/EventDelegate.qml

@@ -18,9 +18,7 @@ Column {
         nextItem     = eventList.model.get(model.index - 1)
     }
 
-    property var senderInfo: senderInfo = users.find(model.senderId)
+    property bool isOwn: chatPage.userId === model.sender_id
-
-    property bool isOwn: chatPage.userId === model.senderId
     property bool onRight: eventList.ownEventsOnRight && isOwn
     property bool combine: eventList.canCombine(previousItem, model)
     property bool talkBreak: eventList.canTalkBreak(previousItem, model)
@@ -28,22 +26,22 @@ Column {
 
     readonly property bool smallAvatar:
         eventList.canCombine(model, nextItem) &&
-        (model.eventType == "RoomMessageEmote" ||
+        (model.event_type == "RoomMessageEmote" ||
-         ! model.eventType.startsWith("RoomMessage"))
+         ! model.event_type.startsWith("RoomMessage"))
 
     readonly property bool collapseAvatar: combine
     readonly property bool hideAvatar: onRight
 
     readonly property bool hideNameLine:
-        model.eventType == "RoomMessageEmote" ||
+        model.event_type == "RoomMessageEmote" ||
-        ! model.eventType.startsWith("RoomMessage") ||
+        ! model.event_type.startsWith("RoomMessage") ||
         onRight ||
         combine
 
     width: eventList.width
 
     topPadding:
-        model.eventType == "RoomCreateEvent" ? 0 :
+        model.event_type == "RoomCreateEvent" ? 0 :
         dayBreak  ? theme.spacing * 4 :
         talkBreak ? theme.spacing * 6 :
         combine   ? theme.spacing / 2 :

src/qml/Chat/Timeline/EventList.qml

@@ -2,7 +2,6 @@
 // This file is part of harmonyqml, licensed under LGPLv3.
 
 import QtQuick 2.12
-import SortFilterProxyModel 0.2
 import "../../Base"
 import "../../utils.js" as Utils
 
@@ -22,7 +21,7 @@ HRectangle {
             return Boolean(
                 ! canTalkBreak(item, itemAfter) &&
                 ! canDayBreak(item, itemAfter) &&
-                item.senderId === itemAfter.senderId &&
+                item.sender_id === itemAfter.sender_id &&
                 Utils.minutesBetween(item.date, itemAfter.date) <= 5
             )
         }
@@ -42,18 +41,15 @@ HRectangle {
             }
 
             return Boolean(
-                itemAfter.eventType == "RoomCreateEvent" ||
+                itemAfter.event_type == "RoomCreateEvent" ||
                 item.date.getDate() != itemAfter.date.getDate()
             )
         }
 
         model: HListModel {
-            sourceModel: timelines
+            keyField: "client_id"
-
+            source:
-            filters: ValueFilter {
+                modelSources[["Event", chatPage.userId, chatPage.roomId]] || []
-                roleName: "roomId"
-                value: chatPage.roomId
-            }
         }
 
         property bool ownEventsOnRight:
@@ -76,23 +72,20 @@ HRectangle {
         // Declaring this as "alias" provides the on... signal
         property real yPos: visibleArea.yPosition
         property bool canLoad: true
-        // property int zz: 0
+        onYPosChanged: Qt.callLater(loadPastEvents)
 
-        onYPosChanged: {
+        function loadPastEvents() {
-            if (chatPage.category != "Invites" && canLoad && yPos <= 0.1) {
+            if (chatPage.invited_id || ! canLoad || yPos > 0.1) { return }
-                // zz += 1
+            eventList.canLoad = false
-                // print(canLoad, zz)
+            py.callClientCoro(
-                eventList.canLoad = false
+                chatPage.userId, "load_past_events", [chatPage.roomId],
-                py.callClientCoro(
+                moreToLoad => { eventList.canLoad = moreToLoad }
-                    chatPage.userId, "load_past_events", [chatPage.roomId],
+            )
-                    moreToLoad => { eventList.canLoad = moreToLoad }
-                )
-            }
         }
     }
 
     HNoticePage {
-        text: qsTr("Nothing to show here yet...")
+        text: qsTr("Nothing here yet...")
 
         visible: eventList.model.count < 1
         anchors.fill: parent

src/qml/Chat/TypingMembersBar.qml

@@ -11,31 +11,42 @@ HRectangle {
     property alias label: typingLabel
 
     color: theme.chat.typingMembers.background
-    implicitHeight: typingLabel.text ? typingLabel.height : 0
+    implicitHeight: typingLabel.text ? rowLayout.height : 0
 
     Behavior on implicitHeight { HNumberAnimation {} }
 
     HRowLayout {
+        id: rowLayout
         spacing: theme.spacing
-        anchors.fill: parent
-        Layout.leftMargin: spacing
-        Layout.rightMargin: spacing
-        Layout.topMargin: spacing / 4
-        Layout.bottomMargin: spacing / 4
 
         HIcon {
             id: icon
             svgName: "typing"  // TODO: animate
-            height: typingLabel.height
+
+            Layout.fillHeight: true
+            Layout.leftMargin: rowLayout.spacing / 2
         }
 
         HLabel {
             id: typingLabel
-            text: chatPage.roomInfo.typingText
             textFormat: Text.StyledText
             elide: Text.ElideRight
+            text: {
+                let tm = chatPage.roomInfo.typing_members
+
+                if (tm.length == 0) return ""
+                if (tm.length == 1) return qsTr("%1 is typing...").arg(tm[0])
+
+                return qsTr("%1 and %2 are typing...")
+                       .arg(tm.slice(0, -1).join(", ")).arg(tm.slice(-1)[0])
+            }
 
             Layout.fillWidth: true
+            Layout.fillHeight: true
+            Layout.topMargin: rowLayout.spacing / 4
+            Layout.bottomMargin: rowLayout.spacing / 4
+            Layout.leftMargin: rowLayout.spacing / 2
+            Layout.rightMargin: rowLayout.spacing / 2
         }
     }
 }

src/qml/EventHandlers/includes.js

@@ -1,9 +0,0 @@
-// Copyright 2019 miruka
-// This file is part of harmonyqml, licensed under LGPLv3.
-
-"use strict"
-
-// FIXME: Obsolete method, but need Qt 5.12+ for standard JS modules import
-Qt.include("app.js")
-Qt.include("users.js")
-Qt.include("rooms.js")

src/qml/EventHandlers/rooms.js

@@ -1,113 +0,0 @@
-// Copyright 2019 miruka
-// This file is part of harmonyqml, licensed under LGPLv3.
-
-"use strict"
-
-Qt.include("../utils.js")
-
-function typingTextFor(members, ourUserId) {
-    let ourUsers = []
-    let profiles = []
-    let names    = []
-
-    for (let i = 0;  i < accounts.count; i++) {
-        ourUsers.push(accounts.get(i).userId)
-    }
-
-    for (let member of members) {
-        if (! ourUsers.includes(member)) { profiles.push(users.find(member)) }
-    }
-
-    profiles.sort((left, right) => {
-      if (left.displayName < right.displayName) { return -1 }
-      if (left.displayName > right.displayName) { return +1 }
-      return 0
-    })
-
-    for (let profile of profiles) {
-        names.push(coloredNameHtml(profile.displayName, profile.userId))
-    }
-
-    if (names.length == 0) { return "" }
-    if (names.length == 1) { return qsTr("%1 is typing...").arg(names[0]) }
-
-    let text = qsTr("%1 and %2 are typing...")
-
-    if (names.length == 2) { return text.arg(names[0]).arg(names[1]) }
-
-    return text.arg(names.slice(0, -1).join(", ")).arg(names.slice(-1)[0])
-}
-
-
-function onRoomUpdated(
-    userId, category, roomId, displayName, avatarUrl, topic,
-    members, typingMembers, inviterId
-) {
-    roomCategories.upsert({userId, name: category}, {userId, name: category})
-
-    function find(category) {
-        let found = rooms.getIndices({userId, roomId, category}, 1)
-        return found.length > 0 ? found[0] : null
-    }
-
-    let replace = null
-    if (category == "Invites")    { replace = find("Rooms") || find("Left") }
-    else if (category == "Rooms") { replace = find("Invites") || find("Left") }
-    else if (category == "Left")  { replace = find("Invites") || find("Rooms")}
-
-    let item = {
-        loading:    false,
-        typingText: typingTextFor(typingMembers, userId),
-
-        userId, category, roomId, displayName, avatarUrl, topic, members,
-        inviterId
-    }
-
-    if (replace === null) {
-        rooms.upsert({userId, roomId, category}, item)
-    } else {
-        rooms.set(replace, item)
-    }
-}
-
-
-function onRoomForgotten(userId, roomId) {
-    rooms.popWhere({userId, roomId})
-}
-
-
-function onTimelineEventReceived(
-    eventType, roomId, eventId, senderId, date, content, isLocalEcho,
-    targetUserId
-) {
-    let item = {
-        eventType: py.getattr(eventType, "__name__"),
-        roomId, eventId, senderId, date, content, isLocalEcho, targetUserId
-    }
-
-    if (isLocalEcho) {
-        timelines.append(item)
-        return
-    }
-
-    // Replace first matching local echo
-    let found = timelines.getIndices(
-        {roomId, senderId, content, "isLocalEcho": true}, 1, 250
-    )
-
-    if (found.length > 0) {
-        timelines.set(found[0], item)
-    }
-    // Multiple clients will emit duplicate events with the same eventId
-    else if (item.eventType == "OlmEvent" || item.eventType == "MegolmEvent") {
-        // Don't replace if an item with the same eventId is found in these
-        // cases, because it would be the ecrypted version of the event.
-        timelines.upsert({eventId}, item, false, 250)
-    }
-    else {
-        timelines.upsert({eventId}, item, true, 250)
-    }
-}
-
-
-var onTimelineMessageReceived = onTimelineEventReceived

src/qml/EventHandlers/users.js

@@ -1,23 +0,0 @@
-// Copyright 2019 miruka
-// This file is part of harmonyqml, licensed under LGPLv3.
-
-"use strict"
-
-function onAccountUpdated(userId) {
-    accounts.append({userId})
-}
-
-function onAccountDeleted(userId) {
-    accounts.popWhere({userId}, 1)
-}
-
-function onUserUpdated(userId, displayName, avatarUrl) {
-    users.upsert({userId}, {userId, displayName, avatarUrl, loading: false})
-}
-
-function onDeviceUpdated(userId, deviceId, ed25519Key, trust, displayName,
-                         lastSeenIp, lastSeenDate) {
-}
-
-function onDeviceDeleted(userId, deviceId) {
-}

src/qml/Models/Accounts.qml

@@ -1,13 +0,0 @@
-// Copyright 2019 miruka
-// This file is part of harmonyqml, licensed under LGPLv3.
-
-import QtQuick 2.12
-import SortFilterProxyModel 0.2
-import "../Base"
-
-HListModel {
-    sorters: StringSorter {
-        roleName: "userId"
-        numericMode: true  // human numeric sort
-    }
-}

src/qml/Models/Devices.qml

@@ -1,8 +0,0 @@
-// Copyright 2019 miruka
-// This file is part of harmonyqml, licensed under LGPLv3.
-
-import QtQuick 2.12
-import SortFilterProxyModel 0.2
-import "../Base"
-
-HListModel {}

src/qml/Models/RoomCategories.qml

@@ -1,14 +0,0 @@
-// Copyright 2019 miruka
-// This file is part of harmonyqml, licensed under LGPLv3.
-
-import QtQuick 2.12
-import SortFilterProxyModel 0.2
-import "../Base"
-
-HListModel {
-    sorters: [
-        FilterSorter { ValueFilter { roleName: "name"; value: "Invites" } },
-        FilterSorter { ValueFilter { roleName: "name"; value: "Rooms" } },
-        FilterSorter { ValueFilter { roleName: "name"; value: "Left" } }
-    ]
-}

src/qml/Models/Rooms.qml

@@ -1,32 +0,0 @@
-// Copyright 2019 miruka
-// This file is part of harmonyqml, licensed under LGPLv3.
-
-import QtQuick 2.12
-import SortFilterProxyModel 0.2
-import "../Base"
-
-HListModel {
-    sorters: StringSorter {
-        roleName: "displayName"
-    }
-
-    readonly property ListModel _emptyModel: ListModel {}
-
-    function find(userId, category, roomId) {
-        if (! userId) { return }
-
-        let found = rooms.getWhere({userId, roomId, category}, 1)[0]
-        if (found) { return found }
-
-        return {
-            userId, category, roomId,
-            displayName: "",
-            avatarUrl:   "",
-            topic:       "",
-            members:     _emptyModel,
-            typingText:  "",
-            inviterId:   "",
-            loading:     true,
-        }
-    }
-}

src/qml/Models/Timelines.qml

@@ -1,24 +0,0 @@
-// Copyright 2019 miruka
-// This file is part of harmonyqml, licensed under LGPLv3.
-
-import QtQuick 2.12
-import SortFilterProxyModel 0.2
-import "../Base"
-
-HListModel {
-    function lastEventOf(roomId) {
-        for (let i = 0; i < count; i++) {
-            let item = get(i) // TODO: standardize
-            if (item.roomId == roomId) { return item }
-        }
-        return null
-    }
-
-    sorters: ExpressionSorter {
-        expression: modelLeft.isLocalEcho && ! modelRight.isLocalEcho ?
-                    true :
-                    ! modelLeft.isLocalEcho && modelRight.isLocalEcho ?
-                    false :
-                    modelLeft.date > modelRight.date // descending order
-    }
-}

src/qml/Models/Users.qml

@@ -1,26 +0,0 @@
-// Copyright 2019 miruka
-// This file is part of harmonyqml, licensed under LGPLv3.
-
-import QtQuick 2.12
-import SortFilterProxyModel 0.2
-import "../Base"
-
-HListModel {
-    function find(userId) {
-        // Happens when SortFilterProxyModel ExpressionFilter/Sorter/Role tests
-        // the expression with invalid data to establish property bindings
-        if (! userId) { return }
-
-        let found = getWhere({userId}, 1)[0]
-        if (found) { return found }
-
-        py.callCoro("request_user_update_event", [userId])
-
-        return {
-            userId,
-            displayName: "",
-            avatarUrl:   "",
-            loading:     true,
-        }
-    }
-}

src/qml/Pages/EditAccount/EditAccount.qml

@@ -14,10 +14,14 @@ HPage {
     property int avatarPreferredSize: 256
 
     property string userId: ""
-    readonly property var userInfo: users.find(userId)
-    readonly property bool ready: userInfo && ! userInfo.loading
 
-    property string headerName: userInfo ? userInfo.displayName : ""
+    readonly property bool ready: accountInfo !== "waiting"
+
+    readonly property var accountInfo: Utils.getItem(
+        modelSources["Account"] || [], "user_id", userId
+    ) || "waiting"
+
+    property string headerName: ready ? accountInfo.display_name : userId
 
     hideHeaderUnderHeight: avatarPreferredSize
     headerLabel.text: qsTr("Account settings for %1").arg(
@@ -27,6 +31,7 @@ HPage {
     HSpacer {}
 
     Repeater {
+        id: repeater
         model: ["Profile.qml", "Encryption.qml"]
 
         HRectangle {
@@ -34,6 +39,9 @@ HPage {
             Behavior on color { HColorAnimation {} }
 
             Layout.alignment: Qt.AlignCenter
+            Layout.topMargin: header.visible || index > 0 ? theme.spacing : 0
+            Layout.bottomMargin:
+                header.visible || index < repeater.count - 1? theme.spacing : 0
 
             Layout.maximumWidth: Math.min(parent.width, 640)
             Layout.preferredWidth:

src/qml/Pages/EditAccount/Profile.qml

@@ -16,7 +16,7 @@ HGridLayout {
                 userId, "set_displayname", [nameField.field.text], () => {
                     saveButton.nameChangeRunning = false
                     editAccount.headerName =
-                        Qt.binding(() => userInfo.displayName)
+                        Qt.binding(() => accountInfo.display_name)
                 }
             )
         }
@@ -40,12 +40,12 @@ HGridLayout {
     }
 
     function cancelChanges() {
-        nameField.field.text    = userInfo.displayName
+        nameField.field.text    = accountInfo.display_name
         aliasField.field.text   = aliasField.currentAlias
         fileDialog.selectedFile = ""
         fileDialog.file         = ""
 
-        editAccount.headerName = Qt.binding(() => userInfo.displayName)
+        editAccount.headerName = Qt.binding(() => accountInfo.display_name)
     }
 
     columns: 2
@@ -59,6 +59,8 @@ HGridLayout {
 
         id: avatar
         userId: editAccount.userId
+        displayName: accountInfo.display_name
+        avatarUrl: accountInfo.avatar_url
         imageUrl: fileDialog.selectedFile || fileDialog.file || defaultImageUrl
         toolTipImageUrl: ""
 
@@ -72,16 +74,22 @@ HGridLayout {
             visible: opacity > 0
             opacity: ! fileDialog.dialog.visible &&
                      (! avatar.imageUrl || avatar.hovered) ? 1 : 0
-            Behavior on opacity { HNumberAnimation {} }
 
             anchors.fill: parent
-            color: Utils.hsla(0, 0, 0, avatar.imageUrl ? 0.7 : 1)
+            color: Utils.hsluv(0, 0, 0,
+                (! avatar.imageUrl && overlayHover.hovered) ? 0.9 : 0.7
+            )
+
+            Behavior on opacity { HNumberAnimation {} }
+            Behavior on color { HColorAnimation {} }
 
             HColumnLayout {
                 anchors.centerIn: parent
                 spacing: currentSpacing
                 width: parent.width
 
+                HoverHandler { id: overlayHover }
+
                 HIcon {
                     svgName: "upload-avatar"
                     dimension: 64
@@ -92,7 +100,11 @@ HGridLayout {
 
                 HLabel {
                     text: qsTr("Upload profile picture")
-                    color: Utils.hsla(0, 0, 90, 1)
+                    color: (! avatar.imageUrl && overlayHover.hovered) ?
+                           Qt.lighter(theme.colors.accentText, 1.2) :
+                           Utils.hsluv(0, 0, 90, 1)
+                    Behavior on color { HColorAnimation {} }
+
                     font.pixelSize: theme.fontSize.big *
                                     avatar.height / avatarPreferredSize
                     wrapMode: Text.WordWrap
@@ -107,7 +119,7 @@ HGridLayout {
             id: fileDialog
             fileType: HFileDialogOpener.FileType.Images
             dialog.title: qsTr("Select profile picture for %1")
-                              .arg(userInfo.displayName)
+                              .arg(accountInfo.display_name)
         }
     }
 
@@ -129,14 +141,14 @@ HGridLayout {
             }
 
             HLabeledTextField {
-                property bool changed: field.text != userInfo.displayName
+                property bool changed: field.text != accountInfo.display_name
 
                 property string fText: field.text
                 onFTextChanged: editAccount.headerName = field.text
 
                 id: nameField
                 label.text: qsTr("Display name:")
-                field.text: userInfo.displayName
+                field.text: accountInfo.display_name
                 field.onAccepted: applyChanges()
 
                 Layout.fillWidth: true

src/qml/Python.qml

@@ -4,7 +4,7 @@
 import QtQuick 2.12
 import QtQuick.Controls 2.12
 import io.thp.pyotherside 1.5
-import "EventHandlers/includes.js" as EventHandlers
+import "event_handlers.js" as EventHandlers
 
 Python {
     id: py
@@ -60,21 +60,17 @@ Python {
         addImportPath("src")
         addImportPath("qrc:/")
         importNames("python", ["APP"], () => {
-            call("APP.is_debug_on", [Qt.application.arguments], on => {
+            loadSettings(() => {
-                window.debug = on
+                callCoro("saved_accounts.any_saved", [], any => {
+                    py.ready = true
+                    willLoadAccounts(any)
 
-                loadSettings(() => {
+                    if (any) {
-                    callCoro("saved_accounts.any_saved", [], any => {
+                        py.loadingAccounts = true
-                        py.ready = true
+                        py.callCoro("load_saved_accounts", [], () => {
-                        willLoadAccounts(any)
+                            py.loadingAccounts = false
-
+                        })
-                        if (any) {
+                    }
-                            py.loadingAccounts = true
-                            py.callCoro("load_saved_accounts", [], () => {
-                                py.loadingAccounts = false
-                            })
-                        }
-                    })
                 })
             })
         })

src/qml/Shortcuts.qml

@@ -34,7 +34,7 @@ Item {
 
     Shortcut {
         sequences: settings.keys ? settings.keys.startDebugger : []
-        onActivated: if (window.debug) { py.call("APP.pdb") }
+        onActivated: if (debugMode) { py.call("APP.pdb") }
     }
 
     /*

src/qml/SidePane/AccountDelegate.qml

@@ -10,14 +10,14 @@ Column {
     width: parent.width
     spacing: theme.spacing / 2
 
-    property var userInfo: users.find(model.userId)
     property bool expanded: true
+    readonly property var modelItem: model
 
     Component.onCompleted:
-        expanded = ! window.uiState.collapseAccounts[model.userId]
+        expanded = ! window.uiState.collapseAccounts[model.user_id]
 
     onExpandedChanged: {
-        window.uiState.collapseAccounts[model.userId] = ! expanded
+        window.uiState.collapseAccounts[model.user_id] = ! expanded
         window.uiStateChanged()
     }
 
@@ -28,7 +28,7 @@ Column {
 
         TapHandler {
             onTapped: pageStack.showPage(
-                "EditAccount/EditAccount", { "userId": model.userId }
+                "EditAccount/EditAccount", { "userId": model.user_id }
             )
         }
 
@@ -38,46 +38,22 @@ Column {
 
             HUserAvatar {
                 id: avatar
-                // Need to do this because conflict with the model property
+                userId: model.user_id
-                Component.onCompleted: userId = model.userId
+                displayName: model.display_name
+                avatarUrl: model.avatar_url
             }
 
-            HColumnLayout {
+            HLabel {
+                id: accountLabel
+                color: theme.sidePane.account.name
+                text: model.display_name || model.user_id
+                font.pixelSize: theme.fontSize.big
+                elide: HLabel.ElideRight
+                leftPadding: sidePane.currentSpacing
+                rightPadding: leftPadding
+
                 Layout.fillWidth: true
                 Layout.fillHeight: true
-
-                HLabel {
-                    id: accountLabel
-                    color: theme.sidePane.account.name
-                    text: userInfo.displayName || model.userId
-                    font.pixelSize: theme.fontSize.big
-                    elide: HLabel.ElideRight
-                    leftPadding: sidePane.currentSpacing
-                    rightPadding: leftPadding
-
-                    Layout.fillWidth: true
-                }
-
-                HTextField {
-                    visible: false // TODO
-
-                    id: statusEdit
-                    // text: userInfo.statusMessage
-                    placeholderText: qsTr("Set status message")
-                    font.pixelSize: theme.fontSize.small
-                    background: null
-                    bordered: false
-
-                    padding: 0
-                    leftPadding: accountLabel.leftPadding
-                    rightPadding: leftPadding
-                    Layout.fillWidth: true
-
-                    onEditingFinished: {
-                        //Backend.setStatusMessage(model.userId, text)  TODO
-                        pageStack.forceActiveFocus()
-                    }
-                }
             }
 
             ExpandButton {
@@ -88,17 +64,15 @@ Column {
         }
     }
 
-    RoomCategoriesList {
+    RoomList {
         id: roomCategoriesList
         visible: height > 0
         width: parent.width
         height: childrenRect.height * (accountDelegate.expanded ? 1 : 0)
         clip: heightAnimation.running
 
-        userId: userInfo.userId
+        userId: modelItem.user_id
 
-        Behavior on height {
+        Behavior on height { HNumberAnimation { id: heightAnimation } }
-            HNumberAnimation { id: heightAnimation }
-        }
     }
 }

src/qml/SidePane/AccountList.qml

@@ -9,6 +9,10 @@ HListView {
     id: accountList
     clip: true
 
-    model: accounts
+    model: HListModel {
+        keyField: "user_id"
+        source: modelSources["Account"] || []
+    }
+
     delegate: AccountDelegate {}
 }

src/qml/SidePane/ExpandButton.qml

@@ -2,7 +2,6 @@
 // This file is part of harmonyqml, licensed under LGPLv3.
 
 import QtQuick 2.12
-import QtQuick.Layouts 1.12
 import "../Base"
 
 HUIButton {

src/qml/SidePane/RoomCategoriesList.qml

@@ -1,23 +0,0 @@
-// Copyright 2019 miruka
-// This file is part of harmonyqml, licensed under LGPLv3.
-
-import QtQuick 2.12
-import QtQuick.Layouts 1.12
-import SortFilterProxyModel 0.2
-import "../Base"
-
-HFixedListView {
-    property string userId: ""
-
-    id: roomCategoriesList
-
-    model: SortFilterProxyModel {
-        sourceModel: roomCategories
-        filters: ValueFilter {
-            roleName: "userId"
-            value: userId
-        }
-    }
-
-    delegate: RoomCategoryDelegate {}
-}

src/qml/SidePane/RoomCategoryDelegate.qml

@@ -1,73 +0,0 @@
-// Copyright 2019 miruka
-// This file is part of harmonyqml, licensed under LGPLv3.
-
-import QtQuick 2.12
-import QtQuick.Layouts 1.12
-import "../Base"
-
-Column {
-    id: roomCategoryDelegate
-    width: roomCategoriesList.width
-
-    property int normalHeight: childrenRect.height  // avoid binding loop
-
-    opacity: roomList.model.count > 0 ? 1 : 0
-    height: normalHeight * opacity
-    visible: opacity > 0
-
-    Behavior on opacity { HNumberAnimation {} }
-
-    property string roomListUserId: userId
-    property bool expanded: true
-
-    Component.onCompleted: {
-        if (! window.uiState.collapseCategories[model.userId]) {
-            window.uiState.collapseCategories[model.userId] = {}
-            window.uiStateChanged()
-        }
-
-        expanded = !window.uiState.collapseCategories[model.userId][model.name]
-    }
-
-    onExpandedChanged: {
-        window.uiState.collapseCategories[model.userId][model.name] = !expanded
-        window.uiStateChanged()
-    }
-
-    HRowLayout {
-        width: parent.width
-
-        HLabel {
-            id: roomCategoryLabel
-            text: model.name
-            font.weight: Font.DemiBold
-            elide: Text.ElideRight
-            topPadding: theme.spacing / 2
-            bottomPadding: topPadding
-
-            Layout.leftMargin: sidePane.currentSpacing
-            Layout.fillWidth: true
-        }
-
-        ExpandButton {
-            expandableItem: roomCategoryDelegate
-            iconDimension: 12
-        }
-    }
-
-    RoomList {
-        id: roomList
-        visible: height > 0
-        width: roomCategoriesList.width - accountList.Layout.leftMargin
-        opacity: roomCategoryDelegate.expanded ? 1 : 0
-        height: childrenRect.height * opacity
-        clip: listHeightAnimation.running
-
-        userId: roomListUserId
-        category: name
-
-        Behavior on opacity {
-            HNumberAnimation { id: listHeightAnimation }
-        }
-    }
-}

src/qml/SidePane/RoomDelegate.qml

@@ -12,11 +12,7 @@ HInteractiveRectangle {
     height: childrenRect.height
     color: theme.sidePane.room.background
 
-    TapHandler {
+    TapHandler { onTapped: pageStack.showRoom(userId, model.room_id) }
-        onTapped: pageStack.showRoom(
-            roomList.userId, roomList.category, model.roomId
-        )
-    }
 
     Row {
         width: parent.width - leftPadding * 2
@@ -30,8 +26,8 @@ HInteractiveRectangle {
 
             HRoomAvatar {
                 id: roomAvatar
-                userId: model.userId
+                displayName: model.display_name
-                roomId: model.roomId
+                avatarUrl: model.avatar_url
             }
 
             HColumnLayout {
@@ -40,9 +36,9 @@ HInteractiveRectangle {
                 HLabel {
                     id: roomLabel
                     color: theme.sidePane.room.name
-                    text: model.displayName || "<i>Empty room</i>"
+                    text: model.display_name || "<i>Empty room</i>"
                     textFormat:
-                        model.displayName? Text.PlainText : Text.StyledText
+                        model.display_name? Text.PlainText : Text.StyledText
                     elide: Text.ElideRight
                     verticalAlignment: Qt.AlignVCenter
 
@@ -50,30 +46,27 @@ HInteractiveRectangle {
                 }
 
                 HRichLabel {
-                    function getText(ev) {
-                        if (! ev) { return "" }
-
-                        if (ev.eventType == "RoomMessageEmote" ||
-                            ! ev.eventType.startsWith("RoomMessage"))
-                        {
-                            return Utils.processedEventText(ev)
-                        }
-
-                        return Utils.coloredNameHtml(
-                            users.find(ev.senderId).displayName,
-                            ev.senderId
-                        ) + ": " + py.callSync("inlinify", [ev.content])
-                    }
-
-                    // Have to do it like this to avoid binding loop
-                    property var lastEv: timelines.lastEventOf(model.roomId)
-                    onLastEvChanged: text = getText(lastEv)
-
                     id: subtitleLabel
                     color: theme.sidePane.room.subtitle
                     visible: Boolean(text)
                     textFormat: Text.StyledText
 
+                    text: {
+                        if (! model.last_event) { return "" }
+
+                        let ev = model.last_event
+
+                        if (ev.event_type === "RoomMessageEmote" ||
+                            ! ev.event_type.startsWith("RoomMessage")) {
+                            return Utils.processedEventText(ev)
+                        }
+
+                        return Utils.coloredNameHtml(
+                            ev.sender_name, ev.sender_id
+                        ) + ": " + ev.inline_content
+                    }
+
+
                     font.pixelSize: theme.fontSize.small
                     elide: Text.ElideRight
 

src/qml/SidePane/RoomList.qml

@@ -2,38 +2,19 @@
 // This file is part of harmonyqml, licensed under LGPLv3.
 
 import QtQuick 2.12
-import QtQuick.Layouts 1.12
-import SortFilterProxyModel 0.2
 import "../Base"
 import "../utils.js" as Utils
 
 HFixedListView {
     id: roomList
-
     property string userId: ""
-    property string category: ""
 
-    model: SortFilterProxyModel {
+    model: HListModel {
-        sourceModel: rooms
+        source: Utils.filterModelSource(
-        filters: AllOf {
+            modelSources[["Room", userId]] || [],
-            ValueFilter {
+            paneToolBar.roomFilter,
-                roleName: "category"
+        )
-                value: category
+        keyField: "room_id"
-            }
-
-            ValueFilter {
-                roleName: "userId"
-                value: userId
-            }
-
-            ExpressionFilter {
-                // Utils... won't work directly in expression?
-                function filterIt(filter, text) {
-                    return Utils.filterMatches(filter, text)
-                }
-                expression: filterIt(paneToolBar.roomFilter, displayName)
-            }
-        }
     }
 
     delegate: RoomDelegate {}

src/qml/UI.qml

@@ -37,7 +37,7 @@ HRectangle {
             let props = window.uiState.pageProperties
 
             if (page == "Chat/Chat.qml") {
-                pageStack.showRoom(props.userId, props.category, props.roomId)
+                pageStack.showRoom(props.userId, props.roomId)
             } else {
                 pageStack.show(page, props)
             }
@@ -45,11 +45,11 @@ HRectangle {
     }
 
     property bool accountsPresent:
-        accounts.count > 0 || py.loadingAccounts
+        (modelSources["Account"] || []).length > 0 || py.loadingAccounts
 
     HImage {
-        visible: Boolean(Qt.resolvedUrl(source))
         id: mainUIBackground
+        visible: Boolean(Qt.resolvedUrl(source))
         fillMode: Image.PreserveAspectCrop
         source: theme.ui.image
         sourceSize.width: Screen.width
@@ -99,12 +99,11 @@ HRectangle {
                 window.uiStateChanged()
             }
 
-            function showRoom(userId, category, roomId) {
+            function showRoom(userId, roomId) {
-                let roomInfo = rooms.find(userId, category, roomId)
+                show("Chat/Chat.qml", {userId, roomId})
-                show("Chat/Chat.qml", {roomInfo})
 
                 window.uiState.page           = "Chat/Chat.qml"
-                window.uiState.pageProperties = {userId, category, roomId}
+                window.uiState.pageProperties = {userId, roomId}
                 window.uiStateChanged()
             }
 

src/qml/Window.qml

@@ -4,7 +4,6 @@
 import QtQuick 2.12
 import QtQuick.Controls 2.12
 import "Base"
-import "Models"
 
 ApplicationWindow {
     id: window
@@ -14,23 +13,14 @@ ApplicationWindow {
     width: 640
     height: 480
     visible: true
-    title: "Harmony QML"
     color: "transparent"
 
-    Component.onCompleted: {
+    // Note: For JS object variables, the corresponding method to notify
-        Qt.application.organization = "harmonyqml"
+    // key/value changes must be called manually, e.g. settingsChanged().
-        Qt.application.name         = "harmonyqml"
+    property var modelSources: ({})
-        Qt.application.displayName  = "Harmony QML"
-        Qt.application.version      = "0.1.0"
-        window.ready                = true
-    }
-
-    property bool debug: false
-    property bool ready: false
 
     property var mainUI: null
 
-    // Note: settingsChanged(), uiStateChanged(), etc must be called manually
     property var settings: ({})
     onSettingsChanged: py.saveConfig("ui_settings", settings)
 
@@ -42,27 +32,16 @@ ApplicationWindow {
     Shortcuts { id: shortcuts}
     Python    { id: py }
 
-    // Models
+    Loader {
-    Accounts       { id: accounts }
-    Devices        { id: devices }
-    RoomCategories { id: roomCategories }
-    Rooms          { id: rooms }
-    Timelines      { id: timelines }
-    Users          { id: users }
-
-    LoadingScreen {
-        id: loadingScreen
         anchors.fill: parent
-        visible: uiLoader.scale < 1
+        source: py.ready ? "" : "LoadingScreen.qml"
     }
 
     Loader {
         id: uiLoader
         anchors.fill: parent
-
+        scale: py.ready ? 1 : 0.5
-        property bool ready: window.ready && py.ready
+        source: py.ready ? "UI.qml" : ""
-        scale: uiLoader.ready ? 1 : 0.5
-        source: uiLoader.ready ? "UI.qml" : ""
 
         Behavior on scale { HNumberAnimation {} }
     }

src/qml/event_handlers.js

@@ -3,11 +3,19 @@
 
 "use strict"
 
+
 function onExitRequested(exitCode) {
     Qt.exit(exitCode)
 }
 
+
 function onCoroutineDone(uuid, result) {
     py.pendingCoroutines[uuid](result)
     delete pendingCoroutines[uuid]
 }
+
+
+function onModelUpdated(syncId, data, serializedSyncId) {
+    window.modelSources[serializedSyncId] = data
+    window.modelSourcesChanged()
+}

src/qml/utils.js

@@ -19,20 +19,6 @@ function hsla(hue, saturation, lightness, alpha=1.0) {
 }
 
 
-function arrayToModelItem(keysName, array) {
-    // Convert an array to an object suitable to be in a model, example:
-    // [1, 2, 3] → [{keysName: 1}, {keysName: 2}, {keysName: 3}]
-    let items = []
-
-    for (let item of array) {
-        let obj       = {}
-        obj[keysName] = item
-        items.push(obj)
-    }
-    return items
-}
-
-
 function hueFrom(string) {
     // Calculate and return a unique hue between 0 and 360 for the string
     let hue = 0
@@ -52,7 +38,7 @@ function nameColor(name) {
 }
 
 
-function coloredNameHtml(name, userId, displayText=null) {
+function coloredNameHtml(name, userId, displayText=null, disambiguate=false) {
     // substring: remove leading @
     return "<font color='" + nameColor(name || userId.substring(1)) + "'>" +
            escapeHtml(displayText || name || userId) +
@@ -71,19 +57,20 @@ function escapeHtml(string) {
 
 
 function processedEventText(ev) {
-    if (ev.eventType == "RoomMessageEmote") {
+    if (ev.event_type == "RoomMessageEmote") {
-        let name = users.find(ev.senderId).displayName
+        return "<i>" +
-        return "<i>" + coloredNameHtml(name) + " " + ev.content + "</i>"
+               coloredNameHtml(ev.sender_name, ev.sender_id) + " " +
+               ev.content + "</i>"
     }
 
-    if (ev.eventType.startsWith("RoomMessage")) { return ev.content }
+    if (ev.event_type.startsWith("RoomMessage")) { return ev.content }
 
-    let name = users.find(ev.senderId).displayName
+    let text = qsTr(ev.content).arg(
-    let text = qsTr(ev.content).arg(coloredNameHtml(name, ev.senderId))
+        coloredNameHtml(ev.sender_name, ev.sender_id)
+    )
 
-    if (text.includes("%2") && ev.targetUserId) {
+    if (text.includes("%2") && ev.target_id) {
-        let tname = users.find(ev.targetUserId).displayName
+        text = text.arg(coloredNameHtml(ev.target_name, ev.target_id))
-        text = text.arg(coloredNameHtml(tname, ev.targetUserId))
     }
 
     return text
@@ -105,6 +92,17 @@ function filterMatches(filter, text) {
 }
 
 
+function filterModelSource(source, filter_text, property="filter_string") {
+    if (! filter_text) { return source }
+
+    let results = []
+    for (let item of source) {
+        if (filterMatches(filter_text, item[property])) { results.push(item) }
+    }
+    return results
+}
+
+
 function thumbnailParametersFor(width, height) {
     // https://matrix.org/docs/spec/client_server/latest#thumbnails
 
@@ -127,3 +125,11 @@ function thumbnailParametersFor(width, height) {
 function minutesBetween(date1, date2) {
     return Math.round((((date2 - date1) % 86400000) % 3600000) / 60000)
 }
+
+
+function getItem(array, mainKey, value) {
+    for (let i = 0; i < array.length; i++) {
+        if (array[i][mainKey] === value) { return array[i] }
+    }
+    return undefined
+}

src/themes/Default.qpl

@@ -196,7 +196,7 @@ chat:
         color body:          colors.text
         color date:          colors.dimText
 
-        color greenText: hsluv(80, colors.saturation * 2.25, 80)
+        color greenText: hsluv(135, colors.saturation * 2.25, 80)
         color link:      colors.link
         color code:      colors.code
 

submodules/SortFilterProxyModel

@@ -1 +0,0 @@
-Subproject commit 770789ee484abf69c230cbf1b64f39823e79a181

submodules/qsyncable

@@ -0,0 +1 @@
+Subproject commit f5ca07b71cecda685d0dd4b3c74d2fb2ca71f711