Debugging tips

Warning

These notes exist to assist developers in debugging their hosts in real time. Do not follow any advice here without being prepared for the consequences. Requests for support will be void on improperly-tampered hosts.

Run with full output

Launch Python in developer mode to get stdlib deprecations and asyncio resource debugging:

$ python3 -X dev -X tracemalloc -m immp config.yml

Configure your logging for the runner to output at DEBUG level:

logging:
  version: 1
  filters:
    local:
      (): immp.LocalFilter
  handlers:
    console:
      class: logging.StreamHandler
      filters: [local]
    internal:
      class: logging.FileHandle
      filename: internal.log
  loggers:
    asyncio:
      level: DEBUG
  root:
    level: DEBUG
    handlers: [console, internal]
  disable_existing_loggers: false

This will print local debug output (from the immp namespace) to stdout, and everything (IMMP hooks/plugs as well as their dependencies) to a log file called internal.log.

Enable the async shell

The Shell is an invaluable tool for poking internals, so make sure to have it enabled and accessible.

Checking plug tasks

If a plug is no longer picking up messages or showing signs of life, it may have a background task that failed without restarting. For example, the Telegram plug has a long-poll task to receive updates, stored on the plug’s _receive attribute:

>>> plug._receive
<Task finished coro=<TelegramPlug._poll() done,
      defined at /opt/IMMP/immp/plug/telegram.py:1234>
      exception=RuntimeError('Catastrophic failure',)>

If the task is finished with an exception, call asyncio.Task.result() to get a traceback. A restart of the plug (Plug.stop(), Plug.start()) should get things moving again.

Checking the stream

If a plug is logging creation of its own messages, but the host stream isn’t receiving them, check if its messages are stuck in the queue, and refer to Host._stream:

>>> plug._queue.qsize()
1
>>> host._stream
<PlugStream: 0 done, 12 pending>

If all are pending, then every plug should be idle and awaiting a new message. If more than one are done for more than a moment, then one is likely wedged and holding up the others. You can try transplanting a new stream by re-calling Host.process():

>>> host._stream = host._process = None
>>> host._process = asyncio.ensure_future(host.process())

Replaying messages into a hook

First, fetch the message(s) you’re missing:

>>> msgs = (await host['my-channel'].history())[-3:]

Then feed them into a hook:

>>> for msg in msgs:
...     await host['sync'].on_receive(msg, msg, True)

For a SyncHook, this will send any as-yet-unsynced messages to linked channels.