TL;DR

For the list and search commands you now have three new command-line arguments:

1
2
3
cobib search --decode-latex "naïve"    # will still find `na{\"\i}ve`
cobib search --decode-unicode "naive"  # will still find `naïve`
cobib search --fuzziness 1 "naive"     # also finds `naïve`

For the list command, these argument affect the filter mechanism. You can set your own default values via the respective settings in your configuration file.

What is fuzzy finding?

Fuzzy finding refers to approximate string matching¹ which allows you to relate two strings to each other if they are approximately equal. What exactly this approximation entails can differ; but in general there are three primitive kinds of errors:

• insertions: cot -> cost
• deletions: coat -> cot
• substitutions: coat -> cost

The more errors you allow, the more approximate (or “fuzzy”) your comparison becomes.

How can you use this in coBib?

coBib v5.1.0 has added support for approximate string comparisons for two use cases:

• when using filters to limit the output of the list command
• when searching your database with the search command

In both of these cases, coBib relies on the functionality provided by the regex² package. This is a new optional dependency of coBib, so be sure to install it if you want to use this functionality.

Since the regex package is a drop-in replacement for Python’s builtin re³ module, and coBib already supported standard regex patterns for both of these use cases, the extension to support fuzzy matching was almost straight forward.

In the following, we will explore the new ways for fuzzy finding entries in your database, using the following entry as our example:

1
2
3
4
5
6
7
8
9
@unpublished{Battaglia2024,
 archivePrefix = {arXiv},
 arxivid = {2404.18737v1},
 author = {Battaglia, Stefano and Rossmannek, Max and Rybkin, Vladimir V. and Tavernelli, Ivano and Hutter, J{\"u}rg},
 eprint = {http://arxiv.org/abs/2404.18737v1},
 primaryClass = {physics.chem-ph},
 title = {A general framework for active space embedding methods: applications in quantum computing},
 year = {2024}
}

Using regex patterns directly

You have full control over the regex patterns that are used for filter matching and/or searching. As such, you can use all of the additional features provided by the regex package.

In the following, we will review how to write custom patterns to allow for the different kinds of errors listed above.

Insertion errors

We can allow a number of characters to be inserted into our query string by writing a regex pattern of the form: (Rossmanek){i<2}. This will match all occurrences of Rossmanek with up to one additional arbitrary characters inserted into it, e.g. Rossmannek.

Let’s try it out:

1
cobib list ++author "(Rossmanek){i<2}"

And indeed, the query was able to find our entry above:

1
2
3
4
5
6
7
8
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ label          ┃ title                         ┃ author                        ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Battaglia2024  │ A general framework for       │ Battaglia, Stefano and        │
│                │ active space embedding        │ Rossmannek, Max and Rybkin,   │
│                │ methods: applications in      │ Vladimir V. and Tavernelli,   │
│                │ quantum computing             │ Ivano and Hutter, J{\"u}rg    │
└────────────────┴───────────────────────────────┴───────────────────────────────┘

Deletion errors

Deletions work just like insertions above, but using d instead of i:

1
cobib list ++author "(Rossmanneck){d<2}"

(This matches the entry just like in the previous example.)

Substitution errors

The third kind of error are substitutions which are equivalent to the insertion and removal of a character at the same location. You can specify their number using s:

1
cobib list ++author "(Maz){s<2}"

(This matches the entry just like in the previous example.)

Allowing multiple errors types

You can even combine these errors types. Below we allow an insertion and deletion to occur:

1
cobib list ++author "(Rossmaneck){i<2,d<2}"

(And again, this matches the entry just like in the previous example.)

Generic errors

Finally, you can allow any type of error (i.e. without specifying its type) using e:

1
cobib list ++author "(Rossmaneck){e<3}"

The --fuzziness command-line option (or -z for short)

Since writing these regex patterns can be a bit cumbersome, the --fuzziness argument makes enabling fuzzy finding a lot easier. It is best explained using an example:

1
cobib list --fuzziness 3 ++author "Rossmaneck"

This command is equivalent to the previous example. Here is what happens under the hood:

1. the provided string gets wrapped in parentheses: Rossmaneck -> (Rossmaneck)
2. the provided value of fuzziness gets used as the allowed number of arbitrary errors: {e<3}
3. the two parts get concatenated to form the final query: (Rossmaneck){e<3}

You can also use -z as a short-hand instead of typing out --fuzziness every time.

Setting a default --fuzziness value

If you want to enable approximate string matching by default, you can do so by including the following in your configuration file:

1
2
3
from cobib.config import config

config.commands.list_.fuzziness = 3

And although we only used the list command in our examples above, the search command provides the same features:

1
config.commands.search.fuzziness = 3

Finally, you can always overwrite this default value with the command-line option. For example, --fuzziness 0 will disable fuzzy finding.

Dealing with LaTeX sequences

Accounting for typos by fuzzy matching is nice, but it would require a large threshold of --fuzziness to be able to deal with LaTeX sequences such as the one found in our example: J{\"u}rg.

Instead, coBib v5.1.0 added another way to deal with LaTeX sequences: the --decode-latex command-line option. Setting this will convert LaTeX code to plain text using pylatexenc’s⁴ encoder.

Note, that this only works approximately and only for sequences which can actually be rendered in plain text using Unicode characters.

This allows you to do the following:

1
cobib list --decode-latex ++author "Jürg"

which will again match our entry Battaglia2024 shown above.

The same command-line setting is available for the search command (where it even comes with the -l short-hand).

Configuring the default of --decode-latex

Just like for the --fuzziness setting, you can also configure the default value for the --decode-latex option:

1
2
config.commands.list_.decode_latex = True
config.commands.search.decode_latex = True

If you enable this setting, you can still overwrite it from the command-line using the --no-decode-latex argument (or -D for the search command).

Dealing with Unicode characters

We can take the concept of special character decoding one step further by also leveraging text-unidecode⁵ to convert Unicode characters into their approximate ASCII version.

Pairing this with LaTeX decoding allows us to match our entry with an even simpler query:

1
cobib list --decode-latex --decode-unicode ++author "Jurg"

And, once again, you can do the same within the search command (where it also comes with the -u short-hand).

Configuring the default of --decode-unicode

Of course, you can also configure the defaults for these options:

1
2
config.commands.list_.decode_unicode = True
config.commands.search.decode_unicode = True

And you can overwrite the enabled settings using --no-decode-unicode (or -U for the search command).

Fuzzy search highlights

As a final little example, here you can see that even the highlighting of search results works properly for fuzzy matches:

The short answer: a LOT!

Life in general, has been very busy. But I more or less steadily refactored coBib to integrate it with these two amazing projects by Textualize: rich and textual. This means, coBib now has a shiny new interface in both, its CLI and TUI, and its code has become a lot more maintainable for the future to come!

If you are interested in finding out what this entails, continue reading. If you simply want to see some screenshots, skim through the page as you like.

In any case: be sure to upgrade coBib now, to get v4.0.0 installed on your machine!

The longer answer

It all started when I first heard about textual - a then new project from the developer of rich, a “library for rich text and beautiful formatting in the terminal”¹. I had been keeping an eye on rich for a while as it would bring some nice benefits to coBib’s CLI, and when textual got announced, I knew that I wanted to refactor the archaic ncurses-based TUI of coBib to become more modern and, especially, a much more maintainable codebase!

So even though textual was still in its infancy, the refactoring began…

Rich integration

I first set out to integrate rich. In doing so, I revisit the entire design of coBib’s command classes and also added a “porcelain” output mode for easy parsing. Without getting too much into the gory details, here are some screenshots to gorge yourself in:

Textual integration

While the rich integration was done fairly early in the development cycle, integrating textual took a lot longer, in part also due to it being alpha software with some features I needed simply not being available yet. But it has matured a lot over the last year and I am happy to ship a nice new interface with coBib out today!

Here are some screenshots²:

Functionally, the TUI still works almost exactly as before. You might notice a few minor differences, some of which I plan to iron out in upcoming feature releases, but if you find anything missing or not working, please feel free to open an issue.

Other noteworthy changes

Besides this major UI refactoring, I would like to briefly highlight two other noteworthy changes.

Configuration changes

I refactored the configuration implementation as a Python dataclass³. This makes the code both, more maintainable and significantly easier to document online. However, this means if you used to do the following in your configuration file:

1
config["database"]["git"] = True

You will now need to change this to the following:

1
config.database.git = True

Furthermore, as a consequence of the commands refactoring, a lot of the function signatures of the built-in events have changed. So be sure to update those accordingly.

Label disambiguation

coBib used to be able to detect if an entry already existed in your database and would avoid re-adding it as a duplicate. However, this functionality regressed when I added the label disambiguation feature. With v4.0.0, you will now have the option to decide what to do interactively. And to ease your choice, you will be presented with a side-by-side comparison of the existing and to-be-added entry! Take a look at the screenshot below to see what you might expect from this:

Automatic File-Download

Since version 3.2.0, coBib will attempt to download the PDF of a newly added entry automatically. Out of the box, this feature will work for addition via arXiv IDs. Support for arbitrary DOI entries needs to be configured by providing a dictionary of URL maps. Other parsers do not provide this functionality, yet. But for now, let us dive into a simple example…

Entries added via arXiv IDs

Consider adding a new entry via an arXiv ID using the following command:

1
cobib add --arxiv 2009.10095

This will add the new entry Egger2020 to your database. Furthermore, it will automatically download the PDF version of this article and save it to ~/.local/share/cobib/Egger2020.pdf. You can follow the download progress on the command-line via the commands output:

1
2
3
4
Downloading: [================                        ]   41.8%   1 MB / 2 MB
...
Downloading: [========================================]  100.0%   2 MB / 2 MB
Successfully downloaded ~/.local/share/cobib/Egger2020.pdf

If you want to change the default download location, you can do so by overwriting the config.utils.file_downloader.default_location setting in your config file (read this blogpost for more information on coBibs Python configuration system).

Entries added via DOIs

Now, let us turn towards downloading DOI entries. The reason why this case is more difficult, is that a DOI can resolve to a landing page of any Journal. Thus, there is no single recipe to determine the link to the articles PDF file.

To circumvent this problem, the user needs to manually specify a dictionary of URL mappings in the config.utils.file_downloader.url_map setting. The key-value pairs of this dictionary should be formatted similar to this example:

1
2
3
config.utils.file_downloader.url_map[
    r"(.+)://quantum-journal.org/papers/([^/]+)"
] = r"\1://quantum-journal.org/papers/\2/pdf/"

Let us walk through this snippet to understand what it does:

1. The key r"(.+)://quantum-journal.org/papers/([^/]+)" is a raw Python-string containing a regex pattern. This pattern is made up of the following parts:
   • (.+)://: this matches any file protocol (e.g. http:// or https://) and captures the string preceding ://.
   • quantum-journal.org/papers/: this is the main body of the URL and will be matched as plain text.
   • ([^/]+): this is another regex capture matching any characters other than /.
2. The value of this entry, r"\1://quantum-journal.org/papers/\2/pdf/", is also a regex pattern which is used as a substitution pattern. The placeholders \1 and \2 will be replaced by the first and second captured string, respectively.

Here is a concrete example; when adding a new entry from a DOI via:

1
cobib add --doi 10.22331/q-2021-06-17-479

, the URL of the Journals landing page will resolve to https://quantum-journal.org/papers/q-2021-06-17-479/. This URL matches the key which we discussed above:

1
2
3
https://quantum-journal.org/papers/q-2021-06-17-479/
vvvvv                              vvvvvvvvvvvvvvvv
(.+) ://quantum-journal.org/papers/([^/]+)         /

Thus, after inserting the captured groups into the value pattern, we obtain the URL:

1
2
3
\1   ://quantum-journal.org/papers/\2              /pdf/
^^^^^                              ^^^^^^^^^^^^^^^^
https://quantum-journal.org/papers/q-2021-06-17-479/pdf/

, which is the correct URL pointing to the PDF version of this article.

As you can see, this recipe allows you to map from any Journals landing page to the URL pointing to the articles PDF. However, figuring out the correct regex pattern can be difficult at times. I am planning to collect a list of working patterns on coBibs Wiki in the future and encourage everyone to add their patterns for the benefit of everybody else.

Improvements to the modify Command

Two important changes have been made to how the filtering of the list command and the modifications of the modify command are interpreted:

1. Filter values are now interpreted as regex patterns (which we already looked at above): This means you can perform more advanced filtering in a similar way to how you can search through your database for regex patterns. Here is a simple example:

   1
    cobib list ++label "\D+_\d+"
   

   This will list all entries with a label matching the regex pattern \D+_\d+. In words, this pattern will match any label which starts with multiple non-digit characters, followed by an underscore and ends with multiple digits. This is a common label format used by many Journals, e.g. Rossmannek_2021.

2. Modifications get evaluated as f-strings: These kind of strings are a powerful feature of the Python language and allow you to use variables which will be interpreted literally. Within coBibs scope that means you can use any field name of your entry as a variable which will be replaced by its value in the current entry. Take an example again:

   1
    cobib modify "pages:{pages.replace('--', '-')}" -- ...
   

   In this example, the pages field is modified in such a way, that the occurrence of the string, --, gets replaced by a single dash, -. This showcases well, how having the fields available as proper variables allows you to perform advanced modifications which could otherwise be a long and manual editing task.

Finally, combining these two features, allows you to perform even more powerful modifications to your database. My favorite example is the following:

1
cobib modify "label:{label.replace('_', '')}" -- ++label "\D+_\d+"

This combines our previous two examples into one, renaming all entries which follow the naming pattern <string>_<number> (typically referring to <author>_<year>, e.g. Rossmannek_2021) to follow the pattern <string><number> (e.g. Rossmannek2021).

I consider this a killer feature of coBib making it standout against other bibliography-management tools out there.

Journal Abbreviations

In version 3.2.0, I also added the new Journal Abbreviations. With these, the export command can be used to automatically convert the journal fields of all entries to their abbreviated version. This is a useful functionality to ensure compliance with Journal requirements for BibTeX citation styles.

In order to use this feature, you must configure a list of abbreviations in the config.utils.journal_abbreviations setting. As an example, consider this snippet:

1
config.utils.journal_abbreviations += [("Annalen der Physik", "Ann. Phys.")]

Here, we are adding a new Journal Abbreviation for the “Annalen der Physik” journal, whose short-hand name is “Ann. Phys.”. Note, that we include the punctuation as part of the abbreviation which can be easily removed during exporting, if necessary.

Now, let us look at this example in action! Since we are only interested in the journal field, I will keep the entry to a bare minimum:

1
2
3
4
5
---
einstein:
  ENTRYTYPE: article
  journal: Annalen der Physik
...

We can export it in three different ways to a BibTeX file:

1. cobib export -b einstein.bib -s -- einstein

   1
   2
   3
   @article{einstein,
    journal = {Annalen der Physik},
   }
   

   This is the normal export command, which leaves the Journal name unchanged.

2. cobib export --abbreviate -b einstein.bib -s -- einstein

   1
   2
   3
   @article{einstein,
    journal = {Ann. Phys.},
   }
   

   By specifying the --abbreviate argument, the Journal name gets replaced with our configured abbreviation.

3. cobib export --abbreviate --dotless -b einstein.bib -s -- einstein

   1
   2
   3
   @article{einstein,
    journal = {Ann Phys},
   }
   

   Finally, by adding the --dotless argument, we can even remove the punctuation from the abbreviated name.

coBib will attempt to always store the full Journal name when adding new entries, and will automatically elongate an abbreviated name when encountering it during the add command. It will warn you about entries where it fails to elongate or abbreviate Journal names, so you can gradually grow your list of Journal abbreviations.

Database Format Linter

:scroll: A Linter is a static code analysis tool which can check for stylistic/formatting errors (among other things).

Last but not least, I want to showcase the new _lint_database shell utility added as part of version 3.1.0. It can detect possible shortcomings in your database file formatting. This allows you to keep up-to-date with the latest improvements to coBibs database formatting, including newly supported features like proper number-support for numeric fields and list-support for fields like file, url and tags.

Here is an example database file with several shortcomings:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
---
einstein:
  ENTRYTYPE: article
  ID: einstein
  author: Albert Einstein
  doi: 10.1002/andp.19053221004
  journal: Annalen der Physik
  month: June
  number: "10"
  pages: 891--921
  title: Zur Elektrodynamik bewegter K{\"o}rper
  url: http://dx.doi.org/10.1002/andp.19053221004, https://onlinelibrary.wiley.com/doi/epdf/10.1002/andp.19053221004
  volume: "322"
  year: "1905"
...

You can find the points for improvement by running cobib _lint_database in the terminal. For this example, you will obtain the following output:

1
2
3
4
5
6
literature.yaml:8 Converting field 'month' of entry 'einstein' from 'June' to 'jun'.
literature.yaml:9 Converting field 'number' of entry 'einstein' to integer: 10.
literature.yaml:12 Converted the field 'url' of entry 'einstein' to a list. You can consider storing it as such directly.
literature.yaml:13 Converting field 'volume' of entry 'einstein' to integer: 322.
literature.yaml:14 Converting field 'year' of entry 'einstein' to integer: 1905.
literature.yaml:4 The field 'ID' of entry 'einstein' is no longer required. It will be inferred from the entry label.

As you can see, this command produces lint messages in the format: <database path>:<line number> <message>. You can go through these messages one by one and change your database file as suggested. For example:

1. literature.yaml:8 Converting field 'month' of entry 'einstein' from 'June' to 'jun'.: As of version 3.1.0 coBib always stores the month field as a three-letter code, which ensures that macros from most common BibTeX citation styles will work correctly.

   1
     month: jun
   

2. literature.yaml:9 Converting field 'number' of entry 'einstein' to integer: 10.: Numeric fields are now supported as well, which means that you can properly express numbers as such (and not as strings):

   1
   2
   3
      number: 10
      volume: 322
      year: 1905
   

3. literature.yaml:12 Converted the field 'url' of entry 'einstein' to a list. You can consider storing it as such directly.: coBib can also handle YAML lists properly which greatly improves the readability of the fields file, url and tags:

   1
   2
   3
      url:
        - http://dx.doi.org/10.1002/andp.19053221004
        - https://onlinelibrary.wiley.com/doi/epdf/10.1002/andp.19053221004
   

4. literature.yaml:4 The field 'ID' of entry 'einstein' is no longer required. It will be inferred from the entry label.: Finally, coBib also no longer requires the redundant ID field and correctly infers this information from the label. Thus, you can safely remove these lines from your database.

In the end, the above database file could look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
---
einstein:
  ENTRYTYPE: article
  author: Albert Einstein
  doi: 10.1002/andp.19053221004
  journal: Annalen der Physik
  month: jun
  number: 10
  pages: 891--921
  title: Zur Elektrodynamik bewegter K{\"o}rper
  url:
    - http://dx.doi.org/10.1002/andp.19053221004
    - https://onlinelibrary.wiley.com/doi/epdf/10.1002/andp.19053221004
  volume: 322
  year: 1905
...

I am planning to add a utility to automatically resolve lint warnings in the future, so stay tuned for that! :rocket:

Online documentation

I would like to leave with a final word on the new online documentation of coBib which is now hosted at https://cobib.gitlab.io/cobib/cobib.html. I hope that it may serve you as a useful resource in addition to coBibs manual page :nerd_face:

In this short post I will go over some of the reasons why I chose to re-design the configuration mechanism and provide you detailed instructions on how to migrate from the old INI-style configuration.

Why use a Python-based configuration?

A configuration written in Python provides the user with the greatest flexibility to customize a program which is written in Python. Obviously, this power comes with responsibility, too. In theory, a user could monkey-patch the code to an almost uncontrollable degree, so care just be taken not to copy code into your configuration which you do not understand. Nonetheless, if the user is aware of this responsibility, the power which a Python-based configuration brings to the table is too great of an opportunity to miss out on for coBib.

When re-designing the configuration module I was looking for inspiration in the code of other Python-configured programs which I use myself, too. Namely, qtile and qutebrowser were the two codes which I was mainly looking into. However, I found that both of them (being much larger codes than coBib) had added a lot of code for the configuration parsing which I did not seem to require. Thus, in the end, I ended up with an implementation much different to either of their approaches.

I am not going to go into the details of how I implemented coBib’s new configuration but it essentially boils down to creating a ModuleSpec from the location of the configuration file using importlib.util. I then combined this functionality with an easy-to-use extension of Python’s dict class which allows recursive setting of items via attributes. (See also this Stackoverflow answer.)

How to migrate my existing configuration?

Let me now focus on how you can convert your existing INI-style configuration into a new Python-based one.

:scroll: A short note before we get started: if you do not have an existing configuration, yet, the easiest way to get you started is to copy the well-documented example configuration to the default location and start adapting it to your needs. You can do so with the following command: cobib _example_config > ~/.config/cobib/config.py

However, if you do have an existing configuration it is probably easier to continue reading and migrate it manually.

To get started, open the file ~/.config/cobib/config.py in your text editor of choice. At the top of the file, you have to import coBib’s configuration object:

1
from cobib.config import config

If you are wondering how you can import coBib’s configuration now, before you have written it, you can think of this step as loading the default configuration which you are now free to adapt to your liking.

In the rest of the file, you can now set the configuration options. As mentioned before, coBib’s config object is an extension of a dict object. Thus, the following two statements are equivalent:

1
2
config['database']['git'] = True
config.database.git = True

I personally find the latter to be more convenient for this particular use-case which is why I have added this attribute-setter capability.

If you want to find out what settings exist you can check out coBib’s man-page (man cobib) or the example configuration cobib _example_config). Before I leave you to it, you should know that coBib will validate your configuration before using it. Thus, if you experience an error be sure to read the output as it will likely tell you which setting is causing a problem (e.g. an invalid type).

Finally, here is a comparison of the old and new settings. As you will see, some of the settings have been renamed/moved (these are highlighted in bold).

:warning: Notice, that the KEY_BINDINGS and COLORS sections have been moved into the TUI section. This was not possible in the INI-style configuration but is the more natural location for these settings to be in.

What are we actually trying to do here?

I will start by addressing this very important question right in the beginning. Well, the answer is actually quite simple: we want to unit test as many features of an application as possible. Now, you might ask, what unit testing actually means. To put it in the words of Wikipedia:

<…> unit testing is a software testing method by which individual units of source code <…> are tested to determine whether they are fit for use.

Ah… :thinking: Essentially this boils down to: “we want to test our source code in small logical pieces rather than as one huge black box”. In this way, we can ensure that the individual blocks which make up our code work as intended. Once that is the case, we can add some integration or system tests to ensure that all of our features combine and interact correctly, too.

Testing in Python

When programming in Python you have a variety of testing frameworks to choose from (e.g. unittest, pytest, doctest). For coBib I chose to go with pytest but what I present in this article should work with any of those testing frameworks.

I do not want to go into the details of how pytest works but will show you a very short example test case:

1
2
3
4
5
6
7
8
9
10
from cobib import zsh_helper
import cobib

def test_list_commands():
    """Test listing commands."""
    cmds = zsh_helper.list_commands()
    cmds = [c.split(':')[0] for c in cmds]
    expected = [cmd.replace('Command', '').lower() for
                cmd in cobib.commands.__all__]
    assert sorted(cmds) == sorted(expected)

The unittest above asserts that the zsh utility function list_commands() returns all commands that are available in coBib. If the assert statement on the last line fails, so does the test.

Testing a TUI application

Now, let us assume that we have unittests in place for all of the basic features of our application. However, we have written a new, fancy TUI which combines all of these features. If we want to avoid manually testing all possible workflows in the TUI when integrating new changes, it would be great to test the TUI itself, too. However, this is not as straight forward since we don’t have direct access to the terminal contents.

This is where pyte comes into play! What is pyte, you ask?

It’s an in memory VTXXX-compatible terminal emulator.

This means, pyte allows us to emulate a terminal within which we can run our TUI. This provides us with a programmatic access to the terminal contents, i.e. it allows us to scrape the screen for its contents and assert what is being displayed!

How to setup pyte

The tutorial page of pyte’s documentation shows how straight forward it is to initialize a virtual terminal window:

1
2
3
4
5
import pyte
screen = pyte.Screen(40, 12)
stream = pyte.Stream(screen)
stream.feed(b"Hello World!")
print(screen.display)

The last line will dump the contents of the terminal’s screen like below:

1
2
3
4
5
6
7
8
9
10
11
12
Hello World!                            
                                        
                                        
                                        
                                        
                                        
                                        
                                        
                                        
                                        
                                        
                                        

This appears to be working just as expected! :tada:

Combining pytest and pyte

After a significant amount of research on the web, diving through many Stackoverflow pages and even reaching out to one of the pytest developers on IRC, I finally managed to come up with a solution that seems to be flexible enough for all the testing purposes which I have encountered so far.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
def test_tui():
    """Test TUI."""
    # create pseudo-terminal
    pid, f_d = os.forkpty()
    if pid == 0:
        # child process spawns TUI
        curses.wrapper(TUI)
    else:
        # parent process sets up virtual screen of
        # identical size
        screen = pyte.Screen(80, 24)
        stream = pyte.ByteStream(screen)
        # scrape pseudo-terminal's screen
        while True:
            try:
                [f_d], _, _ = select.select(
                    [f_d], [], [], 1)
            except (KeyboardInterrupt, ValueError):
                # either test was interrupted or the
                # file descriptor of the child process
                # provides nothing to be read
                break
            else:
                try:
                    # scrape screen of child process
                    data = os.read(f_d, 1024)
                    stream.feed(data)
                except OSError:
                    # reading empty
                    break
        for line in screen.display:
            print(line)
        # now, do some assertions (see later)

That’s quite a test function… So let’s break it down:

1. On line 4 we are forking the process into a parent and child process. This is necessary because otherwise our TUI application will take full control of the process and disallow us from actually running any tests on it.
2. On lines 5 and 8 we differentiate between the parent and child process, starting our TUI application on the child process¹ (line 7).
3. In the parent process we can then use pyte to initialize a pseudo-terminal (as explained in How to setup pyte).
4. The endless loop starting on line 14 is then used to scrape the terminal content. We do so by means of the select.select() method which waits for the file descriptor, f_d, until it is ready for reading.
   1. If nothing is available for reading or the test interrupts, we break the loop on line 22 which will eventually lead to a failing test (that is, once we add some assertions as shown below).
   2. The else clause of the try block will execute if no exception was raised. Here, we simply scrape the screen of the child process and feed its contents into the pseudo-terminal. The reason for not directly operating on the data is to also allow easier processing of terminal attributes such as colors, etc.
5. Finally, on line 33 we can start adding some assert statements to check the contents of the terminal window. Note, that I print the contents of the screen prior to this because this will allow easy debugging when a test fails².

Asserting the pseudo-terminal contents

Now, it is finally time to assert the pseudo-terminal contents! :raised_hands: I like to parametrize my test functions which allows me to run the same test with different inputs. In the case of this TUI test it makes sense to have function arguments for the key strokes which should be send to the TUI and for the assertion function which should be used to assert the outcome. This will look something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
@pytest.mark.parametrize(['keys', 'assertion'], [
        ['?', assert_help_screen],
    ])
def test_tui(keys, assertion):
    """Test TUI.

    Args:
        keys (str): keys to be send to the TUI.
        assertion (Callable): function to run the
                              assertions for the keys
                              to be tested.
    """
    pid, f_d = os.forkpty()
    if pid == 0:
        curses.wrapper(TUI)
    else:
        screen = pyte.Screen(80, 24)
        stream = pyte.ByteStream(screen)
        ### SEND KEYS
        # send keys char-wise to TUI
        for key in keys:
            os.write(f_d, str.encode(key))
        ### END
        while True:
            try:
                [f_d], _, _ = select.select(
                    [f_d], [], [], 1)
            except (KeyboardInterrupt, ValueError):
                break
            else:
                try:
                    data = os.read(f_d, 1024)
                    stream.feed(data)
                except OSError:
                    break
        for line in screen.display:
            print(line)
        ### ASSERT OUTCOME
        assertion(screen)
        ### END

As you can see, compared to before we now have two additional sections:

1. On lines 21-22 we iterate a string of key strokes and send them char-wise to the child process, triggering events in the TUI application.
2. And finally, on line 39 we have added a call to an assertion function which is also provided as an input argument. In this specific example the function looks something like this:

   1
   2
   3
   4
   5
   6
   def assert_help_screen(screen):
    """Asserts the contents of the Help screen."""
    assert "coBib TUI Help" in screen.display[2]
    for cmd, desc in TUI.HELP_DICT.items():
        assert any("{:<8} {}".format(cmd+':', desc) in
                   line for line in screen.display[4:21])
   

   This function is just meant as an example to showcase how you could go about asserting the contents of your terminal are correct.

Further extensions

That is all for now! :slightly_smiling_face: I do have a few more features in my coBib testing suite which would have made this post even longer. So please, feel free to check them out! Some examples of what you will be able to find are:

• using pytest fixtures and executing teardown code
• passing additional kwargs to the assertion function (take a look at my test_tui())
• testing terminal color attributes (take a look at test_tui_config_color())
• testing resize events (take a look at test_tui_resize())

Over the past year a few major design features have crystallized out which include:

1. a plain text database (since v0.2)
2. full TUI support (since v2.0.0a1)

Plain Text Storage

The redesign to use a YAML-based plain text database for storage instead of the sqlite3 database in the early proof-of-concept has become an important aspect to me. Not only is it human-readable but it can also be tracked easily with version control and ensures independence of the previous two features with respect to the file format and tool to read it in.

Terminal User Interface

Another major change was the introduction of a simple curses-based TUI which is currently in the beta stage of development: v2.0.0b4. This was an important change since I had realized that handling increasingly large literature database meant a manual and tedious parsing of long stdout outputs to the terminal. The new TUI simplifies this process through an initial loading of the database into the program allowing subsequently instantaneous operations.

It was important to me that the TUI supports all operations that are available on the command line in a meaningful fashion. In the following I would like to introduce a simple workflow with coBib.

Working with coBib

Setup

Before doing anything, you have to run cobib init which will initialize the database file. After this single step, you are good to go!

Using the TUI

When you simply run cobib without any subcommand you will be greeted with the initial screen of the TUI which lists all entries in your database.

:warning: Obviously, when you run this for the first time, this list is most likely empty.

There are a few things going on here:

1. You can see two status lines: one at the top; and one at the bottom.
   • The top one includes coBib’s version information as well as some statistics on your database.
   • The bottom one provides a quick overview of the default key bindings of the most import operations.
2. In between the two status lines, you are presented with a scrollable buffer listing your database entries.
   • You can navigate this buffer with Vim-like bindings of h, j, k and l, but the arrow keys also work if you prefer that.
   • g and G allow you to quickly jump to the beginning and end of the buffer, respectively.
   • You can quit the buffer with q.

On startup, the entry list contains the same information as the result of cobib list -l (by default, that is). Thus, it presents to you a two-column table with the IDs and titles of your database entries.

Adding, Editing and Deleting Entries

You can add (a), edit (e) and delete (d) entries by pressing their respective keys.

Add

This command opens the command prompt below the bottom status line and populates it with :add . Here, you have the full functionality of the command line interface at your hands. This means you can add new entries to your database by providing DOIs, arXiv IDs or even bibtex files. You can also provide custom labels, files and tags. For more information refer to the man page or try cobib add --help.

Edit

This command will open the YAML-representation of the currently selected entry of the buffer in your favorite text editor (as specified by $EDITOR). Here, you get full control over the contents of that entry, so be mindful of your edits!

Delete

This command will delete the currently selected entry of the buffer.

:no_entry: There is no safety net, so unless you have a backup of your database, this can lead to permanent data loss!

Viewing Entries

Now, that you actually have some entries in your database, it is definitely of interest on how to view these!

Show

This command is triggered when you hit Enter or Return. It will replace the contents of the buffer with the bibtex representation of the selected entry. Once again, you can scroll and quit this buffer as before.

Wrap

The wrap command can come in handy especially in this show buffer when your database contains the complete abstracts of your references but it is also available in the normal (the list buffer) which comes in handy when starting coBib in a rather narrow terminal window. By pressing w you can toggle line wrapping: i.e. whether lines continue beyond the right edge of your window and are only visible through scrolling or whether they get wrapped to continue on the next line with a visual indent.

Open

One more command is available through o which will open any associated item of the entry’s file field. This is achieved through xdg-open and open on Linux and Mac OS, respectively. If no item is associated with the selected entry, an error is shown in the command prompt at the bottom of the window.

Filtering the list

Especially when your database becomes very large, being able to narrow the listed entries down through filters is an important function. I will not go into the technical details of the filtering mechanics in too much detail here and instead try to explain the gist of it with some examples. For more information, please take a look at the man page or try cobib list --help.

Filter

When pressing f you are placed into the command prompt at the bottom of the window. There you can add filters to the list query to narrow down the entries to be presented in the buffer above. The filter mechanics are most easily explained with an example:

1
:list ++title Qiskit

This will list only those entries whose title field contains the word “Qiskit”. You can also invert this filter and list only those entries which do not contain the word by typing:

1
:list --title Qiskit

As you can see, keywords with ++ prefixed represent positively matching filters, while those prefixed with -- must negatively match.

For filters you may use any name for a field present in your database. Thus, in the above example, title could be replaced by any word describing such a field. Common examples include author, year, tags, etc. For a complete list of the available filters of your specific database you can run cobib list --help.

There is one more important aspect to filters: combinations! You can combine any number of filters as you please, e.g.

1
:list ++author Bravyi --author Kitaev

will list all entries which have “Bravyi” as their author but not “Kitaev”. Thus, multiple filters are combined with logical ANDs. If you would rather combine all filters with logical ORs, you can add -x or --or as an argument to your list query.

:scroll: If you want to remove a filter from your query to expand the number of listed entries, you can press f once more and edit the filters in the command prompt.

Sorting

Filtering entries would only be half as useful if you could not sort the resulting list. This is what the sort (s) command is for. Again, an example will speak for itself:

1
:list -s year

This will sort the list of entries by year. You can reverse the sorting order by adding the argument -r or --reverse to your list query.

:scroll: Just like removing filters you can remove sorting keys after pressing s once more.

Export

Finally, the export (x) command allows exporting your database to bibtex format or even a zip file which will include all associated file items. You can even combine the filters described above with the export command to selectively export a subset of your database!

1
:export -b 2020.bib -- ++year 2020

In the example above the -- marks the separation of the arguments of the export and the list command. You can find detailed information on how this command works in the man page or with cobib export --help.

Outlook

I hope you enjoyed this short introduction to cOBib and find it to be a useful tool and welcome addition to your terminal workflow! :tada:

Amongst minor improvements to make some of the interactions smoother I have planned several larger ideas which I want to implement for future releases :rocket:

1. implement a search command which should complement the filtering mechanism described earlier to allow field-independent search. This could also be extended to nested searching of associated PDF files.
2. implement a select command allowing to visually select multiple entries to operate on. Obviously, this will be solely a TUI feature.

Resources

You can find more information on everything covered here as well as configuration options and more at:

• the man page: man cobib
• the integrated help: cobib --help and cobib <subcommand> --help
• the Gitlab repository
• the issue tracker