Suggestion for Improving the "Insert Special Character" Feature

[byword]

I posted a suggestion regarding the Insert Special Character feature in the New Features... thread, and I actually tried implementing Calibre's Insert Special Character tool into Sigil. I built this prototype with the help of AI, and since it's just the initial version, the search function hasn't been implemented yet.

I created this sample mainly to explain to KevinH what kind of feature I was envisioning.
Here is what Sigil's current Special Character dialog looks like:


In my line of work, I rarely have a use for the characters currently provided there. Instead, I frequently use circled numbers, various diamond bullets, and mathematical symbols. Because the books I produce are so diverse, I also use characters from multiple languages, such as Greek and Russian, for parallel notation.

Calibre's EPUB editor makes it very easy to find these specific types of characters, which is why I tried applying this feature to Sigil.
In Calibre's Special Character tool, you can view special characters by category, and if you type "circle" in the search bar, you can easily find circled numbers.

This is Calibre's Special Character dialog:


And this is what it looks like when applied to Sigil:


Since I had little use for Sigil's default Special Characters, I previously registered my frequently used codes and used a custom-built version of Sigil. However, because I use such a wide variety of codes, I still often found myself having to search the internet for Unicode values. That is why I wanted to propose merging Calibre's Special Character functionality into Sigil.

I would appreciate hearing other people's opinions on this proposed direction for the feature.

[rjwse@aol.com]

I frequently use circled numbers

Another way to get circled numbers if you are using Ubuntu is to enable the compose key (usually right control key). There are 5936 symbols on tap and you can make your own additional ones, including any which kind of boilerplate, by creating a new file called .XCompose where you make them. To get circled numbers or letters you type right ctrl, (, what you want circled, ) and there it is. Circled 50 is right ctrl, (, 5, 0, )

[KevinH]

This seems overly huge, even with a filter/search feature. It simply provides every single unicode character based on unicode code class categories. This includes characters from dead languages and glyphs/characters that many fonts do not support. My bet is 98% of these are never needed by most users of Sigil.

So you need to apply Pareto's rule here and identify the 20% of characters that cover the 80% of users needs. This is what Sigil's Insert Special Character already tries to do based on a study done a few years back based on already published epubs and character and glyph usage which includes font availability.

You do know you can already copy and paste from any open document and browser to add any character to Sigil, or just type in its numeric entity directly, or create clips for frequently used characters or strings, right? Also most OS's already have built in ways to compose characters and allow for selections. MacOS certainly had. Alternatively you could just keep a text file your commonly used characters that lie outside the mainstream, or just bookmark your favourite unicode website in your browser.

But perhaps scaling this down in some sane way might makes sense. As it stands now it is overkill that duplicates either built in OS compose character tables and functions or duplicates what is available easily in any browser, for very little general user benefit.

But I can not fully judge that as I can not see what code changes are needed and how much memory this takes up.

So please post a diff of your proposed changes or point us to them in your github account repo so we can see for ourselves how much code and memory this takes. If it can be condensed down in some sane way, then "maybe".

[elibrarian]

If the OP is on Windows, there's tools like BabelMap, Unicodia or even the built-in Charmap.exe (in advanced mode). IMHO they're way more sophisticated than the Calibre's Insert Special Character.

If Sigil should be bettered on that point, it would be nice to have the possibility to put one's own most used special characters into the dialog, possibly via a configuration file/ini/whatever.

Regards,

Kim

[byword]

I agree with the feedback that Calibre's Special Characters feature is indeed somewhat excessive.

The approach suggested by elibrarian—allowing users to directly add or remove character codes—seems like a great alternative. Characters that I use frequently might be useless to others, and even if 99% of people use a certain character, it is redundant if I have no need for it. That said, Calibre's "Favorites" feature was definitely appealing.

The special characters required vary widely depending on the language. Circled numbers are just one example; when formatting books, publishing often demands Latin extensions, Cyrillic, units of measurement, various bullet shapes (circles, squares, triangles, diamonds, clovers, etc.), and all sorts of brackets. In Korea, characters like ÄãÑßÞæç are never used. Instead, for digital accessibility, distinct Hanja character codes like 不不, 金金, 樂樂樂樂, and 力力 are often necessary. While they may appear identical, their pronunciations and contexts differ in Korea, so distinct codepoints must be used for screen readers and accessibility tools.

I edit around 30 EPUB titles every month. Because of this, I frequently need to insert a wide variety of characters across different types of books. While I have tried Windows' default character picker as well as third-party character input tools, Sigil's Special Characters feature remains the most convenient.
Currently, I modify the SelectCharacter.cpp file directly to include around 300 characters that I frequently use, and then build Sigil myself. Please refer to the attached image.
The only inconvenience is having to download the source code, replace SelectCharacter.cpp, and rebuild every time a new update is released.

Looking back, the reason I was able to insert characters so quickly was that I had already customized the Special Characters list with the codes I use most. So yes, allowing users to add or remove their preferred character codes seems to be the best solution. If others like this approach, I can give it a try and put together a sample.

[DNSB]

While I don't use special characters all that often, the customizable list sounds like a good idea for extending the list without having 65K characters to choose from.

[KevinH]

So how many custom characters should be allowed. 10, 100, 256, 512, 1024?

Thoughts anyone?

[byword]

Originally Posted by KevinH

So how many custom characters should be allowed. 10, 100, 256, 512, 1024?

Thoughts anyone?

How about 512? I personally added around 400 characters to the SpecialCharacter.cpp file for my workflow.
Just gathering frequently used units like ℃, ㎠, ㎟, and ㎡, along with superscripts and subscripts for editing accessible science e-books, easily adds up to nearly 100. Print book typesetters often combine standard text like 'cm2' or 'mm2', so for accessibility, I need to replace them all with proper symbols like ㎠ and ㎟.
256 seems a bit too small to fit other frequently used code as well.

[BetterRed]

FWIW - thanks to @elibrarian, on Windows I use a third party startup program, Transtools Utilities→Insert Symbol, that pops the following when I press its keyboard shortcut…



The characters and shortcut are configurable.



I like it because I can drive it from the keyboard.

BR

[DiapDealer]

I guess I don't really see the point of re-creating within Sigil the ability to do that which is already very easily accomplished on all three supported platforms: with both inherent and countless 3rd-party--tools that will work with all applications.

[Martinoptic]

I just use windows character map.

[DiapDealer]

Originally Posted by Martinoptic

I just use windows character map.

The Windows Character Map is open on my work machine pretty-much all day. My Linux machine at home has a similar utility that I use a lot.

[KevinH]

Given this would duplicate much of the OS's functionality, perhaps we leave things as is?

But I am open to either but allowing the user to add their own favourites might be useful for some people. But 512 of them seems a bit too much, given this is basically a convenience change only. More like 64 or 128 seems much more reasonable given so many operating system utilities already exist that can handle the full list.

[DiapDealer]

I agree. I'm not opposed to allowing a little customization, but it seems to me the vast majority of users could get by without a ton of available slots. Those who work extensively with this sort of thing typically already have OS-level solutions to customize their workflow.

[DNSB]

The utility I was using to insert special characters moves a used character to it's first page which allows for 64 characters. I'm up to 42 characters in that used list. OTOH, I am not editing non-Latin character set books. Most of the ebooks I get to polish already have any special characters inserted by the creator.