GetThread.pl

With Nitter.net receiving a cease-and-desist from the dark side, it's time I try harder to get something useful from the back-up I made before leaving twitter ... If we do just

sed -e "s/<h1/\n<h1/g" Alltweetsfromblogger.html | \
      grep 935254889919369216 | sed -e "s:</p>:</p>\n:g" | \
      cl xml

we end up with two "conversation" blocks, because the way I proceeded was

1. paste the link
2. paste the copied contents
3. "Conversation" with `<h1>` tag is something that comes from the copy-paste

That was the starting point.  

• Agreed, the command line is ugly, and its output isn't miuch better. 
• Agreed, its scope is fairly limited: Alltweetsfromblogger is a thing I crafted on Google Documents with a manual routine from my list of threads this blog has links to.
• There are still some problems after you extracted as one line the : both before-conversation and in-conversation links are undistinguishable, apart from the fact one points to a date, which is weak

* but * Google Docs also offers .md export, and looking at the .md conversion, I did spot blank lines while there are none in the conversation itself. So maybe adding a blank-line-to-newline converter before we look for the post ID will help ? 

sed -e "s/<h1/\n<h1/g;\
        s:<p class=.c0 c3.><span class=.c2.></span></p>:<hr/>\n:g" Alltweetsfromblogger.html |\
     grep "935254889919369216[^<]" | sed -e "s:</p>:</p>\n:g" | cl xml

That was the start, somewhere quite near after nitter.net disappeared. The idea since then is to extend the "blogpress" scripts, and have the "development stories" as well as chats with fellow developers aside the main blog text.  

I stuck to the html export: the .md file would have been easier to parse, but the embedded pictures in base64 encoding promised to be a pain to work with. It's not the best software in the world, but I managed to couple it with a few lines that scan the json files from my old twitter backup so that I can identify when pictures in the Google Document are actually thumbs of videos, and reinject the link to the videos.

I did some planning-on-calendar that truly helped nailing down some details, but the photo I shot is completely unreadable :-/ Fuzzy blogging for fuzzy coding ... 

It's been about 1 month now and finally I get the kind of output I want. It's time to let you know about it and see how I can actually get those "threads" embedded into the sample chapter. 

• 3/10 - got thread unrolled next to posts and thumbs for "internal" links 

It's blogpress time again

French journalists have a word for that seasonal topic that comes up again in newspapers when every high-end interviewers are on holiday and things are up to interns: "Le marronnier de l'été" ... and it may feel that doing some blog-to-printable/epub conversions pops up every summer here, but hopefully not for the same reason.

The motion started with the late-night idea that "tiled" engine discussions could be a chapter 1 if the blog ever had to be converted into a book, but also that editions could use their own tag, like "ch1" for that chapter 1. On the next day, I was dusting off repositories and working directories for the next "blogpress" round, motivated by the fact that I'll have up to early August before the next automated snapshot/takeout.

I wrote a small introduction to that chapter (french only, atm) and then grew surprised by how little preparation there was on that "tiled" tag. Reader would just be thrown stuff at their eyes, and they'd better understand things. Sure, I introduced it rather late, not as I was writing those posts... I discovered a bit sadly that most of the tutorial I had used by the time are now offline (and even their archive.org copy are firewalled T_T by "my" most powerful computer).

I edited some of the early posts in "tiled" following the idea that "Many posts contain parts that would be useful for a chapter (say, one about tiled game engines) but also parts that don't quite fit. So how about having <span> or <div> tags assigning those items a "no-ch1" (or no-ch5 when I'll be at chapter 5) and let the simple.css make them display: none ?"

Finally, 4 days ago, I managed to get all the issues fixed in my scripts and re-generate a good-looking page that I can use in calibre to export an epub file. And because each issue involved a good deal of html snippets and that I'd rather have a clipboard to compare them when that happens, I decided to use codeberg's issue tracking system for that. 

script output, as uploaded on codeberg	what it looks like on the blog
what it looks like copy-pasted in office	what I had from older (2025) script output

We're not fully done, but it's getting in good shape. I couldn't get it printable, though, as the browser seems to ignore the CSS instructions that make it neat and stylish on screen. 
Next points would be

• recognized purple-colored-text as English
• if a picture is large enough to cover most of the column width, don't try to make it floating.

A sad discovery while working at all this was that the 72x72 thumbs are no longer automatically provided by the takeout. They might have annoyed me when doing the last conversion back in 2025, but no more than I month later I had figured out that they could be a "magical portal" to posts and uploaded them to neocities.

And now I have another project for those thumbs. I had included the "milestone" tag together with "tiled" to get a bit of reminder of what had already happened and what not between two "tiled" posts, but many of the milestones are too large and distracting compared to what I want. The first paragraph (?), the date, title and the thumbnail would likely be sufficient. Hopefully, I can recontruct some thumbs URIs from regular picture URIs, but I'll need to resort on my drive-archived blogthumbs.zip for the missing ones...  

There's one line of all that that I'd like to bring here. It's the incarnation of "make the possible easy" part of Perl:

$post=~s/src=\"([^"]+)"/src="$larger{$pix{$1}}$escaped{$1}" alt="$1 ; $pix{$1}"/sg; 

In that single line, we iterate through (thanks, g modifier) all the image URLs of a single post and patch them using the hash tables that we constructed by parsing "temporary" files that other scripts produced. No need for split-patch-join sort of loop. 

edit: Some commands used to generate the latest version:

THUMBS=../thumbs/thumbz.html perl ../git/list-atoms.pl ../bilou/feed.atom <(cat ../pictures3.lst ../morepix.lst ../pictures2.lst) tiled '(milestone)' > tm3t.html

where "thumbz.html" is the page featuring a link & thumbnail for every post, which list-atom will look for contents. It needs so many .lst files because I worked on list-

grep '<!-- all images' -A9999 tm3t.html | grep ^i | fe - "cp % /d/local/pages/geds/attach_files/ -v"

I added a flat list of filenames with the pictures at the tail of the HTML file so you can "more easily" cherry-pick the files you need (save-as-html in web browser is doing too weird things)

BLOG=https://sylvainhb.blogspot.com perl ../git/thumbify.pl <(perl ../git/summarize-atoms.pl ../bilou/feed.atom) > thumbz.html

Above: how the thumbs page is generated, below, how I got some -0- thumbs from files that had more normal identifiers. (applied in the folder where I unziped blogthumbs.zip I got from that-time-where-thumbs-appeared-in-exported-data.

ls i* | sed -e 's/\(i[0-9][0-9]*\)-\([^0]\)-\(.*\)/mv \1-\2-\3 \1-0-\3/' | grep -v '^i' > renamethumbs.sh

edit+: fine-tuning keeps going on mastodon

Signets et attach

J'ai pour la SchoolZone l'intention d'introduire des signets/marque-pages qui pendouillent... on pourrait s'y accrocher (indispensable), grimper, descendre (sympa) et ils réagiraient comme des cordes plutôt que comme des barres.

J'ai appris dernièrement qu'il fallait que je prototype ce genre de choses dans l'environnement des "three rooms" sous peine d'être démoralisé par le côté "nu et inintéressant" d'un niveau à peine commencé. Et coup de bol, il y a un coin de la salle "école" qui conviendrait bien : le haut. On pourrait atteindre les signets en utilisant la gomme et s'en servir pour rejoindre l'éponge, un peu compliquée à utiliser pour l'instant.

Hanging to a rope-like bookmark, using it to swing around and hop to the next ... that's clearly something I'd like to bring into the school zone and I may have just found the right way to introduce it first in the "three rooms" demo. Yes, I could use one of the levels in which they are featured, but recent development on the green zone shown that a rushed large level doesn't motivate me as much as a small polished screen can. They would be there, in the top corner and providing an alternate way to reach the swinging Spongebop.

Avant de permettre de s'y accrocher et de se balancer, je voulais voir comment m'y prendre pour une interaction plus simple: juste laisser Bilou bousculer le signet et qu'il ondule un peu en reprenant sa place. J'ai le comportement "circular" qui pourrait être pas mal pour ça.

Before I'll let Bilou grab them and swing, I'll have to start with something easier, like just having Bilou disturb them and make them swing stronger when passing nearby. It should be a mere matter of attaching segments to one another with the "circular" controller... except that once Bilou pushes the bottom part of the "rope", segments now need to lookup two objects instead of just one. And that's a bit more tricky to handle for my engine that only allow one object pointer per object.  

Sauf que on tombe dans le cas de figure compliqué où les mouvements d'un segment dépendent à la fois de la position du segment au-dessus et en-dessous de lui, ce qui pose trois questions:

• Est-ce qu'il faut que j'ajoute une zone "solide" à Bilou ?
• Quels flags (pour les collisions) sont disponibles pour ajuster les comportements (j'ai oublié ^^") ?
• de quelles infos est-ce que je dispose après une collision entre deux segments ? Est-ce suffisant ? 

J'avais pensé relire le tag "attach" pour me rafraîchir la mémoire, et j'ai eu la désagréable surprise de le trouver confus, avec quelque malheureuses infos pratiques perdues au milieu du "gamedev story". Il est devenu mon "tag de la semaine", mais j'ai dû reconnaître que ça ne suffisait pas. Je me suis retrouvé à essayer de l'extraire avec les outils "blogpress" et lui appliquer une feuille de style avant de l'utiliser pour tester les "pages web codeberg".

So, that was the idea back on 11/11 when I sat down with my notebook and collected the questions that needed answers before I could get to the coding stage. I thought unrolling the "attach" tag of the thread would give me answers, but it turned out the useful information was somehow lost into trivias, screenshots and other progress reports. And the code documentation wasn't much better, so I ended up restoring a setup for blogpress and tutogit at the same time, uploading them all on codeberg and see whether I could apply simple.css etc.

ça m'a donné le déclic pour reprendre le travail sur le repository "tutogit", en plan depuis 7 ans, vu que maintenant le nouveau système de gestion des propriétés du niveau est arrivé à maturité. Je pense que ce serait là le bon emplacement pour des pages thématiques présentant chaque élément du moteur de jeu ... qui puisse au final s'intégrer à la documentation doxygen. Voilà déjà celle pour attach.

blogpress 0.3

Combinez les scripts de traitement "blogpress" (blogger-vers-epub) et le service d'impression utilisé par Rodrigo Copetti et vous aurez le "side-project" qui me distrait actuellement... Alimenté en plus par une page de recherche sur l'utilisation de XPath pour "travailler plus proprement" ... Sauf que XPath, pour traiter une requête sur le blog, il met 8 secondes. Mes expressions régulières, 1/8ème de seconde.

Ça fait un peu suite à des travaux de septembre dernier pour essayer de voir ce que les "takeout" de photos google ont vraiment dans le ventre, et l'idée serait de "rassembler tout ce qui peut parler de Qui est Bilou, plutôt en Français à destination de la génération suivante.

I have PERL scripts to process the XML data exported from this blog and fetch pictures to get something that could be printed or converted into epub. So far, it had a drawback: if you got an updated version of the xml (i.e. a later backup), all implied filenames for the pictures would change and almost everything would have to be re-downloaded. A shame in a context where some pictures may get lost.

So I opened them again and started replacing "line number" by "uuid generated by blogger", which gives them stable filenames. A curious idea, when you know I was almost stunned by fatigue. I guess I was just too tired to resist the idea of working on a printable book around the idea of "who is Bilou?" that could be printed like the work of Rodrigo Copetti.

Things haven't been all fine, though. One annoyance was those "new" (vs. 2019) 72x72 pixels thumbs that broke some assumptions I made about multiple URLs pointing to the same file in the same post... Then I've tried to use symlinks to refer to contents previously downloaded, but accidentally forgot to tell the "get-pictures" script that it should skip symlinks when they're around, so it tried to overwrite those pictures I already had and I had to plug my backup HDD and enter data recovery mode ... (I mentioned I was tired, right ?)

Plus tôt dans le mois, j'avais passé en revue tous les posts avec le tag "Bilou", donc, pour en retenir à peu près la moitié. 16 Mo de fichier XML, 106 posts retenus (plus les drafts ^^ ) ... des soucis avec les miniatures 72x72 qui essaient de se faire passer pour les images "grand format" ... des soucis avec les fichiers 1600-h qui font semblant d'être des images mais sont en réalité des pages HTML présentant l'image ... des soucis avec les fichiers tirés de screenshots dans firefox qui à force de conversions / -> %2f finissent par produire des noms de fichiers trop long ... les joyeusetés du scripting quoi. Mais au final, j'ai pu avoir un fichier HTML d'à peu près 1Mo référençant la majorité des images voulues convertible en 115 pages de pdf mal fichu (entendez, avec tout de même des images tronquées ou coupées sur deux pages ^^") qui m'a donné envie de voir malgré l'heure avancée "ce que ça donnerait avec la sélection définitive"

Je me prépare donc à télécharger blog-05-07-2025.xml mais à la place ... me voilà débarqué dans le portail "takeout" de google ...  

By late Saturday, I had collected everything again and was ready to apply the scripts on the latest .xml file, the one where I'd have processed all posts with #bilou and assigned some of them to #firstDemo (because they really weren't quite about Bilou himself, but about trees, applemen, scripting ... or something totally unrelated but featuring a small illustration from the 2001 comics). Only to realize that Google has now decided that Blogger export should be managed by the Google Takeout portal. That means waiting for several hours (up to 2 days, but hopefully not in my case), checking boxes and finally receiving a link to a 2GB archive containing .... (drumroll) ... the whole set of pictures uploaded to the blog.

Il me faut donc patienter plusieurs heures pour avoir le nouveau .xml qui est entre-temps devenu un .atom, a laissé tomber certaines informations, regroupé d'autres autrement et est maintenant accompagné ... de l'ensemble (?) des images du blog. Je m'autorise un point d'interrogation, parce que si j'ai finalement pu produire le bilou.html.pdf que je voulais, j'ai quand-même conservé le listing URL->fichiers construit avec blog-28-06-2025.xml et pas avec le nouveau feed.atom ...

Tout ça pour passer le témoin aux p'tits jeunes ? Bah, quand j'ai montré pour la 2eme fois à J.L.N les feuilles qui étaient malgré tout sorties de l'imprimante, il n'y a pour ainsi dire pas regardé et m'a demandé si j'avais fait des progrès avec les cascades, le bug de Bilou-ballerine et tout ça  ^^"

edit: j'ai finalement un petit script thumbify.pl qui rassemble toutes les images 72x72 un peu mieux qu'un oneliner bash et y ajoute les liens vers les posts du blog. ça nous fait 26MiB d'html+png (probablement inutilisable vu le nombre de requêtes HTTP qui seraient nécessaires -- comptez 30' sur le wifi local) ou 11 pages A4 (pas beaucoup plus utiles vu qu'on ne sait pas y faire de recherche ou cliquer sur les liens ^^"). A découvrir sur neocities...

edit++: capturer l'attention de J.L.N avec la page: check ^_^