Pavel's diary

It is now here again. New Year. On December 31 last year, I wrote down some wishes I'd like to achieve in 2005. But as always, I still have 83 issues on My issues list, we have released 2.0 just before the OOoCon in September etc.

Before writing new wishes, I'll summarize events and things that I remember from OpenOffice.org world in 2005.

The most important event for me was release of 2.0 and ODF being accepted as OASIS standard (reverse order would be more appropriate!). The second most important event was OOoCon in Koper. Man, I enjoyed it! Lovely place for OpenOffice.org conference! The release of 2.0.1 has shown us that we have to be more pedantic in QA and also in the release process (timelines are good, but Christmas is not good time to have a release)! AMD64 porting is getting good shape now thanks to Kendy. Porting team for Mac OS X was formed (I even bought my first Mac to be able to help). We finally have Wiki! We had several dramatic failures in our infrastructure. Mail is crucial for communication, but IRC is also important and can help when mail doesn't work!

Forgot something? Let me know.

Now: what I'd like to do/see in 2006?

• AMD64 builds out of the box
• usable Mac OS X universal builds without X11
• I'd like to cut down the list of MyIssues down under 40 at the end of the year (the current status is 83)
• my community build system migrated to newly designed build-farm, more automated
• Czech team at OOoCon 2006 with more presentations and more active members
• at least three big public companies/institutions in Czech migrating to OpenOffice.org and publicly announcing it

Several new things today. ktrace/kdump are strace equivalent. System python loads modules with .so extension, but internal (patched) python is loading .dylib instead. This have to be solved properly - mail to mac@porting.

Long awaited task for all OpenOffice.org translators using POT files is here! Right now, we all use old translate toolkit version that merges all same strings in PO file to one entry. New versions of translate toolkit (0.8 RC5) allows to separate them.

This blog entry will describe, how you should upgrade your tooling and PO files.

So, to update your PO files to new translate toolkit you should prepare the following:

• your latest/current GSI file produced with the old tooling
• new translate toolkit installation

Of course you have your GSI ready - this should be your regular output from the translation process. Installation of the new version of translate toolkit is very easy:

# Download the current version
wget http://switch.dl.sourceforge.net/sourceforge/translate/translate-toolkit-0.8rc5.tar.gz
# Unpack it
tar xvfz translate-toolkit-0.8rc5.tar.gz
# Install it (somewhere)
cd translate-toolkit-0.8rc5
./setup.py install --home=~/.ooo/Translate

I have chosen the directory ~/.ooo/Translate. You should choose the directory yourself.

Now you have everything ready for migration!

Create new (thus empty) temporary directory - I'll use /tmp/migration below and copy your current GSI file there:

p@pc:/tmp> mkdir migration
p@pc:/tmp> cp ~/.ooo/build/Files/SRC680_m147/GSI_cs.sdf.bz2 migration
p@pc:/tmp> cd migration
p@pc:/tmp/migration> bunzip2 GSI_cs.sdf.bz2
p@pc:/tmp/migration> ls -l
total 23628
-rw-r--r--  1 p users 24162376 2005-12-25 17:31 GSI_cs.sdf

Now, we will generate new PO files with the new translate toolkit:

p@pc:/tmp/migration> export PYTHONPATH=/tmp/TRANSLATE/lib/python
p@pc:/tmp/migration> ~/.ooo/Translate/bin/oo2po -i GSI_cs.sdf -o po -l cs
oo2po: warning: Output directory does not exist. Attempting to create
processing 230 files...
[###########################################] 100%

After the utility oo2po finishes the work, the directory po will be populated with new PO files. Verify them and merge them to your CVS or whatever mechanism you use to track the translation process.

This should be everything you need to do to upgrade.

I have uploaded POT files generated with new translate toolkit to ftp.linux.cz with .UPDATED_TRANSLATE suffix so you can update immediately.

For more informations, read my previous blog entries:

• Adding new languages into my build system
• l10n@2.0: PO files and working with them
• l10n@2.0: generating GSI/SDF file from translated PO files

If you have more questions, do not hesitate to contact OpenOffice.org's dev@l10n mailing list.

I have finished first build of OpenOffice.org milestone SRC680_m147. Still manual tweaking needed, this time because of one wrong dependency inside OpenOffice.org source tree, thus the next build will be completely clean with no manual tweaking so I can also upload full build log for future reference and checking.

macmini:~ oo$ ps -xuww|grep s[o]ffice.bin
oo   23483   0.0 14.6   248760  76764  ??  S     9:31AM   0:13.81 \
/Volumes/OpenOffice.org 2/.../program/soffice.bin -
macmini:/Volumes/OpenOffice.org 2/.../program oo$ file soffice.bin
soffice.bin: Mach-O fat file with 2 architectures
soffice.bin (for architecture i386):    Mach-O executable i386
soffice.bin (for architecture ppc):     Mach-O executable ppc

And now the steps needed to get there

• modules bridges, python and pyuno are still not universal. Only PowerPC version. python is doable. pyuno is not universal only because python is PowerPC-only. We do not have bridges for Intel so far.
• I build with disabled mozilla for now.

I'm now uploading the installation disk image to ftp.linux.cz:

83cad15682db414bbc525e272e04fb75 OpenOffice.org-2.0_en-US.dmg 207408798 bytes

The patches you need are in my build system, patches named UB-*.

I do not think it will work on Intel based Macintoshes yet, but you can try it. As I already said several times, you can't bought one yet without limiting your freedom, thus I can't test it (limiting my freedom is unacceptable for me). -----

Apple published several Transition Videos on their developers' page. I enjoyed (French?) accent of the presenter, but in general: very informative, but you should start with the written document first to get good background.

And as promised to be neutral (hi Fridrich ;-), here is an annoyance I found when using Safari.

Safari is a web browser of Mac OS X. On a typical page with links, you see *many* blue texts that are links in fact. Good. When you move the cursor above the link, the link changes its color, it is underlined *and* the cursor changes to small "hand". Good so far. But I wonder why it doesn't change the color, why it is not underlined and why the cursor is still in the form of arrow if you are above the link to QuickTime video. Minor but very visible annoyance. Yes, I know, it is embedded object, but anyway: is it that hard to get it perfect? ;-) -----

SRC680_m147 is the first regular milestone with new strings after 2.0.1 was released. There are several new strings (some only contain fixed typos etc.). So do not forget to update your PO files to this milestone. POT files are here. -----

Jack Loftus dreams that Xen will be killer application. I do not think so. Xen is nice piece of software with several problems. The most important one is that it is not well integrated into the operating system (in terms of tools, etc.).

If I could afford to dream, I'd like to see the VMM built into PC hardware *directly*. Big servers already have it and almost every killer application is an unusual usage of old, known, good technology. Let's try it! Why should I have KVM switch and simulate this behavior for the end-user?

This would be my killer application for 2006! -----

I'll try to summarize important things from my universal binaries investigation.

ChangeLog

2005/12/20 Problem: specifying CFLAGS and LDFLAGS in one command-line (filed as issue number 4387241 in Apple Bug Reporter database).

Why universal?

Apple is going to produce Intel based systems really soon now (right now 87% people on MacPolls expect it to happen in January next year). Previous systems were based on PowerPC processors.

PowerPC is big endian, Intel is little endian. They use different instruction sets. Apple wants the software to run on both platforms in the future. So they invented universal binaries concept.

What is it?

The best description is on small and simple example. We have two files - Makefile and example.c. The Makefile creates several executables:

macmini:~/tmp/example oo$ ls -al example_i386 example_ppc example example_all
-rwxr-xr-x    1 oo       oo          14772 Dec 18 19:35 example_i386
-rwxr-xr-x    1 oo       oo          17748 Dec 18 19:35 example_ppc
-rwxr-xr-x    1 oo       oo          38228 Dec 18 19:35 example
-rwxr-xr-x    1 oo       oo          38228 Dec 18 19:35 example_all
macmini:~/tmp/example oo$

This example is produced on PowerPC G4 based Mac Mini because Intel based system are not available yet (they are available if you accept to limit your freedom, but it is unacceptable for me).

First two files are simple binaries for their respective architectures:

macmini:~/tmp/example oo$ file example_i386 example_ppc
example_i386: Mach-O executable i386
example_ppc:  Mach-O executable ppc
macmini:~/tmp/example oo$

The file example shows one possible way to create universal binary - gcc can do that for you (see Makefile for details).

macmini:~/tmp/example oo$ file example example_all
example: Mach-O fat file with 2 architectures
example (for architecture i386):        Mach-O executable i386
example (for architecture ppc): Mach-O executable ppc
example_all: Mach-O fat file with 2 architectures
example_all (for architecture i386):        Mach-O executable i386
example_all (for architecture ppc): Mach-O executable ppc
macmini:~/tmp/example oo$

The second way is to prepare separate binaries for all architectures and then use the tool lipo to create final, universal binary that will combine them into example_all. The resulting files are the same:

macmini:~/tmp/example oo$ md5sum example example_all
1c1e96a58c9944a409fe093d7f9b436d  example
1c1e96a58c9944a409fe093d7f9b436d  example_all
macmini:~/tmp/example oo$

For my testing purposes, I decided to use the first (gcc) way.

So to sum up, from the porter's point of view, you can think of universal binary as a native binary (PPC in my case) bundled with cross-compiled binary for the other architecture (Intel in my case). And this "cross-compiled" brings several interesting features ;-)

Potential problems, issues and ideas

I'll collect (and update) various issues and problems with universal binaries here as we reach them during the build process.

• Not all tools and programs should be compiled as universal binaries. If the tool is used only at build time and is not bundled with the final product, it can be compiled for native architecture only.
• I wrote simple script that can check if the resulting binaries and/or libraries are universal. It is very simply and stupid but can help you to check the resulting binaries quickly.
• If you are building universal binaries (so your CFLAGS variable contain two -arch flags) you can't create dependencies and use gcc's -MD argument). You can often disable it by using configure's --disable-dependency-tracking option.
• Problems with predefined macros. If you are building on PowerPC, you get POWERPC macro defined. But this doesn't mean that you'll run on PowerPC! Do not forget about cross-compiling part! Intel binary is cross-compiled on your PowerPC machine, so you have to handle it properly. See the above mentioned example.c source code file. It prints different string depending on the architecture it *runs* on.
• The problem mentioned above is also connected with assembler. We use assembler in *many* parts of OpenOffice.org source code. In module sal, we use assembly in interlck.c file. Module sndfile is using assembly to cast (!?) numbers. Mozilla is also using assembly (see Mozilla's universal binary page). All assembler parts have to be modified accordingly.
• Bridges. This is a problem ^2. We still do not have bridges code for MacTel. But imagine we have it. And now we have to compile them both into one! It will be funny task! I think we will use lipo here instead of combining them together. It could be much easier.
• Build prerequisites have to be universal binaries too! Do not forget that if your build depends on e.g. gtk, you must have gtk available as universal binary too, because otherwise the cross-compiling part won't find all symbols. This is a problem right now, because fink doesn't support universal binaries. You have to use DarwinPorts.
• You can't specify both CFLAGS and LDFLAGS on one command-line (modified example):

  macmini:~/tmp/example oo$ gcc -isysroot /Developer/SDKs/MacOSX10.4u.sdk -arch i386 -arch ppc \
                          -Wl,-syslibroot,/Developer/SDKs/MacOSX10.4u.sdk -arch i386 -arch ppc \
                          example.c -o example_allbyone
  /usr/bin/ld: -syslibroot: multiply specified
  collect2: ld returned 1 exit status
  /usr/bin/ld: -syslibroot: multiply specified
  collect2: ld returned 1 exit status
  lipo: can't open input file: /var/tmp//ccslVnnk.out (No such file or directory)
  macmini:~/tmp/example oo$
  

  I still do not know how to solve this properly :-( This affects several modules (like curl and libxml2). They fail in the configure phase.

Sample patches

... are included in my build system. Search in the directory Patches/SRC680* for files named UB-*. Warning: almost everything is brutal hack. I'd like to first introduce the concept to all developers and then collect ideas how to continue properly. Ause? Heiner? Mac OS X porters?

My build compiles many modules as universal binaries, but still some are only for native architecture - like python, curl, sndfile, berkeleydb, libxml2 (see the above mentioned script check_universal!) etc. If you have patch for new modules, do not hesitate to send it to me :-)

BTW - this will be the first porting effort where OpenOffice.org internal only code is much easier to port than OpenOffice.org external modules ;-)

References

Universal Binary Programming Guidelines, Second Edition
Cross-Development Programming Guide
Building Universal Binaries from "configure"-based Open Source Projects - hmm, OpenOffice.org is also based on configure. Rene ? ;-)
Porting UNIX/Linux Applications to Mac OS X

Thanks

I'd like to thank Apple for wonderful weekend!

Did I forgot something? Please tell me so I can put it here for future reference. -----

During this weekend, I was updating almost all compilers on my build systems. This time, it was Apple's Xcode 2.1 on MacOS X. I want to play with Universal Binaries so I have to upgrade to Xcode 2.2.

I repeated the installation several times... I have two users on my system - the first user created during the installation (pavel) and separate user for building OpenOffice.org - user oo. As I downloaded Xcode 2.2 as user oo, I started the installation under its identity as well. Installer asked me for admin user (pavel) and for his password. After entering the password installation continued for random time and then error stopped it. Every time, it was Input/output error. I suspect (wild guess, if you have details, please provide them) that they run the second part of the installation under sudo and it simply timed-out ;-) So I decided to switch to the pavel user identity and finished the installation without problems.

After several experiments in Xcode IDE, I closed it, gcc_select 4.0 to use gcc 4.0.1 and voila, experimenting can start. -----

For testing purposes, I updated my Solaris/SPARC build machine to Sun Studio 11. C and C++ compilers now identify as:

bash-2.05$ cc -V
cc: Sun C 5.8 2005/10/13
usage: cc [ options] files.  Use 'cc -flags' for details
bash-2.05$ CC -V
CC: Sun C++ 5.8 2005/10/13

No patches installed so far.

Heiner told me I need patches from child workspace hr24 to get it working, so I'm saved from suffering there ;-) Unfortunately OpenOffice.org's configure doesn't support Sun Studio 11 (5.8) yes, so I prepared a patch and filed issue for it (#i59476#). -----

I'm now splitting the build system again to support both development and stable release, and found out that the following languages do not update their translations properly. In the child workspace localization02, the macro OOO_LICENSE was removed (of course, because the license is GNU LGPL only now). But still, many "up-to-date" GSI files reference it:

GSI_en-GB.sdf
GSI_ga.sdf
GSI_lo.sdf
GSI_lv.sdf
GSI_sh-YU.sdf
GSI_sr-CS.sdf

Please fix your GSI files and keep them up-to-date! It is easy - during the RC phase, it takes max. 5 minutes to update to the latest POT files! If you have the OOO_LICENSE macro in your translations, the contents of About window will be untranslated. -----

Joost Andrae sent an update to releases mailing list. Hamburg RE team have created new master for 2.0.1 preparations - OOA680. The fifth release candidate will be tagged as OOA680_m1. It is not yet ready though.

It is good that our "do not delay" discussion provoked QA team to higher activity! Thanks. Bugs identified and fixed in RC5 could make OpenOffice.org 2.0.1 even better!

I also cleaned a list of bugs with target 2.0.1. Right now, 48 issues with target 2.0.1 are open, all assigned to translation. -----

Finally I meet with Evan Brown, lead of Gaelic OpenOffice.org localization team. Nice discussions with him and also with his brother. They both enjoy Prague and Czech beer and this is important ;-) -----

I always wondered, how the animal that is on my desktop looks like in reality. Now I know.

Small, nice, funny chameleons. My sister-in-law has two chameleons at home and I took some photos yesterday.

Enjoy them! -----

Another edu-day. Read more details about Dashboard (my previous blog entry about it).

If you'd like to know more about it too, start with this the Dashboard Tutorial. Your next step should be Developing Dashboard Widgets.

After reading both documents, you should be able to create your own simple Dashboard widgets.

The most interesting document is Dashboard Programming Topics. You'll find almost everything you need when implementing your own widget there. Dashboard Reference is another good source of informations. More in-depth informations about debugging is contained in Debugging Dashboard Widgets.

Today, I've prepared skeleton widget for the "OpenOffice.org Latest Development Milestone". I have not finished reading of the above mentioned documents (mainly about packing the widgets). I'll publish the first version when I'm back at home at the end of this week. -----

I started community builds for RC4 (SRC680_m145). The milestone SRC680_m145 was made ready before any of my RC3 builds finished, so I have abandoned building SRC680_m144.

Builds for GNU/Linux are already being uploaded. I uploaded very-pre-beta-do-not-use-it AMD64 build.

GSI files do not contain errors. Only Macedonian one contains some wrong lines. Builds are ready for your QA! -----

The Municipality of Prague 17 (Řepy) is using OpenOffice.org and other free software.

The Municipality met a few problems with OpenOffice.Org during the initial user trainings as it was prepared as a differential training (MS Office vs. OpenOffice.Org).

The problems were mainly based on different level of users' skills so the IT team decided to prepare complete manuals for OpenOffice.Org for its users. Comparative workshops (MS Office/OpenOffice.Org) for users were organised as other changes were truly transparent for them (meaning no difference in the look and feel in the field of network usage excluding that of Internet use).

Source: IDABC -----

Do you know the feeling when you think you have already done that work in the past? This is the feeling I have after working on 64bit patches for module binfilter :-(

We forgot binfilter module for DXArr patches, so many s/long/sal_Int32/, several long/sal_Int32 mismatches that work on 32bit machines, but don't on 64bit. All bf_ builds now with the exception of bf_sw.

I'll continue tomorrow. -----

My son Michal (1 year, 5 months) is able to press Num Lock key when I point my finger to Num Lock LED on the keyboard and Caps Lock when I point to Caps Lock LED! Scroll Lock LED doesn't work in X, so maybe next week, I'll switch into text mode to teach Michal all LEDs ;-)

... and no, my keyboard still isn't Genius SlimStar. -----

I vote against the delay. Here are my reasons.

We are now in the middle of transferring our development model from looong time model to short-time, more often and predictable releases. Thus 2.0.1 is the first (and only one!) that people expect to contain fixes as it would be released after looong time, but in fact it will be released after very short time after our major release 2.0.

It contains many fixes and delaying the release now means that our users won't be able to use them! Only regressions from previous version should be fixed (should there be any). -----

I again spent some time today on learning the basics of Mac OS X Tiger UI (well, in the meantime when I prepared patch for OOo's stlport so it works with both gcc-3.3 and gcc-4.0 from XCode 2.1).

Today, it was the Dashboard, new feature introduced in Mac OS X Tiger. If you do not know it, you can imagine it as a noticeboard above your desktop. If you need something, you'll quickly look at the Dashboard (press F12) and if you are finished with it, you'll again look back to your desktop. What can be there? So called widgets. Quotes, calculator, Wikipedia, system status, weather forecast etc. Widgets are configurable, follow the Apple HIG etc.

Quite handy, very simple idea with perfect working. And what is important: it is very simple to create your own widgets! Another +1 for Apple (of course I have to be neutral: why I feel like in helicopter [citation from #qemu IRC channel] while Mini is doing CPU intensive stuff? ;-). -----

The milestone SRC680_m144 is released, I uploaded POT files (no changes from m143 though). Updated my build system to it, integrated more 64bit stuff, IZ work, cws handling.

I'll work on pj43, pj44 and pj45 child workspaces on weekend. -----

After the first week working with Mac OS X, I found two very annoying features of the default setup. Both are connected with terminal application.

Imagine you do have Terminal in the Dock and do not have any terminal running so far. To run one, you just click on the icon in the Dock. It will open. OK. To open the second one, you can't click on it again. My anticipation was that if I click twice on the icon, two terminals open. Of course you can press Command-N to get new window open, but it is inconsistent - the behavior of click on the icon depends on the system status (it works if there is no black triangle below the icon thus no Terminal application is running, and it doesn't work if it already is running).

What is the second annoyance? Imagine you'd like to open several terminals at once with e.g. five Command-N. My resolution is 1680x1050 so there is plenty of space to place new terminals. But the default placement is to place new terminal over the previous one. So if you want to use them simultaneously, the first thing you have to do is to "distribute" them over your desktop.

Also I had to check "Use option key as meta key" to get the useful xterm's Alt+BackSpace behavior back (delete whole word backwards).

BTW - the default "max user processes" equal to 100 is also not good idea ;-) -----

I have just finished first (still manual tweaking needed) Mac OS X Tiger build on Mac Mini.

Some discussions about Mac OS X version on IRC. Mac OS X porters decided on their meeting in Hamburg that they will skip 2.0.1 release (meeting notes are still not public). I tried to persuade Eric and Oliver that this was wrong decision. The motivation for this decision was to limit the amount of work not directly connected with the future of non-X11 Mac OS X port. I understand this decision from the point of porter, but in general I do not agree with it. The changes between 2.0 and 2.0.1 are so important, that skipping one platform supported in the previous version is impossible. Users will ask many questions (non-informed users will ask "Where can I download 2.0.1 for Mac OS X?", slightly informed users will ask "Why it was skipped?") and the final impact on Mac OS X porters will be even higher. And even worse, Mac OS X users tend to ask their questions on dev@porting, thus also all (not only Mac OS X) porters will be affected.

I hope that they will change their decision. Please try to think about it too and let them know your opinion (be it positive or negative). -----