php-5.3.3 sqlite behaviour

In this post I like to point out to you Bug #340059. It's about using sqlite3 without getting sqlite2 forced on you by portage.

The problem here was that to enable pdo-sqlite, the sqlite driver for PHP Data Objects, you had to enable USE="sqlite". This was fine when there was only one sqlite USE flag. But with both sqlite and sqlite3 selectable, it became an odd choice. PHP Data Objects only fully support sqlite3 in the first place!

So I changed that today and made PDO support for sqlite dependent on USE="sqlite3". The update should now be on a mirror near you. By the virtue of eblits, it is available for all php-5.3 versions we currently have. Simply remerge php (USE="-sqlite sqlite3" emerge -v php) to see the effect - an emerge --depclean should now unmerge sqlite-2* if PHP was the last application requiring it.

And while Dessa kindly pointed out to me Bug #249418, which states that USE="sqlite" should enable either sqlite2 or sqlite3 support and prefer sqlite3 (thus obsoleting the sqlite3 USE flag), I'm still in favor of the current solution.

For one, PHP upstream still enables both sqlite2 and sqlite3 by default. And if you install a Gentoo php without any changes, you'll get sqlite2 and sqlite3, too, because we follow upstreams defaults.

But heads up: future versions of PHP-5.3 will throw E_DEPRECATED notices if you continue to use sqlite2 via sqlite_open() as per this post. Support for it will be gone in future minor versions (whether they will be named 5.4 or 6...)

Fixing /usr/portage/profiles/default/linux/make.defaults

This is a post about probably (to an outsider) boring Gentoo internal development affairs. Feel free to skip or skim it, if you're only here for the PHP status updates ;)

Part of the update to dev-lang/php-5.2.14 was the change to IUSE defaults. This allowed us to provide a PHP build that matched what you could get by simply downloading and compiling the PHP upstream tarball by hand.

It also allowed fixing a long-standing papercut with the make.defaults file. In 2006, the release team discovered php was breaking the release process because certain packages required dev-lang/php to be built with certain USE-flags enabled. For lack of a better solution, the required flags were added to the top-most make.conf template available at that time.

Today, /usr/portage/profiles/default/linux/make.defaults still contains USE="${USE} cli pcre reflection session spl". This results in every linux system having these USE flags enabled by default. At least for spl this is a waste: no other ebuild besides php uses this flag.

Initially, I was going to scrap the whole line, but it turned out to be rather difficult. Not counting spl, all other USE flags are utilized by other packages beside php. So removing the make.defaults line would change their default behavior, too.

Therefore I removed "spl" from the list in make.defaults. For the rest, more consultation is needed. So if you read this and maintain a package using one of "cli pcre reflection session", please comment on this post or better yet, on bug #310383. Expect a mail on gentoo-dev about this, soon, too.

php-5.2.14 USE flag clarification

This is in response to a rescue operation I did yesterday evening, after being notified of Bug #332257. In essence, I've failed to include two USE flags (force-cgi-redirect and discard-path) which should have been there and castrated a third one (cgi).

I snuck those missing two back in and fixed USE="cgi" to also enable fastcgi, like 5.2.13 did. So if your FastCGI setup stopped working after an upgrade to 5.2.14, please resync and remerge php-5.2.14, the regression has been fixed.

A friendly user also asked me about some other changes, I only then realized I failed to document: the sybase, java-external and fastbuild USE flags are gone from 5.2.14. You'll notice that they also don't appear on php-5.3.3. This has a reason, which is probably only appearent if you follow php-bugs@gentoo.org on our Bugzilla, so here goes:

sybase

... was dropped when Bug #281316 came up. I learned that sybase-ct as provided by dev-db/freetds is a near complete stand-in for sybase. If the "near complete" doesn't work for you and you need sybase back, please comment below. But note PHP-5.3 has dropped support for that extension, so you'll need to find another way, anyway.

java-external

... has never done much to the PHP build - it only pulled in dev-php5/php-java-bridge and enabled USE="session". Now, the java-bridge package has not been available in our tree since 2007 and the official status of the "java" extension according to the PHP manual is unmaintained and dead. So I decided to drop this. If you need Java support, please head over to Zend, but I've no experience with their Java stuff, so I can't comment further.

fastbuild

... was an experimental patch to the PHP build system I dropped in a bid to provide a more "vanilla" PHP. To be honest, it wasn't in hoffies eblits for PHP-5.3, as the patch would have probably needed modifications to it in several places. And I was not in favor of adding another eblit to accomodate PHP-5.2 here. To be even more honest, I'm scared by PHP's build system so much I want to destroy and rebuild it from the ashes

News update: php-5.2.14 on the fast lane to stable

Our security team got around to filing Bug #332039, which covers all the security fixes in PHP-5.2.14 and PHP-5.3.3.

It's Gentoo policy to have a normal testing period of approximately 30 days. The period can be shortened or omitted if there are security problems in a current stable version. This is what's happening in the transition from php-5.2.13 to 5.2.14 right now. I intentionally migrated 5.2.14 from the overlay a few days after 5.3.3, so we could test-drive the new eblits backed system with 5.2.14 that makes maintaining PHP so much easier. While all eblits originally were designed for use in 5.3.2, I only needed to update one eblit to accommodate 5.2.14.

But there was another change with 5.2.14 that will enable me to fix bug #310383 once php-5.2.14 is stable on all architectures: 5.2.14 finally makes use of EAPI2. It heavily uses USE dependencies, and also, by popular request, includes USE defaults. This feature auto-selects USE flags if you don't disable them via /etc/portage/package.use. By default, you're now getting the same PHP features you'd get by downloading the source and just running ./configure.

The change to USE defaults also obsoletes settings certain USE flags in the default make.conf, which is what bug #310383 is about.

There's also a downside to the heavy use of EAPI2 features: we now force certain USE flags on, if you enable a specific USE flag. This is a replacement for the manual die message you've probably seen once in a while. An example of this is USE="truetype", which needs the gd (or gd-external) USE flag on. This currently results in two kinds of error messages, a nice one and a very confusing one. Observe (sorry about the wrapping here):

terra mabi # USE="readline libedit" emerge -1 php
These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild N ] dev-libs/libedit-20090923.3.0 456 kB
[ebuild R ] dev-lang/php-5.3.3-r1 USE="libedit* readline" 0 kB

Total: 2 packages (1 new, 1 reinstall), Size of downloads: 456 kB

!!! Multiple package instances within a single package slot have been pulled
!!! into the dependency graph, resulting in a slot conflict:

dev-lang/php:5

('installed', '/', 'dev-lang/php-5.3.3-r1', 'nomerge') pulled in by
=dev-lang/php-5.3.3-r1[-libedit] required by ('ebuild', '/', 'dev-lang/php-5.3.3-r1', 'merge')
(and 1 more)

('ebuild', '/', 'dev-lang/php-5.3.3-r1', 'merge') pulled in by
php

Explanation:

New USE for 'dev-lang/php:5' are incorrectly set. In order to solve
this, adjust USE to satisfy '=dev-lang/php-5.3.3-r1[-libedit]'.

While I find the fact that portage is complaining about a "slot" problem misleading, it gives a nice and clear explanation about what to do to recover. Compare to this:

terra mabi # USE="sharedext sharedmem threads" emerge =dev-lang/php-5.2.14

These are the packages that would be merged, in order:

Calculating dependencies... done!

emerge: there are no ebuilds built with USE flags to satisfy "=dev-lang/php-5.2.14[-threads]".
!!! One of the following packages is required to complete your request:
- dev-lang/php-5.2.14 (Change USE: -threads)
(dependency required by "dev-lang/php-5.2.14" [ebuild])
(dependency required by "=dev-lang/php-5.2.14" [argument])

Duh! I've no clue about the root issue by looking at the error message. Turns out USE="sharedmem" clashes with USE="threads", but you can only discover this by looking at the ebuild. I'm currently working out how to get you a more informative message.

That's the news for today. Stay tuned while we work out the details about slotting PHP to minor versions (certainly the topic of one of the next posts), which we hope to have out before PHP gets a 5.4 version.

EOL for USE=concurrentmodphp

As previously mentioned, concurrentmodphp is currently available as a USE flag for dev-lang/php. It enables you to run two versions of PHP in parallel, loaded into your apache's mod_php.

With the advent of FastCGI/fpm in PHP-5.3.3 (very soon on a mirror near you), the preferred way to run multiple versions of PHP in parallel is CGI. Not only can you run several PHP versions independently, you also get proper script isolation (running each script as a different user, if you wish so) and thus enhanced security.

This is why I've decided to end support for USE=concurrentmodphp after php-5.3.3 and php-5.2.14. Versions after those will simply stop shipping the patches required to support this USE flag. Please use the time with PHP-5.3.3 to switch to fpm, if you need the functionality currently provided by concurrentmodphp.

Using multiple PHP versions

This is another "state of php in gentoo" article. Specifically, I want to talk about using SLOTs with PHP.

The current situation was created during the switch from PHP-4 to PHP-5, one of the biggest and more drastic in terms of features. PHP-5 added a full blown object oriented system, ending PHP's dubious fame as a highly advanced template engine and easing work on large code base projects. Clearly, porting your code from using PHP-4 to 5 was a tremendous effort if you wanted to take advantage of all the new and powerful features PHP-5 offered. So the then gentoo php team decided to make it possible to install PHP-4 and 5 alongside, easing migration testing and extension usage.
What they did was to slot PHP to it's major version. This is why today you're left with dev-lang/php in one slot: "5".

In a meeting on 11th April, the gentoo php team decided to halt decisions on slotting less (i.e. removing slotting) or more (i.e. on minor versions), due to manpower restrictions. This was chiefly due to our desire to get php-5.3.2 into the tree as fast as possible and we needed every minute we could contribute to gentoo at that time to this task.

Now that php-5.3.2 is in the tree and I'm slowly killing the remaining bugs left (helped, and mostly driven, by many friendly contributors, thank you!), we can focus on other areas to improve the overall quality of gentoo's PHP offers.

So slotting became an issue once again.

In an informal meeting, we agreed that removing slotting was the easiest solution, but not the best. While slotting to minor versions increases ebuild complexity and comes with it's own problems (you need a way to select the "active" PHP installation to run for the apache module, for example), the advantages in migration testing and also for shared hosting providers, who want to offer more than one PHP version, trumped the relatively low cost of slotting.

Here's a short pro-contra overview of minor version slotting I came up with:
pro

• test your code against the most current PHP version on the same machine


• provide your customers the PHP they want

contra

• need to remove slots you don't need (or they'll take up hard disk space)


• increased complexity in configuration

That's pretty short and I'm sure you can come up with more benefits and disadvantages. Do let me know in the comments!

I'm still somewhat disappointed with the lack of "end-user" benefits in this move. If you just want to run that cool webservice your friend has been telling you about, you'll emerge php and will receive slot updates instead of the normal ones thereafter. You regularly want to run emerge --depclean, so you don't keep around PHP slots you no longer need (thanks to darkside for helping me with this one).

Another hurdle will be that we require you to "enable" the PHP version you want via eselect to benefit from the slotting. This affects for what PHP version your PHP extensions like apc or suhosin will get compiled (each minor version has a different ABI, so for each SLOT you need a special build). We're trying to make it so that not "enabling" anything will guess the correct PHP version (probably your highest slot), but that's not sure right now. You will also use eselect to switch from one php apache module to another.

Speaking of apache modules, I've another proposal: currently, Gentoo supports a highly experimental feature called concurrentmodphp. It's a USE flag for dev-lang/php and enables you to run multiple versions of php in the same apache. Usually, this results in a symbol name clash of the php modules you try to load, but with this USE flag enabled, it should work. That said, I never used it, did not test it and upstream will tell you to rebuild without it if you do and run into problems. It's in no way supported neither by Apache or PHP. What is supported is running multiple versions of whatever you like via cgi (fpm, for that matter). So if you want that functionality, you can get it anyway, without causing maintenance problems for the team.

So here's the proposal: I will drop the concurrentmodphp functionality starting in August, if nobody objects with a technical argument on why to keep it. I'd like to hear cases where concurrentmodphp solves a problem fpm can't handle.

Anyway, if you want to help with the slotting efforts, please visit #gentoo-php on freenode, tell us to pull your ideas from your git repository (you can clone your local copy off of our slotting branch) or open bugs explaining why such and such change in said branch won't work, kills kittens or is otherwise undesirable. As always: comments welcome!

Why you want PHP-5.3.2

In this post, I'm going to spend some bytes telling you why PHP-5.3(.2) is awesome. The goal is to get you wanting to type echo "=dev-lang/php-5.3.2" >> /etc/portage/package.keywords followed by emerge php after reading this.

In my current little survey, 6 out of 7 who have voted so far, said that their code is ready for PHP-5.3. So they probably know about the goodies already and hopefully have already tried Gentoo's all new and shiny php-5.3.2 ebuild or even the php-5.3.3 RCs Ole maintains in our overlay.

For the rest of you, I'll highlight some points of the PHP-5.3 migration list:

Late Static Binding

Here's what the PHP website says: [...] can be used to reference the called class in a context of static inheritance [...] So far, so unclear. But it continues:

[...] static:: will no longer be resolved using the class where the method is defined but it will rather be computed using runtime information.

This is most useful in a class hierarchy, where your (abstract) base class does most of the work, but needs bits of information from the child class. Using static::, you can call back to methods of your child class from the base class without your base class having to know about the particular child class at all! This saves you numerous instanceof tests.

I use this in a MVC-Framework, where a BaseController holds a generic way to generate pagination data and forward it to the view. It also collects cookie and database information to save and restore user's preferences. This forced the cookie and database query methods into the ChildControllers, as the particular cookie or database column name was specified there. With PHP-5.3 I can now access those bits via static:: calls in the base class! No need for code duplication anymore.

Closures

If PHP-5.3 had contained no new features but this one, I'd still be happy. Closures are a powerful concept, but as the Late Static Binding above, can take a little bit of time to get into. In short, you can do everything you could do with create_function but instead of hiding all your code in a big string (loosing all the syntax highlighting and error checking of your editor), you now have perfectly valid PHP code inside a function you can declare about anywhere you could use a statement. You can assign your function like so:

$f = function($v) { return $v*$v; }
$f(2); // 4

Or you can use it directly, like so:

$squares = array_map(function($v){ return $v*$v; }, range(1,10));

This even saved me some precious horizontal space compared to array_map(create_function('$v', 'return $v*$v;'), range(1,10));

For additional coolness, you can let the closure use data from the parent scope (the context it is defined in, not the one it is run in), something you could achieve using create_function and embedding the extern variable in the string - it's so ugly I won't repeat it here. But look at the cleanliness of the closure approach (taken from the PHP manual):

$callback =
function ($quantity, $product) use ($tax, &$total)
{
$pricePerItem = constant(__CLASS__ . "::PRICE_" .
strtoupper($product));
$total += ($pricePerItem * $quantity) * ($tax + 1.0);
};

array_walk($this->products, $callback);

You definitely want to give closures a try!

Ternary default operator

Folks have lobbied long and hard for this. In short, it's your well-known ternary operator (bool ? if-true : if-false) shortened to return the evaluated ("bool" in my case) statement if you supply nothing for the "if-true" part but "bool" actually evaluated to true. Here's an example:

return 'gentoo is cool' ?: 'who wants to read this, anyway?';

This will return "gentoo is cool", as a non-zero-length string evaluates to true and nothing was given to be return in this case.

New date/time functions

Not much to see here: you can now use date_add(), date_diff() and date_sub() for which I've been waiting for ever since I needed to display dates and calculate expiry times. strtotime() is good for a lot of things, but sometimes you need more.

Misc

Some new stuff I still need to test out:

• The new mysqlnd library now returns correctly typed data instead of strings only for a query result


• SPL gained a host of basic data structures like Heap, PriorityQueue or Stack
  
• Phar archive support is now shipped with PHP
  

As expected, some things stopped to work in PHP-5.3 - they are documented in the list of Backward Incompatible Changes. The most notable changes for my code were various array functions (like usort) not accepting anything besides arrays anymore and call_user_method() now issuing E_DEPRECATED notices. Also note that the POSIX regular expressions are now deprecated in favor of Perl regexes.

That's it for now, I hope you're on your way to emerge php (if you haven't done so already)!

PHP-5.3.2 availability in ~arch imminent

So, another status update on PHP... If you're following PHP in Gentoo closely, you might already fear this will turn into a never-ending story. It's been over a year bug #274512, requesting 5.3 to be added, has been filed and we're now three days after the release of PHP-5.3.0. Well, sorry! Real-life has taken a heavy toll with all of the php team members.

Okay, on the bright side, the PHP team has had tremendous support in the last months. Thanks again to Vladimir Tsisaruk and Ole Markus With, who I already mentioned in the last post. They were joined by more awesome tinderbox testing by Diego (flameeyes) - many thanks for the dozen bugs you found and filed! Thanks to Christian Hoffmann for background discussion and insights, too! As well as an uncounted number of users who helped on bugs, you were most helpful.

And now the news: we currently have 60 bugs assigned to the php team, and no more critical bugs left on the PHP-5.3 Tracker. And as promised, I'll push php-5.3.2 out of package.mask accordingly. This is your one-time heads-up that php-5.3.2 will hit your testing system tomorrow, 04.07.2010, around 10.00GMT.

In other news: I touched a few packages today, updating them for PHP-5.3. If I stepped on someone's toes while doing this: I'm sorry if I broke something! If you find a message in your package's Changelog mentioning bug #298205 that means I updated your package to EAPI2 or just updated your {R,}DEPENDs to directly depend on "dev-lang/php" and whatever USE-flags you needed. To get the gory details, just have a look at the linked bug.

So, to sum it up: php-5.3.2 will be available without more work on your part if you're running a testing amd64 or x86 system tomorrow (4.7.2010)! Sorry to the other arches, please help your respective arch-team test the PHP-5.3 awesomeness on bug #321743. It will still have raw edges and some packages still fail to compile with it (have a look at the bugs assigned to php-bugs@gentoo.org to get the full picture), but for the overwhelming part of our tree it works just fine now! Enjoy!

P.S.: I'll start on a post detailing the changes coming your way with PHP-5.3.2 right away...

PHP-5.3 status

In this post, I'm going to talk a bit about what the current status of PHP-5.3 in gentoo is.

In summary, dev-lang/php-5.3.2 will be the version to enter testing (~arch) on 20th June if nothing special happens.

Now, what's that "special" thing that shouldn't happen?

If you follow our Bugzilla, in this case Bug 312775, you will see quite some bugs attached to this tracker bug. This is where the php team collects everything that relates to getting our users PHP-5.3. The team, and I'd like to extend that to mean our non-dev heavy-duty contributors Ole Markus With and Vladimir Tsisaruk along with our very own Christian Hoffmann, has put quite some effort into testing PHP-5.3 already. Everything they and others have found is filed in a separate bug per issue and added to that tracker bug. Just take a look.

When Christian Hoffmann designed the php-5.3 ebuild, one decision was to minimize the "eclass-clutter" caused by it. Currently, every (minor-)release of PHP needs a new eclass, so the micro releases (5.2.1, 5.2.2, and so forth) won't duplicate an enormous amount of code (e.g. 734 lines in php5_2-sapi.eclass). These eclasses were specific to the dev-lang/php package and a hold-over from the earliest days of PHP in gentoo. Hoffie exchanged the eclass with the concept of eblits, used in sys-libs/glibc and sys-kernel/mips-sources. So in dev-lang/php-5.3.2, all phases like src_unpack, src_compile, etc. are literally outsourced to different files residing in files/eblits. This allows us to share common code without the need to clutter the "global" eclass space. Horray!

But as with every drastic change, bugs will show up. This is why the php-5.3 ebuilds were developed out of the view of most of our users. Nobody likes their production site to go offline because the team missed some crucial files, mistyped variables and released a general mayhem, right?
Currently, dev-lang/php-5.3.2 is package.mask'ed in the portage tree. This means that you need to enter it into your package.unmask file to install it. If you're a standard portage user

echo '=dev-lang/php-5.3.2' >> /etc/portage/package.unmask

should do just that.

The current php-5.3.2 still needs all the testing it can get!

If you want to help, please unmask the ebuild locally as shown above and

emerge -av php

to see what it will do. Note that

pcre, spl, reflection

USE flags are gone - their functionality is now part of the PHP core.

If you encounter a bug while using or installing php-5.3.2, please report it on our Bugzilla!

Still want to know what "special things" can prevent php-5.3.2 from being available without the manual unmasking step above? We will not unmask the package if there is a blocking bug on the above mentioned Tracker present. We want to make sure most of the tree will "just work[tm]" when you update PHP. What exactly constitutes a blocking bug is determined by impact - php update ate your system? apache crashes after php update? your application doesn't work anymore? We want to fix that before it hits more users. So please send us your bug reports!

One last special note: I lack the hardware to test PHP on the less common architectures like SPARC or PowerPC. If you own such a machine, our arch teams will be delighted to see your test report attached to Bug 321743. To test, keyword php-5.3.2 locally (use "x86" even if you're not on x86 machines!):

echo '=dev-lang/php-5.3.2 ~x86 x86' >> /etc/portage/package.keywords

and unmask it as described above. Continue to install PHP normally. Please attach the output of "emerge --info php" in any case and a full build.log in the event of a build-failure.

Thanks for reading and stay tuned for the next post where I report on the changes that will come with PHP-5.3

Introduction aka Why yet another blog?

In short: I've been updating Gentoo's PHP-related packages lately. The communication mostly happened on IRC or via the team's wiki and I wanted to reach a wider audience as we near the introduction of PHP-5.3 and some other high impact stuff.

I already mentioned IRC and wiki as two "news" outlets, so why another one?
Well, for the first, IRC is transitory, time-dependent and a high-noise medium. It is not suited for one-to-many communication and anyone who has ever tried reading an IRC-log values a well written summary. The PHP team currently does this on the team's wiki. And if it were just for technical details and archive functionality, the wiki would be totally sufficient.

So this blog is for reaching out to developers and users alike to get a (hopefully) current grasp of all the things happening inside the gentoo PHP team. Expect to see explanations and heads-up for new PHP versions, eclass/eblits/package RFCs and the occasional general PHP post on stuff currently in the works upstream.

I'm not going to discuss PHP programming here. This is not a tutorial space. However, if you have cool (read: big, heavily modified, servings truckloads of users, etc.) applications of gentoo PHP somewhere, just email me at mabi@gentoo.org and I may feature it here.

See you on the flip side, Matti