![Popular open source software is more secure than unpopular open source software, because insecure software becomes unpopular fast. [That doesn't happen for proprietary software.]](../sites/all/themes/drupal4hu/images/bg-center/bg-center_4.png)
We have written some very frustrated blog posts. Time to reflect on them and outline some causes and plans to move forward.
1. Why are we so frustrated?
There have been a small number of people consistently fighting the major/critical bug queue for a long time. We got Drupal 7 released (through patching and reviewing some 8000 issues) but after that we needed to continue the bug fighting and this really got on people's nerves. For example the upgrade path has been critically broken for two years without a break.
We went through a hellish period in Drupal 7 where we had hundreds of critical bugs - both from letting the queue get out of hand, and actually having a lot of bugs in core. To try to stop this happening again for Drupal 8, and to limit what can happen there while so many outstanding bugs are in D7, we put thresholds in place. But this comes at a cost: people like writing cool new features a lot more than fixing these radioactive, ugly bugs and core development have been stuck fixing the bugs for a couple of years already (or bowed out while that happens and not come back yet). Although if you want a gold star on your resume, fix some of these :)
So what we really need here are two things: get more new people into the bugfighting trenches, and unblock day-to-day development of core for non-bugfixes. We already started office hours, we have just started the core announcement feed, we will announce specific bugs we will work on a given day (week?) and mentor people to work on them. Getting more people involved in bugfixing is definitely not a pipe dream, boobaa fixed a very gnarly upgrade critical as his first core contribution, this needs to be sustainable though so people don't leave as quickly as others get involved.
Second, once we are down to something like 70 instead of 100 majors, and Drupal 8 progress quits being blocked every other day, and Drupal 8 advances and becomes more discernible from Drupal 7, more people will come. See 3. for more on this.
Finally, webchick is going to write a blog post about some other some efforts to rope in more core contributors. Stay tuned! But also please share your ideas about how to get more.
2. Dries was unavailable for a long time, unable to commit sufficient time to core and meta issues like process you read a lot about in this blog post (like the tresholds above). However, Dries is actively working on this, and his availability will sharply increase as he is done with reorganizing the Drupal Association and also he is hiring people (like webchick and Moshe) to take over some of his Acquia responsibilities.
3. There's little doubt that Drupal needs some major refactoring, most importantly we want to change the extremely tightly coupled nature of Drupal.
We think that even the most drastic refactoring can happen within the current process without a ground-up rewrite. Major changes happened in Drupal 7 such as DBTNG without starting again from scratch. However core processes need to be updated to incorporate refactoring as well.
We discussed such drastic use cases as changing the API and internals for module_invoke_all() and drupal_alter() -- even that can be done step by step by introducing first a module_invoke2, converting core hook by hook and at the end rename. Once an API change is agreed, if it's possible to add a backwards compatibility layer temporarily, it allows work to be done in manageable chunks. This worked for the database API in Drupal 7 - which was committed around 18 months before the bc layer was removed. It needs be very clear such things are temporary -- we still dont want to be bloated permanently by being backwards compatible -- some ideas include throwing deprecated errors from the old implementation and adding doxygen indicated the deprecated state.
We may also need to allow for some breakage when major changes are committed, but the issue thresholds will allow us to do that without things getting completely out of hand. If thresholds are breached unintentionally then it will hopefully lead to an 'all hands on deck' situation.
To avoid months of reroll hell we started scheduling big changes -- once we have agreed on a given API, a given change there is a specific date when it will get in.
4. We do not want to move many modules into contrib because it's unclear who would maintain them there. Of course, it's a burden on core developers but keeping API consumers around is a good way to verify whether our APIs work. However, we will evaluate modules like php, profile and trigger. But, we want the new D7UX modules to stay because we are a product and not a framework. Drupal actually is a sucky framework, Zend, Symfony 2 etc. are much better frameworks. Our strength is we are a product and we need to focus on that. Right now these modules are a burden but we hope that as outlined in 1. the core team (whatever that means) will work hard to find maintainers for them. We might want to spin them off in the future but only when we know who is going to maintain them because we need them for the product to succeed. Success is a multi facet issue: also it's easier to suck people into core development by getting them fix user facing bugs. webchick just called these gateway drugs.
Also, look at Linux on the desktop! If you do not focus on having a convincing product for end users, guess what? Technical prowess wont impress them. The year of Desktop Linux never happened. Not even when Windows Vista appeared after so many years and it was crap. This a big warning for us. (Yes, the Linux kernel is a success but I am talking of the desktop.) Also note that while the previous points were a result of a community discussion, this paragraph is mine only.
Commenting on this Story is closed.
Given there's already an effort to use some of Symfony for D8 (HTTP layer, more too IIRC), could Drupal itself be written as a layer on top of Symfony? Clearly, though, there'd be a lot of compatibility to deal with, and Drupal is far beyond the days of being just nodes + comment + users, but it could be done and might be a good fit. This would give us what appears to be an excellent "true" framework + Drupaly goodness at the bottom layer, plus lots of CMS goodness at the top.
"Our strength is we are a product and we need to focus on that."
But if Drupal's strength is being a product, who is that product targeted at? What is it supposed to do?
Make websites? Make making websites easy? Make making complex websites easy? Should I go on?
Don't make it more difficult than it is.
To me, Drupal is a half-baked, kludgy (but still flexible) framework with a lot of tools glued on, some of them great and usable and powerful (like Views) and some half-finished (like Media). In fact, far too many tools are REALLY unfinished or have lame UI and there are so many tools in general that I cannot image in Drupal will EVER become a polished "product." But Drupal allows some sites to be built faster than if starting from scratch with a framework but not nearly as fast as a site that can be built with Wordpress or the like. So it seems Drupal is neither a great framework nor a polished product (of any kind) but it is a serviceable (and at times fantastic) tool for making an awful lot of different kinds of sites. I know you know this already.
What I really want to say is I think you are unreasonably dismissing the small core movement by way of saying Drupal is a product. I know you of all people desperately want to improve the core's framework. But can't we go small core and still keep a "standard installation" product or choice of several that is maintained in a "core-like" fashion? similar to Acquia Drupal's bundled contrib modules I suppose. But it could perhaps have an independent release schedule, especially if those modules and features were completely uncoupled from the core framework. Many people in the community say we an have our cake and eat it, too. So what prevents this (designating a "Small core" and then de-coupled product on top) actually? Is it Dries? Is it a matter of a vote? Who gets to vote? Who (besides, apparently, you) is actually against it? Just curious.
You move the product to contrib right now and it dies. First we need to get people to maintain it and then we discuss whether we want to spin it off. Because it's going to be a sizable effort to find and mentor the people to maintain them even using the people who focus on core. You put it in contrib now and then who will find those people and get them to the point where they can actually take responsibility for the product?
What happened to scratch your own itch? Surely, if some of these modules/features are really needed by enough people/developers, then they will feel compelled to contribute to maintaining them, won't they? I am asking naively. If experience has taught you otherwise, I respect that.
You move the product to contrib right now and it dies.
Bullshit. We are talking about core crisis, not contrib crisis, right ? Contrib is OK, core is not. So it works opposite: have the product in core and it dies, because people building sites don't want to develop the product they need to fight with.
Totally agree, more developers work on contrib because there's shorter release cycles and its easier to get involved.
Some things need to be allowed to die. This doesn't include every module that core ships with, but it applies to several.
"You move the product to contrib right now and it dies."
Agreed with Anon1
Surely, if some of these modules/features are really needed by enough people/developers, then they will feel compelled to contribute to maintaining them, won't they?
Agreed with Anon2
Bullshit. We are talking about core crisis, not contrib crisis, right ? Contrib is OK, core is not.
Agreed with Anon3
More developers work on contrib because there's shorter release cycles and its easier to get involved.
Agreed with Anon4
Some things need to be allowed to die.
Agreed with bojanz
Then let them die. This is open source. Everything worth maintaining is being maintained. If there is code that nobody wants to touch with a ten foot pole, then there's a reason for that.
Drupal may not be a PHP framework, but it's definitely a content management framework.
Indeed. It's certainly being used as a framework over Zend and Symfony by many organisations. Before we develop an inferiority complex about lower level PHP frameworks modelled on Rails, it's worth noting that there are no successful CMS or CMF projects based on them - and this is despite every serious developer you talk to having written multiple CMS's. It's an apple and pears comparison in any case, Zend is more of a library, both use a WebMVC ActiveRecord approach, and neither is designed to model event-driven systems. When the key words that people use to describe Drupal are 'flexible' and 'extensible', it seems pretty off-the-mark to claim it is not a framework.
Nice to see a constructive approach on how to move forward.
Could you elaborate on ... and mentor people to work on them ...?
Will this be ad-hoc mentoring using IRC stuff (like currently is happening) or have there been discussion on more structured support for mentoring ?
I've been listening to several people that have been tutoring friends (who didn't know drupal) and they expressed they would considere tutoring other people if there was an easy way to do it .... thus a request for more structured support for tutoring exist. It may not only be relevant for bug fixes but also for growing the hobbyists. We actually have tutoring hobbyist that we are not able to contribute at the moment (who would have guested).
One way we could have a simple and quick mentoring tool would be an issue queue -- call it a student queue -- where different interest (e.g. contributing to core, a particular module, documentation, etc.) can be expressed and current skill level (novice, basic, advanced) can be indicated. The student could elaborate what they are interested in contributing/learning/solving. Assigning to a student means you take tutoring responsibility of course.
The closest we have to structured mentoring is core office hours - two slots per week where people aim to be on irc. This has just started in the past couple of weeks so it is still very early days - we could definitely use more volunteers, and potentially more slots as well. This is partly based on the views bug squad which was created to solve similar issues. There has been ad-hoc mentoring in irc for years, office hours are mainly about making it more obvious that it exists.
It's a bit different to mentoring people on how to use Drupal for something, that feels more in the realm of training/tutoring than mentoring.
I like the separation between training an mentoring. To me training is a business issue while mentoring is a community issue.
I see how it should work for core, but maybe this is exactly the problem, it narrow down the potential bug fixers to people that can directly work with all the required tools (IRC, git) and have all the prerequisite knowledge (5 years ago Drupal was not that complex). This is increasingly becoming a huge gap. More structured mentoring tools would be a stepping stone. Just as we have core and contrib. development, can't we have contrib. mentoring too please?
I can name you a hand full of people who would love to be such kind of mentor. They are not experts on core, but they know a big deal about Drupal. Some of them work on contrib. modules, but other work on documentation or other non-core contribution.
There's the Views bug squad, which core office hours is partially lifted from: http://drupal.org/node/945414
There's also some discussion about trying to convert the project application process to more of a mentoring thing, the whole process is being discussed at DrupalCon in some depth and it'll be interesting to see what comes out of that.
A couple of years ago there was a plan to make GHOP a year-round thing (and call it drop), that didn't really happen but probably could if a few people put effort in to revive it.
Could you give a link about the project application process?
In the mean time we created an infrastructure issue (http://drupal.org/node/1261188). This is really something I like to work on.
Half agreed. Now, here is my constructive.
We are on Quantum entanglement here:
1.) No, Drupal is a platform for many companies.
2.) Yes, Drupal is a product for many companies.
Then:
- 1.) Is a need 2.) is want.
- 1.) CAN happened WITHOUT 2.) in short-term, but for long term Drupal can't grow.
- 2.) CAN happened WITHOUT 1.) in short-term, but instantly Drupal Platform Fork.
Then 2 Core Team:
1.) Core Drupal Platform Team. Don't have to maintain and fix the issue queue for product modules.
2.) Core Drupal Product Team. Maintain and fix the issue queue for product modules.
Then 2 priority, seperate issue queue:
1.) Core Drupal Platform Team. Create, maintain and fix issue queue for the Drupal platform.
2.) Core Drupal Product Team. Create, maintain and fix issue queue for the Drupal product modules.
Then:
1.) Developers who like to work more on platform, no product issue overwhelm.
2.) Developers who like to work more on product, no product rant here.
Then:
1.) Drupal Platform can be independent from Drupal product on developments, features, timeframe, launch date etc.
2.) Drupal Product can be independent from Drupal product developments, features, timeframe, launch date etc.
Then:
1.) Current Drupal Core developer happy and and take responsibility to work on Drupal Platform.
2.) Acquia and ????_company happy and take responsibility to work on Drupal product.
Conclusion:
1.) CHX, are you sure you still want to work on both 1.) and 2.)?
2.) Why Developer who are not happy about Drupal the product still have to work on it without payment? Let's the company who benefit ten millions from it RESPONSE THEMSELF, HERE IN CORE ;)
That's fair to me.
To do any of that, you need to decide what goes in the product and what goes in the platform, and it's at this point that working on that up front completely breaks down.
Probably everyone could agree that forum module is a 'product' module. But what about taxonomy, 90% of user module, node, comment, Field API, large parts of system mdoule. As soon as you look at those modules, what they are, how they're used, then an up front split is impossible. To figure that out requires http://drupal.org/node/1224666 to be done first. This doesn't mean we can't look at killing off some core modules that no-one loves, like PHP or trigger regardless of later decisions.
Install profiles should be the product, not the default Drupal download with old modules like forum, trigger and blog modules. If you look at it like this lot of core modules could be put to contrib right away.
You can easily build a blog layout and a forum layout with simple nodes via Views and CCK.
Right, several could, but some couldn't. So you should go down the list of core modules and pick them out one by one. It's likely to will end up with "yes", "no" and "maybe" columns. I don't remember anyone who has posted about how easy it would be to have a core 'product' profile actually doing that and posting the results.
Nice post, adding to the flow of voices raising concerns about where Drupal is headed. I would add 2 comments to this;
4. We do not want to move many modules into contrib because it's unclear who would maintain them there.
Then let them die. This is open source. Everything worth maintaining is being maintained. If there is code that nobody wants to touch with a ten foot pole, then there's a reason for that.
The talks about Drupal as a product are very silly at times. Drupal without contrib is a very sucky product. Find me people who use Drupal core with no contrib modules added. I will buy beer for both of them.
I've mainly been watching the various blog posts on the "Drupal Crisis" via Drupal Planet. I haven't had time to find where the real discussions have been happening. Especially since a lot goes on in IRC and voice chat.
So, here's what I think things are settling down to: Essentially three levels. core, then official install profiles, then Contrib modules and install profiles.
Is that correct?
Also, when I see posts about fixing Drupal, most of them talk about it as part of D8. D7 seems to get very little consideration. Snowman is trying to be ready for inclusion in D8. The initiatives that Dries started are all targeted at D8. Essentially, if it's a real problem that needs fixing really badly, the fix is being put off until D8. The only major D8 fix I know of that is also trying to do work in D7 as well is the configuration management initiative.
Am I just missing something? Again, I haven't had time to spend looking through core issues or the various groups where discussions probably are happening. (Oh, another thing to fix, make it easier to participate in this kind of discussion. ;) )
The reason this concerns me is that I don't know when D8 will be ready for production. And most of the issues Snowman and smallcore are trying to solve are issues that I need fixed really soon. Are these thing too complicated to do anything with in D7? Or could some of the work get pulled into D7 before D8 is released?