tag:blogger.com,1999:blog-35858418383399629092024-03-13T22:49:39.922-07:00PloneChixEmpowered by Ploneeleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.comBlogger20125tag:blogger.com,1999:blog-3585841838339962909.post-53357910932339195512014-05-01T16:13:00.001-07:002014-05-01T16:13:14.915-07:002006 Called and Wants its Apache Config Back<h3>
<b>Summary</b></h3>
Upgrade Apache, stop using Rewrite Rules and go back to ProxyPass.<br />
<h3>
Details</h3>
This is a quicky but hopefully it will save someone else from my pain. We upgraded our apache instance from 2.2 to 2.4 in the last few months (with some, but not a ton of pain). The biggest change IMO was the way permissions work, moving away from <span style="background-color: #f3f5f7; font-family: courier, monospace; font-size: 16px; white-space: pre-wrap;">Order allow,deny</span> syntax and towards <a href="http://httpd.apache.org/docs/2.4/mod/mod_authz_host.html">Require</a> syntax. I followed the upgrade guide and everything started normally and Rewrite rules were all good so I did the lazy admin thing and wrapped it up.<br />
<div>
<br /></div>
<div>
My papertrail alerts notified me with this log, on a completely blocked off site:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://www.evernote.com/shard/s1/sh/5857444a-291e-4a9d-b57f-9dc865b85f24/aaf4ac496b8304b5d2b09e758e13e1a4/res/fc587337-967a-4434-a54e-a9765222921e/skitch.png?resizeSmall&width=832" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="128" src="https://www.evernote.com/shard/s1/sh/5857444a-291e-4a9d-b57f-9dc865b85f24/aaf4ac496b8304b5d2b09e758e13e1a4/res/fc587337-967a-4434-a54e-a9765222921e/skitch.png?resizeSmall&width=832" width="640" /></a></div>
<div>
<br /></div>
<div>
Sadly, this error is actually just a canary. I discovered that someone in Poznan [1] has been using our server as a <a href="https://wiki.apache.org/httpd/ProxyAbuse">Proxy</a> [3]. <i>Guh</i>.</div>
<div>
<br /></div>
<div>
I validated this by setting my browser proxy for that subomain on for our http, not https domain (https didn't work) and then attempting to browse the internet [2]. Which worked. Moar <i>guh</i>. You could see in the logs that I was using our server as a passthrough, which is bad juju. I have no idea how or why, since we have ProxyRequests off and we were using rewrite with [L,P] and I was set on a traditional tried and true<a href="https://gist.github.com/eleddy/46717fae3a4a853008c7"> recommended Plone rewrite method</a>. I used ProxyPass rules then I could NOT use that server as a proxy, but if I used Rewrite with [P] then I could.<br />
<br />
I honestly don't understand 100%, but it has to do with the fact that <b>Require all granted</b> is needed to get to the rewrite rules, and because we are using apache like a proxy using <b>Require all denied</b> and then <b>Require local</b> just doesn't work (tail the auth logs at trace3 and you'll see something like <span style="font-size: x-small;"><i>client denied by server configuration: proxy:http://127.0.0.1</i></span> where that proxy prefix makes things "not work as expected"<span style="font-size: xx-small;">(tm)</span>. The docs recommend using <a href="http://httpd.apache.org/docs/trunk/mod/mod_remoteip.html">mod_remoteip</a> to get around this limitation but I tried unsuccessfully many variations of this and just got nowhere.</div>
<div>
<br />
During this long ass investigation, I also learned that using rewrite rules are <a href="http://httpd.apache.org/docs/current/rewrite/proxy.html">not the recommended</a> way to do things anymore anyways and furthermore are less performant because rewrites don't use Apache's worker pool. That's obviously no good.<br />
<br />
It would be easy enough to fix if we didn't have a lot of domain shit going on (who doesn't these days), which turns out to be pretty annoying and hard to resolve. I come to you with this config snippet, which should for just about anyone who wants to update to 2.4.<br />
<br />
<script src="https://gist.github.com/eleddy/9d2b899bea886c30f8f1.js"></script></div>
<div>
<br /></div>
<div>
If you use <a href="https://forge.puppetlabs.com/puppetlabs/apache">puppetlabs apache module</a> this will look like:</div>
<div>
<script src="https://gist.github.com/eleddy/5972128bfa6673845216.js"></script>
</div>
<div>
I still have not figured out a sexy one off solution for managing multiple subdomains this way and rewriting them to plone sites so you would need to rely on your trusty musty config management tool to do that for you (which you are using right???).<br />
<br />
I know I usually use this blog for more definitive advice but this is actually NOT one of those. I am extremely surprised at the ease which I set my own server up as a proxy and the pain it took to undo that. This post is part alternative method, part warning, and part documentation as to why I'm switching over from Rewrite to ProxyPass again.<br />
<br />
[1] To do this, look in your apache access logs and then perform a whois on the requesters ip.<br />
[2] You can also do this with telnet, but I got mixed results that way. I found the browser was just easier to jerry rig the test case.<br />
[3] Note that this doesn't necessarily mean that your server is compromised, but rather that some baddies can proxy through your servers and do baddie things. More importantly, your service host can find out and slap you with something if you are lucky or suspend your sever if you are not. DO NOT FEED THE BADDIES! Unless you are a sympathizer. Then honeypot up!</div>
eleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.com6tag:blogger.com,1999:blog-3585841838339962909.post-65612376188113297442013-10-18T14:32:00.001-07:002013-10-18T14:32:12.360-07:00Team Modernize<h3>
&TLDR;</h3>
Have you wanted to contribute to a major Plone release but felt nervous or perhaps feeling a bit like you don't have the chops (aka impostor syndrome)? Can you spare a 1-4 hours? Have I got a deal for you!<br />
<h3>
Details</h3>
It has been a VERY busy summer and I <i>may</i> have disappeared for a bit to actually make money but I'm getting back into the swing of things and rediscovering a <a href="https://dev.plone.org/ticket/13260">PLIP</a> I proposed, thanks to some lovely nagging by @bloodbare.<br />
<br />
In one year, many things have changed and Plone 5 is real. We started at the Plone conference in Arnhem and things just lagged - MY BAD! The problem with this PLIP was that it is a bit overwhelming and I just couldn't get motivated to continue. To be fair no one else did either because its a mega poopy task. It was difficult to track and the branches were dizzying. Lucky for us all, we can move forward differently in a way that is a bit cleaner.<br />
<br />
I have carefully broken apart this PLIP* into a lots of <a href="https://github.com/plone/Products.CMFPlone/issues?labels=PLIP+13260%3A+Remove+CMF&state=open">tiny little pieces</a>. Each task should be around 1-4 hours. Some may even be shorter than that. If you want to help with this PLIP all you have to do is:<br />
<br />
<ol>
<li>Get the Plone 5 buildout <a href="https://buildoutcoredev.readthedocs.org/en/latest/intro.html#setting-up-your-development-environment">running on your machine</a>. This is the master branch (it's so exciting!). If you have not signed a contributor agreement yet don't let that stop you. Send it my way to assignments@plone.org and we will get you started.</li>
<li>Accept a ticket from the <a href="https://github.com/plone/Products.CMFPlone/issues?labels=PLIP+13260%3A+Remove+CMF&state=open">list of baddies</a>. Some tickets are literally just "Delete these files and make sure no tests break". Yep. Get those before they are gone!</li>
<li>Modernize like mad</li>
<li>Run test cases locally</li>
<li>Commit the changes and make a pull request. Make sure to indicate the ticket number in the request. The pull requests go to master branch.</li>
<li>Make sure to followup with jenkins and the other robots to verify you didn't break the build. In theory they actually yell at you!</li>
<li>GOTO 2**</li>
</ol>
<br />
And that's it! For your help you will get:<br />
<br />
<ol>
<li>My undying admiration and appreciation</li>
<li>A chance to participate in a major open source software release (very cool stuff)</li>
<li>Upon Plone 5 launch, a t-shirt, laptop sticker, or fez celebrating the release of Plone 5 and indicating that you were a core contributor in the decrufting of this bad boy. ***</li>
<li>The chance to save that dirt old feature you love, or remove that buggy old feature you hate</li>
<li>A chance to go through the pull request cycle with a group of people who are committed to be very sweet while giving you code review and pointers. What other PLIP can guarantee that???</li>
</ol>
<div>
I am here to help through the whole process. If you are interested and still don't know where to start, just find <b>eleddy</b> on IRC and I'll walk you through it. These tickets are a once in a lifetime way to get involved in a major framework and a major release.</div>
<div>
<br /></div>
<div>
I can't do it alone, Ploners. Together, we can decruft this bad boy all the way to Plone 5. </div>
<br />
<br />
<br />
<span style="font-size: xx-small;">* There are indeed still some tickets remaining to be made and I promise I'll get to those soon! That doesn't mean I don't need help now!</span><br />
<span style="font-size: xx-small;">** or not. Seriously, if you just picked one ticket I would still jump for joy!</span><br />
<span style="font-size: xx-small;">*** I promise to think of something clever</span>eleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.com0tag:blogger.com,1999:blog-3585841838339962909.post-9744149280360769862013-06-14T12:11:00.000-07:002013-06-14T12:11:19.840-07:00Operation Flawless Login<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://www.evernote.com/shard/s1/sh/1f63bb67-63d3-4fd0-b30b-f2c6d243972e/e2699a76c491fbd51a242a9ea4b1ea41/deep/0/Screenshot%206/14/13%2011:35%20AM.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="200" src="https://www.evernote.com/shard/s1/sh/1f63bb67-63d3-4fd0-b30b-f2c6d243972e/e2699a76c491fbd51a242a9ea4b1ea41/deep/0/Screenshot%206/14/13%2011:35%20AM.png" width="173" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">No booze, no caffeine, training,<br />bad wifi, jetlag, and 100+ users<br />on every type of device under<br />the sun who couldn't reliably<br />log onto the system. PAIN.</td></tr>
</tbody></table>
<div class="p1">
I was going to do the whole play it cool thing and wait 3 days for the next post but I am just so excited about this new "blogging" trend. I'm hoping to make it a Friday tradition (preceded by 2 cups of drip coffee and followed by a nap).</div>
<div class="p2">
<br /></div>
<div class="p1">
One thing I'm super excited to start working on as part of the migration away from cpy/cpt is redoing login. As far as Plone goes, you are likely in 1 or 2 camps: </div>
<ol class="ol1">
<li class="li1">Works fine for me!</li>
<li class="li1">Rot in hell you !@#$ old code</li>
</ol>
<div class="p1">
I suspect most of you are in camp 1, or just have no opinion whatsoever so I'm hoping I can convince you to jump over to camp 2. Most of the following was originally written with an intense amount of pain (see photo) so if it seems like I'm shouting at times its because I was: I probably qualified as temporarily insane. Here we go!</div>
<div class="p1">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://www.evernote.com/shard/s1/sh/5ebd86b3-7af1-4eb0-aac1-1b1429fa5194/a512b7f3ffce423ce314c1eae521b3b5/deep/0/Screenshot%206/14/13%2011:48%20AM.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="316" src="https://www.evernote.com/shard/s1/sh/5ebd86b3-7af1-4eb0-aac1-1b1429fa5194/a512b7f3ffce423ce314c1eae521b3b5/deep/0/Screenshot%206/14/13%2011:48%20AM.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Ryan Foster (@dextermilo) recently proposed this beautiful, simple, login page for a client which has all css/js/etc embedded making it load very fast (1 template + logo). Tell me you don't want this login page HARD. I know I do.</td></tr>
</tbody></table>
<h3>
Login as a First Impression</h3>
<div class="p1">
For sites that are completely blocked off by default, login is the first experience with a new system/site. For anyone who is invited to a system, its the same case. In fact, I would argue that <b>a seamless login is the most important introduction to any web based system</b>. It should be flawless. Absolutely flawless. </div>
<div class="p2">
<br /></div>
<div class="p1">
And this is where "we the Plone" fall flat on our face. A sampling of the issues, mostly focused on the ones dear to my heart at the moment:</div>
<ul class="ul1">
<li class="li3">The login, registration, password reset pages are too slow from relying on a framework that it doesn't need, causing more complications with frustrated users. We can't login 100 people at once even with load balancing. Trainers around the world know exactly what I'm talking about. How can you ramp up a system that people can't login to fast and accurately?</li>
<li class="li1">We are 100% not ready for mobile. There are some issues with random phones that we will never be able to address but we also need to catch up with HOW people use mobile devices. The best example is the shitty network experience. What happens when you click on a registration link and the network goes out before you can complete? You go back to the email invite and RE-CLICK. But wait… whats that? Plone only allows you to click once and the link expires? 1999 called and wants their user restrictions back. </li>
<li class="li1">The messaging/wording of invite and password reset emails is robotic and offers no indication to what the user just signed up for or were invited to. They get the email and have questions like "Wait, so how to I get to the home page?" and "What is my username?" and "Who is admin?". If you have signed up for basically any service at any point lately I'm sure you know what a <a href="https://www.evernote.com/shard/s1/sh/90641af7-4a7b-4fa4-b9d5-0b6b2442a81c/4bf6e4d9b0b67c5d7a1f960ddc34a130/deep/0/Screenshot%206/14/13%2011:15%20AM.png">great</a> <a href="https://www.evernote.com/shard/s1/sh/6d15672c-832a-4b44-857c-48097bb69af4/8922b3d134a59b28486d26320a59ec38/deep/0/Screenshot%206/14/13%2011:16%20AM.png">invite</a> <a href="https://www.evernote.com/shard/s1/sh/cc6eb357-21e3-4a03-b7fb-60bdedfe06e0/c069bd4c14ba995bdf4b46b302f86bbb/deep/0/Screenshot%206/14/13%2011:17%20AM.png">email</a> feels like.</li>
<li class="li1">Testing? What's that? </li>
<li class="li3">Have you seen this code? <a href="https://github.com/plone/Products.CMFPlone/tree/master/Products/CMFPlone/skins/plone_login">I mean seriously</a> (WARNING: NSFW) look at that shit. It's 10 years of whacked out random bug fixes that no one has any idea if its ok to rip out or not. Its 20+ different files spewed throughout the code in portal_skins style that requires a diagram the size of my kitchen table to print out. Bugs languish in the tracker for years because everyone is terrified to touch it. It is officially <b>unmaintainable code</b>. Don't get me wrong, it was great in its hay day. But I'm sorry login_form.pt, it's time for us to go our separate ways.</li>
<li class="li1">Customization. Ugh. Enough said.</li>
<li class="li1">There is no OOB way to block off a site.</li>
<li class="li3">and so much more… I don't have time to list them all. Have a favorite login gripe? Comment below!</li>
</ul>
<div class="p1">
<b>Summary</b>: It's a liability to Plone and more importantly my projects, and my job these days is to minimize liability. I'm coming for you login!</div>
<div class="p1">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://www.evernote.com/shard/s1/sh/ada99401-566f-44ae-8e14-ae0ce0444a08/b631b4e0da1c5d3076d5816dec29e12d/deep/0/Screenshot%206/14/13%2011:30%20AM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="270" src="https://www.evernote.com/shard/s1/sh/ada99401-566f-44ae-8e14-ae0ce0444a08/b631b4e0da1c5d3076d5816dec29e12d/deep/0/Screenshot%206/14/13%2011:30%20AM.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Battling the login monster one head at a time takes too long. Instead we cut out the heart, then roast the heads in olive oil for 8 hours, top with a dollop of creme freche and a sprinkle of truffle oil. Serve immediately with a spicy Triple.</td></tr>
</tbody></table>
<h3>
I Love You but I've Chosen Rewrite</h3>
<div class="p1">
In good faith of open source complaining, I am determined to help fix it. In fact, <a href="https://github.com/hammertoe">Matt Hamilton</a> has already agreed to help and get this handled and I secretly suspect that he has more deep down pain than I since he is taking off with it and already planning a quick weekend sprint to kick it off. Are you one a member of the login burned? <a href="https://github.com/collective/plone.future.login">Help us out</a>!</div>
<div class="p2">
<br /></div>
<div class="p1">
Here is a brain dump of the things I personally want to accomplish with login rework:</div>
<ul class="ul1">
<li class="li1">Login gets dirt fast. It should handle 100 simultaneous logins on 1 site, 1 box, standard hardware. I want to say 300 but 100 is a good target and we can't have mass training sessions without this. I think a lot of the other options can help on this. I am tired of pointing to caching as the solution. There are a lot of transaction issues in this but we have to start somewhere. The skeletons oh the skeletons!</li>
<li class="li1">Stop basing on main template (this also applies to 404, 500, etc). If a site is closed off and resources were just recooked, thats 20+ resources the browser has to load and thats the immediate impression for the users. Get this down to less than 3 files (a login splash type thing, ideally 1 page with embedded resources). Ability to customize VERY easily (diazo?) but more importantly having a good default. </li>
<li class="li1">Mobile. Responsive, quick. Avoid redirects at all costs and where they are make it sane. I recently saw password reset on initial login fail on a variety of mobile devices. We are doing something wrong/old school/something. </li>
<li class="li1">Integrate iw.rejectanynmous to core. </li>
<li class="li1">Sane invite messages. Default system invite messages are horrid. They don't have site urls, nor indicate the username or who or what or… anything really. Ideally this message can be customized TTW (e.g. welcome to the site. this is our new intranet and I want you to use it like x,y,z your login is x the url is y and I am your new overlord)</li>
<li class="li1">Forgotten password. Oh god. Expire on click is a disaster, and the methodology is generally buggy.</li>
<li class="li1">Tests. Tests everywhere. I want robot going crazy too.</li>
<li class="li1">Security. Lots of it.</li>
<li class="li1">Login/reset/all forms rewritten as z3c. This allows us to extend and add things like site agreements. </li>
<li class="li1">Simplicity. There are too many options to be sane at the moment. We need to not even look at the code, and think "what do people REALLY need to configure in login" (especially that can't be done in PAS). We need to worry about our 80% use case first. I will be rabid about this. Login, login, login. And that's it. For now.</li>
<li class="li1">Rip out openid as it is. A simple pluggable solution is the answer to proper integration.</li>
<li class="li1">Default captcha for open registration.</li>
<li class="li1">Integrate some of the <a href="https://dev.plone.org/ticket/10959">login api work</a> that has been done. </li>
</ul>
<div class="p1">
And on my "after that is done" wish list:</div>
<ul class="ul1">
<li class="li1">Must reset password is weird and has never worked properly as I have wanted it implemented. The ability to expire passwords from some control panel has been on a wish list for many clients over the years. Time based expiration as well. That works, not that is a hack into the current whacky code base.</li>
<li class="li1">Tracking for site admins. Who hasn't logged in in 2 weeks? 1 month? Who has NEVER logged in? How can I make sure people are actually using the site? What kind of data do we need to capture to answer security compliance questions? We can't hide behind LDAP integrations for that forever. Also not first pass, but the ability to generate these reports must be considered.</li>
</ul>
<div class="p1">
In the category of "really dreamy some day if I can find the time":</div>
<ul class="ul1">
<li class="li3">Plone as oauth consumer. It's 2013 and "login with Facebook" isn't going away. *sigh*. I don't know that this should be in core but the ability to do setup oauth is HAWT.</li>
<li class="li3">Plone as an authentication provider. </li>
</ul>
<div class="p1">
We are ripping out portal_skins piecemeal so that we can get the new stuff out to integrators ahead of time to test then merge it back into core for the release. Login is its own package, <a href="https://github.com/collective/plone.future.login">plone.future.login</a> and anyone is welcome to help the rewrite. The theme is release early, release often, simple, fast, extensible.</div>
<div class="p2">
<br /></div>
<div class="p1">
Easy enough right?</div>
eleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.com2tag:blogger.com,1999:blog-3585841838339962909.post-58340560431779100882013-06-11T23:17:00.000-07:002013-06-13T15:19:39.516-07:00PSM 2013 Days 1-4: Everything you Never Wanted to Know About Plone 5<div class="p1">
<br /></div>
<div class="p2">
My, my, my it's been a while Ploners. I deeply apologize for lack of communication on my part and my only excuse is laziness, impatience, and hubris.</div>
<div class="p3">
<br /></div>
<div class="p1">
How does one sum up the realization that communication in the Plone community has hit rock bottom? One can't. So this will likely be the longest blog post I ever write and hopefully the last one of this length. Anyone who has ever said they get pissed that "we" don't communicate will leave this post with sincere regrets about asking for more info. If you are lucky I will put a TOC on this manifesto, but don't hold your breath. It's technical, it's marketing, and everything in between. Grab a cup of coffee and join me on the 7 day adventure that was Plone Symposium Midwest 2013 (PSM13).</div>
<div class="p1">
<br /></div>
<div class="p1">
Technical note: unlike certain content management systems like Plone, blogger does not have pull quotes. I'll do my best to mimic them for you lovelies, but I'm limited here. </div>
<h3>
<b>Pre Sprint</b></h3>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-HgRctXUy1_o/UbgLtuwt7kI/AAAAAAAAL9c/v2y8Xnj_9tM/s1600/IMAG0056.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="180" src="http://3.bp.blogspot.com/-HgRctXUy1_o/UbgLtuwt7kI/AAAAAAAAL9c/v2y8Xnj_9tM/s320/IMAG0056.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Pre-sprinters at the park</td></tr>
</tbody></table>
<span style="text-align: center;">PSM13 was kicked off by a pre-sprint. 10+ of us rolled into Oshkosh early, with the intention of focusing on the </span><a href="https://dev.plone.org/ticket/13260" style="text-align: center;">portal_skins removal plip</a><span style="text-align: center;">. I arrived a day later than the usual suspects to see that things were already kicked off on redefining the javascript integration story for Plone which generally distracted us from the goal of coding for and drove most of us to the overarching question of... </span><i>Seriously, can we define Plone 5 for real and really mean it?</i><br />
<div class="p3">
<br /></div>
<div class="p1">
Plotting, coffee, arguments, persuasion, a whole lot of momentum, and 2 days later, <a href="https://docs.google.com/document/d/1ctopG52U7ieicCPvavRAshgkAW68uIDa9zXGTGqhmyc">a document</a> was born. That's right folks, its in writing, and its happening. Get ready.</div>
<blockquote class="tr_bq">
<b><span style="font-size: x-small;">Alanis Morisette Interlude</span></b><i><span style="font-size: x-small;">This may seem like a very trivial document folks, but I assure you its one of a kind. Not only does it list what we plan to do, but it even has marketing reasons and measures of completeness. Get your lottery ticket now. Book a flight to loch ness and prepare to capture sasquatch. This could never happen again.</span></i></blockquote>
<div class="p1">
I'm sure you don't trust us, and considering that the first talk of Plone 5 like things started in <i>December of 2007</i>, I totally understand. Why is this time different? First, all of these features are already in progress, and in fact many of them are complete. There was talk of a 4.4 release and that is likely hitting the wayside in exchange for the opportunity to make many overdue, much needed improvements (read:change is coming). </div>
<blockquote class="tr_bq">
<b><span style="font-size: x-small;">This Time, in the Key of A Major</span></b><i><span style="font-size: x-small;">What is a major release? A major release is an opportunity for us as developers to break some backwards compatibility in exchange for overdue large infrastructure upgrades and extra fancy new features. These changes affect highly customized sites the most so if you are running stock Plone, chances are that you are going to be A-OK. In general, we do our best not to break anything for ANY upgrades but some changes can't be avoided. Hence - major release!</span></i></blockquote>
<div class="p1">
Second, we reached out to integrators at the conference, business people, developers, etc to get feedback. What you see documented is by no means final, but rather a general consensus of what generally sounds reasonable to cut a major release. Think something is missing? <a href="http://plone.org/community/processes/plips">Submit a PLIP</a> NOW. Not sure why something is or isn't there? Ask a question in the comments and I will gladly explain. The posted document has open comments and if you want to have access to help us gather collective business reasoning, <a href="mailto:eleddy@plone.org">just ask</a>.</div>
<blockquote class="tr_bq">
<b><span style="font-size: x-small;">Deco vs Collective.Cover or How I learned to Stop Hating and Start WAITING</span></b><i><span style="font-size: x-small;">Oh dear. This was very much debated, but you'll notice these are both omitted from Plone 5. If we were to include either one of them, there is a good chance the release would be delayed even more, and both technologies have some serious technical issues before they can really be integrated. In addition, it was crossing the line of "maybe too much at once". They can still both be used as ad ons and I encourage the continuing development of both.</span></i></blockquote>
<div class="p1">
Third, we realized that the basic changes present a need to prepare add-on developers and integrators for the set of changes that are coming up. We need to get the community on best practices NOW so that the shock of Plone 5 doesn't hit us later. Almost all of the technologies and framework to make your custom integrations and add-ons compatible with Plone 5 are available for use TODAY. However, modern best practices are not documented well, and we realize that the success of Plone 5 relies on people upgrading their Plone 2.5 habits to modern development and integration practices. Documenting these best practices is at the top of all of our lists and we will do what we can to make sure that you aren't shocked. </div>
<blockquote class="tr_bq">
<b><span style="font-size: x-small;">Tough Love Interlude</span></b><i><span style="font-size: x-small;">I know integrators, it's hard to change old habits. But, progress demands it. If you are particularly concerned about a change to the infrastructure and how it will affect your deploys, we will be putting together a FAQ to go with the release. Start asking here or on the PLIPs or on the Plone 5 document. We'll make sure that you're "story" has a translation. We aren't ask you to go back and upgrade all sites, just that you become familiar with modern Plone practices and start off new sites with them. A quick list:</span></i><br />
<ul class="ul1">
<li class="li1" style="text-align: justify;"><i><span style="font-size: x-small;">Theme with Diazo</span></i></li>
</ul>
<ul class="ul1">
<li class="li1" style="text-align: justify;"><i><span style="font-size: x-small;">Use z3c.form for forms (not formlib)</span></i></li>
</ul>
<ul class="ul1">
<li class="li1" style="text-align: justify;"><i><span style="font-size: x-small;">Don't make CPT workflows (not like you wanted to anyways)</span></i></li>
</ul>
<ul class="ul1">
<li class="li1" style="text-align: justify;"><i><span style="font-size: x-small;">Use BrowserViews not portal_skins. Once you start, the fun won't stop. </span></i></li>
</ul>
<ul class="ul1">
<li class="li1" style="text-align: justify;"><i><span style="font-size: x-small;">Stop making custom archetypes and use Dexterity</span></i></li>
</ul>
<ul class="ul1">
<li class="li1" style="text-align: justify;"><i><span style="font-size: x-small;">Think about your reliance on z3c.jbot: in theory that is a task for Diazo these days. UPDATE: People are bitching about this statement. I don't know what to tell you. I prefixed it with "in theory" and a little "Think about" because I have never seen a need for jbot myself. That's right, I have never used it. I'm sure it's great, but my general thought is that it is a crutch for a more real problem that needs to be solved. This is not part of my expertise or something I care to fight about or elaborate on so here is my advice: do whatever you want. This blog post is not here to prevent you from thinking.</span></i></li>
</ul>
</blockquote>
<div class="p1">
Fourth, we have no BDFL or dominant company at the moment to drive development and set a roadmap (and we are excited about it). Many historical leaders and companies have moved on to different places in life. All of the statistics show the community moving to a less centralized development pattern and at a faster rate (more on that research in a later blog post). When you see the data, you'll get tingles I'm sure. This means that we need to coordinate differently and make decisions differently.</div>
<div class="p3">
<br /></div>
<div class="p1">
Because of this, we realize the need to reach out for help to hit this vision so here is your ample notice and opportunity for all of the community to help us reach this goal. We aren't the community that can lock Martin Aspeli in a room and expect Deco to pop out a week later anymore. We have to be a bit more deliberate and communicate much more clearly. Someone said "failing to plan is planning to fail" and I almost spit out my beer. Fuck that shit, I'm planning to kick ass.</div>
<div class="p3">
<br /></div>
<div class="p1">
In particular, we are reaching out to companies to help us support this vision by offering resources to help out a particular PLIP. For example, <a href="http://www.sixfeetup.com/">SixFeetUp</a> is offering QA time for new features like <a href="https://dev.plone.org/ticket/10886">plone.app.events</a> and <a href="http://www.netsight.co.uk/">Netsight</a> is interested in helping get the Plone login story fixed (with one of my clients also having a vested interest). Please check out the story and ask your boss, clients, etc if there is some way that your company, institution, whatever can get behind a change and help is make Plone 5 better. I'll update this post with anyone who pledges to chip in!</div>
<blockquote class="tr_bq">
<b><span style="font-size: x-small;">We Need Marketers, not Developers</span></b><i><span style="font-size: x-small;">One thing that you will hear more of as these blog posts start pouring out of my butt is the notion that we aren't really suffering from lack of developers. We can always use more of course, but what we REALLY need are people with "soft skills". We need people who can start getting ready to market and push, who can make campaigns, and translate developer speak to business leaders and decision makers. Got some of those lying around? <a href="mailto:eleddy@plone.org">Contact me</a> and we'll take advantage of them tomorrow.</span></i></blockquote>
<div class="p1">
In summary, Plone 5 is happening this time because we now understand the importance of communicating with all of you NOW and not after. All of us have promised to blog, talk, and communicate more. In exchange, we are asking for your help. Help as developers, integrators, and most importantly, business people and marketers. Key contacts are listed on the document and hopefully each leader will write a post explaining the change and what it means to you. When in doubt, just ask them.</div>
<div class="p3">
<br /></div>
<div class="p1">
Which is a perfect segue into my <span class="s1"><strike>personal vendetta</strike></span> Plone 5 project: removal of portal_skins.</div>
<h3>
<b>Goodbye CPY</b></h3>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-tafVtGk8gk4/UbgLscN2TxI/AAAAAAAAL9M/t05_yTSVFU8/s1600/IMAG0045.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="320" src="http://4.bp.blogspot.com/-tafVtGk8gk4/UbgLscN2TxI/AAAAAAAAL9M/t05_yTSVFU8/s320/IMAG0045.jpg" width="181" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Wildcards fantastic home brew</td></tr>
</tbody></table>
<div class="p1">
At the Plone Conference in 2012, I started working on the r<a href="https://dev.plone.org/ticket/13260">emoval of cpy/vpy scripts</a>. It's been slow and I confess I haven't dedicated the time I wanted. At this moment I think its for the better because there have been some realizations in the meanwhile on the right way to do this.</div>
<div class="p3">
<br /></div>
<div class="p1">
Initially I was focused on only upgrading cpy/vpy scripts but have broadened the scope to idealism in the way that only a masturbatory developer could: Down with portal_skins!</div>
<div class="p3">
<br /></div>
<div class="p1">
Now before you freak out, let me say that YOU, integrator, developer, all around good guy can still upgrade your plone product which uses things in portal_skins. However, my vision has a portal_skins free Products.CMFPlone. Everything that is there will be moved into plone_deprecated and you can let acquisition be all magical and do weird stuff still. In addition, your custom skins and what not will still work (there is no guarantee that this will still be the case in Plone NextBigRelease though). However, <b><i>a major goal of Plone 5 is to return core to represent the current best practice of developing in Plone</i></b>. </div>
<div class="p3">
<br /></div>
<div class="p1">
And now the most important question: WHY? portal_skins are part of this notion of acquisition. Security and performance reasons have show us that maybe this isn't the best way forward. In addition, modernizing the underlying framework relies on us moving beyond this notion. But wait, there's more. Most of that stuff existed before test cases were culturally enforced. The code is untested and intricate, so people are terrified to touch it. We can't fix bugs on files that everyone is afraid to modify. It get's better. Customization of page templates in portal_skins is shotty at best. It doesn't survive upgrades. Moving all forms to z3c.form gets us on one form technology that can be plugged and extended, and should survive future upgrades much better. Last but not least, times have changed. Resources like js and css shouldn't be managed in the registry. Anything bigger than a small site should be compressing with modern tools and hosting in a web server and/or a CDN. We need to recognize that the <i>industry</i> standards have changed and we are behind. Time to play a little catch up.</div>
<div class="p3">
<br /></div>
<div class="p1">
Moving all of portal_skins is a much larger task and this meant changing the approach to the change. At first it was a branch on Products.CMFPlone but I had a couple of realizations that led me to believe that the majority of changes should actually be done in a new package, which will then be merged back into core at the end. I'll bullet point the reasons since I'm running out of transition words:</div>
<ol class="ol1">
<li class="li1">Constantly merging back in changes in every package made me want to throw my laptop out of the window. Git has been merging like a dickhead lately and I had several errors from that waste a lot of time. </li>
<li class="li1">I want these changes for my own projects. Yesterday. I am willing and have the capacity to put them in live sites. Mostly performance reasons, but also integration pain points. I suspect other people do as well.</li>
<li class="li1">We can document and keep track of the effort in a more sane matter. When merging in commits I was losing track of the W5H of what happened because this is such a long ranging project.</li>
<li class="li1">Integrators and add on developers can pull the changes at any time and see if their shizzle works.</li>
<li class="li1">We have a lot of test cases to write. I'm emotionally tied to this thought that we can start fresh and outside of core will just seem more approachable.</li>
</ol>
<div class="p1">
All changes will go into plone.future.* packages that will be grouped by general integration points.</div>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-wcbAAgziT5c/UbgLtiavaTI/AAAAAAAAL9k/tbC_QoFmYzU/s1600/IMAG0057.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="200" src="http://3.bp.blogspot.com/-wcbAAgziT5c/UbgLtiavaTI/AAAAAAAAL9k/tbC_QoFmYzU/s200/IMAG0057.jpg" width="113" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Grace Lim, the Plone Ranger</td></tr>
</tbody></table>
People will be able to grab them at any point. At the moment I see these as:<br />
<ul class="ul1">
<li class="li1">plone.future.formscripts: All of our core actions like a add, move, delete, etc.. are on code that hasn't been touched in YEARS. These are things that are highly unlikely to be uncustomized. This also includes control panel forms and their conversion to z3c forms. I can't imagine that most people can't start using this as soon as a release is cut. At first it will be just a few scripts and I will cut releases after each new thing is added. Ideally we can get these out in the wild, slowly, before the big bang.</li>
<li class="li1">plone.future.login: Oh god, login is worth a whole blogpost at least this length. It can't be approached in the same set of changes with any sort of sanity. </li>
<li class="li1">plone.future.resources: Plone 5 will have a new diazo theme so this won't affect you. Old themes will still work, but if you have properly based your site on Sunburst, then you have the OPTION to use this package, which will contain 1 properly minified js file, 1 properly minified css, and various icons moved out of portal_skins and into resource registries. I know I have several sites that likely won't redesign for a while and will benefit immensely from this. </li>
</ul>
<div class="p1">
In order to smooth this transition and to allow you as an add on developer and/or integrator there are a few other things that I intend to include:</div>
<ul class="ul1">
<li class="li1">A script to scan a directory of code that issues warnings for usages of things that have been deprecated.</li>
<li class="li1">A script to scan a directory of code that issue warnings for all plone.future references once the packages have all been merged back into core.</li>
<li class="li1">An intense upgrade guide that matches said warnings and indicates old school vs new school ways of doing things.</li>
<li class="li1">Tests, tests, tests!</li>
</ul>
<div class="p1">
Ideally each one of these items is updated AS the feature is removed to limit the last minute documentation work. I'm in the process of using cut, copy, paste, delete, and rename as an example and setting up this framework, after which I'll start cherry picking work that has already been done and calling for more help. The hard work and dedication that went into plone.api has been inspiring and its my hope to walk away from what <i>could </i>be a tedious task with an immense amount of <b>pride</b>. This is a chance to fix a whole helluva lot of things.</div>
<div class="p3">
<br /></div>
<div class="p1">
Overall, a pre-sprint was the best thing to happen to a symposium/conference since reliable wifi. </div>
<h3>
<b>Fundraising</b></h3>
<div class="p1">
When you thought this blog post couldn't get any longer, it did. It will continue to grow until you all REGRET knowing what's going on. Should you have made it to this point, obviously you haven't learned. </div>
<div class="p3">
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-yUz4aUrAg-M/UbgLt4z1GmI/AAAAAAAAL9s/pOyOu1mkIb4/s1600/IMAG0058.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="181" src="http://4.bp.blogspot.com/-yUz4aUrAg-M/UbgLt4z1GmI/AAAAAAAAL9s/pOyOu1mkIb4/s320/IMAG0058.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">It's like the city of Oshkosh just knew...</td></tr>
</tbody></table>
<div class="p1">
Being on the Plone Foundation board has been fruitful this year, and I'm super ecstatic about the push to spend money on sprints and kickstart this Plone 5 process. I was so excited in fact, that I <a href="http://www.slideshare.net/eleddy/uwosh">gave a talk</a> about it. With that victory has come a little burden to bear, and that burden is a bear by the name of "fundraising".</div>
<div class="p3">
<br /></div>
<div class="p1">
In order to dump money into sprints this year, I promised to raise the money we dumped for next year, so we can do the same thing. Turns out that I am a TERRIBLE fundraiser with a serious time deficit and intense lack of skills. Enter <a href="https://twitter.com/jasontlantz">Jason Lantz</a>, who gave a talk on how he built a <a href="https://github.com/innocenceproject/collective.salesforce.fundraising">fundraising platform</a> on Plone for <a href="https://secure.innocenceproject.org/donate">the innocence project</a>. How disgustingly appropriate! After spending a full day hashing out ideas with him I realized how little I knew about raising money for non-profits and how much beer I will owe him the next time we meet up.</div>
<div class="p3">
<br /></div>
<div class="p1">
He has been a community lurker for over 10 years and is very excited to help us, the good people of Plone, get our fundraising shit together. His work is open and he is working on repackaging the add on for use by everyone (if you are a non-profit and you are interested, head over to <a href="https://github.com/innocenceproject/collective.salesforce.fundraising">github</a> to see the progress). I am excited to work with Jason and in addition a whole new team of people excited to kick off fundraising efforts. Expect to hear more from us on that later and remember: you don't have to wait for a fancy donation website to support the foundation. Go to <a href="http://plone.org/foundation/foundation-donations">plone.org</a> to help out now. The money that you donate is going to things like bringing K.K. Dhanesh over to Oshkosh from India to sprint for the first time!!</div>
<h3>
<b>Communications</b></h3>
<div class="p1">
I've hinted at the severe lack of communication that goes on in Plone and we have lots of people (including myself) that have vowed to make this a little easier on people. This includes but is not limited too:</div>
<ul class="ul1">
<li class="li1">Eric Steele, Rok Garbas, David Glick, and myself vowing to blog more. Twitter has killed the blog and we need to go back to putting complete thoughts and ideas on paper for those that can't explain MultiAdapters in under 140 characters. I encourage all developers to get back to the paragraph.</li>
<li class="li1">Eric Steele will be trying to get together a regular call of team leaders, which will be attended by Rose Pryune who will act as a translator and post back to the community. </li>
<li class="li1">A revamp of the marketing team to be led by Eric Rozeboom. There is a lot to be done here and I'm sure he will post a wonderful plan soon and start recruiting. I know in particular that <i>I</i> am looking for someone to help us better utilize our adwords account. Know someone???</li>
<li class="li1">Heidi Reinke of UW Oshkosh College of Business has volunteered to have her marketing class for fall semester take on Plone as a use case for marketing. I can't wait to see what she comes up with and if you get contacted out of the blue by people doing research - don't be shocked! </li>
</ul>
<div class="p1">
And look guys, its in writing. No backing out now!</div>
<div class="p3">
<br /></div>
<div class="p1">
Lots of talk about communication led to some solid realizations on all sides. Developers don't like to communicate in lengthy posts like this and a lot of "translation" gets lost because of it. Forcing people to write docs is, however, not the solution. The oral interview is MUCH more effective and we have to spread that word. Want to know what's going on in Plone? Don't wait for a blog post. Ping a core member, ask them on a call, and write up a post for everyone else to consume. </div>
<h3>
<b>That Stinking Contributor Agreement</b></h3>
<div class="p1">
In addition to being a Plone Foundation Sponsor, hosting a kick ass pre-sprint barbecue complete with home brew, and maintaining several high quality Plone plugins, Talin Senner of <a href="https://www.wildcardcorp.com/">Wildcard Corp</a> has volunteered to help us finally kick this long standing issue with the Plone Contributor Agreement. I am eternally grateful for them picking this up and rolling with it since it was making my head spin and I was dangerously spread thin.</div>
<h3>
<b>Mailing Lists</b></h3>
<div class="p1">
We REALLY need someone to put a nice front end on mailing lists. The lovely Heidi made a statement along the lines of "why would i use plone? signing up for the mailing list makes this software look like its from 1990". OUCH. Interested? Ping me. We just need a plone.org initial UI into the basic mailing lists. </div>
<div class="p3">
<br /></div>
<div class="p1">
In addition, we should think about making plone-accounce a mail chimp list, considering that:</div>
<br />
<ol class="ol1">
<li class="li1">The UI doesn't suck at all</li>
<li class="li1">It's for communicating out, not within</li>
</ol>
<h3>
I'm Running out of Steam</h3>
<div>
I didn't even get to talk about Grace Lims video project for coders and their kids, the Plone Ranger, the packages that were released and the intense code sessions, the work on EVERYTHING from coredev docks to mock, realization that we need help with adwords... oh god. That's another 6 hour plane ride of info. For now, sleep.<br />
<br />
<b>More sprint reports at</b>:<br />
<ul>
<li><a href="http://www.uwosh.edu/ploneprojects/plone-activities-blog/">http://www.uwosh.edu/ploneprojects/plone-activities-blog/</a></li>
</ul>
<div>
<b>Followup Awesomeness</b>:</div>
</div>
<div>
<ul>
<li><a href="http://pigeonflight.blogspot.com/2013/06/modern-plone-best-practices-or-how-to.html">http://pigeonflight.blogspot.com/2013/06/modern-plone-best-practices-or-how-to.html</a></li>
<li><a href="http://pigeonflight.blogspot.com/2013/06/modern-plone-practices-poster-now.html">http://pigeonflight.blogspot.com/2013/06/modern-plone-practices-poster-now.html</a></li>
</ul>
</div>
eleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.com4tag:blogger.com,1999:blog-3585841838339962909.post-6756731141065854412012-08-25T17:45:00.000-07:002012-08-25T17:45:12.207-07:00Plone South America Sprint Report: Day 1!Holy Moly! Plone Symposium South America has been amazing, and the strong local community really shined through today at the sprints. I usually don't do these report things but I'm just so golly proud I'm going to gush about all that got done today at the sprint that has no name!<br />
<br />
Fun Facts:<br />
* 18+ new contributor agreements already<br />
* 6 people made their first commit to Plone <b>core</b> today already. Many of these people have been plone users/integrators for 5+ years so I'm really excited to get their input into core going forward<br />
* Plone has an amazing community in South America that takes names and kicks ass<br />
* Only 2 machines had buildout issues and both were resolved in under 10 minutes. That has to be some kind of a record.<br />
<br />
<a href="http://ploneapi.readthedocs.org/en/latest/index.html">plone.api</a> gathered even more momentum. A sampling of what got done:<br />
<br />
* closing open tickets and adding several new methods<br />
* update error reporting so that errors are useful<br />
* support for localized documentation in sphinx<br />
* tests for dexterity support<br />
* code for granting roles locally and globally<br />
<br />
<br />
<a href="https://github.com/xiru/plone.api.json">plone.api.json</a> was born and core implementation completed. Testing, docs, and hopefully a demo will be worked on tomorrow.<br />
<br />
The <a href="http://tutorialtodoapp.readthedocs.org/en/latest/">Plone TODO tutorial</a> is almost done. We did a coding dojo with 6 people, 3 of which were php developers, to test the documentation and find holes. It could use a couple more passes and some refining but please check it out and report bugs there! It is fully tested, and will be guaranteed to work with new releases of Plone.<br />
<br />
Internationalization of <a href="http://buildoutcoredev.readthedocs.org/en/latest/index.html">buildout.coredev</a> docs into <a href="https://github.com/plone/buildout.coredev/tree/4.2/docs/es">Spanish</a> and <a href="https://github.com/plone/buildout.coredev/tree/4.2/docs/pt_BR">Portugese</a> has been started and a LOT of progress is made. It's become very clear to me personally that getting started in Plone has been hindered for a long time by assuming proficient knowledge of English. If anyone is looking to translate key docs for the coredev buildout and plone.api into a local language, please join us in #sprint tomorrow or just holler a bit and I'll find you.<br />
<br />
<a href="https://github.com/collective/collective.ploneide">Plone IDE</a> debugger has been fixed and now working on integrating rope.<br />
<br />
In addition we got several <a href="https://dev.plone.org/ticket/13010">major bug fixes</a> from for plone.app.collections, excellent support from Interlegis for wifi and a place to crash, and super yummy pao de queijo.<br />
<br />
Who knows what will come of tomorrow. Having a boring weekend? Join us in #sprint!eleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.com1tag:blogger.com,1999:blog-3585841838339962909.post-28891559504490957542012-05-07T12:16:00.000-07:002012-05-09T13:59:45.445-07:00Plone Contributor Agreements<i>This is an open letter about my frustrations with the Plone contributor agreement process. I have tried many times to discuss this in a private setting but I feel no movement and have resorted to stupid antics like writing open letters as blog posts. This is my last effort to push this forward.</i><br />
<i><br /></i><br />
Dear Plone Foundation Board -<br />
<br />
The current state of the Plone contributor agreement process is abysmal. It is careless, out of date, and undocumented at best. Don't worry, I am not talking about e-signing. We all gave up on that. I am talking about simple things that make me want to go postal. If the developers in #plone can get it together and be nice to newbies, you can do the same.<br />
<br />
The steps to make it sane are trivial:<br />
<ol>
<li>Update the <a href="http://plone.org/foundation/contributors-agreement/agreement.pdf/view">agreement</a>. Get rid of the stuff that no one fills in and the fields that you will never follow up on (e.g. "<i>Please write your physical mailing address here, so we can mail a physical copy back to you.</i>" Ahem...). Every time some one fills this out at a sprint, they ask what "Program" is and I always respond with "Pretty sure it's Plone... I think...". I also have to ask them to add their github user name and put it somewhere on the bottom (or I forget then have to contact them later). Other gems in this legal document include: "<i>If
you have any questions, or would like something to be changed, ask [Plone Foundation Contact for Assignments
Email Address] via email.</i>" In its current state, the agreement is a joke. Please spend 5 minutes and update it.</li>
<li>Update the documentation for new contributors. From our communications, I understand that scanned agreements can be accepted. Maybe you should write that somewhere since everything now points to snail mail (used to be a joke, now just sad). You could also have an email address set up so that it doesn't take 3 emails to find out who "gets the agreements now". You could set up one email, such as contribute@plone.org, that redirects to whomever is in charge of the agreements at the time being. They could even answer questions and you could update <a href="http://plone.org/foundation/contributors-agreement/agreement.pdf/view">certain documents</a> to have contact information once and never have to update it again. Related: internationally mailing agreements is cruel and unusual punishment. </li>
<li>Update the documentation for superusers. I have run several sprints and have superuser access to everything needed to make things work. I try to do the right thing but I constantly feel at risk. Let me count the ways:</li>
<ol>
<li>When triaging tickets with patches, I have no idea which people have signed an agreement and we can accept the patch and who has not. This is not fair to me OR the poor ticket reporters that I pester asking if they have signed or not. </li>
<li>During virtual sprints, such as tuneups, I often spend time recruiting people to send in their agreements. Does this count as a normal sprint? Can I give them perms once I see the agreement? No one knows the rules. Write them down, please. </li>
<li>Similarly, there is no list of who has and who hasn't signed the agreement. It would make my life so much easier if I could just validate that. People forget if they signed it in the past and then its a whole debacle. Keep in mind that as a sprint host, I am the one who has to answer questions about whether or not agreements are received or what the status is and can they have permissions yet. </li>
</ol>
<li>Be responsive. When people sign agreements, especially mail it in, its because they have something they want to commit. A one month turnaround time is horrid. Motivation is a fleeting thing. Furthermore, please don't make people open a ticket for core access in trac. It's annoying. If you get their github username upfront in a <a href="http://plone.org/foundation/contributors-agreement/agreement.pdf/view">certain document</a>, you can just add them. DONE.</li>
<li><b>Bonus Round</b>: Follow up. How about a nice welcome email? Maybe a link to <a href="http://readthedocs.org/docs/buildoutcoredev/en/latest/">instructions on getting the buildout going</a> and how to fix tickets? This can be very easily done with a mailing list which could then be used to send updates like "hey contributors, we moved from svn a year ago" instead of people opening tickets in trac. </li>
</ol>
<div>
In summary, I'm asking you to give a shit about this poop stain of a process. It is a frustrating, out of date introduction to the Plone developer community and we deserve better.</div>
<div>
<br /></div>
<div>
Liz</div>
<div>
<br /></div>
<div>
PS. Have word processor, will edit.<br />
<br />
<b>UPDATE #1</b>: The email exists! It's assignments@plone.org. If your agreement is in limbo, email them today and ask whats up.<br />
<br />
Also, apparently we technically don't accept scanned documents yet. I hear conflicting things so at the moment I guess just... no.<br />
<br />
<b>UPDATE #2</b>: There is a list of approved contributors at <span class="s1"><a href="https://plone.org/team/Committers">https://plone.org/team/Committers</a></span><span class="s2">. Quick, everyone click it and watch plone.org crawl! It's a half solution since it doesn't actually list the trac username for quick search but at least it's something.</span><br />
</div>eleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.com6tag:blogger.com,1999:blog-3585841838339962909.post-36114180579352749542011-08-24T12:39:00.000-07:002011-08-24T12:41:24.171-07:00Talking about Talks<div>When we started planning the conference I intended to blog about EVERYTHING as a form of documentation. Whoops. No time like the present to catch up!</div><div><br />
</div>Without a doubt, setting up talks has been the most time consuming process in the conference planning so far. I want to shed some light on what I did this year, why it took so long and why it was so painful. This will give me some much needed catharsis and I would also be interested in hearing your feedback in the comments or elsewhere. The goal is to provide a solid, pain free template for next years organizers and for anyone else planning a conference in the future.<br />
<div><br />
</div><div><span class="Apple-style-span" style="font-size: x-large;">The Plan</span></div><div>The plan was simple enough: setup a Google moderator account and get feedback on what people want to see. Everyone who is planning on submitting will vet or obtain ideas there. Then, have people submit the talk to the ploneconf.org website with their bio and their talk (which will be magically normalized of course). The netsight product will take care of voting, scheduling, and display. Clean hands, everything automated, go to the pub.</div><div><br />
</div><div><span class="Apple-style-span" style="font-size: x-large;">The Reality</span></div><div>Google moderator worked "just alright". The main issue was that most people thought this was the talk submission process, and were pimping for votes SXSW style. This made it super confusing when we actually asked for talks to be submitted. Nonetheless, feedback was gathered and there were a lot of interesting ideas thrown around. </div><div><br />
</div><div><span class="Apple-style-span" style="font-size: large;">Talk Submissions</span></div><div>Our "plan" did not include validating what was done last year code-wise ahead of time. We didn't realize that no permissions were set up for an open site, and everything was completely customized for Bristol. We never got logins integrated like we wanted either. Whoops. Deadline and pressure was mounting so we threw up a Google Form. Enter stage left: the vile mistress Regret.</div><div><br />
</div><div>Immediately we realized that people couldn't edit their submissions after clicking submit and started emailing us changes instead. Also, people had to triplicate their bios and many of them said things like "see other bio". This was not really fair to the submitters or us. At least the form looked nice, was embed-able, and I could add fields as people reminded me that I was missing important things like email address and first time speaker status (thanks to all those who took the time to send feedback - this is all now documented for next year). Totally not worth it though.</div><div><br />
</div><div><span class="Apple-style-span" style="font-size: large;">Deadlines</span></div><div>We decided up front there would be no extensions so that lit a fire under us to make sure people submitted (I personally get annoyed by our "deadline extended!" community habit). The post for talks went up and they start coming in quickly. At 3.5 days and 2 rooms, we were looking at around 65 talk slots to fill (not including related technologies track which was organized a bit differently). There was a big bump in submissions at first but then things started to trickle down to the point where we were nervous wouldn't reach our target number of talks. </div><div><br />
</div><div>So we set up a raffle for a free conference ticket and doubled newbie chances to win. This went REALLY well, and encouraged people to submit multiple talks. Later, this gave me the opportunity to pick the strongest talks for a particular person and not have to turn down a great speaker that just picked a bad topic. This gave us plenty of great talks to chose from.</div><div><br />
</div><div>The other thing that happened is that people didn't believe the deadline was real and/or missed it. Some people submitted by email. Others by Google docs. Some talks were "implied". This process went on far too long. To give you an idea what I was dealing with, here are talks to format stats:</div><div><ul><li>Google Forms: 72 (4 were submitted late)</li>
<li>Google Docs: 17</li>
<li>Email: 4 (2 never responded with bios)</li>
<li>IRC: 1</li>
<li>Chat: 1</li>
</ul><div>If I messed up your talk, I deeply apologize but hopefully you understand why things slipped through the cracks. This goes back to the earlier point on not being able to edit talks. Lesson learned. </div></div><div><br />
</div><div><span class="Apple-style-span" style="font-size: large;">Voting</span></div><div>Once all the talks were in we had to import them into the conference website for voting (for those keeping count, this is the 3rd e-format for talks). It took a good 2/3 of a day to set up the styles, update the CSV importer and Dexterity types, and deploy. It was easy drudgery. At some point a unicode problem was introduced we couldn't have any authors or titles with non-ascii letters. By that point I said "f*ck it", ran the import, and asked for help in manually cleaning up the bad encoding. Many thanks to bdbaddog on that one. This could all be avoided in the future.</div><div><br />
</div><div>Voting again was somewhat confusing since "we already voted". The controls aren't super intuitive but luckily they were the same as last year and didn't require authentication. All the reports were already there that I needed and people seemed to figure it out just fine. I will mention that there wasn't a single surprise result in there. Everything lined up just nearly exactly as I thought it would. It did give me numbers to work with for scheduling, which was a huge benefit (technique will be described below).</div><div><br />
</div><div><span class="Apple-style-span" style="font-size: large;">Scheduling</span></div><div>After all the votes had been cast, I stared at the computer for at least 2 hours thinking about where to go next. No matter what, code had to be updated (last years schedule was a hard coded html template) and it would be nice to make something reusable but I wasn't sure if it was possible in a timely manner. I looked at a couple calendar integration tools but none of them played well with the Dexterity type we were using. In the end I resorted to <a href="http://twitpic.com/6a5qm7">notecards and giant blocks of paper.</a></div><div><br />
</div><div>Process pseudocode (maybe someone can code this up for next year): </div><div><ol><li>Transfer talk title, speaker, and vote count to postcards (format #4). If the talk was 45 minutes, leave it fill size. If it was 30 minutes, cut it in half. Talks go on 1 of 4 colored postcards: All, Beginner, Intermediate, or Advanced. This refers to level of development experience needed to see the talk. This was the easiest way for me to think of preventing talk conflicts. Note: I read what the speaker claimed their audience was but 9 times out of 10 there was a discrepancy from what was claimed and what the extended talk summary said. In those cases, I went with my gut.</li>
<li>Sort each pile by number of votes. For each pile with too many talks, move the worst rated <b>n</b> talks to a separate pile. They aren't excluded, just removed from consideration until all the most popular talks are handled. Try to make all the piles an equalish size. Set aside 1/3 of the worse rated talks for multiple submitters for the same reason in a different pile. Talks that are almost exact duplicates get the same treatment. Side note: the person who submitted the most talks was Carlos de la Guradia at a whopping 6 talks!</li>
<li>Draw a big calendar. I used some GIANT post it notes. Lay out the tracks and physically write in things that can't be changed such as keynotes, lightening talks, foundation meetings. Set up a plan ahead of time. I decided I would put "All" and "Beginner" talks in track 1, "Intermediate" and "Advanced" talks in track 2, and "Related Technology" talks in track 3. For each track I alternated between all and beginning, intermediate and advanced as much as possible. I pitted "All" talks against "Intermediate", and "Beginner" against "Advanced" as much as possible in attempt to avoid beginner/intermediate talk clashes. </li>
<li>Work in breaks and lunches. I went for a break or lunch about once every 2 hours.</li>
<li>After getting all the super high rated talks mapped out and un-conflicted, pull in the reserve stacks and added them in by popularity and amount of space left in the track. Flex time talks really came in handy. The size of the cards fit perfectly for my mocked up calendar.</li>
</ol></div><div>With this strategy, the hardest scheduling was the related technology track, since there would inevitably be clashes between the more advanced talks and those. When I had a conundrum, there was almost always a similar or duplicate talk in my backpocket that I just threw into the schedule at a different time. It took me a moment to get over the fact that there is no harm in having almost the same talk twice if they are both highly rated and well respected speakers: a conference schedule is not an ER diagram!</div><div><br />
</div><div>Last but not the very least was putting these talks in electronic form. I gave up on using the conference talks add-on as soon as I found SCHED.org. After an hour of beer and wallowing over the fact that I should have used that from the beginning, I spent the next 8 hours adding speakers, talks, keywords, etc to the new site (format #5). There is an API but all of the talks needed to be keyworded, slotted, and mapped to users so I sucked it up and just hammered it in. So much for automated.</div><div><br />
</div><div>And the 2011 <a href="http://2011ploneconference.sched.org/">Plone Conference Schedule</a> was born.</div><div><br />
</div><div><span class="Apple-style-span" style="font-size: x-large;">Hindsight</span></div><div>No one is paying me hourly to make decisions for this conference so I have a tendency to revert to geeky habits and "make" or "wrangle" things instead of searching the web for tools and starting there. At first I was fitting conference stuff in at the end of the day and making poor choices. Now I tend to block of entire "conference days" to help reduce the decision fatigue factor. It helps a lot.</div><div><br />
</div><div>There is also a natural tendency with the conference to just get things done so people "stop bugging me about it" or because "if we don't do this soon we are screwed". I can't speak for past organizers but I have a feeling it's one of the core reasons why there is no reusable conference framework despite all the years of the conference being around. Day job money always trumps conference "<painful_joke>profit" [<-- hilarious!]</painful_joke>. It really takes a solid effort and motivation to do the right thing for the future. Just writing this post helps keep me focused and remembering to reuse where possible, be it our own code or someone else's.</div><div><br />
</div><div><span class="Apple-style-span" style="font-size: x-large;">Suggestions for Next Year</span></div><div><ul><li>Get feedback on whether or not moderator actually influenced talks this year before going that route again. I have a sneaking suspicion it didn't make that much difference. Instead, use feedback from THIS years conference to purposely direct the speakers for next year. You could also encourage people to submit multiple talks from the beginning and weed out the weakest ones later.</li>
<li>Beware of the seductive siren that is Google Forms. </li>
<li>Do the raffle for a free speaker ticket again. In fact, give away several if you can manage.</li>
<li>As a potential conference speaker, please please please use whatever form was submitted and submit your talk on time. I highly encourage next years organizers to set and enforce rules on talk submissions. "Go here and do it by X or it's out". The favors and special considerations will kill you. They killed me, and I didn't even honor all of them I think. I don't even know if I did or not!</li>
<li>Use sched.org or something like it from the get go. Plone is not good for this. SCHED prints out nicely and can be print out and cut up for manual organizing. It is mobile ready. It takes talk submissions and has a voting process. It is everything. I paid a $99 fee to get an add free version. Totally worth it. If someone steps up to sponsor the mobile version, we will be even better off.</li>
<li>There are still some things missing with SCHED. Do your research before jumping on it again. I have already started bothering them for key missing features so maybe next year they will have their shizzle together more. In your research, don't forget things like: voting, editing your own profile and talk, multiple tracks and venues, mobile support. Bonus if you find ajaxy drag and drop and auto buffers for talks.</li>
<li>As far as scheduling, the alternate talk sizes was a slight bit more overhead but in the end it actually gave me a bit of play room and the ability to fill empty slots. I will be interested to see how that goes.</li>
<li>We set up a ton of mailing lists for speakers, sponsors, etc with mailchimp. At the end of the conference, don't let 2011 organizers forget to get feedback for 2012. We tried a lot of new things like variable talks, staggered schedules, related tech - all of them need to be evaluated and reconsidered for next year.</li>
<li>If you have a bunch of people working dev, make sure someone is NOT developing and can be a manager. We know this on contracts: don't forget to apply the same philosophy here. (If I say it enough, I won't let it happen again). The perfect example is when we started "integrating" Brown Paper Tickets into the site. Put a bunch of people super geeked to give back into one room and all the sudden you've integrated something that doesn't need integrated to begin with (thanks to Jon Stahl for bringing some reason to that situation before it got out of hand). I should not have been even trying to dev anything in this scenario - role mixing is like vodka and redbull: it feels awesome at the time but the aftermath often involves vomit. </li>
<li>While Brown Paper Tickets is local and a bit cheaper, Event Brite is integrated into just about everything already. Mailing list tools, scheduling tools, etc all support importing users directly with the API. Something to consider.</li>
</ul><div><span class="Apple-style-span" style="font-size: x-large;">FAQ</span></div><div><ul><li><b>Shouldn't "The State of Plone" talk be a keynote? </b>We have a unique situation with space this year in that none of the rooms can <i>technically</i> hold more than 300 people (fire marshall Liz says they can). For the main keynote we rented out a space in the movie theater that is attached to SFSU. It will be awesome: you can drag your hungover butt in, eat popcorn and nachos, and get your mind melted. But it's $1000/hour. Yeah. Time to get in the movie theater business. </li>
<li><b>Does that mean multiple tracks for Lightening Tracks and popular talks?</b> No. We will be broadcasting live to overflow rooms in these cases. The crowd tends to thin out for these so hopefully we won't need them. We will use the preliminary results from scheduling on sched.org to decide which talks get overflow rooms in general. In those cases, we ask that people who are not actually paying attention use the overflow. You know who you are Mr/s. Email McYoutube.</li>
<li><b>My bio is weird - I didn't write one - what happened?</b> If you didn't write your bio, there is a good chance that I did (under the influence of beer and exhaustion of course). Feel free to log in and change it. </li>
<li><b>Can you fix talk X clashing with talk Y and move talk Z? </b>Sure. If you give me the full swap formula and it passes through my "we can't do that because XYZ drama" filter at talks@ploneconf.org I'll seriously consider it. <chuckle> Also, please note that these will all be recorded so you can go back and watch something at any point in the future.</chuckle></li>
<li><b>Will this schedule be changing?</b> Likely. The general timeline is not going to change and key events will not move. Some talks may get shuffled here and there though and I fully expect some last minute additions/subtractions.</li>
<li><b>Wait, did you say reusing from last year? </b>Yep. All the work we do on the site is open and started with an injection from the 2010 Plone Conference site code by Netsight. I encourage anyone who wants to change something to contribute at https://github.com/Plone-Conference-Devs. Little by little we can make a difference for the future. Hopefully in 5 years, everything will be easy peasy. Well, the tech part anyways. The dealing with people part needs its own documentation.</li>
</ul></div><div>Many thanks to all the feedback, advice, and help along the way. Feel free to comment below on changes, suggestions, etc. </div></div>eleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.com4tag:blogger.com,1999:blog-3585841838339962909.post-73406716383878002462011-08-08T20:15:00.000-07:002011-08-09T10:36:17.349-07:0010 Minute Caching with ApacheThe problem with caching is that it encompasses 3 layers of the stack, and requires a good solid read to understand what's really happening. You should enable browser caching on every site with more than 10 users, and proxy caching for resources pretty soon thereafter. This is my attempt to distill this down into a method that will get you "far enough".<br />
<br />
<b>Goal</b>: cache everything, except that which is content in the browser <i>and</i> in a disk cache.<br />
<ol><li><b>Install and activate plone.app.caching (aka HTTP Caching)</b>. This is included by default after Plone 4.1. Use it. On the first tab: enable gzip and enable caching. Leave caching proxies and in memory cache at the default. On the third tab: Content files and images get moderate, content folder and content items get weak, and file and image and stable file and image get strong. You may want unstable file and image to be moderate but... I doubt it. When in doubt, use less caching. Last but not least, in the last tab, weak caching is configured to use last modified (that's it), moderate caching is default with just max age set, strong caching is same as moderate but up the expiration time to 864,000, and also check store in RAM cache. This number means those resources will be 10 days to be in the proxy/browser cache, and you can probably up that significantly as your caching chops get sharper. You may be tempted to put <i>everything</i> in ram cache at this point in time, but don't. That's for people who like to debug caches.</li>
<li><b>Install Apache 2 and Configure Your Site</b>. You only need to do this if this is a new site and/or it's not already configured. Many linux installs actually have this by default. Some good ubuntu instructions are <a href="https://help.ubuntu.com/8.04/serverguide/C/httpd.html">here</a>. You probably want to enable rewrite and proxy as well, in which case you will need to a2enmod rewrite, proxy, proxy_http as well if you want to do rewriting, but this post isn't about that :)</li>
<li><b>Configure Apache as a disk cache</b>. Basic instructions are <a href="http://pashabd.blogspot.com/2011/06/apache-modcache-ubuntu-10.html">here</a>. You will need to add one key line: "CacheIgnoreNoLastMod On". This is because the js and css resources don't have a modification time by default and we still want them cached on disk. In fact, these are the resources that we care the MOST about being cached. Make sure to set up the cache cleaning process. Check out the <a href="https://gist.github.com/1133260">example site config</a> to see how simple it can be. Note that this is just a balls to the wall quick write up. All apache is doing here is looking at your caching headers from p.a.caching and doing the same thing that the browser would. When configured properly, after the first fetch, Plone will never render any css, js, or image files again.</li>
<li><b>Validate</b>. Use LiveHeaders in Firefox or look at the network diagrams in Chrome/Safari. JS, CSS, and [non-content] images should never 304. They should download on the first pass and then always pull from cache. Pages, folders, or any other content should always 200. If you see any 404s in the meanwhile, stop immediately and fix them. Clear the cache. Validate that all new page requests come from the cache.</li>
</ol><div><span class="Apple-style-span" style="font-size: large;">FAQ</span></div><div><ol><li><b>How do I know its hitting the cache? </b>I'm so glad I pretended like you asked. There is no built in way to do this but with some clever jiggering it's easy. From the command line, tail the Z2.log(s) for each running zope instance. (Pst: you can tail multiple logs at once by just listing them with a space in between). Then, open up the browser. Load a page. Notice that everything is pulled the first time. Clear the cache. Load the page again. Note that all the resources have 200 OK status (meaning they were actually retrieved from the servers) but there is no entry in your Z2.log. It's a festivus miracle! This means that they were served from apache.</li>
<li><b>What about HAProxy? </b>That's nice, and I totally recommend it. But not required. </li>
<li><b>Should I use an apache in buildout? </b>No. Unless you want to hate yourself or you are an expert sysadmin [that is trying to get fired]. Tutorials don't care about buildout and you want to have access to tutorials.</li>
<li><b>Shouldn't I be using varnish too?</b> Perhaps. I personally have never liked it, and there is no one config out of the box that makes me happy. All of my sites are closed, pretty dynamic and include 3rd party integration so caching individual pages never suits me. Remember, this is a 10 minute way to get basic results. Varnish is really overkill for a lot of sites and will just give you more headache than its worth. </li>
<li><b>Purging?</b> That's for another tutorial. </li>
<li><b>What exactly is cached here and where?</b> This does NOT cache and views or pages. That's the point. It's simple enough for debugging and small setups while still redirecting "crap traffic" to another server. It caches just enough and no more. I'm pretty sure there is a your mom joke in here somewhere...</li>
<li><b>Isn't apache for old people?</b> Beat it punk.</li>
<li><b>Did you test these instructions?</b> No, they were written in anger and rush. Comment if I messed up and I'll help you then update this. If there is significant interest I can do a more detailed post. This assumes a minimum sysadmin knowledge.</li>
<li><b>What if I want to force fresh css/js?</b> Easy. Just reinstall your product <i>OR</i> turn development mode on then off for js and/or css. This causes a new file name to be generated and the caching cycle to start over again. For static images, you may be in a harder situation. I have just relegated to create a new name for the images each time. I find that these rarely change since these images are almost always going to be from design. If you can wait the 10 days (or whatever time you set for strong caching) you don't need to do anything. \o/</li>
</ol></div>eleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.com4tag:blogger.com,1999:blog-3585841838339962909.post-35291249592157143552011-06-09T19:10:00.000-07:002011-06-09T19:10:11.900-07:00Programmatically Adding Image Content in Generic SetupHave you been sitting around for the last hour thinking "man, it would sure be great if I could add Images to my site on install but I just can't find a code example"? I know I was. Consider it documented!<br />
<br />
There will be a slideshow in the Plone Conference 2011 website (it's coming soon I swear!) that we wanted to work on install since it was 1. on the front page and 2. a PITA to set up every time. The code below gives "a" way to do this, although it could be taken to a much more abstracted level (what couldn't?!?).<br />
<br />
This assumes that there is a folder called images in your generic setup profile folder, which is likely to be in ... > profiles > default > images. The nice thing is that it doesn't rely on the magical, mystical .content jiggery. I won't rehash the code comments, but if you are using the <a href="http://plone.org/products/collective.easyslideshow">slideshow product</a> then maybe this cantation is for you!<br />
<br />
<script src="https://gist.github.com/1018109.js?file=fart.py">
</script>eleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.com2tag:blogger.com,1999:blog-3585841838339962909.post-7481677736828910142010-11-17T15:42:00.000-08:002010-11-17T15:42:04.341-08:00Plone, JQuery and Prototype: Easier Done Than SaidIt's hard to believe, but jQuery hasn't always existed. So some of us used other libraries, like Prototype[.js]. We even built entire systems on them. And then upgrade day came...<br />
<br />
I sat back with my largest bottle of beer and got ready for what was sure to be a javascript apocalypse. How could I merge 4 years of Prototype based javascript with Plones new JQuery libs? Surely it would be painful.<br />
<br />
And yet it wasn't. It will take me longer to write this blog post. But just in case there is some other straggling soul like myself who just can't get enough Event.observe(), fear not! Follow these simple steps:<br />
<br />
<ol><li>Create a new js file and add one line: jQuery.noConflict();</li>
<li>Then register that js file with your site. Prototype must go first or it will whine (remember not to compress or cook!). The noConflict js file must be listed before using any custom scripts. I recommend loading it right after jQuery.js. See below for a sample.</li>
<li>Reinstall. </li>
</ol><br />
Much can be said about jQuery playing nice and especially to the Plone javascript gurus who were kind enough to work within a namespace. Thank you. I owe you a beer.<br />
<br />
<script src="https://gist.github.com/704345.js?file=jsregistry.xml">
</script>eleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.com1tag:blogger.com,1999:blog-3585841838339962909.post-26246409820092070302010-09-14T17:24:00.000-07:002010-09-14T17:30:25.663-07:00LA Theme Sprint Report<div class="" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: left;"><a href="http://2.bp.blogspot.com/_3RIl2f8RMJQ/TJAQ5J0ffHI/AAAAAAAAIbE/ToRYz0VEWwM/s1600/Sparkling-xdv-theme.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="200" src="http://2.bp.blogspot.com/_3RIl2f8RMJQ/TJAQ5J0ffHI/AAAAAAAAIbE/ToRYz0VEWwM/s200/Sparkling-xdv-theme.png" width="148" /></a>Kudos to Michael Miller and Luke Brannon of the <a href="http://laplone.org/">LA Plone</a> Meetup for hosting the first (annual?) Plone Theming Sprint in LA. So many things about it were successful that I have had major writers block on how to get things started so I'm just going to jump write in.<br />
<br />
The most exciting part of the sprint was not only the diversity of the people, but the diversity of experience as well. Plenty of newbies overcame their fears to tackle new skills. For example, Heather released her very <a href="http://pypi.python.org/pypi/collective.fsdsimplifier/1.0">first product to pypi</a>, almost the whole crew picked up and ran with XDV, and Alice was an end user who just showed up and by the end of the first day was familiar with installing products through buildout, basic site admin, and fixing borked databases!<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/_3RIl2f8RMJQ/TJARUaB5sNI/AAAAAAAAIbM/AatPR2gVHh8/s1600/Welcome+to+Plone+%E2%80%94+Site.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="200" src="http://1.bp.blogspot.com/_3RIl2f8RMJQ/TJARUaB5sNI/AAAAAAAAIbM/AatPR2gVHh8/s200/Welcome+to+Plone+%E2%80%94+Site.png" width="132" /></a></div>Why would her database get borked you ask? Because she went through and installed all 52 themes listed on plone.org to evaluate quality, take screenshots where necessary, and took copious notes of all posted themes that didn't uninstall cleanly or didn't install at all.<br />
<br />
Now you are asking, why would someone do that? Only because we tackled the revamping PloneSoftwareCenter! This included adding ratings and we needed some data for when it goes live. Oh yeah. Alec took <a href="http://pypi.python.org/pypi/plone.contentratings/1.0-rc1">plone.contentratings</a> from bad ass to bad asser while I worked on integrating and revamping the UI. Check out the <a href="http://www.screencast.com/users/eleddy/folders/Jing/media/98b33d12-45b9-4d89-8c2a-4024fe60a21f">screencast</a> to get a preview of the changes. This will be a great accompaniment to the Plone 4 release so if you have any time at all to help get this deployed - ping me!<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/_3RIl2f8RMJQ/TJASe16vJEI/AAAAAAAAIbU/b8kH8Ci0SY0/s1600/plone-indication.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="200" src="http://1.bp.blogspot.com/_3RIl2f8RMJQ/TJASe16vJEI/AAAAAAAAIbU/b8kH8Ci0SY0/s200/plone-indication.jpg" width="108" /></a></div>And finally you ask, why would you want to revamp PSC? Glad you asked. Plone.org has less than spectacular support for navigating themes and we needed to have a beautiful place to display all of the new themes that the rest of the crew came up with. 3 Mikes, Trish, Tyler, and Albert banged away on creating 3 beautiful XDV themes (coming your way soon after solving the packaging issue).<br />
<br />
The best part of all? At the end all anyone wanted to know was when the next sprint would be. That's a trend I'm happy to obligue. Thanks to everyone who showed up and good work everyone!<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/_3RIl2f8RMJQ/TJAFSZzzoUI/AAAAAAAAIa8/HJ_UGQuSVWk/s1600/2010-08-28_20-09-39_901.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="225" src="http://3.bp.blogspot.com/_3RIl2f8RMJQ/TJAFSZzzoUI/AAAAAAAAIa8/HJ_UGQuSVWk/s400/2010-08-28_20-09-39_901.jpg" width="400" /></a></div><br />
</div>eleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.com11tag:blogger.com,1999:blog-3585841838339962909.post-69061988400959562592010-09-04T14:35:00.000-07:002010-09-04T14:35:50.601-07:00Feed a QA SprinterThe 2010 Plone Conference is just around the corner and sprint planning is well under way. One of the sprints will consist of creating Selenium tests and <a href="http://sourceforge.net/mailarchive/message.php?msg_name=5A77FD09-4C29-4826-9CEF-072B77B79B3E%40psu.edu">integrating with Hudson</a>. As Plone moves towards more and more Javascript and Ajax interface these tests will be essential for maintaining a high quality code base. This sprint will be great for newbies and veterans alike with some potentially boring but definitely worthwhile QA work.<br />
<div><br />
</div><div>Ideally this sprint is crowded and short so that people can get back to their regularly scheduled sprints. If you will be in Bristol for the sprints, please consider donating 1-2 hours of time to help us get up and running. If you won't be in Bristol and want to remote sprint stay tuned! However, if you can't do either (or just hate doing that stuff) but want to support the effort, please consider chipping in to <a href="http://ploneconf2010qasprint.chipin.com/plone-qa-sprint">feed a sprinter</a>.<br />
<br />
If anyone has other ideas on how to lure people in or has items to donate as door prizes/giveaways/whatever I'm all ears. Additionally, if you have a company and/or want to sponsor this sprint directly let me know and I'll cancel the chip in.<br />
<br />
Happy bug hunting!</div><div><br />
</div><object height="250" width="250"><param name="movie" value="http://widget.chipin.com/widget/id/8f32c4bf4b73fa7d"></param><param name="allowScriptAccess" value="always"></param><param name="wmode" value="transparent"></param><param name="event_title" value="Feed%20a%20QA%20Sprinter"></param><param name="event_desc" value="Plone%20Conference%202010%20at%20Bristol"></param><param name="color_scheme" value="blue"></param><embed src="http://widget.chipin.com/widget/id/8f32c4bf4b73fa7d" flashVars="event_title=Feed%20a%20QA%20Sprinter&event_desc=Plone%20Conference%202010%20at%20Bristol&color_scheme=blue" type="application/x-shockwave-flash" allowScriptAccess="always" wmode="transparent" width="250" height="250"></embed></object>eleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.com0tag:blogger.com,1999:blog-3585841838339962909.post-50228521608059448682010-07-21T13:29:00.000-07:002010-07-21T13:29:59.970-07:00Changing the Order of Fields in ArchetypesI wish I knew this 4 years ago.<br />
<br />
When creating new content based on archetypes, I usually start by copying the base schema, or one of my own base schemas with something like MyNewSchema = BaseSchema.copy(). One thing that has been historically hard in my case is that the schema order in the code directly correlates to the order in which its displayed. This means that the base copied data always goes first when displayed, which was not always what I wanted. The solution was custom templates and blech all over.<br />
<br />
Apparently you can just reorder these things. Deep in the bowels of archetypes there is a function called moveField. Now, reordering the schema can me as easy as MyNewSchema.moveField('description', pos='bottom'). Hot dog! Pasting the interface for reference.<br />
<br />
<script src="http://gist.github.com/485067.js">
</script>eleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.com6tag:blogger.com,1999:blog-3585841838339962909.post-43435270013618419612010-07-16T13:05:00.000-07:002010-07-16T14:33:58.907-07:00The Culture of Reporting Bugs<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Did you know that Plone has a QA team? It does! The <a href="http://plone-qa.tool.net/">QA team</a> is just getting off the ground, so it's a great time to talk about the culture of quality assurance. Since I don't use enough old english in my posts, I want to kick-off the topic with a couple of <b>QA Commandments </b>that when recognized by all members of the community ensure a yummy culture of quality. The first part involves bug reporting, and they go something like this:</span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b><br />
</b></span><br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"></span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>1.</b></span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>Thou shalt report every bug, big and small, content, documentation, infrastructure, and beyond</b></span><br />
<div><div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Is it a spelling mistake? Report it. Is the interface about as useful as a bag of sand? Report it. Is it impossible to find the documentation you are looking for? Report it!</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span> </div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Will your bug get fixed in a timely manner? Probably not. But that's ok. The point is that you noticed something that was below perfection, and you cared enough to document it by writing a bug report.</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span> </div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">I worked on a QA team right out of college and I am proud to say we worked just as hard (if not more!) as the developers. There was constant arguing between the teams but together we delivered a super stable end product. A key component of our strategy was competition within the QA team itself over who could report the most bugs. There was no prize, just pride. And it worked! Every nook and cranny of that system that wasn't perfect was documented.</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span> </div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>2. Thou shalt not bitch about bug reports</b></span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">This follows from commandment 1. Good QA people care even </span><b><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">more</span></b><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"> than the end user about a perfect experience, and caring takes a lot of effort. The thing that happens when developers stop complaining about bug reports is that it actually encourages people to report more bugs - imagine that!</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span> </div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">A good set of bug reports should be treated like problem documentation. The better documented the faults are the easier it is to fix and the easier it is to provide support until that fix goes through (since it may be a while). </span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span> </div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">The best part is that it's easy to encourage this culture. The next time you close a bug, big or small, thank the original reporter. </span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span> </div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>3. Thou shalt make it easy </b></span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>for everyone </b></span><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>to report bugs </b></span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Yes you! If you want to find lots of bugs you need to make it easy to report them. This </span><a href="http://elizabethleddy.blogspot.com/2008/04/testc.html"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">makes me nuts</span></a><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"> with just about every software product known to man. One click to report a bug. No screens, no searching, just get the info and get them outta there. And yes, it's on my list of things to change with Plone.org.</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span> </div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Unfortunately, I think Plone unnecessarily suffers from the bug collecting of zope and zodb. Searching for a zope bug collector without the exact keywords in google returns two bug collectors, one from 2003 and one from 2005. Both abandoned. And if you report in the wrong one they just close the ticket and that's it. Screw you end user for not being able to differentiate between a plone, zope, or zodb bug. Boo.</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span> </div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">So I am putting a list of collectors here for my reference as well as yours. Go ahead, report that bug you've been sitting on for a while. </span></div><div><ul><li><a href="http://dev.plone.org/plone/newticket"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Plone bug</span></a></li>
<li><a href="https://dev.plone.org/plone.org/newticket"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Plone.org bug</span></a></li>
<li><a href="http://dev.plone.org/plone/newticket?component=Documentation"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Plone Documentation Bug</span></a></li>
<li><a href="https://bugs.launchpad.net/zope2"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Zope2 Bug</span></a></li>
<li><a href="https://bugs.launchpad.net/zope-pas/"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">PAS</span></a></li>
<li><a href="https://bugs.launchpad.net/zope-cmf"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">CMF Bug</span></a></li>
<li><a href="https://bugs.launchpad.net/zodb"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">ZODB Bug</span></a></li>
<li><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Zope3 -> </span><a href="https://bugs.launchpad.net/bluebream"><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Bluebream</span></a></li>
</ul><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Now, if a bug gets marked invalid because it's someone else's problem you can just move on to the next tracker and follow through there. Did I miss any? </span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span> </div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>4. Thou shalt stand by that bug and not be bullied by developers</b></span></div></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">This is the hardest part of any QA effort. It's easy to give up when a developer spouts some mojo that you just don't understand. Nonsense poopy pants! Good QA people are so annoying developers wish they would go bury themselves in a hole in a forrest. And you know what? They are often wrong. But a lot of the time they are right, and a HUGE bug manifests from what seems like a template error. That's when it pays off to be a PIA.</span></div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><br />
</span> </div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">So stick to your guns QA jedi's and do what's right for the end user. If you always keep them in mind the end result will always be what's best for the product and the community.</span></div><div><div><div><span class="Apple-style-span" style="font-family: 'Lucida Grande';"><br />
</span></div></div></div></div></div>eleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.com4tag:blogger.com,1999:blog-3585841838339962909.post-23820952609167444772010-06-10T13:20:00.000-07:002010-06-10T14:25:29.118-07:00Diversity in IRCConfession: I was a closet female for a long time. My nick is purposefully ambiguous and when people assumed I was 'Eric' I never corrected them. I never signed my full name, not because I was afraid or insecure in my answer, but because I was afraid I would get treated differently for it. None of my profiles had my picture until just a few months ago, when I "came out" at the Plone Budapest conference.<br />
<div><br />
<div><a href="http://4.bp.blogspot.com/_3RIl2f8RMJQ/TBE1wPREvdI/AAAAAAAAIWA/jlbr0yrA5NM/s1600/areyouahotchick.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="400" src="http://4.bp.blogspot.com/_3RIl2f8RMJQ/TBE1wPREvdI/AAAAAAAAIWA/jlbr0yrA5NM/s400/areyouahotchick.jpg" width="187" /></a></div><div>I usually feel bad about it. For 7 years I could have been supporting women in technology. It seemed the easiest way to get by at the time. Was I being a coward? Maybe I was making it all up?</div><div><br />
</div><div>Then there are days like today, where a I get a gentle reminder as to where it all started. To the left is a screenshot of an #plone IRC chat with someone who was just completely lost in their task (probably still is). Many of us were helping out this person and at some point they moved it to a private conversation. Fine - that happens a lot. </div><div><br />
Click the image to get a "x-large" more readable version. I'll wait...<br />
<br />
</div><div>I'm not posting this to complain* - in fact I see it as a victory. I have no hard feelings and a simple correction not only took us back to business but raised the standard for getting help from that point forward.</div><div><br />
</div><div>Rather, I want people to understand why it's important for groups like PloneChix to exist. Diversity is scarce in the community and dealing with it is still new to many people. And guess what? The only way to make things better is to be more visible and pull through awkward moments like this. </div><div><br />
</div><div>It's easy to dismiss this as "not a Plone problem" when it's plone "users" that are the offenders but that is a cop out. If someone is new to helping out on IRC and is treated like this, we risk permanently losing their help and expertise. If it's normal to have women in IRC, at conferences, etc... then the standard for behavior will be higher to begin with.</div><div><br />
</div><div>So ladies, help me out - I'm tired of dealing with this alone. Take a moment to be seen: get in IRC, go to a conference, talk at a local meetup and invite your friends. One at a time, we can make Plone a stress free place for everyone.</div><div><br />
</div><div><span class="Apple-style-span" style="font-size: small;">*I spend way too much time arguing with people about whether or not this stuff still exists. Many think that it's a problem of the past because it doesn't happen in public forums or chats anymore. My response is usually "Duh! Of course it doesn't." Most incidents DO go unreported for that exact reason.</span></div></div>eleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.com5tag:blogger.com,1999:blog-3585841838339962909.post-40686007058609062272010-06-08T12:28:00.000-07:002010-06-08T12:28:52.166-07:00Searching the Plone3 APIAfter kvetching about the Plone API documentation, I realized that maybe it's not the docs that need updated, but rather just a better way to navigate through them. Inspired by an <a href="http://dukebody.com/2009/09/search-plugin-for-ploneorg-documentation/">awesome article by dukebody</a>, I looked into how chrome could better serve me. To set up API search in the url bar:<br />
<br />
<ol><li>Chrome > Preferences > Default Search: Manage</li>
<li>Click the + sign</li>
<li>Choose your name and keyword. I used 'Plone 3 API' and 'api'. Use this beast for the url: http://www.google.com/search?hl=en&q=site:http://api.plone.org/Plone/3.0/private/products/+%s&aq=f&aqi=&aql=&oq=&gs_rfai=</li>
<li>Save and enjoy.</li>
</ol><div>In my case to use it I just enter 'api PloneModuleName' into the url bar and click enter. This gives me instant gratification and filters out the weeds of versions I don't want. Plone on.</div>eleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.com4tag:blogger.com,1999:blog-3585841838339962909.post-35617846027109583872010-05-18T10:54:00.000-07:002010-05-18T10:54:58.479-07:00Adventures in Buildout: Zope 2.12 meets SOAPAfter 3 years of hanging around in Plone 2.5, I am finally getting around to upgrading our infrastructure to Plone 4. Part of that upgrade includes pulling out all of our services and api calls into a separate buildout, a solution that does not actually need the Plone package. A big problem with this migration was that we still use SOAP to talk to external vendors, most python SOAP modules suck, and none of them are fully integrated with a database/framework.<br />
<br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">As a satisfied user of <a href="http://www.zope.org/Members/Dirk.Datzert/SOAPSuppport">SOAPSupport</a> for many years, my first inclination was just to continue with an old version of Zope and continue to use that. However, the project hasn't been supported/updated in a long time and I was excited to take advantage of memory improvements in Python 2.6, a fully eggified Zope, and Blob storage among other things. There is a great article on <a href="http://plone.org/documentation/kb/soap-support-for-plone">configuring SOAP in Plone</a>, which targets Zope 2.11 and custom products, however its still very mind bending for those of us with simpler needs.</div><br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">While updating the z3c.soap package for Zope 2.12 compatability, I was also busy trying to grok "new" buildout concepts (I know - I've been in the closet for a while). I couldn't find one really solid simple tutorial on getting started with buildout. After a roller coaster ride of emotions and awesome help from the Plone community, I *think* I've figured out the simplest way to get any python SOAP server up and running with (or without) buildout. I have not addressed the WSDL or complex types questions yet, since I don't need them. I also said simplest, which means I had little regard for proper classing and blah blah blah... yell at me in the comments if it's grossly off-key or if I missed something - it's bloody likely.</div><br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">Below is an introduction to configuring SOAP in Zope as well as an extra detailed buildout process for old-schoolers like myself who are used to Products based install. I hope it helps someone get up and running with a python soap server fast, in a repeatable, buildout style.<br />
<br />
</div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">FYI, this can also easily be customized to support Plone and other zope based apps since its buildout based. My goal was to make this like <a href="http://plone.org/products/mysite">MySite</a>, where you can start and build off of it. In the end I don't think I have even come close the greatness that was MySite but you can still <a href="http://plonechix.pbworks.com/f/zsoap.zip">download the finished package</a> if you don't care about the how and just want the now, or if you want to follow along easier.<br />
<br />
</div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">NOTE: At the time of writing virtualenv has a bug with this buildout so if you get an ImportError for shutil, just use regular old python. This is also why this tutorial is not using virtualenv, although its perfect for this scenario otherwise.<br />
<br />
</div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: x-large;">Summary</span></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">For those who don't care about buildout or the details, <a href="http://plonechix.pbworks.com/ZopeSoapBuilout">here is the recipe</a> for simple zope + soap. Customize away!<br />
<br />
</div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: x-large;">The Details: Creating the Buildout</span></div>I won't go into the philosophy of buildout since its all over so we can get to implementing. I keep all of my buildouts in a directory in my home folder, called 'buildouts', so I always know where to start. Also, before you get to the end and kick yourself, I'm using Python 2.6 here. I have not tested it with any other version. Caveat Emptor.<br />
<br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">First, make sure that you have the buildout package installed:<br />
<br />
</div></div><pre>> sudo easy_install zc.buildout
</pre><br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">Create a buildout environment, with all the scripts we need to get started</div></div><pre># create a new directory for our new buildout
> cd ~/buildouts
> mkdir zsoap
> cd zsoap
# initialize the buildout environment
> buildout init
Creating '/Users/eleddy/buildouts/zsoap/buildout.cfg'.
Creating directory '/Users/eleddy/buildouts/zsoap/bin'.
Creating directory '/Users/eleddy/buildouts/zsoap/parts'.
Creating directory '/Users/eleddy/buildouts/zsoap/eggs'.
Creating directory '/Users/eleddy/buildouts/zsoap/develop-eggs'.
Generated script '/Users/eleddy/buildouts/zsoap/bin/buildout'.
# optional: include bootstrap.py
> wget http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py
</pre><br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">You don't have to include the bootstrap.py file, but it makes things hella easier and I'll assume that you have it for the rest of this. After initializing the buildout, you will have a bunch of extra directories in your zsoap folder, including bin, buildout.cfg, develop-eggs, eggs, and parts. Don't worry about them for now - just make sure they are there.</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">Next, edit buildout.cfg so it looks like <a href="http://plonechix.pbworks.com/ZopeSoapBuilout">this</a>. I'll explain it all while we wait for the buildout to run.</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">Finally run the buildout:</div></div><pre>> python bootstrap.py
> ./bin/buildout
</pre><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">Note that whatever version of python you run bootstrap with is the version that zope will be running with. This recipe was created in Python 2.6, since Zope 2.12 runs swimmingly on it. If your system python is not 2.6, make sure to alter your bootstrap command to reflect that. </div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">While its running, you may see errors like "SyntaxError: ("'return' outside function",...". Don't worry about those. It's simply trying to compile scripts, which you can't do.</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: x-large;">The Breakdown</span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">Once the buildout starts going, go ahead and take a shower, walk the dog, or read the line by line description of what is happening:</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">1. [buildout]</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">2. parts = scripts</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">3. instance</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">4. test</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">5. </span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">6. extends = http://svn.zope.org/*checkout*/Zope/tags/2.12.5/versions.cfg</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">7. versions = versions</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">8. eggs = z3c.soap</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">9. extensions = buildout.threatlevel</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">Lines 1-4 say that this buildout has three parts that need to be run. When you run this, you will see that the "parts" directory that was automagically created from initializing the environment now has 3 folders with the exact same names. Coincidence? You decide. </div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">Line 6 is a link to download an automatically configured buildout for installing zope 2 in a delicious eggy format. To change versions of zope, just modify this url. Go ahead. <a href="http://svn.zope.org/*checkout*/Zope/tags/2.12.5/versions.cfg">Try it</a>.</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">Line 7 Indicates that we are going to force some packages to get a certain version, instead of the latest. This is called "pinning" a version. Some packages have conflicting versions, and you may need to pin the version which satisfies everything. This line simply tells buildout that there is a section, called versions, which will do just that.</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">Line 8 indicates which additional eggs we need to get the project going. It will ask <a href="http://pypi.python.org/pypi">PyPi</a> for the latest version of this eggs (unless we pinned it) and install them. Notice that we didn't have to include Zope2 because it's handled with the scripts section with a special recipe that you'll see later.</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">10. [versions]</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">11. Zope2 = 2.12.5</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">12. </span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">10-11 show how to pin a version of a package. This is not necessary for this case since the Zope2 recipe does this for us, but other packages probably won't have this luxury. Maybe you want to pin to a minor version, for example. Furthermore I already numbered these lines and I'm too lazy to go through and renumber them.</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Times;"><span class="Apple-style-span" style="font-size: medium;"><br />
</span></span></span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">13. [scripts]</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">14. recipe = zc.recipe.egg:scripts</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">15. eggs = Zope2</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">Lines 13-15 might as well be magic. They install Zope2, the inner workings of which I have no idea.</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Times;"><span class="Apple-style-span" style="font-size: medium;"><br />
</span></span></span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">16. [instance]</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">17. recipe = plone.recipe.zope2instance</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">18. user = admin:admin</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">19. http-address = 8081</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">20. products = ${buildout:directory}/products</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">21. debug-mode=on</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">22. zcml = z3c.soap</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">23. eggs = ${buildout:eggs}</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">Lines 16-23 create a new zope2 instance. It's pretty self-explanatory except for a couple lines. Line 22 males sure that we include the z3c.soap package, and run the zcml slug that initializes it. This patches the publisher to make it accept soap requests. Line 23 makes sure that all the eggs created in other places of this buildout are put in the path of the instance. Without this, it won't be able to import anything.</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">24. [test]</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">25. recipe = zc.recipe.testrunner</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">26. eggs = z3c.soap</span></span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">And finally, we must not forget to include test cases to make sure everything is running smashingly. </div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: x-large;">Up and Running</span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">Assuming everything runs correctly, we can start testing things out. First we want to validate that everything looks like its running ok.<br />
<br />
</div></div><pre>> ./bin/test
</pre><br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">It's that easy. This should run fine and pass all tests. Yeehaw! Let's start the new zope 2 instance in foreground mode:</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><pre>> ./bin/instance fg
</pre><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">Still easy. Woohoo! If the slug was installed correctly, z3c.soap will have a line in the trackback that says something like "INFO Zope z3c.soap: modified ZPublisher.HTTPRequest.processInputs". Congratulations, you are ready to serve SOAP requests.</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: x-large;">Taking it a Step Awesomer</span></div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">Now what? Well, you can follow the <a href="http://plone.org/documentation/kb/soap-support-for-plone">original tutoria</a>l and add wsdls and custom type and all that fancy stuff. Since I'm lazy I prefer to do everything through scripts. In the ZMI, I like to add a folder called 'services' and then just pile up python scripts in there. Err... I mean Script (Python)'s. Let's verify it's all SOAPed up like we want. Add a script called 'test' in a folder called 'scripts' that simply returns a string 'hello world!'. Then you can use the <a href="http://sourceforge.net/projects/pywebsvcs/files/SOAP.py/">SOAPpy</a> package to see what's really going on. From the python prompt:<br />
<br />
</div></div><pre>>>> from SOAPpy import SOAPProxy, URLopener
>>> url = "http://localhost:8081/services"
>>> namespace = "http://plone.org"
>>> server = SOAPProxy(url)
>>> server.config.dumpSOAPOut = 1
>>> server.config.dumpSOAPIn = 1
>>> print server._ns(namespace).test()[0]
hello world!
</pre><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">Every script you add will have a SOAP interface built in. How cool is that? Also, a tip from the trenches, use dictionaries to pass back values for more complex values. In my experience, it's more compatible with Java stacks, which I assume you have to deal with, because otherwise you wouldn't be using SOAP. I digress...</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">So I know you're thinking, what is the awesomer part of "taking it a step awesomer"? Well, if you want a repeatable buildout you don't want to store your scripts in the ZMI. I know I didn't, since we have an SVN repository that is way better for such things (again, I'm behind, I'm not cool enough to Git or Hg yet). We can do is use the CMF Filesystem Directory view and store all of our scripts on the filesystem. While they are not editable in the ZMI, but the can be executed and customized.</div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">I tried my hardest to find a way to do this without making a product, but I can't find a way to register a CMF directory view without being in the context of a product. So let's walk through making one - you probably need it anyways. It's slightly annoying, but at least this way it won't get wiped by re-running buildout. If you already have a product then skip the paster part and just add the necessary slugs.<br />
<br />
First make sure that you have access to paster. If you can't use paster from the command line,<br />
<pre></pre><pre>> sudo easy_install -U ZopeSkel
</pre><br />
Then decide a name for the product, and run paster. I will use z.soap for this purpose.<br />
<pre></pre><pre>> cd develop-eggs
> paster create -t basic_zope
> ...blah blah blah
</pre><br />
Next we need to add our new egg to the buildout.cfg, mark it as a development package, and include it in our zope instance. Update your buildout.cfg to look like <a href="http://plonechix.pbworks.com/ZopeSoapBuilout2">this</a>, with the bold lines indicating the changes. Hopefully by now the buildout thing is starting to make sense. But we aren't done yet.<br />
<br />
Now we need to actually register a services directory. Let's make the directories first<br />
<pre>> mkdir ~/buildouts/zsoap/develop-eggs/z.soap/z/soap/skins
> mkdir ~/buildouts/zsoap/develop-eggs/z.soap/z/soap/skins/services
</pre><br />
All of the Script (Python)'s that we want to be version controlled can go in the services directory. You can version control the whole buildout, or of course register your new egg with PyPi and replace that whole mess with eggs = my.egg.<br />
<br />
Tell zope about the directory that you created by editing your new eggs configure.zcml, which will be in ~buildouts/zsoap/develop-eggs/z.soap/z/soap/configure.zcml. Add the registerDirectory directive to point to the folder you just created, adding the cmf namespace if needed. The final configure.zcml should look like <a href="http://plonechix.pbworks.com/ConfigureZCMLZopeSoap">this</a>, with the changes highlighted.<br />
<br />
Phew! Are we done yet? No. Re-run buildout and restart your instance<br />
<br />
<pre>> ./bin/buildout
> ./bin/instance fg
</pre><br />
At the root of the ZMI, add a Filesystem Directory View and you will see that your directory is listed. Let's put the same test script that was in the zodb the first time in the filesystem now. In your new skins, services directory, add a file, test.py with the contents below and run the SOAPpy test above. Viola! Now you can add scripts until you are sick. They can compute, forward info, act as an API, and even interact with your products. No restart needed to add/change scripts since its a filesystem view.<br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"></div><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">##Script (Python) "test"
##bind container=container
##bind context=context
##bind namespace=
##bind script=script
##bind subpath=traverse_subpath
##parameters=
##title=
##
return "hello world!"</pre><br />
But wait, you don't want to do that last step manually? You want that automated? I did too! Last but not least, let's add the directory view on startup if it's not already there. There is a special initialize function that we can call on startup. Open up develop-eggs/z.soap/z/soap/__init__.py and make it look like <a href="http://plonechix.pbworks.com/ZopeSoapInitialize">this</a>. Now instead of calling zope2 initialization, we need to call our own package initializer by modifying the configure.zcml to look like <a href="http://plonechix.pbworks.com/ConfirgureZCMLZopeSoap2">this</a>. Now start up the zope instance and it will come alive!<br />
<br />
So, if there is anything wrong here or it could be done easier, please comment and I will update. The last part was definitely not as easy to setup as I would have hoped. And because I'm so nice, you can <a href="http://plonechix.pbworks.com/f/zsoap.zip">download this package</a>, untar, bootstrap, build, and go crazy with customizing. </div></div>eleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.com4tag:blogger.com,1999:blog-3585841838339962909.post-86946734861447194552010-02-22T17:51:00.000-08:002010-02-24T15:05:02.218-08:00Catalog Key ErrorsApparently I am the patron saint of <a href="http://plonechix.blogspot.com/2009/12/definitive-guide-to-poskeyerror.html">key errors</a>. The latest one was with the catalog, and it was way nicer than some of the previous monsters because at least it didn't bring down the whole site or involve sweaty, hand shaking stress. To the end user, certain keyword searches didn't in livesearch, advanced queries. This key error manifested in the logs as:<br />
<br />
<pre>Module Products.ZCTextIndex.BaseIndex, line 203, in search_phrase
Module Products.ZCTextIndex.OkapiIndex, line 161, in _search_wids
KeyError: -1525983394
</pre><br />
I had a feeling that this would happen because a few days earlier, I came across a few weird errors trying to emply some jedi mind tricks in the db. An example of one:<br />
<br />
<pre>2010-02-17 21:46:36,505 ERROR @/opt/Zope-2.9/lib/python/Products/PluginIndexes/common/UnIndex.py/UnIndex.py UnIndex 194 : DateIndex: unindex_object could not remove documentId -1324564720 from index modified. This should not happen.
Traceback (most recent call last):
File "/opt/Zope-2.9/lib/python/Products/PluginIndexes/common/UnIndex.py", line 168, in removeForwardIndexEntry
indexRow.remove(documentId)
KeyError: -1825924534
</pre><br />
Whoops. They didn't cause anything obvious to happen so I did what every lazy sys admin does and ignored it. As far as I can tell there isn't a required correlation between the two errors, but it will at least point you in the right direction.<br />
<br />
That was Friday night. Then customers got on the system Monday and we started getting the ZCTextIndex KeyError sporadically. It affected all searches with the word 'jackson' and using indexes set up with OkapiIndex algorithm (PloneLexicon). This is the default choice for the Title, SearchableText, and Description indexes.<br />
<br />
Looking this error up on Google shows a few answers, which involve <a href="http://markmail.org/message/iupk5lo3bpr2wnow#query:Products.ZCTextIndex.OkapiIndex%20KeyError+page:1+mid:v5dv44gmbhzhabzn+state:results">clearing and rebuilding</a> the catalog. I'm sure this works, but our index alone is 1.2GB so that was really a last resort option. It would take so long that before reindexing we probably just would have monkeypatched the catalog code to just ignore those keys. I was determined to find a way to fix without rebuilding. And I did.<br />
<br />
Warning: the following explanation is based on eye crossing interpretations of zope code. I may be off a little but I think its a worthy attempt nonethless.<br />
<br />
The text indexes need a way to correlate a word set (what the user is searching for) with a document and vice-versa. For example, when I lookup 'jackson', I need to know that 17 documents have that keyword. If you have been mucking around with the catalogs or something went crazy on your catalog (like your sys admin) then you can get this words list of sync with other indexes and the real catalog. To confirm this, try idx.getEntryForObject(docid) on the failing index, where docid is the KeyError integer in your error log. In my example, the keyword 'jackson' in the words BTree pointed to a list of documents, all of which were valid except -1525983394. By just extracting the bad reference to this non-existing document, then there is no need to reindex everything.<br />
<br />
Note that for this to work you need to know the word that triggers the error. You can't do a reverse lookup to find out which words reference that document because that mapping is the one that is still correct. It will always return empty due to a silent fail. If your users won't tell you, then you can just log it when it errors out later. If its really bad, iterate through all words and what they point to and see if they throw a key error (hint: use the _wordinfo.iterkeys() iterator + other tools listed <a href="http://docs.zope.org/zope3/Code/BTrees/IIBTree/IIBTree.1">here</a>). Or just reindex.<br />
<br />
As always, backup before hand and if you are nervous work on a copy first. I'm sure this can easily be modified for other indexes/schemes as well.<br />
<br />
From the debug prompt:<br />
<pre>'''
@docids is a list of the key error items as ints, not strings
@word is the word that triggers the index error
@cat is the catalog object that has the error, i.e. app.foo.portal_catalog
'''
def removeFromWordCatalog(docids, word, cat):
import transaction
for docid in docids:
for indexObj in cat.getIndexObjects():
try:
itype = indexObj.getIndexType()
except AttributeError:
continue # some indexes don't implement this method
lex = indexObj.getLexicon()
if itype == 'Okapi BM25 Rank':
wids = lex.termToWordIds(word)
idx = indexObj.index
for wid in wids:
try:
# using excepts because BTrees key lookup is annoying
# see if its even really an error before ousting
blah = idx._wordinfo[wid][docid]
except KeyError: # not listed - nothing wrong
continue
idx._wordinfo[wid].pop(docid)
idx._wordinfo[wid] # persistance
try:
idx._wordinfo[wid][docid]
except KeyError:
print "sucessfully removed"
transaction.savepoint(1)
transaction.commit()
docids = [-1525983376, -3423423445, ...]
word = 'jackson'
cat = app.foo.portal_catalog
removeFromWordCatalog(docids, word, cat) </pre><br />
UPDATE: I already got to the point where the word by word thing became annoying so I wrote a generic function to "unword" a catalog entry. Much nicer, and although I thought it would take a long time it was actually not toooooo bad. I also pieced out the code so I could get as generic or specific as I want in the case that something else happens. Which it will. Woohoo!<br />
<br />
<pre>def removeDocFromWordlist(idx, wid, docid):
import transaction
try:
# using excepts because BTrees key lookup is annoying
# see if its even really an error before ousting
blah = idx._wordinfo[wid][docid]
except KeyError: # not listed - nothing wrong
return
idx._wordinfo[wid].pop(docid)
idx._wordinfo[wid] # persistance
try:
idx._wordinfo[wid][docid]
except KeyError:
print "sucessfully removed"
transaction.savepoint(1)
transaction.commit()
def cleanUpBadDocId(docid, idx):
iterati = idx._wordinfo.iterkeys()
while True:
try:
wid = iterati.next()
except StopIteration:
break
if idx._wordinfo[wid].has_key(docid):
print "removing %s from word list %s"%(docid, wid)
removeDocFromWordlist(idx, wid, docid)
def removeFromWordCatalogs(docids, cat):
for docid in docids:
for indexObj in cat.getIndexObjects():
try:
itype = indexObj.getIndexType()
except AttributeError:
continue # some indexes don't implement this method
lex = indexObj.getLexicon()
if itype == 'Okapi BM25 Rank':
idx = indexObj.index
cleanUpBadDocId(docid, idx)
docids = [-1525983376, -3423423445, ...]
cat = app.foo.portal_catalog
removeFromWordCatalogs(docids, cat)
</pre>eleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.com2tag:blogger.com,1999:blog-3585841838339962909.post-54281459121171414692010-02-16T23:37:00.000-08:002010-02-17T08:34:20.862-08:00PloneChix: Hot and Not<div><div><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">Instead of boring you with a lengthy explanation of our official <a href="http://plonechix.pbworks.com/Our-Purpose">purpose</a>, I'd like to address what makes PloneChix exciting and unique and then tackle some misconceptions about what people think PloneChix represents. As a followup post, I'd like to invite all PloneChix members to explain why PloneChix exists for them. There will be more meaning and inspiration from their stories than reading an tedious political statement.</span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;">What is so hot about PloneChix? PloneChix is ...</span><br />
<div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><ul><li>... a no flame zone. That means no RTFMs, no stupid questions, no negative feedback, and no smack talk. This is the most important aspect of who we are. Women in technology, especially open source, are more likely to withdraw from a community because of actions/comments/answers perceived as harsh. By making a special effort to make a no flame zone, we are encouraging people to ask for help and giving them helpful, thorough, honest answers by any means possible. Who knows, maybe all this kindness will be contagious. </li>
<li><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">... a place to be heard.</span></span><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;"> Speak up, let us help you, stick around Plone for a little bit longer and we think you'll like what you see.</span></span></li>
<li><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">... a place to ask non-technical questions (although we like technical too). For example: What is the job situation in the bay area for Plone integrators? Does company X have a supportive female developer community? What can I do to market myself better as an integrator/developer/documenter? </span></span></li>
<li><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">... an open and supportive learning environment. This is the perfect arena to tell a story and get honest feedback. Did you blow that interview because you underrepresented yourself or did you really not have enough experience? Let's talk about it. It's a place to get advice from women with different experiences and backgrounds.</span></span></li>
<li><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">... an opportunity to tackle that project with a group instead of doing it on your own. We have just started working on defining our </span></span><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><a href="http://plonechix.pbworks.com/Project-Ideas">projects for this year</a>.</span></span><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;"> Why not add yours? If you have wanted to commit to the core for a while but just didn't want to tackle such a huge project on your own or want some help, we'll make sure to hook you up with a veteran who will show you the ropes.</span></span></li>
<li><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">... a place to network. PloneChix is here to share and encourage each other to write that first book or give a talk at that conference. In my opinion, visibility is the #1 issue with women in the Plone community. Be visible, be heard, and get your name out there. There are lots of special opportunities for women in technology, and if we don't take advantage of them women in other communities will (i.e. Ruby women are fierce). Posting job opportunities, calls for papers/talks, scholarships, et al is highly encouraged.</span></span></li>
</ul><div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">Hot indeed. There are already some strange rumors about PloneChix that have surfaced that I'd like to take a moment to address. PloneChix is NOT...</span></span></div></div></div></span></span></div></div><ul><li><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">... a group of man-hating femi-nazis. In fact it would be way easier to justify if we were. However, we do have a tendency to focus on the needs of women and womens issues in the Plone, python, and Open Source Software (OSS) communities.</span></span></li>
<li><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">... a derogatory name. It's an homage to </span></span><a href="http://www.linuxchix.org/about-linuxchix.html"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">LinuxChix</span></span></a><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">, one of the finest female-friendly OSS movements to this day. There are many other groups that pay tribute to the original, including DevChix, DrupalChix and CodeChix, just to name a few. The word "Chix" has become synonymous with female fronted groups that discourage friendly fire (no RTFMs) and encourage low barrier to entry.</span></span></li>
<li><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">... a "women only" club. Anyone is welcome as long as they are supportive of the cause and contribute positively back to it and/or the Plone community.</span></span></li>
<li><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">... a support group for complaining or flaming, especially about other members of the community. PloneChix is a place for positive feedback, encouragement, and help. We are about positive change, not negative reinforcement.</span></span></li>
<li><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">... looking for 50:50 male to female ratio just for the sake of having a 50:50 male to female ratio. On the contrary, we are all interested in seeing the entire Plone community grow and thrive in the way that </span></span><a href="http://infotrope.net/blog/2009/07/25/standing-out-in-the-crowd-my-oscon-keynote/"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">Kirrily Roberts</span></span></a><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">* describes.</span></span></li>
<li><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">... a replacement for the mailing list, irc, or other more formal means of communication about Plone. It is an additional communication channel lead by like-minded individuals. It is also a place to talk about careers, meetups, projects, or even just to brag about a new project in a friendly, open environment. </span></span></li>
</ul><div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">This is the part of the post where I start practicing what I've been preaching. Girls, boys, aliens, machines: how does this make you feel? Hint: I was shooting for all warm and tingly inside.</span></div></div><div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br />
</span></span></div></div><div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">If you are interested in participating, male or female, check out </span></span><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><a href="http://plonechix.pbworks.com/">our wiki</a>,</span></span><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;"> which has all the information you need to get started.</span></span></div></div><div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br />
</span></span></div></div><div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: small;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">* Please read this post 50 times. Take a nap. Then 50 more. Then we'll talk.</span></span></div></div></div></div>eleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.com0tag:blogger.com,1999:blog-3585841838339962909.post-15721538814733464912009-12-02T18:31:00.000-08:002010-02-05T16:52:35.752-08:00The Definitive Guide to POSKeyErrorTrying to get PloneChix off the ground during the holidays has been a tough start so I figured we could get the blog a good head start with a much needed post for anyone who has had the hair pulling, spirit breaking chance to work with a POSKeyError. Happen upon one of these bad boys and your life is filled with countless threads, code snippits, and half finished <a href="http://plone.org/documentation/error/poskeyerror/">reference articles</a>. Here is a shot at explaining the source of the problem to start with (prevention), and giving a consolidated page of tools to tackle the beast that is POSKeyError head on (treatment).<br />
<br />
<span style="font-size: large;">Getting Started</span><br />
If you are on this page, there is a good chance you are pooping yourself right now because the database isn't responding, things aren't happy, systems are failing. Let's take this moment to make a copy of everything in its messed up state and while that copies - breathe. Here's your worst case scenario: everything is screwed and you have to restore from backups. I've dealt with several different KeyErrors and never had to restore but that doesn't mean you won't.<br />
<br />
In fact, if you have recent enough backups to where you can restore with minimal data loss, do it. This is not a fast process (set aside at least 3 hours, eons more if you don't have giant cojones). You will not likely fix the source of the problem but at least your dbs will be back online. If you have any scheduled cron jobs for packing the databases or taking backups make sure they are disabled since there is a good chance that one of those could re-trigger the error.<br />
<br />
So start copying and if your db is huge go tell someone to start pulling that restore from tape. Breathe. <br />
<br />
<span style="font-size: large;">Types of Key Errors</span><br />
In my experience, there are two types of key errors - POSKeyError and [connection] KeyError. It's hard to pinpoint which one you have sometimes because they can mask as one another but once you get into debugging you'll know right away.<br />
<br />
<a href="http://spamsch.blogspot.com/2009/04/how-to-repair-broken-zodb-poskeyerror.html">In theory</a>, <b>POSKeyError</b> is short for POSitional Key Error. It's likely to happen if the database gets corrupted, you blow out the file system storage space, your iscsi drive flip out, etc... It <i>may</i> happen when you cross reference database (more on that later) but I've only seen a more traditional KeyError in that case. Your trackback probably looks something like this*:<br />
<br />
<pre>Traceback (most recent call last):
File "/usr/local/lib/python/ZEO/zrpc/connection.py", line 421, in handle_request
ret = meth(*args)
File "/usr/local/lib/python/ZODB/FileStorage/FileStorage.py", line 625, in modifiedInVersion
pos = self._lookup_pos(oid)
File "/usr/local/lib/python/ZODB/FileStorage/FileStorage.py", line 514, in _lookup_pos
raise POSKeyError(oid)
POSKeyError: 0x172024 </pre><br />
Notice that this is referencing the object that its looking up and the error is not happening in a connection handling phase. This type of error may or may not cause your whole site to fart, depending on where the corrupted object is. For example, if the error is in your portal_skins/custom folder then you are screwed since aquisition is probably picking it up for every site.<br />
<br />
A quick digression on the 0x172024 "thing". This is the oid (object id, in the zodb) of the object that is the offender. For this to be useful in any database manipulation, we will most certainly need to convert it using the p64 function in ZODB.utils. A quick docstring excerpt of two very useful functions we'll use later to convert this oid to an 8-byte string*:<br />
<br />
<pre>def p64(v):
"""Pack an integer or long into a 8-byte string"""</pre><pre>def u64(v):
"""Unpack an 8-byte string into a 64-bit long integer."""
</pre><br />
The other <b>KeyError</b> (which manifests as POSKeyError and just KeyError, but I'll refer to as just KeyError for clarity) usually has some sort of connection traceback info in the error message like so:<br />
<br />
<pre>2009-12-01 16:51:18,056 ERROR @/opt/Zope-2.9/lib/python/ZODB/Connection.py/Connection.py Connection 737 : Couldn't load state for 0x01
Traceback (most recent call last):
File "/opt/Zope-2.9/lib/python/ZODB/Connection.py", line 732, in setstate
self._setstate(obj)
File "/opt/Zope-2.9/lib/python/ZODB/Connection.py", line 786, in _setstate
self._reader.setGhostState(obj, p)
File "/opt/Zope-2.9/lib/python/ZODB/serialize.py", line 604, in setGhostState
state = self.getState(pickle)
File "/opt/Zope-2.9/lib/python/ZODB/serialize.py", line 597, in getState
return unpickler.load()
File "/opt/Zope-2.9/lib/python/ZODB/serialize.py", line 479, in _persistent_load
return self.loaders[reference_type](self, *args)
File "/opt/Zope-2.9/lib/python/ZODB/serialize.py", line 540, in load_multi_oid
conn = self._conn.get_connection(database_name)
File "/opt/Zope-2.9/lib/python/ZODB/Connection.py", line 305, in <b>get_connection
</b> new_con = self._db.databases[database_name].open(
KeyError: 'mysite'
</pre><br />
Ewe. This ones a little more mucky. It may or may not have an oid. These <a href="http://plone.org/support/forums/general#nabble-td3658431">connection KeyErrors</a> happen when you have multiple database mounts and one database is <a href="http://docs.zope.org/zope3/Code/persistent/wref/WeakRef.1">weak referencing</a> another. !@#$ -> what the hell does that mean, right? Let's say you have two Plone Sites, foo and bar, both mounted onto your main db. You are messing around one day and you customize logo.jpg for foo. But you messed up and that was actually meant to be for bar so being a good lazy dev you just use plone's sweet cut and paste functionality to move logo.jpg from foo into bar. Everything is working perfect (and this is indeed "legal"). This will continue to work, until the fateful day that you pack foo and then BAMN! It's key error time.<br />
<br />
When you cut and pasted logo.jpg from foo to bar, you didn't actually create a new object. You created a weak reference to foo from bar. As long as that reference wasn't garbage collected in foo, you were golden. When you <a href="http://old.nabble.com/Missing-loader-for-multidatabase-refs--tt9744503.html#a9744503">run pack</a>, it destroys references to the object. The official explanation from ZODB/cross-database-references.txt<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;"><span style="font-size: x-small;"> Garbage collection is done on a database by database basis.</span></div><div style="font-family: "Courier New",Courier,monospace;"><span style="font-size: x-small;"> If an object on a database only has references to it from other</span></div><div style="font-family: "Courier New",Courier,monospace;"><span style="font-size: x-small;"> databases, then the object will be garbage collected when its</span></div><div style="font-family: "Courier New",Courier,monospace;"><span style="font-size: x-small;"> database is packed. The cross-database references to it will be</span></div><div style="font-family: "Courier New",Courier,monospace;"><span style="font-size: x-small;"> broken.</span></div><br />
What is bar to do now? It can't find the logo.jpg object, and since every page in your site has a logo you are screwed.<br />
<br />
I've seen this happen for non cut/paste reasons as well. If you get the error after a fresh restart on one zope and the others are running fine, then something happened in the mounting of your databases that messed up the stream. I don't know the exact details of why, but I've seen rapid restarts of a zope instance invoke this hella error. Do not restart the other zopes if you can avoid it and use those to start salvaging data just in case. If you are lucky, just go into the undo at the root of the site and undo the last couple of transactions, especially if you see a suspicious addMount transaction. That might be all you need!<br />
<br />
<span style="font-size: large;">Verifying</span><br />
Let's get to fixing then shall we! First things first, if you know you had some kind of activity on the disk that may have corrupted the data, we can easily verify by running <a href="http://wiki.zope.org/ZODB/FileStorageBackup">fstest.py</a>:<br />
<br />
<pre> /usr/local/bin/fstest.py /srv/zeo/zeo0/Data.fs
</pre><br />
If nothing is corrupted, it will return nothing. Bad data will return something like<br />
<br />
<pre>/srv/zeo/zeo0/Data.fs truncated possibly because of damaged records at 1670660850
</pre><br />
Yuck! But at least we know what's going on. If this is indeed the case, you may want to start with <a href="http://spamsch.blogspot.com/2009/04/how-to-repair-broken-zodb-poskeyerror.html">@spamsch's great tutorial</a> on dealing with transaction corruption and then come back here if that doesn't work.<br />
<br />
To check for bad references, we can use <a href="http://wiki.zope.org/ZODB/FileStorageBackup">fsrefs.py</a>:<br />
<br />
<pre> /usr/local/bin/fsrefs.py /srv/zeo/zeo0/Data.fs
</pre><br />
Unfortunately I lost my example of what a bad ref error looks like but its similar to fstest. Most importantly it will tell you the oid of the item that is causing things to break. If you're error message has no oid in it, you will absolutely need this!<br />
<br />
I have not tried it, but it may be worth checking out <a href="http://pypi.python.org/pypi/zc.zodbdgc/0.4.0">zc.zodbgc</a> and its multi-zodb-check-refs script. <br />
<br />
If you are certain its a [connection] key error and nothing shows up, you can try to <a href="http://www.zopelabs.com/cookbook/1054240694">manually search</a> for it from within the [zopectl] debug console:<br />
<br />
<pre>from ZODB.POSException import POSKeyError
def error_finder(folder, exception=POSKeyError, stop_on_first=None):
""" start at the given folderish object.
If stop_on_first is true, quit after one exception;
otherwise, keep going through the whole tree."""
for id, next_item in folder.objectItems():
try:
print id
next_item.getId()
except exception:
print `exception`, "in folder",
print '/'.join(folder.getPhysicalpath()),
print "at id:", id
if stop_on_first:
raise "done" # hack to break out of recursion
else:
# no error, recurse if it's objectManagerish
if hasattr(next_item.aq_base, 'objectItems'):
error_finder(next_item, exception, stop_on_first)
error_finder(app.mysite, stop_on_first=True)
</pre><span style="font-size: large;">Fix 'Er Up</span><br />
The goal here is simple: delete the offending object. This can be tricky since its hard to delete something that doesn't exist. This can all done from the [zopectl] debug console. If you think this is a connection key error and you are working from zopectl debug, first and foremost modify your config file to get rid of any unnecessary mounts. Broken objects are better than key errors and if it allows you to recover certain parts then even better.<br />
<br />
Once in the debug prompt we can <a href="http://www.zopezone.com/discussions/general/00000168">find out which object</a> that oid represents:<br />
<br />
<pre>from ZODB.utils import p64
o = root._p_jar[p64(0x172024)
print o.getId()
print o.bobobase_modification_time()
print o.meta_type
print o.__ac_local_roles__
</pre><br />
Hopefully you now know the id of the offending object and lets try to delete it<br />
<br />
<pre>from AccessControl.SecurityManagement import newSecurityManager
admin = app.acl_users.getUserById('admin')
admin = admin.__of__(app.acl_users)
newSecurityManager(None, admin)
from Testing import makerequest
req=makerequest.makerequest(app.mysite.bad_folder)
req.manage_delObjects(['bad_folder'])
import transaction
transaction.commit()
</pre><br />
If that worked and your stuff now loads - lucky you. If all you got was another pos key error, try to delete the parent. If the deletion strategy doesn't work, the next strategy is to fill the void of that object with a fake one, and then delete it. For example:<br />
<br />
<pre>app.mysite._setOb('bad_folder', newFolderInstance)
</pre><br />
If neither of the above work, you are probably looking at a connection Key Error. Feel free to try <a href="http://www.mail-archive.com/zodb-dev@zope.org/msg02175.html">what is described here</a>, but I have no idea what that code looks like so I won't mention it. Instead let's leave the zope environment and go directly to <a href="http://www.zope.org/Documentation/Articles/ZODB1">manipulating the ZODB</a> by hand.<br />
<br />
In the python shell, we can create a connection to the database and manually inspect the objects. We can try manually delete there, and go back in time transaction by transaction. You are probably going to have to poke (use dir!) around so I'm just going to give some getting started code. To manually open the db:<br />
<br />
<pre>from ZODB import FileStorage, DB
from ZODB.utils import u64
storage = FileStorage.FileStorage('Data.fs')</pre><pre> </pre><pre># don't use this code if you don't have to, especially</pre><pre># if you are having a connection error. You'll know because</pre><pre># app won't let you list its objects without throwing</pre><pre># the db_connection error
</pre><pre>db = DB(storage)
connection = db.open()
root = connection.root()
app = root['Application']
</pre><br />
To view the contents of 'app', you must use list to get your keys in sanely readable form:<br />
<br />
<pre>list(app.keys()) # returns child nodes of the app
</pre><br />
If at any point listing the keys fails, you are getting warmer to finding the broken object. Try to delete things if possible, remembering to commit the transaction. Sometimes thats enough to get by. Also note that since you are no longer in a zope environment, all of the objects will be listed as "broken". Don't get obsessed with this - its just because you don't have the proper modules loaded. These are not your key error items and don't get obsessed with fixing them unless you need to traverse in.<br />
<br />
Another route to try is undoing transactions one at a time. Since its <a href="http://docs.zope.org/zodb/zodbguide/transactions.html#undoing-changes">perfectly explained in the docs</a>, I won't replicate it but this can do wonders. Remember to commit frequently and take the time to check all the time.<br />
<br />
If you are trying to find a lost object, take advantage of the iterators in zodb. For example, to iterate through the pickle of each object:<br />
<br />
<pre>it = storage.iterator()
found = False
while not found:
try:
item = it.next().next()
data = item.data
poo = "%s"%data
if poo.count('broken_object_id'):
print "OID: ", u64(item.oid)
print "TID: ", u64(item.tid)
maybeThisIsWhatIWant = root._p_jar[item.oid]
print maybeThisIsWhatIWant
</pre><br />
If your db screwed the pooch and lost a references, maybe you can use this id to map it back. At this point the options are endless and you will have to be creative so use a copy, take a risk, and jump in to coding with ZODB. Remember: the goal is to delete or restore the reference. <br />
<br />
If something looks incorrect, there is a chance it is since this material exists only in the bermuda triangle so please leave a comment and I'll make sure to update or clarify.<br />
<br />
<span style="font-size: large;">Prevention</span><br />
<span style="font-size: small;">With multi-reference</span>, the best thing to do is to NOT cut and paste between mounts. I know its easier said than done. Just remember, copy-paste-delete. In a <a href="http://groups.google.com/group/plonechix/browse_thread/thread/217c71fe849301b4#">followup discussion</a>, @davisagli mentioned that in ZODB 3.9 you can disallow cross-references by setting the allow-implicit-cross-referenced. If you're looking into mounting and you have a newer version of plone I would enable that by default.<br />
<br />
[UPDATE] I have reason to believe that the key error from mounting can come from a multi zope setup where the config files are not lined up with each other. I've been able to recreate this on my local server where 2 zopes share zodbs, including the main db, but where one zope has another mount configured. Going to the manage screen and viewing the mount form in each instance show different results but there is also a transaction that occurs in the background, which you can see in the undo log. After just listing the mount points in each instance, the error triggers on the other instance. The simple fix to this is to undo the last transaction, which is labeled "addMountForm". I think anyways...<br />
<br />
Happy Hunting Ploners!<span style="font-size: large;"><span style="font-size: small;"> <br />
</span></span><br />
<span style="font-size: large;"><span style="font-size: small;"> </span> <br />
</span><br />
<span style="font-size: large;">Other Useful Links</span><br />
<ul><li><a href="http://david.wglick.org/2009/recombining-zodb-storages/">Recombining ZODB Storages</a> </li>
</ul><br />
* No DBs were harmed in the generation of these traces. They are 100% real and, yes, you are allowed to feel empathetic to my pain of having to deal with all of them. These are just a few of my favorites. Some day I'll put together a "best-of" compilation, under the influence of course.<br />
* I don't think this applies to Python 3 because it handles unicode differently. I think @davisagli mentioned that but I can't recall at the moment why.eleddyhttp://www.blogger.com/profile/12829844320869028742noreply@blogger.com7